GRIB2: the good, the bad, and the ugly

Jennifer Adams jma at COLA.IGES.ORG
Wed Oct 10 13:47:14 EDT 2007 

On Oct 10, 2007, at 1:23 PM, Wesley Ebisuzaki wrote:

> Jennifer,
>
> Jennifer Adams wrote:
>>
>> That's a very good point. But is there a quick and easy way to change
>> the packing technique? You'd probably have to write out a new file to
>> do that. And if you have to write out a new file, why not just create
>> a binary file instead?
> But grib2 complex packing can give very good compression with roughly
> the same decode time as grib1.
>>
>> Inside GrADS, grib2 is handled like a completely new data format --
>> borrowing from grib1, but at the I/O level it is completely
>> separate. GrADS relies heavily on g2clib, which does all the
>> unpacking
>> and expanding for us, regardless of what packing technique is used.
> g2clib will handle JPEG2000 and complex packing automatically, so you
> don't have to change your code.

Right -- I'm not going to change the GrADS code -- what I want to
find out Is if g2clib routines (called by GrADS) will return a grid
faster when it has complex packing compared to JPEG compression.

>> I would like to find out whether it would make a difference to GrADS
>> if the grib2 data were not compressed. If changing the packing from
>> JPEG to noJPEG doesn't improve the performance ... then we would have
>> to tweak the grib2 interface for optimization down the road.
>>
>> Are there any grib2 files being distributed to the public that don't
>> use JPEG compression?
> You can use
>
>    cnvgrib -p31 -g22 old.grb2 new.grb2
>
> I works on our AIX.  May have issues on non-AIX machines.
> If you need, I can send you a test file.

I will ask my fortran guru to build cnvgrib for me and will convert a
GFS 0.5-degree forecast and see how it compares.

Jennifer


>
>          Wesley
>
>>
>> Jennifer
>>
>> On Oct 10, 2007, at 9:56 AM, Wesley Ebisuzaki wrote:
>>
>>> Jennifer,
>>>
>>>     The NCEP grib2 files are packed using JPEG2000 compression.
>>> This
>>> provides very good compression
>>> at the expense of being much slower to decode.  If you go to complex
>>> packing with (1st or 2nd order
>>> differences)  a sample file grows from 25MB to 30MB; however, the
>>> decode
>>> time decreases from 30 seconds
>>> to 12 seconds (1 second more than grib1).   For reasonable
>>> performance,
>>> you may change the packing.
>>
>>
>>
>>>
>>>      Wesley
>>>
>>> Jennifer Adams wrote:
>>>>
>>>> The size of grib2 files is impressive given what's packed inside.
>>>> However, once you bring those files to your local disk lickety-
>>>> split
>>>> there will be some drawbacks when it comes to doing the I/O.
>>>> Because
>>>> the grids are compressed, you need to unpack and expand them before
>>>> you can read any data. In GrADS terms, that means you must read the
>>>> entire lat/lon grid into memory before you can pick out a single
>>>> grid
>>>> point. For displays in the X and Y dimensions, there is no evident
>>>> performance hit. For displays that have Z or T varying, it means
>>>> you
>>>> must read grids at all levels or times before drawing the plot
>>>> -- this
>>>> is bad, but tolerable, and once you've got those grids in
>>>> memory, the
>>>> 2nd plot you draw is very fast. The ugly part is when you want
>>>> to draw
>>>> a Z-T plot -- each point in the Z-T domain requires a global
>>>> grid read
>>>> into memory. This is where you make up for the time saved when you
>>>> downloaded the files. There is no such thing as a free lunch --
>>>> high
>>>> resolution data gobbles resources no matter what data format you
>>>> use.
>>>>
>>>> By the way, the above is based on two days of testing -- Monday was
>>>> the first day that the GRIB2 interface in GrADS 2.0 was working. I
>>>> still have a lot of testing to do, but am working hard to get a
>>>> source
>>>> tar file out there for alpha testing as soon as I can.
>>>>
>>>> Jennifer
>>>>
>>>>
>>>>
>>>> On Oct 10, 2007, at 4:04 AM, Bernd Becker wrote:
>>>>
>>>>> Wesley,
>>>>>
>>>>>
>>>>> Hmmm,  assume you maintain a forecast  archive
>>>>> and from one day to the next, you switch from grib 1 data to
>>>>> grib 2
>>>>> data. Each grib  file has its meta data stored separately in one
>>>>> ctl and
>>>>> idx file each.
>>>>> Is it  then possible to read the whole archive in one grads job,
>>>>> despite the mix of grib1 and grib 2 data?
>>>>> (when) is grads able to read grib2 data?
>>>>
>>>>>
>>>>> Many thanks,
>>>>> Bernd.
>>>>>
>>>>> On Tue, 2007-10-09 at 13:55 -0400, Wesley Ebisuzaki wrote:
>>>>>> Bernd,
>>>>>>
>>>>>>      IMHO grib1 is doomed as grib2's compression is too good
>>>>>> (~1/5
>>>>>> of grib1
>>>>>> size for operational model files)  and compression will increase
>>>>>> as the
>>>>>> world goes
>>>>>> to higher resolution files.  Since grib1 is 1/3 the size of a
>>>>>> netcdf
>>>>>> file, you will
>>>>>> see pretty big (uncompressed) netcdf files.   (Your 4x
>>>>>> expansion is
>>>>>> a big
>>>>>> underestimate.)
>>>>>>
>>>>>>        Wesley
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Bernd Becker wrote:
>>>>>>> Wesley,
>>>>>>>
>>>>>>> see below:
>>>>>>>
>>>>>>> On Tue, 2007-10-09 at 10:33 -0400, Wesley Ebisuzaki wrote:
>>>>>>>
>>>>>>>> Patrick,
>>>>>>>>
>>>>>>>> Patrick Reuter wrote:
>>>>>>>>
>>>>>>>>> Dear all,
>>>>>>>>>
>>>>>>>>> I download grib2 files and have problems to degrib into Grads.
>>>>>>>>>
>>>>>>>>> I have the following inventory :
>>>>>>>>>
>>>>>>>>> 1:115:d=2007100706:HGT:850 mb:15 hour fcst
>>>>>>>>> 2:227347:d=2007100706:TMP:850 mb:15 hour fcst
>>>>>>>>> 3:315793:d=2007100706:RH:850 mb:15 hour fcst
>>>>>>>>> 4.1:411471:d=2007100706:UGRD:850 mb:15 hour fcst
>>>>>>>>> 4.2:411471:d=2007100706:VGRD:850 mb:15 hour fcst
>>>>>>>>>
>>>>>>>>> And I want to extract BOTH the UGRD/VGRD values. Here are
>>>>>>>>> my degrib
>>>>>>>>> commands :
>>>>>>>>>
>>>>>>>>> ./degrib grib2file -C -msg 4.1 -Flt -Interp -GrADS -out UGRD
>>>>>>>>> ./degrib grib2file -C -msg 4.2 -Flt -Interp -GrADS -out VGRD
>>>>>>>>>
>>>>>>>>>
>>>>>>>> The notation 4.1 (message #4, submessage #1) is unique to
>>>>>>>> wgrib2.  You
>>>>>>>> should consult the degrib documentation to see how they handle
>>>>>>>> submessages.
>>>>>>>>
>>>>>>>> If all else fails, you can also use wgrib2 to convert
>>>>>>>> submessages to
>>>>>>>> messages by
>>>>>>>>
>>>>>>>>           wgrib2 -grib new_grib old_grib
>>>>>>>>
>>>>>>>>
>>>>>>>>> The first command works fine, but in the second command, I
>>>>>>>>> always get
>>>>>>>>> the following error message :
>>>>>>>>>
>>>>>>>>> ERROR: In call to Grib2Convert.
>>>>>>>>> ERROR: In call to ReadGrib2Record.
>>>>>>>>> ERROR: Unpack library error code (0 2)
>>>>>>>>>
>>>>>>>>> In other words, since the VGRD data is always stored in GRIB2
>>>>>>>>> submessages, I can't access to it.
>>>>>>>>>
>>>>>>>>> Is anybody familiar with my problem ?
>>>>>>>>>
>>>>>>>>> Thanks in advance
>>>>>>>>>
>>>>>>>>>   Patrick
>>>>>>>>>
>>>>>>>> BTW if the grib2 files are lat-lon/mercator/gaussian-grid
>>>>>>>> then you
>>>>>>>> should
>>>>>>>> consider converting the files into netcdf using wgrib2.  The
>>>>>>>> netcdf files
>>>>>>>> can be opened by GrADS using the sdfopen command.  I believe
>>>>>>>> that
>>>>>>>> degrib can produce netcdf files too.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> Is netcdf the death-knell for GRIB1 and 2 data on a global
>>>>>>> scale?
>>>>>>> I suppose the NEtcdf files will be about 4 times larger  on
>>>>>>> average.
>>>>>>> This would increase I/O times and storage requirements.
>>>>>>>
>>>>>>> When will NetCdf use compression?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Bernd.
>>>>>>>
>>>>>>>
>>>>>>>>    Wesley
>>>>>>>>
>>>>>>> --
>>>>>>> Bernd Becker   The Monthly Outlook
>>>>>>> Met Office  FitzRoy Road  Exeter   Devon EX1 3PB  United Kingdom
>>>>>>> Tel.: +44 (0) 1392 884511 Fax: +44 (0)870 900 5050
>>>>>>> E-mail:bernd.becker at metoffice.com
>>>>>>> <mailto:bernd.becker at metoffice.com>
>>>>>>> <mailto:bernd.becker at metoffice.com>  -  http://www.metoffice.com
>>>>>>>
>>>>> --
>>>>> Bernd Becker   The Monthly Outlook
>>>>> Met Office  FitzRoy Road  Exeter   Devon EX1 3PB  United Kingdom
>>>>> Tel.: +44 (0) 1392 884511 Fax: +44 (0)870 900 5050
>>>>> E-mail:bernd.becker at metoffice.com
>>>>> <mailto:bernd.becker at metoffice.com>
>>>>> <mailto:bernd.becker at metoffice.com>  -  http://www.metoffice.com
>>>>
>>>> --
>>>> Jennifer M. Adams
>>>> IGES/COLA
>>>> 4041 Powder Mill Road, Suite 302
>>>> Calverton, MD 20705
>>>> jma at cola.iges.org <mailto:jma at cola.iges.org>
>>>>
>>>>
>>>>
>>
>> --
>> Jennifer M. Adams
>> IGES/COLA
>> 4041 Powder Mill Road, Suite 302
>> Calverton, MD 20705
>> jma at cola.iges.org <mailto:jma at cola.iges.org>
>>
>>
>>

--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma at cola.iges.org



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20071010/0c8a774c/attachment.html

More information about the gradsusr mailing list