GRIB2: the good, the bad, and the ugly

Wed Oct 10 09:56:50 EDT 2007

Jennifer,

     The NCEP grib2 files are packed using JPEG2000 compression.  This
provides very good compression
at the expense of being much slower to decode.  If you go to complex
packing with (1st or 2nd order
differences)  a sample file grows from 25MB to 30MB; however, the decode
time decreases from 30 seconds
to 12 seconds (1 second more than grib1).   For reasonable performance,
you may change the packing.

      Wesley

Jennifer Adams wrote:
>
> The size of grib2 files is impressive given what's packed inside.
> However, once you bring those files to your local disk lickety-split
> there will be some drawbacks when it comes to doing the I/O. Because
> the grids are compressed, you need to unpack and expand them before
> you can read any data. In GrADS terms, that means you must read the
> entire lat/lon grid into memory before you can pick out a single grid
> point. For displays in the X and Y dimensions, there is no evident
> performance hit. For displays that have Z or T varying, it means you
> must read grids at all levels or times before drawing the plot -- this
> is bad, but tolerable, and once you've got those grids in memory, the
> 2nd plot you draw is very fast. The ugly part is when you want to draw
> a Z-T plot -- each point in the Z-T domain requires a global grid read
> into memory. This is where you make up for the time saved when you
> downloaded the files. There is no such thing as a free lunch -- high
> resolution data gobbles resources no matter what data format you use.
>
> By the way, the above is based on two days of testing -- Monday was
> the first day that the GRIB2 interface in GrADS 2.0 was working. I
> still have a lot of testing to do, but am working hard to get a source
> tar file out there for alpha testing as soon as I can.
>
> Jennifer
>
>
>
> On Oct 10, 2007, at 4:04 AM, Bernd Becker wrote:
>
>> Wesley,
>>
>>
>> Hmmm,  assume you maintain a forecast  archive
>> and from one day to the next, you switch from grib 1 data to grib 2
>> data. Each grib  file has its meta data stored separately in one ctl and
>> idx file each.
>> Is it  then possible to read the whole archive in one grads job,
>> despite the mix of grib1 and grib 2 data?
>> (when) is grads able to read grib2 data?
>
>>
>> Many thanks,
>> Bernd.
>>
>> On Tue, 2007-10-09 at 13:55 -0400, Wesley Ebisuzaki wrote:
>>> Bernd,
>>>
>>>      IMHO grib1 is doomed as grib2's compression is too good (~1/5
>>> of grib1
>>> size for operational model files)  and compression will increase as the
>>> world goes
>>> to higher resolution files.  Since grib1 is 1/3 the size of a netcdf
>>> file, you will
>>> see pretty big (uncompressed) netcdf files.   (Your 4x expansion is
>>> a big
>>> underestimate.)
>>>
>>>        Wesley
>>>
>>>
>>>
>>>
>>>
>>>
>>> Bernd Becker wrote:
>>>> Wesley,
>>>>
>>>> see below:
>>>>
>>>> On Tue, 2007-10-09 at 10:33 -0400, Wesley Ebisuzaki wrote:
>>>>
>>>>> Patrick,
>>>>>
>>>>> Patrick Reuter wrote:
>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> I download grib2 files and have problems to degrib into Grads.
>>>>>>
>>>>>> I have the following inventory :
>>>>>>
>>>>>> 1:115:d=2007100706:HGT:850 mb:15 hour fcst
>>>>>> 2:227347:d=2007100706:TMP:850 mb:15 hour fcst
>>>>>> 3:315793:d=2007100706:RH:850 mb:15 hour fcst
>>>>>> 4.1:411471:d=2007100706:UGRD:850 mb:15 hour fcst
>>>>>> 4.2:411471:d=2007100706:VGRD:850 mb:15 hour fcst
>>>>>>
>>>>>> And I want to extract BOTH the UGRD/VGRD values. Here are my degrib
>>>>>> commands :
>>>>>>
>>>>>> ./degrib grib2file -C -msg 4.1 -Flt -Interp -GrADS -out UGRD
>>>>>> ./degrib grib2file -C -msg 4.2 -Flt -Interp -GrADS -out VGRD
>>>>>>
>>>>>>
>>>>> The notation 4.1 (message #4, submessage #1) is unique to wgrib2.  You
>>>>> should consult the degrib documentation to see how they handle
>>>>> submessages.
>>>>>
>>>>> If all else fails, you can also use wgrib2 to convert submessages to
>>>>> messages by
>>>>>
>>>>>           wgrib2 -grib new_grib old_grib
>>>>>
>>>>>
>>>>>> The first command works fine, but in the second command, I always get
>>>>>> the following error message :
>>>>>>
>>>>>> ERROR: In call to Grib2Convert.
>>>>>> ERROR: In call to ReadGrib2Record.
>>>>>> ERROR: Unpack library error code (0 2)
>>>>>>
>>>>>> In other words, since the VGRD data is always stored in GRIB2
>>>>>> submessages, I can't access to it.
>>>>>>
>>>>>> Is anybody familiar with my problem ?
>>>>>>
>>>>>> Thanks in advance
>>>>>>
>>>>>>   Patrick
>>>>>>
>>>>> BTW if the grib2 files are lat-lon/mercator/gaussian-grid then you
>>>>> should
>>>>> consider converting the files into netcdf using wgrib2.  The
>>>>> netcdf files
>>>>> can be opened by GrADS using the sdfopen command.  I believe that
>>>>> degrib can produce netcdf files too.
>>>>>
>>>>>
>>>>
>>>> Is netcdf the death-knell for GRIB1 and 2 data on a global scale?
>>>> I suppose the NEtcdf files will be about 4 times larger  on average.
>>>> This would increase I/O times and storage requirements.
>>>>
>>>> When will NetCdf use compression?
>>>>
>>>> Thanks,
>>>> Bernd.
>>>>
>>>>
>>>>>    Wesley
>>>>>
>>>> --
>>>> Bernd Becker   The Monthly Outlook
>>>> Met Office  FitzRoy Road  Exeter   Devon EX1 3PB  United Kingdom
>>>> Tel.: +44 (0) 1392 884511 Fax: +44 (0)870 900 5050
>>>> E-mail:bernd.becker at metoffice.com
>>>> <mailto:bernd.becker at metoffice.com>  -  http://www.metoffice.com
>>>>
>> --
>> Bernd Becker   The Monthly Outlook
>> Met Office  FitzRoy Road  Exeter   Devon EX1 3PB  United Kingdom
>> Tel.: +44 (0) 1392 884511 Fax: +44 (0)870 900 5050
>> E-mail:bernd.becker at metoffice.com
>> <mailto:bernd.becker at metoffice.com>  -  http://www.metoffice.com
>
> --
> Jennifer M. Adams
> IGES/COLA
> 4041 Powder Mill Road, Suite 302
> Calverton, MD 20705
> jma at cola.iges.org <mailto:jma at cola.iges.org>
>
>
>