GRIB2: the good, the bad, and the ugly

Wed Oct 10 08:57:43 EDT 2007

The size of grib2 files is impressive given what's packed inside.
However, once you bring those files to your local disk lickety-split
there will be some drawbacks when it comes to doing the I/O. Because
the grids are compressed, you need to unpack and expand them before
you can read any data. In GrADS terms, that means you must read the
entire lat/lon grid into memory before you can pick out a single grid
point. For displays in the X and Y dimensions, there is no evident
performance hit. For displays that have Z or T varying, it means you
must read grids at all levels or times before drawing the plot --
this is bad, but tolerable, and once you've got those grids in
memory, the 2nd plot you draw is very fast. The ugly part is when you
want to draw a Z-T plot -- each point in the Z-T domain requires a
global grid read into memory. This is where you make up for the time
saved when you downloaded the files. There is no such thing as a free
lunch -- high resolution data gobbles resources no matter what data
format you use.

By the way, the above is based on two days of testing -- Monday was
the first day that the GRIB2 interface in GrADS 2.0 was working. I
still have a lot of testing to do, but am working hard to get a
source tar file out there for alpha testing as soon as I can.

Jennifer

On Oct 10, 2007, at 4:04 AM, Bernd Becker wrote:

> Wesley,
>
>
> Hmmm,  assume you maintain a forecast  archive
> and from one day to the next, you switch from grib 1 data to grib 2
> data. Each grib  file has its meta data stored separately in one
> ctl and
> idx file each.
> Is it  then possible to read the whole archive in one grads job,
> despite the mix of grib1 and grib 2 data?
> (when) is grads able to read grib2 data?

>
> Many thanks,
> Bernd.
>
> On Tue, 2007-10-09 at 13:55 -0400, Wesley Ebisuzaki wrote:
>> Bernd,
>>
>>      IMHO grib1 is doomed as grib2's compression is too good (~1/5
>> of grib1
>> size for operational model files)  and compression will increase
>> as the
>> world goes
>> to higher resolution files.  Since grib1 is 1/3 the size of a netcdf
>> file, you will
>> see pretty big (uncompressed) netcdf files.   (Your 4x expansion
>> is a big
>> underestimate.)
>>
>>        Wesley
>>
>>
>>
>>
>>
>>
>> Bernd Becker wrote:
>>> Wesley,
>>>
>>> see below:
>>>
>>> On Tue, 2007-10-09 at 10:33 -0400, Wesley Ebisuzaki wrote:
>>>
>>>> Patrick,
>>>>
>>>> Patrick Reuter wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> I download grib2 files and have problems to degrib into Grads.
>>>>>
>>>>> I have the following inventory :
>>>>>
>>>>> 1:115:d=2007100706:HGT:850 mb:15 hour fcst
>>>>> 2:227347:d=2007100706:TMP:850 mb:15 hour fcst
>>>>> 3:315793:d=2007100706:RH:850 mb:15 hour fcst
>>>>> 4.1:411471:d=2007100706:UGRD:850 mb:15 hour fcst
>>>>> 4.2:411471:d=2007100706:VGRD:850 mb:15 hour fcst
>>>>>
>>>>> And I want to extract BOTH the UGRD/VGRD values. Here are my
>>>>> degrib
>>>>> commands :
>>>>>
>>>>> ./degrib grib2file -C -msg 4.1 -Flt -Interp -GrADS -out UGRD
>>>>> ./degrib grib2file -C -msg 4.2 -Flt -Interp -GrADS -out VGRD
>>>>>
>>>>>
>>>> The notation 4.1 (message #4, submessage #1) is unique to
>>>> wgrib2.  You
>>>> should consult the degrib documentation to see how they handle
>>>> submessages.
>>>>
>>>> If all else fails, you can also use wgrib2 to convert
>>>> submessages to
>>>> messages by
>>>>
>>>>           wgrib2 -grib new_grib old_grib
>>>>
>>>>
>>>>> The first command works fine, but in the second command, I
>>>>> always get
>>>>> the following error message :
>>>>>
>>>>> ERROR: In call to Grib2Convert.
>>>>> ERROR: In call to ReadGrib2Record.
>>>>> ERROR: Unpack library error code (0 2)
>>>>>
>>>>> In other words, since the VGRD data is always stored in GRIB2
>>>>> submessages, I can't access to it.
>>>>>
>>>>> Is anybody familiar with my problem ?
>>>>>
>>>>> Thanks in advance
>>>>>
>>>>>   Patrick
>>>>>
>>>> BTW if the grib2 files are lat-lon/mercator/gaussian-grid then
>>>> you should
>>>> consider converting the files into netcdf using wgrib2.  The
>>>> netcdf files
>>>> can be opened by GrADS using the sdfopen command.  I believe that
>>>> degrib can produce netcdf files too.
>>>>
>>>>
>>>
>>> Is netcdf the death-knell for GRIB1 and 2 data on a global scale?
>>> I suppose the NEtcdf files will be about 4 times larger  on average.
>>> This would increase I/O times and storage requirements.
>>>
>>> When will NetCdf use compression?
>>>
>>> Thanks,
>>> Bernd.
>>>
>>>
>>>>    Wesley
>>>>
>>> --
>>> Bernd Becker   The Monthly Outlook
>>> Met Office  FitzRoy Road  Exeter   Devon EX1 3PB  United Kingdom
>>> Tel.: +44 (0) 1392 884511 Fax: +44 (0)870 900 5050
>>> E-mail:bernd.becker at metoffice.com  -  http://www.metoffice.com
>>>
> --
> Bernd Becker   The Monthly Outlook
> Met Office  FitzRoy Road  Exeter   Devon EX1 3PB  United Kingdom
> Tel.: +44 (0) 1392 884511 Fax: +44 (0)870 900 5050
> E-mail:bernd.becker at metoffice.com  -  http://www.metoffice.com

--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma at cola.iges.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20071010/7de44dd5/attachment.html 