GRIB2: the good, the bad, and the ugly

Wed Oct 10 12:11:23 EDT 2007

That's a very good point. But is there a quick and easy way to change
the packing technique? You'd probably have to write out a new file to
do that. And if you have to write out a new file, why not just create
a binary file instead?

Inside GrADS, grib2 is handled like a completely new data format --
borrowing from grib1, but at the I/O level it is completely separate.
GrADS relies heavily on g2clib, which does all the unpacking and
expanding for us, regardless of what packing technique is used. I
would like to find out whether it would make a difference to GrADS if
the grib2 data were not compressed. If changing the packing from JPEG
to noJPEG doesn't improve the performance ... then we would have to
tweak the grib2 interface for optimization down the road.

Are there any grib2 files being distributed to the public that don't
use JPEG compression?

Jennifer

On Oct 10, 2007, at 9:56 AM, Wesley Ebisuzaki wrote:

> Jennifer,
>
>     The NCEP grib2 files are packed using JPEG2000 compression.  This
> provides very good compression
> at the expense of being much slower to decode.  If you go to complex
> packing with (1st or 2nd order
> differences)  a sample file grows from 25MB to 30MB; however, the
> decode
> time decreases from 30 seconds
> to 12 seconds (1 second more than grib1).   For reasonable
> performance,
> you may change the packing.

>
>      Wesley
>
> Jennifer Adams wrote:
>>
>> The size of grib2 files is impressive given what's packed inside.
>> However, once you bring those files to your local disk lickety-split
>> there will be some drawbacks when it comes to doing the I/O. Because
>> the grids are compressed, you need to unpack and expand them before
>> you can read any data. In GrADS terms, that means you must read the
>> entire lat/lon grid into memory before you can pick out a single grid
>> point. For displays in the X and Y dimensions, there is no evident
>> performance hit. For displays that have Z or T varying, it means you
>> must read grids at all levels or times before drawing the plot --
>> this
>> is bad, but tolerable, and once you've got those grids in memory, the
>> 2nd plot you draw is very fast. The ugly part is when you want to
>> draw
>> a Z-T plot -- each point in the Z-T domain requires a global grid
>> read
>> into memory. This is where you make up for the time saved when you
>> downloaded the files. There is no such thing as a free lunch -- high
>> resolution data gobbles resources no matter what data format you use.
>>
>> By the way, the above is based on two days of testing -- Monday was
>> the first day that the GRIB2 interface in GrADS 2.0 was working. I
>> still have a lot of testing to do, but am working hard to get a
>> source
>> tar file out there for alpha testing as soon as I can.
>>
>> Jennifer
>>
>>
>>
>> On Oct 10, 2007, at 4:04 AM, Bernd Becker wrote:
>>
>>> Wesley,
>>>
>>>
>>> Hmmm,  assume you maintain a forecast  archive
>>> and from one day to the next, you switch from grib 1 data to grib 2
>>> data. Each grib  file has its meta data stored separately in one
>>> ctl and
>>> idx file each.
>>> Is it  then possible to read the whole archive in one grads job,
>>> despite the mix of grib1 and grib 2 data?
>>> (when) is grads able to read grib2 data?
>>
>>>
>>> Many thanks,
>>> Bernd.
>>>
>>> On Tue, 2007-10-09 at 13:55 -0400, Wesley Ebisuzaki wrote:
>>>> Bernd,
>>>>
>>>>      IMHO grib1 is doomed as grib2's compression is too good (~1/5
>>>> of grib1
>>>> size for operational model files)  and compression will increase
>>>> as the
>>>> world goes
>>>> to higher resolution files.  Since grib1 is 1/3 the size of a
>>>> netcdf
>>>> file, you will
>>>> see pretty big (uncompressed) netcdf files.   (Your 4x expansion is
>>>> a big
>>>> underestimate.)
>>>>
>>>>        Wesley
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Bernd Becker wrote:
>>>>> Wesley,
>>>>>
>>>>> see below:
>>>>>
>>>>> On Tue, 2007-10-09 at 10:33 -0400, Wesley Ebisuzaki wrote:
>>>>>
>>>>>> Patrick,
>>>>>>
>>>>>> Patrick Reuter wrote:
>>>>>>
>>>>>>> Dear all,
>>>>>>>
>>>>>>> I download grib2 files and have problems to degrib into Grads.
>>>>>>>
>>>>>>> I have the following inventory :
>>>>>>>
>>>>>>> 1:115:d=2007100706:HGT:850 mb:15 hour fcst
>>>>>>> 2:227347:d=2007100706:TMP:850 mb:15 hour fcst
>>>>>>> 3:315793:d=2007100706:RH:850 mb:15 hour fcst
>>>>>>> 4.1:411471:d=2007100706:UGRD:850 mb:15 hour fcst
>>>>>>> 4.2:411471:d=2007100706:VGRD:850 mb:15 hour fcst
>>>>>>>
>>>>>>> And I want to extract BOTH the UGRD/VGRD values. Here are my
>>>>>>> degrib
>>>>>>> commands :
>>>>>>>
>>>>>>> ./degrib grib2file -C -msg 4.1 -Flt -Interp -GrADS -out UGRD
>>>>>>> ./degrib grib2file -C -msg 4.2 -Flt -Interp -GrADS -out VGRD
>>>>>>>
>>>>>>>
>>>>>> The notation 4.1 (message #4, submessage #1) is unique to
>>>>>> wgrib2.  You
>>>>>> should consult the degrib documentation to see how they handle
>>>>>> submessages.
>>>>>>
>>>>>> If all else fails, you can also use wgrib2 to convert
>>>>>> submessages to
>>>>>> messages by
>>>>>>
>>>>>>           wgrib2 -grib new_grib old_grib
>>>>>>
>>>>>>
>>>>>>> The first command works fine, but in the second command, I
>>>>>>> always get
>>>>>>> the following error message :
>>>>>>>
>>>>>>> ERROR: In call to Grib2Convert.
>>>>>>> ERROR: In call to ReadGrib2Record.
>>>>>>> ERROR: Unpack library error code (0 2)
>>>>>>>
>>>>>>> In other words, since the VGRD data is always stored in GRIB2
>>>>>>> submessages, I can't access to it.
>>>>>>>
>>>>>>> Is anybody familiar with my problem ?
>>>>>>>
>>>>>>> Thanks in advance
>>>>>>>
>>>>>>>   Patrick
>>>>>>>
>>>>>> BTW if the grib2 files are lat-lon/mercator/gaussian-grid then
>>>>>> you
>>>>>> should
>>>>>> consider converting the files into netcdf using wgrib2.  The
>>>>>> netcdf files
>>>>>> can be opened by GrADS using the sdfopen command.  I believe that
>>>>>> degrib can produce netcdf files too.
>>>>>>
>>>>>>
>>>>>
>>>>> Is netcdf the death-knell for GRIB1 and 2 data on a global scale?
>>>>> I suppose the NEtcdf files will be about 4 times larger  on
>>>>> average.
>>>>> This would increase I/O times and storage requirements.
>>>>>
>>>>> When will NetCdf use compression?
>>>>>
>>>>> Thanks,
>>>>> Bernd.
>>>>>
>>>>>
>>>>>>    Wesley
>>>>>>
>>>>> --
>>>>> Bernd Becker   The Monthly Outlook
>>>>> Met Office  FitzRoy Road  Exeter   Devon EX1 3PB  United Kingdom
>>>>> Tel.: +44 (0) 1392 884511 Fax: +44 (0)870 900 5050
>>>>> E-mail:bernd.becker at metoffice.com
>>>>> <mailto:bernd.becker at metoffice.com>  -  http://www.metoffice.com
>>>>>
>>> --
>>> Bernd Becker   The Monthly Outlook
>>> Met Office  FitzRoy Road  Exeter   Devon EX1 3PB  United Kingdom
>>> Tel.: +44 (0) 1392 884511 Fax: +44 (0)870 900 5050
>>> E-mail:bernd.becker at metoffice.com
>>> <mailto:bernd.becker at metoffice.com>  -  http://www.metoffice.com
>>
>> --
>> Jennifer M. Adams
>> IGES/COLA
>> 4041 Powder Mill Road, Suite 302
>> Calverton, MD 20705
>> jma at cola.iges.org <mailto:jma at cola.iges.org>
>>
>>
>>

--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma at cola.iges.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20071010/f2eaf40c/attachment.html 