GRIB2: the good, the bad, and the ugly

Wed Oct 10 13:23:49 EDT 2007

Jennifer,

Jennifer Adams wrote:
>
> That's a very good point. But is there a quick and easy way to change
> the packing technique? You'd probably have to write out a new file to
> do that. And if you have to write out a new file, why not just create
> a binary file instead?
But grib2 complex packing can give very good compression with roughly
the same decode time as grib1.
>
> Inside GrADS, grib2 is handled like a completely new data format --
> borrowing from grib1, but at the I/O level it is completely
> separate. GrADS relies heavily on g2clib, which does all the unpacking
> and expanding for us, regardless of what packing technique is used.
g2clib will handle JPEG2000 and complex packing automatically, so you
don't have to change your code.

> I would like to find out whether it would make a difference to GrADS
> if the grib2 data were not compressed. If changing the packing from
> JPEG to noJPEG doesn't improve the performance ... then we would have
> to tweak the grib2 interface for optimization down the road.
>
> Are there any grib2 files being distributed to the public that don't
> use JPEG compression?
You can use

    cnvgrib -p31 -g22 old.grb2 new.grb2

I works on our AIX.  May have issues on non-AIX machines.
If you need, I can send you a test file.

          Wesley

>
> Jennifer
>
> On Oct 10, 2007, at 9:56 AM, Wesley Ebisuzaki wrote:
>
>> Jennifer,
>>
>>     The NCEP grib2 files are packed using JPEG2000 compression.  This
>> provides very good compression
>> at the expense of being much slower to decode.  If you go to complex
>> packing with (1st or 2nd order
>> differences)  a sample file grows from 25MB to 30MB; however, the decode
>> time decreases from 30 seconds
>> to 12 seconds (1 second more than grib1).   For reasonable performance,
>> you may change the packing.
>
>
>
>>
>>      Wesley
>>
>> Jennifer Adams wrote:
>>>
>>> The size of grib2 files is impressive given what's packed inside.
>>> However, once you bring those files to your local disk lickety-split
>>> there will be some drawbacks when it comes to doing the I/O. Because
>>> the grids are compressed, you need to unpack and expand them before
>>> you can read any data. In GrADS terms, that means you must read the
>>> entire lat/lon grid into memory before you can pick out a single grid
>>> point. For displays in the X and Y dimensions, there is no evident
>>> performance hit. For displays that have Z or T varying, it means you
>>> must read grids at all levels or times before drawing the plot -- this
>>> is bad, but tolerable, and once you've got those grids in memory, the
>>> 2nd plot you draw is very fast. The ugly part is when you want to draw
>>> a Z-T plot -- each point in the Z-T domain requires a global grid read
>>> into memory. This is where you make up for the time saved when you
>>> downloaded the files. There is no such thing as a free lunch -- high
>>> resolution data gobbles resources no matter what data format you use.
>>>
>>> By the way, the above is based on two days of testing -- Monday was
>>> the first day that the GRIB2 interface in GrADS 2.0 was working. I
>>> still have a lot of testing to do, but am working hard to get a source
>>> tar file out there for alpha testing as soon as I can.
>>>
>>> Jennifer
>>>
>>>
>>>
>>> On Oct 10, 2007, at 4:04 AM, Bernd Becker wrote:
>>>
>>>> Wesley,
>>>>
>>>>
>>>> Hmmm,  assume you maintain a forecast  archive
>>>> and from one day to the next, you switch from grib 1 data to grib 2
>>>> data. Each grib  file has its meta data stored separately in one
>>>> ctl and
>>>> idx file each.
>>>> Is it  then possible to read the whole archive in one grads job,
>>>> despite the mix of grib1 and grib 2 data?
>>>> (when) is grads able to read grib2 data?
>>>
>>>>
>>>> Many thanks,
>>>> Bernd.
>>>>
>>>> On Tue, 2007-10-09 at 13:55 -0400, Wesley Ebisuzaki wrote:
>>>>> Bernd,
>>>>>
>>>>>      IMHO grib1 is doomed as grib2's compression is too good (~1/5
>>>>> of grib1
>>>>> size for operational model files)  and compression will increase
>>>>> as the
>>>>> world goes
>>>>> to higher resolution files.  Since grib1 is 1/3 the size of a netcdf
>>>>> file, you will
>>>>> see pretty big (uncompressed) netcdf files.   (Your 4x expansion is
>>>>> a big
>>>>> underestimate.)
>>>>>
>>>>>        Wesley
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Bernd Becker wrote:
>>>>>> Wesley,
>>>>>>
>>>>>> see below:
>>>>>>
>>>>>> On Tue, 2007-10-09 at 10:33 -0400, Wesley Ebisuzaki wrote:
>>>>>>
>>>>>>> Patrick,
>>>>>>>
>>>>>>> Patrick Reuter wrote:
>>>>>>>
>>>>>>>> Dear all,
>>>>>>>>
>>>>>>>> I download grib2 files and have problems to degrib into Grads.
>>>>>>>>
>>>>>>>> I have the following inventory :
>>>>>>>>
>>>>>>>> 1:115:d=2007100706:HGT:850 mb:15 hour fcst
>>>>>>>> 2:227347:d=2007100706:TMP:850 mb:15 hour fcst
>>>>>>>> 3:315793:d=2007100706:RH:850 mb:15 hour fcst
>>>>>>>> 4.1:411471:d=2007100706:UGRD:850 mb:15 hour fcst
>>>>>>>> 4.2:411471:d=2007100706:VGRD:850 mb:15 hour fcst
>>>>>>>>
>>>>>>>> And I want to extract BOTH the UGRD/VGRD values. Here are my degrib
>>>>>>>> commands :
>>>>>>>>
>>>>>>>> ./degrib grib2file -C -msg 4.1 -Flt -Interp -GrADS -out UGRD
>>>>>>>> ./degrib grib2file -C -msg 4.2 -Flt -Interp -GrADS -out VGRD
>>>>>>>>
>>>>>>>>
>>>>>>> The notation 4.1 (message #4, submessage #1) is unique to
>>>>>>> wgrib2.  You
>>>>>>> should consult the degrib documentation to see how they handle
>>>>>>> submessages.
>>>>>>>
>>>>>>> If all else fails, you can also use wgrib2 to convert submessages to
>>>>>>> messages by
>>>>>>>
>>>>>>>           wgrib2 -grib new_grib old_grib
>>>>>>>
>>>>>>>
>>>>>>>> The first command works fine, but in the second command, I
>>>>>>>> always get
>>>>>>>> the following error message :
>>>>>>>>
>>>>>>>> ERROR: In call to Grib2Convert.
>>>>>>>> ERROR: In call to ReadGrib2Record.
>>>>>>>> ERROR: Unpack library error code (0 2)
>>>>>>>>
>>>>>>>> In other words, since the VGRD data is always stored in GRIB2
>>>>>>>> submessages, I can't access to it.
>>>>>>>>
>>>>>>>> Is anybody familiar with my problem ?
>>>>>>>>
>>>>>>>> Thanks in advance
>>>>>>>>
>>>>>>>>   Patrick
>>>>>>>>
>>>>>>> BTW if the grib2 files are lat-lon/mercator/gaussian-grid then you
>>>>>>> should
>>>>>>> consider converting the files into netcdf using wgrib2.  The
>>>>>>> netcdf files
>>>>>>> can be opened by GrADS using the sdfopen command.  I believe that
>>>>>>> degrib can produce netcdf files too.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Is netcdf the death-knell for GRIB1 and 2 data on a global scale?
>>>>>> I suppose the NEtcdf files will be about 4 times larger  on average.
>>>>>> This would increase I/O times and storage requirements.
>>>>>>
>>>>>> When will NetCdf use compression?
>>>>>>
>>>>>> Thanks,
>>>>>> Bernd.
>>>>>>
>>>>>>
>>>>>>>    Wesley
>>>>>>>
>>>>>> --
>>>>>> Bernd Becker   The Monthly Outlook
>>>>>> Met Office  FitzRoy Road  Exeter   Devon EX1 3PB  United Kingdom
>>>>>> Tel.: +44 (0) 1392 884511 Fax: +44 (0)870 900 5050
>>>>>> E-mail:bernd.becker at metoffice.com <mailto:bernd.becker at metoffice.com>
>>>>>> <mailto:bernd.becker at metoffice.com>  -  http://www.metoffice.com
>>>>>>
>>>> --
>>>> Bernd Becker   The Monthly Outlook
>>>> Met Office  FitzRoy Road  Exeter   Devon EX1 3PB  United Kingdom
>>>> Tel.: +44 (0) 1392 884511 Fax: +44 (0)870 900 5050
>>>> E-mail:bernd.becker at metoffice.com <mailto:bernd.becker at metoffice.com>
>>>> <mailto:bernd.becker at metoffice.com>  -  http://www.metoffice.com
>>>
>>> --
>>> Jennifer M. Adams
>>> IGES/COLA
>>> 4041 Powder Mill Road, Suite 302
>>> Calverton, MD 20705
>>> jma at cola.iges.org <mailto:jma at cola.iges.org>
>>>
>>>
>>>
>
> --
> Jennifer M. Adams
> IGES/COLA
> 4041 Powder Mill Road, Suite 302
> Calverton, MD 20705
> jma at cola.iges.org <mailto:jma at cola.iges.org>
>
>
>