[gradsusr] Faster point output from grb data files

[Sam Wilson]

Hi Wesley,

Thank you for your reply and sorry for the slow response.

I checked the packing like you suggested and the NAM model files I'm using are compressed with jpeg2000, hence the slowness.

I did what you suggested by converting the grb2 files to complex packing, but I'm still having problems with slowness – it didn't seem to buy me much.

I'm thinking that if there was a way I could change the grid used from Lambert Conformal to just a regular lat/lon grid, which would eliminate the need for a PDEF entry in the ctl file, it would speed things up a lot.  I've checked around online, but haven't had any luck.

Do you know if there is a way to do that using lats4d or the re() function? Or anything else?

Thanks again for your help.

Best,
Sam


From: Wesley Ebisuzaki - NOAA Federal <wesley.ebisuzaki at noaa.gov<mailto:wesley.ebisuzaki at noaa.gov>>
Reply-To: GrADS Users Forum <gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>>
Date: Wed, 17 Apr 2013 07:54:29 -0400
To: GrADS Users Forum <gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>>
Subject: Re: [gradsusr] Faster point output from grb data files

Sam,

    Grib2 files compressed with jpeg2000 are slow to read.   You can determine
the type of compressing by using

    wgrib2 IN.grb -packing

You can speed up the reading of jpeg compressed grib2 files by converting
the file to complex packing.

    wgrib2 IN.grb -set_grib_type c3 -grib_out OUT.grb

c3 is good for smooth fields and c1 is good for noisy fields.  Many people
use jpeg compression because it often makes the smallest files.

After converting the file, the index files will have to be remade.

 Wesley



On Tue, Apr 16, 2013 at 7:16 PM, Sam Wilson <sam at surfline.com<mailto:sam at surfline.com>> wrote:
Hi,

I have a set of grads scripts that generate point output of various variables (wind, pressure, etc.) for many locations using different model datasets in grb format.  I'm using the gr2stn function to get the output at the station locations.  Most of my scripts run very quickly and generate data for thousands of locations.  However, some of them are very slow and will take many hours to get output at all the points I want.  The only difference between the scripts are the dataset that they are accessing and using to generate the output.

I'm trying to determine why some of them run very slow and others do not.  I assumed the reason was the size of the grb files being used – if the grb files were very large then it would be slower and vice versa.  As a test, I decreased the number of variables being included in the grb file set that was very large to reduce the size of those grb files.  I then ran the script to generate the point output from the smaller grb files and it didn't seem to buy me much run time.

I did notice that the grb files that are associated with the slow processing have a PDEF entry in the .ctl file, whereas the grb files that process very fast have no PDEF entry.  Could this be the cause?  Does it take a lot longer to get point output from data that has to be mapped to a rectilinear lat/lon grid (has a PDEF entry) versus data that is already on a rectilinear lat/lon grid (no PDEF entry)?  If this is the case, does anyone know of a way around this to get faster point output from grb files?

Thanks for your time and any help you can provide.

Best,
Sam

_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr


_______________________________________________ gradsusr mailing list gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org> http://gradsusr.org/mailman/listinfo/gradsusr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20130425/51f73efa/attachment-0003.html