[gradsusr] Faster point output from grb data files

[Wesley Ebisuzaki - NOAA Federal]

Sam,

    Grib2 files compressed with jpeg2000 are slow to read.   You can
determine
the type of compressing by using

    wgrib2 IN.grb -packing

You can speed up the reading of jpeg compressed grib2 files by converting
the file to complex packing.

    wgrib2 IN.grb -set_grib_type c3 -grib_out OUT.grb

c3 is good for smooth fields and c1 is good for noisy fields.  Many people
use jpeg compression because it often makes the smallest files.

After converting the file, the index files will have to be remade.

 Wesley



On Tue, Apr 16, 2013 at 7:16 PM, Sam Wilson <sam at surfline.com> wrote:

>   Hi,
>
>  I have a set of grads scripts that generate point output of various
> variables (wind, pressure, etc.) for many locations using different model
> datasets in grb format.  I'm using the gr2stn function to get the output at
> the station locations.  Most of my scripts run very quickly and generate
> data for thousands of locations.  However, some of them are very slow and
> will take many hours to get output at all the points I want.  The only
> difference between the scripts are the dataset that they are accessing and
> using to generate the output.
>
>  I'm trying to determine why some of them run very slow and others do
> not.  I assumed the reason was the size of the grb files being used – if
> the grb files were very large then it would be slower and vice versa.  As a
> test, I decreased the number of variables being included in the grb file
> set that was very large to reduce the size of those grb files.  I then ran
> the script to generate the point output from the smaller grb files and it
> didn't seem to buy me much run time.
>
>  I did notice that the grb files that are associated with the slow
> processing have a PDEF entry in the .ctl file, whereas the grb files that
> process very fast have no PDEF entry.  Could this be the cause?  Does it
> take a lot longer to get point output from data that has to be mapped to a
> rectilinear lat/lon grid (has a PDEF entry) versus data that is already
> on a rectilinear lat/lon grid (no PDEF entry)?  If this is the case, does
> anyone know of a way around this to get faster point output from grb files?
>
>  Thanks for your time and any help you can provide.
>
>  Best,
>  Sam
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20130417/ce745e86/attachment-0003.html