[gradsusr] Faster point output from grb data files

Fri Apr 26 08:11:47 EDT 2013

Sam,

  I guess that you didn't see much of a speed up because the data had been
decoded and
then cached.  You can regrid the data using wgrib2.

   wgrib2 IN.grb -set_grib_type c3 -new_grid_winds earth -new_grid latlon
LON0:NLON:DLON  LAT0:NLAT:DLAT OUT.grb

      Wesley

On Thu, Apr 25, 2013 at 5:46 PM, Sam Wilson <sam at surfline.com> wrote:

>   Hi Wesley,
>
>  Thank you for your reply and sorry for the slow response.
>
>  I checked the packing like you suggested and the NAM model files I'm
> using are compressed with jpeg2000, hence the slowness.
>
>  I did what you suggested by converting the grb2 files to complex
> packing, but I'm still having problems with slowness – it didn't seem to
> buy me much.
>
>  I'm thinking that if there was a way I could change the grid used from
> Lambert Conformal to just a regular lat/lon grid, which would eliminate the
> need for a PDEF entry in the ctl file, it would speed things up a lot.
>  I've checked around online, but haven't had any luck.
>
>  Do you know if there is a way to do that using lats4d or the re()
> function? Or anything else?
>
>  Thanks again for your help.
>
>  Best,
> Sam
>
>
>   From: Wesley Ebisuzaki - NOAA Federal <wesley.ebisuzaki at noaa.gov>
> Reply-To: GrADS Users Forum <gradsusr at gradsusr.org>
> Date: Wed, 17 Apr 2013 07:54:29 -0400
> To: GrADS Users Forum <gradsusr at gradsusr.org>
> Subject: Re: [gradsusr] Faster point output from grb data files
>
>   Sam,
>
>      Grib2 files compressed with jpeg2000 are slow to read.   You can
> determine
> the type of compressing by using
>
>      wgrib2 IN.grb -packing
>
>  You can speed up the reading of jpeg compressed grib2 files by converting
> the file to complex packing.
>
>      wgrib2 IN.grb -set_grib_type c3 -grib_out OUT.grb
>
>  c3 is good for smooth fields and c1 is good for noisy fields.  Many
> people
> use jpeg compression because it often makes the smallest files.
>
>  After converting the file, the index files will have to be remade.
>
>   Wesley
>
>
>
> On Tue, Apr 16, 2013 at 7:16 PM, Sam Wilson <sam at surfline.com> wrote:
>
>>   Hi,
>>
>>  I have a set of grads scripts that generate point output of various
>> variables (wind, pressure, etc.) for many locations using different model
>> datasets in grb format.  I'm using the gr2stn function to get the output at
>> the station locations.  Most of my scripts run very quickly and generate
>> data for thousands of locations.  However, some of them are very slow and
>> will take many hours to get output at all the points I want.  The only
>> difference between the scripts are the dataset that they are accessing and
>> using to generate the output.
>>
>>  I'm trying to determine why some of them run very slow and others do
>> not.  I assumed the reason was the size of the grb files being used – if
>> the grb files were very large then it would be slower and vice versa.  As a
>> test, I decreased the number of variables being included in the grb file
>> set that was very large to reduce the size of those grb files.  I then ran
>> the script to generate the point output from the smaller grb files and it
>> didn't seem to buy me much run time.
>>
>>  I did notice that the grb files that are associated with the slow
>> processing have a PDEF entry in the .ctl file, whereas the grb files that
>> process very fast have no PDEF entry.  Could this be the cause?  Does it
>> take a lot longer to get point output from data that has to be mapped to a
>> rectilinear lat/lon grid (has a PDEF entry) versus data that is already
>> on a rectilinear lat/lon grid (no PDEF entry)?  If this is the case, does
>> anyone know of a way around this to get faster point output from grb files?
>>
>>  Thanks for your time and any help you can provide.
>>
>>  Best,
>>  Sam
>>
>> _______________________________________________
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>>
>>
>  _______________________________________________ gradsusr mailing list
> gradsusr at gradsusr.org http://gradsusr.org/mailman/listinfo/gradsusr
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20130426/2df080d7/attachment-0003.html 