[gradsusr] Speed optimizations

[Jennifer M Adams]

I am quite familiar with that 1km radar data in grib2 format. It is behind a few of COLA’s operational products (e.g. http://wxmaps.org/pix2/currad.png).

The reason why it takes GrADS so long to open the file is because of PDEF — while you watch that ’scanning descriptor file’ message and sit there waiting for the prompt to return, GrADS is generating a database of indices to use for the pdef’d I/O. The interpolation is bilinear, so for every grid point in the destination grid, GrADS needs to know which 4 data points to read from the native data file. Once that array of indices has been calculated, the prompt returns and then the I/O is pretty fast because the entire native grib2 grid gets cached.

Jim Potemra’s example of a high res data set that loads instantly but takes a long time with the I/O — that data set probably has no pdef entry, but the I/O is slow because it’s classic netcdf, which is not cached by GrADS (it’s on my list of things to implement.) If you combine both of these features (i.e. you have a classic-format high-res netcdf file that requires a PDEF entry), it’s going to be painful to read with GrADS.

I suggest you try a 1-time format translation of the 1km grib2 file into a compressed, chunked netcdf4 file on a regular grid. Suppose the XDEF and YDEF entries in your descriptor file (1kmradar.ctl) looked like this:
xdef 6550 linear -129.0 0.01
ydef 3300 linear   22.0 0.01

Here’s an UNTESTED script fragment to do the format translation:
‘open 1kmradar.ctl’
‘define base=BREFl1_1’
‘set chunksize 655 330’
‘set sdfwrite -3dt -flt -nc4 -chunk -zip latest_1kmradar.nc4’
‘sdfwrite base’

Now you can do all your I/O from the web server on latest_1kmradar.nc4 using sdfopen and I bet it will be surprisingly quick. Not only have you gotten rid of PDEF, you are taking advantage of the super-efficient I/O with the chunked data file — the library does the subsetting for you, because it only reads the chunks it needs to cover a given region. If you display the data more than once in a script, the library is also cacheing (so GrADS doesn’t have to).

—Jennifer




On Jan 14, 2016, at 7:05 PM, James T. Potemra <jimp at hawaii.edu<mailto:jimp at hawaii.edu>> wrote:

Hi Travis,

This seems a little strange to me.  I did a quick test with one of our higher res data sets (global bathy) that is 21600 by 64800 (x,y) and it loads almost instantly in GrADS.  Of course generating a display takes a long time, and my test was done on a netCDF file.

This aside, could you run the data through an OPeNDAP server?  Even though it would be on the same machine, you could use the server to subset via the DAP call, rather than relying on GrADS to load the entire file.

Jim

On 1/14/16 12:54 PM, Travis Wilson - NOAA Federal wrote:
Hi,

At the NWS we have been using grads to create images from grib files on the fly.  Users use a front end interface to select a model (which is just a grib file on a server), field, and view, and submits a job.  Within 3-4 seconds the server is able to launch grads, open the grib file, and produce 50-100 images for the user to view.  This works extremely well for CONUS datasets with medium to coarse resolutions (up to 6km).
We have started using higher resolution grib files (1 km radar files are attached) and they are substantially slower.  The main problem comes from opening the file.  If a user selects radar for Northern California, it takes grads approximately 18 seconds just to open the 1km CONUS radar file (scanning descriptor file)..and then grads plots rather quickly.

A workaround we have been using is to subset the data on the fly (since we don’t know the area the user will select).  This example regrids the CONUS radar for northern California.

wgrib2 1kmradar_201512241600.grib2 -small_grib 233:243 36.8:42.8 1kmradar_201512241600.grib2.small

The small grib file will open in GrADS in about 1 second.  Regridding, and remaking the ctl and idx is substantially faster, but still requires a lot of time.  My question is, is there any way to speed up the scanning of the CONUS descriptor file without subset the grib file.  For example, grads would not scan the entire conus descriptor file when we just need to look at Northern California (or whatever portion of the US the users selects).

We appreciate all the help,
Travis



_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr


_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20160115/3a1d5ae2/attachment-0001.html