[gradsusr] Performance Tips

[Jennifer M Adams]

Dear Travis, Wesley, et al.,

I have done some testing with the high-res fnexrad 1km radar data, comparing the use of ‘pdef lccr’ (where the interpolation weights are calculated internally) and ‘pdef bilin’ (where interpolation weights are provided by the user in an external file. Reading the weights from a file was significantly faster — something like 30x faster!

The tricky part of taking advantage of this performance gain is creating the pdef file itself, which depends on you being able to calculate non-integer i,j values in the native grid that correspond to each grid point in the destination grid, which is defined by what you put in your XDEF and YDEF statements. This is not necessarily simple.

The good news is that GrADS does this calculation for you every time you open a descriptor with a pdef statement that doesn’t point to an external file — lcc, lccr, nps, sps, etc. I am going to implement a command ‘pdefwrite’ that will write out the interpolation weights calculated internally for these types of PDEF entries so that the file can be used with ‘pdef bilin’ instead. The protocol will be something like this:
1. Create a descriptor that has a pdef statement like this:
pdef 4736 3000 lccr 23.0 -120 1 1 40.0 40.0 -100 1016.2360 1016.150
2. Open it with grads
3. Invoke pdefwrite with a file name as an argument
4. Rewrite your descriptor to use this pdef statment instead:
pdef 4736 3000 bilin stream binary your-filename-here
5. Don’t change the XDEF and YDEF statements — those match the pdef file you created in step 3.
6. Open the new descriptor with GrADS and start working right away.

Additional comments on Travis’s email:

Shade1 may be faster than shade2 in some cases, but it won’t look right with transparent colors because the polygons in the old algorithm overlap. By the way, in the newer versions of GrADS, ‘gxout shaded’ is an alias for shade2, so if you want to use shade1 you have to say so explicitly.

For regridding, the new code in lterp() does just about everything re() does only it is faster and more accurate. It is true that the destination grid definition requires an open file, but I use something like this all the time:
dset ^foo.bin
options template
undef -9.99e8
xdef 90 linear 2 4
ydef 45 linear -88 4
tdef 1 linear 01Jan0001 1dy
zdef 1 linear 1 1
vars 1
foo 0 99 foo
endvars

You can even create that dummy descriptor on the fly, depending on what destination grid you need at the time. Also, if you are using pdef, it is a waste of resources to use lterp(), just put your desired destination grid in the XDEF and YDEF statements.

High res data sets take longer to render because they have more data to grind through to calculate where to draw the contours. But if your data is high res, don’t you want to see that reflected in your plot?

I like ‘gxout grfill’ to really see the finer details in the data. Contours over highly variable data (e.g. temperature in the Rocky Mountains) can look really noisy but grfill lets you see that variability without all the annoying squiggly contour lines.

Regarding the resolution of the image output — there is no point to write out really high res data to a small image file; you just end up drawing over the same pixel multiple times. If image file dimensions are your limiting factor, then it might make sense to downgrade the resolution of your grid. I don’t think the optimal ratio between grid size and image size is 1:1, however. There’s probably a sweet spot somewhere where you can still see all the details in your data and the image size is lean enough. I think 800x600 is pretty small, and it is also not quite the same aspect ratio as 11x8.5 so your image will be a bit distorted from what you see in the display window.

Don’t forget about the utility ‘pngquant' for making the image output files (from v2.1+) less bulky so you can store more of them and they will load faster in a browser.

—Jennifer


On Feb 11, 2016, at 10:40 AM, Wesley Ebisuzaki - NOAA Federal <wesley.ebisuzaki at noaa.gov<mailto:wesley.ebisuzaki at noaa.gov>> wrote:

Travis,

   I haven't tried this but it may work.

   Instead of regridding your hi-res lat-lon data, make a new control file
which has a PDEF .. BILIN.  This PDEF would map low_res(i,j) -> hi_res(n*i, n*j)

     low_res() : the low-res x-y grid which is defined in the low-res ctl file.
     hi_res(): the hi-res grib file grid

I don't remember if grids start at grid(0,0) or grid(1,1).  If grids start at (1,1) then
the above formula would have to be changed.

Wesley



On Tue, Feb 9, 2016 at 5:37 PM, Travis Wilson - NOAA Federal <travis.wilson at noaa.gov<mailto:travis.wilson at noaa.gov>> wrote:
Hi All,

Attached is a very short ppt on grads performance vs python using grib files.  In most cases, grads blows python away.  Times are relative to our machine and consider everything from starting grads/opening the file, to closing the file.

- In particular we have found that shaded1 is much faster.  Up to 40% faster on our machines.
- Wesley Ebisuzaki recommended converting the grib files to a lat/lon grid to eliminate the PDEF entry to significantly speed up the opening time of high resolution grib files. http://gradsusr.org/pipermail/gradsusr/2016-January/039339.html
- Again noted by Wesley, grib packing can have an impact on performance http://gradsusr.org/pipermail/gradsusr/2010-May/027683.html

One thing we show in the ppt is that as the view gets wider (i.e. the number of points that are plotted increase), the slower grads is relative to python.  At some point, python will become faster.   Anyways, to battle with this, regridding (using the re() function) the data within grads significantly speeds up the plotting time (see last slide) when you have a lot of points.  As far as I know, you can’t use re() in grads 2.1a3.  You do have lterp() but a grid is needed.  Is there anything that will allow me to lterp to my image dimensions?  Say my image dimensions are x800 y600 then lterp would interpolate my high resolution grib file to x800 y600 (or some multiple of) when a view exceeds 800 points across.  This will significantly speed up the plotting time when viewing a wide view of a high resolution grib file while not degrading the image quality by much (again, see last slide).

Also, if anyone has other performance tips on plotting high resolution grib files we would love to hear them.

Thanks,
Travis

_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr


_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr

--
Jennifer Miletta Adams
Center for Ocean-Land-Atmosphere Studies (COLA)
George Mason University



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20160211/ef9ce3b6/attachment.html