[gradsusr] Performance Tips

Fri Feb 12 12:55:29 EST 2016

Hi All,

These are great tips.  Changing xdef and ydef is a great option but if I
understand correctly, it won’t work with Jennifer’s new code since
pdefwrite would have to be redone.  Making a dummy grid on the fly and
using lterp looks to be the next best option for us.

The most surprising thing I found from my python test is that plotting
performance doesn’t really degrade as you view a larger area of a high
resolution grib file.

HRRR example (attached in original email)

Grads =  0.86s (California view)  --> 6.36 (Conus View)

Python = 4.7 (California view) --> 6.8 (Conus View)

It may be beneficial if grads had a grid-to-xwindow ratio and/or a
grid-to-image ratio setting to acknowledge the fact that we don’t want to
keep writing over the same pixel for high resolution grids (basically
regrid or start skipping grib points for the plot/xwindow when things
become redundant).  GrADS could possibly allow users to turn this option
on/off and set their desired ratio.  This would make grads very snappy with
possibly little to no image/xwindow quality loss.  A good example is when
someone is doing an analysis with a HRRR grib file and grads performance
would change very little whether someone is looking at the entire conus or
just a small region.   Right now, grads performance changes by a factor of
7 in the examples I sent in the PDF.  Python’s performance changes by only
a factor of 1.5, so I suspect it is doing some regridding or selective grib
point plotting on the fly to keep things speedy.  Anyways, it is just a
thought and may be beneficial as we head towards higher resolution
datasets.  Thank you all for your help, I really appreciate it.

Travis

On Thu, Feb 11, 2016 at 11:12 AM, Jennifer M Adams <jadams21 at gmu.edu> wrote:

> Dear Travis, Wesley, et al.,
>
> I have done some testing with the high-res fnexrad 1km radar data,
> comparing the use of ‘pdef lccr’ (where the interpolation weights are
> calculated internally) and ‘pdef bilin’ (where interpolation weights are
> provided by the user in an external file. Reading the weights from a file
> was significantly faster — something like 30x faster!
>
> The tricky part of taking advantage of this performance gain is creating
> the pdef file itself, which depends on you being able to calculate
> non-integer i,j values in the native grid that correspond to each grid
> point in the destination grid, which is defined by what you put in your
> XDEF and YDEF statements. This is not necessarily simple.
>
> The good news is that GrADS does this calculation for you every time you
> open a descriptor with a pdef statement that doesn’t point to an external
> file — lcc, lccr, nps, sps, etc. I am going to implement a command
> ‘pdefwrite’ that will write out the interpolation weights calculated
> internally for these types of PDEF entries so that the file can be used
> with ‘pdef bilin’ instead. The protocol will be something like this:
> 1. Create a descriptor that has a pdef statement like this:
> pdef 4736 3000 lccr 23.0 -120 1 1 40.0 40.0 -100 1016.2360 1016.150
> 2. Open it with grads
> 3. Invoke pdefwrite with a file name as an argument
> 4. Rewrite your descriptor to use this pdef statment instead:
> pdef 4736 3000 bilin stream binary *your-filename-here*
> 5. Don’t change the XDEF and YDEF statements — those match the pdef file
> you created in step 3.
> 6. Open the new descriptor with GrADS and start working right away.
>
> Additional comments on Travis’s email:
>
> Shade1 may be faster than shade2 in some cases, but it won’t look right
> with transparent colors because the polygons in the old algorithm overlap.
> By the way, in the newer versions of GrADS, ‘gxout shaded’ is an alias for
> shade2, so if you want to use shade1 you have to say so explicitly.
>
> For regridding, the new code in lterp() does just about everything re()
> does only it is faster and more accurate. It is true that the destination
> grid definition requires an open file, but I use something like this all
> the time:
> dset ^foo.bin
> options template
> undef -9.99e8
> xdef 90 linear 2 4
> ydef 45 linear -88 4
> tdef 1 linear 01Jan0001 1dy
> zdef 1 linear 1 1
> vars 1
> foo 0 99 foo
> endvars
>
> You can even create that dummy descriptor on the fly, depending on what
> destination grid you need at the time. Also, if you are using pdef, it is a
> waste of resources to use lterp(), just put your desired destination grid
> in the XDEF and YDEF statements.
>
> High res data sets take longer to render because they have more data to
> grind through to calculate where to draw the contours. But if your data is
> high res, don’t you want to see that reflected in your plot?
>
> I like ‘gxout grfill’ to really see the finer details in the data.
> Contours over highly variable data (e.g. temperature in the Rocky
> Mountains) can look really noisy but grfill lets you see that variability
> without all the annoying squiggly contour lines.
>
> Regarding the resolution of the image output — there is no point to write
> out really high res data to a small image file; you just end up drawing
> over the same pixel multiple times. If image file dimensions are your
> limiting factor, then it might make sense to downgrade the resolution of
> your grid. I don’t think the optimal ratio between grid size and image size
> is 1:1, however. There’s probably a sweet spot somewhere where you can
> still see all the details in your data and the image size is lean enough. I
> think 800x600 is pretty small, and it is also not quite the same aspect
> ratio as 11x8.5 so your image will be a bit distorted from what you see in
> the display window.
>
> Don’t forget about the utility ‘pngquant' for making the image output
> files (from v2.1+) less bulky so you can store more of them and they will
> load faster in a browser.
>
> —Jennifer
>
>
> On Feb 11, 2016, at 10:40 AM, Wesley Ebisuzaki - NOAA Federal <
> wesley.ebisuzaki at noaa.gov> wrote:
>
> Travis,
>
>    I haven't tried this but it may work.
>
>    Instead of regridding your hi-res lat-lon data, make a new control file
> which has a PDEF .. BILIN.  This PDEF would map low_res(i,j) ->
> hi_res(n*i, n*j)
>
>      low_res() : the low-res x-y grid which is defined in the low-res ctl
> file.
>      hi_res(): the hi-res grib file grid
>
> I don't remember if grids start at grid(0,0) or grid(1,1).  If grids start
> at (1,1) then
> the above formula would have to be changed.
>
> Wesley
>
>
>
> On Tue, Feb 9, 2016 at 5:37 PM, Travis Wilson - NOAA Federal <
> travis.wilson at noaa.gov> wrote:
>
>> Hi All,
>>
>>
>> Attached is a very short ppt on grads performance vs python using grib
>> files.  In most cases, grads blows python away.  Times are relative to our
>> machine and consider everything from starting grads/opening the file, to
>> closing the file.
>>
>>
>> - In particular we have found that shaded1 is much faster.  Up to 40%
>> faster on our machines.
>>
>> - Wesley Ebisuzaki recommended converting the grib files to a lat/lon
>> grid to eliminate the PDEF entry to significantly speed up the opening time
>> of high resolution grib files.
>> http://gradsusr.org/pipermail/gradsusr/2016-January/039339.html
>>
>> - Again noted by Wesley, grib packing can have an impact on performance
>> http://gradsusr.org/pipermail/gradsusr/2010-May/027683.html
>>
>>
>> One thing we show in the ppt is that as the view gets wider (i.e. the
>> number of points that are plotted increase), the slower grads is relative
>> to python.  At some point, python will become faster.   Anyways, to battle
>> with this, regridding (using the re() function) the data within grads
>> significantly speeds up the plotting time (see last slide) when you have a
>> lot of points.  As far as I know, you can’t use re() in grads 2.1a3.  You
>> do have lterp() but a grid is needed.  Is there anything that will allow me
>> to lterp to my image dimensions?  Say my image dimensions are x800 y600
>> then lterp would interpolate my high resolution grib file to x800 y600 (or
>> some multiple of) when a view exceeds 800 points across.  This will
>> significantly speed up the plotting time when viewing a wide view of a high
>> resolution grib file while not degrading the image quality by much (again,
>> see last slide).
>>
>>
>> Also, if anyone has other performance tips on plotting high resolution
>> grib files we would love to hear them.
>>
>>
>> Thanks,
>>
>> Travis
>>
>> _______________________________________________
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>>
>>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
>
>
> --
> Jennifer Miletta Adams
> Center for Ocean-Land-Atmosphere Studies (COLA)
> George Mason University
>
>
>
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20160212/dfe9abfd/attachment.html 