[gradsusr] Performance Tips

Fri Feb 12 13:38:30 EST 2016

Travis,

    I think this is how Jennifer wants it to work.

You have a 1 km grid and a ctl for the 1km grid.  This works great for my
town
but is super slow for the CONUS where your sceen can't resolve 1 km.

You make a ctl that uses the 1km file but uses xdef/ydef for a 10 km grid.
This will require a pdef line.  This control file will be good for state
maps.

You can also make a ctl that uses the 1km file but uses xdef/ydef for a 100
km grid.
This will also require a pdef line.  This ctl file will be good for CONUS
maps.

How you make the pdef line/file is the subject of another email.

Wesley

On Fri, Feb 12, 2016 at 12:55 PM, Travis Wilson - NOAA Federal <
travis.wilson at noaa.gov> wrote:

> Hi All,
>
>
> These are great tips.  Changing xdef and ydef is a great option but if I
> understand correctly, it won’t work with Jennifer’s new code since
> pdefwrite would have to be redone.  Making a dummy grid on the fly and
> using lterp looks to be the next best option for us.
>
>
> The most surprising thing I found from my python test is that plotting
> performance doesn’t really degrade as you view a larger area of a high
> resolution grib file.
>
>
> HRRR example (attached in original email)
>
> Grads =  0.86s (California view)  --> 6.36 (Conus View)
>
> Python = 4.7 (California view) --> 6.8 (Conus View)
>
>
>
> It may be beneficial if grads had a grid-to-xwindow ratio and/or a
> grid-to-image ratio setting to acknowledge the fact that we don’t want to
> keep writing over the same pixel for high resolution grids (basically
> regrid or start skipping grib points for the plot/xwindow when things
> become redundant).  GrADS could possibly allow users to turn this option
> on/off and set their desired ratio.  This would make grads very snappy with
> possibly little to no image/xwindow quality loss.  A good example is when
> someone is doing an analysis with a HRRR grib file and grads performance
> would change very little whether someone is looking at the entire conus or
> just a small region.   Right now, grads performance changes by a factor of
> 7 in the examples I sent in the PDF.  Python’s performance changes by only
> a factor of 1.5, so I suspect it is doing some regridding or selective grib
> point plotting on the fly to keep things speedy.  Anyways, it is just a
> thought and may be beneficial as we head towards higher resolution
> datasets.  Thank you all for your help, I really appreciate it.
>
>
>
> Travis
>
> On Thu, Feb 11, 2016 at 11:12 AM, Jennifer M Adams <jadams21 at gmu.edu>
> wrote:
>
>> Dear Travis, Wesley, et al.,
>>
>> I have done some testing with the high-res fnexrad 1km radar data,
>> comparing the use of ‘pdef lccr’ (where the interpolation weights are
>> calculated internally) and ‘pdef bilin’ (where interpolation weights are
>> provided by the user in an external file. Reading the weights from a file
>> was significantly faster — something like 30x faster!
>>
>> The tricky part of taking advantage of this performance gain is creating
>> the pdef file itself, which depends on you being able to calculate
>> non-integer i,j values in the native grid that correspond to each grid
>> point in the destination grid, which is defined by what you put in your
>> XDEF and YDEF statements. This is not necessarily simple.
>>
>> The good news is that GrADS does this calculation for you every time you
>> open a descriptor with a pdef statement that doesn’t point to an external
>> file — lcc, lccr, nps, sps, etc. I am going to implement a command
>> ‘pdefwrite’ that will write out the interpolation weights calculated
>> internally for these types of PDEF entries so that the file can be used
>> with ‘pdef bilin’ instead. The protocol will be something like this:
>> 1. Create a descriptor that has a pdef statement like this:
>> pdef 4736 3000 lccr 23.0 -120 1 1 40.0 40.0 -100 1016.2360 1016.150
>> 2. Open it with grads
>> 3. Invoke pdefwrite with a file name as an argument
>> 4. Rewrite your descriptor to use this pdef statment instead:
>> pdef 4736 3000 bilin stream binary *your-filename-here*
>> 5. Don’t change the XDEF and YDEF statements — those match the pdef file
>> you created in step 3.
>> 6. Open the new descriptor with GrADS and start working right away.
>>
>> Additional comments on Travis’s email:
>>
>> Shade1 may be faster than shade2 in some cases, but it won’t look right
>> with transparent colors because the polygons in the old algorithm overlap.
>> By the way, in the newer versions of GrADS, ‘gxout shaded’ is an alias for
>> shade2, so if you want to use shade1 you have to say so explicitly.
>>
>> For regridding, the new code in lterp() does just about everything re()
>> does only it is faster and more accurate. It is true that the destination
>> grid definition requires an open file, but I use something like this all
>> the time:
>> dset ^foo.bin
>> options template
>> undef -9.99e8
>> xdef 90 linear 2 4
>> ydef 45 linear -88 4
>> tdef 1 linear 01Jan0001 1dy
>> zdef 1 linear 1 1
>> vars 1
>> foo 0 99 foo
>> endvars
>>
>> You can even create that dummy descriptor on the fly, depending on what
>> destination grid you need at the time. Also, if you are using pdef, it is a
>> waste of resources to use lterp(), just put your desired destination grid
>> in the XDEF and YDEF statements.
>>
>> High res data sets take longer to render because they have more data to
>> grind through to calculate where to draw the contours. But if your data is
>> high res, don’t you want to see that reflected in your plot?
>>
>> I like ‘gxout grfill’ to really see the finer details in the data.
>> Contours over highly variable data (e.g. temperature in the Rocky
>> Mountains) can look really noisy but grfill lets you see that variability
>> without all the annoying squiggly contour lines.
>>
>> Regarding the resolution of the image output — there is no point to write
>> out really high res data to a small image file; you just end up drawing
>> over the same pixel multiple times. If image file dimensions are your
>> limiting factor, then it might make sense to downgrade the resolution of
>> your grid. I don’t think the optimal ratio between grid size and image size
>> is 1:1, however. There’s probably a sweet spot somewhere where you can
>> still see all the details in your data and the image size is lean enough. I
>> think 800x600 is pretty small, and it is also not quite the same aspect
>> ratio as 11x8.5 so your image will be a bit distorted from what you see in
>> the display window.
>>
>> Don’t forget about the utility ‘pngquant' for making the image output
>> files (from v2.1+) less bulky so you can store more of them and they will
>> load faster in a browser.
>>
>> —Jennifer
>>
>>
>> On Feb 11, 2016, at 10:40 AM, Wesley Ebisuzaki - NOAA Federal <
>> wesley.ebisuzaki at noaa.gov> wrote:
>>
>> Travis,
>>
>>    I haven't tried this but it may work.
>>
>>    Instead of regridding your hi-res lat-lon data, make a new control file
>> which has a PDEF .. BILIN.  This PDEF would map low_res(i,j) ->
>> hi_res(n*i, n*j)
>>
>>      low_res() : the low-res x-y grid which is defined in the low-res ctl
>> file.
>>      hi_res(): the hi-res grib file grid
>>
>> I don't remember if grids start at grid(0,0) or grid(1,1).  If grids
>> start at (1,1) then
>> the above formula would have to be changed.
>>
>> Wesley
>>
>>
>>
>> On Tue, Feb 9, 2016 at 5:37 PM, Travis Wilson - NOAA Federal <
>> travis.wilson at noaa.gov> wrote:
>>
>>> Hi All,
>>>
>>>
>>> Attached is a very short ppt on grads performance vs python using grib
>>> files.  In most cases, grads blows python away.  Times are relative to our
>>> machine and consider everything from starting grads/opening the file, to
>>> closing the file.
>>>
>>>
>>> - In particular we have found that shaded1 is much faster.  Up to 40%
>>> faster on our machines.
>>>
>>> - Wesley Ebisuzaki recommended converting the grib files to a lat/lon
>>> grid to eliminate the PDEF entry to significantly speed up the opening time
>>> of high resolution grib files.
>>> http://gradsusr.org/pipermail/gradsusr/2016-January/039339.html
>>>
>>> - Again noted by Wesley, grib packing can have an impact on performance
>>> http://gradsusr.org/pipermail/gradsusr/2010-May/027683.html
>>>
>>>
>>> One thing we show in the ppt is that as the view gets wider (i.e. the
>>> number of points that are plotted increase), the slower grads is relative
>>> to python.  At some point, python will become faster.   Anyways, to battle
>>> with this, regridding (using the re() function) the data within grads
>>> significantly speeds up the plotting time (see last slide) when you have a
>>> lot of points.  As far as I know, you can’t use re() in grads 2.1a3.  You
>>> do have lterp() but a grid is needed.  Is there anything that will allow me
>>> to lterp to my image dimensions?  Say my image dimensions are x800 y600
>>> then lterp would interpolate my high resolution grib file to x800 y600 (or
>>> some multiple of) when a view exceeds 800 points across.  This will
>>> significantly speed up the plotting time when viewing a wide view of a high
>>> resolution grib file while not degrading the image quality by much (again,
>>> see last slide).
>>>
>>>
>>> Also, if anyone has other performance tips on plotting high resolution
>>> grib files we would love to hear them.
>>>
>>>
>>> Thanks,
>>>
>>> Travis
>>>
>>> _______________________________________________
>>> gradsusr mailing list
>>> gradsusr at gradsusr.org
>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>
>>>
>> _______________________________________________
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>>
>>
>> --
>> Jennifer Miletta Adams
>> Center for Ocean-Land-Atmosphere Studies (COLA)
>> George Mason University
>>
>>
>>
>>
>> _______________________________________________
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>>
>>
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20160212/d3a2d383/attachment-0001.html 