<div dir="ltr">I figured this would be the perfect post to ask these questions in:<div><br></div><div>1.) "Defining" a variable actually 'costs' nothing in terms of performance, correct? It only actually gets calculated (the cost of processing) when you display it, right?</div><div><br></div><div>2.) Is there anyway to process the display once, but use it's output multiple times without incurring additional processing time? Simple example would be, display t step 2, but also fwrite the processed data to a file so when you process t step 3 you can just open the already processed data and simply add t step 3's data to it? Think of doing a total accumulation plot, the further out it gets the longer it takes to process (presumably) because it's calculating all previous hours data (which technically has already been processed) before it even calculates the current hours.</div><div><br></div><div>3.) All of the pdef talk in this, without defining a smaller pdef is the actual entire file read as soon as you open the corresponding ctl file? From the way the talks have been in this, it seems so but I figured I'd ask. Perhaps I'm confused on what setting an actual lat/long 'does', I assume it calculated the display output based around those dimensions and from the way I'm reading this it seems like the lat/lon is simply "controlling" grads in terms of the actual bounding dimension to display the data?</div><div><br></div><div>For example, we do this:</div><div><br></div><div><div>'set lat 10 68'</div><div>'set lon -138 -55'</div></div><div><br></div><div><div>'define radar1kagl = refd1000m.1'</div><div>'define hgt850tmp = tmpprs.1(lev=850)-273.15'</div></div><div><br></div><div><div>'set lat 20 58'</div><div>'set lon -128 -65'</div></div><div><br></div><div>'d hgt850tmp'</div><div><br></div><div>As you guys are probably aware, if we were to make the lat/lon before the define half the size the second lat/lon would still be the bounding box but the data would only be calculated from the first set, which is also why I'm questioning #3.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 12, 2016 at 1:38 PM, Wesley Ebisuzaki - NOAA Federal <span dir="ltr"><<a href="mailto:wesley.ebisuzaki@noaa.gov" target="_blank">wesley.ebisuzaki@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div><div>Travis,<br><br></div> I think this is how Jennifer wants it to work.<br><br></div>You have a 1 km grid and a ctl for the 1km grid. This works great for my town<br></div>but is super slow for the CONUS where your sceen can't resolve 1 km.<br><br></div>You make a ctl that uses the 1km file but uses xdef/ydef for a 10 km grid. <br></div>This will require a pdef line. This control file will be good for state maps.<br><br></div>You can also make a ctl that uses the 1km file but uses xdef/ydef for a 100 km grid.<br></div>This will also require a pdef line. This ctl file will be good for CONUS maps.<br><br></div>How you make the pdef line/file is the subject of another email.<span class="HOEnZb"><font color="#888888"><br><br></font></span></div><span class="HOEnZb"><font color="#888888">Wesley</font></span><div><div class="h5"><br><div><div><br><br><div><div><div><div><div><div><div><div><div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 12, 2016 at 12:55 PM, Travis Wilson - NOAA Federal <span dir="ltr"><<a href="mailto:travis.wilson@noaa.gov" target="_blank">travis.wilson@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><p class="MsoNormal">Hi All,</p>
<p class="MsoNormal"><br></p><p class="MsoNormal">These are great tips.
Changing xdef and ydef is a great option but if I understand correctly, it
won’t work with Jennifer’s new code since pdefwrite would have to be
redone. Making a dummy grid on the fly
and using lterp looks to be the next best option for us. </p>
<p class="MsoNormal"><br></p><p class="MsoNormal">The most surprising thing I found from my python test is
that plotting performance doesn’t really degrade as you view a larger area of a
high resolution grib file. </p>
<p class="MsoNormal" style="margin-bottom:0.0001pt"><br></p><p class="MsoNormal" style="margin-bottom:0.0001pt">HRRR example
(attached in original email)</p>
<p class="MsoNormal" style="margin-bottom:0.0001pt">Grads = 0.86s (California view) --> 6.36 (Conus View)</p>
<p class="MsoNormal" style="margin-bottom:0.0001pt">Python = 4.7
(California view) --> 6.8 (Conus View)</p>
<p class="MsoNormal" style="margin-bottom:0.0001pt"> </p>
<p class="MsoNormal" style="margin-bottom:0.0001pt">It may be
beneficial if grads had a grid-to-xwindow ratio and/or a grid-to-image ratio
setting to acknowledge the fact that we don’t want to keep writing over the
same pixel for high resolution grids (basically regrid or start skipping grib points
for the plot/xwindow when things become redundant). GrADS could possibly allow users to turn this
option on/off and set their desired ratio.
This would make grads very snappy with possibly little to no
image/xwindow quality loss. A good
example is when someone is doing an analysis with a HRRR grib file and grads performance
would change very little whether someone is looking at the entire conus or just
a small region. Right now, grads performance changes by a
factor of 7 in the examples I sent in the PDF.
Python’s performance changes by only a factor of 1.5, so I suspect it is
doing some regridding or selective grib point plotting on the fly to keep
things speedy. Anyways, it is just a
thought and may be beneficial as we head towards higher resolution datasets. Thank you all for your help, I really
appreciate it.</p>
<p class="MsoNormal" style="margin-bottom:0.0001pt"> </p>
<p class="MsoNormal" style="margin-bottom:0.0001pt">Travis </p><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 11, 2016 at 11:12 AM, Jennifer M Adams <span dir="ltr"><<a href="mailto:jadams21@gmu.edu" target="_blank">jadams21@gmu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word">
Dear Travis, Wesley, et al.,
<div><br>
</div>
<div>I have done some testing with the high-res fnexrad 1km radar data, comparing the use of ‘pdef lccr’ (where the interpolation weights are calculated internally) and ‘pdef bilin’ (where interpolation weights are provided by the user in an external file.
Reading the weights from a file was significantly faster — something like 30x faster!
<div><br>
</div>
<div>The tricky part of taking advantage of this performance gain is creating the pdef file itself, which depends on you being able to calculate non-integer i,j values in the native grid that correspond to each grid point in the destination grid, which is defined
by what you put in your XDEF and YDEF statements. This is not necessarily simple. </div>
<div><br>
</div>
<div>The good news is that GrADS does this calculation for you every time you open a descriptor with a pdef statement that doesn’t point to an external file — lcc, lccr, nps, sps, etc. I am going to implement a command ‘pdefwrite’ that will write out the interpolation
weights calculated internally for these types of PDEF entries so that the file can be used with ‘pdef bilin’ instead. The protocol will be something like this:</div>
<div>1. Create a descriptor that has a pdef statement like this:</div>
<div><font face="Courier">pdef 4736 3000 lccr 23.0 -120 1 1 40.0 40.0 -100 1016.2360 1016.150</font></div>
<div>2. Open it with grads</div>
<div>3. Invoke pdefwrite with a file name as an argument</div>
<div>4. Rewrite your descriptor to use this pdef statment instead:</div>
<div><font face="Courier">pdef 4736 3000 bilin stream binary <i>your-filename-here</i></font><br>
<div>5. Don’t change the XDEF and YDEF statements — those match the pdef file you created in step 3. </div>
<div>6. Open the new descriptor with GrADS and start working right away. </div>
<div><br>
</div>
<div>Additional comments on Travis’s email: </div>
<div><br>
</div>
<div>Shade1 may be faster than shade2 in some cases, but it won’t look right with transparent colors because the polygons in the old algorithm overlap. By the way, in the newer versions of GrADS, ‘gxout shaded’ is an alias for shade2, so if you want to use
shade1 you have to say so explicitly. </div>
<div><br>
</div>
<div>For regridding, the new code in lterp() does just about everything re() does only it is faster and more accurate. It is true that the destination grid definition requires an open file, but I use something like this all the time:</div>
<div>
<div><font face="Courier" size="2">dset ^foo.bin</font></div>
<div><font face="Courier" size="2">options template</font></div>
<div><font face="Courier" size="2">undef -9.99e8</font></div>
<div><font face="Courier" size="2">xdef 90 linear 2 4</font></div>
<div><font face="Courier" size="2">ydef 45 linear -88 4</font></div>
<div><font face="Courier" size="2">tdef 1 linear 01Jan0001 1dy</font></div>
<div><font face="Courier" size="2">zdef 1 linear 1 1</font></div>
<div><font face="Courier" size="2">vars 1</font></div>
<div><font face="Courier" size="2">foo 0 99 foo</font></div>
<div><font face="Courier" size="2">endvars</font></div>
</div>
<div><br>
</div>
<div>You can even create that dummy descriptor on the fly, depending on what destination grid you need at the time. Also, if you are using pdef, it is a waste of resources to use lterp(), just put your desired destination grid in the XDEF and YDEF statements. </div>
<div><br>
</div>
<div>High res data sets take longer to render because they have more data to grind through to calculate where to draw the contours. But if your data is high res, don’t you want to see that reflected in your plot? </div>
<div><br>
</div>
<div>I like ‘gxout grfill’ to really see the finer details in the data. Contours over highly variable data (e.g. temperature in the Rocky Mountains) can look really noisy but grfill lets you see that variability without all the annoying squiggly contour lines. </div>
<div><br>
</div>
<div>Regarding the resolution of the image output — there is no point to write out really high res data to a small image file; you just end up drawing over the same pixel multiple times. If image file dimensions are your limiting factor, then it might make
sense to downgrade the resolution of your grid. I don’t think the optimal ratio between grid size and image size is 1:1, however. There’s probably a sweet spot somewhere where you can still see all the details in your data and the image size is lean enough.
I think 800x600 is pretty small, and it is also not quite the same aspect ratio as 11x8.5 so your image will be a bit distorted from what you see in the display window. </div>
<div><br>
</div>
<div>Don’t forget about the utility ‘pngquant' for making the image output files (from v2.1+) less bulky so you can store more of them and they will load faster in a browser.</div>
<div><br>
</div>
<div>—Jennifer</div>
<div><br>
</div>
<div><div><div><br>
<div>
<div>On Feb 11, 2016, at 10:40 AM, Wesley Ebisuzaki - NOAA Federal <<a href="mailto:wesley.ebisuzaki@noaa.gov" target="_blank">wesley.ebisuzaki@noaa.gov</a>> wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>Travis,<br>
<br>
</div>
I haven't tried this but it may work.<br>
<br>
</div>
Instead of regridding your hi-res lat-lon data, make a new control file<br>
</div>
which has a PDEF .. BILIN. This PDEF would map low_res(i,j) -> hi_res(n*i, n*j)<br>
<br>
</div>
<div> low_res() : the low-res x-y grid which is defined in the low-res ctl file.<br>
</div>
<div> hi_res(): the hi-res grib file grid<br>
</div>
<div><br>
</div>
<div>I don't remember if grids start at grid(0,0) or grid(1,1). If grids start at (1,1) then<br>
</div>
<div>the above formula would have to be changed.<br>
</div>
<div><br>
</div>
<div>Wesley<br>
</div>
<div><br>
<br>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Feb 9, 2016 at 5:37 PM, Travis Wilson - NOAA Federal
<span dir="ltr"><<a href="mailto:travis.wilson@noaa.gov" target="_blank">travis.wilson@noaa.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<p class="MsoNormal" style="margin-bottom:0.0001pt"><span style="font-size:9.5pt;font-family:Arial,sans-serif;background-image:initial;background-repeat:initial">Hi All,</span><span style="font-size:12pt;font-family:'Times New Roman',serif"></span></p>
<div style="margin-bottom:0.0001pt"><span style="font-size:9.5pt;font-family:Arial,sans-serif"> </span><br>
</div>
<p class="MsoNormal" style="margin-bottom:0.0001pt;background-image:initial;background-repeat:initial">
<span style="font-size:9.5pt;font-family:Arial,sans-serif">Attached is a very short ppt on grads performance vs python using grib files. In most cases, grads blows python away. Times are relative to our machine and consider everything from starting grads/opening
the file, to closing the file. </span></p>
<div style="margin-bottom:0.0001pt"><span style="font-size:9.5pt;font-family:Arial,sans-serif"> </span><br>
</div>
<p class="MsoNormal" style="margin-bottom:0.0001pt;background-image:initial;background-repeat:initial">
<span style="font-size:9.5pt;font-family:Arial,sans-serif">- In particular we have found that shaded1 is much faster. Up to 40% faster on our machines. </span></p>
<p class="MsoNormal" style="margin-bottom:0.0001pt;background-image:initial;background-repeat:initial">
<span style="font-size:9.5pt;font-family:Arial,sans-serif">- Wesley Ebisuzaki recommended converting the grib files to a lat/lon grid to eliminate the PDEF entry to significantly speed up the opening time of high resolution grib files. <a href="http://gradsusr.org/pipermail/gradsusr/2016-January/039339.html" target="_blank">http://gradsusr.org/pipermail/gradsusr/2016-January/039339.html</a></span></p>
<p class="MsoNormal" style="margin-bottom:0.0001pt;background-image:initial;background-repeat:initial">
<span style="font-size:9.5pt;font-family:Arial,sans-serif">- Again noted by Wesley, grib packing can have an impact on performance
<a href="http://gradsusr.org/pipermail/gradsusr/2010-May/027683.html" target="_blank">
http://gradsusr.org/pipermail/gradsusr/2010-May/027683.html</a> </span></p>
<div style="margin-bottom:0.0001pt"><span style="font-size:9.5pt;font-family:Arial,sans-serif"> </span><br>
</div>
<p class="MsoNormal" style="margin-bottom:0.0001pt;background-image:initial;background-repeat:initial">
<span style="font-size:9.5pt;font-family:Arial,sans-serif">One thing we show in the ppt is that as the view gets wider (i.e. the number of points that are plotted increase), the slower grads is relative to python. At some point, python will become faster.
Anyways, to battle with this, regridding (using the re() function) the data within grads significantly speeds up the plotting time (see last slide) when you have a lot of points. As far as I know, you can’t use re() in grads 2.1a3. You do have lterp()
but a grid is needed. Is there anything that will allow me to lterp to my image dimensions? Say my image dimensions are x800 y600 then lterp would interpolate my high resolution grib file to x800 y600 (or some multiple of) when a view exceeds 800 points
across. This will significantly speed up the plotting time when viewing a wide view of a high resolution grib file while not degrading the image quality by much (again, see last slide).
</span></p>
<div style="margin-bottom:0.0001pt"><span style="font-size:9.5pt;font-family:Arial,sans-serif"> </span><br>
</div>
<p class="MsoNormal" style="margin-bottom:0.0001pt;background-image:initial;background-repeat:initial">
<span style="font-size:9.5pt;font-family:Arial,sans-serif">Also, if anyone has other performance tips on plotting high resolution grib files we would love to hear them.</span></p>
<div style="margin-bottom:0.0001pt"><span style="font-size:9.5pt;font-family:Arial,sans-serif"> </span><br>
</div>
<p class="MsoNormal" style="margin-bottom:0.0001pt;background-image:initial;background-repeat:initial">
<span style="font-size:9.5pt;font-family:Arial,sans-serif">Thanks,</span></p>
<p class="MsoNormal" style="margin-bottom:0.0001pt;background-image:initial;background-repeat:initial">
<span style="font-size:9.5pt;font-family:Arial,sans-serif">Travis</span></p>
</div>
<br>
_______________________________________________<br>
gradsusr mailing list<br>
<a href="mailto:gradsusr@gradsusr.org" target="_blank">gradsusr@gradsusr.org</a><br>
<a href="http://gradsusr.org/mailman/listinfo/gradsusr" rel="noreferrer" target="_blank">http://gradsusr.org/mailman/listinfo/gradsusr</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
_______________________________________________<br>
gradsusr mailing list<br>
<a href="mailto:gradsusr@gradsusr.org" target="_blank">gradsusr@gradsusr.org</a><br>
<a href="http://gradsusr.org/mailman/listinfo/gradsusr" target="_blank">http://gradsusr.org/mailman/listinfo/gradsusr</a><br>
</blockquote>
</div>
<br>
</div></div><div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div style="font-size:12px">--</div>
<div style="font-size:12px">Jennifer Miletta Adams<br>
Center for Ocean-Land-Atmosphere Studies (COLA)<br>
George Mason University<br>
<br>
<br>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
</div>
<br>_______________________________________________<br>
gradsusr mailing list<br>
<a href="mailto:gradsusr@gradsusr.org" target="_blank">gradsusr@gradsusr.org</a><br>
<a href="http://gradsusr.org/mailman/listinfo/gradsusr" rel="noreferrer" target="_blank">http://gradsusr.org/mailman/listinfo/gradsusr</a><br>
<br></blockquote></div><br></div></div>
<br>_______________________________________________<br>
gradsusr mailing list<br>
<a href="mailto:gradsusr@gradsusr.org" target="_blank">gradsusr@gradsusr.org</a><br>
<a href="http://gradsusr.org/mailman/listinfo/gradsusr" rel="noreferrer" target="_blank">http://gradsusr.org/mailman/listinfo/gradsusr</a><br>
<br></blockquote></div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
<br>_______________________________________________<br>
gradsusr mailing list<br>
<a href="mailto:gradsusr@gradsusr.org">gradsusr@gradsusr.org</a><br>
<a href="http://gradsusr.org/mailman/listinfo/gradsusr" rel="noreferrer" target="_blank">http://gradsusr.org/mailman/listinfo/gradsusr</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">-Chris A. Gilroy</div>
</div>