<div dir="ltr">Hi Jennifer and Wesley,<div><br></div><div>I appreciate your comments, they are very helpful. </div><div><br></div><div>Wesley, the conversion to a lat/lon grid is exactly what we needed. As what Jennifer said, eliminating PDEF solves the problem! Also, thank you for the suggestion on converting the grib packing to c1, this is going to save us quite a bit of space! </div><div><br></div><div>Thanks,</div><div>Travis</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 15, 2016 at 4:59 AM, Wesley Ebisuzaki - NOAA Federal <span dir="ltr"><<a href="mailto:wesley.ebisuzaki@noaa.gov" target="_blank">wesley.ebisuzaki@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Travis,<br><br></div> Opening your ctl file takes about 6 seconds on my linux machine. A display takes about 2 seconds.<br></div>The memory requirements for displaying the field ranges from 55% to 92% of the total memory<br></div>of a 16 GB machine. Perhaps your displays are slow because you are using swap/virtual memory.<br><br></div> You mentioned that 1km radar fields are part of an archive. You can reduce the file size by<br></div><div>using special values for undefined values rather than a bitmap and by using wgrib2's<br></div><div>more efficient packing routines. For your sample file, you can save 56%. This will speed<br></div><div>up the I/O time. The downside is that using special values for undefined is not supported by<br></div><div>all programs. Most grib software supports special values because special values have been used<br>by NDFD grib2 files for several years. However, YMMV.<br><br>bash-4.1$ wgrib2 1kmradar_201512241600.grib2 -set_grib_type c1 -grib_out 1kmradar_201512241600.grib2.c1<br>1:189:d=<a href="tel:2015122416" value="+12015122416" target="_blank">2015122416</a>:BREF:surface - surface:anl:<br>bash-4.1$ ls -l 1kmradar_201512241600.grib2*<br>-rw-r--r--. 1 wd51we wd5 2597063 Jan 15 07:01 1kmradar_201512241600.grib2<br>-rw-r--r--. 1 wd51we wd5 1152668 Jan 15 07:18 1kmradar_201512241600.grib2.c1<br><div><div class="gmail_extra"><br></div><div class="gmail_extra">Jennifer also mentioned that the PDEFS require calculation by GrADS. You can use<br></div><div class="gmail_extra">wgrib2 to interpolate to a lat-lon grib file. This has the advantage of eliminating the<br></div><div class="gmail_extra">pdef calculation and reducing memory footprint of GrADS. If your computer is<br></div><div class="gmail_extra">running out of memory, this should be a big speed up.<br><br>wgrib2 1kmradar_201512241600.grib2 -set_grib_type c1 -new_grid_winds earth <br> -new_grid latlon -129.209875:6541:0.0100363502228856 <br> 21.747560:3298:0.00923772727272727 1kmradar_201512241600.grib2.latlon<br><br></div><div class="gmail_extra">I used the same weird lat-lon grid as used in your ctl file. Opening the lat-lon<br></div><div class="gmail_extra">grid was probably less than 1 second. Plotting started immediately.<br><br></div><div class="gmail_extra">So the slowness of in your system was probably caused by too little computer<br></div><div class="gmail_extra">memory. The slowness in my system was caused by the interpolation from<br></div><div class="gmail_extra">lambert-conformal to the lat-lon grid. NetCDF was probably faster because<br></div><div class="gmail_extra">(1) it allowed partial grid to be extracted which reduced the memory footprint<br></div><div class="gmail_extra">and avoided swapping and (2) the NetCDF data was already stored in a lat-lon grid. <br>You mentioned that the NetCDF file was 3x times larger. You had not used a bitmap, you<br></div><div class="gmail_extra">would have found that the NetCDF file was 6x larger. (Now waiting for a<br></div><div class="gmail_extra">some NetCDF user to reduce that number.)<span class="HOEnZb"><font color="#888888"><br></font></span></div><span class="HOEnZb"><font color="#888888"><div class="gmail_extra"><br></div><div class="gmail_extra">Wesley<br></div></font></span><div><div class="h5"><div class="gmail_extra"><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jan 14, 2016 at 8:57 PM, Jennifer M Adams <span dir="ltr"><<a href="mailto:jadams21@gmu.edu" target="_blank">jadams21@gmu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word">
<br>
<div>
<div>On Jan 14, 2016, at 8:25 PM, Travis Wilson - NOAA Federal <<a href="mailto:travis.wilson@noaa.gov" target="_blank">travis.wilson@noaa.gov</a>> wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">Hi James,
<div><br>
</div>
<div>I converted the file to a netcdf via sdfwrite and you are right. The file opens almost instantly. I wonder why grib files are so slow to open and if this could be avoided?
</div>
</div>
</blockquote>
Please see my earlier email for an explanation. </div>
<div><br>
<blockquote type="cite">
<div dir="ltr">
<div>It is worth noting that the netcdf file is about 3x larger even with using the -zip option. Unfortunately since this is an archive, we can't just increase the data stored by a factor of 3. </div>
</div>
</blockquote>
Did you use the -flt option? </div>
<div>If file size is really a problem, you can archive the grib2 files but use the netcdf file for the real-time web server — overwrite the netcdf file every time a new radar file comes in. </div>
<div><br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>I like your idea of the OPenDAP server, we may have to check into this if there is no other option. </div>
</div>
</blockquote>
I don’t think this is going to save you any time because under the hood of the opendap server, the I/O problems remain. You’re just adding overhead by putting it behind GDS, which takes extra time to write out the subsets in flat binary and then send them to
the opendap client. </div>
<div>—Jennifer</div>
<div><br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Jan 14, 2016 at 4:05 PM, James T. Potemra <span dir="ltr">
<<a href="mailto:jimp@hawaii.edu" target="_blank">jimp@hawaii.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">Hi Travis,<br>
<br>
This seems a little strange to me. I did a quick test with one of our higher res data sets (global bathy) that is 21600 by 64800 (x,y) and it loads almost instantly in GrADS. Of course generating a display takes a long time, and my test was done on a netCDF
file.<br>
<br>
This aside, could you run the data through an OPeNDAP server? Even though it would be on the same machine, you could use the server to subset via the DAP call, rather than relying on GrADS to load the entire file.<br>
<br>
Jim
<div>
<div><br>
<br>
<div>On 1/14/16 12:54 PM, Travis Wilson - NOAA Federal wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div>
<div dir="ltr">
<p class="MsoNormal">Hi,</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">At the NWS we have been using grads to create images from grib files on the fly. Users use a front end interface to select a model (which is just a grib file on a server), field, and view, and submits a job. Within 3-4 seconds the server
is able to launch grads, open the grib file, and produce 50-100 images for the user to view. This works extremely well for CONUS datasets with medium to coarse resolutions (up to 6km).
</p>
<p class="MsoNormal">We have started using higher resolution grib files (1 km radar files are attached) and they are substantially slower. The main problem comes from opening the file. If a user selects radar for Northern California, it takes grads approximately
18 seconds just to open the 1km CONUS radar file (scanning descriptor file)..and then grads plots rather quickly.
</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">A workaround we have been using is to subset the data on the fly (since we don’t know the area the user will select). This example regrids the CONUS radar for northern California. </p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">wgrib2 1kmradar_201512241600.grib2 -small_grib 233:243 36.8:42.8 1kmradar_201512241600.grib2.small</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">The small grib file will open in GrADS in about 1 second. Regridding, and remaking the ctl and idx is substantially faster, but still requires a lot of time. My question is, is there any way to speed up the scanning of the CONUS descriptor
file without subset the grib file. For example, grads would not scan the entire conus descriptor file when we just need to look at Northern California (or whatever portion of the US the users selects).
</p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal">We appreciate all the help,</p>
<p class="MsoNormal">Travis</p>
</div>
<br>
<fieldset></fieldset> <br>
</div>
</div>
<pre>_______________________________________________
gradsusr mailing list
<a href="mailto:gradsusr@gradsusr.org" target="_blank">gradsusr@gradsusr.org</a>
<a href="http://gradsusr.org/mailman/listinfo/gradsusr" target="_blank">http://gradsusr.org/mailman/listinfo/gradsusr</a>
</pre>
</blockquote>
<br>
</div>
<br>
_______________________________________________<br>
gradsusr mailing list<br>
<a href="mailto:gradsusr@gradsusr.org" target="_blank">gradsusr@gradsusr.org</a><br>
<a href="http://gradsusr.org/mailman/listinfo/gradsusr" rel="noreferrer" target="_blank">http://gradsusr.org/mailman/listinfo/gradsusr</a><br>
<br>
</blockquote>
</div>
<br>
</div>
_______________________________________________<br>
gradsusr mailing list<br>
<a href="mailto:gradsusr@gradsusr.org" target="_blank">gradsusr@gradsusr.org</a><br>
<a href="http://gradsusr.org/mailman/listinfo/gradsusr" target="_blank">http://gradsusr.org/mailman/listinfo/gradsusr</a><br>
</blockquote>
</div>
<br>
</div>
<br>_______________________________________________<br>
gradsusr mailing list<br>
<a href="mailto:gradsusr@gradsusr.org" target="_blank">gradsusr@gradsusr.org</a><br>
<a href="http://gradsusr.org/mailman/listinfo/gradsusr" rel="noreferrer" target="_blank">http://gradsusr.org/mailman/listinfo/gradsusr</a><br>
<br></blockquote></div><br></div></div></div></div></div></div>
<br>_______________________________________________<br>
gradsusr mailing list<br>
<a href="mailto:gradsusr@gradsusr.org">gradsusr@gradsusr.org</a><br>
<a href="http://gradsusr.org/mailman/listinfo/gradsusr" rel="noreferrer" target="_blank">http://gradsusr.org/mailman/listinfo/gradsusr</a><br>
<br></blockquote></div><br></div>