<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Hi, Brandon — <div>There is some code built into GrADS to help us developers track memory use. It is accessible with the undocumented ‘q mem’ command. We keep a list of memory blocks that have been allocated (not all, but most), and ‘q mem’ prints the list. The output from ‘q mem’ includes the address, the size, and the type of the memory block — the type is really a name that makes it convenient for finding the place in the code where that block was allocated, and also for knowing what that block of memory is for. If you compare the output from ‘q mem’ on startup and then after opening a file, and then again after displaying a variable, you will get a sense for what is responsible for the size of your executable as it is running. Items associated with the grib2 cache have names g2buff1, anchor, g2fld1, and g2mask1. If you invoke ‘flush’ and then check ‘q mem’ you can see immediately whether or not the flush command has done its job. </div><div><br></div><div>I tried a sequence of commands similar to what you showed using the 0.25-degree GFS output, and could not see any problems with ‘flush’. (Aside — thanks for teaching me about ‘pgrep’!) I suspect that the memory use in your case is not due to an ever-increasing grib2 cache, but to increasing allocations of metabuffer blocks (named mbufbuff in the ‘q mem’ output). The metabuffer contains all the instructions for drawing a graphic, and if your graphic is very detailed with shaded plots and contour overlays, the use of shapefiles or the basemap script, or if it contains many panels, then the metabuffer expands accordingly and additional blocks are allocated, and you will see lots of ‘mbufbuff’ entries when checking ‘q mem’. The metabuffer blocks are not released on ‘reinit’ — we assume if you needed them once, you may need them again, so we leave them chained up, ready to be re-used. A ‘clear’ command will flush the metabuffer and reset the pointer to the beginning of the first block; it may be that all your script needs is a ‘clear’ at the beginning of each loop. </div><div><br></div><div>—Jennifer</div><div><br><div><div>On Mar 19, 2015, at 2:01 AM, Brandon Schmidt <<a href="mailto:admin@wilmingtonwx.com">admin@wilmingtonwx.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">Through extensive debugging, I've found that the GrADS <i>flush</i> function does not work properly. This first came to my realization when I began to automate my GrADS scripts. Each script processes a single hour by downloading a GRIB file, creating a control file, opening the control file, and displaying/outputting the needed data. The scripts initialize with right around 100 MB of memory but through execution, they can rack up as much as 1.4 GB. Of course when processing multiple hours, this slowed my server to a crawl and in some cases caused crashes (especially with high-res HRRR data for example). Anyway, I traced the excessive memory usage to the display commands. I am aware that with each display command, the displayed variable is stored to the GrADS GRIB2 cache to eliminate further decompression overhead. This is all well given the <i>flush</i> command but it seems to only work half of the time. One can easily see this by noting the initial memory, displaying a few variables, flushing, and then noting the final memory. Seemingly randomly, the memory following a flush will slowly creep up. The behavior memory-wise that I expected from <i>flush</i> can only be obtained with <i>reinit</i> but calling that function thousands of times is not an option. Any ideas as to a work-around or fix? I am using the latest 2.1.a3 version. The first <i>flush</i> below works but the second does not.<div><br><div>-Brandon<br></div><div><br></div><div>---------------------------------------------------------------------------------</div><div><p class=""><span class=""><font face="monospace, monospace" size="1">ga-> !pgrep grads</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">7879</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">ga-> ! ps -p 7879 -o %mem,size </font></span></p><p class=""><span style="font-family:monospace,monospace;font-size:x-small">%MEM</span><span style="font-family:monospace,monospace;font-size:x-small"> </span><span style="font-family:monospace,monospace;font-size:x-small">SZ</span></p><p class=""><span class=""><font face="monospace, monospace" size="1"> 1.3 87276</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">ga-> d tmp2m</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">Notice: Automatic Grid Interpolation Taking Place</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">Contouring: 255 to 305 interval 5 </font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">ga-> ! ps -p 7879 -o %mem,size</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">%MEM SZ</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1"> 1.7 108024</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">ga-> d capesfc</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">Notice: Automatic Grid Interpolation Taking Place</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">Contouring: 0 to 3500 interval 500 </font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">ga-> set lat 50 70</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">LAT set to 50 70 </font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">ga-> d gustsfc</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">Notice: Automatic Grid Interpolation Taking Place</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">Contouring: 1 to 13 interval 1 </font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">ga-> set lat 20 90</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">LAT set to 20 90 </font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">ga-> d tmp2m</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">Notice: Automatic Grid Interpolation Taking Place</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">Contouring: 255 to 305 interval 5 </font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">ga-> flush</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">grib2 cache cleared</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">ga-> ! ps -p 7879 -o %mem,size</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">%MEM SZ</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1"> 1.7 108024</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">ga-> d tmp2m</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">Notice: Automatic Grid Interpolation Taking Place</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">Contouring: 255 to 305 interval 5 </font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">ga-> ! ps -p 7879 -o %mem,size</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">%MEM SZ</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1"> 1.9 122908</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">ga-> flush</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">grib2 cache cleared</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">ga-> ! ps -p 7879 -o %mem,size</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">%MEM SZ</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1"> 1.9 122908</font></span></p><p class=""><span class=""><font face="monospace, monospace" size="1">ga-> </font></span></p></div></div></div>
_______________________________________________<br>gradsusr mailing list<br><a href="mailto:gradsusr@gradsusr.org">gradsusr@gradsusr.org</a><br>http://gradsusr.org/mailman/listinfo/gradsusr<br></blockquote></div><br><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px;"><span class="Apple-style-span" style="border-collapse: separate; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; -webkit-text-decorations-in-effect: none; text-indent: 0px; -webkit-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><div>--</div><div>Jennifer M. Adams<br>Center for Ocean-Land-Atmosphere Studies (COLA)<br>111 Research Hall, Mail Stop 2B3<br>George Mason University<br>4400 University Drive<br>Fairfax, VA 22030 <br><br></div><div><br class="khtml-block-placeholder"></div><br class="Apple-interchange-newline"></span></span><br class="Apple-interchange-newline">
</div>
<br></div></body></html>