[gradsusr] gradsusr Digest, Vol 61, Issue 25
Jennifer Adams
jma at cola.iges.org
Thu Mar 26 13:17:40 EDT 2015
For the record: I had mistaken ‘shade2b’ for ‘shade2’ in this reply. The gxout option ‘shade2b’ is the one that leaves all the sub-grid scale polygons as they are. Shade2 is the new contouring algorithm that is the default in all versions of 2.1. Sorry for the confusion!
—Jennifer
On Mar 23, 2015, at 6:34 AM, Jennifer Adams <jma at cola.iges.org> wrote:
> Brandon,
> Try using gxout shaded instead of shade2. The shade2 algorithm will have many, many more small polygons (on the sub-grid box scale.) That should help quite a bit. There is no real benefit for using shade2 — it was originally left in just in case the algorithm for merging the smaller polygons into bigger ones had flaws, but that has turned out to be a non-issue. If you use a lot of memory with regular use, then adding a feature to release the metabuffer blocks on ‘clear’ or ‘reinit’ will only slow your performance, because they will need to be re-allocated again for each new graphic.
> —Jennifer
>
> On Mar 22, 2015, at 8:01 PM, Brandon Schmidt <admin at wilmingtonwx.com> wrote:
>
>> I only display one product at a time. I occasionally overlap 2 - one shaded and one contour - but the metabuffer blocks are caused by regular use. I narrowed down my graphics command by command and I've found that the following combination still creates unnecessarily high memory use. The model data is HRRR 3 km, my gxout is set to 'shade2', my clab is set to 'masked', my cint is set to 1, cterp is set to off, parea is set to the full page, mproj is set to scaled, mpdset is set to hires, and finally I draw a county shapefile with most graphics. In the end, the full metabuffer is not necessary for the duration of the script. I only process one variable at a time so storing the entire buffer for tmp2m display commands is unnecessary when drawing a few MSLP contours later on in the script. Is it at all possible to set an option to clear the metabuffer in future releases? If not, where is the metabuffer source located? I could perhaps take a look myself and attempt to build in my own mod.
>>
>> Thanks - Brandon
>>
>> On Thu, Mar 19, 2015 at 2:35 PM, <gradsusr-request at gradsusr.org> wrote:
>> Send gradsusr mailing list submissions to
>> gradsusr at gradsusr.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> http://gradsusr.org/mailman/listinfo/gradsusr
>> or, via email, send a message with subject or body 'help' to
>> gradsusr-request at gradsusr.org
>>
>> You can reach the person managing the list at
>> gradsusr-owner at gradsusr.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of gradsusr digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Re: 'flush' command does not work (Jennifer Adams)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Thu, 19 Mar 2015 14:35:46 -0400
>> From: Jennifer Adams <jma at cola.iges.org>
>> Subject: Re: [gradsusr] 'flush' command does not work
>> To: GrADS Users Forum <gradsusr at gradsusr.org>
>> Message-ID: <6F09B5F5-BCDE-48FC-8723-769F48982865 at cola.iges.org>
>> Content-Type: text/plain; charset="windows-1252"
>>
>> The metabuffer blocks do not get freed once they are allocated. What happens with ?clear? or ?reinit? is that the pointers are reset to the beginning of the first block and the allocated blocks are re-used. New blocks are only allocated when all the existing blocks are filled up with graphics commands. You must be drawing one incredible graphic with so many metabuffer blocks allocated. Are you sure you?re not overlapping too many layers?
>> ?Jennifer
>>
>> On Mar 19, 2015, at 1:56 PM, Brandon Schmidt <admin at wilmingtonwx.com> wrote:
>>
>> > Hi Jennifer,
>> >
>> > Thank you for the 'q mem' command...that has helped in seeing what you've described. I did some more testing with the 'flush' command and it does seem to work. However, I already 'clear' after each display yet memory continues to rise. Should all "mbufbuff" entries in the memory query disappear after a 'clear' command? If so, they aren't. I trimmed mbufbuff entries in the middle of each result below but they were identical in each case.
>> >
>> > -Brandon
>> >
>> > ga-> q mem
>> >
>> > pos=0 ptr=0x5ef8580 len=7 type=setmpds
>> >
>> > pos=1 ptr=0x5ef8840 len=24 type=gxmbuf
>> >
>> > pos=2 ptr=0x7f3ca256c010 len=1000000 type=mbufbuff
>> >
>> > pos=3 ptr=0x7582a60 len=7 type=cmd
>> >
>> > pos=5 ptr=0x5f11090 len=23416 type=pfi
>> >
>> > pos=6 ptr=0x5f10020 len=57 type=mnam
>> >
>> > pos=7 ptr=0x5f10360 len=48 type=vals1
>> >
>> > pos=8 ptr=0x5f103a0 len=48 type=vals1
>> >
>> > pos=9 ptr=0x5f103e0 len=64 type=tvals5
>> >
>> > pos=10 ptr=0x5f10430 len=80 type=vals2
>> >
>> > pos=11 ptr=0x5f16c20 len=50505 type=pvar2
>> >
>> > pos=12 ptr=0x5f0fbd0 len=48 type=evals3
>> >
>> > pos=13 ptr=0x5f0fc10 len=60 type=ens5
>> >
>> > pos=14 ptr=0x5f0fc60 len=40 type=g2indx
>> >
>> > pos=15 ptr=0x5f106d0 len=568 type=g2intpnt
>> >
>> > pos=16 ptr=0x5f23180 len=20024 type=rbuf1
>> >
>> > pos=17 ptr=0x5f27fd0 len=20024 type=pbuf1
>> >
>> > pos=18 ptr=0x5f2ce20 len=20024 type=ubuf1
>> >
>> > pos=19 ptr=0x7f3c99c61010 len=11563860 type=ppi0
>> >
>> > pos=20 ptr=0x7f3c98652010 len=23127720 type=ppf0
>> >
>> > pos=21 ptr=0x7f3c97043010 len=23127720 type=ppf1
>> >
>> > pos=22 ptr=0x7f3c95a34010 len=23127720 type=ppw
>> >
>> > pos=24 ptr=0x5f32d20 len=24 type=gxmbuf
>> >
>> > pos=25 ptr=0x69f5c80 len=1000000 type=mbufbuff
>> >
>> > pos=26 ptr=0x5f32590 len=24 type=gxmbuf
>> >
>> > pos=27 ptr=0x6e38ed0 len=1000000 type=mbufbuff
>> >
>> > ...
>> >
>> > pos=181 ptr=0x7b3ca90 len=24 type=gxmbuf
>> >
>> > pos=182 ptr=0xd2f2bb0 len=1000000 type=mbufbuff
>> >
>> > pos=183 ptr=0x7b3c240 len=24 type=gxmbuf
>> >
>> > pos=184 ptr=0xd3e6e00 len=1000000 type=mbufbuff
>> >
>> > ga-> clear
>> >
>> > ga-> q mem
>> >
>> > pos=0 ptr=0x5ef8580 len=7 type=setmpds
>> >
>> > pos=1 ptr=0x5ef8840 len=24 type=gxmbuf
>> >
>> > pos=2 ptr=0x7f3ca256c010 len=1000000 type=mbufbuff
>> >
>> > pos=3 ptr=0x7b3c330 len=7 type=cmd
>> >
>> > pos=5 ptr=0x5f11090 len=23416 type=pfi
>> >
>> > pos=6 ptr=0x5f10020 len=57 type=mnam
>> >
>> > pos=7 ptr=0x5f10360 len=48 type=vals1
>> >
>> > pos=8 ptr=0x5f103a0 len=48 type=vals1
>> >
>> > pos=9 ptr=0x5f103e0 len=64 type=tvals5
>> >
>> > pos=10 ptr=0x5f10430 len=80 type=vals2
>> >
>> > pos=11 ptr=0x5f16c20 len=50505 type=pvar2
>> >
>> > pos=12 ptr=0x5f0fbd0 len=48 type=evals3
>> >
>> > pos=13 ptr=0x5f0fc10 len=60 type=ens5
>> >
>> > pos=14 ptr=0x5f0fc60 len=40 type=g2indx
>> >
>> > pos=15 ptr=0x5f106d0 len=568 type=g2intpnt
>> >
>> > pos=16 ptr=0x5f23180 len=20024 type=rbuf1
>> >
>> > pos=17 ptr=0x5f27fd0 len=20024 type=pbuf1
>> >
>> > pos=18 ptr=0x5f2ce20 len=20024 type=ubuf1
>> >
>> > pos=19 ptr=0x7f3c99c61010 len=11563860 type=ppi0
>> >
>> > pos=20 ptr=0x7f3c98652010 len=23127720 type=ppf0
>> >
>> > pos=21 ptr=0x7f3c97043010 len=23127720 type=ppf1
>> >
>> > pos=22 ptr=0x7f3c95a34010 len=23127720 type=ppw
>> >
>> > pos=24 ptr=0x5f32d20 len=24 type=gxmbuf
>> >
>> > pos=25 ptr=0x69f5c80 len=1000000 type=mbufbuff
>> >
>> > pos=26 ptr=0x5f32590 len=24 type=gxmbuf
>> >
>> > pos=27 ptr=0x6e38ed0 len=1000000 type=mbufbuff
>> >
>> > .........
>> >
>> > pos=181 ptr=0x7b3ca90 len=24 type=gxmbuf
>> >
>> > pos=182 ptr=0xd2f2bb0 len=1000000 type=mbufbuff
>> >
>> > pos=183 ptr=0x7b3c240 len=24 type=gxmbuf
>> >
>> > pos=184 ptr=0xd3e6e00 len=1000000 type=mbufbuff
>> >
>> >
>> > On Thu, Mar 19, 2015 at 12:00 PM, <gradsusr-request at gradsusr.org> wrote:
>> > Send gradsusr mailing list submissions to
>> > gradsusr at gradsusr.org
>> >
>> > To subscribe or unsubscribe via the World Wide Web, visit
>> > http://gradsusr.org/mailman/listinfo/gradsusr
>> > or, via email, send a message with subject or body 'help' to
>> > gradsusr-request at gradsusr.org
>> >
>> > You can reach the person managing the list at
>> > gradsusr-owner at gradsusr.org
>> >
>> > When replying, please edit your Subject line so it is more specific
>> > than "Re: Contents of gradsusr digest..."
>> >
>> >
>> > Today's Topics:
>> >
>> > 1. Re: 'flush' command does not work (Jennifer Adams)
>> >
>> >
>> > ----------------------------------------------------------------------
>> >
>> > Message: 1
>> > Date: Thu, 19 Mar 2015 10:40:22 -0400
>> > From: Jennifer Adams <jma at cola.iges.org>
>> > Subject: Re: [gradsusr] 'flush' command does not work
>> > To: GrADS Users Forum <gradsusr at gradsusr.org>
>> > Message-ID: <DCBDC890-E502-4901-8579-949C454BE3DF at cola.iges.org>
>> > Content-Type: text/plain; charset="windows-1252"
>> >
>> > Hi, Brandon ?
>> > There is some code built into GrADS to help us developers track memory use. It is accessible with the undocumented ?q mem? command. We keep a list of memory blocks that have been allocated (not all, but most), and ?q mem? prints the list. The output from ?q mem? includes the address, the size, and the type of the memory block ? the type is really a name that makes it convenient for finding the place in the code where that block was allocated, and also for knowing what that block of memory is for. If you compare the output from ?q mem? on startup and then after opening a file, and then again after displaying a variable, you will get a sense for what is responsible for the size of your executable as it is running. Items associated with the grib2 cache have names g2buff1, anchor, g2fld1, and g2mask1. If you invoke ?flush? and then check ?q mem? you can see immediately whether or not the flush command has done its job.
>> >
>> > I tried a sequence of commands similar to what you showed using the 0.25-degree GFS output, and could not see any problems with ?flush?. (Aside ? thanks for teaching me about ?pgrep?!) I suspect that the memory use in your case is not due to an ever-increasing grib2 cache, but to increasing allocations of metabuffer blocks (named mbufbuff in the ?q mem? output). The metabuffer contains all the instructions for drawing a graphic, and if your graphic is very detailed with shaded plots and contour overlays, the use of shapefiles or the basemap script, or if it contains many panels, then the metabuffer expands accordingly and additional blocks are allocated, and you will see lots of ?mbufbuff? entries when checking ?q mem?. The metabuffer blocks are not released on ?reinit? ? we assume if you needed them once, you may need them again, so we leave them chained up, ready to be re-used. A ?clear? command will flush the metabuffer and reset the pointer to the beginning of the firs!
>> t !
>> > block; it may be that all your script needs is a ?clear? at the beginning of each loop.
>> >
>> > ?Jennifer
>> >
>> > On Mar 19, 2015, at 2:01 AM, Brandon Schmidt <admin at wilmingtonwx.com> wrote:
>> >
>> > > Through extensive debugging, I've found that the GrADS flush function does not work properly. This first came to my realization when I began to automate my GrADS scripts. Each script processes a single hour by downloading a GRIB file, creating a control file, opening the control file, and displaying/outputting the needed data. The scripts initialize with right around 100 MB of memory but through execution, they can rack up as much as 1.4 GB. Of course when processing multiple hours, this slowed my server to a crawl and in some cases caused crashes (especially with high-res HRRR data for example). Anyway, I traced the excessive memory usage to the display commands. I am aware that with each display command, the displayed variable is stored to the GrADS GRIB2 cache to eliminate further decompression overhead. This is all well given the flush command but it seems to only work half of the time. One can easily see this by noting the initial memory, displaying a few variables,!
>> f!
>> > lushing, and then noting the final memory. Seemingly randomly, the memory following a flush will slowly creep up. The behavior memory-wise that I expected from flush can only be obtained with reinit but calling that function thousands of times is not an option. Any ideas as to a work-around or fix? I am using the latest 2.1.a3 version. The first flush below works but the second does not.
>> > >
>> > > -Brandon
>> > >
>> > > ---------------------------------------------------------------------------------
>> > > ga-> !pgrep grads
>> > >
>> > > 7879
>> > >
>> > > ga-> ! ps -p 7879 -o %mem,size
>> > >
>> > > %MEM SZ
>> > >
>> > > 1.3 87276
>> > >
>> > > ga-> d tmp2m
>> > >
>> > > Notice: Automatic Grid Interpolation Taking Place
>> > >
>> > > Contouring: 255 to 305 interval 5
>> > >
>> > > ga-> ! ps -p 7879 -o %mem,size
>> > >
>> > > %MEM SZ
>> > >
>> > > 1.7 108024
>> > >
>> > > ga-> d capesfc
>> > >
>> > > Notice: Automatic Grid Interpolation Taking Place
>> > >
>> > > Contouring: 0 to 3500 interval 500
>> > >
>> > > ga-> set lat 50 70
>> > >
>> > > LAT set to 50 70
>> > >
>> > > ga-> d gustsfc
>> > >
>> > > Notice: Automatic Grid Interpolation Taking Place
>> > >
>> > > Contouring: 1 to 13 interval 1
>> > >
>> > > ga-> set lat 20 90
>> > >
>> > > LAT set to 20 90
>> > >
>> > > ga-> d tmp2m
>> > >
>> > > Notice: Automatic Grid Interpolation Taking Place
>> > >
>> > > Contouring: 255 to 305 interval 5
>> > >
>> > > ga-> flush
>> > >
>> > > grib2 cache cleared
>> > >
>> > > ga-> ! ps -p 7879 -o %mem,size
>> > >
>> > > %MEM SZ
>> > >
>> > > 1.7 108024
>> > >
>> > > ga-> d tmp2m
>> > >
>> > > Notice: Automatic Grid Interpolation Taking Place
>> > >
>> > > Contouring: 255 to 305 interval 5
>> > >
>> > > ga-> ! ps -p 7879 -o %mem,size
>> > >
>> > > %MEM SZ
>> > >
>> > > 1.9 122908
>> > >
>> > > ga-> flush
>> > >
>> > > grib2 cache cleared
>> > >
>> > > ga-> ! ps -p 7879 -o %mem,size
>> > >
>> > > %MEM SZ
>> > >
>> > > 1.9 122908
>> > >
>> > > ga->
>> > >
>> > > _______________________________________________
>> > > gradsusr mailing list
>> > > gradsusr at gradsusr.org
>> > > http://gradsusr.org/mailman/listinfo/gradsusr
>> >
>> > --
>> > Jennifer M. Adams
>> > Center for Ocean-Land-Atmosphere Studies (COLA)
>> > 111 Research Hall, Mail Stop 2B3
>> > George Mason University
>> > 4400 University Drive
>> > Fairfax, VA 22030
>> >
>> >
>> >
>> >
>> >
>> > -------------- next part --------------
>> > An HTML attachment was scrubbed...
>> > URL: http://gradsusr.org/pipermail/gradsusr/attachments/20150319/82b935a5/attachment-0001.html
>> >
>> > ------------------------------
>> >
>> > _______________________________________________
>> > gradsusr mailing list
>> > gradsusr at gradsusr.org
>> > http://gradsusr.org/mailman/listinfo/gradsusr
>> >
>> >
>> > End of gradsusr Digest, Vol 61, Issue 23
>> > ****************************************
>> >
>> > _______________________________________________
>> > gradsusr mailing list
>> > gradsusr at gradsusr.org
>> > http://gradsusr.org/mailman/listinfo/gradsusr
>>
>> --
>> Jennifer M. Adams
>> Center for Ocean-Land-Atmosphere Studies (COLA)
>> 111 Research Hall, Mail Stop 2B3
>> George Mason University
>> 4400 University Drive
>> Fairfax, VA 22030
>>
>>
>>
>>
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: http://gradsusr.org/pipermail/gradsusr/attachments/20150319/f6fed08c/attachment.html
>>
>> ------------------------------
>>
>> _______________________________________________
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>>
>>
>> End of gradsusr Digest, Vol 61, Issue 25
>> ****************************************
>>
>> _______________________________________________
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>
> --
> Jennifer M. Adams
> Center for Ocean-Land-Atmosphere Studies (COLA)
> 111 Research Hall, Mail Stop 2B3
> George Mason University
> 4400 University Drive
> Fairfax, VA 22030
>
>
>
>
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
--
Jennifer M. Adams
Center for Ocean-Land-Atmosphere Studies (COLA)
111 Research Hall, Mail Stop 2B3
George Mason University
4400 University Drive
Fairfax, VA 22030
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20150326/c6af63c6/attachment-0001.html
More information about the gradsusr
mailing list