[gradsusr] Performance issue when working with multiple files on GrADS 2.1

[Jennifer Adams]

Of course you can draw two variables in the same image — two grids are not going to max out your cache size. 

loop over file names
  open file
    loop over lat/lon domains 
      draw variables
        print image
    end
  close file
end

—Jennifer



On Mar 26, 2015, at 5:56 PM, Brian Gaze <brian.gaze at NTLWORLD.COM> wrote:

Jennifer,

Thanks - but that pattern won't work if I want to draw both of the variables on the same image will it? 

Brian


On 26 March 2015 at 17:49, Jennifer Adams <jma at cola.iges.org> wrote:
Hi, Brian — 
The grib2 cache is a linked list of 2-D grids that are described by the file number, the coordinates of the non-varying dimensions (Z, T, and E), the size of the grid, and the name of the variable. If you are doing more than one display from that grid, the data are read from the cache instead of re-doing the I/O. Each new grid that is read is added to the list, most recent first. But because of the size limit of the grib2 cache, if you exceed the size, the grid at the end of the chain is released to make room for the new grid. So, if you are displaying data from 22 different grids, that is probably enough to exceed the size of the cache and when you go back to grid #1 the I/O has to be done again because that grid is no longer in the cache. It therefore makes a lot of sense for you to re-arrange the order of your nesting loops and do all your displays of a single cached grid before moving to the next variable. 

loop over file names
  open file
  loop over variables
    loop over lat/lon domains 
       draw plot 
       print image
    end
  end
  close file
end

The metafile buffer size option is parsed but not looked at in version 2.1. I should probably add a warning to alert the user, or remove it altogether. 
--Jennifer



On Mar 26, 2015, at 12:26 PM, Brian Gaze <brian.gaze at NTLWORLD.COM> wrote:

Hi,

When working concurrently with multiple grib2 files I'm encountering a performance issue. I initially thought this was caused by the grib2 cache but now wonder if it is related to the metafile buffer. 

The sample script below helps to explain. Scenario is....

1) Open all files
2) Outer loop where latitude and longitude for plot are set
3) Inner loop through all open files plotting 2 variables from each one for the latitude and longitude currently set
4) Write output to an image file
5) Iterate

If I reduce the filenumber count to be lower than 13 OR only plot 1 variable the image files are generated quickly, approx 1.5 per second. 

If I leave the filenumber count as it is and plot the 2 variables the plot plot is very slow taking approx 50 seconds to be written to the file system.
 
 

*Open files 1 to 22
'open file00.ctl'
'open file01.ctl'
'open filexx.ctl'


while(b<60)

filenumber=1
while (filenumber<23)
'set dfile ' %filenumber
'd var1'
'd var2'
filenumber=filenumber+1
endwhile

op = ' C:\'%imagefilename% '.png png x800 y600'
'gxprint ' %op ' white'

B=B+1
endwhile 


When running the .gs I've tried using the -m NNN option to increase the metafile buffer size (e.g. to 3000000) but this doesn't seem to make any difference. 

Any ideas??

Thanks

Brian



_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org
http://gradsusr.org/mailman/listinfo/gradsusr

--
Jennifer M. Adams
Center for Ocean-Land-Atmosphere Studies (COLA)
111 Research Hall, Mail Stop 2B3
George Mason University
4400 University Drive
Fairfax, VA 22030 






_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org
http://gradsusr.org/mailman/listinfo/gradsusr


_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org
http://gradsusr.org/mailman/listinfo/gradsusr

--
Jennifer M. Adams
Center for Ocean-Land-Atmosphere Studies (COLA)
111 Research Hall, Mail Stop 2B3
George Mason University
4400 University Drive
Fairfax, VA 22030 





-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20150326/55b4fa9f/attachment-0001.html