loop over x,y,z in user defined function (opengrads)

[Brian Doty]

The expression handling part of grads is fundamentally limited to  
grids with a maximum of 2D varying dimensions.  This goes back to the  
early 90s when I designed and implemented the language known as the  
"grads expression".  This design has stood the test of time -- if  
anything, experience since then has shown this to be extremely  
important, as it limits the amount of memory that grads consumes.  You  
can, of course, override this at any time by staging data in memory  
via the "define" feature.  Define can use great amounts of memory very  
rapidly, but it is under your control.

There are functions, such as ave, that loop over some dimension and  
evaluate a target expression that results in a 2D or less grid, and  
then does some operation on those grids (such as sum them and divide  
by n).  Note that at no time is there a grid present that is more than  
2D.  The looping results in potentially a lot of I/O.  If the data is  
going to be looped through more than once (or sliced in some non- 
efficient way), then putting the data into memory in advance, via  
define, can yield large performance improvements.  (It is important to  
understand the way grads evaluates expressions and what implications  
this has for I/O if you are working with a lot of data -- the way you  
do certain things can have huge performance differences.)

The functions that loop through a dimension (eg, ave) operate  
recursively.  They are thus not suitable as UDF functions, since I  
don't see how to easily provide a standardized interface without very  
high long term maintenance costs.  Thus I would personally recommend  
that you implement such a function as a source code mod, this being  
much safer in the long term, as it will make clear to future users the  
nature of the function and the level of burden for maintaining it.

As a side note, it is also important that no function outputs a grid  
that has fundamentally different dimensions than the input dimension  
environment.  This is a necessary restriction to preserve the internal  
consistency of the expression handling.   What I mean by this can be  
shown by an example:  if the input dimension environment is X and Y  
varying, it would be invalid to output a grid with some dimension  
other than X and Y varying (eg, T).   It is ok to output a grid that  
is a subset, ie, with just X varying.

Another approach is to do what I do in this situation:  I write a  
script and a C program, and my script outputs the needed data into a  
file, and the C program does the calculation, and the script reads in  
the data output by the C program.  Somewhat kludgy, but not all  
functions are worth the time and effort to implement within grads and  
maintain over the longer term,  IMO.

I am working through the design of some sort of interface to allow  
tasks to be done directly on "defined" objects.  This is an outgrowth  
of our recent implementation of netCDF file output.  Jennifer and I  
had extensive design discussions on this feature, and it became clear  
that we needed to collect large amounts of data in memory to optimize  
the I/O, and define was the obvious way to do this.  Now I am working  
on a way to do a "sort" operation on a "defined" data object, where  
the sort would operate though one of the dimensions.  I am also  
thinking that this type of thing could be generalized and a "UDF"  
style capability could be provided to perform some arbitrary task on a  
defined object.  This would not be part of the expression capability,  
but would be for one-shot types of things.  I've been trying to figure  
out how to provide a sort feature and this is the only way I can see  
to do it cleanly... Brian


On Jan 13, 2010, at 3:37 PM, Ming Pan wrote:

> Dear Users/Developers,
>
> I'm using grads-2.0.a7.oga.3 and trying to write a customized function
> to compute low-level cape. In this case, I need to loop over 3
> dimension x, y, and z, but it seems the current udf interface only
> allows passing 2-D grids (isiz, jsiz). Anyone ran into a similar
> situation and came up with a solution/work-around?
>
> Thanks!
>
> ming