Memory optimization (was Re: Performance tuning)

[Arlindo da Silva]

On Wed, Sep 17, 2008 at 6:09 AM, Stefan Gofferje <
gradsusers at gofferje.homelinux.org> wrote:

> Hi,
>
> Jennifer Adams schrieb:
> > The 'reinit' command will release all memory, including the grib2 cache.
> > The maximum size of the cache is hardcoded in gaio.c:
> > #define MAXG2CACHE 500100100
> > You can change this to something smaller and recompile GrADS. At some
> > point, I will add a command-line switch to set the cache size on
> > startup, but haven't gotten to that yet. (The parsing of switches and
> > args on startup is a bit of a mess and needs a thorough overhaul).
>
> Now I got a statement from the provider which indicates, the memory
> usage was the problem. In my tests at home, grads didn't work with less
> than 2M of cache.
> I would now need some help for optimization.
>
> At the moment I do the following:
> I download 61 timesteps of grib2 data from NOAA and cat them into one
> big file. This one big file I feed into my grads script to create
> weather charts and meteograms. I know I could significantly reduce the
> mem footprint by downloading one timestep, render the charts, end grads,
> get the next timestep, render the charts, etc. This would even reduce
> the continuos CPU load.
> BUT for the meteograms, I need everything in one file...
>

In your case, you could use lats4d with the g2 extension to create a
scandinaian subset for your meteograms.


Anybody any ideas?
>

One very simple solution is to add a GRIB-2 to GRIB-1 filter along the way,
to convert the GRIB-2 to GRIB-1 format. Grib-2 is a good format for data
transmission but is far less less optimal for day to day use. For example,
the performance of your meteograms scripts will go up by a lot with grib-1.
Of course, you have to factor in the cost of conversion.

  Arlindo


--
Arlindo da Silva
dasilva at alum.mit.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20080917/f618f58e/attachment.html