Memory optimization (was Re: Performance tuning)

[Stefan Gofferje]

Hi,

Arlindo da Silva schrieb:
> In your case, you could use lats4d with the g2 extension to create a
> scandinaian subset for your meteograms.

I already do some limiting in the download sript:

while [ $i -le 180 ]; do

  $GETGFS data ${DATE}${RUN} $i $i 0 all 'TMP.(2 m above ground|850
mb|500 mb)|RH.(2 m above ground|850 mb|500 mb)|PRMSL.mean sea
level|PRES.surface|CAPE|CIN|4LFTX.surface|UGRD.(10 m above ground|400
mb|450 mb|500 mb|550 mb|600 mb|650 mb|700 mb|750 mb|800 mb|850 mb|900
mb|925 mb|950 mb|1000 mb)|VGRD.(10 m above ground|400 mb|450 mb|500
mb|550 mb|600 mb|650 mb|700 mb|750 mb|800 mb|850 mb|900 mb|925 mb|950
mb|1000 mb)|VVEL.(700
mb)|TCDC|PRATE.surface|CPRAT.surface|CSNOW.surface|CRAIN.surface|CFRZR.surface|CICEP.surface'
${GRIBDIR}

  if [ $i -lt 10 ]; then
    FILENAME="${GRIBDIR}/gfs.t${RUN}z.pgrb2f0${i}"
  else
    FILENAME="${GRIBDIR}/gfs.t${RUN}z.pgrb2f${i}"
  fi
  $WGRIB2 ${FILENAME} -undefine out-box -60:100 43:90 -grib_out
${FILENAME}.extracted
  cat ${FILENAME}.extracted >> ${GRIB2FILE}
  rm ${FILENAME} ${FILENAME}.extracted
  i=$(echo "$i+3"|bc)
done

> One very simple solution is to add a GRIB-2 to GRIB-1 filter along the
> way, to convert the GRIB-2 to GRIB-1 format. Grib-2 is a good format for
> data transmission but is far less less optimal for day to day use. For
> example, the performance of your meteograms scripts will go up by a lot
> with grib-1. Of course, you have to factor in the cost of conversion.

That sounds rather complex. I did some modifications and I wait for the
feedback from my provider. If there still is a problem, I consider to
take the meteograms out.
I was thinking, if there is another option.

grads can write data to disk, right?
Would it be possible to change the procedure like that

while timestep <= 61
  Dowload timestep data
  start grads
  render charts
  write the relevant data for the cities for this timestep to disk
done
start another grads script
read the data for the meteograms and render those

What would be an appropiate format to write the data to disk so another
script can easily read them?

Terve,
Stefan

--
Last words of a stormchaser:
"Where is that rotation on the radar?!"