<div dir="ltr">On Wed, Sep 17, 2008 at 1:33 PM, Kevin M Levey <span dir="ltr"><<a href="mailto:klevey@customweather.com">klevey@customweather.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
WED 17SEP08: 1030PDT<br>
<br>
Hi Arlindo<br>
<br>
Below is a snippet you contributed and I'd like to know in a little<br>
more detail what you mean by<div class="Ih2E3d"><br>
<br>
"Grib-2 is a good format for data transmission but is far less less<br>
optimal for day to day use. For example, the performance of your<br>
meteograms scripts will go up by a lot with grib-1"<br>
<br></div>
Are you saying that GRADS processes GRIB1 data much better/faster than<br>
GRIB2 data?</blockquote><div><br>Yes. Grib-1 compression is simpler.<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> Also more specifically, does GRADS 2x take longer to<br>
process GRIB2 data than say GRADS 1.9RC1 takes to process GRIB1 using<br>
the same grads script?<br>
</blockquote><div><br>Due to better compression algorithms, Grib-2 files are smaller (therefore better for transmission and storage) but you have to pay a price to decompress it. Grib-2 is particularly problematic to process single point time series such as those used in meteograms. Even if you only need one point, typically Grib-2 requires that you read in the whole globe in order in order to apply the compression. GrADS v2 does a lot of caching behind the scenes to minimize this, at the expense of increased memory usage. The g2() extension for GrADS v1.9 does not do any such caching.<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
Right now I've run into a time issue using the GFS 0.5x0.5 GRIB2 data<br>
using your GRADS 1.9RC1 + GEX - I thought it was merely the result of<br>
using higher res data that caused the time issue (i.e. taking very<br>
long to process all the necessary scripts).</blockquote><div><br>Run a test to convince yourself. Get one of your Grib-2 datasets, run it through lats4d and create a grib-1 version and rerun your usual scripts. If your script is i/o bound you should see a dramatic increased in speed. (If you do that, post your numbers here for future reference.)<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
Would it be more efficient to use GRIB1 data instead of GRIB2 data? </blockquote><div><br>Grib-1 takes more disk space, so it is not more efficient in terms of storage.<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
As<br>
you said, the conversion factor is often the problem point here as<br>
does take some time to convert the 0.5x0.5 GFS GRIB2 data to GRIB1 and<br>
is not possible to do in our case as it takes too long to convert 61<br>
time steps between 0 and 180 hours, even with high end Linux servers.<br>
</blockquote><div><br><br>When disk space is not a concern, the only situation I found this grib-1 conversion to pay off is when you are doing a lot of single grid point analysis like for meteograms, or accessing the same datasets over and over again. If you only read your files once it does not matter, this extra grib-2 to grib-1 conversion just adds complexity to your system. On the other hand, if you have a dynamic website where the plots are done on the fly, or a GDS site for that matter, having your files in Grib-1 could increase your response time by a lot. These days storage is getting cheaper while individual processor speed has flatten out, so using more disk space for grib-1 files seems a viable alternative for increasing performance.<br>
<br> Does anybody out there have a different experience/opinion about this?<br><br> Arlindo<br><br><br> ps: BTW, internally gzipped HDF-4 files has a similar issue with single point data.<br></div></div><br clear="all">
<br>-- <br>Arlindo da Silva<br><a href="mailto:dasilva@alum.mit.edu">dasilva@alum.mit.edu</a><br>
</div>