[gradsusr] NOMADS download speeds

[Joey Woodson]

Turns out the subsetting *is* the culprit... changed it from the CONUS to
US/Canada/Alaska and it actually downloads (insignificantly) faster... I'll
try testing raw data downloads sometime too (especially since I can use
parallel connections without bogging down the NOMADS server). Oh, and I
implemented the staggered downloads and now have a lag of about 2-4 seconds
for the first map and 12-20 seconds for the last map (GrADS is really fast
at generating maps... at least when the right settings are used on printim).

Thanks,
Joey
http://www.instantweathermaps.com

On Tue, Jul 12, 2011 at 2:42 PM, Wesley Ebisuzaki <Wesley.Ebisuzaki at noaa.gov
> wrote:

> Joey,
>
>       My impression of the nomads server is that it is very fast.
> However, I have a very fast link to the internet and
> the site is only 10 miles away.
>
>      You are using the GFS which has 0.5 degree resolution (260K
> points) and you are using the filter program
> which can create a regional subset.   If you are asking for a 5x5 grid,
> then your download speeds will be
> slow.  After all you have to process 10K points for every point sent to
> you.
>
>     Another possibility is that you may have run afoul of some
> fair-usage filter.   Every site needs some protection
> against denial-of-service attacks and poorly programmed web scripts.
> Anyways the nomads help desk is a more
> appropriate source of information than the GrADS user forum.
>
>    Wesley Ebisuzaki
>
>
> Joey Woodson wrote:
> > Is there any way to speed up downloads from the NOMADS server
> > (http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_hd.pl)? I currently
> > run a site that spends around 6-8 seconds per forecast hour generating
> > maps. Having said that, it is quite frustrating to see download times
> > for 1 MB of GRIB data exceed 10 seconds at times. I currently use 4
> > wget sessions in parallel to handle the downloading, and I only sleep
> > for 2 seconds instead of 10 when polling the NOMADS server to check
> > when the next file is available, so as you can see I am very
> > speed-oriented. Is there any way to download faster, or is
> > partial-downloading (downloading part of the GRIB file, generating the
> > relevant maps for the first part while the rest downloads, then
> > generating the maps for the rest) the fastest way of doing this?
> >
> > Joey
> > http://www.instantweathermaps.com <http://www.instantweathermaps.com/>
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > gradsusr mailing list
> > gradsusr at gradsusr.org
> > http://gradsusr.org/mailman/listinfo/gradsusr
> >
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20110728/eaeaed09/attachment-0003.html