[gradsusr] (no subject)

[Mark Sponsler]

Hi Jeff,
Regarding connection limits and downloading data versus building charts, I keep them as 2 separate scripts.

 I have a script that just downloads the data without any transformation and stores it as a file and the builds a control file for that data file. It also sets a flag that indicates whether the data file and control file were successfully created. For almost every data bit downloaded I check the response code for a positive outcome (if rc = 0 then. ...if rc=1 then...).  I check the data too ( if varx > or < some value then the data is corrupt). Then I have other scripts that read the resulting output file to build images. 

The thought being that in the past downloading data from the Dods server was occasionally problematic.  If the download process failed, or grabbed corrupt data, or any of multiple scenarios I've experienced in the past, I didn't want to then build corrupt images and post them to my site and then receive the requesite flood of emails from people complaining about bad images. The amount of quality checking does not need to be exhaustive. Typically if a few variables are suspect, that means it's all corrupt and something hinkey is going on at the data center. 

So downloading data and quality checking that data is one workflow, and building product is a separate workflow. At least that's my strategy.

Hope that helps,
Mark

On October 3, 2021 5:50:59 AM PDT, Jeff Chabot <jsc219 at gmail.com> wrote:
>Hello GrADS Users,
>
>I asked this question back in July and didn't see any responses, so I
>will try again.
>
>I have a question about pulling data from NOMADs with the new
>connection limits.  I have worked around the limit using the following
>sleep command between variables like this:
>
>say 'Define Precipitation'
>'define precip = apcpsfc/25.4'
>say result
>
>'!sleep 60'
>
>say 'Define Categorical ice pellets'
>'define ice = precip * cicepsfc'
>say result
>
>I am defining variables before a while statement:
>
>while ( t <= TIME_STEPS )
>
>'set t 't
>
>'display precip'
>'display ice'
>
>t = t = t+1
>
>endwhile
>
>This works but it slows the process down of course.  Would it be more
>efficient to not define the variables before the
>while statement, and just use them this way:
>
>while ( t <= TIME_STEPS )
>
>'set t 't
>
>'display apcpsfc/25.4'
>'display apcpsfc/25.4 * cicepsfc'
>
>t = t = t+1
>
>endwhile
>
>Thoughts?  I have tried it both ways with very mixed results, so I am
>just wondering what others are doing?
>
>Sincerely,
>
>Jeff Chabot
>_______________________________________________
>gradsusr mailing list
>gradsusr at gradsusr.org
>http://gradsusr.org/mailman/listinfo/gradsusr

-- 
Thanks 
Mark Sponsler
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gradsusr.org/pipermail/gradsusr/attachments/20211003/ff3a1e20/attachment.html>