[gradsusr] NOMADS/Dods 120 Connection Limit Info

[MARK SPONSLER]

Hi Everyone,
On 4/20/2021 with the 12z run of all models, NOAA implemented a brick wall limiter on the OpenDODs server to prevent some users from over-utilizing computer resources and to enable equitable access to everyone.

I thought I'd share some info on that process.

The limiter is set to prevent more than 120 connections in a 60 second period.

A 'connection' in Grads terminology is a request for a single variable. If you access DODs via the sdfopen command, enable fwrite, and then perform a 'display' on a single variable, that constitutes a single 'connection'.

While in fwrite (using sdfopen), if you set time = 1 and then display 5 variables, then start a loop and set time = 2 and display the same 5 variables, that is 10 connections from the limiters perspective.

Sdfopen moves pretty quick, and if you're downloading data in loops, it's very easy to exceed the 120 connection limit.

If you exceed the limit, the limiter will block your access to the 121st variable.

Here's the error you will see in Grads (in this case the limiter tripped on the display of variable hgtprs):

gancgrid error: nc_get_vara_double failed; NetCDF: Malformed or inaccessible DAP DATADDS
Data Request Error: Error for variable 'hgtprs'
Error ocurred at column 1
DISPLAY error: Invalid expression
Expression = hgtprs

The solution is simple.

I added a few lines of code to all my sdfopen/fwrite GRads scripts to pause the script after 100 variables have been displayed/downloaded.

That grads command (using the Windows sleep executable sleep.exe) would be something like:

say 'Sleep for 60 secs to Prevent Hitting 120 connection limit'
'!sleep.exe 60'
say 'Continue Processing'

There's probably a much more eloquent way to do this in UNIX and MacOS.

Anyway - thought I'd share this tidbit of info.

Thanks,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gradsusr.org/pipermail/gradsusr/attachments/20210427/aadc96c5/attachment.html>