[gradsusr] Mask/Scripting error
Jennifer M Adams
jadams21 at gmu.edu
Mon Jan 9 16:36:57 EST 2017
Justin,
Now your script is producing some meaningful error messages that indicate a problem with the time axis metadata.
The sdfopen code will retrieve the first and second time coordinate values and calculate an increment; then it parses the string in the units attribute (e.g. “minutes since…”, “months since…”, et al.) and makes sure that these two make sense together. The error message you are seeing happens when the time axis unit is “minutes since” but the increment between the first and second time steps is larger than (60 minutes * 24 hours * 360 days).
Check the output from ncdump -c for the mask files that it is choking on and see if you can find the differences between the metadata in the files that do work.
Also, it is probably worth the bother to switch the order of your open files — open the GLDAS data first keeping it open throughout the whole script as file 1, then inside the mask loop open each mask as file 2 and just keep opening and closing file 2. You may notice an improvement in performace.
—Jennifer
On Jan 9, 2017, at 3:03 PM, Justin Hicks <jhicks2014 at gmail.com<mailto:jhicks2014 at gmail.com>> wrote:
Jennifer,
Thanks, I never thought about making a list of the mask names and using 'subwrd' to call them. I used the draft you provided and changed a few lines, and this is what it currently looks like:
'reinit'
outfile='/home/jhicks/Downloads/GLDAS_MASK_DATA.txt'
'!/bin/rm -f 'outfile
masks='00_WLD 01_ART 02_GOM 03_Med 04_blk 05_Cas 06_GLK 07_NPC 08_NAT 09_IND 10_SPC 11_SAT 12_ANT 13_SAm 14_NAm 15_AFR 16_EUA 17_AUS 18_OCN 19_GRN 20_NHL 21_NHO 22_SHL 23_SHO 24_NHm 25_SHm 26_WOL 27_ATL 28_PAC 29_NIO 30_SIO 31_MNP 32_AR2 33_GOB 34_SAH 35_SWM 36_SPD 37_NMB 38_WAS 39_AUD 40_AN2 41_NWP 42_KUR 43_GLF 44_ECS 45_BEN 46_EQP 47_INC 48_GTP 49_WIN 50_AMZ 51_TAL 52_EIO 53_NIG 54_EAF 55_AZ2 56_BOR'
nmasks=57
m=1
while (m<=nmasks)
maskname=subwrd(masks,m)
infile='/data2/control/Region'maskname'_mask_GLDAS.nc'
'open 'infile
if (rc)
say 'open failed for mask file 'm': 'infile', rc='rc
return
endif
'set dfile 1'
'set t 1 756'
'set lon -179.5 179.5'
'set lat -89.5 89.5'
'sdfopen /data2/control/GLDAS_NOAH10_M.020.ctl_1Deg_RainfallRateSfc.nc<http://gldas_noah10_m.020.ctl_1deg_rainfallratesfc.nc/>'
if (rc)
say 'sdfopen failed for GLDAS file with mask number 'm
return
endif
'set dfile 2'
'set lon -179.5 179.5'
'set lat -89.5 89.5'
'set time 00Z01JAN1948'
'set dfile 1'
'q file 1'
region = sublin(result,2)
region = subwrd(region,2)
'q file 2'
dataset = sublin(result,2)
dataset = subwrd(dataset,2)
'q dims'
rectime = sublin(result,5)
recdate = subwrd(rectime,6)
'd aave(maskout(rainfsfc.2*86400,lterp(mask.1(time=00Z01JAN1948),rainfsfc.2)),lon=-179.5,lon=179.5,lat=-89.5,lat=89.5)'
say result
rainfsfc = sublin(result,1)
rainfsfc = subwrd(rainfsfc,4)
string = dataset' 'region' 'rectime' 'rainfsfc
rc=write(outfile , string ,'append')
'close 2'
'close 1'
m=m+1
endwhile
rc=close(outfile)
*
*
*
When I run this script in GrADS, the result is strange...some results are printed, but I get this error:
gadsdf: Time increment too large for 'minutes since' time units attribute
SET DFILE error: file 2 not open
sdfopen failed for GLDAS file with mask number 3
*
*
However, if I run this script again, I will get an error pertaining to, for example mask number 6:
gadsdf: Time increment too large for 'minutes since' time units attribute
SET DFILE error: file 2 not open
sdfopen failed for GLDAS file with mask number 6
I'm starting to think the issue is memory related or possibly with the masks themselves (which were not produced by me), but I'm not quite sure.
-Justin
On Thu, Jan 5, 2017 at 5:45 PM, Jennifer M Adams <jadams21 at gmu.edu<mailto:jadams21 at gmu.edu>> wrote:
Hi, Justin —
I have a couple of suggestions for you.
0. Please use indents to make your code readable for yourself and others helping you.
1. Rather than duplicate the code block over and over again, I would try adding a wrapper around it to loop over the mask names.
2. You are missing a statement to close your text output file:
rc=close(“gldas_mask_data.txt”)
And it looks like your ‘append’ option is never set for your write statements. Be careful that you don’t overwrite this output file with every write call. I would set the name up front, delete the file with a call to the shell, then proceed with all the mask results appending to the file every time, then close it when you are done. If you are creating a separate output text file for each mask, put this code inside the wrapper loop and create a uniqe outfile name that uses the mask name.
3. You should add a check after every ‘open’ and ‘sdfopen’ call to make sure they worked (test if rc!=0), and if not, exit the script with an informative message. Then you will be able to see more clearly where things are going wrong.
Here’s an untested draft of a re-organization of your script:
‘reinit’
* set the output file name, delete it if it already exists
outfile=‘GLDAS_MASK_DATA.txt’
‘!/bin/rm -f ‘outfile
* create a list the different mask names for constructing filenames
masks=’00_WLD 01_ART 02_ABC 03_XYZ’
nmasks=4
* loop over all the masks
m=1
while (m<=nmasks)
maskname=subwrd(masks,m)
infile=‘/data2/control/Region’maskname’_mask_GLDAS.nc'
‘open ‘infile
* make sure the file open worked without any errors
if (rc)
say ‘open failed for mask file ‘m’: ‘infile’, rc=‘rc
return
endif
* add your code here
‘sdfopen GLDASfile.nc’
if (rc)
say ‘sdfopen failed for GLDAS file with mask number ‘m
return
endif
* add more of your code here
string=blah’ ‘blah
rc=write(outfile,string,append)
* no need to keep calling reinit, just clean up properly, reset if necessary
‘close 2’
‘close 1’
mask=mask+1
endwhile
* close the text file with all the mask data
rc=close(“text_output_for_this_mask”)
—Jennifer
On Jan 5, 2017, at 12:46 PM, Justin Hicks <jhicks2014 at gmail.com<mailto:jhicks2014 at gmail.com>> wrote:
Jeff,
The seventh file exists; this is an issue that pertains to any combination of seven masks. Also, I begin each new block of commands with 'reinit', which should close any previously open mask files. If I run all 56 masks (one at a time, each closing before the next mask is opened), some of the masks still output data, and others just output that error statement. That's why I don't believe it is a memory issue. Each mask file runs perfectly fine when run in its own gradscript; I just get this error when grouping them all into one script.
-Justin
On Thu, Jan 5, 2017 at 12:32 PM, Jeff Duda <jeffduda319 at gmail.com<mailto:jeffduda319 at gmail.com>> wrote:
-Does the seventh file exist on your system?
-Are you opening too many files? I don't see any 'close' statements in your script, so you may want to consider the ramifications of never closing files if you're opening a large number of them.
-Are you running out of memory? Usually Grads just crashes entirely when there's a memory error, but perhaps it's not in this instance.
Usually when it comes to debugging, the most comprehensive thing you can do is a line-by-line print statement to see what's going on in the script after each and every line. You can simplify by printing out only specific lines that you think are critical, then subdivide sections of code down to the source of the problem so that you don't have to literally issue a print statement after every line.
Jeff Duda
On Thu, Jan 5, 2017 at 10:19 AM, Justin Hicks <jhicks2014 at gmail.com<mailto:jhicks2014 at gmail.com>> wrote:
GrADS users,
I have come across a strange problem while running a gradscript. I am using this script to output average monthly data of different regions to a text file. Each block of commands contains a different mask, but uses the same datafile. The first block of commands is shown below:
'reinit'
'open /data2/control/Region00_WLD_mask_GLDAS.nc'
'set t 1 756'
'set dfile 1'
'set lon -179.5 179.5'
'set lat -89.5 89.5'
'sdfopen /data2/control/GLDAS_NOAH10_M.020.ctl_1Deg_RainfallRateSfc.nc'
'set dfile 2'
'set lon -179.5 179.5'
'set lat -89.5 89.5'
'set time 00Z01JAN1948'
'set dfile 1'
nmonths=756
i=1
while (i<=nmonths)
'set t 'i
'q file 1'
region = sublin(result,2)
region = subwrd(region,2)
'q file 2'
dataset = sublin(result,2)
dataset = subwrd(dataset,2)
'q dims'
rectime = sublin(result,5)
recdate = subwrd(rectime,6)
'd aave(maskout(rainfsfc.2*86400,lterp(mask.1(time=00Z01JAN1948),rainfsfc.2)),lon=-179.5,lon=179.5,lat=-89.5,lat=89.5)'
say 't='i
say result
rainfsfc = sublin(result,1)
rainfsfc = subwrd(rainfsfc,4)
string = dataset' 'region' 'rectime' 'rainfsfc
rc=write('GLDAS_MASK_DATA.txt' , string ,'append')
i = i + 1
endwhile
I repeat these commands in the same script over and over, only changing the second line (ex. the next one would read 'open /data2/control/Region01_ART_mask_GLDAS.nc'). The data is output perfectly when I have six masks in the script, but when I include the seventh, the data output from the third mask / block of commands is altered, and I get the following error:
QUERY FILE Error: file 2 not open
Data Request Error: File number out of range
Variable = rainfsfc.2
Operation Error: Error from maskout function
Operation Error: Error from aave function
Error ocurred at column 1
DISPLAY error: Invalid expression
Expression = aave(maskout(rainfsfc.2*86400,lterp(mask.1(time=00z01jan1948),rainfsfc.2)),lon=-179.5,lon=179.5,lat=-89.5,lat=89.5)
t=703
Data Request Error: File number out of range
Variable = rainfsfc.2
Operation Error: Error from maskout function
Operation Error: Error from aave function
Error ocurred at column 1
DISPLAY error: Invalid expression
Expression = aave(maskout(rainfsfc.2*86400,lterp(mask.1(time=00z01jan1948),rainfsfc.2)),lon=-179.5,lon=179.5,lat=-89.5,lat=89.5)
I don't believe the error lies within the third mask file itself because it plots fine when it is run independently from the other masks. I also get the same output issue when I run, for example, masks 50-56 (one or two of these will come up with the error, but running masks 50-55 will yield no problems).
Any help would be appreciated to me solve this issue, as I'm baffled.
-Justin
--
Justin Hicks
Undergraduate
Atmospheric Sciences Major
GIS Minor
University of Maryland-College Park
_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr
--
Jeff Duda
Post-doctoral research fellow
University of Oklahoma School of Meteorology
_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr
--
Justin Hicks
Undergraduate
Atmospheric Sciences Major
GIS Minor
University of Maryland-College Park
_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr
--
Jennifer Miletta Adams
Center for Ocean-Land-Atmosphere Studies (COLA)
George Mason University
_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr
--
Justin Hicks
Undergraduate
Atmospheric Sciences Major
GIS Minor
University of Maryland-College Park
_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr
--
Jennifer Miletta Adams
Center for Ocean-Land-Atmosphere Studies (COLA)
George Mason University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20170109/c3bcf826/attachment-0001.html
More information about the gradsusr
mailing list