[gradsusr] Mask/Scripting error

Tue Jan 10 10:23:16 EST 2017

Jennifer,

I switched the order of the open files, and that seemed to fix the issue I
was having. I just need to add a few commands so that each time step (756
month) is covered.

Thank you!

-Justin

On Mon, Jan 9, 2017 at 4:36 PM, Jennifer M Adams <jadams21 at gmu.edu> wrote:

> Justin,
> Now your script is producing some meaningful error messages that indicate
> a problem with the time axis metadata.
>
> The sdfopen code will retrieve the first and second time coordinate values
> and calculate an increment; then it parses the string in the units
> attribute (e.g. “minutes since…”,  “months since…”, et al.) and makes sure
> that these two make sense together. The error message you are seeing
> happens when the time axis unit is “minutes since” but the increment
> between the first and second time steps is larger than (60 minutes * 24
> hours * 360 days).
>
> Check the output from ncdump -c for the mask files that it is choking on
> and see if you can find the differences between the metadata in the files
> that do work.
>
> Also, it is probably worth the bother to switch the order of your open
> files — open the GLDAS data first keeping it open throughout the whole
> script as file 1, then inside the mask loop open each mask as file 2 and
> just keep opening and closing file 2.  You may notice an improvement in
> performace.
> —Jennifer
>
>
> On Jan 9, 2017, at 3:03 PM, Justin Hicks <jhicks2014 at gmail.com> wrote:
>
> Jennifer,
>
> Thanks, I never thought about making a list of the mask names and using
> 'subwrd' to call them. I used the draft you provided and changed a few
> lines, and this is what it currently looks like:
>
> 'reinit'
> outfile='/home/jhicks/Downloads/GLDAS_MASK_DATA.txt'
> '!/bin/rm -f 'outfile
>
> masks='00_WLD 01_ART 02_GOM 03_Med 04_blk 05_Cas 06_GLK 07_NPC 08_NAT
> 09_IND 10_SPC 11_SAT 12_ANT 13_SAm 14_NAm 15_AFR 16_EUA 17_AUS 18_OCN
> 19_GRN 20_NHL 21_NHO 22_SHL 23_SHO 24_NHm 25_SHm 26_WOL 27_ATL 28_PAC
> 29_NIO 30_SIO 31_MNP 32_AR2 33_GOB 34_SAH 35_SWM 36_SPD 37_NMB 38_WAS
> 39_AUD 40_AN2 41_NWP 42_KUR 43_GLF 44_ECS 45_BEN 46_EQP 47_INC 48_GTP
> 49_WIN 50_AMZ 51_TAL 52_EIO 53_NIG 54_EAF 55_AZ2 56_BOR'
>
> nmasks=57
>
> m=1
> while (m<=nmasks)
>   maskname=subwrd(masks,m)
>   infile='/data2/control/Region'maskname'_mask_GLDAS.nc'
>   'open 'infile
>
>
> if (rc)
>     say 'open failed for mask file 'm': 'infile', rc='rc
>     return
>   endif
>
> 'set dfile 1'
> 'set t 1 756'
> 'set lon -179.5 179.5'
> 'set lat -89.5 89.5'
>
> 'sdfopen /data2/control/GLDAS_NOAH10_M.020.ctl_1Deg_RainfallRateSfc.nc
> <http://gldas_noah10_m.020.ctl_1deg_rainfallratesfc.nc/>'
>
>
> if (rc)
>     say 'sdfopen failed for GLDAS file with mask number 'm
>     return
>   endif
>
> 'set dfile 2'
> 'set lon -179.5 179.5'
> 'set lat -89.5 89.5'
> 'set time 00Z01JAN1948'
> 'set dfile 1'
>
> 'q file 1'
> region = sublin(result,2)
> region = subwrd(region,2)
> 'q file 2'
> dataset = sublin(result,2)
> dataset = subwrd(dataset,2)
>  'q dims'
>  rectime = sublin(result,5)
>  recdate = subwrd(rectime,6)
> 'd aave(maskout(rainfsfc.2*86400,lterp(mask.1(time=
> 00Z01JAN1948),rainfsfc.2)),lon=-179.5,lon=179.5,lat=-89.5,lat=89.5)'
>
>  say result
>  rainfsfc = sublin(result,1)
>  rainfsfc = subwrd(rainfsfc,4)
> string = dataset' 'region' 'rectime' 'rainfsfc
>  rc=write(outfile ,  string ,'append')
> 'close 2'
> 'close 1'
>
>   m=m+1
> endwhile
>
> rc=close(outfile)
>
> *
> *
> *
>
> When I run this script in GrADS, the result is strange...some results are
> printed, but I get this error:
>
>  gadsdf: Time increment too large for 'minutes since' time units attribute
> SET DFILE error:  file 2 not open
> sdfopen failed for GLDAS file with mask number 3
>
> *
> *
> However, if I run this script again, I will get an error pertaining to,
> for example mask number 6:
>
> gadsdf: Time increment too large for 'minutes since' time units attribute
> SET DFILE error:  file 2 not open
> sdfopen failed for GLDAS file with mask number 6
>
>
> I'm starting to think the issue is memory related or possibly with the
> masks themselves (which were not produced by me), but I'm not quite sure.
>
> -Justin
>
> On Thu, Jan 5, 2017 at 5:45 PM, Jennifer M Adams <jadams21 at gmu.edu> wrote:
>
>> Hi, Justin —
>> I have a couple of suggestions for you.
>>
>> 0. Please use indents to make your code readable for yourself and others
>> helping you.
>>
>> 1. Rather than duplicate the code block over and over again, I would try
>> adding a wrapper around it to loop over the mask names.
>>
>> 2. You are missing a statement to close your text output file:
>>   rc=close(“gldas_mask_data.txt”)
>> And it looks like your ‘append’ option is never set for your write
>> statements. Be careful that you don’t overwrite this output file with every
>> write call. I would set the name up front, delete the file with a call to
>> the shell, then proceed with all the mask results appending to the file
>> every time, then close it when you are done. If you are creating a separate
>> output text file for each mask, put this code inside the wrapper loop and
>> create a uniqe outfile name that uses the mask name.
>>
>> 3. You should add a check after every ‘open’ and ‘sdfopen’ call to make
>> sure they worked (test if rc!=0), and if not, exit the script with an
>> informative message. Then you will be able to see more clearly where things
>> are going wrong.
>>
>> Here’s an untested draft of a re-organization of your script:
>>
>> ‘reinit’
>>
>> * set the output file name, delete it if it already exists
>> outfile=‘GLDAS_MASK_DATA.txt’
>> ‘!/bin/rm -f ‘outfile
>>
>> * create a list the different mask names for constructing filenames
>> masks=’00_WLD 01_ART 02_ABC 03_XYZ’
>> nmasks=4
>>
>> * loop over all the masks
>> m=1
>> while (m<=nmasks)
>>   maskname=subwrd(masks,m)
>>   infile=‘/data2/control/Region’maskname’_mask_GLDAS.nc'
>>   ‘open ‘infile
>> * make sure the file open worked without any errors
>>   if (rc)
>>     say ‘open failed for mask file ‘m’: ‘infile’, rc=‘rc
>>     return
>>   endif
>>
>> * add your code here
>>
>>   ‘sdfopen GLDASfile.nc’
>>   if (rc)
>>     say ‘sdfopen failed for GLDAS file with mask number ‘m
>>     return
>>   endif
>>
>> * add more of your code here
>>   string=blah’ ‘blah
>>   rc=write(outfile,string,append)
>>
>> * no need to keep calling reinit, just clean up properly, reset if
>> necessary
>>   ‘close 2’
>>   ‘close 1’
>>
>>   mask=mask+1
>> endwhile
>>
>> * close the text file with all the mask data
>> rc=close(“text_output_for_this_mask”)
>>
>> —Jennifer
>>
>>
>> On Jan 5, 2017, at 12:46 PM, Justin Hicks <jhicks2014 at gmail.com> wrote:
>>
>> Jeff,
>>
>> The seventh file exists; this is an issue that pertains to any
>> combination of seven masks. Also, I begin each new block of commands with
>> 'reinit', which should close any previously open mask files. If I run all
>> 56 masks (one at a time, each closing before the next mask is opened), some
>> of the masks still output data, and others just output that error
>> statement. That's why I don't believe it is a memory issue. Each mask file
>> runs perfectly fine when run in its own gradscript; I just get this error
>> when grouping them all into one script.
>>
>> -Justin
>>
>> On Thu, Jan 5, 2017 at 12:32 PM, Jeff Duda <jeffduda319 at gmail.com> wrote:
>>
>>> -Does the seventh file exist on your system?
>>> -Are you opening too many files? I don't see any 'close' statements in
>>> your script, so you may want to consider the ramifications of never closing
>>> files if you're opening a large number of them.
>>> -Are you running out of memory? Usually Grads just crashes entirely when
>>> there's a memory error, but perhaps it's not in this instance.
>>>
>>> Usually when it comes to debugging, the most comprehensive thing you can
>>> do is a line-by-line print statement to see what's going on in the script
>>> after each and every line. You can simplify by printing out only specific
>>> lines that you think are critical, then subdivide sections of code down to
>>> the source of the problem so that you don't have to literally issue a print
>>> statement after every line.
>>>
>>> Jeff Duda
>>>
>>> On Thu, Jan 5, 2017 at 10:19 AM, Justin Hicks <jhicks2014 at gmail.com>
>>> wrote:
>>>
>>>> GrADS users,
>>>>
>>>> I have come across a strange problem while running a gradscript. I am
>>>> using this script to output average monthly data of different regions to a
>>>> text file. Each block of commands contains a different mask, but uses the
>>>> same datafile. The first block of commands is shown below:
>>>>
>>>>  'reinit'
>>>> 'open /data2/control/Region00_WLD_mask_GLDAS.nc'
>>>> 'set t 1 756'
>>>> 'set dfile 1'
>>>> 'set lon -179.5 179.5'
>>>> 'set lat -89.5 89.5'
>>>> 'sdfopen /data2/control/GLDAS_NOAH10_M.020.ctl_1Deg_RainfallRateSfc.nc'
>>>> 'set dfile 2'
>>>> 'set lon -179.5 179.5'
>>>> 'set lat -89.5 89.5'
>>>> 'set time 00Z01JAN1948'
>>>> 'set dfile 1'
>>>>  nmonths=756
>>>>  i=1
>>>>  while (i<=nmonths)
>>>>  'set t 'i
>>>> 'q file 1'
>>>> region = sublin(result,2)
>>>> region = subwrd(region,2)
>>>> 'q file 2'
>>>> dataset = sublin(result,2)
>>>> dataset = subwrd(dataset,2)
>>>>  'q dims'
>>>>  rectime = sublin(result,5)
>>>>  recdate = subwrd(rectime,6)
>>>> 'd aave(maskout(rainfsfc.2*86400,lterp(mask.1(time=00Z01JAN1948
>>>> ),rainfsfc.2)),lon=-179.5,lon=179.5,lat=-89.5,lat=89.5)'
>>>>   say 't='i
>>>>  say result
>>>>  rainfsfc = sublin(result,1)
>>>>  rainfsfc = subwrd(rainfsfc,4)
>>>>  string = dataset' 'region' 'rectime' 'rainfsfc
>>>>  rc=write('GLDAS_MASK_DATA.txt' ,  string ,'append')
>>>>  i = i + 1
>>>>  endwhile
>>>>
>>>> I repeat these commands in the same script over and over, only changing
>>>> the second line (ex. the next one would read 'open
>>>> /data2/control/Region01_ART_mask_GLDAS.nc'). The data is output
>>>> perfectly when I have six masks in the script, but when I include the
>>>> seventh, the data output from the third mask / block of commands is
>>>> altered, and I get the following error:
>>>>
>>>> QUERY FILE Error:  file 2 not open
>>>> Data Request Error:  File number out of range
>>>>   Variable = rainfsfc.2
>>>> Operation Error:  Error from maskout function
>>>> Operation Error:  Error from aave function
>>>>   Error ocurred at column 1
>>>> DISPLAY error:  Invalid expression
>>>>   Expression = aave(maskout(rainfsfc.2*86400,
>>>> lterp(mask.1(time=00z01jan1948),rainfsfc.2)),lon=-179.5,lon=
>>>> 179.5,lat=-89.5,lat=89.5)
>>>> t=703
>>>> Data Request Error:  File number out of range
>>>>   Variable = rainfsfc.2
>>>> Operation Error:  Error from maskout function
>>>> Operation Error:  Error from aave function
>>>>   Error ocurred at column 1
>>>> DISPLAY error:  Invalid expression
>>>>   Expression = aave(maskout(rainfsfc.2*86400,
>>>> lterp(mask.1(time=00z01jan1948),rainfsfc.2)),lon=-179.5,lon=
>>>> 179.5,lat=-89.5,lat=89.5)
>>>>
>>>>
>>>> I don't believe the error lies within the third mask file itself
>>>> because it plots fine when it is run independently from the other masks. I
>>>> also get the same output issue when I run, for example, masks 50-56 (one or
>>>> two of these will come up with the error, but running masks 50-55 will
>>>> yield no problems).
>>>>
>>>> Any help would be appreciated to me solve this issue, as I'm baffled.
>>>>
>>>> -Justin
>>>>
>>>> --
>>>> *Justin Hicks*
>>>>
>>>> *Undergraduate Atmospheric Sciences Major*
>>>>
>>>> *GIS Minor University of Maryland-College Park*
>>>>
>>>> _______________________________________________
>>>> gradsusr mailing list
>>>> gradsusr at gradsusr.org
>>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>>
>>>>
>>>
>>>
>>> --
>>> Jeff Duda
>>> Post-doctoral research fellow
>>> University of Oklahoma School of Meteorology
>>>
>>> _______________________________________________
>>> gradsusr mailing list
>>> gradsusr at gradsusr.org
>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>
>>>
>>
>>
>> --
>> *Justin Hicks*
>>
>> *Undergraduate Atmospheric Sciences Major*
>>
>> *GIS Minor University of Maryland-College Park*
>> _______________________________________________
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>>
>>
>> --
>> Jennifer Miletta Adams
>> Center for Ocean-Land-Atmosphere Studies (COLA)
>> George Mason University
>>
>>
>>
>>
>> _______________________________________________
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>>
>>
>
>
> --
> *Justin Hicks*
>
> *Undergraduate Atmospheric Sciences Major*
>
> *GIS Minor University of Maryland-College Park*
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
>
>
> --
> Jennifer Miletta Adams
> Center for Ocean-Land-Atmosphere Studies (COLA)
> George Mason University
>
>
>
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
>
>

-- 
*Justin Hicks*

*UndergraduateAtmospheric Sciences Major*

*GIS MinorUniversity of Maryland-College Park*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20170110/ec7f4e25/attachment-0001.html 