<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div><div id="AppleMailSignature"><span style="background-color: rgba(255, 255, 255, 0);">Fantastic! Thanks Jennifer!</span></div><div id="AppleMailSignature"><span style="background-color: rgba(255, 255, 255, 0);"><br></span></div><div id="AppleMailSignature"><span style="background-color: rgba(255, 255, 255, 0);">Paul</span></div><br><br></div><div><br>On Apr 13, 2017, at 7:25 PM, Jennifer M Adams <<a href="mailto:jadams21@gmu.edu">jadams21@gmu.edu</a>> wrote:<br><br></div><blockquote type="cite"><div>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<div>I figured out what the problem is. For each variable in a binary file, GrADS calculates an offset — the number of bytes that you must skip over before you read that particular variable. The offset is for the first variable in the file is 0. For each successive
variable, it is the offset from the previous variable plus its nlevs*nlons*nlats. In your data file, nlevs is very large (127), and the xsize and ysize are also large enough so that the offset grows very quickly. Unfortunately, it is stored internally as a
4-byte integer, and after the 31st variable, it runs out of bits and the offset values become garbage. I added a debugging print statement which shows the offset for each variable, note it goes bad for the last 6 variables: </div>
<div><br>
</div>
<div>
<div><font face="Courier New" style="font-size: 14px;"> cref 0</font></div>
<div><font face="Courier New" style="font-size: 14px;"> zh 573160</font></div>
<div><font face="Courier New" style="font-size: 14px;"> th 73364480</font></div>
<div><font face="Courier New" style="font-size: 14px;"> thpert 146155800</font></div>
<div><font face="Courier New" style="font-size: 14px;"> prs 218947120</font></div>
<div><font face="Courier New" style="font-size: 14px;"> prspert 291738440</font></div>
<div><font face="Courier New" style="font-size: 14px;"> qv 364529760</font></div>
<div><font face="Courier New" style="font-size: 14px;"> qc 437321080</font></div>
<div><font face="Courier New" style="font-size: 14px;"> qr 510112400</font></div>
<div><font face="Courier New" style="font-size: 14px;"> qi 582903720</font></div>
<div><font face="Courier New" style="font-size: 14px;"> qs 655695040</font></div>
<div><font face="Courier New" style="font-size: 14px;"> qg 728486360</font></div>
<div><font face="Courier New" style="font-size: 14px;"> qhl 801277680</font></div>
<div><font face="Courier New" style="font-size: 14px;"> ccn 874069000</font></div>
<div><font face="Courier New" style="font-size: 14px;"> ccw 946860320</font></div>
<div><font face="Courier New" style="font-size: 14px;"> crw 1019651640</font></div>
<div><font face="Courier New" style="font-size: 14px;"> cci 1092442960</font></div>
<div><font face="Courier New" style="font-size: 14px;"> csw 1165234280</font></div>
<div><font face="Courier New" style="font-size: 14px;"> chw 1238025600</font></div>
<div><font face="Courier New" style="font-size: 14px;"> chl 1310816920</font></div>
<div><font face="Courier New" style="font-size: 14px;"> vhw 1383608240</font></div>
<div><font face="Courier New" style="font-size: 14px;"> vhl 1456399560</font></div>
<div><font face="Courier New" style="font-size: 14px;"> dbz 1529190880</font></div>
<div><font face="Courier New" style="font-size: 14px;"> uinterp 1601982200</font></div>
<div><font face="Courier New" style="font-size: 14px;"> uturb 1674773520</font></div>
<div><font face="Courier New" style="font-size: 14px;"> udiff 1747564840</font></div>
<div><font face="Courier New" style="font-size: 14px;"> upwp 1820356160</font></div>
<div><font face="Courier New" style="font-size: 14px;"> vinterp 1893147480</font></div>
<div><font face="Courier New" style="font-size: 14px;"> vturb 1965938800</font></div>
<div><font face="Courier New" style="font-size: 14px;"> vdiff 2038730120</font></div>
<div><font face="Courier New" style="font-size: 14px;"> vpwp 2111521440</font></div>
<div><font face="Courier New" style="font-size: 14px;"> winterp -2110654536</font></div>
<div><font face="Courier New" style="font-size: 14px;"> wturb -2037863216</font></div>
<div><font face="Courier New" style="font-size: 14px;"> wdiff -1965071896</font></div>
<div><font face="Courier New" style="font-size: 14px;"> xvort -1892280576</font></div>
<div><font face="Courier New" style="font-size: 14px;"> yvort -1819489256</font></div>
<div><font face="Courier New" style="font-size: 14px;"> zvort -1746697936</font></div>
</div>
<div><br>
</div>
<div>So, I will have to patch this bug and use an 8-byte integer to store the offset. Until then, your workaround is to write out fewer Z levels, or separate the variables into two distinct files. The output from ‘q file’ was very helpful, and so was the clue
about the final six variables showing the error. </div>
<div>—Jennifer</div>
<div><br>
</div>
<br>
<div>
<div>On Apr 13, 2017, at 12:20 PM, Paul Markowski <<a href="mailto:pmarkowski@psu.edu">pmarkowski@psu.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000">The commands 'q file' and 'q dims' don't reveal anything out of the ordinary either (I've pasted the results below in case anyone wants to see). FWIW, this error appears only when I attempt to display the last 6 variables
of the 37 variables in the file, which maybe suggests that the issue is file-size related. The file is 10.5 GB (I misspoke earlier when I said that it was ~7 GB). What has me stumped is that this doesn't appear to be a 2 GB (32-bit) issue. I routinely view
files >2 GB, even >1 TB (!). My machine is 64-bit, linux, and has 132 GB of RAM. About the only thing unusual here is the large number of variables (37, 36 of which are 3D). I don't believe I've ever written that many to output before, and when I reduce
the number of variables, the error goes away. I might look to see if there's some strange bug in CM1 such that an extra field is or isn't getting written to output, such that there's a mismatch between the grads ctl file and the dat file (this seems like
a longshot though, given how many years I've used CM1 with Grads). <br>
<br>
Paul<br>
<br>
<br>
ga-> open cm1out_000001_s.ctl<br>
Scanning description file: cm1out_000001_s.ctl<br>
Data file cm1out_000001_s.dat is open as file 1<br>
LON set to -122.3 122.3 <br>
LAT set to -74.85 74.85 <br>
LEV set to 0.005 0.005 <br>
Time values set: 1:7:3:0 1:7:3:0 <br>
E set to 1 1 <br>
<br>
ga-> q file<br>
File 1 : CM1 output<br>
Descriptor: cm1out_000001_s.ctl<br>
Binary: cm1out_000001_s.dat<br>
Type = Gridded<br>
Xsize = 890 Ysize = 644 Zsize = 127 Tsize = 13 Esize = 1<br>
Number of Variables = 37<br>
cref 0 99 composite reflectivity (dBZ)<br>
zh 127 99 height on model levels (m)<br>
th 127 99 potential temp. (K)<br>
thpert 127 99 potential temp. pert. (K)<br>
prs 127 99 pressure (Pa)<br>
prspert 127 99 pressure pert. (Pa)<br>
qv 127 99 water vapor mixing ratio<br>
qc 127 99 qc<br>
qr 127 99 qr<br>
qi 127 99 qi<br>
qs 127 99 qs<br>
qg 127 99 qg<br>
qhl 127 99 qhl<br>
ccn 127 99 ccn<br>
ccw 127 99 ccw<br>
crw 127 99 crw<br>
cci 127 99 cci<br>
csw 127 99 csw<br>
chw 127 99 chw<br>
chl 127 99 chl<br>
vhw 127 99 vhw<br>
vhl 127 99 vhl<br>
dbz 127 99 reflectivity (dBZ)<br>
uinterp 127 99 u interp. to scalar points<br>
uturb 127 99 u turb forcing (m/s^2)<br>
udiff 127 99 u diff forcing (m/s^2)<br>
upwp 127 99 uprime-wprime (m^2/s^2)<br>
vinterp 127 99 v interp. to scalar points<br>
vturb 127 99 v turb forcing (m/s^2)<br>
vdiff 127 99 v diff forcing (m/s^2)<br>
vpwp 127 99 vprime-wprime (m^2/s^2)<br>
winterp 127 99 w interp. to scalar points<br>
wturb 127 99 w turb forcing (m/s^2)<br>
wdiff 127 99 w turb forcing (m/s^2)<br>
xvort 127 99 horiz vorticity (x) (s^-1)<br>
yvort 127 99 horiz vorticity (y) (s^-1)<br>
zvort 127 99 vertical vorticity (s^-1)<br>
<br>
ga-> q dims<br>
Default file number is: 1 <br>
X is varying Lon = -122.3 to 122.3 X = 1 to 890<br>
Y is varying Lat = -74.85 to 74.85 Y = 1 to 644<br>
Z is fixed Lev = 0.005 Z = 1<br>
T is fixed Time = 00Z03JUL0001 T = 1<br>
E is fixed Ens = 1 E = 1<br>
<br>
ga-> d zvort<br>
Low Level I/O Error: Seek error on data file <br>
Data file name = cm1out_000001_s.dat <br>
650 rc=-1 fpos=-1746697936 pfi->fhdr=0<br>
Error occurred when seeking to byte -6986791744 <br>
Data Request Error: Error for variable 'zvort'<br>
Error ocurred at column 1<br>
DISPLAY error: Invalid expression <br>
Expression = zvort<br>
<br>
<br>
<br>
<br>
<br>
On 04/13/2017 11:00 AM, Jennifer M Adams wrote:
<blockquote cite="mid:DB4E525D-5134-42A9-A691-888A31BB827B@gmu.edu" type="cite">Hi, Paul —
<div>GrADS is having trouble figuring out where to set the pointer in the data file to read the desired grid — that variable fpos should not be -2110654536. I would try the following strategy:</div>
<div>1. create a separate non-templated descriptor file for this particular data file</div>
<div>2. try to display zvort for the first and last time steps and see if you are still getting the error</div>
<div>3. the fpos variable is calculated based on the current dimension environment and the structure of the data file (order of variables, etc.) so if you can duplicate the error with the one file, please also send the output from ‘q file' and ‘q dims’ before
you try your display. </div>
<div>—Jennifer</div>
<div><br>
<div>
<div>On Apr 13, 2017, at 9:48 AM, Paul Markowski <<a moz-do-not-send="true" href="mailto:pmarkowski@psu.edu">pmarkowski@psu.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">Hello Grads users,<br>
<br>
I was wondering if anyone has ever seen this Grads error before:<br>
<br>
ga-> d zvort<br>
Low Level I/O Error: Seek error on data file<br>
Data file name = cm1out_00%y4_s.dat<br>
650 rc=-1 fpos=-2110654536 pfi->fhdr=0<br>
Error occurred when seeking to byte -8442618144<br>
Data Request Error: Error for variable 'zvort'<br>
Error ocurred at column 1<br>
DISPLAY error: Invalid expression<br>
Expression = zvort<br>
<br>
In case it matters, I'm trying to display output from George Bryan's CM1 <br>
model (CM1 is a popular cloud-resolving model used heavily in the <br>
convective storms community, but is also used by the boundary layer and <br>
mountain meteorology communities). I've used Grads to display CM1 <br>
output for 15 years, and this is the first time I've ever seen this <br>
strange error. It might be noteworthy that I get this error only for <br>
some of the variables that I'm trying to display. My output file is ~7 <br>
GB. I've tried both Grads 2.0.1 and 2.1.a3; same error with both. I <br>
don't know if it has to do with the large file size; perhaps there's a <br>
"large memory" compile option that wasn't properly used when the code <br>
was compiled? I'd be grateful for any insight you might have!<br>
<br>
Paul<br>
<br>
<br>
<br>
<br>
-- <br>
Paul Markowski<br>
Professor of Meteorology<br>
Department of Meteorology and Atmospheric Science<br>
Pennsylvania State University<br>
520 Walker Building<br>
University Park, PA 16802<br>
email: <a moz-do-not-send="true" href="mailto:pmarkowski@psu.edu">pmarkowski@psu.edu</a><br>
phone: 814.865.9736<br>
web: <a moz-do-not-send="true" href="http://sites.psu.edu/pmarkowski">http://sites.psu.edu/pmarkowski</a><br>
<br>
_______________________________________________<br>
gradsusr mailing list<br>
<a moz-do-not-send="true" href="mailto:gradsusr@gradsusr.org">gradsusr@gradsusr.org</a><br>
<a class="moz-txt-link-freetext" href="http://gradsusr.org/mailman/listinfo/gradsusr">http://gradsusr.org/mailman/listinfo/gradsusr</a><br>
</blockquote>
</div>
<br>
<div apple-content-edited="true">
<div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="font-size: 12px; orphans: 2; widows: 2;">--</div>
<div style="font-size: 12px; orphans: 2; widows: 2;">Jennifer Miletta Adams<br>
Center for Ocean-Land-Atmosphere Studies (COLA)<br>
George Mason University<br>
<br>
<br>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset> <br>
<pre wrap="">_______________________________________________
gradsusr mailing list
<a class="moz-txt-link-abbreviated" href="mailto:gradsusr@gradsusr.org">gradsusr@gradsusr.org</a>
<a class="moz-txt-link-freetext" href="http://gradsusr.org/mailman/listinfo/gradsusr">http://gradsusr.org/mailman/listinfo/gradsusr</a>
</pre>
</blockquote>
<br>
<br>
</div>
</blockquote>
</div>
<br>
<div apple-content-edited="true">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="font-size: 12px; orphans: 2; widows: 2;">--</div>
<div style="font-size: 12px; orphans: 2; widows: 2;">Jennifer Miletta Adams<br>
Center for Ocean-Land-Atmosphere Studies (COLA)<br>
George Mason University<br>
<br>
<br>
</div>
</div>
</div>
</div>
<br>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>gradsusr mailing list</span><br><span><a href="mailto:gradsusr@gradsusr.org">gradsusr@gradsusr.org</a></span><br><span><a href="http://gradsusr.org/mailman/listinfo/gradsusr">http://gradsusr.org/mailman/listinfo/gradsusr</a></span><br></div></blockquote></body></html>