[gradsusr] question about "Low Level I/O Error: Seek error on data file"
Jennifer M Adams
jadams21 at gmu.edu
Thu Apr 13 19:25:13 EDT 2017
I figured out what the problem is. For each variable in a binary file, GrADS calculates an offset — the number of bytes that you must skip over before you read that particular variable. The offset is for the first variable in the file is 0. For each successive variable, it is the offset from the previous variable plus its nlevs*nlons*nlats. In your data file, nlevs is very large (127), and the xsize and ysize are also large enough so that the offset grows very quickly. Unfortunately, it is stored internally as a 4-byte integer, and after the 31st variable, it runs out of bits and the offset values become garbage. I added a debugging print statement which shows the offset for each variable, note it goes bad for the last 6 variables:
cref 0
zh 573160
th 73364480
thpert 146155800
prs 218947120
prspert 291738440
qv 364529760
qc 437321080
qr 510112400
qi 582903720
qs 655695040
qg 728486360
qhl 801277680
ccn 874069000
ccw 946860320
crw 1019651640
cci 1092442960
csw 1165234280
chw 1238025600
chl 1310816920
vhw 1383608240
vhl 1456399560
dbz 1529190880
uinterp 1601982200
uturb 1674773520
udiff 1747564840
upwp 1820356160
vinterp 1893147480
vturb 1965938800
vdiff 2038730120
vpwp 2111521440
winterp -2110654536
wturb -2037863216
wdiff -1965071896
xvort -1892280576
yvort -1819489256
zvort -1746697936
So, I will have to patch this bug and use an 8-byte integer to store the offset. Until then, your workaround is to write out fewer Z levels, or separate the variables into two distinct files. The output from ‘q file’ was very helpful, and so was the clue about the final six variables showing the error.
—Jennifer
On Apr 13, 2017, at 12:20 PM, Paul Markowski <pmarkowski at psu.edu<mailto:pmarkowski at psu.edu>> wrote:
The commands 'q file' and 'q dims' don't reveal anything out of the ordinary either (I've pasted the results below in case anyone wants to see). FWIW, this error appears only when I attempt to display the last 6 variables of the 37 variables in the file, which maybe suggests that the issue is file-size related. The file is 10.5 GB (I misspoke earlier when I said that it was ~7 GB). What has me stumped is that this doesn't appear to be a 2 GB (32-bit) issue. I routinely view files >2 GB, even >1 TB (!). My machine is 64-bit, linux, and has 132 GB of RAM. About the only thing unusual here is the large number of variables (37, 36 of which are 3D). I don't believe I've ever written that many to output before, and when I reduce the number of variables, the error goes away. I might look to see if there's some strange bug in CM1 such that an extra field is or isn't getting written to output, such that there's a mismatch between the grads ctl file and the dat file (this seems like a longshot though, given how many years I've used CM1 with Grads).
Paul
ga-> open cm1out_000001_s.ctl
Scanning description file: cm1out_000001_s.ctl
Data file cm1out_000001_s.dat is open as file 1
LON set to -122.3 122.3
LAT set to -74.85 74.85
LEV set to 0.005 0.005
Time values set: 1:7:3:0 1:7:3:0
E set to 1 1
ga-> q file
File 1 : CM1 output
Descriptor: cm1out_000001_s.ctl
Binary: cm1out_000001_s.dat
Type = Gridded
Xsize = 890 Ysize = 644 Zsize = 127 Tsize = 13 Esize = 1
Number of Variables = 37
cref 0 99 composite reflectivity (dBZ)
zh 127 99 height on model levels (m)
th 127 99 potential temp. (K)
thpert 127 99 potential temp. pert. (K)
prs 127 99 pressure (Pa)
prspert 127 99 pressure pert. (Pa)
qv 127 99 water vapor mixing ratio
qc 127 99 qc
qr 127 99 qr
qi 127 99 qi
qs 127 99 qs
qg 127 99 qg
qhl 127 99 qhl
ccn 127 99 ccn
ccw 127 99 ccw
crw 127 99 crw
cci 127 99 cci
csw 127 99 csw
chw 127 99 chw
chl 127 99 chl
vhw 127 99 vhw
vhl 127 99 vhl
dbz 127 99 reflectivity (dBZ)
uinterp 127 99 u interp. to scalar points
uturb 127 99 u turb forcing (m/s^2)
udiff 127 99 u diff forcing (m/s^2)
upwp 127 99 uprime-wprime (m^2/s^2)
vinterp 127 99 v interp. to scalar points
vturb 127 99 v turb forcing (m/s^2)
vdiff 127 99 v diff forcing (m/s^2)
vpwp 127 99 vprime-wprime (m^2/s^2)
winterp 127 99 w interp. to scalar points
wturb 127 99 w turb forcing (m/s^2)
wdiff 127 99 w turb forcing (m/s^2)
xvort 127 99 horiz vorticity (x) (s^-1)
yvort 127 99 horiz vorticity (y) (s^-1)
zvort 127 99 vertical vorticity (s^-1)
ga-> q dims
Default file number is: 1
X is varying Lon = -122.3 to 122.3 X = 1 to 890
Y is varying Lat = -74.85 to 74.85 Y = 1 to 644
Z is fixed Lev = 0.005 Z = 1
T is fixed Time = 00Z03JUL0001 T = 1
E is fixed Ens = 1 E = 1
ga-> d zvort
Low Level I/O Error: Seek error on data file
Data file name = cm1out_000001_s.dat
650 rc=-1 fpos=-1746697936 pfi->fhdr=0
Error occurred when seeking to byte -6986791744
Data Request Error: Error for variable 'zvort'
Error ocurred at column 1
DISPLAY error: Invalid expression
Expression = zvort
On 04/13/2017 11:00 AM, Jennifer M Adams wrote:
Hi, Paul —
GrADS is having trouble figuring out where to set the pointer in the data file to read the desired grid — that variable fpos should not be -2110654536. I would try the following strategy:
1. create a separate non-templated descriptor file for this particular data file
2. try to display zvort for the first and last time steps and see if you are still getting the error
3. the fpos variable is calculated based on the current dimension environment and the structure of the data file (order of variables, etc.) so if you can duplicate the error with the one file, please also send the output from ‘q file' and ‘q dims’ before you try your display.
—Jennifer
On Apr 13, 2017, at 9:48 AM, Paul Markowski <pmarkowski at psu.edu<mailto:pmarkowski at psu.edu>> wrote:
Hello Grads users,
I was wondering if anyone has ever seen this Grads error before:
ga-> d zvort
Low Level I/O Error: Seek error on data file
Data file name = cm1out_00%y4_s.dat
650 rc=-1 fpos=-2110654536 pfi->fhdr=0
Error occurred when seeking to byte -8442618144
Data Request Error: Error for variable 'zvort'
Error ocurred at column 1
DISPLAY error: Invalid expression
Expression = zvort
In case it matters, I'm trying to display output from George Bryan's CM1
model (CM1 is a popular cloud-resolving model used heavily in the
convective storms community, but is also used by the boundary layer and
mountain meteorology communities). I've used Grads to display CM1
output for 15 years, and this is the first time I've ever seen this
strange error. It might be noteworthy that I get this error only for
some of the variables that I'm trying to display. My output file is ~7
GB. I've tried both Grads 2.0.1 and 2.1.a3; same error with both. I
don't know if it has to do with the large file size; perhaps there's a
"large memory" compile option that wasn't properly used when the code
was compiled? I'd be grateful for any insight you might have!
Paul
--
Paul Markowski
Professor of Meteorology
Department of Meteorology and Atmospheric Science
Pennsylvania State University
520 Walker Building
University Park, PA 16802
email: pmarkowski at psu.edu<mailto:pmarkowski at psu.edu>
phone: 814.865.9736
web: http://sites.psu.edu/pmarkowski
_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr
--
Jennifer Miletta Adams
Center for Ocean-Land-Atmosphere Studies (COLA)
George Mason University
_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr
--
Jennifer Miletta Adams
Center for Ocean-Land-Atmosphere Studies (COLA)
George Mason University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20170413/640233a3/attachment-0001.html
More information about the gradsusr
mailing list