[gradsusr] question about "Low Level I/O Error: Seek error on data file"
Paul Markowski
pmarkowski at psu.edu
Thu Apr 13 21:33:59 EDT 2017
Fantastic! Thanks Jennifer!
Paul
> On Apr 13, 2017, at 7:25 PM, Jennifer M Adams <jadams21 at gmu.edu> wrote:
>
> I figured out what the problem is. For each variable in a binary file, GrADS calculates an offset — the number of bytes that you must skip over before you read that particular variable. The offset is for the first variable in the file is 0. For each successive variable, it is the offset from the previous variable plus its nlevs*nlons*nlats. In your data file, nlevs is very large (127), and the xsize and ysize are also large enough so that the offset grows very quickly. Unfortunately, it is stored internally as a 4-byte integer, and after the 31st variable, it runs out of bits and the offset values become garbage. I added a debugging print statement which shows the offset for each variable, note it goes bad for the last 6 variables:
>
> cref 0
> zh 573160
> th 73364480
> thpert 146155800
> prs 218947120
> prspert 291738440
> qv 364529760
> qc 437321080
> qr 510112400
> qi 582903720
> qs 655695040
> qg 728486360
> qhl 801277680
> ccn 874069000
> ccw 946860320
> crw 1019651640
> cci 1092442960
> csw 1165234280
> chw 1238025600
> chl 1310816920
> vhw 1383608240
> vhl 1456399560
> dbz 1529190880
> uinterp 1601982200
> uturb 1674773520
> udiff 1747564840
> upwp 1820356160
> vinterp 1893147480
> vturb 1965938800
> vdiff 2038730120
> vpwp 2111521440
> winterp -2110654536
> wturb -2037863216
> wdiff -1965071896
> xvort -1892280576
> yvort -1819489256
> zvort -1746697936
>
> So, I will have to patch this bug and use an 8-byte integer to store the offset. Until then, your workaround is to write out fewer Z levels, or separate the variables into two distinct files. The output from ‘q file’ was very helpful, and so was the clue about the final six variables showing the error.
> —Jennifer
>
>
>> On Apr 13, 2017, at 12:20 PM, Paul Markowski <pmarkowski at psu.edu> wrote:
>>
>> The commands 'q file' and 'q dims' don't reveal anything out of the ordinary either (I've pasted the results below in case anyone wants to see). FWIW, this error appears only when I attempt to display the last 6 variables of the 37 variables in the file, which maybe suggests that the issue is file-size related. The file is 10.5 GB (I misspoke earlier when I said that it was ~7 GB). What has me stumped is that this doesn't appear to be a 2 GB (32-bit) issue. I routinely view files >2 GB, even >1 TB (!). My machine is 64-bit, linux, and has 132 GB of RAM. About the only thing unusual here is the large number of variables (37, 36 of which are 3D). I don't believe I've ever written that many to output before, and when I reduce the number of variables, the error goes away. I might look to see if there's some strange bug in CM1 such that an extra field is or isn't getting written to output, such that there's a mismatch between the grads ctl file and the dat file (this seems like a longshot though, given how many years I've used CM1 with Grads).
>>
>> Paul
>>
>>
>> ga-> open cm1out_000001_s.ctl
>> Scanning description file: cm1out_000001_s.ctl
>> Data file cm1out_000001_s.dat is open as file 1
>> LON set to -122.3 122.3
>> LAT set to -74.85 74.85
>> LEV set to 0.005 0.005
>> Time values set: 1:7:3:0 1:7:3:0
>> E set to 1 1
>>
>> ga-> q file
>> File 1 : CM1 output
>> Descriptor: cm1out_000001_s.ctl
>> Binary: cm1out_000001_s.dat
>> Type = Gridded
>> Xsize = 890 Ysize = 644 Zsize = 127 Tsize = 13 Esize = 1
>> Number of Variables = 37
>> cref 0 99 composite reflectivity (dBZ)
>> zh 127 99 height on model levels (m)
>> th 127 99 potential temp. (K)
>> thpert 127 99 potential temp. pert. (K)
>> prs 127 99 pressure (Pa)
>> prspert 127 99 pressure pert. (Pa)
>> qv 127 99 water vapor mixing ratio
>> qc 127 99 qc
>> qr 127 99 qr
>> qi 127 99 qi
>> qs 127 99 qs
>> qg 127 99 qg
>> qhl 127 99 qhl
>> ccn 127 99 ccn
>> ccw 127 99 ccw
>> crw 127 99 crw
>> cci 127 99 cci
>> csw 127 99 csw
>> chw 127 99 chw
>> chl 127 99 chl
>> vhw 127 99 vhw
>> vhl 127 99 vhl
>> dbz 127 99 reflectivity (dBZ)
>> uinterp 127 99 u interp. to scalar points
>> uturb 127 99 u turb forcing (m/s^2)
>> udiff 127 99 u diff forcing (m/s^2)
>> upwp 127 99 uprime-wprime (m^2/s^2)
>> vinterp 127 99 v interp. to scalar points
>> vturb 127 99 v turb forcing (m/s^2)
>> vdiff 127 99 v diff forcing (m/s^2)
>> vpwp 127 99 vprime-wprime (m^2/s^2)
>> winterp 127 99 w interp. to scalar points
>> wturb 127 99 w turb forcing (m/s^2)
>> wdiff 127 99 w turb forcing (m/s^2)
>> xvort 127 99 horiz vorticity (x) (s^-1)
>> yvort 127 99 horiz vorticity (y) (s^-1)
>> zvort 127 99 vertical vorticity (s^-1)
>>
>> ga-> q dims
>> Default file number is: 1
>> X is varying Lon = -122.3 to 122.3 X = 1 to 890
>> Y is varying Lat = -74.85 to 74.85 Y = 1 to 644
>> Z is fixed Lev = 0.005 Z = 1
>> T is fixed Time = 00Z03JUL0001 T = 1
>> E is fixed Ens = 1 E = 1
>>
>> ga-> d zvort
>> Low Level I/O Error: Seek error on data file
>> Data file name = cm1out_000001_s.dat
>> 650 rc=-1 fpos=-1746697936 pfi->fhdr=0
>> Error occurred when seeking to byte -6986791744
>> Data Request Error: Error for variable 'zvort'
>> Error ocurred at column 1
>> DISPLAY error: Invalid expression
>> Expression = zvort
>>
>>
>>
>>
>>
>>> On 04/13/2017 11:00 AM, Jennifer M Adams wrote:
>>> Hi, Paul —
>>> GrADS is having trouble figuring out where to set the pointer in the data file to read the desired grid — that variable fpos should not be -2110654536. I would try the following strategy:
>>> 1. create a separate non-templated descriptor file for this particular data file
>>> 2. try to display zvort for the first and last time steps and see if you are still getting the error
>>> 3. the fpos variable is calculated based on the current dimension environment and the structure of the data file (order of variables, etc.) so if you can duplicate the error with the one file, please also send the output from ‘q file' and ‘q dims’ before you try your display.
>>> —Jennifer
>>>
>>>> On Apr 13, 2017, at 9:48 AM, Paul Markowski <pmarkowski at psu.edu> wrote:
>>>>
>>>> Hello Grads users,
>>>>
>>>> I was wondering if anyone has ever seen this Grads error before:
>>>>
>>>> ga-> d zvort
>>>> Low Level I/O Error: Seek error on data file
>>>> Data file name = cm1out_00%y4_s.dat
>>>> 650 rc=-1 fpos=-2110654536 pfi->fhdr=0
>>>> Error occurred when seeking to byte -8442618144
>>>> Data Request Error: Error for variable 'zvort'
>>>> Error ocurred at column 1
>>>> DISPLAY error: Invalid expression
>>>> Expression = zvort
>>>>
>>>> In case it matters, I'm trying to display output from George Bryan's CM1
>>>> model (CM1 is a popular cloud-resolving model used heavily in the
>>>> convective storms community, but is also used by the boundary layer and
>>>> mountain meteorology communities). I've used Grads to display CM1
>>>> output for 15 years, and this is the first time I've ever seen this
>>>> strange error. It might be noteworthy that I get this error only for
>>>> some of the variables that I'm trying to display. My output file is ~7
>>>> GB. I've tried both Grads 2.0.1 and 2.1.a3; same error with both. I
>>>> don't know if it has to do with the large file size; perhaps there's a
>>>> "large memory" compile option that wasn't properly used when the code
>>>> was compiled? I'd be grateful for any insight you might have!
>>>>
>>>> Paul
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Paul Markowski
>>>> Professor of Meteorology
>>>> Department of Meteorology and Atmospheric Science
>>>> Pennsylvania State University
>>>> 520 Walker Building
>>>> University Park, PA 16802
>>>> email: pmarkowski at psu.edu
>>>> phone: 814.865.9736
>>>> web: http://sites.psu.edu/pmarkowski
>>>>
>>>> _______________________________________________
>>>> gradsusr mailing list
>>>> gradsusr at gradsusr.org
>>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>
>>> --
>>> Jennifer Miletta Adams
>>> Center for Ocean-Land-Atmosphere Studies (COLA)
>>> George Mason University
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> gradsusr mailing list
>>> gradsusr at gradsusr.org
>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>
>>
>
> --
> Jennifer Miletta Adams
> Center for Ocean-Land-Atmosphere Studies (COLA)
> George Mason University
>
>
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20170413/42d12b0e/attachment.html
More information about the gradsusr
mailing list