[gradsusr] Low level I/O error: read error on data file

[Mark Sponsler]

Hi Jennifer,
I've reported the issue to NOAA multiple times. They fix it, but it returns. So at this point I'm working to instead build defenses into my code. 

Your idea below sounds like a good one:

"Well, the file I/O error shouldn’t crash GrADS. You can try a display with ’set gxout stat’ and check if you get a non-zero return code. "

I'll give that a try and see if it can spot the error. If so, then I'll just move to the next time step and repeat till I get to the end of the file.

Great suggestion - Thanks!


On February 4, 2016 10:41:43 AM PST, Jennifer M Adams <jadams21 at gmu.edu> wrote:
>
>On Feb 4, 2016, at 11:02 AM, Mark Sponsler
><msponsler at COMCAST.NET<mailto:msponsler at COMCAST.NET>> wrote:
>
>Hi Jennifer,
>
>Good info..
>
>I don't want to hijack Annas thread but.....
>
>In my instance the error is usually due to the data being corrupted on
>the Dods server in later time steps in the file. I can download and
>process the file fine for weeks, then for whatevet reason it will get
>corrupted and syat that way for most every model run until NOAA
>realizes the error and fixes it. I suspect it's due to some batch
>process that actually puts the files on the dods server.
>That sounds like something you should discuss with the administrator of
>the NOAA GDS.
>
>
>The file is typically 228 meg total, so relatively small in size. 3 hr
>NAM data/4 variables.
>
>Was wondering if there is a way to insert a test of a single variable
>for each timestep that I execute before I try to execute the grads
>display function. It seems some of the time step data is fine, but then
>gets corrupt in later time steps. But I'm guessing just the test would
>cause grads to crash.
>Well, the file I/O error shouldn’t crash GrADS. You can try a display
>with ’set gxout stat’ and check if you get a non-zero return code.
>
>
>The data I've downloaded this morning looks good. But I'll monitor to
>see if there is a significant difference in file size between corrupt
>and 'good' files. Maybe that's another 'safety check' option just
>checking bulk file size).
>
>Any other ideas are welcome.
>It really depends a lot on how you are downloading the data. Your bug
>report to NOAA should have the exact time when the corrupted file was
>downloaded, they could (in theory) check the server logs to see if
>there’s an explanation.
>—Jennifer
>
>On February 4, 2016 5:44:43 AM PST, Jennifer M Adams
><jadams21 at gmu.edu<mailto:jadams21 at gmu.edu>> wrote:
>The “Low level I/O error” message usually means your flat binary file
>is the wrong size. GrADS makes a calculation of where to offset the
>pointer — i.e. how many bytes deep into the file it needs to go to read
>the desired grid. If the file read command returns an error, you get
>the message that tells you how many bytes it was trying to read from
>the file (in this case 1144) and the offset it was using (in this case
>-2129859208, which isn’t right and looks like some kind of precision
>error).
>
>The first thing you should do is make a calculation of what the file
>size should be:
>xsize * ysize * zsize * tsize * esize * number-of-variables * 4 (bytes)
>If your file has some z-varying variables and some non-z-varying
>variables of if your data is sequential this calculation is a little
>trickier, but you get the idea. If your file size doesn’t match, then
>figure out where you went wrong in describing the grid or in creating
>your file. If it does match, then the file may be corrputed somewhere.
>Try displaying one time step at a time and figure out which T index
>causes it to fail — if that T value is where the file size crosses the
>2GB mark, then you probably have a precision issue. GrADS 2.0.2 should
>easily be able to read a big binary file, but it may depend on the
>system you’re using to read it.
>—Jennifer
>
>
>On Feb 1, 2016, at 7:50 AM, Anna Lukianova
><romashe4ka04 at rambler.ru<mailto:romashe4ka04 at rambler.ru>> wrote:
>
>
>Hi, everyone!
>
>I have 2-D data file with a lot of records in time.
>
>When I want to plot variables for the last records of time, I obtain
>such an error  :
>
>
>Low level I/O error: read error on data file. Error reading 1144 bytes
>at location -2129859208/
>
>
>Is this error linked to some file limits in GrADS (I have 2.0.2
>version) ?
>
>
>Thanks in advance, Lukianova Ann.
>
>_______________________________________________
>gradsusr mailing list
>gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
>http://gradsusr.org/mailman/listinfo/gradsusr
>
>--
>Jennifer Miletta Adams
>Center for Ocean-Land-Atmosphere Studies (COLA)
>George Mason University
>
>
>
>
>
>________________________________
>
>gradsusr mailing list
>gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
>http://gradsusr.org/mailman/listinfo/gradsusr
>
>Thanks,
>Mark Sponsler
>
>--
>Jennifer Miletta Adams
>Center for Ocean-Land-Atmosphere Studies (COLA)
>George Mason University

Thanks,
Mark Sponsler
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20160204/f5458d13/attachment-0001.html