[gradsusr] Maximum data file size

[Paul Markowski]

Hi Chris,

This same error was the subject of a discussion on April 13.  I've 
pasted Jennifer Adams' reply below.  It appears to be a subtle Grads bug 
(maybe "bug" isn't the right word--perhaps "limitation" is better?).  
Jennifer, any updates on where things stand regarding a patch?  Thanks!!

Paul


On Apr 13, 2017, at 7:25 PM, Jennifer M Adams <jadams21 at gmu.edu 
<mailto:jadams21 at gmu.edu>> wrote:

I figured out what the problem is. For each variable in a binary file, 
GrADS calculates an offset --- the number of bytes that you must skip 
over before you read that particular variable. The offset is for the 
first variable in the file is 0. For each successive variable, it is the 
offset from the previous variable plus its nlevs*nlons*nlats. In your 
data file, nlevs is very large (127), and the xsize and ysize are also 
large enough so that the offset grows very quickly. Unfortunately, it is 
stored internally as a 4-byte integer, and after the 31st variable, it 
runs out of bits and the offset values become garbage. I added a 
debugging print statement which shows the offset for each variable, note 
it goes bad for the last 6 variables:

     cref   0
       zh   573160
       th   73364480
   thpert   146155800
      prs   218947120
  prspert   291738440
       qv   364529760
       qc   437321080
       qr   510112400
       qi   582903720
       qs   655695040
       qg   728486360
      qhl   801277680
      ccn   874069000
      ccw   946860320
      crw   1019651640
      cci   1092442960
      csw   1165234280
      chw   1238025600
      chl   1310816920
      vhw   1383608240
      vhl   1456399560
      dbz   1529190880
  uinterp   1601982200
    uturb   1674773520
    udiff   1747564840
     upwp   1820356160
  vinterp   1893147480
    vturb   1965938800
    vdiff   2038730120
     vpwp   2111521440
  winterp   -2110654536
    wturb   -2037863216
    wdiff   -1965071896
    xvort   -1892280576
    yvort   -1819489256
    zvort   -1746697936

So, I will have to patch this bug and use an 8-byte integer to store the 
offset. Until then, your workaround is to write out fewer Z levels, or 
separate the variables into two distinct files. The output from 'q file' 
was very helpful, and so was the clue about the final six variables 
showing the error.
---Jennifer




On 5/31/17, 11:20 AM, Chris Nowotarski wrote:
> Hello,
>
> I'm running grads 2.0.2 and I appear to be running into file size 
> limitations.  Is there a maximum file size (or record number) for 
> binary files that are interpretable by Grads?  I had heard there was 
> previously a 2 GB limit, but that this limit is alleviated in newer 
> versions of Grads.  I seem to have an issue once files exceed 8 GB.  
> Is there any way around this besides breaking up files?
>
> -Chris
>
> -- 
> Chris Nowotarski
> Assistant Professor
> Department of Atmospheric Sciences
> Texas A&M University
> MS 3150
> College Station, TX 77843
> 979-845-3305
>
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr


-- 
Paul Markowski
Professor of Meteorology
Department of Meteorology and Atmospheric Science
Pennsylvania State University
520 Walker Building
University Park, PA 16802
email: pmarkowski at psu.edu
phone: 814.865.9736
web: http://sites.psu.edu/pmarkowski

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20170531/40f8a5fd/attachment.html