[gradsusr] Maximum data file size

[Eric Aligo]

Paul, Chris,

I also have the same problems, and I too am looking forward to the bug fix.

Eric
On 05/31/2017 12:29 PM, Paul Markowski wrote:
> Hi Chris,
>
> This same error was the subject of a discussion on April 13. I've 
> pasted Jennifer Adams' reply below.  It appears to be a subtle Grads 
> bug (maybe "bug" isn't the right word--perhaps "limitation" is 
> better?).  Jennifer, any updates on where things stand regarding a 
> patch?  Thanks!!
>
> Paul
>
>
> On Apr 13, 2017, at 7:25 PM, Jennifer M Adams <jadams21 at gmu.edu 
> <mailto:jadams21 at gmu.edu>> wrote:
>
> I figured out what the problem is. For each variable in a binary file, 
> GrADS calculates an offset — the number of bytes that you must skip 
> over before you read that particular variable. The offset is for the 
> first variable in the file is 0. For each successive variable, it is 
> the offset from the previous variable plus its nlevs*nlons*nlats. In 
> your data file, nlevs is very large (127), and the xsize and ysize are 
> also large enough so that the offset grows very quickly. 
> Unfortunately, it is stored internally as a 4-byte integer, and after 
> the 31st variable, it runs out of bits and the offset values become 
> garbage. I added a debugging print statement which shows the offset 
> for each variable, note it goes bad for the last 6 variables:
>
> cref   0
> zh   573160
> th   73364480
> thpert   146155800
>  prs   218947120
>  prspert   291738440
> qv   364529760
> qc   437321080
> qr   510112400
> qi   582903720
> qs   655695040
> qg   728486360
>  qhl   801277680
>  ccn   874069000
>  ccw   946860320
>  crw   1019651640
>  cci   1092442960
>  csw   1165234280
>  chw   1238025600
>  chl   1310816920
>  vhw   1383608240
>  vhl   1456399560
>  dbz   1529190880
>  uinterp   1601982200
>  uturb   1674773520
>  udiff   1747564840
> upwp   1820356160
>  vinterp   1893147480
>  vturb   1965938800
>  vdiff   2038730120
> vpwp   2111521440
>  winterp   -2110654536
>  wturb   -2037863216
>  wdiff   -1965071896
>  xvort   -1892280576
>  yvort   -1819489256
>  zvort   -1746697936
>
> So, I will have to patch this bug and use an 8-byte integer to store 
> the offset. Until then, your workaround is to write out fewer Z 
> levels, or separate the variables into two distinct files. The output 
> from ‘q file’ was very helpful, and so was the clue about the final 
> six variables showing the error.
> —Jennifer
>
>
>
>
> On 5/31/17, 11:20 AM, Chris Nowotarski wrote:
>> Hello,
>>
>> I'm running grads 2.0.2 and I appear to be running into file size 
>> limitations.  Is there a maximum file size (or record number) for 
>> binary files that are interpretable by Grads?  I had heard there was 
>> previously a 2 GB limit, but that this limit is alleviated in newer 
>> versions of Grads.  I seem to have an issue once files exceed 8 GB.  
>> Is there any way around this besides breaking up files?
>>
>> -Chris
>>
>> -- 
>> Chris Nowotarski
>> Assistant Professor
>> Department of Atmospheric Sciences
>> Texas A&M University
>> MS 3150
>> College Station, TX 77843
>> 979-845-3305
>>
>>
>> _______________________________________________
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>
>
> -- 
> Paul Markowski
> Professor of Meteorology
> Department of Meteorology and Atmospheric Science
> Pennsylvania State University
> 520 Walker Building
> University Park, PA 16802
> email:pmarkowski at psu.edu
> phone: 814.865.9736
> web:http://sites.psu.edu/pmarkowski
>
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr


-- 
Dr. Eric Aligo
I. M. Systems Group, Regional Model Developer
NCEP/EMC
5830 University Research Court
College Park, MD  20740, USA
Phone (301) 683-3686

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20170602/a7d1f08c/attachment.html