[gradsusr] strange behavior with GrADS 2.0.a8

Tue Aug 31 19:36:24 EDT 2010

Hi John,

Many thanks for the cygwin binaries!  I'm not convinced though that curl is causing the grib2 headache.  I actually downloaded a regular NAM file nam.t00z.awphys00.grb2.tm00 from http://www.ftp.ncep.noaa.gov/data/nccf/com/nam/prod/ using wget, and it also had that junk stuff in it, so my guess is NCEP is putting that in there for some reason.  Junk bytes also appear to be in the RTMA files located at http://weather.noaa.gov/pub/SL.us008001/ST.expr/DF.gr2/DC.ndgd/GT.rtma/AR.conus/ which gives grads2a8 problems, so that's another dataset to look at for testing purposes.  The only possibilities that I can see are that either NCEP is putting it in there, or both wget and curl are putting it in there for selected files for some strange reason.  GFS files from http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/ seem to work fine, and they don't have the junk bytes in them.

Don Van Dyke
NWS Tallahassee, FL

---------------------------------------

Hi Don,

I did some research and it appears that curl is the placing the information in the header.

There is an option using the perl wwwcurl library that suppresses the information, it is CURLOPT_HEADER=0

Since Wesley calls curl from a perl system call and not using the wwwcurl library; I was not successful in figuring out how to call curl and suppress it.

Maybe some curl expert on this list can help out.

I make the Cygwin binaries for Jennifer; although, I believe that Arlindo has the OpenGrads version.

See http://vista.cira.colostate.edu/nco/grads-2.0.a8-i686-pc-cygwin.tar.gz for simply the binaries. You must have Cygwin installed.

I changed to /usr/local and un-tarred them into the bin directory.

John

John Huddleston, PhD
Cooperative Institute for Research in the Atmosphere

------------------------
----- Original Message -----
From: <Don.VanDyke at noaa.gov>
Date: Tuesday, August 31, 2010 2:42 pm
Subject: Re: [gradsusr] strange behavior with GrADS 2.0.a8
To: GrADS Users Forum <gradsusr at gradsusr.org>

> Hi Jennifer/John,
> 
> Interesting, thanks for the replies.  I would have never figured that 
> one out in a million years!  I didn't even know a unix command called 
> 'od' existed, so thanks for the tip there as well.  Interestingly, all 
> of those junk bytes are coming from the original NCEP grib2 file, not 
> my nam1hr.sh script.  The place to download them for testing is at  .  
> They're located in the nam.yyyymmdd folders and have the exact same 
> naming convention as my files.  I downloaded a full file to try to 
> just do a simple plot of tmp2m, and it gave me the same error with 
> grads2a8.  This would also seem to explain why I was only having 
> trouble with the NAM files and not GFS files, because downloading a 
> GFS file and using that 'od' command to look at it reveals no "junk" 
> bytes before the "GRIB" string.  By the way, I'm using the precompiled 
> CentOS5.4-x86_64 binaries on a Red Hat Enterprise 5 computer at work.  
> However, I have cygwin installed at hom
> e and use the precompiled grads2a71 binaries for cygwin, but I would 
> really love if someone donated their GrADS 2.0.a8 precompiled cygwin 
> binaries if they have them. :-)
> 
> To be honest, I have no idea how to remove those junk bytes out of the 
> grib2 files, but if this can be fixed on the next release, then I'll 
> probably just wait for that.  Thanks again for all the help!
> 
> Don Van Dyke
> NWS Tallahassee, FL  
> 
> 
> ----- Original Message -----
> From: Jennifer Adams <jma at cola.iges.org>
> Date: Tuesday, August 31, 2010 10:43 am
> Subject: Re: [gradsusr] strange behavior with GrADS 2.0.a8
> To: GrADS Users Forum <gradsusr at gradsusr.org>
> 
> 
> > Hi, Don --
> > This one was tricky! Thank you for providing all the info I needed 
> to  
> > 
> > diagnose the problem.
> > 
> > First of all, your grib files have 112 bytes of junk at the 
> beginning  
> > 
> > of the file. One of my favorite debugging tools, the unix command 
> 'od' 
> >  
> > gives me this:
> > 
> >  > od -c -N 116 nam.t00z.awphys01.grb2.tm00
> > 0000000   \r  \n   -   -   4   c   7   c   8   0   6   5   3   d   e 
>   
> > 3
> > 0000020   \r  \n   C   o   n   t   e   n   t   -   t   y   p   e   :
> > 0000040    t   e   x   t   /   p   l   a   i   n   ;       c   h   a 
>   
> > r
> > 0000060    s   e   t   =   i   s   o   -   8   8   5   9   -   1  \r 
>  
> > \n
> > 0000100    C   o   n   t   e   n   t   -   r   a   n   g   e   :     
>   
> > b
> > 0000120    y   t   e   s       6   6   9   5   3   5   -   1   0   4 
>   
> > 7
> > 0000140    9   1   0   /   2   6   7   7   4   7   2   2  \r  \n  \r 
>  
> > \n
> > 0000160    G   R   I   B
> > 
> > In other words, your GRIB file has the following text prefixed:
> > 
> > --4c7c80653de3
> > Content-type: text/plain; charset=iso-8859-1
> > Content-range: bytes 669535-1047910/26774722
> > 
> > Check your nam1hr.sh script and see if you can figure out how to get 
>  
> > 
> > rid of that.
> > 
> > Secondly, as far as GrADS is concerned, the GRIB interface changed  
> 
> > between a7 and a8 in order to handle files > 2Gb. This meant we had 
> to 
> >  
> > stop using a routine in the GRIB2 library (seekgb) because it does 
> not 
> >  
> > use 8-byte integers for file position offsets. The work normally 
> done  
> > 
> > by seekgb is coded directly in the GrADS I/O layer for version a8.  
> 
> > Unfortunately, one thing that seekgb does that the new code in a8 
> does 
> >  
> > not is check for junk bytes before the 4-character string "GRIB" at  
> 
> > the beginning of the record. I will fix this for the new release.
> > 
> > In the meanwhile, try to avoid contaminating your GRIB file with 
> junk. 
> >  
> > And thanks for the bug report!
> > --Jennifer
> > 
> > 
> > 
> > 
> > On Aug 31, 2010, at 1:04 AM, Don.VanDyke at noaa.gov wrote:
> > 
> > > Hello,
> > >
> > > I am noticing some strange behavior with GrADS 2.0.a8 when I try 
> to  
> > 
> > > write a netCDF file from the following grib2 files at this link.   
> 
> > >   I created those grib2 files with the nam1hr.sh script at that 
> link 
> >  
> > > following the example shown at  
> > >  .  Basically, they were created from standard NCEP 12 km NAM 
> grib2  
> > 
> > > files, but they only have certain variables contained within them 
> in 
> >  
> > > order to save bandwidth.  My goal is to write those variables to  
> 
> > > netCDF files, but my writenc.gs example script at  
> > >  does not work with grads2a8.  It produces an error message shown 
> at 
> >  
> > >  .
> > >
> > > The interesting thing is that everything works perfectly using the 
>  
> > 
> > > previous version of GrADS (2.0.a7.1).  However, I would like to 
> use  
> > 
> > > the -flt option with the sdfwrite command, and that's only 
> available 
> >  
> > > for 2.0.a8 and higher.  Can anybody explain what I'm doing wrong 
> or  
> > 
> > > if this is a bug introduced with grads2a8?  Thanks!
> > >
> > > Don Van Dyke
> > > NWS Tallahassee, FL
> > > _______________________________________________
> > > gradsusr mailing list
> > > gradsusr at gradsusr.org
> > > 
> > 
> > --
> > Jennifer M. Adams
> > IGES/COLA
> > 4041 Powder Mill Road, Suite 302
> > Calverton, MD 20705
> > jma at cola.iges.org
> > 
> > 
> > 
> > _______________________________________________
> > gradsusr mailing list
> > gradsusr at gradsusr.org> 