OT: wgrib to check grib errors?

[Dave Allured]

Federico,

I had a similar question last year.  I never found a fully satisfying
answer.  However, a colleague made this brief script that worked well
enough for my application:

#!/bin/sh
FILESIZE=`ls -Ll $1 | awk '{print $5}'`
SKIPCNT=`expr ${FILESIZE} - 4`
# echo FILESIZE=$FILESIZE SKIPCNT=$SKIPCNT
FILENAME=lastfour$$
rm -f /var/tmp/${FILENAME}
dd if=$1 ibs=1 iseek=$SKIPCNT of=/var/tmp/${FILENAME} >/dev/null 2>&1
LAST4=`cat /var/tmp/${FILENAME}`
if [ "x${LAST4}" = "x7777" ] ; then
     exit 0
else
     exit 1
fi
rm -f /var/tmp/${FILENAME}

Notes:

1.  Here's how it works.  The script simply checks to see whether the
last four bytes of the file are ASCII "7777".  This is the expected
value of the end message on the last data record of any standard grib
file.  (See grib docs.)

2.  This script returns the Unix result codes ($status) that you requested:

   1 if the grib file is ok
   0 if the grib file is corrupted

3.  The ONLY thing that is checked is whether a grib file has been
truncated.  The script does not look at other available measures of
internal consistency, such as good looking start and end messages on
every grib record, or record byte counts.

However, in my experience with many grib files downloaded across the
internet, the most common type of corruption is file truncation.  In
fact I don't think I have ever seen a corrupted grib file that was not
also truncated.  Therefore, this simple method is probably "good enough"
for applications like yours.

4.  If by any chance you are using ECMWF data, many of their files are
padded with an arbitrary number of zero bytes (nulls) after the 7777 end
message.  The above script will need enhancement before it can correctly
judge such files.

5.  Once I experimented with throwing some garbage binary files at
wgrib.  I got highly inconsistent results including crashes (seg faults)
IIRC.  You may want to avoid using wgrib as a first screen, for this reason.

HTH.

--Dave A.
CDC/CIRES/NOAA


Federico Ceccarelli wrote:
> Hi everybody,
>
> I receive daily grib files from a numerical model by ftp.
>
> I receive a single file for every variable and every timestep, so I have
> to combine all small grib files in a bigger file before processing it.
>
> Sometimes it happens that some small  grib file seems to be somehow
> corrupted, and analyzing it with wgrib I receive a message like:
>
> 'missing end section'
>
> Have you ever experienced a problem such this? IS there a way to fix it?
>
> Is there a result code that wgrib gives back referred to the quality of
> the grib files?
>
> for example:
> 1 if the grib file is ok
> 0 if the grib file is corrupted
>
> so that I could cut away the corrupted files before joining them in a
> bigger grib file...
>
> thanks in advance!
>
> federico