Problem to open grib2 Nomad/CFSR data with opengrads

Fri Feb 12 14:11:42 EST 2010

All,

I don't know if this is a related problem. I downloaded model and observation climate files to use with the CFS climate model to adjust the model for bias. Most of the time the monthly climate files work fine. However, it looks like the date in the grib1 files are wrong at times. This example I have below is taken from file available on the 30th of January. The February data, in grib1 clearly has the date of 19810230, an impossible date. If the code used to generate the new CFSR monthly data sets is unchanged then the same error will continue. My guess is that the code increments the month, not looking at the day. 

wgrib -verf -4yr dd.xxxx03_pa.modclim_tl.sflux | more

1:0:d=1981023000:PRATE:kpds5=59:kpds6=1:kpds7=0:TR=133:P1=1:P2=2:TimeU=3:sfc:time?:NAve=22
2:33924:d=1981023000:TMP:kpds5=11:kpds6=1:kpds7=0:TR=133:P1=1:P2=2:TimeU=3:sfc:time?:NAve=22
3:70104:d=1981023000:TMP:kpds5=11:kpds6=105:kpds7=2:TR=133:P1=1:P2=2:TimeU=3:2 m above gnd:time?:NAve=22
4:106284:d=1981023000:DLWRF:kpds5=205:kpds6=1:kpds7=0:TR=133:P1=1:P2=2:TimeU=3:sfc:time?:NAve=22
5:142464:d=1981023000:DSWRF:kpds5=204:kpds6=1:kpds7=0:TR=133:P1=1:P2=2:TimeU=3:sfc:time?:NAve=22
6:176388:d=1981023000:GFLUX:kpds5=155:kpds6=1:kpds7=0:TR=133:P1=1:P2=2:TimeU=3:sfc:time?:NAve=22
7:192936:d=1981023000:HPBL:kpds5=221:kpds6=1:kpds7=0:TR=133:P1=1:P2=2:TimeU=3:sfc:time?:NAve=22
...

As you can see the grib file contains an invalid date of 1981023000. It's this date that causes the problem. When I change the date in the control *ctl file it will not map properly because I get 'Entire Grid Undefined' because gribmap reads the grib file and generates an error that time is out of bounds, no matter what I put into the *ctl file. The problem is the invalid time of 30Feb in the grib file.

I hope that I'm on the right track and not adding to confusion.

Joe Pietrowicz, Ph.D.
Senior Research Scientist
ZedX, Inc.

----- Original Message -----
From: "Jennifer Adams" <jma at COLA.IGES.ORG>
To: GRADSUSR at LIST.CINECA.IT
Sent: Friday, February 12, 2010 1:48:13 PM GMT -05:00 US/Canada Eastern
Subject: Re: Problem to open grib2 Nomad/CFSR data with opengrads

Dear All, 
Everyone at COLA is interested in looking at the new CFS Reanalysis, so you can be sure that GrADS will be able handle it. The -b option for gribmap has not been implemented for grib2 yet, so that will have to be patched quickly. We're working on some other changes to gribmap to handle large (>2GB) grib files, so the timing will be just about right for a new release with these features. Please don't ask when, I don't know. 

As for the CFSR data behind TDS at NCDC, GrADS works fine as a client, but you will likely have to create a descriptor file, especially if the data are not on a regular lat/lon grid and need a PDEF statement. The variable names are very long and need to be aliased. The time axis metadata from TDS may also be incompatible with 'sdfopen'. I was able to open the URL posted earlier in this thread and display several variables. 

I have not had a chance to grab CFSR samples and see where the problems are (as far as GrADS is concerned). It sounds like it will depend a lot on which file/product you are looking at. I know the data wranglers at NCEP and NCDC are working hard to make this data available and usable and it's not an easy job. If you are eager to look at this data, make sure you are using the most recent versions of all the available tools. And if you post questions to the list, be specific about the version you're using and the operating system you're running on as well as the data file (or URL) you're trying to read. 

--Jennifer 

On Feb 12, 2010, at 11:52 AM, Dan Swank wrote: 

Hello, 

I need to take back a few statements I made in 
my previous post. 

I have gotten several CFSR/CFSRMON files to plot successfully. 
Here are some samples: 

http://nomads.ncdc.noaa.gov/data/cfsr/200001/wnd10m.gdas.200001.grb2 
http://nomads.ncdc.noaa.gov/data/cfsr/200001/wnd10m.gdas.200001.ctl 
http://nomads.ncdc.noaa.gov/data/cfsr/200001/pwat.gdas.200001.grb2.idx 

http://nomads.ncdc.noaa.gov/data/cfsr/200001/pwat.gdas.200001.grb2 
http://nomads.ncdc.noaa.gov/data/cfsr/200001/pwat.gdas.200001.ctl 
http://nomads.ncdc.noaa.gov/data/cfsr/200001/pwat.gdas.200001.grb2.idx 

http://nomads.ncdc.noaa.gov/data/cfsrmon/200001/ipvh02.gdas.200001.ctl 
http://nomads.ncdc.noaa.gov/data/cfsrmon/200001/ipvh02.gdas.200001.grb2.idx 
http://nomads.ncdc.noaa.gov/data/cfsrmon/200001/ipvh02.gdas.200001.grb2 

http://nomads.ncdc.noaa.gov/data/cfsrmon/200909/ocnf04.gdas.200909.06Z.grb2 
http://nomads.ncdc.noaa.gov/data/cfsrmon/200909/ocnf04.gdas.200909.06Z.ctl 
http://nomads.ncdc.noaa.gov/data/cfsrmon/200909/ocnf04.gdas.200909.06Z.grb2.idx 

The -b should be used when invoking GrADS2 gribmap. 
The -0 opt should be set when working with monthly means 
(cfsrmon) and diurnal fraction monthlies. 

Here are the known gotchas: 
wgrib2 v.1.8.2 or earlier may produce control files 
with incorrect TDEF's when used with g2ctl.pl . 

The original CFSR was produced in GRIB1 at NCEP. 
NCEP converted this data to GRIB2 for transmission 
and storage purposes. There are possible issues 
in the translation between GRIB1 and GRIB2 [indicator 
of statistical process] header. This affects monthly 
means (which are all statistically processed in some way). 
gribmap's -0 option nulls this *potential* issue by matching 
using the reference time. Again, this issue is unproven. 

The GRIB2 headers contain local extensions 
used by Table 4.10 (-fix_ncep wgrib2 option). 
I had thought this might cause issues with gribmap 
but am now not certain. 

-Dan 

Henrique Barbosa wrote the following on 2/10/2010 1:59 PM: 

Dear Dan Swank, 

Thank you for clarifying the problem. The difference between GDS and TDS 

servers was not clear to me. I have not mentioned in my previous email, 

but I already tried to download the data and write a ctl... 

... and I got the time coordinate error you mentioned! 

I hope the GrADS dev-team will take some time to get the 

CFSR working in GrADS. 

I can only enforce your request. +1 here. 

[]'s 

Henrique 

On Tue, Feb 9, 2010 at 6:34 PM, Dan Swank <Dan.Swank at noaa.gov> wrote: 

Hello, 

GrADS in general is not ready for the CFSR dataset. 

I was able to use gradsdap alpha5 to sdfopen the source 

you listed ~ but this is a THREDDS server, which is 

presenting variable names which are too long for 

grads to handle. [(q file) snipit]: 

relative_humidi 0 t,y,x Relative humidity @ sigma 

relative_humidi 0 t,y,x Relative humidity @ sigma_layer 

relative_humidi 0 t,y,x Relative humidity @ zeroDegC_isotherm 

specific_humidi 0 ty,x Specific humidity @ pressure 

specific_humidi 0 ty,x Specific humidity @ pressure_difference_layer 

storm_relative_ 0 ty,x Storm relative helicity @ 

height_above_ground_layer 

stream_function 0 ty,x Stream function @ pressure 

surface_lifted_ 0 t,y,x Surface Lifted Index @ surface 

temperature 0 ty,x Temperature @ pressure 

temperature_alt 4 t,z,y,x Temperature @ altitude_above_msl 

temperature_pot 0 ty,x Temperature @ potential_vorticity_surface 

temperature_max 0 t,y,x Temperature @ maximum_wind 

So even if you managed to open it, you wouldn't get much further. 

Connecting a GrADS client to a THREDDS server has many known issues. 

It will flat out refuse to identify the x/y coordinates when opening 

non-lat/lon grids served from THREDDS, for instance. Although the 

source you specified IS a lat/lon, some CFSR sets are on 

gaussian coordinates. The problem there is that THREDDS presents 

the native grid x/y's without projecting it to a lat/lon 

like a GDS does. 

Attempting to download a GRIB2 file and creating a control 

file will result in a gribmap2 not being able to MATCH the 

time coordinates because of the new local GRIB table the headers 

use. This is especially evident in the monthly means. This 

pretty much stops us from putting the CFSR behind a GDS2. 

I hope the GrADS dev-team will take some time to get the 

CFSR working in GrADS. 

-Dan Swank 

NCDC-NOMADS Support 

Henrique Barbosa wrote the following on 2/9/2010 12:12 PM: 

Dear All, 

I need help to access CFSR (NCEP reanalysis 3) over opendap with opengrads. 

I get an horrendous error message (see below) when I try to: 

ga-> sdfopen http://nomads.ncdc.noaa.gov/thredds/dodsC/cfsrmon/200912/pgblnl.gdas.200912.grb2 

I am using v2.0.a7.oga.3 downloaded from opengrads website. I do 

have opendap enabled (see output of 'q config' below). 

The strange thing is that I can open other sources of data, for example 

ga-> sdfopen http://nomads.ncep.noaa.gov:9090/dods/gfs/gfs20100111/gfs_00z 

works perfectly and I can plot, subset, etc... 

Any ideas? 

[]s 

Henrique 

-------------------------------------------------------------------------------------------------------------------------------------------------- 

ga-> q config 

Config: v2.0.a7.oga.3 little-endian readline printim grib2 netcdf 

hdf4-sds hdf5 opendap-grids,stn athena geotiff 

Grid Analysis and Display System (GrADS) Version 2.0.a7.oga.3 

Copyright (c) 1988-2009 by Brian Doty and the 

Institute for Global Environment and Society (IGES) 

This program is distributed WITHOUT ANY WARRANTY 

See file COPYRIGHT for more information. 

Built Thu Oct 29 17:57:08 EDT 2009 for x86_64-unknown-linux-gnu 

This version of GrADS has been configured with the following options: 

o Built on a LITTLE ENDIAN machine 

o Command line editing ENABLED 

http://tiswww.case.edu/php/chet/readline/rltop.html 

o printim command for image output ENABLED 

http://www.zlib.net 

http://www.libpng.org/pub/png/libpng.html 

http://www.libgd.org/Main_Page 

o GRIB2 interface ENABLED 

http://www.ijg.org 

http://www.ece.uvic.ca/~mdadams/jasper 

http://www.nco.ncep.noaa.gov/pmb/codes/GRIB2 

g2clib-1.0.5 

o NetCDF interface ENABLED 

http://www.opendap.org 

libnc-dap 4.0.1-beta3-snapshot2009021712 of Mar 3 2009 14:13:33 $ 

o HDF interface ENABLED 

http://hdfgroup.org 

HDF 4.2r3 

HDF5 1.8.2 

o Athena Widget GUI ENABLED 

o OPeNDAP gridded data interface ENABLED 

http://www.opendap.org 

libdap 3.7.10 

o OPeNDAP station data interface ENABLED 

http://iges.org/grads/gadoc/supplibs.html 

libgadap 2.0.oga.1 

o GeoTIFF and KML output ENABLED 

http://www.libtiff.org 

http://geotiff.osgeo.org 

For additional information please consult http://iges.org/grads 

-------------------------------------------------------------------------------------------------------------------------------------------------- 

ga-> sdfopen http://nomads.ncdc.noaa.gov/thredds/dodsC/cfsrmon/2001912/pgblnl.gdas.200912.grb2 

Scanning self-describing file: 

http://nomads.ncdc.noaa.gov/thredds/dodsC/cfsrmon/200912/pgblnl.gdas.200912.grb2 

*** glibc detected *** grads: free(): invalid pointer: 0x00000000018ff354 *** 

======= Backtrace: ========= 

/lib/libc.so.6[0x2b6884e9fcb8] 

/lib/libc.so.6(cfree+0x76)[0x2b6884ea2276] 

grads(_Z15convert_nc_type7nc_typeiiPvS0_+0x2ca)[0x8011ea] 

grads(_ZN7NCArray14extract_valuesEPviiRK12ClientParams+0x2e9)[0x833a05] 

grads(_Z8DODvarioiiPKmS0_PKlPvi+0xc6e)[0x803aea] 

grads(nc3_get_vara_double+0x9e)[0x80e3b8] 

grads(nc_get_vara_double+0x1a)[0x60bcca] 

grads(read_one_dimension+0x89)[0x541ac1] 

grads(gadsdf+0x17d6)[0x540080] 

grads(gasdfopen+0x1d4)[0x53e700] 

grads(gacmd+0xfb2)[0x524504] 

grads(Main+0x4e1)[0x496f01] 

grads(main+0x2e)[0x496a16] 

/lib/libc.so.6(__libc_start_main+0xe6)[0x2b6884e465a6] 

grads(__gxx_personality_v0+0x1a9)[0x496959] 

======= Memory map: ======== 

00400000-00cca000 r-xp 00000000 08:01 604869 

/usr/local/grads-2.0.a7.oga.3/bin/grads 

00dc9000-00dea000 rwxp 008c9000 08:01 604869 

/usr/local/grads-2.0.a7.oga.3/bin/grads 

00dea000-010aa000 rwxp 00dea000 00:00 0 

0180f000-01a04000 rwxp 0180f000 00:00 0 [heap] 

2b6882c7a000-2b6882c9a000 r-xp 00000000 08:01 83056 

/lib/ld-2.9.so 

2b6882c9a000-2b6882c9d000 rwxp 2b6882c9a000 00:00 0 

-- 

Dan Swank 

STG, Incorporated - Government Contractor 

NCDC-NOMADS Project: Software & Data Management 

Data Access Branch 

National Climatic Data Center 

Veach-Baley Federal Building 

151 Patton Avenue 

Asheville, NC 28801-5001 

Email: Dan.Swank at noaa.gov 

Phone: 828-271-4007 

-- 
Dan Swank 
STG, Incorporated - Government Contractor 
NCDC-NOMADS Project: Software & Data Management 
Data Access Branch 
National Climatic Data Center 
Veach-Baley Federal Building 
151 Patton Avenue 
Asheville, NC 28801-5001 
Email: Dan.Swank at noaa.gov 
Phone: 828-271-4007 

-- 
Jennifer M. Adams 
IGES/COLA 
4041 Powder Mill Road, Suite 302 
Calverton, MD 20705 
jma at cola.iges.org 