[gradsusr] Problem reading sub-hourly WRF grib files (NCEP code table for 15/30 min records)
Wesley Ebisuzaki - NOAA Federal
wesley.ebisuzaki at noaa.gov
Tue Nov 21 15:47:49 EST 2017
Ivan,
New plan: don't fix g2ctl and alt_g2ctl.
Turns out that the problem was in your grib1 files. Your grib1
20 minute forecast file had items like "20min fcst" and "valid 0-0min".
The last one should have been "valid 10-20min" or "valid 0-20min".
This error makes templating impossible.
Wesley
On Tue, Nov 21, 2017 at 2:42 PM, Wesley Ebisuzaki - NOAA Federal <
wesley.ebisuzaki at noaa.gov> wrote:
> Ivan,
>
> The error message, "add_time: undefined time unit 14", comes from wgrib2.
>
> So you have to be processing the grib2 files.
>
> I don't work with files that are sub-hourly. So the support for
>
> minutes is not robust. There is no support for minutes by
>
> the program grib2ctl.pl. G2ctl has some support for minutes
>
> but it appears to be broken for your files.
>
> Plan: fix g2ctl and alt_g2ctl. I stopped working
>
> on grib1 utilities about 10 years ago.
>
>
> Wesley
>
>
> On Tue, Nov 21, 2017 at 1:20 PM, Ivan Toman <ivtoman at inet.hr> wrote:
>
>> Wesley,
>>
>> Thank you very much for your efforts to help. However I'm still not sure
>> we understand each other well? It is not important to me to use grib2
>> format. I just want to be
>> able to get data from model using GrADS, regardless of format or tools
>> used. But I can't postprocess grib1 also, if I use grib2ctl.pl on grib1
>> format, I again get same error like when I use g2ctl on grib2, as I pasted
>> below.
>>
>> So basically for me neither grib2 nor grib1 work.
>>
>> Ivan
>>
>>
>>
>> On 11/21/2017 04:48 PM, Wesley Ebisuzaki - NOAA Federal wrote:
>>
>> Ivan,
>>
>> I just modified grb1to2.pl to add support for minutes time units. Both
>> g2ctl and alt_g2ctl need some changes to add support for templates
>> that use minutes and forecast minutes. I'll get to it soon.
>>
>> The problem is that you are using cnvgrib to convert from grib1
>> to grib2. Cnvgrib does not handle grib1 files that use a 30-minute
>> time units (value=14) and produces invalid grib2 files.
>>
>> I talked with the person who use to support cnvgrib and he stated that
>> another person complained about a similar problem and that cnvgrib
>> is no longer supported. So you are not going to get the fix from
>> the official source of the program. The fix is straight forward;
>> however, you should reconsider how you do your processing.
>>
>> Wesley
>>
>> On Mon, Nov 20, 2017 at 6:47 PM, Ivan Toman <ivtoman at inet.hr> wrote:
>>
>>> Thank you Wesley,
>>>
>>> but it seems that same problem happens with grib1 format, directly from
>>> UPP postprocessor:
>>>
>>> grib2ctl.pl -verf WRFPRS_d01.00_30 >30.ctl
>>> add_time: undefined time unit 14
>>> PDS_date: problem
>>> add_time: undefined time unit 14
>>> PDS_date: problem
>>>
>>> wgrib WRFPRS_d01.00_20
>>> 1:0:d=17111700:RWMR:kpds5=170:kpds6=109:kpds7=1:TR=10:P1=0:P2=20:TimeU=0:hybrid lev 1:20min fcst:NAve=0
>>> 2:44496:d=17111700:SNMR:kpds5=171:kpds6=109:kpds7=1:TR=10:P1=0:P2=20:TimeU=0:hybrid lev 1:20min fcst:NAve=0
>>> 3:88992:d=17111700:REFC:kpds5=212:kpds6=200:kpds7=0:TR=10:P1=0:P2=20:TimeU=0:atmos col:20min fcst:NAve=0
>>>
>>> But:
>>>
>>> wgrib WRFPRS_d01.00_30
>>> 1:0:d=17111700:RWMR:kpds5=170:kpds6=109:kpds7=1:TR=10:P1=0:P2=1:TimeU=14:hybrid lev 1:30min fcst:NAve=0
>>> 2:44496:d=17111700:SNMR:kpds5=171:kpds6=109:kpds7=1:TR=10:P1=0:P2=1:TimeU=14:hybrid lev 1:30min fcst:NAve=0
>>> 3:88992:d=17111700:REFC:kpds5=212:kpds6=200:kpds7=0:TR=10:P1=0:P2=1:TimeU=14:atmos col:30min fcst:NAve=0
>>>
>>>
>>> I believe, pretty much the same response on time units on grib1 format.
>>>
>>> Is this expected behaviour or grib files look wrong? I uploaded them
>>> here if you want to take a look:
>>> http://gamma.meteoadriatic.net/tmp/jennifer/
>>>
>>> Thx
>>> Ivan
>>>
>>>
>>> On 11/20/2017 01:52 AM, Wesley Ebisuzaki - NOAA Federal wrote:
>>>
>>> Ivan,
>>>
>>> Back from a conference. 14 = 1/2 hour for grib1. However
>>> there isn't a 1/2 hour time unit for grib2.
>>>
>>> http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_table4-4.shtml
>>>
>>> The code needs to be changed to use minutes as the grib2 time units.
>>> Note: grib1 only used 1 byte or 2 bytes to store the forecast length.
>>> Therefore they invented a large number of time units. Grib2 uses
>>> 4 bytes to store the number of time units.
>>>
>>> You can try to convince NCO to fix cnvgrib. I know the person
>>> responsible for cnvgrib and good luck.
>>>
>>> My suggestion #1 doesn't work
>>>
>>> $ wgrib2 wrfprs_d01.00_30.g2 -if ":30 min fcst:" -set_ftime "30 min
>>> fcst" -fi -grib wrfprs_d01.00_30.g2.con
>>> add_time: undefined time unit 14
>>> add_time: undefined time unit 14
>>> add_time: undefined time unit 14
>>>
>>> -if ":30 min fcst:" will always fail because wgrib2 doesn't understand
>>> the time code 14.
>>> The error message "add_time: undefined time unit 14" comes from the
>>> calculation of
>>> the verification time.
>>>
>>> Wesley
>>>
>>>
>>> On Fri, Nov 17, 2017 at 3:34 AM, Ivan Toman <ivtoman at inet.hr> wrote:
>>>
>>>> Wesley,
>>>>
>>>> I think this comes from here:
>>>>
>>>> On grib1 output from UPP - this is for 10min record, all normal:
>>>>
>>>> $ wgrib WRFPRS_d01.00_10
>>>> 1:0:d=17111700:RWMR:kpds5=170:kpds6=109:kpds7=1:TR=10:P1=0:P2=10:TimeU=0:hybrid
>>>> lev 1:10min fcst:NAve=0
>>>> 2:44496:d=17111700:SNMR:kpds5=171:kpds6=109:kpds7=1:TR=10:P1=0:P2=10:TimeU=0:hybrid
>>>> lev 1:10min fcst:NAve=0
>>>> 3:88992:d=17111700:REFC:kpds5=212:kpds6=200:kpds7=0:TR=10:P1=0:P2=10:TimeU=0:atmos
>>>> col:10min fcst:NAve=0
>>>> ...
>>>>
>>>> however, for 30min record:
>>>>
>>>> $ wgrib WRFPRS_d01.00_30
>>>> 1:0:d=17111700:RWMR:kpds5=170:kpds6=109:kpds7=1:TR=10:P1=0:P2=1:TimeU=14:hybrid
>>>> lev 1:30min fcst:NAve=0
>>>> 2:44496:d=17111700:SNMR:kpds5=171:kpds6=109:kpds7=1:TR=10:P1=0:P2=1:TimeU=14:hybrid
>>>> lev 1:30min fcst:NAve=0
>>>> 3:88992:d=17111700:REFC:kpds5=212:kpds6=200:kpds7=0:TR=10:P1=0:P2=1:TimeU=14:atmos
>>>> col:30min fcst:NAve=0
>>>> ...
>>>>
>>>> TimeU=14
>>>>
>>>>
>>>> Where this comes from?
>>>>
>>>> As far as I understand it, UPP follows this time designation for time
>>>> units:
>>>> http://www.nco.ncep.noaa.gov/pmb/docs/on388/table4.html
>>>> 14 = "Half an hour"
>>>>
>>>> This is from UPP code that sets it:
>>>>
>>>> ! J. HALLEY GOTWAY, MODIFY HOW THE TIME INFORMATION IS STORED IN
>>>> ID(17-20),
>>>> ! (FCST TIME UNIT, P1, P2, TIME RANGE INDICATOR).
>>>> ! CHECK IF THE NUMBER OF FORECAST MINUTES IS ZERO FOR HOURS OR
>>>> NON-ZERO FOR
>>>> ! OFF-HOUR FORECASTS. FOR OFF-HOUR FORECASTS, CHECK IF THE TOTAL
>>>> NUMBER
>>>> ! OF MINUTES IS DIVISIBLE BY 30, 15, OR NEITHER, AND USE THE
>>>> APPROPRIATE
>>>> ! FCST TIME UNIT VALUE. FOR ANY FIELD OTHER THAN AN INSTANTANEOUS
>>>> FIELD,
>>>> ! ASSUME ID(18-20), ARE PASSED IN CORRECTLY.
>>>>
>>>> IF(IFMIN .GE. 1)THEN
>>>> ! ID(17) = 0
>>>> ! COMPUTE THE TOTAL FORECAST MINUTES.
>>>> TOTMIN=IFHR*60+IFMIN
>>>>
>>>> ! CHECK FOR 1/2 HOURLY INCREMENTS.
>>>> IF (MOD(TOTMIN, 30) == 0) THEN
>>>> ID(17) = 14
>>>> DIV = 30
>>>> ! CHECK FOR 1/4 HOURLY INCREMENTS.
>>>> ELSEIF (MOD(TOTMIN, 15) == 0) THEN
>>>> ID(17) = 13
>>>> DIV = 15
>>>> ! OTHERWISE, USE MINUTES.
>>>> ELSE
>>>> ID(17) = 0
>>>> DIV = 1
>>>> ENDIF
>>>>
>>>>
>>>> Does this look OK to you?
>>>>
>>>> Thx
>>>> Ivan
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 11/15/2017 02:39 PM, Wesley Ebisuzaki - NOAA Federal wrote:
>>>>
>>>> Ivan,
>>>>
>>>> I am not aware of a time unit 14. It's not in NCEP's documentation.
>>>>
>>>> Wesley
>>>>
>>>> http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_table4-4.shtml
>>>>
>>>> On Wed, Nov 15, 2017 at 4:27 AM, Ivan Toman <ivtoman at inet.hr> wrote:
>>>>
>>>>> Wesley,
>>>>>
>>>>> I finally found time to dig into this.
>>>>>
>>>>> I actually get "undefined time unit 14" from wgrib2 when trying to
>>>>> handle 30 min records. That's probably why g2ctl, alt_g2ctl also does not
>>>>> do the job (throwing this message) and so on.
>>>>>
>>>>> For example, your suggestion #2:
>>>>>
>>>>> $ wgrib2 wrfprs_d01.00_30.g2 -if ":30 min fcst:" -set_ftime "30 min
>>>>> fcst" -fi -grib wrfprs_d01.00_30.g2.con
>>>>> add_time: undefined time unit 14
>>>>> add_time: undefined time unit 14
>>>>> add_time: undefined time unit 14
>>>>> 1:0:d=2017111500 <%28201%29%20711-1500>:RWMR:1 hybrid level:1 ? fcst:
>>>>> add_time: undefined time unit 14
>>>>> add_time: undefined time unit 14
>>>>> add_time: undefined time unit 14
>>>>> 2:2310:d=2017111500 <%28201%29%20711-1500>:SNMR:1 hybrid level:1 ?
>>>>> fcst:
>>>>> add_time: undefined time unit 14
>>>>> ...
>>>>>
>>>>>
>>>>> Or...
>>>>>
>>>>> $ alt_gmp -i wrfprs_d01.grib2.ctl
>>>>> wgrib2_flags=-npts -set_ext_name 1 -end_FT -ext_name -lev
>>>>> wgrib2_inv=.invd01
>>>>> dtype: dtype grib2
>>>>> pdef: pdef 149 149 lccr -29.363000 45.203000 1 1 -21.633000
>>>>> -21.633000 54.288000 12000.000000 12000.000000tdef: nt=7
>>>>> start=00Z15nov2017 by=10mn
>>>>> zdef: nlevel=25
>>>>> resolve_dsets dset=wrfprs_d01.grib2 inctime=10mn
>>>>> resolve_dsets: no template
>>>>> scanning wrfprs_d01.grib2 (process=0)
>>>>> add_time: undefined time unit 14
>>>>> add_time: undefined time unit 14
>>>>> add_time: undefined time unit 14
>>>>> add_time: undefined time unit 14
>>>>> add_time: undefined time unit 14
>>>>>
>>>>>
>>>>>
>>>>> grib1to2.pl did not work either, it failed:
>>>>>
>>>>> $ for i in WRFPRS_d01.0* ; do ./grb1to2.pl -o $i.g2 $i ; done
>>>>> (((rec 1:0:date 2017111500 <%28201%29%20711-1500> RWMR kpds5=170
>>>>> kpds6=109 kpds7=1 levels=(0,1) grid=255 hybrid lev 1 anl:
>>>>> RWMR=Rain water mixing ratio [kg/kg]
>>>>> timerange 0 P1 0 P2 0 TimeU 1 nx 149 ny 149 GDS grid 3 num_in_ave 0
>>>>> missing 0
>>>>> center 7 subcenter 0 process 125 Table 2 scan: WE:SN winds(grid)
>>>>> Lambert Conf: Lat1 -29.363000 Lon1 45.203000 Lov 54.288000
>>>>> Latin1 -21.633000 Latin2 -21.633000 LatSP -90.000000 LonSP
>>>>> 0.000000
>>>>> South Pole (149 x 149) Dx 12.000000 Dy 12.000000 scan 64 mode 136
>>>>> min/max data 0 0 num bits 0 BDS_Ref 0 DecScale 0 BinScale 0
>>>>>
>>>>> )))
>>>>> >> grib2_metadata --- WRFPRS_d01.00_00 wgrib=./wgrib wgrib2=./wgrib2
>>>>> Use of uninitialized value $1 in negation (-) at ./
>>>>> grib1to2_metadata.pl line 74, <INV> line 1.
>>>>> no scan mode at ./grib1to2_metadata.pl line 83, <INV> line 1.
>>>>> missing GRIB record(s)
>>>>> ((()))
>>>>> Problem, winds not defined! old ./wgrib?
>>>>> grib2 message ignored (use wgrib2)
>>>>> missing GRIB record(s)
>>>>> ((()))
>>>>> Problem, winds not defined! old ./wgrib?
>>>>> (((rec 1:0:date 2017111500 <%28201%29%20711-1500> RWMR kpds5=170
>>>>> kpds6=109 kpds7=1 levels=(0,1) grid=255 hybrid lev 1 10min fcst:
>>>>> RWMR=Rain water mixing ratio [kg/kg]
>>>>> timerange 10 P1 0 P2 10 TimeU 0 nx 149 ny 149 GDS grid 3 num_in_ave
>>>>> 0 missing 0
>>>>> center 7 subcenter 0 process 125 Table 2 scan: WE:SN winds(grid)
>>>>> Lambert Conf: Lat1 -29.363000 Lon1 45.203000 Lov 54.288000
>>>>> Latin1 -21.633000 Latin2 -21.633000 LatSP -90.000000 LonSP
>>>>> 0.000000
>>>>> South Pole (149 x 149) Dx 12.000000 Dy 12.000000 scan 64 mode 136
>>>>> min/max data 0 5.31824e-06 num bits 16 BDS_Ref 0 DecScale 11
>>>>> BinScale 4
>>>>>
>>>>>
>>>>>
>>>>> I have no idea how to proceed further.
>>>>> Tks,
>>>>> Ivan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 11/07/2017 08:12 PM, Wesley Ebisuzaki - NOAA Federal wrote:
>>>>>
>>>>> Ivan,
>>>>>
>>>>> "UPP follows NCEP code table for timing grib records - for 15 and 30
>>>>> minute records use codes 13 and 14 respectively instead of simple
>>>>> minutes. This seems to confuse GrADS (or g2ctl/gribmap, I'm not sure)."
>>>>>
>>>>> "Yes it is fixed. I'm currently need to output 10 min intervals. So,
>>>>> for
>>>>> full hour, 10m, 20m, 40m and 50m I get OK data. But for 30m I do not."
>>>>>
>>>>> These quotes suggests that it is a problem with gribmap expecting
>>>>> 30 minute_time_units and not being able to handle 1
>>>>> 30_minute_time_units.
>>>>>
>>>>> Suggestion #1
>>>>>
>>>>> use alt_g2ctl/alt_gmp
>>>>>
>>>>> Both are based on wgrib2 which uses english rather than code table
>>>>> numbers.
>>>>>
>>>>> Suggestion #2
>>>>>
>>>>> convert forecast time from 1 (30 minutes) to 30 (minutes)
>>>>>
>>>>> wgrib2 IN.grb -if ":30 min fcst:" -set_ftime "30 min fcst" -fi
>>>>> -grib OUT.grb
>>>>>
>>>>> Wesley
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Nov 7, 2017 at 1:21 PM, Jeff Duda <jeffduda319 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> What result do you get when you do
>>>>>>
>>>>>> wgrib2 -T (grib file)
>>>>>>
>>>>>> ?
>>>>>>
>>>>>> Look after the "D=", as the represents the time in the GRIB2 file.
>>>>>>
>>>>>> Jeff
>>>>>>
>>>>>> On Tue, Nov 7, 2017 at 12:10 PM, Ivan Toman <ivtoman at inet.hr> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Yes it is fixed. I'm currently need to output 10 min intervals. So,
>>>>>>> for full hour, 10m, 20m, 40m and 50m I get OK data. But for 30m I do not.
>>>>>>>
>>>>>>> Thank you!
>>>>>>> Ivan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 11/07/2017 05:50 PM, Jeff Duda wrote:
>>>>>>>
>>>>>>> What is your time interval? If it's not fixed, that's one problem.
>>>>>>> If it is fixed, then I don't see why regular templating wouldn't work
>>>>>>> unless the time being written to the grib2 file is not right. But there are
>>>>>>> ways to fix that.
>>>>>>>
>>>>>>> Jeff Duda
>>>>>>>
>>>>>>> On Tue, Nov 7, 2017 at 10:12 AM, Ivan Toman <ivtoman at inet.hr> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I'm having difficulties reading sub-hourly time records from WRF
>>>>>>>> grib
>>>>>>>> files in GrADS.
>>>>>>>>
>>>>>>>> UPP follows NCEP code table for timing grib records - for 15 and 30
>>>>>>>> minute records use codes 13 and 14 respectively instead of simple
>>>>>>>> minutes. This seems to confuse GrADS (or g2ctl/gribmap, I'm not
>>>>>>>> sure).
>>>>>>>>
>>>>>>>> What I get as result is that I can read any sub-hourly record as
>>>>>>>> long as
>>>>>>>> it is not 15 or 30 minute record. For those, I get undefined grid
>>>>>>>> instead of data.
>>>>>>>>
>>>>>>>> I use this workflow for postprocessing: wfrout >(UPP)> grib1
>>>>>>>> >(cnvgrib)>
>>>>>>>> grib2 >(g2ctl,gribmap)> GrADS
>>>>>>>>
>>>>>>>> Does anybody know what is going on there?
>>>>>>>>
>>>>>>>> Thank you in advance
>>>>>>>>
>>>>>>>> Ivan Toman
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> gradsusr mailing list
>>>>>>>> gradsusr at gradsusr.org
>>>>>>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Jeff Duda
>>>>>>> Post-doctoral research fellow
>>>>>>> University of Oklahoma School of Meteorology
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> gradsusr mailing listgradsusr at gradsusr.orghttp://gradsusr.org/mailman/listinfo/gradsusr
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> gradsusr mailing list
>>>>>>> gradsusr at gradsusr.org
>>>>>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jeff Duda
>>>>>> Post-doctoral research fellow
>>>>>> University of Oklahoma School of Meteorology
>>>>>>
>>>>>> _______________________________________________
>>>>>> gradsusr mailing list
>>>>>> gradsusr at gradsusr.org
>>>>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> gradsusr mailing listgradsusr at gradsusr.orghttp://gradsusr.org/mailman/listinfo/gradsusr
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> gradsusr mailing list
>>>>> gradsusr at gradsusr.org
>>>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>>>
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> gradsusr mailing listgradsusr at gradsusr.orghttp://gradsusr.org/mailman/listinfo/gradsusr
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> gradsusr mailing list
>>>> gradsusr at gradsusr.org
>>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> gradsusr mailing listgradsusr at gradsusr.orghttp://gradsusr.org/mailman/listinfo/gradsusr
>>>
>>>
>>>
>>> _______________________________________________
>>> gradsusr mailing list
>>> gradsusr at gradsusr.org
>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>
>>>
>>
>>
>> _______________________________________________
>> gradsusr mailing listgradsusr at gradsusr.orghttp://gradsusr.org/mailman/listinfo/gradsusr
>>
>>
>>
>> _______________________________________________
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20171121/8076c593/attachment-0001.html
More information about the gradsusr
mailing list