[gradsusr] Problem with Ensemble Files
Tom Hultquist
weathertom at gmail.com
Mon Jan 20 17:40:29 EST 2014
Yeah, I saw that but was lazy and hoping g2ctl would do all my work for me :-). But it won't take much effort to get things automated and working again. Thank you!
Jennifer Adams <jma at cola.iges.org> wrote:
>That's good. I does say in the documentation -- "The axis described in
>the TDEF entry is an envelope that spans the time ranges of all
>ensemble members." So if you want to keep the tdef size to match the
>latest forecast, you will have to trim the lengths of the other members
>so they fit in that envelope.
>
>For operational generation of descriptor files, I create a single
>template that has strings like XXX and YYY where the date strings
>should be and then use 'sed' to substitute those with real time dates
>to create a new version of my template that is relevant for the current
>time. I also use the date handling in GrADS scripting to generate those
>substitution strings, or short C programs like the following:
>
>#include <stdio.h>
>#include <string.h>
>#include <time.h>
>
>char *cmons[12] = {"jan","feb","mar","apr","may","jun","jul","aug",
> "sep","oct","nov","dec"};
>
>main () {
>time_t *tsec;
>time_t tt;
>struct tm *tm;
>char sname[50];
>int rc;
>
> tt = time((long *)0);
> tsec = &tt;
> tm = gmtime(tsec);
> if (tm->tm_year>99) tm->tm_year = tm->tm_year-100;
> sprintf (sname,"%02i%s%4i",
> tm->tm_mday,cmons[tm->tm_mon],tm->tm_year+2000);
> printf ("%s\n",sname);
>}
>
>This program returns a string like "20jan2014".
>--Jennifer
>
>
>On Jan 20, 2014, at 4:24 PM, Tom Hultquist wrote:
>
>> Jennifer,
>>
>> Ok, so I did wind up looking at it today after all. In any event,
>your suggestion definitely helped and I "think" it is fixed (at least
>the oddity I noticed before is gone, and the output I've checked thus
>far looks ok). In a nutshell, I needed to change the TDEF definition so
>that it would use the earliest time in any of the members, and adjust
>the number of times up (by 19) to cover the full range of time steps
>(even though each individual member has 149, the range covered by all
>of them is 168). Luckily, I can get g2ctl.pl to do most of this for me
>by simply reversing the order of the names in the -ens templating
>option (oldest first, newest last). This gives me a ctl file that works
>with the exception of needing to change tdef 149... to tdef 168...,
>which I can accomplish easily enough with a quick sed command after
>g2ctl runs. Now I just need to modify the scripts I'd written, since
>all of my time related coding needs to be increased by 19. That's not a
>big deal, will just take a bunch of search/replaces. So, for now I
>think things are ok. Thanks again.
>>
>> tom
>>
>>
>> On Mon, Jan 20, 2014 at 12:36 PM, Jennifer Adams <jma at cola.iges.org>
>wrote:
>> Yes, I agree that the behavior for e03 is weird, but for the purpose
>of debugging, it would be helpful to rule out the other things I point
>out.
>> 1. Make sure that TDEF is big enough to cover all time steps in all
>members.
>> 2. Use wgrib2 to make sure that all data files have the expected
>number of time steps for all variables.
>> 3. Run gribmap with the -v option and see if every record in every
>file gets matched.
>> --Jennifer
>>
>>
>>
>> On Jan 20, 2014, at 12:21 PM, Tom Hultquist wrote:
>>
>>> Jennifer,
>>>
>>> Thanks for the reply. With regard to what you mentioned...
>>>
>>> 1) The TDEF was defined by g2ctl, and seems to match how its
>documentation says it would do things for a time-lagged ensemble
>(whereby it is set to the 1st/most recent member). I can certainly try
>adjusting it though, to see if it helps. It is my intention, however,
>to only do calculations on the overlapping time period, essentially
>from the beginning time of member 1 (most recent) out 4 weeks. The
>previous runs (members 2-20) all overlap that time time period, but
>contain earlier data, since at one time each of them was member #1
>(every six hours that changes). I'm not re-downloading the files, but
>am simply using the "realtime" files that were downloaded earlier. I'll
>play around, and see if I see any more meaningful info from gribmap.
>>>
>>> 2) Yes, I copy/rename the files each run so that the templating is
>easier, just the %e is different between each member.
>>>
>>> 3) I'm not sure of this either, but I am only downloading the
>realtime output (basically 12 hours after run time). So, my older
>members are not re-downloaded, but are simply the files I previously
>downloaded.
>>>
>>> It is just odd that everything works fine, except for what winds up
>as member 3 each time, whether that is from 00, 06, 12, or 18Z run.
>And, when that member becomes member 2, it's fine as well.
>>>
>>> tom
>>>
>>>
>>> On Mon, Jan 20, 2014 at 9:00 AM, Jennifer Adams <jma at cola.iges.org>
>wrote:
>>> Tom,
>>> I have a couple of suggestions:
>>>
>>> 1. You have set your TDEF to cover the entire range of times for
>ensemble e01, but not for the entire span of all the other members. If
>you were to expand your tdef so that it started at 18z28jan2014 (the
>start of e20) and ended with the final time step of e01, then you would
>not see the "valid time outside of file limits" messages. However, if
>it is your intention to only look at data from the older runs (members
>e02 - e20) that overlap with the most recent run, then you should
>change the length of those members from 149 to match the actual number
>of time steps that fit inside the range of e01. Expanding your TDEF to
>include the whole length of all members will clean up the gribmap
>output to make it easier to see if there are other problems.
>>>
>>> 2. You have a fixed date string in your file name
>"/home/mdladmin/CFS/Input/cfspgb.01.20140119%e.grb2" . Are you renaming
>the older forecast files to match this? Make sure that your naming
>convention matches your DSET entry.
>>>
>>> 3. I can't verify this at the moment, but I have noticed that the
>CFS real time data files are not the same when they are brand new
>compared to when they are a couple of days old. I think they add some
>additional time steps after some time has elapsed since the forecast
>was initially posted. In my own operational scripts to grab this data I
>wait for 48 hours before getting the data -- for our purposes, looking
>at data in real time is not strictly necessary, so it's not big deal to
>wait. But see if the number of records in each file in your ensemble
>set really covers the nominal 149 time steps. A careful look at the
>wgrib2 output should help you see if this may be an issue in your case.
>
>>>
>>> --Jennifer
>>>
>>> On Jan 19, 2014, at 10:32 PM, Tom Hultquist wrote:
>>>
>>>> Jennifer,
>>>>
>>>> No problem, I just didn't want to provide too many details
>initially. The GRIB files are from the CFS off of NOMADS. I'm grabbing
>6-hourly files from the 4x per day runs, and just grabbing member 1
>(for a certain window of time, whereby the previous files over the past
>five days will overlap the same time period for which I'm computing
>things). The files are at
>http://nomads.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20140117/00/6hrly_grib_01/
>with YYYYMMDD/CC changing depending on date/cycle downloaded.
>>>>
>>>> The ctl file for the most recent "run" is attached. It works ok
>except for the issue I described (member #3 always causes the issue).
>GrADS is version 2.1.a1, although I also had the issue with 2.0.2. As
>best I can tell g2ctl.pl is 0.0.4o, so perhaps that is an issue (since
>I do see a more recent version, 0.0.8.8 online). I suppose I'll give
>that a shot in the meantime.
>>>>
>>>> tom
>>>>
>>>>
>>>> On Sun, Jan 19, 2014 at 8:39 PM, Jennifer Adams <jma at cola.iges.org>
>wrote:
>>>> Tom,
>>>> Can you please provide the source of your grib files, and the
>descriptor file that doesn't work? I can't resolve a problem that can't
>be reproduced. It would also be useful to know the version of GrADS and
>g2ctl you're using.
>>>> --Jennifer
>>>>
>>>> On Jan 19, 2014, at 3:12 PM, Tom Hultquist wrote:
>>>>
>>>>> I have a set of files with different initial times, whose forecast
>times overlap so that I can use them as an ensemble dataset. I am using
>g2ctl.pl with the -ens option, and it is creating a proper ctl file. I
>thought everything was working fine, but then noticed some oddities in
>a few calculations I was doing. So, I decided to view data from each
>member interactively in GrADS, using the ctl file which was created by
>g2ctl.pl. For some reason, it would not display anything from the third
>ensemble member (entire grid undefined). There was, however, nothing
>wrong with the file, since it worked fine if viewed individually
>outside of the ensemble set. In addition, when a new ctl is created
>later, and that member 3 became member 4, it no longer caused a
>problem. However, the "new" member 3 (which was previously member 2)
>had the same issue. When looking at the gribmap feedback (with -v
>turned on), I can see it runs into issues with valid times being
>outside of file limits (which indeed is true, since the files overlap,
>but don't cover the exact same time frame as member 1). When gribmap
>has this error, it shows it= (e.g. it=10) vs t= for the valid time
>conflict. I couldn't find any info on what it= references. The same
>error happens with subsequent files (members 4-20), but gribmap appears
>to scan those files twice and they display properly for me.
>>>>>
>>>>> Has anyone experienced anything like this, and/or have any
>suggestions?
>>>>> _______________________________________________
>>>>> gradsusr mailing list
>>>>> gradsusr at gradsusr.org
>>>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>>
>>>> --
>>>> Jennifer M. Adams
>>>> Center for Ocean-Land-Atmosphere Studies (COLA)
>>>> 111 Research Hall, Mail Stop 2B3
>>>> George Mason University
>>>> 4400 University Drive
>>>> Fairfax, VA 22030
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> gradsusr mailing list
>>>> gradsusr at gradsusr.org
>>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>>
>>>>
>>>> <CFS_ensemble1.ctl>_______________________________________________
>>>>
>>>> gradsusr mailing list
>>>> gradsusr at gradsusr.org
>>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>
>>>
>>> --
>>> Jennifer M. Adams
>>> Center for Ocean-Land-Atmosphere Studies (COLA)
>>> 111 Research Hall, Mail Stop 2B3
>>> George Mason University
>>> 4400 University Drive
>>> Fairfax, VA 22030
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> gradsusr mailing list
>>> gradsusr at gradsusr.org
>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>
>>>
>>> _______________________________________________
>>> gradsusr mailing list
>>> gradsusr at gradsusr.org
>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>
>> --
>> Jennifer M. Adams
>> Center for Ocean-Land-Atmosphere Studies (COLA)
>> 111 Research Hall, Mail Stop 2B3
>> George Mason University
>> 4400 University Drive
>> Fairfax, VA 22030
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>>
>>
>> _______________________________________________
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>
>--
>Jennifer M. Adams
>Center for Ocean-Land-Atmosphere Studies (COLA)
>111 Research Hall, Mail Stop 2B3
>George Mason University
>4400 University Drive
>Fairfax, VA 22030
>
>
>
>
>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>gradsusr mailing list
>gradsusr at gradsusr.org
>http://gradsusr.org/mailman/listinfo/gradsusr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20140120/ba014029/attachment-0001.html
More information about the gradsusr
mailing list