[gradsusr] Problem with Ensemble Files

Mon Jan 20 13:51:53 EST 2014

Yep, will do, thanks again (just won't get to it today).

On Mon, Jan 20, 2014 at 12:36 PM, Jennifer Adams <jma at cola.iges.org> wrote:

> Yes, I agree that the behavior for e03 is weird, but for the purpose of
> debugging, it would be helpful to rule out the other things I point out.
> 1. Make sure that TDEF is big enough to cover all time steps in all
> members.
> 2. Use wgrib2 to make sure that all data files have the expected number of
> time steps for all variables.
> 3. Run gribmap with the -v option and see if every record in every file
> gets matched.
> --Jennifer
>
>
>
> On Jan 20, 2014, at 12:21 PM, Tom Hultquist wrote:
>
> Jennifer,
>
> Thanks for the reply. With regard to what you mentioned...
>
> 1) The TDEF was defined by g2ctl, and seems to match how its documentation
> says it would do things for a time-lagged ensemble (whereby it is set to
> the 1st/most recent member). I can certainly try adjusting it though, to
> see if it helps. It is my intention, however, to only do calculations on
> the overlapping time period, essentially from the beginning time of member
> 1 (most recent) out 4 weeks. The previous runs (members 2-20) all overlap
> that time time period, but contain earlier data, since at one time each of
> them was member #1 (every six hours that changes). I'm not re-downloading
> the files, but am simply using the "realtime" files that were downloaded
> earlier. I'll play around, and see if I see any more meaningful info from
> gribmap.
>
> 2) Yes, I copy/rename the files each run so that the templating is easier,
> just the %e is different between each member.
>
> 3) I'm not sure of this either, but I am only downloading the realtime
> output (basically 12 hours after run time). So, my older members are not
> re-downloaded, but are simply the files I previously downloaded.
>
> It is just odd that everything works fine, except for what winds up as
> member 3 each time, whether that is from 00, 06, 12, or 18Z run. And, when
> that member becomes member 2, it's fine as well.
>
> tom
>
>
> On Mon, Jan 20, 2014 at 9:00 AM, Jennifer Adams <jma at cola.iges.org> wrote:
>
>> Tom,
>> I have a couple of suggestions:
>>
>> 1. You have set your TDEF to cover the entire range of times for ensemble
>> e01, but not for the entire span of all the other members. If you were to
>> expand your tdef so that it started at 18z28jan2014 (the start of e20) and
>> ended with the final time step of e01, then you would not see the "valid
>> time outside of file limits" messages. However, if it is your intention to
>> only look at data from the older runs (members e02 - e20) that overlap with
>> the most recent run, then you should change the length of those members
>> from 149 to match the actual number of time steps that fit inside the range
>> of e01. Expanding your TDEF to include the whole length of all members will
>> clean up the gribmap output to make it easier to see if there are other
>> problems.
>>
>> 2. You have a fixed date string in your file name
>> "/home/mdladmin/CFS/Input/cfspgb.01.20140119%e.grb2" . Are you renaming the
>> older forecast files to match this? Make sure that your naming convention
>> matches your DSET entry.
>>
>> 3. I can't verify this at the moment, but I have noticed that the CFS
>> real time data files are not the same when they are brand new compared to
>> when they are a couple of days old. I think they add some additional time
>> steps after some time has elapsed since the forecast was initially posted.
>> In my own operational scripts to grab this data I wait for 48 hours before
>> getting the data -- for our purposes, looking at data in real time is not
>> strictly necessary, so it's not big deal to wait. But see if the number of
>> records in each file in your ensemble set really covers the nominal 149
>> time steps. A careful look at the wgrib2 output should help you see if this
>> may be an issue in your case.
>>
>> --Jennifer
>>
>> On Jan 19, 2014, at 10:32 PM, Tom Hultquist wrote:
>>
>> Jennifer,
>>
>> No problem, I just didn't want to provide too many details initially. The
>> GRIB files are from the CFS off of NOMADS. I'm grabbing 6-hourly files from
>> the 4x per day runs, and just grabbing member 1 (for a certain window of
>> time, whereby the previous files over the past five days will overlap the
>> same time period for which I'm computing things). The files are at
>> http://nomads.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20140117/00/6hrly_grib_01/with YYYYMMDD/CC changing depending on date/cycle downloaded.
>>
>> The ctl file for the most recent "run" is attached. It works ok except
>> for the issue I described (member #3 always causes the issue). GrADS is
>> version 2.1.a1, although I also had the issue with 2.0.2. As best I can
>> tell g2ctl.pl is 0.0.4o, so perhaps that is an issue (since I do see a
>> more recent version, 0.0.8.8 online). I suppose I'll give that a shot in
>> the meantime.
>>
>> tom
>>
>>
>> On Sun, Jan 19, 2014 at 8:39 PM, Jennifer Adams <jma at cola.iges.org>wrote:
>>
>>> Tom,
>>> Can you please provide the source of your grib files, and the descriptor
>>> file that doesn't work? I can't resolve a problem that can't be reproduced.
>>> It would also be useful to know the version of GrADS and g2ctl you're using.
>>> --Jennifer
>>>
>>> On Jan 19, 2014, at 3:12 PM, Tom Hultquist wrote:
>>>
>>> I have a set of files with different initial times, whose forecast times
>>> overlap so that I can use them as an ensemble dataset. I am using
>>> g2ctl.pl with the -ens option, and it is creating a proper ctl file. I
>>> thought everything was working fine, but then noticed some oddities in a
>>> few calculations I was doing. So, I decided to view data from each member
>>> interactively in GrADS, using the ctl file which was created by g2ctl.pl.
>>> For some reason, it would not display anything from the third ensemble
>>> member (entire grid undefined). There was, however, nothing wrong with the
>>> file, since it worked fine if viewed individually outside of the ensemble
>>> set. In addition, when a new ctl is created later, and that member 3 became
>>> member 4, it no longer caused a problem. However, the "new" member 3 (which
>>> was previously member 2) had the same issue. When looking at the gribmap
>>> feedback (with -v turned on), I can see it runs into issues with valid
>>> times being outside of file limits (which indeed is true, since the files
>>> overlap, but don't cover the exact same time frame as member 1). When
>>> gribmap has this error, it shows it= (e.g. it=10) vs t= for the valid time
>>> conflict. I couldn't find any info on what it= references. The same error
>>> happens with subsequent files (members 4-20), but gribmap appears to scan
>>> those files twice and they display properly for me.
>>>
>>> Has anyone experienced anything like this, and/or have any suggestions?
>>> _______________________________________________
>>> gradsusr mailing list
>>> gradsusr at gradsusr.org
>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>
>>>
>>>  --
>>> Jennifer M. Adams
>>> Center for Ocean-Land-Atmosphere Studies (COLA)
>>> 111 Research Hall, Mail Stop 2B3
>>> George Mason University
>>> 4400 University Drive
>>> Fairfax, VA 22030
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> gradsusr mailing list
>>> gradsusr at gradsusr.org
>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>
>>>
>> <CFS_ensemble1.ctl>_______________________________________________
>>
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>>
>>
>>  --
>> Jennifer M. Adams
>> Center for Ocean-Land-Atmosphere Studies (COLA)
>> 111 Research Hall, Mail Stop 2B3
>> George Mason University
>> 4400 University Drive
>> Fairfax, VA 22030
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>>
>>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
>
>
>  --
> Jennifer M. Adams
> Center for Ocean-Land-Atmosphere Studies (COLA)
> 111 Research Hall, Mail Stop 2B3
> George Mason University
> 4400 University Drive
> Fairfax, VA 22030
>
>
>
>
>
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20140120/d7eb3fb5/attachment.html 