[gradsusr] Problem with Ensemble Files

Mon Jan 20 13:36:02 EST 2014

Yes, I agree that the behavior for e03 is weird, but for the purpose of debugging, it would be helpful to rule out the other things I point out. 
1. Make sure that TDEF is big enough to cover all time steps in all members. 
2. Use wgrib2 to make sure that all data files have the expected number of time steps for all variables.
3. Run gribmap with the -v option and see if every record in every file gets matched. 
--Jennifer

On Jan 20, 2014, at 12:21 PM, Tom Hultquist wrote:

> Jennifer, 
> 
> Thanks for the reply. With regard to what you mentioned...
> 
> 1) The TDEF was defined by g2ctl, and seems to match how its documentation says it would do things for a time-lagged ensemble (whereby it is set to the 1st/most recent member). I can certainly try adjusting it though, to see if it helps. It is my intention, however, to only do calculations on the overlapping time period, essentially from the beginning time of member 1 (most recent) out 4 weeks. The previous runs (members 2-20) all overlap that time time period, but contain earlier data, since at one time each of them was member #1 (every six hours that changes). I'm not re-downloading the files, but am simply using the "realtime" files that were downloaded earlier. I'll play around, and see if I see any more meaningful info from gribmap.
> 
> 2) Yes, I copy/rename the files each run so that the templating is easier, just the %e is different between each member. 
> 
> 3) I'm not sure of this either, but I am only downloading the realtime output (basically 12 hours after run time). So, my older members are not re-downloaded, but are simply the files I previously downloaded.
> 
> It is just odd that everything works fine, except for what winds up as member 3 each time, whether that is from 00, 06, 12, or 18Z run. And, when that member becomes member 2, it's fine as well.
> 
> tom
> 
> 
> On Mon, Jan 20, 2014 at 9:00 AM, Jennifer Adams <jma at cola.iges.org> wrote:
> Tom,
> I have a couple of suggestions:
> 
> 1. You have set your TDEF to cover the entire range of times for ensemble e01, but not for the entire span of all the other members. If you were to expand your tdef so that it started at 18z28jan2014 (the start of e20) and ended with the final time step of e01, then you would not see the "valid time outside of file limits" messages. However, if it is your intention to only look at data from the older runs (members e02 - e20) that overlap with the most recent run, then you should change the length of those members from 149 to match the actual number of time steps that fit inside the range of e01. Expanding your TDEF to include the whole length of all members will clean up the gribmap output to make it easier to see if there are other problems. 
> 
> 2. You have a fixed date string in your file name "/home/mdladmin/CFS/Input/cfspgb.01.20140119%e.grb2" . Are you renaming the older forecast files to match this? Make sure that your naming convention matches your DSET entry. 
> 
> 3. I can't verify this at the moment, but I have noticed that the CFS real time data files are not the same when they are brand new compared to when they are a couple of days old. I think they add some additional time steps after some time has elapsed since the forecast was initially posted. In my own operational scripts to grab this data I wait for 48 hours before getting the data -- for our purposes, looking at data in real time is not strictly necessary, so it's not big deal to wait. But see if the number of records in each file in your ensemble set really covers the nominal 149 time steps. A careful look at the wgrib2 output should help you see if this may be an issue in your case. 
> 
> --Jennifer
> 
> On Jan 19, 2014, at 10:32 PM, Tom Hultquist wrote:
> 
>> Jennifer,
>> 
>> No problem, I just didn't want to provide too many details initially. The GRIB files are from the CFS off of NOMADS. I'm grabbing 6-hourly files from the 4x per day runs, and just grabbing member 1 (for a certain window of time, whereby the previous files over the past five days will overlap the same time period for which I'm computing things). The files are at http://nomads.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20140117/00/6hrly_grib_01/ with YYYYMMDD/CC changing depending on date/cycle downloaded.
>> 
>> The ctl file for the most recent "run" is attached. It works ok except for the issue I described (member #3 always causes the issue). GrADS is version 2.1.a1, although I also had the issue with 2.0.2. As best I can tell g2ctl.pl is 0.0.4o, so perhaps that is an issue (since I do see a more recent version, 0.0.8.8 online). I suppose I'll give that a shot in the meantime.
>> 
>> tom
>> 
>> 
>> On Sun, Jan 19, 2014 at 8:39 PM, Jennifer Adams <jma at cola.iges.org> wrote:
>> Tom,
>> Can you please provide the source of your grib files, and the descriptor file that doesn't work? I can't resolve a problem that can't be reproduced. It would also be useful to know the version of GrADS and g2ctl you're using.
>> --Jennifer
>> 
>> On Jan 19, 2014, at 3:12 PM, Tom Hultquist wrote:
>> 
>>> I have a set of files with different initial times, whose forecast times overlap so that I can use them as an ensemble dataset. I am using g2ctl.pl with the -ens option, and it is creating a proper ctl file. I thought everything was working fine, but then noticed some oddities in a few calculations I was doing. So, I decided to view data from each member interactively in GrADS, using the ctl file which was created by g2ctl.pl. For some reason, it would not display anything from the third ensemble member (entire grid undefined). There was, however, nothing wrong with the file, since it worked fine if viewed individually outside of the ensemble set. In addition, when a new ctl is created later, and that member 3 became member 4, it no longer caused a problem. However, the "new" member 3 (which was previously member 2) had the same issue. When looking at the gribmap feedback (with -v turned on), I can see it runs into issues with valid times being outside of file limits (which indeed is true, since the files overlap, but don't cover the exact same time frame as member 1). When gribmap has this error, it shows it= (e.g. it=10) vs t= for the valid time conflict. I couldn't find any info on what it= references. The same error happens with subsequent files (members 4-20), but gribmap appears to scan those files twice and they display properly for me.
>>> 
>>> Has anyone experienced anything like this, and/or have any suggestions?
>>> _______________________________________________
>>> gradsusr mailing list
>>> gradsusr at gradsusr.org
>>> http://gradsusr.org/mailman/listinfo/gradsusr
>> 
>> --
>> Jennifer M. Adams
>> Center for Ocean-Land-Atmosphere Studies (COLA)
>> 111 Research Hall, Mail Stop 2B3
>> George Mason University
>> 4400 University Drive
>> Fairfax, VA 22030 
>> 
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>> 
>> 
>> <CFS_ensemble1.ctl>_______________________________________________
>> 
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
> 
> 
> --
> Jennifer M. Adams
> Center for Ocean-Land-Atmosphere Studies (COLA)
> 111 Research Hall, Mail Stop 2B3
> George Mason University
> 4400 University Drive
> Fairfax, VA 22030 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
> 
> 
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr

--
Jennifer M. Adams
Center for Ocean-Land-Atmosphere Studies (COLA)
111 Research Hall, Mail Stop 2B3
George Mason University
4400 University Drive
Fairfax, VA 22030 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20140120/e9ded247/attachment.html 