Potential bug(?) in GrADS when sub-hour data-file names are generated from template

Wed Jul 5 22:14:18 EDT 2006

Jennifer,

> I hope this clarifies the situation. There's no bug in GrADS, there's
> only a lack of support for a forecast minute template substitution
> string. And that's why we added the extremely general %ch
> substitution,
> to solve tricky problems like this one.

With all due respect, but there is not a single word that grads does
not support such a situation in your documentation (http://
grads.iges.org/grads/gadoc/templates.html). That is why, I will treat
it as a bug in the code.

> So %f2 and %f3 are not going to work for forecast increments less
> than 1hr.

It does not support anything that is not an equal hour increment and/
or does not start on an equal hour. Here is the example:

With:
DSET ^g4.%iy2%im2%id2%ih2%in2_%f2%n2.dat
TDEF   121 LINEAR 00:00Z02JUL2006   65MN

You get:
g4.0607020000_0000.dat
g4.0607020000_0105.dat
g4.0607020000_0210.dat
g4.0607020000_0315.dat
g4.0607020000_0420.dat
g4.0607020000_0525.dat
????????
g4.0607020000_0730.dat
g4.0607020000_0835.dat
g4.0607020000_0940.dat
g4.0607020000_1045.dat
g4.0607020000_1150.dat
g4.0607020000_1255.dat
g4.0607020000_1300.dat
g4.0607020000_1405.dat
g4.0607020000_1510.dat
g4.0607020000_1615.dat
g4.0607020000_1720.dat
g4.0607020000_1825.dat
g4.0607020000_2030.dat
g4.0607020000_2135.dat
g4.0607020000_2240.dat
..........

Note, that minutes are wrong past the missing file (marked with ??????).

> And that's why we added the extremely general %ch substitution,
> to solve tricky problems like this one.

There is nothing tricky about the case. There is simply a mistake in
the code, that was most probably developed when nobody was thinking
about using sub-hour increments and/or off-equal-hour initialization.
Mistakes are normal, refusing to try to fix them is not.

Here are the changes that should be enough for your code to 'support'
forecast time in the template. At least they worked for me just fine.
The bad logic is in the following lines in gautil.c:
line 1799: tdif = (tdif+30)/60;
line 1812: tdif = (tdif+30)/60;

To fix it, simply replace both lines with:
tdif = (tdif+stim.mn)/60;

I tested it with different cases on %in2 and %n2 between 1 and 99 on
Linux. Just for the peace of mind.

It took me roughly 30mn to find/fix it in the code without knowing
the code. You would probably do it in time shorter than it took you
to explain what is not 'supported'. If you find a case/architecture,
where it does not work, I will try to make it work for you. Just let
me know.

I have wasted too much time on this issue already. Unfortunately, I
have problems with compilation of full GrADS package on our version
of Linux, so I will spend more. But, I will ultimately get this one
working too. Hopefully, without need for your 'support'.

Cheers,
Henryk
_______________________________________________________
Henryk Modzelewski, UBC EOS, Atmospheric Science Programme
Contact info: http://www.eos.ubc.ca/~henryk/

"For what are we left with if there remains no mystery?
What hope might we find if we know all of the answers?"
                                         Drizzt Do'Urden
_______________________________________________________

On Jun 30, 2006, at 1:32 PM, Jennifer Adams wrote:

> Henryk (and anyone else who cares about templating minutiae),
>
> The way that GrADS handles templating is to construct an array of
> filenames based on the substitution strings in the DSET entry -- the
> array of filenames correspond to each time step in the TDEF.
>
> In your case, at t=1, the valid time is the base time
> (00:00z21jun2006)
> plus zero. At t=2 the valid time is the base time plus one increment
> (30 mn), thus 00:30z31june2006. In the gafndt routine, the base
> time (a
> structure called stim) and the valid time (a structure called dtim)
> are
> used to determine how to construct the filename string.
>
> The %f2 and %f3 templating strings substitute a forecast hour -- this
> is calculated in gafndt as the difference in minutes between the valid
> time and the base time for any given time step. Here's where the
> 'bug(?)' happens -- the units for the %f2 and %f3 substitution strings
> are hours, so it rounds up to the nearest hour if the difference is
> not
> exactly divisible by 60. Thus for your t=2, a difference of 00:30
> rounds up to a forecast hour of "01", and that's what gets added to
> the
> data file name where the %f2 appears. Then, you have appended the %n,
> which does handle the 30 minute increment, so it appends that to the
> filename and that's how you end up with "...0130..." for your 2nd time
> step.
>
> So %f2 and %f3 are not going to work for forecast increments less than
> 1hr. In your case, you have to use the template substitution strings
> that correspond to the valid time instead of forecast hour. And you
> are
> correct, my earlier suggestion won't work if you don't start at 00z
> and
> if your forecast goes beyond 24 hours.  So you can ignore my previous
> post on this topic.
>
> The way I see it, you have two options for solving this problem:
> 1. Use the "%ch" substitution string and then add 121 CHSUB entries to
> map every single file name to a time step.
> 2. Change your file naming scheme to include a valid time in this
> format: yyyymmddhhmm and use that instead of the 4-digit forecast
> 'hhmm'.
>
> My guess is that option #1 would be easier, since the "0000 0030 0100
> 0130 ..." strings would correspond to times 1,2,3,4 ... no matter what
> the initial time is, and you could use the same set of CHSUB entries
> for every single forecast you generate. The downside of option #1 is
> that you'll have to upgrade to 1.9b4 for your daily use.
>
> I hope this clarifies the situation. There's no bug in GrADS, there's
> only a lack of support for a forecast minute template substitution
> string. And that's why we added the extremely general %ch
> substitution,
> to solve tricky problems like this one.
>
> Jennifer
>
>
>
>
>
> On Jun 28, 2006, at 5:21 PM, Henryk Modzelewski wrote:
>
>> Jennifer,
>>
>> Thanks a lot for the reply.
>>
>> Unfortunately, most of hour grids do not start at 0Z and are 60hr
>> long. Sorry if I miss something, but your suggestion does not seem to
>> work for me.
>>
>> Henryk
>> _______________________________________________________
>> Henryk Modzelewski, UBC EOS, Atmospheric Science Programme
>> Contact info: http://www.eos.ubc.ca/~henryk/
>>
>> "For what are we left with if there remains no mystery?
>> What hope might we find if we know all of the answers?"
>>                                         Drizzt Do'Urden
>> _______________________________________________________
>>
>>
>> On Jun 28, 2006, at 10:27 AM, Jennifer M. Adams wrote:
>>
>>> Henryk,
>>> Thanks for the clear explanation of the problem. I was able to
>>> reproduce it and will dig a little deeper to see why the %f template
>>> isn't getting the hour right. In the meanwhile, change the %f2 in
>>> your template string to %h2 and it should work fine.
>>> Jennifer
>>>
>>> Jennifer M. Adams
>>> IGES/COLA
>>> 4041 Powder Mill Road, Suite 302
>>> Beltsville, MD 20705
>>> jma at cola.iges.org
>>
>>
> --
> Jennifer Miletta Adams
> Center for Ocean-Land-Atmosphere Studies (COLA)
> 4041 Powder Mill Road, Suite 302
> Calverton, MD 20705 USA
> jma at cola.iges.org