lats4d

[Jennifer Adams]

We know that the loss of the LATS interface severely impacts many
users. But our intention is to *replace* it with something better,
not to eliminate the functionality completely. Consider this:

set x; set y; set z; set t; set e
set gxout fwrite
set fwrite -nc file.nc
d var
disable fwrite

You've just written out a subset in netcdf with all the necessary
metadata. Other formats such as HDF could be added as a -h4 or -h5
option. When 'var' is part of an unmanageably large grib2 data set
behind a remote server (several possibilities come to mind), you've
just saved yourself a week (or more) of work.

Jennifer


On Jan 30, 2008, at 8:53 AM, Mike Bosilovich wrote:

> Thanks for the prompt reply Brian,
>
> I realize and greatly appreciate the amount of work you, Jennifer
> and others have done, with very little support. Grads is a great
> tool, hands down. And I tell that to anyone who will listen.
>
> I assumed that you all, as developers, would like feedback on what
> users need to do. Lats4d fills a very important role in my work,
> and I expect others as well. I wanted to pass on a point of view.
> Certainly, I'm not trying to tell you what you must do, only
> providing feedback on the capability that I find useful.
>
> Now, as for what Jennifer said, there was a post that says lats
> will not be supported. That was the impetus for my post. If I
> misunderstood, and there will be lats4d capability in some way, I
> apologize for over reacting.
>
> In any event, I also need to clarify my statement on the new NCEP
> reanalysis. They will have 1000TB of Grib1 format data at NCDC
> (this is still changing, as I understand). In Grib2, it comes down
> to 500Tb or so. For what that's worth, it's still immense. I got
> the sense that they are planning on grib1, because of the lack of
> utilities with grib2 capability.
>
> Again, this is merely my opinion as a user, and I hoped the opinion
> and the information would be useful to the developers.
>
> Sincerely,
>
> Mike Bosilovich
>
> On Jan 29, 2008 10:02 PM, Brian Doty <doty at cola.iges.org> wrote:
> Mike, the new Version 2 of GrADS is being released in alpha version
> and incomplete so that people would have access to the new GRIB2
> support.   With no funding for many years and key personnel working
> limited part time hours we do not have the resources to support old
> code and also move forward to provide essential new functionality for
> present and future research needs.  GrADS development first started
> in the late 1980s and it has taken us several years of work to clean
> up the 15 years of accumulated mods and patches on top of the
> original code base in order to come out with a new code base so we
> could move forward.   I have made every effort over the years to
> ensure upward compatibility and maintain stability.  There is no way
> that a 20 year old code base can be patched and hacked for another 20
> years or even 5 years and maintain stability and still support the
> 100TB data sets you talk about and the other projects probably in the
> pipeline.   As new versions of data formats appear -- grib2, hdf5,
> new netcdf on the way, etc ---  we have no choice.  We cannot keep
> using old libraries and maintain them ourselves.  We cannot patch and
> hack old code that we didn't write to begin with and which in some
> cases has restrictive copyrights.
>
> As I say, version 2 as it stands now is incomplete.   Key areas of
> functionality which need to be re-implemented have not yet been
> done.  Other major parts of grads have not yet been upgraded, such as
> the graphics package.   As Jennifer said in her original email, the
> output of various data formats is high on the list of things yet to
> be done...   Brian
>
> On Jan 29, 2008, at 4:34 PM, Mike Bosilovich wrote:
>
> >
> > I have to admit an ignorance to some of the inner workings of Grads
> > development. But I am quite surprised to hear that lats4d is not
> > presently supported in Grads 2.
> >
> > I have been using Grads for 10 years now (I can still recall the
> > relief after years of ncargf77 programming :-) Of course, I still
> > write code when appropriate, but grads and lats4d have been the
> > main tools in my work. Lats4d fills a critical void. By pointing it
> > at any grads formatted file (netcdf, hdf, binary or grib), I can
> > reformat the data to what ever a colleague may need. I can also,
> > easily, no, effortlessly subset variables space or time. Coupled
> > with a call to regrid, this subsetting utility is beyond compare.
> > In a c shell, it can rip through huge data files with simplicity. I
> > hope this does not sound like exaggeration, lats4d is the strongest
> > data tool I have used.
> >
> > In our office, we have just begun production of a new reanalysis
> > data product. While it will take some time to complete, we are
> > beginning to develop examples on how users can access and analyze
> > the data. Comparison with other reanalyses is obvious and will be
> > in high demand. To accomplish that, they will need to regrid our
> > data sets to the existing coarser reanalysis data sets. Or, they
> > may need to change the format out of our native HDF. The easiest
> > way to explain to others how to do this is with lats4d.
> >
> > There will be 100Tb of reanalysis data available through a GDS.
> > With Lats4d and gradsdods (or gradsdap), this would be much more
> > accessible. Users, with some examples, will access the data through
> > online capabilities, rather than bulk downloading of the native HDF
> > files (a throttle may be needed for access if that is the preferred
> > by users). In addition, a plan is being prepared to develop a DVD,
> > similar to the NCEP reanalyses CD and NARR DVD. Personally, I would
> > like to see a flavor of grads and lats4d included therein
> > (admittedly, I have not gotten to discuss this with the grads
> > developers yet).
> >
> > It doesn't stop there. At AMS last week, NCEP and NCDC held a town
> > hall meeting to discuss their plans for the next reanalyses. They
> > expect to have nearly 1000Tb of data from three different
> > reanalyses, and their production has also started. I don't see
> > storage as a barrier, but bandwidth is. Too many users making too
> > many big requests will limit accessibility. Lats4d access to their
> > GDS will become an important function.
> >
> > Again, I have to admit I do not know the extent of the issues here.
> > By necessity, I will have to use versions of grads that include
> > lats4d not only in my work, but as I show other how to use our
> > data. I felt the need to speak up and I hope that these issues will
> > be considered.
> >
> > A sincere grads and lats4d user,
> >
> > Mike Bosilovich
> >
>

--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma at cola.iges.org



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20080130/199d2107/attachment.html