lats4d

[Wesley Ebisuzaki]

Mike,

      I hate to disappoint you but the NOMADS sites (NCEP, NCDC) don't
use lats4d for the distribution of NARR, global reanalyses and
other model products.  With the recent release of GrADS v2.0a, all the
building blocks are in place for NOMADS to support grib-2.
For many users, grib-2 will be more fun than grib-1.

    Wesley Ebisuzaki


Mike Bosilovich wrote:
> Thanks for the prompt reply Brian,
>
> I realize and greatly appreciate the amount of work you, Jennifer and
> others have done, with very little support. Grads is a great tool,
> hands down. And I tell that to anyone who will listen.
>
> I assumed that you all, as developers, would like feedback on what
> users need to do. Lats4d fills a very important role in my work, and I
> expect others as well. I wanted to pass on a point of view. Certainly,
> I'm not trying to tell you what you must do, only providing feedback
> on the capability that I find useful.
>
> Now, as for what Jennifer said, there was a post that says lats will
> not be supported. That was the impetus for my post. If I
> misunderstood, and there will be lats4d capability in some way, I
> apologize for over reacting.
>
> In any event, I also need to clarify my statement on the new NCEP
> reanalysis. They will have 1000TB of Grib1 format data at NCDC (this
> is still changing, as I understand). In Grib2, it comes down to 500Tb
> or so. For what that's worth, it's still immense. I got the sense that
> they are planning on grib1, because of the lack of utilities with
> grib2 capability.
>
> Again, this is merely my opinion as a user, and I hoped the opinion
> and the information would be useful to the developers.
>
> Sincerely,
>
> Mike Bosilovich
>
> On Jan 29, 2008 10:02 PM, Brian Doty <doty at cola.iges.org
> <mailto:doty at cola.iges.org>> wrote:
>
>     Mike, the new Version 2 of GrADS is being released in alpha version
>     and incomplete so that people would have access to the new GRIB2
>     support.   With no funding for many years and key personnel working
>     limited part time hours we do not have the resources to support old
>     code and also move forward to provide essential new functionality for
>     present and future research needs.  GrADS development first started
>     in the late 1980s and it has taken us several years of work to clean
>     up the 15 years of accumulated mods and patches on top of the
>     original code base in order to come out with a new code base so we
>     could move forward.   I have made every effort over the years to
>     ensure upward compatibility and maintain stability.  There is no way
>     that a 20 year old code base can be patched and hacked for another 20
>     years or even 5 years and maintain stability and still support the
>     100TB data sets you talk about and the other projects probably in the
>     pipeline.   As new versions of data formats appear -- grib2, hdf5,
>     new netcdf on the way, etc ---  we have no choice.  We cannot keep
>     using old libraries and maintain them ourselves.  We cannot patch and
>     hack old code that we didn't write to begin with and which in some
>     cases has restrictive copyrights.
>
>     As I say, version 2 as it stands now is incomplete.   Key areas of
>     functionality which need to be re-implemented have not yet been
>     done.  Other major parts of grads have not yet been upgraded, such as
>     the graphics package.   As Jennifer said in her original email, the
>     output of various data formats is high on the list of things yet to
>     be done...   Brian
>
>     On Jan 29, 2008, at 4:34 PM, Mike Bosilovich wrote:
>
>     >
>     > I have to admit an ignorance to some of the inner workings of Grads
>     > development. But I am quite surprised to hear that lats4d is not
>     > presently supported in Grads 2.
>     >
>     > I have been using Grads for 10 years now (I can still recall the
>     > relief after years of ncargf77 programming :-) Of course, I still
>     > write code when appropriate, but grads and lats4d have been the
>     > main tools in my work. Lats4d fills a critical void. By pointing it
>     > at any grads formatted file (netcdf, hdf, binary or grib), I can
>     > reformat the data to what ever a colleague may need. I can also,
>     > easily, no, effortlessly subset variables space or time. Coupled
>     > with a call to regrid, this subsetting utility is beyond compare.
>     > In a c shell, it can rip through huge data files with simplicity. I
>     > hope this does not sound like exaggeration, lats4d is the strongest
>     > data tool I have used.
>     >
>     > In our office, we have just begun production of a new reanalysis
>     > data product. While it will take some time to complete, we are
>     > beginning to develop examples on how users can access and analyze
>     > the data. Comparison with other reanalyses is obvious and will be
>     > in high demand. To accomplish that, they will need to regrid our
>     > data sets to the existing coarser reanalysis data sets. Or, they
>     > may need to change the format out of our native HDF. The easiest
>     > way to explain to others how to do this is with lats4d.
>     >
>     > There will be 100Tb of reanalysis data available through a GDS.
>     > With Lats4d and gradsdods (or gradsdap), this would be much more
>     > accessible. Users, with some examples, will access the data through
>     > online capabilities, rather than bulk downloading of the native HDF
>     > files (a throttle may be needed for access if that is the preferred
>     > by users). In addition, a plan is being prepared to develop a DVD,
>     > similar to the NCEP reanalyses CD and NARR DVD. Personally, I would
>     > like to see a flavor of grads and lats4d included therein
>     > (admittedly, I have not gotten to discuss this with the grads
>     > developers yet).
>     >
>     > It doesn't stop there. At AMS last week, NCEP and NCDC held a town
>     > hall meeting to discuss their plans for the next reanalyses. They
>     > expect to have nearly 1000Tb of data from three different
>     > reanalyses, and their production has also started. I don't see
>     > storage as a barrier, but bandwidth is. Too many users making too
>     > many big requests will limit accessibility. Lats4d access to their
>     > GDS will become an important function.
>     >
>     > Again, I have to admit I do not know the extent of the issues here.
>     > By necessity, I will have to use versions of grads that include
>     > lats4d not only in my work, but as I show other how to use our
>     > data. I felt the need to speak up and I hope that these issues will
>     > be considered.
>     >
>     > A sincere grads and lats4d user,
>     >
>     > Mike Bosilovich
>     >
>
>