lats4d

[Glenn.Rutledge]

Hello Mike and all-
I'm the lucky guy in the middle of planning for the receipt of the CFSRR
(the next reanal) at NCDC.  I have asked the data be provided to us in
Grib2 to save space.  The number you are stating are correct-
approximately 915TB in Grib1 so there is an approximate 40-50%
reduction.   NCDC and NCEP conducted a User Workshop at AMS Town Hall on
Jan 21.  The slides I prepared for our Director and working with NCEP
are attached (and the numbers are listed as _Grib1_ for consistency)

What I and others here planning for this dataset is that we would like
to hear how some of these data could be best organized.  Our suggested
file level is attached in the presentation (not sure gradsusr list will
pass the presentation thru however).  In any event- NCDC plans to make
as much of this avbl when it's released to the public in June of 2009
(as I understand it from NCEP).  SOme I want on disk (about 150TB for
on-line access) via NOMADS, and the rest will have to be staged off from
tape.  Hope this helps. Glenn

Mike Bosilovich wrote the following on 1/30/2008 8:53 AM:
> Thanks for the prompt reply Brian,
>
> I realize and greatly appreciate the amount of work you, Jennifer and
> others have done, with very little support. Grads is a great tool,
> hands down. And I tell that to anyone who will listen.
>
> I assumed that you all, as developers, would like feedback on what
> users need to do. Lats4d fills a very important role in my work, and I
> expect others as well. I wanted to pass on a point of view. Certainly,
> I'm not trying to tell you what you must do, only providing feedback
> on the capability that I find useful.
>
> Now, as for what Jennifer said, there was a post that says lats will
> not be supported. That was the impetus for my post. If I
> misunderstood, and there will be lats4d capability in some way, I
> apologize for over reacting.
>
> In any event, I also need to clarify my statement on the new NCEP
> reanalysis. They will have 1000TB of Grib1 format data at NCDC (this
> is still changing, as I understand). In Grib2, it comes down to 500Tb
> or so. For what that's worth, it's still immense. I got the sense that
> they are planning on grib1, because of the lack of utilities with
> grib2 capability.
>
> Again, this is merely my opinion as a user, and I hoped the opinion
> and the information would be useful to the developers.
>
> Sincerely,
>
> Mike Bosilovich
>
> On Jan 29, 2008 10:02 PM, Brian Doty <doty at cola.iges.org
> <mailto:doty at cola.iges.org>> wrote:
>
>     Mike, the new Version 2 of GrADS is being released in alpha version
>     and incomplete so that people would have access to the new GRIB2
>     support.   With no funding for many years and key personnel working
>     limited part time hours we do not have the resources to support old
>     code and also move forward to provide essential new functionality for
>     present and future research needs.  GrADS development first started
>     in the late 1980s and it has taken us several years of work to clean
>     up the 15 years of accumulated mods and patches on top of the
>     original code base in order to come out with a new code base so we
>     could move forward.   I have made every effort over the years to
>     ensure upward compatibility and maintain stability.  There is no way
>     that a 20 year old code base can be patched and hacked for another 20
>     years or even 5 years and maintain stability and still support the
>     100TB data sets you talk about and the other projects probably in the
>     pipeline.   As new versions of data formats appear -- grib2, hdf5,
>     new netcdf on the way, etc ---  we have no choice.  We cannot keep
>     using old libraries and maintain them ourselves.  We cannot patch and
>     hack old code that we didn't write to begin with and which in some
>     cases has restrictive copyrights.
>
>     As I say, version 2 as it stands now is incomplete.   Key areas of
>     functionality which need to be re-implemented have not yet been
>     done.  Other major parts of grads have not yet been upgraded, such as
>     the graphics package.   As Jennifer said in her original email, the
>     output of various data formats is high on the list of things yet to
>     be done...   Brian
>
>     On Jan 29, 2008, at 4:34 PM, Mike Bosilovich wrote:
>
>     >
>     > I have to admit an ignorance to some of the inner workings of Grads
>     > development. But I am quite surprised to hear that lats4d is not
>     > presently supported in Grads 2.
>     >
>     > I have been using Grads for 10 years now (I can still recall the
>     > relief after years of ncargf77 programming :-) Of course, I still
>     > write code when appropriate, but grads and lats4d have been the
>     > main tools in my work. Lats4d fills a critical void. By pointing it
>     > at any grads formatted file (netcdf, hdf, binary or grib), I can
>     > reformat the data to what ever a colleague may need. I can also,
>     > easily, no, effortlessly subset variables space or time. Coupled
>     > with a call to regrid, this subsetting utility is beyond compare.
>     > In a c shell, it can rip through huge data files with simplicity. I
>     > hope this does not sound like exaggeration, lats4d is the strongest
>     > data tool I have used.
>     >
>     > In our office, we have just begun production of a new reanalysis
>     > data product. While it will take some time to complete, we are
>     > beginning to develop examples on how users can access and analyze
>     > the data. Comparison with other reanalyses is obvious and will be
>     > in high demand. To accomplish that, they will need to regrid our
>     > data sets to the existing coarser reanalysis data sets. Or, they
>     > may need to change the format out of our native HDF. The easiest
>     > way to explain to others how to do this is with lats4d.
>     >
>     > There will be 100Tb of reanalysis data available through a GDS.
>     > With Lats4d and gradsdods (or gradsdap), this would be much more
>     > accessible. Users, with some examples, will access the data through
>     > online capabilities, rather than bulk downloading of the native HDF
>     > files (a throttle may be needed for access if that is the preferred
>     > by users). In addition, a plan is being prepared to develop a DVD,
>     > similar to the NCEP reanalyses CD and NARR DVD. Personally, I would
>     > like to see a flavor of grads and lats4d included therein
>     > (admittedly, I have not gotten to discuss this with the grads
>     > developers yet).
>     >
>     > It doesn't stop there. At AMS last week, NCEP and NCDC held a town
>     > hall meeting to discuss their plans for the next reanalyses. They
>     > expect to have nearly 1000Tb of data from three different
>     > reanalyses, and their production has also started. I don't see
>     > storage as a barrier, but bandwidth is. Too many users making too
>     > many big requests will limit accessibility. Lats4d access to their
>     > GDS will become an important function.
>     >
>     > Again, I have to admit I do not know the extent of the issues here.
>     > By necessity, I will have to use versions of grads that include
>     > lats4d not only in my work, but as I show other how to use our
>     > data. I felt the need to speak up and I hope that these issues will
>     > be considered.
>     >
>     > A sincere grads and lats4d user,
>     >
>     > Mike Bosilovich
>     >
>
>

--
Glenn K. Rutledge
Services Team Leader
Remote Sensing and Applications Division
NOMADS Project Manager
National Oceanic and Atmospheric Administration
National Climatic Data Center
151 Patton Ave
Asheville NC 28801
Phone: (828) 271-4097
Fax: (828) 271-4328

NOMADS: http://nomads.ncdc.noaa.gov/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20080130/a723fda3/attachment.html