lats4d

Wed Jan 30 09:18:09 EST 2008

with attachment......

Glenn.Rutledge wrote the following on 1/30/2008 9:14 AM:
> Hello Mike and all-
> I'm the lucky guy in the middle of planning for the receipt of the
> CFSRR (the next reanal) at NCDC.  I have asked the data be provided to
> us in Grib2 to save space.  The number you are stating are correct-
> approximately 915TB in Grib1 so there is an approximate 40-50%
> reduction.   NCDC and NCEP conducted a User Workshop at AMS Town Hall
> on Jan 21.  The slides I prepared for our Director and working with
> NCEP are attached (and the numbers are listed as _Grib1_ for consistency)
>
> What I and others here planning for this dataset is that we would like
> to hear how some of these data could be best organized.  Our suggested
> file level is attached in the presentation (not sure gradsusr list
> will pass the presentation thru however).  In any event- NCDC plans to
> make as much of this avbl when it's released to the public in June of
> 2009 (as I understand it from NCEP).  SOme I want on disk (about 150TB
> for on-line access) via NOMADS, and the rest will have to be staged
> off from tape.  Hope this helps. Glenn
>
> Mike Bosilovich wrote the following on 1/30/2008 8:53 AM:
>> Thanks for the prompt reply Brian,
>>
>> I realize and greatly appreciate the amount of work you, Jennifer and
>> others have done, with very little support. Grads is a great tool,
>> hands down. And I tell that to anyone who will listen.
>>
>> I assumed that you all, as developers, would like feedback on what
>> users need to do. Lats4d fills a very important role in my work, and
>> I expect others as well. I wanted to pass on a point of view.
>> Certainly, I'm not trying to tell you what you must do, only
>> providing feedback on the capability that I find useful.
>>
>> Now, as for what Jennifer said, there was a post that says lats will
>> not be supported. That was the impetus for my post. If I
>> misunderstood, and there will be lats4d capability in some way, I
>> apologize for over reacting.
>>
>> In any event, I also need to clarify my statement on the new NCEP
>> reanalysis. They will have 1000TB of Grib1 format data at NCDC (this
>> is still changing, as I understand). In Grib2, it comes down to 500Tb
>> or so. For what that's worth, it's still immense. I got the sense
>> that they are planning on grib1, because of the lack of utilities
>> with grib2 capability.
>>
>> Again, this is merely my opinion as a user, and I hoped the opinion
>> and the information would be useful to the developers.
>>
>> Sincerely,
>>
>> Mike Bosilovich
>>
>> On Jan 29, 2008 10:02 PM, Brian Doty <doty at cola.iges.org
>> <mailto:doty at cola.iges.org>> wrote:
>>
>>     Mike, the new Version 2 of GrADS is being released in alpha version
>>     and incomplete so that people would have access to the new GRIB2
>>     support.   With no funding for many years and key personnel working
>>     limited part time hours we do not have the resources to support old
>>     code and also move forward to provide essential new functionality for
>>     present and future research needs.  GrADS development first started
>>     in the late 1980s and it has taken us several years of work to clean
>>     up the 15 years of accumulated mods and patches on top of the
>>     original code base in order to come out with a new code base so we
>>     could move forward.   I have made every effort over the years to
>>     ensure upward compatibility and maintain stability.  There is no way
>>     that a 20 year old code base can be patched and hacked for another 20
>>     years or even 5 years and maintain stability and still support the
>>     100TB data sets you talk about and the other projects probably in the
>>     pipeline.   As new versions of data formats appear -- grib2, hdf5,
>>     new netcdf on the way, etc ---  we have no choice.  We cannot keep
>>     using old libraries and maintain them ourselves.  We cannot patch and
>>     hack old code that we didn't write to begin with and which in some
>>     cases has restrictive copyrights.
>>
>>     As I say, version 2 as it stands now is incomplete.   Key areas of
>>     functionality which need to be re-implemented have not yet been
>>     done.  Other major parts of grads have not yet been upgraded, such as
>>     the graphics package.   As Jennifer said in her original email, the
>>     output of various data formats is high on the list of things yet to
>>     be done...   Brian
>>
>>     On Jan 29, 2008, at 4:34 PM, Mike Bosilovich wrote:
>>
>>     >
>>     > I have to admit an ignorance to some of the inner workings of Grads
>>     > development. But I am quite surprised to hear that lats4d is not
>>     > presently supported in Grads 2.
>>     >
>>     > I have been using Grads for 10 years now (I can still recall the
>>     > relief after years of ncargf77 programming :-) Of course, I still
>>     > write code when appropriate, but grads and lats4d have been the
>>     > main tools in my work. Lats4d fills a critical void. By pointing it
>>     > at any grads formatted file (netcdf, hdf, binary or grib), I can
>>     > reformat the data to what ever a colleague may need. I can also,
>>     > easily, no, effortlessly subset variables space or time. Coupled
>>     > with a call to regrid, this subsetting utility is beyond compare.
>>     > In a c shell, it can rip through huge data files with simplicity. I
>>     > hope this does not sound like exaggeration, lats4d is the strongest
>>     > data tool I have used.
>>     >
>>     > In our office, we have just begun production of a new reanalysis
>>     > data product. While it will take some time to complete, we are
>>     > beginning to develop examples on how users can access and analyze
>>     > the data. Comparison with other reanalyses is obvious and will be
>>     > in high demand. To accomplish that, they will need to regrid our
>>     > data sets to the existing coarser reanalysis data sets. Or, they
>>     > may need to change the format out of our native HDF. The easiest
>>     > way to explain to others how to do this is with lats4d.
>>     >
>>     > There will be 100Tb of reanalysis data available through a GDS.
>>     > With Lats4d and gradsdods (or gradsdap), this would be much more
>>     > accessible. Users, with some examples, will access the data through
>>     > online capabilities, rather than bulk downloading of the native HDF
>>     > files (a throttle may be needed for access if that is the preferred
>>     > by users). In addition, a plan is being prepared to develop a DVD,
>>     > similar to the NCEP reanalyses CD and NARR DVD. Personally, I would
>>     > like to see a flavor of grads and lats4d included therein
>>     > (admittedly, I have not gotten to discuss this with the grads
>>     > developers yet).
>>     >
>>     > It doesn't stop there. At AMS last week, NCEP and NCDC held a town
>>     > hall meeting to discuss their plans for the next reanalyses. They
>>     > expect to have nearly 1000Tb of data from three different
>>     > reanalyses, and their production has also started. I don't see
>>     > storage as a barrier, but bandwidth is. Too many users making too
>>     > many big requests will limit accessibility. Lats4d access to their
>>     > GDS will become an important function.
>>     >
>>     > Again, I have to admit I do not know the extent of the issues here.
>>     > By necessity, I will have to use versions of grads that include
>>     > lats4d not only in my work, but as I show other how to use our
>>     > data. I felt the need to speak up and I hope that these issues will
>>     > be considered.
>>     >
>>     > A sincere grads and lats4d user,
>>     >
>>     > Mike Bosilovich
>>     >
>>
>>
>
> --
> Glenn K. Rutledge
> Services Team Leader
> Remote Sensing and Applications Division
> NOMADS Project Manager
> National Oceanic and Atmospheric Administration
> National Climatic Data Center
> 151 Patton Ave
> Asheville NC 28801
> Phone: (828) 271-4097
> Fax: (828) 271-4328
>
> NOMADS: http://nomads.ncdc.noaa.gov/
>

--
Glenn K. Rutledge
Services Team Leader
Remote Sensing and Applications Division
NOMADS Project Manager
National Oceanic and Atmospheric Administration
National Climatic Data Center
151 Patton Ave
Asheville NC 28801
Phone: (828) 271-4097
Fax: (828) 271-4328

NOMADS: http://nomads.ncdc.noaa.gov/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20080130/ea472cda/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Karl AMS-2008 Reanalysis TownHall Access FINAL1.ppt
Type: application/vnd.ms-powerpoint
Size: 1457664 bytes
Desc: not available
Url : http://gradsusr.org/pipermail/gradsusr/attachments/20080130/ea472cda/attachment.ppt 