[gradsusr] Will grads be in spack?

Tue Jan 8 14:27:35 EST 2019

I think Wesley knows more about this than I do, but here’s my understanding:

The patches to jasper-1.900.1 for supporting JPEG2000 didn’t change the version number, so it’s a little hard to know which one you have. I got my src tarball from Wesley, and haven’t touched it in years. There is a small but non-zero chance you will encounter a GRIB2 file that employs PNG compression. The RTGSST data from NCEP is another example. I would build your grib2 library with the Spack jasper version 1.900.1 and do some testing later with some grib2 files you need to read. Or you could use wgrib2 to change the compression algorithm in your data file.

As for which PNG version, upgrading to libpng15 solved some image printing issues with Cairo, and I and have not had any problems with g2clib using libpng15.  Just make sure that the libpng version is consistent across all the libraries that use it that are linked with grads. I make heavy use of the ‘ldd’ command on unix systems to see what shared libraries my executables and libraries depend on (‘otool -L’ is the Mac equivalent). When you’re done testing the I/O of your grib2 file by displaying a variable, test the image printing with the ‘gxprint’ command.
--Jennifer

--
Jennifer Miletta Adams
ADNET Systems, Inc.
NASA/GSFC, Code 610.2
Building 32, Room S159
(301) 614-6070

From: gradsusr <gradsusr-bounces at gradsusr.org> on behalf of "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson at nasa.gov>
Reply-To: GrADS Users Forum <gradsusr at gradsusr.org>
Date: Tuesday, January 8, 2019 at 1:36 PM
To: Wesley Ebisuzaki - NOAA Federal <wesley.ebisuzaki at noaa.gov>, GrADS Users Forum <gradsusr at gradsusr.org>
Subject: Re: [gradsusr] Will grads be in spack?

Wesley,

Spack seems to have 1.25.7 as a Safe Version. Is that sufficient?

Matt

--
Matt Thompson, SSAI, Sr Scientific Programmer/Analyst
NASA GSFC,    Global Modeling and Assimilation Office
Code 610.1,  8800 Greenbelt Rd,  Greenbelt,  MD 20771
Phone: 301-614-6712                 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

From: Wesley Ebisuzaki - NOAA Federal <wesley.ebisuzaki at noaa.gov>
Date: Tuesday, January 8, 2019 at 12:17 PM
To: GrADS Users Forum <gradsusr at gradsusr.org>
Cc: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson at nasa.gov>
Subject: Re: [gradsusr] Will grads be in spack?

Matt,

The original jasper 1.900.1 libraries from the univ of victoria had problems and
both the redhat and debian updates fixed the problems.  After many years,
the original author started updating jasper and fixing the bugs.  The last
time that I looked (1+ years ago), there was a small API change which would
require a change in g2clib.  So you will have to check which version of jasper
is already in spark.

G2clib requires libpng 1.2.xx.  Not sure 1.5.xx will work.  PNG grib files are
rare but not extinct.  ESRL uses them for processed radar fields.

Wesley Ebisuzaki

On Tue, Jan 8, 2019 at 11:56 AM Adams, Jennifer M. (GSFC-610.2)[ADNET SYSTEMS INC] <jennifer.m.adams at nasa.gov<mailto:jennifer.m.adams at nasa.gov>> wrote:
Hi, Matt --
1. The shapelib library has been around for a while, and it's very easy to build, if a Spack package doesn't exist, it won't be hard to create.
2. You definitely want to enable opendap in the NetCDF library. The dap/gadap libs in GrADS are for accessing station data (as opposed to grids) via GDS, but I don't think there is anyone out there putting station data behind GDS anymore.
3. GrADS will require some code changes to use udunits2.  For now, you'll have to wrangle udunits1 into submission.
4. As for GRIB2,  forget ECMWF; GrADS relies on NCEP's g2clib to handle the I/O.  In sickness and in health, for better or worse, in good times and bad, etc. To properly support JPEG2000 compression (which is used in some NCEP products), a special version of the jasper library is needed (ftp://cola.gmu.edu/grads/Supplibs/2.2/src/jasper-1.900.1-14ubuntu3.2.debian.tar.gz).

--Jennifer

--
Jennifer Miletta Adams
ADNET Systems, Inc.
NASA/GSFC, Code 610.2
Building 32, Room S159
(301) 614-6070

On 1/8/19, 8:45 AM, "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson at nasa.gov<mailto:matthew.thompson at nasa.gov>> wrote:

    Jennifer, et al,

    I just did a quick scan of this page:

      http://cola.gmu.edu/grads/gadoc/supplibs2.html

    and tried to match with spack packages. Most had pretty good matches
    (though they might be called "libX" instead of "X"), and the only
    libraries I couldn't find are:

      dap
      gadap
      shapelib

    and there are two that I don't think are quite right:

      udunits ==> spack has udunits2
      g2clib  ==> spack has grib-api, eccodes

    You said to ignore dap/gadap, so I'm happy with that. Note that in my
    testing I'm always building netcdf with the dap variant and it works for
    that at least:

    [(513) 08:43 AM] $ ncdump -hsc
    https://opendap.nccs.nasa.gov/dods/GEOS-5/fp/0.25_deg/fcast/inst1_2d_hwl_Nx/inst1_2d_hwl_Nx.20181222_12
    | head -20
    netcdf inst1_2d_hwl_Nx {
    dimensions:
        lat = 721 ;
        lon = 1152 ;
        time = 123 ;
    variables:
        double time(time) ;
                time:grads_dim = "t" ;
                time:grads_mapping = "linear" ;
                time:grads_size = "123" ;

    shapelib seems pretty standard in its build, so a package for it seems
    possible.

    udunits was the precursor to udunits2, so a package for it probably
    isn't impossible either though we might need to talk with spack about
    naming it as "udunits" nowadays usually refers to the udunits2 package
    (udunits1?).

    You have instructions for g2clib as well so it's probably doable as
    well, but ECMWF's (deprecated) grib-api and eccodes are in spack. I'm
    guessing, though, that the APIs are not the same?

    Matt

    On 1/7/19 12:17 PM, Adams, Jennifer M. (GSFC-610.2)[ADNET SYSTEMS INC]
    wrote:
    > Matt, I see you are taking on the role of GMAO’s Fairy Godmother … Bravo!
    >
    > The m4 macros used by the GrADS autoconfigure scripts with the --enable-dyn-supplibs option are not perfect. I was rushing to get them into some kind of working state when releasing version 2.2.1 right before my position at COLA ended. Please use the source code at https://github.com/j-m-adams/GrADS, and not the tarball from COLA, which is missing a few scripts. I will try to incorporate the patches from Steve Woodbridge as quickly as I can. Not all libraries use the PKG_CONFIG model, so it is a mix of strategies for finding what is needed. I definitely rely on $LDFLAGS, $CFLAGS, $CPPFLAGS, and $LD_LIBRARY_PATH to find everything. The grib2 library is a good example of one with non-standard features -- that lib is barely supported within NCEP and has zero bells and whistles for simplifying your life; you have to edit the makefile to tweak the build and then copy files manually to install them outside of the local directory. Don't bother with dap/gadap. What else is not available in Spack?
    > --Jennifer
    >
    >
    > --
    > Jennifer Miletta Adams
    > ADNET Systems, Inc.
    > NASA/GSFC, Code 610.2
    > Building 32, Room S159
    > (301) 614-6070
    >
    > ________________________________________
    > From: gradsusr [gradsusr-bounces at gradsusr.org<mailto:gradsusr-bounces at gradsusr.org>] on behalf of Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] [matthew.thompson at nasa.gov<mailto:matthew.thompson at nasa.gov>]
    > Sent: Monday, January 07, 2019 10:47 AM
    > To: GrADS Users Forum
    > Subject: Re: [gradsusr] Will grads be in spack?
    >
    > All,
    >
    > I've been trying to slowly move GMAO's GEOS model's Baselibs to spack. This has involved learning how to make new spack packages, which has been interesting (and not quite fully successful yet, to be honest, but I'm learning!). Now, Baselibs started as an offshoot of Supplibs, so I am learning some useful and perhaps relevant lessons.
    >
    > Looking at how GrADS is built (which I've never done as the binaries work fine for my needs), the main issue I see at present is figuring out how to build GrADS without the SUPPLIBS being in one place. GEOS inherited this as it expects Baselibs to all be under one place as well. With GEOS I have the "luxury" of knowing its very odd non-standard GNU Make build system and could probably hack it to find spack's libraries without too much trouble, but GEOS relies on fewer libraries than GrADS.
    >
    > GrADS, though, uses configure. Yay for us builders and it looks like some of the libraries seem exposed to configure (--with-netcdf, etc.) but not all. Does anyone out there have experience building GrADS with --enable-dyn-supplibs? If you've documented how to get GrADS to find and use system libraries, spack might be okay with that (with, say, loooong LDFLAGS, LIBS, CPPFLAGS lines). The only other thought would be finding an autotools hacker who could expose all the supplibs with --with-PKG options.
    >
    > As an aside, it looks like most of Supplibs is in spack, but not all (g2clib, a few others), so eventually a limited GrADS might be doable, with a nigh-complete build if other Supplibs were ported.
    >
    > Matt
    >
    >
    > --
    > Matt Thompson, SSAI, Sr Scientific Programmer/Analyst
    > NASA GSFC,    Global Modeling and Assimilation Office
    > Code 610.1,  8800 Greenbelt Rd,  Greenbelt,  MD 20771
    > Phone: 301-614-6712                 Fax: 301-614-6246
    > http://science.gsfc.nasa.gov/sed/bio/matthew.thompson
    >
    > On 1/4/19, 2:08 PM, "gradsusr on behalf of Tom Robinson" <gradsusr-bounces at gradsusr.org<mailto:gradsusr-bounces at gradsusr.org> on behalf of ter at hawaii.edu<mailto:ter at hawaii.edu>> wrote:
    >
    >      I have recently been using spack to install scientific software. It really makes things easy. Is there any plan to add grads to spack?
    >
    >      -Tom
    >
    >      Sent from my iPhone
    >      _______________________________________________
    >      gradsusr mailing list
    >      gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
    >      http://gradsusr.org/mailman/listinfo/gradsusr
    >
    >
    >
    > _______________________________________________
    > gradsusr mailing list
    > gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
    > http://gradsusr.org/mailman/listinfo/gradsusr
    >
    > _______________________________________________
    > gradsusr mailing list
    > gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
    > http://gradsusr.org/mailman/listinfo/gradsusr
    >

    --
    Matt Thompson, SSAI, Sr Scientific Programmer/Analyst
    NASA GSFC,    Global Modeling and Assimilation Office
    Code 610.1,  8800 Greenbelt Rd,  Greenbelt,  MD 20771
    Phone: 301-614-6712                 Fax: 301-614-6246
    http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gradsusr.org/pipermail/gradsusr/attachments/20190108/c70b254e/attachment-0001.html>