[gradsusr] Control File Issues for HDF5 file

Jennifer M Adams jadams21 at gmu.edu
Fri Jan 15 12:48:00 EST 2016


Hi, Justin —
You can get rid of the warning messages about cache size by adding a CACHESIZE entry to your descriptor file. The other message, the one about src and dest having different data spaces … that may be a problem with hdf5 and using the integer array values in the units field of the variable declaration. Could you please write to me offline and attach your data file (don’t copy the forum with that attachment.) —Jennifer

On Jan 15, 2016, at 11:04 AM, Justin Hicks <jhicks2014 at gmail.com<mailto:jhicks2014 at gmail.com>> wrote:

Jennifer,

I appreciate the help. I am still pretty new to Grads and have taught myself a majority of what I know.

I believe the EDEF is not operational with the 'linear' option, so I tried the 'NAMES' option instead.

My updated control file is below:

DSET /data2/GPM/GPM_3CMB.03/2015/3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5
OPTIONS template
Title   GPM_3CMB.03
UNDEF   -9999.0
DTYPE  hdf5_grid
xdef 28 linear 1 1
ydef 72 linear 1 1
zdef 2 linear 1 1
tdef 16 linear 01mar2015 1mo
edef 3 names plrcnt0 plrcnt1 plrcnt2
vars 3
/Grids/G1/precipLiqRate/count=>plrcnt0  2   0,e,t,z,y,x   plrcnt0
/Grids/G1/precipLiqRate/count=>plrcnt1  2   1,e,t,z,y,x   plrcnt1
/Grids/G1/precipLiqRate/count=>plrcnt2  2   2,e,t,z,y,x   plrcnt2
endvars


I also tried to play around with the edef names. However, with this control file, I get the following error below in grads:

d plrcnt0

*** WARNING! ***
*** The I/O for this variable will be extremely slow
*** because the chunks are too big to fit in the cache
*** chunk size = 27648 bytes  (72 * 2 * 16 * 3 * 1 * 4 bytes)
*** cache size = 16128 bytes
*** There are two ways to control the cache size:
*** add a CACHESIZE entry to the descriptor file
*** or use the 'set cachesf' command
*** Please read http://iges.org/grads/gadoc/compression.html
HDF5-DIAG: Error detected in HDF5 (1.8.7) thread 0:
  #000: H5Dio.c line 174 in H5Dread(): can't read data
    major: Dataset
    minor: Read failed
  #001: H5Dio.c line 348 in H5D_read(): src and dest data spaces have different sizes
    major: Invalid arguments to routine
    minor: Bad value
H5Dread failed for int data
Data Request Error:  Error for variable 'plrcnt0'
  Error ocurred at column 1
DISPLAY error:  Invalid expression
  Expression = plrcnt0


-Justin

On Thu, Jan 14, 2016 at 11:52 AM, Jennifer M Adams <jadams21 at gmu.edu<mailto:jadams21 at gmu.edu>> wrote:
The output from h5dump should have what you need, although what you included in your original post is missing some vital information.

For the sake of example (so you know what to look for), below is some of the h5dump output that includes DATA with each attribute, which tells me how to map this grid into the GrADS data model.  The descriptor file entries (also included below the h5dump output)  illustrate how the data in the hdf5 attributes map to the GrADS descriptor file. Note how your h5dump output lists the attribute “DimensionNames” but doesn’t print out the actual values of that attribute (the DATA).

  DATASET "gaugeRelativeWeighting" {
         DATATYPE  H5T_STD_I16LE
         DATASPACE  SIMPLE { ( 3600, 1800 ) / ( 3600, 1800 ) }
         ATTRIBUTE "CodeMissingValue" {
            DATATYPE  H5T_STRING {
                  STRSIZE 6;
                  STRPAD H5T_STR_NULLTERM;
                  CSET H5T_CSET_ASCII;
                  CTYPE H5T_C_S1;
               }
            DATASPACE  SCALAR
            DATA {
            (0): "-9999"
            }
         }
         ATTRIBUTE "DIMENSION_LIST" {
            DATATYPE  H5T_VLEN { H5T_REFERENCE}
            DATASPACE  SIMPLE { ( 2 ) / ( 2 ) }
            DATA {
            (0): (DATASET 598 /Grid/lon ), (DATASET 1147 /Grid/lat )
            }
         }
         ATTRIBUTE "DimensionNames" {
            DATATYPE  H5T_STRING {
                  STRSIZE 8;
                  STRPAD H5T_STR_NULLTERM;
                  CSET H5T_CSET_ASCII;
                  CTYPE H5T_C_S1;
               }
            DATASPACE  SCALAR
            DATA {
            (0): "lon,lat"
            }
         }
         ATTRIBUTE "Units" {
            DATATYPE  H5T_STRING {
                  STRSIZE 8;
                  STRPAD H5T_STR_NULLTERM;
                  CSET H5T_CSET_ASCII;
                  CTYPE H5T_C_S1;
               }
            DATASPACE  SCALAR
            DATA {
            (0): "percent"
            }
         }

dset ^3B-MO.MS.MRG.3IMERG.20140312-S000000-E235959.03.V03D.HDF5
dtype hdf5_grid
undef -9999.9
xdef 3600 linear -179.95 0.10
ydef 1800 linear -89.95 0.10
zdef 1 linear 1 1
tdef 1 linear 01mar2014 1mo
vars 1
/Grid/precipitation=>p 0 x,y precipitation [mm/hr]
endvars

Your variables have a shape like this:
3, 3, 16, 2, 72, 28
so you need to figure out how you’re going to map those into X, Y, Z, T, and E and then use numbers for the remaining dimensions.

Without knowing anything about this data, you could try a complete abstract grid:

xdef 28 linear 1 1
ydef 72 linear 1 1
zdef 2 linear 1 1
tdef 16 linear 01jan0001 1mo
edef 3 linear 1 1
...
vars 3
/Grids/G1/precipLiqRate/count=>plrcnt0  2   0,e,t,z,y,x   plrcnt0
/Grids/G1/precipLiqRate/count=>plrcnt1  2   1,e,t,z,y,x   plrcnt1
/Grids/G1/precipLiqRate/count=>plrcnt2  2   2,e,t,z,y,x   plrcnt2
endvars

The variable that has an extra dimension (‘hist’ with DATASPACE  SIMPLE { ( 30, 3, 3, 16, 2, 72, 28 ) / ( 30, 3, 3, 16, 2, 72, 28 ) }) is going to be divided further into more variables with an extra set of integers in the comma-delimited units entry of your descriptor file.
Good luck!!
—Jennifer



On Jan 14, 2016, at 11:04 AM, Justin Hicks <jhicks2014 at gmail.com<mailto:jhicks2014 at gmail.com>> wrote:

Jennifer,

Thank you. I will play around with this. However, it seems I would have to know the xdef, ydef, and zdef to take advantage of this. Any hints on how I could find out what these would be? What I have in my control file isn't correct; I just placed it there with the intention of fixing it later after the variables worked.

-Justin

On Thu, Jan 14, 2016 at 10:42 AM, Jennifer M Adams <jadams21 at gmu.edu<mailto:jadams21 at gmu.edu>> wrote:
Hi, Justin —
That message looks like your delcaration of the coordinates (“x,y”) is wrong. Look at the DATASPACE SIMPLE attribute in the h5dump output, that should give you a clue as to how many and what size the dimensions are. It looks like ‘hist’ has one more dimension than ‘mean’ and these variables have 6 or 7 dimensions.  Have a look at item #4 in the Usage Notes of http://cola.gmu.edu/grads/gadoc/SDFdescriptorfile.html for how to handle this situation.
—Jennifer

On Jan 14, 2016, at 10:04 AM, Justin Hicks <jhicks2014 at gmail.com<mailto:jhicks2014 at gmail.com>> wrote:

Jennifer,

I put in the dataset names, like you said, but I still get a different error. The error says:

ga-> d plrcnt
HDF5-DIAG: Error detected in HDF5 (1.8.7) thread 0:
  #000: H5Dio.c line 153 in H5Dread(): selection+offset not within extent
    major: Dataspace
    minor: Out of range
H5Dread failed for int data
Data Request Error:  Error for variable 'plrcnt'
  Error ocurred at column 1
DISPLAY error:  Invalid expression
  Expression = plrcnt



-Justin

On Wed, Jan 13, 2016 at 8:07 AM, Jennifer M Adams <jadams21 at gmu.edu<mailto:jadams21 at gmu.edu>> wrote:
Justin,
It looks like the variables in your descriptor files are group names not dataset names. Try this instead:
/Grids/G1/precipLiqRate/count=>plrcnt            0 x,y  liquid precipitation rate count
/Grids/G1/precipLiqRate/hist=>plrhist            0 x,y  liquid precipitation rate hist
/Grids/G1/precipLiqRate/mean=>plrmean            0 x,y  liquid precipitation rate mean
/Grids/G1/precipLiqRate/stdev=>plrcstd           0 x,y  liquid precipitation rate stdev
—Jennifer

On Jan 11, 2016, at 2:42 PM, Justin Hicks <jhicks2014 at gmail.com<mailto:jhicks2014 at gmail.com>> wrote:

Grads Users,

I was able to open the control file in grads, but i get an error when I try to display a variable:

ga-> d plrate
HDF5-DIAG: Error detected in HDF5 (1.8.7) thread 0:
  #000: H5D.c line 341 in H5Dopen2(): not a dataset
    major: Dataset
    minor: Inappropriate type
Error: H5Dopen2 failed for variable /Grids/G1/precipLiqRate
Error: Variable /Grids/G1/precipLiqRate not in HDF5 file
Data Request Error:  Error for variable 'plrate'
  Error ocurred at column 1
DISPLAY error:  Invalid expression
  Expression = plrate

The control file is below:

DSET /data2/GPM/GPM_3CMB.03/2015/3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5
OPTIONS template
DTYPE hdf5_grid
TITLE GPM precip
UNDEF -999.0
XDEF 3600 linear -180 0.1
YDEF 300 linear -20 0.1
ZDEF 1 levels 1000
TDEF 1 linear 01mar2015 1dy
VARS 4
/Grids/G1/precipLiqRate=>plrate            0 x,y  liquid precipitation rate
/Grids/G1/precipLiqWaterContent=>plcon     0 x,y   liquid precipitation content
/Grids/G1/precipTotDm=>ptot                0 x,y   precipitation total
/Grids/G1/precipTotRate=>ptotr             0 x,y  precipitation total rate
ENDVARS


I don't think the xdef and ydef are completely accurate, but I shouldn't get that error message even if the edef and ydef are wrong. The file contents of h5dump are below:

/home/jhicks/h5dump -n 3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5
HDF5 "3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5" {
FILE_CONTENTS {
 group      /
 group      /Grids
 group      /Grids/G1
 dataset    /Grids/G1/precipAllObs
 group      /Grids/G1/precipLiqRate
 dataset    /Grids/G1/precipLiqRate/count
 dataset    /Grids/G1/precipLiqRate/hist
 dataset    /Grids/G1/precipLiqRate/mean
 dataset    /Grids/G1/precipLiqRate/stdev
 group      /Grids/G1/precipLiqWaterContent
 dataset    /Grids/G1/precipLiqWaterContent/count
 dataset    /Grids/G1/precipLiqWaterContent/hist
 dataset    /Grids/G1/precipLiqWaterContent/mean
 dataset    /Grids/G1/precipLiqWaterContent/stdev
 group      /Grids/G1/precipTotDm
 dataset    /Grids/G1/precipTotDm/count
 dataset    /Grids/G1/precipTotDm/hist
 dataset    /Grids/G1/precipTotDm/mean
 dataset    /Grids/G1/precipTotDm/stdev
 group      /Grids/G1/precipTotLogNw
 dataset    /Grids/G1/precipTotLogNw/count
 dataset    /Grids/G1/precipTotLogNw/hist
 dataset    /Grids/G1/precipTotLogNw/mean
 dataset    /Grids/G1/precipTotLogNw/stdev
 group      /Grids/G1/precipTotRate
 dataset    /Grids/G1/precipTotRate/count
 dataset    /Grids/G1/precipTotRate/hist
 dataset    /Grids/G1/precipTotRate/mean
 dataset    /Grids/G1/precipTotRate/stdev
 group      /Grids/G1/precipTotWaterContent
 dataset    /Grids/G1/precipTotWaterContent/count
 dataset    /Grids/G1/precipTotWaterContent/hist
 dataset    /Grids/G1/precipTotWaterContent/mean
 dataset    /Grids/G1/precipTotWaterContent/stdev
 dataset    /Grids/G1/surfPrecipLiqRateProb
 dataset    /Grids/G1/surfPrecipLiqRateUn
 group      /Grids/G1/surfPrecipTotRateDiurnal
 dataset    /Grids/G1/surfPrecipTotRateDiurnal/count
 dataset    /Grids/G1/surfPrecipTotRateDiurnal/mean
 dataset    /Grids/G1/surfPrecipTotRateDiurnal/stdev
 dataset    /Grids/G1/surfPrecipTotRateDiurnalAllObs
 dataset    /Grids/G1/surfPrecipTotRateProb
 dataset    /Grids/G1/surfPrecipTotRateUn
 group      /Grids/G2
 dataset    /Grids/G2/precipAllObs
 group      /Grids/G2/precipLiqRate
 dataset    /Grids/G2/precipLiqRate/count
 dataset    /Grids/G2/precipLiqRate/mean
 dataset    /Grids/G2/precipLiqRate/stdev
 group      /Grids/G2/precipLiqWaterContent
 dataset    /Grids/G2/precipLiqWaterContent/count
 dataset    /Grids/G2/precipLiqWaterContent/mean
 dataset    /Grids/G2/precipLiqWaterContent/stdev
 group      /Grids/G2/precipTotDm
 dataset    /Grids/G2/precipTotDm/count
 dataset    /Grids/G2/precipTotDm/mean
 dataset    /Grids/G2/precipTotDm/stdev
 group      /Grids/G2/precipTotLogNw
 dataset    /Grids/G2/precipTotLogNw/count
 dataset    /Grids/G2/precipTotLogNw/mean
 dataset    /Grids/G2/precipTotLogNw/stdev
 group      /Grids/G2/precipTotRate
 dataset    /Grids/G2/precipTotRate/count
 dataset    /Grids/G2/precipTotRate/mean
 dataset    /Grids/G2/precipTotRate/stdev
 group      /Grids/G2/precipTotWaterContent
 dataset    /Grids/G2/precipTotWaterContent/count
 dataset    /Grids/G2/precipTotWaterContent/mean
 dataset    /Grids/G2/precipTotWaterContent/stdev
 dataset    /Grids/G2/surfPrecipLiqRateProb
 dataset    /Grids/G2/surfPrecipLiqRateUn
 group      /Grids/G2/surfPrecipTotRateDiurnal
 dataset    /Grids/G2/surfPrecipTotRateDiurnal/count
 dataset    /Grids/G2/surfPrecipTotRateDiurnal/mean
 dataset    /Grids/G2/surfPrecipTotRateDiurnal/stdev
 dataset    /Grids/G2/surfPrecipTotRateDiurnalAllObs
 dataset    /Grids/G2/surfPrecipTotRateProb
 dataset    /Grids/G2/surfPrecipTotRateUn
 dataset    /InputAlgorithmVersions
 dataset    /InputFileNames
 dataset    /InputGenerationDateTimes
 }
}


The group contents of /Grids/G1/precipLiqRate are below:

h5dump -H -g/Grids/G1/precipLiqRate 3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5
HDF5 "3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5" {
GROUP "/Grids/G1/precipLiqRate" {
   DATASET "count" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SIMPLE { ( 3, 3, 16, 2, 72, 28 ) / ( 3, 3, 16, 2, 72, 28 ) }
      ATTRIBUTE "CodeMissingValue" {
         DATATYPE  H5T_STRING {
               STRSIZE 101;
               STRPAD H5T_STR_NULLTERM;
               CSET H5T_CSET_ASCII;
               CTYPE H5T_C_S1;
            }
         DATASPACE  SCALAR
      }
      ATTRIBUTE "DimensionNames" {
         DATATYPE  H5T_STRING {
               STRSIZE 21;
               STRPAD H5T_STR_NULLTERM;
               CSET H5T_CSET_ASCII;
               CTYPE H5T_C_S1;
            }
         DATASPACE  SCALAR
      }
   }
   DATASET "hist" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SIMPLE { ( 30, 3, 3, 16, 2, 72, 28 ) / ( 30, 3, 3, 16, 2, 72, 28 ) }
      ATTRIBUTE "CodeMissingValue" {
         DATATYPE  H5T_STRING {
               STRSIZE 101;
               STRPAD H5T_STR_NULLTERM;
               CSET H5T_CSET_ASCII;
               CTYPE H5T_C_S1;
            }
         DATASPACE  SCALAR
      }
      ATTRIBUTE "DimensionNames" {
         DATATYPE  H5T_STRING {
               STRSIZE 25;
               STRPAD H5T_STR_NULLTERM;
               CSET H5T_CSET_ASCII;
               CTYPE H5T_C_S1;
            }
         DATASPACE  SCALAR
      }
   }
   DATASET "mean" {
      DATATYPE  H5T_IEEE_F32LE
      DATASPACE  SIMPLE { ( 3, 3, 16, 2, 72, 28 ) / ( 3, 3, 16, 2, 72, 28 ) }
      ATTRIBUTE "CodeMissingValue" {
         DATATYPE  H5T_STRING {
               STRSIZE 101;
               STRPAD H5T_STR_NULLTERM;
               CSET H5T_CSET_ASCII;
               CTYPE H5T_C_S1;
            }
         DATASPACE  SCALAR
      }
      ATTRIBUTE "DimensionNames" {
         DATATYPE  H5T_STRING {
               STRSIZE 21;
               STRPAD H5T_STR_NULLTERM;
               CSET H5T_CSET_ASCII;
               CTYPE H5T_C_S1;
            }
         DATASPACE  SCALAR
      }
   }
   DATASET "stdev" {
      DATATYPE  H5T_IEEE_F32LE
      DATASPACE  SIMPLE { ( 3, 3, 16, 2, 72, 28 ) / ( 3, 3, 16, 2, 72, 28 ) }
      ATTRIBUTE "CodeMissingValue" {
         DATATYPE  H5T_STRING {
               STRSIZE 101;
               STRPAD H5T_STR_NULLTERM;
               CSET H5T_CSET_ASCII;
               CTYPE H5T_C_S1;
            }
         DATASPACE  SCALAR
      }
      ATTRIBUTE "DimensionNames" {
         DATATYPE  H5T_STRING {
               STRSIZE 21;
               STRPAD H5T_STR_NULLTERM;
               CSET H5T_CSET_ASCII;
               CTYPE H5T_C_S1;
            }
         DATASPACE  SCALAR
      }
   }
}
}


Any help would be appreciated.

Thanks,

-Justin
_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr


_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr


_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr


_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr


_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr


_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr


_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20160115/6bac193d/attachment-0001.html 


More information about the gradsusr mailing list