[gradsusr] Control File Issues for HDF5 file
Jennifer M Adams
jadams21 at gmu.edu
Fri Jan 15 12:48:00 EST 2016
Hi, Justin —
You can get rid of the warning messages about cache size by adding a CACHESIZE entry to your descriptor file. The other message, the one about src and dest having different data spaces … that may be a problem with hdf5 and using the integer array values in the units field of the variable declaration. Could you please write to me offline and attach your data file (don’t copy the forum with that attachment.) —Jennifer
On Jan 15, 2016, at 11:04 AM, Justin Hicks <jhicks2014 at gmail.com<mailto:jhicks2014 at gmail.com>> wrote:
Jennifer,
I appreciate the help. I am still pretty new to Grads and have taught myself a majority of what I know.
I believe the EDEF is not operational with the 'linear' option, so I tried the 'NAMES' option instead.
My updated control file is below:
DSET /data2/GPM/GPM_3CMB.03/2015/3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5
OPTIONS template
Title GPM_3CMB.03
UNDEF -9999.0
DTYPE hdf5_grid
xdef 28 linear 1 1
ydef 72 linear 1 1
zdef 2 linear 1 1
tdef 16 linear 01mar2015 1mo
edef 3 names plrcnt0 plrcnt1 plrcnt2
vars 3
/Grids/G1/precipLiqRate/count=>plrcnt0 2 0,e,t,z,y,x plrcnt0
/Grids/G1/precipLiqRate/count=>plrcnt1 2 1,e,t,z,y,x plrcnt1
/Grids/G1/precipLiqRate/count=>plrcnt2 2 2,e,t,z,y,x plrcnt2
endvars
I also tried to play around with the edef names. However, with this control file, I get the following error below in grads:
d plrcnt0
*** WARNING! ***
*** The I/O for this variable will be extremely slow
*** because the chunks are too big to fit in the cache
*** chunk size = 27648 bytes (72 * 2 * 16 * 3 * 1 * 4 bytes)
*** cache size = 16128 bytes
*** There are two ways to control the cache size:
*** add a CACHESIZE entry to the descriptor file
*** or use the 'set cachesf' command
*** Please read http://iges.org/grads/gadoc/compression.html
HDF5-DIAG: Error detected in HDF5 (1.8.7) thread 0:
#000: H5Dio.c line 174 in H5Dread(): can't read data
major: Dataset
minor: Read failed
#001: H5Dio.c line 348 in H5D_read(): src and dest data spaces have different sizes
major: Invalid arguments to routine
minor: Bad value
H5Dread failed for int data
Data Request Error: Error for variable 'plrcnt0'
Error ocurred at column 1
DISPLAY error: Invalid expression
Expression = plrcnt0
-Justin
On Thu, Jan 14, 2016 at 11:52 AM, Jennifer M Adams <jadams21 at gmu.edu<mailto:jadams21 at gmu.edu>> wrote:
The output from h5dump should have what you need, although what you included in your original post is missing some vital information.
For the sake of example (so you know what to look for), below is some of the h5dump output that includes DATA with each attribute, which tells me how to map this grid into the GrADS data model. The descriptor file entries (also included below the h5dump output) illustrate how the data in the hdf5 attributes map to the GrADS descriptor file. Note how your h5dump output lists the attribute “DimensionNames” but doesn’t print out the actual values of that attribute (the DATA).
DATASET "gaugeRelativeWeighting" {
DATATYPE H5T_STD_I16LE
DATASPACE SIMPLE { ( 3600, 1800 ) / ( 3600, 1800 ) }
ATTRIBUTE "CodeMissingValue" {
DATATYPE H5T_STRING {
STRSIZE 6;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "-9999"
}
}
ATTRIBUTE "DIMENSION_LIST" {
DATATYPE H5T_VLEN { H5T_REFERENCE}
DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
DATA {
(0): (DATASET 598 /Grid/lon ), (DATASET 1147 /Grid/lat )
}
}
ATTRIBUTE "DimensionNames" {
DATATYPE H5T_STRING {
STRSIZE 8;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "lon,lat"
}
}
ATTRIBUTE "Units" {
DATATYPE H5T_STRING {
STRSIZE 8;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "percent"
}
}
dset ^3B-MO.MS.MRG.3IMERG.20140312-S000000-E235959.03.V03D.HDF5
dtype hdf5_grid
undef -9999.9
xdef 3600 linear -179.95 0.10
ydef 1800 linear -89.95 0.10
zdef 1 linear 1 1
tdef 1 linear 01mar2014 1mo
vars 1
/Grid/precipitation=>p 0 x,y precipitation [mm/hr]
endvars
Your variables have a shape like this:
3, 3, 16, 2, 72, 28
so you need to figure out how you’re going to map those into X, Y, Z, T, and E and then use numbers for the remaining dimensions.
Without knowing anything about this data, you could try a complete abstract grid:
xdef 28 linear 1 1
ydef 72 linear 1 1
zdef 2 linear 1 1
tdef 16 linear 01jan0001 1mo
edef 3 linear 1 1
...
vars 3
/Grids/G1/precipLiqRate/count=>plrcnt0 2 0,e,t,z,y,x plrcnt0
/Grids/G1/precipLiqRate/count=>plrcnt1 2 1,e,t,z,y,x plrcnt1
/Grids/G1/precipLiqRate/count=>plrcnt2 2 2,e,t,z,y,x plrcnt2
endvars
The variable that has an extra dimension (‘hist’ with DATASPACE SIMPLE { ( 30, 3, 3, 16, 2, 72, 28 ) / ( 30, 3, 3, 16, 2, 72, 28 ) }) is going to be divided further into more variables with an extra set of integers in the comma-delimited units entry of your descriptor file.
Good luck!!
—Jennifer
On Jan 14, 2016, at 11:04 AM, Justin Hicks <jhicks2014 at gmail.com<mailto:jhicks2014 at gmail.com>> wrote:
Jennifer,
Thank you. I will play around with this. However, it seems I would have to know the xdef, ydef, and zdef to take advantage of this. Any hints on how I could find out what these would be? What I have in my control file isn't correct; I just placed it there with the intention of fixing it later after the variables worked.
-Justin
On Thu, Jan 14, 2016 at 10:42 AM, Jennifer M Adams <jadams21 at gmu.edu<mailto:jadams21 at gmu.edu>> wrote:
Hi, Justin —
That message looks like your delcaration of the coordinates (“x,y”) is wrong. Look at the DATASPACE SIMPLE attribute in the h5dump output, that should give you a clue as to how many and what size the dimensions are. It looks like ‘hist’ has one more dimension than ‘mean’ and these variables have 6 or 7 dimensions. Have a look at item #4 in the Usage Notes of http://cola.gmu.edu/grads/gadoc/SDFdescriptorfile.html for how to handle this situation.
—Jennifer
On Jan 14, 2016, at 10:04 AM, Justin Hicks <jhicks2014 at gmail.com<mailto:jhicks2014 at gmail.com>> wrote:
Jennifer,
I put in the dataset names, like you said, but I still get a different error. The error says:
ga-> d plrcnt
HDF5-DIAG: Error detected in HDF5 (1.8.7) thread 0:
#000: H5Dio.c line 153 in H5Dread(): selection+offset not within extent
major: Dataspace
minor: Out of range
H5Dread failed for int data
Data Request Error: Error for variable 'plrcnt'
Error ocurred at column 1
DISPLAY error: Invalid expression
Expression = plrcnt
-Justin
On Wed, Jan 13, 2016 at 8:07 AM, Jennifer M Adams <jadams21 at gmu.edu<mailto:jadams21 at gmu.edu>> wrote:
Justin,
It looks like the variables in your descriptor files are group names not dataset names. Try this instead:
/Grids/G1/precipLiqRate/count=>plrcnt 0 x,y liquid precipitation rate count
/Grids/G1/precipLiqRate/hist=>plrhist 0 x,y liquid precipitation rate hist
/Grids/G1/precipLiqRate/mean=>plrmean 0 x,y liquid precipitation rate mean
/Grids/G1/precipLiqRate/stdev=>plrcstd 0 x,y liquid precipitation rate stdev
—Jennifer
On Jan 11, 2016, at 2:42 PM, Justin Hicks <jhicks2014 at gmail.com<mailto:jhicks2014 at gmail.com>> wrote:
Grads Users,
I was able to open the control file in grads, but i get an error when I try to display a variable:
ga-> d plrate
HDF5-DIAG: Error detected in HDF5 (1.8.7) thread 0:
#000: H5D.c line 341 in H5Dopen2(): not a dataset
major: Dataset
minor: Inappropriate type
Error: H5Dopen2 failed for variable /Grids/G1/precipLiqRate
Error: Variable /Grids/G1/precipLiqRate not in HDF5 file
Data Request Error: Error for variable 'plrate'
Error ocurred at column 1
DISPLAY error: Invalid expression
Expression = plrate
The control file is below:
DSET /data2/GPM/GPM_3CMB.03/2015/3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5
OPTIONS template
DTYPE hdf5_grid
TITLE GPM precip
UNDEF -999.0
XDEF 3600 linear -180 0.1
YDEF 300 linear -20 0.1
ZDEF 1 levels 1000
TDEF 1 linear 01mar2015 1dy
VARS 4
/Grids/G1/precipLiqRate=>plrate 0 x,y liquid precipitation rate
/Grids/G1/precipLiqWaterContent=>plcon 0 x,y liquid precipitation content
/Grids/G1/precipTotDm=>ptot 0 x,y precipitation total
/Grids/G1/precipTotRate=>ptotr 0 x,y precipitation total rate
ENDVARS
I don't think the xdef and ydef are completely accurate, but I shouldn't get that error message even if the edef and ydef are wrong. The file contents of h5dump are below:
/home/jhicks/h5dump -n 3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5
HDF5 "3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5" {
FILE_CONTENTS {
group /
group /Grids
group /Grids/G1
dataset /Grids/G1/precipAllObs
group /Grids/G1/precipLiqRate
dataset /Grids/G1/precipLiqRate/count
dataset /Grids/G1/precipLiqRate/hist
dataset /Grids/G1/precipLiqRate/mean
dataset /Grids/G1/precipLiqRate/stdev
group /Grids/G1/precipLiqWaterContent
dataset /Grids/G1/precipLiqWaterContent/count
dataset /Grids/G1/precipLiqWaterContent/hist
dataset /Grids/G1/precipLiqWaterContent/mean
dataset /Grids/G1/precipLiqWaterContent/stdev
group /Grids/G1/precipTotDm
dataset /Grids/G1/precipTotDm/count
dataset /Grids/G1/precipTotDm/hist
dataset /Grids/G1/precipTotDm/mean
dataset /Grids/G1/precipTotDm/stdev
group /Grids/G1/precipTotLogNw
dataset /Grids/G1/precipTotLogNw/count
dataset /Grids/G1/precipTotLogNw/hist
dataset /Grids/G1/precipTotLogNw/mean
dataset /Grids/G1/precipTotLogNw/stdev
group /Grids/G1/precipTotRate
dataset /Grids/G1/precipTotRate/count
dataset /Grids/G1/precipTotRate/hist
dataset /Grids/G1/precipTotRate/mean
dataset /Grids/G1/precipTotRate/stdev
group /Grids/G1/precipTotWaterContent
dataset /Grids/G1/precipTotWaterContent/count
dataset /Grids/G1/precipTotWaterContent/hist
dataset /Grids/G1/precipTotWaterContent/mean
dataset /Grids/G1/precipTotWaterContent/stdev
dataset /Grids/G1/surfPrecipLiqRateProb
dataset /Grids/G1/surfPrecipLiqRateUn
group /Grids/G1/surfPrecipTotRateDiurnal
dataset /Grids/G1/surfPrecipTotRateDiurnal/count
dataset /Grids/G1/surfPrecipTotRateDiurnal/mean
dataset /Grids/G1/surfPrecipTotRateDiurnal/stdev
dataset /Grids/G1/surfPrecipTotRateDiurnalAllObs
dataset /Grids/G1/surfPrecipTotRateProb
dataset /Grids/G1/surfPrecipTotRateUn
group /Grids/G2
dataset /Grids/G2/precipAllObs
group /Grids/G2/precipLiqRate
dataset /Grids/G2/precipLiqRate/count
dataset /Grids/G2/precipLiqRate/mean
dataset /Grids/G2/precipLiqRate/stdev
group /Grids/G2/precipLiqWaterContent
dataset /Grids/G2/precipLiqWaterContent/count
dataset /Grids/G2/precipLiqWaterContent/mean
dataset /Grids/G2/precipLiqWaterContent/stdev
group /Grids/G2/precipTotDm
dataset /Grids/G2/precipTotDm/count
dataset /Grids/G2/precipTotDm/mean
dataset /Grids/G2/precipTotDm/stdev
group /Grids/G2/precipTotLogNw
dataset /Grids/G2/precipTotLogNw/count
dataset /Grids/G2/precipTotLogNw/mean
dataset /Grids/G2/precipTotLogNw/stdev
group /Grids/G2/precipTotRate
dataset /Grids/G2/precipTotRate/count
dataset /Grids/G2/precipTotRate/mean
dataset /Grids/G2/precipTotRate/stdev
group /Grids/G2/precipTotWaterContent
dataset /Grids/G2/precipTotWaterContent/count
dataset /Grids/G2/precipTotWaterContent/mean
dataset /Grids/G2/precipTotWaterContent/stdev
dataset /Grids/G2/surfPrecipLiqRateProb
dataset /Grids/G2/surfPrecipLiqRateUn
group /Grids/G2/surfPrecipTotRateDiurnal
dataset /Grids/G2/surfPrecipTotRateDiurnal/count
dataset /Grids/G2/surfPrecipTotRateDiurnal/mean
dataset /Grids/G2/surfPrecipTotRateDiurnal/stdev
dataset /Grids/G2/surfPrecipTotRateDiurnalAllObs
dataset /Grids/G2/surfPrecipTotRateProb
dataset /Grids/G2/surfPrecipTotRateUn
dataset /InputAlgorithmVersions
dataset /InputFileNames
dataset /InputGenerationDateTimes
}
}
The group contents of /Grids/G1/precipLiqRate are below:
h5dump -H -g/Grids/G1/precipLiqRate 3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5
HDF5 "3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5" {
GROUP "/Grids/G1/precipLiqRate" {
DATASET "count" {
DATATYPE H5T_STD_I32LE
DATASPACE SIMPLE { ( 3, 3, 16, 2, 72, 28 ) / ( 3, 3, 16, 2, 72, 28 ) }
ATTRIBUTE "CodeMissingValue" {
DATATYPE H5T_STRING {
STRSIZE 101;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
ATTRIBUTE "DimensionNames" {
DATATYPE H5T_STRING {
STRSIZE 21;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
}
DATASET "hist" {
DATATYPE H5T_STD_I32LE
DATASPACE SIMPLE { ( 30, 3, 3, 16, 2, 72, 28 ) / ( 30, 3, 3, 16, 2, 72, 28 ) }
ATTRIBUTE "CodeMissingValue" {
DATATYPE H5T_STRING {
STRSIZE 101;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
ATTRIBUTE "DimensionNames" {
DATATYPE H5T_STRING {
STRSIZE 25;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
}
DATASET "mean" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 3, 3, 16, 2, 72, 28 ) / ( 3, 3, 16, 2, 72, 28 ) }
ATTRIBUTE "CodeMissingValue" {
DATATYPE H5T_STRING {
STRSIZE 101;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
ATTRIBUTE "DimensionNames" {
DATATYPE H5T_STRING {
STRSIZE 21;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
}
DATASET "stdev" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 3, 3, 16, 2, 72, 28 ) / ( 3, 3, 16, 2, 72, 28 ) }
ATTRIBUTE "CodeMissingValue" {
DATATYPE H5T_STRING {
STRSIZE 101;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
ATTRIBUTE "DimensionNames" {
DATATYPE H5T_STRING {
STRSIZE 21;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
}
}
}
Any help would be appreciated.
Thanks,
-Justin
_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr
_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr
_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr
_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr
_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr
_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr
_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20160115/6bac193d/attachment-0001.html
More information about the gradsusr
mailing list