[gradsusr] Control File Issues for HDF5 file
Justin Hicks
jhicks2014 at gmail.com
Fri Jan 15 11:04:07 EST 2016
Jennifer,
I appreciate the help. I am still pretty new to Grads and have taught
myself a majority of what I know.
I believe the EDEF is not operational with the 'linear' option, so I tried
the 'NAMES' option instead.
My updated control file is below:
DSET
/data2/GPM/GPM_3CMB.03/2015/3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5
OPTIONS template
Title GPM_3CMB.03
UNDEF -9999.0
DTYPE hdf5_grid
xdef 28 linear 1 1
ydef 72 linear 1 1
zdef 2 linear 1 1
tdef 16 linear 01mar2015 1mo
edef 3 names plrcnt0 plrcnt1 plrcnt2
vars 3
/Grids/G1/precipLiqRate/count=>plrcnt0 2 0,e,t,z,y,x plrcnt0
/Grids/G1/precipLiqRate/count=>plrcnt1 2 1,e,t,z,y,x plrcnt1
/Grids/G1/precipLiqRate/count=>plrcnt2 2 2,e,t,z,y,x plrcnt2
endvars
I also tried to play around with the edef names. However, with this control
file, I get the following error below in grads:
d plrcnt0
*** WARNING! ***
*** The I/O for this variable will be extremely slow
*** because the chunks are too big to fit in the cache
*** chunk size = 27648 bytes (72 * 2 * 16 * 3 * 1 * 4 bytes)
*** cache size = 16128 bytes
*** There are two ways to control the cache size:
*** add a CACHESIZE entry to the descriptor file
*** or use the 'set cachesf' command
*** Please read http://iges.org/grads/gadoc/compression.html
HDF5-DIAG: Error detected in HDF5 (1.8.7) thread 0:
#000: H5Dio.c line 174 in H5Dread(): can't read data
major: Dataset
minor: Read failed
#001: H5Dio.c line 348 in H5D_read(): src and dest data spaces have
different sizes
major: Invalid arguments to routine
minor: Bad value
H5Dread failed for int data
Data Request Error: Error for variable 'plrcnt0'
Error ocurred at column 1
DISPLAY error: Invalid expression
Expression = plrcnt0
-Justin
On Thu, Jan 14, 2016 at 11:52 AM, Jennifer M Adams <jadams21 at gmu.edu> wrote:
> The output from h5dump should have what you need, although what you
> included in your original post is missing some vital information.
>
> For the sake of example (so you know what to look for), below is some of
> the h5dump output that includes *DATA *with each attribute, which tells
> me how to map this grid into the GrADS data model. The descriptor file
> entries (also included below the h5dump output) illustrate how the data in
> the hdf5 attributes map to the GrADS descriptor file. Note how your h5dump
> output lists the attribute “DimensionNames” but doesn’t print out the
> actual values of that attribute (the DATA).
>
> DATASET "gaugeRelativeWeighting" {
> DATATYPE H5T_STD_I16LE
> DATASPACE SIMPLE { ( 3600, 1800 ) / ( 3600, 1800 ) }
> ATTRIBUTE "CodeMissingValue" {
> DATATYPE H5T_STRING {
> STRSIZE 6;
> STRPAD H5T_STR_NULLTERM;
> CSET H5T_CSET_ASCII;
> CTYPE H5T_C_S1;
> }
> DATASPACE SCALAR
> *DATA* {
> (0): "-9999"
> }
> }
> ATTRIBUTE "DIMENSION_LIST" {
> DATATYPE H5T_VLEN { H5T_REFERENCE}
> DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
> *DATA *{
> (0): (DATASET 598 /Grid/lon ), (DATASET 1147 /Grid/lat )
> }
> }
> ATTRIBUTE "DimensionNames" {
> DATATYPE H5T_STRING {
> STRSIZE 8;
> STRPAD H5T_STR_NULLTERM;
> CSET H5T_CSET_ASCII;
> CTYPE H5T_C_S1;
> }
> DATASPACE SCALAR
> *DATA* {
> (0): "lon,lat"
> }
> }
> ATTRIBUTE "Units" {
> DATATYPE H5T_STRING {
> STRSIZE 8;
> STRPAD H5T_STR_NULLTERM;
> CSET H5T_CSET_ASCII;
> CTYPE H5T_C_S1;
> }
> DATASPACE SCALAR
> *DATA *{
> (0): "percent"
> }
> }
>
> dset ^3B-MO.MS.MRG.3IMERG.20140312-S000000-E235959.03.V03D.HDF5
> dtype hdf5_grid
> undef -9999.9
> xdef 3600 linear -179.95 0.10
> ydef 1800 linear -89.95 0.10
> zdef 1 linear 1 1
> tdef 1 linear 01mar2014 1mo
> vars 1
> /Grid/precipitation=>p 0 x,y precipitation [mm/hr]
> endvars
>
> Your variables have a shape like this:
>
> 3, 3, 16, 2, 72, 28
>>>
>>> so you need to figure out how you’re going to map those into X, Y, Z, T,
> and E and then use numbers for the remaining dimensions.
>
> Without knowing anything about this data, you could try a complete
> abstract grid:
>
> xdef 28 linear 1 1
> ydef 72 linear 1 1
> zdef 2 linear 1 1
> tdef 16 linear 01jan0001 1mo
> edef 3 linear 1 1
> ...
> vars 3
> /Grids/G1/precipLiqRate/count=>plrcnt0 2 0,e,t,z,y,x plrcnt0
> /Grids/G1/precipLiqRate/count=>plrcnt1 2 1,e,t,z,y,x plrcnt1
> /Grids/G1/precipLiqRate/count=>plrcnt2 2 2,e,t,z,y,x plrcnt2
> endvars
>
> The variable that has an extra dimension (‘hist’ with DATASPACE SIMPLE {
> ( 30, 3, 3, 16, 2, 72, 28 ) / ( 30, 3, 3, 16, 2, 72, 28 ) }) is going to be
> divided further into more variables with an extra set of integers in the
> comma-delimited units entry of your descriptor file.
> Good luck!!
> —Jennifer
>
>
>
> On Jan 14, 2016, at 11:04 AM, Justin Hicks <jhicks2014 at gmail.com> wrote:
>
> Jennifer,
>
> Thank you. I will play around with this. However, it seems I would have to
> know the xdef, ydef, and zdef to take advantage of this. Any hints on how I
> could find out what these would be? What I have in my control file isn't
> correct; I just placed it there with the intention of fixing it later after
> the variables worked.
>
> -Justin
>
> On Thu, Jan 14, 2016 at 10:42 AM, Jennifer M Adams <jadams21 at gmu.edu>
> wrote:
>
>> Hi, Justin —
>> That message looks like your delcaration of the coordinates (“x,y”) is
>> wrong. Look at the DATASPACE SIMPLE attribute in the h5dump output, that
>> should give you a clue as to how many and what size the dimensions are. It
>> looks like ‘hist’ has one more dimension than ‘mean’ and these variables
>> have 6 or 7 dimensions. Have a look at item #4 in the Usage Notes of
>> http://cola.gmu.edu/grads/gadoc/SDFdescriptorfile.html for how to handle
>> this situation.
>> —Jennifer
>>
>> On Jan 14, 2016, at 10:04 AM, Justin Hicks <jhicks2014 at gmail.com> wrote:
>>
>> Jennifer,
>>
>> I put in the dataset names, like you said, but I still get a different
>> error. The error says:
>>
>> ga-> d plrcnt
>> HDF5-DIAG: Error detected in HDF5 (1.8.7) thread 0:
>> #000: H5Dio.c line 153 in H5Dread(): selection+offset not within extent
>> major: Dataspace
>> minor: Out of range
>> H5Dread failed for int data
>> Data Request Error: Error for variable 'plrcnt'
>> Error ocurred at column 1
>> DISPLAY error: Invalid expression
>> Expression = plrcnt
>>
>>
>>
>> -Justin
>>
>> On Wed, Jan 13, 2016 at 8:07 AM, Jennifer M Adams <jadams21 at gmu.edu>
>> wrote:
>>
>>> Justin,
>>> It looks like the variables in your descriptor files are group names not
>>> dataset names. Try this instead:
>>> /Grids/G1/precipLiqRate/count=>plrcnt 0 x,y liquid
>>> precipitation rate count
>>> /Grids/G1/precipLiqRate/hist=>plrhist 0 x,y liquid
>>> precipitation rate hist
>>> /Grids/G1/precipLiqRate/mean=>plrmean 0 x,y liquid
>>> precipitation rate mean
>>> /Grids/G1/precipLiqRate/stdev=>plrcstd 0 x,y liquid
>>> precipitation rate stdev
>>> —Jennifer
>>>
>>> On Jan 11, 2016, at 2:42 PM, Justin Hicks <jhicks2014 at gmail.com> wrote:
>>>
>>> Grads Users,
>>>
>>> I was able to open the control file in grads, but i get an error when I
>>> try to display a variable:
>>>
>>> ga-> d plrate
>>> HDF5-DIAG: Error detected in HDF5 (1.8.7) thread 0:
>>> #000: H5D.c line 341 in H5Dopen2(): not a dataset
>>> major: Dataset
>>> minor: Inappropriate type
>>> Error: H5Dopen2 failed for variable /Grids/G1/precipLiqRate
>>> Error: Variable /Grids/G1/precipLiqRate not in HDF5 file
>>> Data Request Error: Error for variable 'plrate'
>>> Error ocurred at column 1
>>> DISPLAY error: Invalid expression
>>> Expression = plrate
>>>
>>> The control file is below:
>>>
>>> DSET
>>> /data2/GPM/GPM_3CMB.03/2015/3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5
>>> OPTIONS template
>>> DTYPE hdf5_grid
>>> TITLE GPM precip
>>> UNDEF -999.0
>>> XDEF 3600 linear -180 0.1
>>> YDEF 300 linear -20 0.1
>>> ZDEF 1 levels 1000
>>> TDEF 1 linear 01mar2015 1dy
>>> VARS 4
>>> /Grids/G1/precipLiqRate=>plrate 0 x,y liquid precipitation
>>> rate
>>> /Grids/G1/precipLiqWaterContent=>plcon 0 x,y liquid precipitation
>>> content
>>> /Grids/G1/precipTotDm=>ptot 0 x,y precipitation total
>>> /Grids/G1/precipTotRate=>ptotr 0 x,y precipitation total
>>> rate
>>> ENDVARS
>>>
>>>
>>> I don't think the xdef and ydef are completely accurate, but I shouldn't
>>> get that error message even if the edef and ydef are wrong. The file
>>> contents of h5dump are below:
>>>
>>> /home/jhicks/h5dump -n
>>> 3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5
>>> HDF5 "3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5" {
>>> FILE_CONTENTS {
>>> group /
>>> group /Grids
>>> group /Grids/G1
>>> dataset /Grids/G1/precipAllObs
>>> group /Grids/G1/precipLiqRate
>>> dataset /Grids/G1/precipLiqRate/count
>>> dataset /Grids/G1/precipLiqRate/hist
>>> dataset /Grids/G1/precipLiqRate/mean
>>> dataset /Grids/G1/precipLiqRate/stdev
>>> group /Grids/G1/precipLiqWaterContent
>>> dataset /Grids/G1/precipLiqWaterContent/count
>>> dataset /Grids/G1/precipLiqWaterContent/hist
>>> dataset /Grids/G1/precipLiqWaterContent/mean
>>> dataset /Grids/G1/precipLiqWaterContent/stdev
>>> group /Grids/G1/precipTotDm
>>> dataset /Grids/G1/precipTotDm/count
>>> dataset /Grids/G1/precipTotDm/hist
>>> dataset /Grids/G1/precipTotDm/mean
>>> dataset /Grids/G1/precipTotDm/stdev
>>> group /Grids/G1/precipTotLogNw
>>> dataset /Grids/G1/precipTotLogNw/count
>>> dataset /Grids/G1/precipTotLogNw/hist
>>> dataset /Grids/G1/precipTotLogNw/mean
>>> dataset /Grids/G1/precipTotLogNw/stdev
>>> group /Grids/G1/precipTotRate
>>> dataset /Grids/G1/precipTotRate/count
>>> dataset /Grids/G1/precipTotRate/hist
>>> dataset /Grids/G1/precipTotRate/mean
>>> dataset /Grids/G1/precipTotRate/stdev
>>> group /Grids/G1/precipTotWaterContent
>>> dataset /Grids/G1/precipTotWaterContent/count
>>> dataset /Grids/G1/precipTotWaterContent/hist
>>> dataset /Grids/G1/precipTotWaterContent/mean
>>> dataset /Grids/G1/precipTotWaterContent/stdev
>>> dataset /Grids/G1/surfPrecipLiqRateProb
>>> dataset /Grids/G1/surfPrecipLiqRateUn
>>> group /Grids/G1/surfPrecipTotRateDiurnal
>>> dataset /Grids/G1/surfPrecipTotRateDiurnal/count
>>> dataset /Grids/G1/surfPrecipTotRateDiurnal/mean
>>> dataset /Grids/G1/surfPrecipTotRateDiurnal/stdev
>>> dataset /Grids/G1/surfPrecipTotRateDiurnalAllObs
>>> dataset /Grids/G1/surfPrecipTotRateProb
>>> dataset /Grids/G1/surfPrecipTotRateUn
>>> group /Grids/G2
>>> dataset /Grids/G2/precipAllObs
>>> group /Grids/G2/precipLiqRate
>>> dataset /Grids/G2/precipLiqRate/count
>>> dataset /Grids/G2/precipLiqRate/mean
>>> dataset /Grids/G2/precipLiqRate/stdev
>>> group /Grids/G2/precipLiqWaterContent
>>> dataset /Grids/G2/precipLiqWaterContent/count
>>> dataset /Grids/G2/precipLiqWaterContent/mean
>>> dataset /Grids/G2/precipLiqWaterContent/stdev
>>> group /Grids/G2/precipTotDm
>>> dataset /Grids/G2/precipTotDm/count
>>> dataset /Grids/G2/precipTotDm/mean
>>> dataset /Grids/G2/precipTotDm/stdev
>>> group /Grids/G2/precipTotLogNw
>>> dataset /Grids/G2/precipTotLogNw/count
>>> dataset /Grids/G2/precipTotLogNw/mean
>>> dataset /Grids/G2/precipTotLogNw/stdev
>>> group /Grids/G2/precipTotRate
>>> dataset /Grids/G2/precipTotRate/count
>>> dataset /Grids/G2/precipTotRate/mean
>>> dataset /Grids/G2/precipTotRate/stdev
>>> group /Grids/G2/precipTotWaterContent
>>> dataset /Grids/G2/precipTotWaterContent/count
>>> dataset /Grids/G2/precipTotWaterContent/mean
>>> dataset /Grids/G2/precipTotWaterContent/stdev
>>> dataset /Grids/G2/surfPrecipLiqRateProb
>>> dataset /Grids/G2/surfPrecipLiqRateUn
>>> group /Grids/G2/surfPrecipTotRateDiurnal
>>> dataset /Grids/G2/surfPrecipTotRateDiurnal/count
>>> dataset /Grids/G2/surfPrecipTotRateDiurnal/mean
>>> dataset /Grids/G2/surfPrecipTotRateDiurnal/stdev
>>> dataset /Grids/G2/surfPrecipTotRateDiurnalAllObs
>>> dataset /Grids/G2/surfPrecipTotRateProb
>>> dataset /Grids/G2/surfPrecipTotRateUn
>>> dataset /InputAlgorithmVersions
>>> dataset /InputFileNames
>>> dataset /InputGenerationDateTimes
>>> }
>>> }
>>>
>>>
>>> The group contents of /Grids/G1/precipLiqRate are below:
>>>
>>> h5dump -H -g/Grids/G1/precipLiqRate
>>> 3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5
>>> HDF5 "3B-MO.GPM.DPRGMI.CORRAGM.20150301-S000000-E235959.03.V03D.HDF5" {
>>> GROUP "/Grids/G1/precipLiqRate" {
>>> DATASET "count" {
>>> DATATYPE H5T_STD_I32LE
>>> DATASPACE SIMPLE { ( 3, 3, 16, 2, 72, 28 ) / ( 3, 3, 16, 2, 72,
>>> 28 ) }
>>> ATTRIBUTE "CodeMissingValue" {
>>> DATATYPE H5T_STRING {
>>> STRSIZE 101;
>>> STRPAD H5T_STR_NULLTERM;
>>> CSET H5T_CSET_ASCII;
>>> CTYPE H5T_C_S1;
>>> }
>>> DATASPACE SCALAR
>>> }
>>> ATTRIBUTE "DimensionNames" {
>>> DATATYPE H5T_STRING {
>>> STRSIZE 21;
>>> STRPAD H5T_STR_NULLTERM;
>>> CSET H5T_CSET_ASCII;
>>> CTYPE H5T_C_S1;
>>> }
>>> DATASPACE SCALAR
>>> }
>>> }
>>> DATASET "hist" {
>>> DATATYPE H5T_STD_I32LE
>>> DATASPACE SIMPLE { ( 30, 3, 3, 16, 2, 72, 28 ) / ( 30, 3, 3, 16,
>>> 2, 72, 28 ) }
>>> ATTRIBUTE "CodeMissingValue" {
>>> DATATYPE H5T_STRING {
>>> STRSIZE 101;
>>> STRPAD H5T_STR_NULLTERM;
>>> CSET H5T_CSET_ASCII;
>>> CTYPE H5T_C_S1;
>>> }
>>> DATASPACE SCALAR
>>> }
>>> ATTRIBUTE "DimensionNames" {
>>> DATATYPE H5T_STRING {
>>> STRSIZE 25;
>>> STRPAD H5T_STR_NULLTERM;
>>> CSET H5T_CSET_ASCII;
>>> CTYPE H5T_C_S1;
>>> }
>>> DATASPACE SCALAR
>>> }
>>> }
>>> DATASET "mean" {
>>> DATATYPE H5T_IEEE_F32LE
>>> DATASPACE SIMPLE { ( 3, 3, 16, 2, 72, 28 ) / ( 3, 3, 16, 2, 72,
>>> 28 ) }
>>> ATTRIBUTE "CodeMissingValue" {
>>> DATATYPE H5T_STRING {
>>> STRSIZE 101;
>>> STRPAD H5T_STR_NULLTERM;
>>> CSET H5T_CSET_ASCII;
>>> CTYPE H5T_C_S1;
>>> }
>>> DATASPACE SCALAR
>>> }
>>> ATTRIBUTE "DimensionNames" {
>>> DATATYPE H5T_STRING {
>>> STRSIZE 21;
>>> STRPAD H5T_STR_NULLTERM;
>>> CSET H5T_CSET_ASCII;
>>> CTYPE H5T_C_S1;
>>> }
>>> DATASPACE SCALAR
>>> }
>>> }
>>> DATASET "stdev" {
>>> DATATYPE H5T_IEEE_F32LE
>>> DATASPACE SIMPLE { ( 3, 3, 16, 2, 72, 28 ) / ( 3, 3, 16, 2, 72,
>>> 28 ) }
>>> ATTRIBUTE "CodeMissingValue" {
>>> DATATYPE H5T_STRING {
>>> STRSIZE 101;
>>> STRPAD H5T_STR_NULLTERM;
>>> CSET H5T_CSET_ASCII;
>>> CTYPE H5T_C_S1;
>>> }
>>> DATASPACE SCALAR
>>> }
>>> ATTRIBUTE "DimensionNames" {
>>> DATATYPE H5T_STRING {
>>> STRSIZE 21;
>>> STRPAD H5T_STR_NULLTERM;
>>> CSET H5T_CSET_ASCII;
>>> CTYPE H5T_C_S1;
>>> }
>>> DATASPACE SCALAR
>>> }
>>> }
>>> }
>>> }
>>>
>>>
>>> Any help would be appreciated.
>>>
>>> Thanks,
>>>
>>> -Justin
>>> _______________________________________________
>>> gradsusr mailing list
>>> gradsusr at gradsusr.org
>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>
>>>
>>>
>>> _______________________________________________
>>> gradsusr mailing list
>>> gradsusr at gradsusr.org
>>> http://gradsusr.org/mailman/listinfo/gradsusr
>>>
>>>
>> _______________________________________________
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>>
>>
>>
>> _______________________________________________
>> gradsusr mailing list
>> gradsusr at gradsusr.org
>> http://gradsusr.org/mailman/listinfo/gradsusr
>>
>>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
>
>
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20160115/5ae7327d/attachment-0001.html
More information about the gradsusr
mailing list