[gradsusr] HDF5 to GrADS ... how to create the descriptor file?
Karsten Haustein
karsten.haustein at bsc.es
Fri Nov 26 11:12:23 EST 2010
Hello,
I'm in trouble with HDF5 format since it is completely different to
HDF4. I'm working with version GrADS 2.0a9. As a matter of fact, I have
to create a descriptor file with UNPACK command etc. in order to open
the file (open command works only). The problem is, that I really don't
know, how to do that, although the GrADS manual is explaining things a
bit: http://iges.org/grads/gadoc/SDFdescriptorfile
Well, let's first have a look into the dumpfile created with n5dump-command:
HDF5 "LDA_PROD_LEV1_20040301.HDF5" {
GROUP "/" {
GROUP "LDA_PROD_LEV1_20040301" {
GROUP "CompositProduct" {
GROUP "stPROD1" {
DATASET "Latitude" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 3242096 ) / ( 3242096 ) }
DATA {
}
}
DATASET "Longitude" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 3242096 ) / ( 3242096 ) }
DATA {
}
}
}
GROUP "stPROD7" {
DATASET "AOD" {
DATATYPE H5T_STD_U8LE
DATASPACE SIMPLE { ( 3242096 ) / ( 3242096 ) }
DATA {
}
ATTRIBUTE "OFFSET" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): 0.05
}
}
ATTRIBUTE "SLOPE" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): 0.00570866
}
}
}
DATASET "AOD_ERR" {
DATATYPE H5T_STD_U16BE
DATASPACE SIMPLE { ( 3242096 ) / ( 3242096 ) }
DATA {
}
ATTRIBUTE "OFFSET" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): 0
}
}
ATTRIBUTE "SLOPE" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): 0.000152593
}
}
}
DATASET "BHR_ERR_VIS06" {
DATATYPE H5T_STD_I32BE
DATASPACE SIMPLE { ( 3242096 ) / ( 3242096 ) }
DATA {
}
ATTRIBUTE "OFFSET" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): 0
}
}
ATTRIBUTE "SLOPE" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): 0.000152593
}
}
}
DATASET "BHR_VIS06" {
DATATYPE H5T_STD_I32BE
DATASPACE SIMPLE { ( 3242096 ) / ( 3242096 ) }
DATA {
}
ATTRIBUTE "OFFSET" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): 0
}
}
ATTRIBUTE "SLOPE" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): 1.52593e-05
}
}
}
DATASET "CLASS" {
DATATYPE H5T_STD_U8LE
DATASPACE SIMPLE { ( 3242096 ) / ( 3242096 ) }
DATA {
}
ATTRIBUTE "SAEROTYPE" {
DATATYPE H5T_STRING {
STRSIZE 15;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SIMPLE { ( 4 ) / ( 4 ) }
DATA {
(0): "LDA_SPH_ABSORB", "LDA_SPH_MODABS",
"LDA_SPH_NONABS", "LDA_NSP_MEDRAD"
}
}
}
DATASET "FREPTIME" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 3242096 ) / ( 3242096 ) }
DATA {
}
}
DATASET "MaxAOD" {
DATATYPE H5T_STD_U8LE
DATASPACE SIMPLE { ( 3242096 ) / ( 3242096 ) }
DATA {
}
ATTRIBUTE "OFFSET" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): 0.05
}
}
ATTRIBUTE "SLOPE" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): 0.00570866
}
}
}
DATASET "MinAOD" {
DATATYPE H5T_STD_U8LE
DATASPACE SIMPLE { ( 3242096 ) / ( 3242096 ) }
DATA {
}
ATTRIBUTE "OFFSET" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): 0.05
}
}
ATTRIBUTE "SLOPE" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): 0.00570866
}
}
}
DATASET "PROBA" {
DATATYPE H5T_STD_I32BE
DATASPACE SIMPLE { ( 3242096 ) / ( 3242096 ) }
DATA {
}
ATTRIBUTE "OFFSET" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): 0
}
}
ATTRIBUTE "SLOPE" {
DATATYPE H5T_IEEE_F32BE
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): 1.52593e-05
}
}
}
DATASET "UISLOTSERIES" {
DATATYPE H5T_STD_I32BE
DATASPACE SIMPLE { ( 3242096 ) / ( 3242096 ) }
DATA {
}
}
}
}
}
}
}
Let me further give you some numbers extracted with HDFview (only the
AOD is of interest for me):
Longitude and Latitude are 1-dim arrays with the dimension 3162397
(32-bit float = H5T_IEEE_F32BE)
AOD is of the same size (1-dim with 3162397 entries), but
8-bit-unsigned-character (= H5T_STD_U8LE). Slope and offset according
dumpfile. Apart from the different entry size of the array, it is the same.
Thus, I strongly assume that one has to make use of the UNPACK command
since it seems to be packed non-float data which is obviously the
typical HDF5 feature (the two spatial dimensions are merged into one
large 1-dim array). But, how to use the UNPACK command? According
http://iges.org/grads/gadoc/descriptorfile.html
... I tried 2 versions:
UNPACK SLOPE OFFSET
UNPACK 0.00570866 0.05
It yields "Low Level I/O Error: Read error on data file" in each case.
But this might not be the problem, since I have to put some typical
XDEF, YDEF values which I don't know. I only know maximum and minimum
Lat/Lon values, but not the grid spacing. Maybe a specific DTYPE entry
would not require XDEF, YDEF anymore, but the GrADS version I'm using
(2.0a9) doesn't cope with entries like "hdf5_grid" or "hdfsds"! Instead,
it returns: "The invalid description file record is --> DTYPE
hdf5_grid". For this reason, I commented DTYPE to be able to open the
file at least. Up to now, I were using "DTYPE netcdf", which works fine
also for HDF4 files. Perhaps I have to use "units" for non-standard
binary files instead, but again, it doesn't work. I tried sth like
"AOD=>aod = -1,40,8". May be I'm not applying it correctly!?
Alternatively, any suggestion on how to convert HDF5 into HDF4 or NetCDF
would be highly appreciated. I tried "ncl_convert2nc" which works fine
for HDF4, but of course not for HDF5.
I'm using GrADS for all my model output (NetCDF as well as HDF4 files),
but now I got these HDF5 files (only available in this format
unfortunately) and I ran completely out of ideas how to get it read and
processed by GrADS. Since the gradsusr-archive also didn't help, I send
you this message, hoping that anybody out there have a clue or hint how
to settle this issue. It's basically nothing more than translating the
content of the HDF5-dumpfile into the GrADS descriptor file ... which
I'm unfortunately incapable of.
Thank you very much!
Best regards,
KarSteN
--
______________________________________________________________________
Karsten Haustein
Earth Sciences Division
Barcelona Supercomputing Center - Centro Nacional de Supercomputación
Building Nexus II -- Floor 1 -- Area C
c/ Jordi Girona 29, E-08034 Barcelona, Spain
phone: +34-934137581 fax +34-934137721
email: karsten.haustein at bsc.es
BSC web page: http://www.bsc.es/
DREAM model: http://www.bsc.es/projects/earthscience/DREAM/
______________________________________________________________________
WARNING / LEGAL TEXT: This message is intended only for the use of the
individual or entity to which it is addressed and may contain
information which is privileged, confidential, proprietary, or exempt
from disclosure under applicable law. If you are not the intended
recipient or the person responsible for delivering the message to the
intended recipient, you are strictly prohibited from disclosing,
distributing, copying, or in any way using this message. If you have
received this communication in error, please notify the sender and
destroy and delete any copies you may have received.
http://www.bsc.es/disclaimer.htm
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20101126/b0b5e8c5/attachment-0003.html
More information about the gradsusr
mailing list