[gradsusr] HDF5 to GrADS ... how to create the descriptor file?

Karsten Haustein karsten.haustein at bsc.es
Fri Nov 26 11:12:23 EST 2010


Hello,

I'm in trouble with HDF5 format since it is completely different to 
HDF4. I'm working with version GrADS 2.0a9. As a matter of fact,  I have 
to create a descriptor file with UNPACK command etc. in order to open 
the file (open command works only). The problem is, that I really don't 
know, how to do that, although the GrADS manual is explaining things a 
bit: http://iges.org/grads/gadoc/SDFdescriptorfile

Well, let's first have a look into the dumpfile created with n5dump-command:

HDF5 "LDA_PROD_LEV1_20040301.HDF5" {
GROUP "/" {
    GROUP "LDA_PROD_LEV1_20040301" {
       GROUP "CompositProduct" {
          GROUP "stPROD1" {
             DATASET "Latitude" {
                DATATYPE  H5T_IEEE_F32BE
                DATASPACE  SIMPLE { ( 3242096 ) / ( 3242096 ) }
                DATA {
                }
             }
             DATASET "Longitude" {
                DATATYPE  H5T_IEEE_F32BE
                DATASPACE  SIMPLE { ( 3242096 ) / ( 3242096 ) }
                DATA {
                }
             }
          }
          GROUP "stPROD7" {
             DATASET "AOD" {
                DATATYPE  H5T_STD_U8LE
                DATASPACE  SIMPLE { ( 3242096 ) / ( 3242096 ) }
                DATA {
                }
                ATTRIBUTE "OFFSET" {
                   DATATYPE  H5T_IEEE_F32BE
                   DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
                   DATA {
                   (0): 0.05
                   }
                }
                ATTRIBUTE "SLOPE" {
                   DATATYPE  H5T_IEEE_F32BE
                   DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
                   DATA {
                   (0): 0.00570866
                   }
                }
             }
             DATASET "AOD_ERR" {
                DATATYPE  H5T_STD_U16BE
                DATASPACE  SIMPLE { ( 3242096 ) / ( 3242096 ) }
                DATA {
                }
                ATTRIBUTE "OFFSET" {
                   DATATYPE  H5T_IEEE_F32BE
                   DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
                   DATA {
                   (0): 0
                   }
                }
                ATTRIBUTE "SLOPE" {
                   DATATYPE  H5T_IEEE_F32BE
                   DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
                   DATA {
                   (0): 0.000152593
                   }
                }
             }
             DATASET "BHR_ERR_VIS06" {
                DATATYPE  H5T_STD_I32BE
                DATASPACE  SIMPLE { ( 3242096 ) / ( 3242096 ) }
                DATA {
                }
                ATTRIBUTE "OFFSET" {
                   DATATYPE  H5T_IEEE_F32BE
                   DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
                   DATA {
                   (0): 0
                   }
                }
                ATTRIBUTE "SLOPE" {
                   DATATYPE  H5T_IEEE_F32BE
                   DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
                   DATA {
                   (0): 0.000152593
                   }
                }
             }
             DATASET "BHR_VIS06" {
                DATATYPE  H5T_STD_I32BE
                DATASPACE  SIMPLE { ( 3242096 ) / ( 3242096 ) }
                DATA {
                }
                ATTRIBUTE "OFFSET" {
                   DATATYPE  H5T_IEEE_F32BE
                   DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
                   DATA {
                   (0): 0
                   }
                }
                ATTRIBUTE "SLOPE" {
                   DATATYPE  H5T_IEEE_F32BE
                   DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
                   DATA {
                   (0): 1.52593e-05
                   }
                }
             }
             DATASET "CLASS" {
                DATATYPE  H5T_STD_U8LE
                DATASPACE  SIMPLE { ( 3242096 ) / ( 3242096 ) }
                DATA {
                }
                ATTRIBUTE "SAEROTYPE" {
                   DATATYPE  H5T_STRING {
                         STRSIZE 15;
                         STRPAD H5T_STR_NULLTERM;
                         CSET H5T_CSET_ASCII;
                         CTYPE H5T_C_S1;
                      }
                   DATASPACE  SIMPLE { ( 4 ) / ( 4 ) }
                   DATA {
                   (0): "LDA_SPH_ABSORB", "LDA_SPH_MODABS", 
"LDA_SPH_NONABS", "LDA_NSP_MEDRAD"
                   }
                }
             }
             DATASET "FREPTIME" {
                DATATYPE  H5T_IEEE_F32BE
                DATASPACE  SIMPLE { ( 3242096 ) / ( 3242096 ) }
                DATA {
                }
             }
             DATASET "MaxAOD" {
                DATATYPE  H5T_STD_U8LE
                DATASPACE  SIMPLE { ( 3242096 ) / ( 3242096 ) }
                DATA {
                }
                ATTRIBUTE "OFFSET" {
                   DATATYPE  H5T_IEEE_F32BE
                   DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
                   DATA {
                   (0): 0.05
                   }
                }
                ATTRIBUTE "SLOPE" {
                   DATATYPE  H5T_IEEE_F32BE
                   DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
                   DATA {
                   (0): 0.00570866
                   }
                }
             }
             DATASET "MinAOD" {
                DATATYPE  H5T_STD_U8LE
                DATASPACE  SIMPLE { ( 3242096 ) / ( 3242096 ) }
                DATA {
                }
                ATTRIBUTE "OFFSET" {
                   DATATYPE  H5T_IEEE_F32BE
                   DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
                   DATA {
                   (0): 0.05
                   }
                }
                ATTRIBUTE "SLOPE" {
                   DATATYPE  H5T_IEEE_F32BE
                   DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
                   DATA {
                   (0): 0.00570866
                   }
                }
             }
             DATASET "PROBA" {
                DATATYPE  H5T_STD_I32BE
                DATASPACE  SIMPLE { ( 3242096 ) / ( 3242096 ) }
                DATA {
                }
                ATTRIBUTE "OFFSET" {
                   DATATYPE  H5T_IEEE_F32BE
                   DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
                   DATA {
                   (0): 0
                   }
                }
                ATTRIBUTE "SLOPE" {
                   DATATYPE  H5T_IEEE_F32BE
                   DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
                   DATA {
                   (0): 1.52593e-05
                   }
                }
             }
             DATASET "UISLOTSERIES" {
                DATATYPE  H5T_STD_I32BE
                DATASPACE  SIMPLE { ( 3242096 ) / ( 3242096 ) }
                DATA {
                }
             }
          }
       }
    }
}
}

Let me further give you some numbers extracted with HDFview (only the 
AOD is of interest for me):
Longitude and Latitude are 1-dim arrays with the dimension 3162397 
(32-bit float = H5T_IEEE_F32BE)
AOD is of the same size (1-dim with 3162397 entries), but 
8-bit-unsigned-character (= H5T_STD_U8LE). Slope and offset according 
dumpfile. Apart from the different entry size of the array, it is the same.

Thus, I strongly assume that one has to make use of the UNPACK command 
since it seems to be packed non-float data which is obviously the 
typical HDF5 feature (the two spatial dimensions are merged into one 
large 1-dim array). But, how to use the UNPACK command? According 
http://iges.org/grads/gadoc/descriptorfile.html
... I tried 2 versions:
UNPACK SLOPE OFFSET
UNPACK 0.00570866 0.05
It yields "Low Level I/O Error:  Read error on data file" in each case.

But this might not be the problem, since I have to put some typical 
XDEF, YDEF values which I don't know. I only know maximum and minimum 
Lat/Lon values, but not the grid spacing. Maybe a specific DTYPE entry 
would not require XDEF, YDEF anymore, but the GrADS version I'm using 
(2.0a9) doesn't cope with entries like "hdf5_grid" or "hdfsds"! Instead, 
it returns: "The invalid description file record is --> DTYPE 
hdf5_grid". For this reason, I commented DTYPE to be able to open the 
file at least. Up to now, I were using "DTYPE netcdf", which works fine 
also for HDF4 files. Perhaps I have to use "units" for non-standard 
binary files instead, but again, it doesn't work. I tried sth like 
"AOD=>aod = -1,40,8". May be I'm not applying it correctly!?

Alternatively, any suggestion on how to convert HDF5 into HDF4 or NetCDF 
would be highly appreciated. I tried "ncl_convert2nc" which works fine 
for HDF4, but of course not for HDF5.

I'm using GrADS for all my model output (NetCDF as well as HDF4 files), 
but now I got these HDF5 files (only available in this format 
unfortunately) and I ran completely out of ideas how to get it read and 
processed by GrADS. Since the gradsusr-archive also didn't help, I send 
you this message, hoping that anybody out there have a clue or hint how 
to settle this issue. It's basically nothing more than translating the 
content of the HDF5-dumpfile into the GrADS descriptor file ... which 
I'm unfortunately incapable of.

Thank you very much!

Best regards,
KarSteN


-- 
______________________________________________________________________

Karsten Haustein
Earth Sciences Division
Barcelona Supercomputing Center - Centro Nacional de Supercomputación
Building Nexus II -- Floor 1 -- Area C
c/ Jordi Girona 29, E-08034 Barcelona, Spain
phone: +34-934137581 fax +34-934137721
email: karsten.haustein at bsc.es

BSC web page: http://www.bsc.es/
DREAM model: http://www.bsc.es/projects/earthscience/DREAM/
______________________________________________________________________



WARNING / LEGAL TEXT: This message is intended only for the use of the
individual or entity to which it is addressed and may contain
information which is privileged, confidential, proprietary, or exempt
from disclosure under applicable law. If you are not the intended
recipient or the person responsible for delivering the message to the
intended recipient, you are strictly prohibited from disclosing,
distributing, copying, or in any way using this message. If you have
received this communication in error, please notify the sender and
destroy and delete any copies you may have received.

http://www.bsc.es/disclaimer.htm
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20101126/b0b5e8c5/attachment-0003.html 


More information about the gradsusr mailing list