[gradsusr] Performance hit with data descriptor files?
Colarco, Peter R. (GSFC-6133)
peter.r.colarco at nasa.gov
Mon Jul 12 21:26:30 EDT 2010
Hi-
Apologies, here is the requested information.
Thanks,
Pete
OS:
% uname -a
Linux redacted.gsfc.nasa.gov<http://redacted.gsfc.nasa.gov> 2.6.32-gentoo-r7 #2 SMP Thu Jul 8 11:39:03 EDT 2010 x86_64 Intel(R) Xeon(R) CPU X5460 @ 3.16GHz GenuineIntel GNU/Linux
File Information:
% ncdump -hs /path/to/file/Y2003/M01/MYD04_L2_ocn.aero_tc8_051.qawt.20030101.nc4
netcdf MYD04_L2_ocn.aero_tc8_051.qawt.20030101 {
dimensions:
time = UNLIMITED ; // (8 currently)
levels = 7 ;
longitude = 1152 ;
latitude = 721 ;
variables:
double time(time) ;
time:units = "hours since 2003-1-1 1" ;
time:_Storage = "chunked" ;
time:_ChunkSizes = 1 ;
double levels(levels) ;
levels:units = "hPa" ;
levels:description = "Pressure level" ;
levels:type = "plev" ;
levels:long_name = "Level" ;
levels:positive = "down" ;
levels:_Storage = "contiguous" ;
double longitude(longitude) ;
longitude:units = "degrees_east" ;
longitude:long_name = "Longitude" ;
longitude:_Storage = "contiguous" ;
double latitude(latitude) ;
latitude:units = "degrees_north" ;
latitude:long_name = "Latitude" ;
latitude:_Storage = "contiguous" ;
float aodtau(time, levels, latitude, longitude) ;
aodtau:comments = "Unknown1 variable comment" ;
aodtau:long_name = "aodtau" ;
aodtau:units = "" ;
aodtau:grid_name = "grid01" ;
aodtau:grid_type = "linear" ;
aodtau:time_statistic = "instantaneous" ;
aodtau:missing_value = 1.e+15f ;
aodtau:_Storage = "chunked" ;
aodtau:_ChunkSizes = 1, 1, 721, 1152 ;
aodtau:_DeflateLevel = 2 ;
// global attributes:
:Conventions = "COARDS" ;
:calendar = "standard" ;
:comments = "File" ;
:model = "geos/das" ;
:center = "gsfc" ;
:_Format = "netCDF-4" ;
}
ga-> q config
Config: v2.0.a7.oga.3 little-endian readline printim grib2 netcdf hdf4-sds hdf5 opendap-grids,stn athena geotiff
Grid Analysis and Display System (GrADS) Version 2.0.a7.oga.3
Copyright (c) 1988-2009 by Brian Doty and the
Institute for Global Environment and Society (IGES)
This program is distributed WITHOUT ANY WARRANTY
See file COPYRIGHT for more information.
Built Thu Oct 29 17:57:08 EDT 2009 for x86_64-unknown-linux-gnu
This version of GrADS has been configured with the following options:
o Built on a LITTLE ENDIAN machine
o Command line editing ENABLED
http://tiswww.case.edu/php/chet/readline/rltop.html
o printim command for image output ENABLED
http://www.zlib.net
http://www.libpng.org/pub/png/libpng.html
http://www.libgd.org/Main_Page
o GRIB2 interface ENABLED
http://www.ijg.org
http://www.ece.uvic.ca/~mdadams/jasper
http://www.nco.ncep.noaa.gov/pmb/codes/GRIB2
g2clib-1.0.5
o NetCDF interface ENABLED
http://www.opendap.org
libnc-dap 4.0.1-beta3-snapshot2009021712 of Mar 3 2009 14:13:33 $
o HDF interface ENABLED
http://hdfgroup.org
HDF 4.2r3
HDF5 1.8.2
o Athena Widget GUI ENABLED
o OPeNDAP gridded data interface ENABLED
http://www.opendap.org
libdap 3.7.10
o OPeNDAP station data interface ENABLED
http://iges.org/grads/gadoc/supplibs.html
libgadap 2.0.oga.1
o GeoTIFF and KML output ENABLED
http://www.libtiff.org
http://geotiff.osgeo.org
For additional information please consult http://iges.org/grads
On Jul 12, 2010, at 4:51 PM, Jennifer Adams wrote:
Could you please send the output from ncdump -hs (the s is for compression info, since you have netcdf-4 files). Also send the output from 'q config' and the specs of the OS you are running on. These things are important for solving every problem.
--Jennifer
On Jul 12, 2010, at 4:42 PM, Colarco, Peter R. (GSFC-6133) wrote:
Hi-
I am running opengrads 2.0.a7.oga.3/x86_64.
I have a very simple data descriptor file (below) to template a set of large-ish files (gridded 1152 x 721 x 7 points). When I simply "sdfopen" the first file and plot the first variable/first time/first level, I get my result essentially immediately. When I instead "xdfopen" the data descriptor file and try to plot the same, it takes some time (~10 sec) to plot. When I repeat the same sort of thing but operate on several times the difference in performance between the two methods is of course very noticeable.
The issue seems to be in the "options template" line below; taking that out and explicitly putting the first file in I can "xdfopen" the data descriptor file and plot the first variable essentially instantly. Additional metadata in the data descriptor file (e.g., from make_ctl.sh) does not help or matter as far as I can tell.
Can anyone suggest a solution to this problem? Am I doing something weird or wrong here?
Data Descriptor File:
dset /path/to/my/data/Y2003/M01/MYD04_L2_ocn.aero_tc8_051.qawt.%y4%m2%d2.nc4
options template
tdef time 248 linear 0z01jan2003 3hr
Thank you,
Pete Colarco
--------------------------------------------------------------------
Peter Colarco
NASA GSFC
Code 613.3
NASA Goddard Space Flight Center
Greenbelt, MD 20771
301.614.6382 (ph)
301.614.5903 (fax)
peter.r.colarco at nasa.gov<mailto:peter.r.colarco at nasa.gov>
http://hyperion.gsfc.nasa.gov/People/Colarco
--------------------------------------------------------------------
_______________________________________________
gradsusr mailing list
gradsusr at gradsusr.org<mailto:gradsusr at gradsusr.org>
http://gradsusr.org/mailman/listinfo/gradsusr
--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma at cola.iges.org<mailto:jma at cola.iges.org>
<ATT00001..txt>
--------------------------------------------------------------------
Peter Colarco
NASA GSFC
Code 613.3
NASA Goddard Space Flight Center
Greenbelt, MD 20771
301.614.6382 (ph)
301.614.5903 (fax)
peter.r.colarco at nasa.gov<mailto:peter.r.colarco at nasa.gov>
http://hyperion.gsfc.nasa.gov/People/Colarco
--------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://gradsusr.org/pipermail/gradsusr/attachments/20100712/29c4a173/attachment-0003.html
More information about the gradsusr
mailing list