[gradsusr] template option suddenly slow for nc files
Jennifer M Adams
jadams21 at gmu.edu
Thu Apr 18 21:05:40 EDT 2024
Eric, can you try running ncdump with the -s option to show the chunking parameters? It may be that the chunk shapes and sizes are different and not entirely compatible with the GrADS I/O pattern.
--Jennifer
________________________________
From: gradsusr <gradsusr-bounces at gradsusr.org> on behalf of Eric A. Aligo <eric.aligo at noaa.gov>
Sent: Thursday, April 18, 2024 8:37 AM
To: gradsusr at gradsusr.org <gradsusr at gradsusr.org>
Subject: [gradsusr] template option suddenly slow for nc files
Plotting one level and one time from a 3D field takes 3 seconds from a netcdf file using one version of FV3 code, but now it takes 2.5 minutes with a new version of FV3 code. The differences I see in the header of the netcdf file are: the version of netcdf and hdf5, ncnsto, and grid_id. The fill/undefined values are also different. My control file is below. Is this a known issue with GraDS and netcdf/hdf5? A bug?
dset ^dynf%h3.nc
dtype netcdf
options template
undef 9.99e+20
title FV3 history
xdef 2501 linear -135 0.03
ydef 1051 linear 22.0 0.03
tdef 3 linear 00Z20JUN2019 1hr
zdef 65 linear 1 1
VARS 1
dzdt 65 t,z,y,x Vertical velocity
endvars
ncdump -h:
:_NCProperties = "version=2,netcdf=4.9.2,hdf5=1.14.0" ; <-----takes 2.5 minutes to plot field
:_NCProperties = "version=2,netcdf=4.7.4,hdf5=1.10.6," ; <-----takes 3 seconds to plot field
Here's some of the header of the netcdf file that behaves normally:
netcdf dynf002 {
dimensions:
grid_xt = 2501 ;
grid_yt = 1051 ;
nchars = 20 ;
pfull = 65 ;
phalf = 66 ;
time = 1 ;
float dzdt(time, pfull, grid_yt, grid_xt) ;
dzdt:_FillValue = -1.e+10f ;
dzdt:cell_methods = "time: point" ;
dzdt:long_name = "vertical wind" ;
dzdt:missing_value = -1.e+10f ;
dzdt:output_file = "dyn" ;
dzdt:units = "m/sec" ;
// global attributes:
:_NCProperties = "version=2,netcdf=4.7.4,hdf5=1.10.6," ;
:ak = 212.637f, 452.613f, 723.285f, 1028.378f, 1372.014f, 1758.738f, 2193.544f, 2681.893f, 3229.725f, 3843.468f, 4530.025f, 5296.374f, 6132.608f, 7004.872f, 7877.419f, 8714.452f, 9480.727f, 10155.56f, 10734.38f, 11213.79f, 11591.01f, 11863.99f, 12031.62f, 12093.93f, 12052.24f, 11909.32f, 11669.5f, 11338.72f, 10924.5f, 10435.84f, 9883.087f, 9277.633f, 8631.631f, 7957.623f, 7268.15f, 6575.364f, 5890.665f, 5224.384f, 4585.535f, 3981.638f, 3418.626f, 2900.827f, 2431.016f, 2010.516f, 1639.35f, 1316.413f, 1039.659f, 806.289f, 612.932f, 455.809f, 330.883f, 233.98f, 160.901f, 107.5f, 69.754f, 43.805f, 26.13f, 14.45f, 7.211f, 3.103f, 1.058f, 0.234f, 0.017f, 0.f, 0.f, 0.f ;
:bk = 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 3.79e-06f, 0.00018806f, 0.00097247f, 0.00279793f, 0.00610706f, 0.01133629f, 0.01877378f, 0.02854147f, 0.04074101f, 0.05545674f, 0.07275035f, 0.09265514f, 0.1151704f, 0.1402563f, 0.1678291f, 0.1977584f, 0.2298655f, 0.2639235f, 0.2996609f, 0.3367662f, 0.3748959f, 0.4136837f, 0.4527515f, 0.4917214f, 0.530227f, 0.5679249f, 0.6045035f, 0.6396912f, 0.6732613f, 0.7050351f, 0.7348821f, 0.7627185f, 0.7885046f, 0.8122396f, 0.833957f, 0.8537182f, 0.8716077f, 0.8877271f, 0.9021903f, 0.9151191f, 0.926639f, 0.9368768f, 0.9459575f, 0.9540022f, 0.9611273f, 0.9674414f, 0.9730366f, 0.9779925f, 0.9823816f, 0.9862693f, 0.9897139f, 0.992767f, 0.9954739f, 0.9978736f, 1.f ;
:dlat = 0.03f ;
:dlon = 0.03f ;
:grid = "latlon" ;
:grid_id = 1 ;
:hydrostatic = "non-hydrostatic" ;
:lat1 = 22.f ;
:lat2 = 53.5f ;
:lon1 = -135.f ;
:lon2 = -60.f ;
:ncnsto = 13 ;
:source = "FV3GFS" ;
Here's the header of the netcdf file that doesn't behave normally:
netcdf dynf002 {
dimensions:
grid_xt = 2501 ;
grid_yt = 1051 ;
nchars = 20 ;
pfull = 65 ;
phalf = 66 ;
time = 1 ;
float dzdt(time, pfull, grid_yt, grid_xt) ;
dzdt:_FillValue = 9.99e+20f ;
dzdt:cell_methods = "time: point" ;
dzdt:long_name = "vertical wind" ;
dzdt:missing_value = 9.99e+20f ;
dzdt:output_file = "dyn" ;
dzdt:units = "m/sec" ;
// global attributes:
:ak = 212.637f, 452.613f, 723.285f, 1028.378f, 1372.014f, 1758.738f, 2193.544f, 2681.893f, 3229.725f, 3843.468f, 4530.025f, 5296.374f, 6132.608f, 7004.872f, 7877.419f, 8714.452f, 9480.727f, 10155.56f, 10734.38f, 11213.79f, 11591.01f, 11863.99f, 12031.62f, 12093.93f, 12052.24f, 11909.32f, 11669.5f, 11338.72f, 10924.5f, 10435.84f, 9883.087f, 9277.633f, 8631.631f, 7957.623f, 7268.15f, 6575.364f, 5890.665f, 5224.384f, 4585.535f, 3981.638f, 3418.626f, 2900.827f, 2431.016f, 2010.516f, 1639.35f, 1316.413f, 1039.659f, 806.289f, 612.932f, 455.809f, 330.883f, 233.98f, 160.901f, 107.5f, 69.754f, 43.805f, 26.13f, 14.45f, 7.211f, 3.103f, 1.058f, 0.234f, 0.017f, 0.f, 0.f, 0.f ;
:bk = 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 3.79e-06f, 0.00018806f, 0.00097247f, 0.00279793f, 0.00610706f, 0.01133629f, 0.01877378f, 0.02854147f, 0.04074101f, 0.05545674f, 0.07275035f, 0.09265514f, 0.1151704f, 0.1402563f, 0.1678291f, 0.1977584f, 0.2298655f, 0.2639235f, 0.2996609f, 0.3367662f, 0.3748959f, 0.4136837f, 0.4527515f, 0.4917214f, 0.530227f, 0.5679249f, 0.6045035f, 0.6396912f, 0.6732613f, 0.7050351f, 0.7348821f, 0.7627185f, 0.7885046f, 0.8122396f, 0.833957f, 0.8537182f, 0.8716077f, 0.8877271f, 0.9021903f, 0.9151191f, 0.926639f, 0.9368768f, 0.9459575f, 0.9540022f, 0.9611273f, 0.9674414f, 0.9730366f, 0.9779925f, 0.9823816f, 0.9862693f, 0.9897139f, 0.992767f, 0.9954739f, 0.9978736f, 1.f ;
:dlat = 0.03f ;
:dlon = 0.03f ;
:grid = "latlon" ;
:grid_id = 1L ;
:hydrostatic = "non-hydrostatic" ;
:lat1 = 22.f ;
:lat2 = 53.5f ;
:lon1 = -135.f ;
:lon2 = -60.f ;
:ncnsto = 16 ;
:source = "FV3GFS" ;
:_NCProperties = "version=2,netcdf=4.9.2,hdf5=1.14.0" ;
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gradsusr.org/pipermail/gradsusr/attachments/20240419/a77c543b/attachment.html>
More information about the gradsusr
mailing list