[gradsusr] template option suddenly slow for nc files
Eric A. Aligo
eric.aligo at noaa.gov
Fri Apr 19 10:43:01 EDT 2024
Thanks for the response! The first list below is from a file that gives
me a hard time with templating in GrADS. The second list below works
well in GrADS, and I had to set ideflate=0 (no compression). Is there
something that I can do to use compression and still be able to use the
template option efficiently or am I stuck with no compression?
With compression:
float dzdt(time, pfull, grid_yt, grid_xt) ;
dzdt:_QuantizeBitRoundNumberOfSignificantBits = 18 ;
dzdt:_FillValue = 9.99e+20f ;
dzdt:cell_methods = "time: point" ;
dzdt:long_name = "vertical wind" ;
dzdt:missing_value = 9.99e+20f ;
dzdt:output_file = "dyn" ;
dzdt:units = "m/sec" ;
dzdt:_Storage = "chunked" ;
dzdt:_ChunkSizes = 1, 17, 263, 626 ;
dzdt:_DeflateLevel = 1 ;
dzdt:_Shuffle = "true" ;
dzdt:_Endianness = "little" ;
dzdt:_NoFill = "true" ;
Without compression:
float dzdt(time, pfull, grid_yt, grid_xt) ;
dzdt:_FillValue = 9.99e+20f ;
dzdt:cell_methods = "time: point" ;
dzdt:long_name = "vertical wind" ;
dzdt:missing_value = 9.99e+20f ;
dzdt:output_file = "dyn" ;
dzdt:units = "m/sec" ;
dzdt:_Storage = "contiguous" ;
dzdt:_Endianness = "little" ;
dzdt:_NoFill = "true" ;
On 4/18/2024 9:05 PM, Jennifer M Adams wrote:
> Eric, can you try running ncdump with the -s option to show the
> chunking parameters? It may be that the chunk shapes and sizes are
> different and not entirely compatible with the GrADS I/O pattern.
> --Jennifer
> ------------------------------------------------------------------------
> *From:* gradsusr <gradsusr-bounces at gradsusr.org> on behalf of Eric A.
> Aligo <eric.aligo at noaa.gov>
> *Sent:* Thursday, April 18, 2024 8:37 AM
> *To:* gradsusr at gradsusr.org <gradsusr at gradsusr.org>
> *Subject:* [gradsusr] template option suddenly slow for nc files
>
> Plotting one level and one time from a 3D field takes 3 seconds from a
> netcdf file using one version of FV3 code, but now it takes 2.5
> minutes with a new version of FV3 code. The differences I see in the
> header of the netcdf file are: the version of netcdf and hdf5,
> ncnsto, and grid_id. The fill/undefined values are also different. My
> control file is below. Is this a known issue with GraDS and
> netcdf/hdf5? A bug?
>
>
> dset ^dynf%h3.nc
> dtype netcdf
> options template
> undef 9.99e+20
> title FV3 history
> xdef 2501 linear -135 0.03
> ydef 1051 linear 22.0 0.03
> tdef 3 linear 00Z20JUN2019 1hr
> zdef 65 linear 1 1
> VARS 1
> dzdt 65 t,z,y,x Vertical velocity
> endvars
>
>
> ncdump -h:
>
> :_NCProperties = "version=2,netcdf=4.9.2,hdf5=1.14.0" ; *<-----takes
> 2.5 minutes to plot field*
>
> :_NCProperties = "version=2,netcdf=4.7.4,hdf5=1.10.6," ; *<-----takes
> 3 seconds to plot field*
>
> Here's some of the header of the netcdf file that behaves normally:
>
> netcdf dynf002 {
> dimensions:
> grid_xt = 2501 ;
> grid_yt = 1051 ;
> nchars = 20 ;
> pfull = 65 ;
> phalf = 66 ;
> time = 1 ;
>
> float dzdt(time, pfull, grid_yt, grid_xt) ;
> dzdt:_FillValue = -1.e+10f ;
> dzdt:cell_methods = "time: point" ;
> dzdt:long_name = "vertical wind" ;
> dzdt:missing_value = -1.e+10f ;
> dzdt:output_file = "dyn" ;
> dzdt:units = "m/sec" ;
> // global attributes:
> :_NCProperties = "version=2,netcdf=4.7.4,hdf5=1.10.6," ;
> :ak = 212.637f, 452.613f, 723.285f, 1028.378f,
> 1372.014f, 1758.738f, 2193.544f, 2681.893f, 3229.725f, 3843.468f,
> 4530.025f, 5296.374f, 6132.608f, 7004.872f, 7877.419f, 8714.452f,
> 9480.727f, 10155.56f, 10734.38f, 11213.79f, 11591.01f, 11863.99f,
> 12031.62f, 12093.93f, 12052.24f, 11909.32f, 11669.5f, 11338.72f,
> 10924.5f, 10435.84f, 9883.087f, 9277.633f, 8631.631f, 7957.623f,
> 7268.15f, 6575.364f, 5890.665f, 5224.384f, 4585.535f, 3981.638f,
> 3418.626f, 2900.827f, 2431.016f, 2010.516f, 1639.35f, 1316.413f,
> 1039.659f, 806.289f, 612.932f, 455.809f, 330.883f, 233.98f, 160.901f,
> 107.5f, 69.754f, 43.805f, 26.13f, 14.45f, 7.211f, 3.103f, 1.058f,
> 0.234f, 0.017f, 0.f, 0.f, 0.f ;
> :bk = 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f,
> 0.f, 0.f, 3.79e-06f, 0.00018806f, 0.00097247f, 0.00279793f,
> 0.00610706f, 0.01133629f, 0.01877378f, 0.02854147f, 0.04074101f,
> 0.05545674f, 0.07275035f, 0.09265514f, 0.1151704f, 0.1402563f,
> 0.1678291f, 0.1977584f, 0.2298655f, 0.2639235f, 0.2996609f,
> 0.3367662f, 0.3748959f, 0.4136837f, 0.4527515f, 0.4917214f, 0.530227f,
> 0.5679249f, 0.6045035f, 0.6396912f, 0.6732613f, 0.7050351f,
> 0.7348821f, 0.7627185f, 0.7885046f, 0.8122396f, 0.833957f, 0.8537182f,
> 0.8716077f, 0.8877271f, 0.9021903f, 0.9151191f, 0.926639f, 0.9368768f,
> 0.9459575f, 0.9540022f, 0.9611273f, 0.9674414f, 0.9730366f,
> 0.9779925f, 0.9823816f, 0.9862693f, 0.9897139f, 0.992767f, 0.9954739f,
> 0.9978736f, 1.f ;
> :dlat = 0.03f ;
> :dlon = 0.03f ;
> :grid = "latlon" ;
> :*grid_id = 1 ;*
> :hydrostatic = "non-hydrostatic" ;
> :lat1 = 22.f ;
> :lat2 = 53.5f ;
> :lon1 = -135.f ;
> :lon2 = -60.f ;
> :*ncnsto = 13 ;*
> :source = "FV3GFS" ;
>
> Here's the header of the netcdf file that doesn't behave normally:
>
> netcdf dynf002 {
> dimensions:
> grid_xt = 2501 ;
> grid_yt = 1051 ;
> nchars = 20 ;
> pfull = 65 ;
> phalf = 66 ;
> time = 1 ;
> float dzdt(time, pfull, grid_yt, grid_xt) ;
> dzdt:_FillValue = *9.99e+20*f ;
> dzdt:cell_methods = "time: point" ;
> dzdt:long_name = "vertical wind" ;
> dzdt:missing_value = *9.99e+20*f ;
> dzdt:output_file = "dyn" ;
> dzdt:units = "m/sec" ;
>
> // global attributes:
> :ak = 212.637f, 452.613f, 723.285f, 1028.378f,
> 1372.014f, 1758.738f, 2193.544f, 2681.893f, 3229.725f, 3843.468f,
> 4530.025f, 5296.374f, 6132.608f, 7004.872f, 7877.419f, 8714.452f,
> 9480.727f, 10155.56f, 10734.38f, 11213.79f, 11591.01f, 11863.99f,
> 12031.62f, 12093.93f, 12052.24f, 11909.32f, 11669.5f, 11338.72f,
> 10924.5f, 10435.84f, 9883.087f, 9277.633f, 8631.631f, 7957.623f,
> 7268.15f, 6575.364f, 5890.665f, 5224.384f, 4585.535f, 3981.638f,
> 3418.626f, 2900.827f, 2431.016f, 2010.516f, 1639.35f, 1316.413f,
> 1039.659f, 806.289f, 612.932f, 455.809f, 330.883f, 233.98f, 160.901f,
> 107.5f, 69.754f, 43.805f, 26.13f, 14.45f, 7.211f, 3.103f, 1.058f,
> 0.234f, 0.017f, 0.f, 0.f, 0.f ;
> :bk = 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f,
> 0.f, 0.f, 3.79e-06f, 0.00018806f, 0.00097247f, 0.00279793f,
> 0.00610706f, 0.01133629f, 0.01877378f, 0.02854147f, 0.04074101f,
> 0.05545674f, 0.07275035f, 0.09265514f, 0.1151704f, 0.1402563f,
> 0.1678291f, 0.1977584f, 0.2298655f, 0.2639235f, 0.2996609f,
> 0.3367662f, 0.3748959f, 0.4136837f, 0.4527515f, 0.4917214f, 0.530227f,
> 0.5679249f, 0.6045035f, 0.6396912f, 0.6732613f, 0.7050351f,
> 0.7348821f, 0.7627185f, 0.7885046f, 0.8122396f, 0.833957f, 0.8537182f,
> 0.8716077f, 0.8877271f, 0.9021903f, 0.9151191f, 0.926639f, 0.9368768f,
> 0.9459575f, 0.9540022f, 0.9611273f, 0.9674414f, 0.9730366f,
> 0.9779925f, 0.9823816f, 0.9862693f, 0.9897139f, 0.992767f, 0.9954739f,
> 0.9978736f, 1.f ;
> :dlat = 0.03f ;
> :dlon = 0.03f ;
> :grid = "latlon" ;
> :*grid_id = 1L ;*
> :hydrostatic = "non-hydrostatic" ;
> :lat1 = 22.f ;
> :lat2 = 53.5f ;
> :lon1 = -135.f ;
> :lon2 = -60.f ;
> :*ncnsto = 16* ;
> :source = "FV3GFS" ;
> :_NCProperties = "version=2,netcdf=4.9.2,hdf5=1.14.0" ;
>
> Eric
>
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gradsusr.org/pipermail/gradsusr/attachments/20240419/0c1cd0e7/attachment-0001.html>
More information about the gradsusr
mailing list