undef with unpack in dtype netcdf
Jennifer Adams
jma at COLA.IGES.ORG
Thu Mar 10 15:11:51 EST 2005
Hi, Diane --
First, a clarification of terminology:
> undef -1.e+34f missing_value
In this undef entry, you've provided a "file-wide" undef value of
-1e+34 and an "undef attribute name". When you provide the attribute
name, this tells GrADS that variables in the file may have different
undef values and it needs to retrieve those along with the data when it
does the I/O. Then, each variable is tested against its own missing
value. If data are determined to be missing, then the missing value is
changed to the file-wide undef, so that after I/O all variables appear
to have the same undef value. This makes it possible to compare
variables with different undefs without confusing GrADS. If no undef
attribute name is given, the file-wide undef is used for each variable,
and the missing value attribute is not retrieved during I/O.
I believe your issue has to do with whether the missing value attribute
corresponds to a packed or unpacked data value. The way it has been
implemented in the dtype netcdf interface is the latter: GrADS unpacks
first, then does a "fuzzy" test to see if data values are within
epsilon of the missing value -- if so, then they are assigned the
file-wide undef.
Since the missing value in your data set corresponds to a packed data
value, then I suggest trying this as a solution: put the unpacked value
of the undef as your file-wide undef without the undef attribute name:
undef -10.0017
I have no idea what the NetCDF convention is for missing_value and
_FillValue -- whether these are meant to correspond to packed or
unpacked values. Anyone else know the answer to this?
Jennifer
On Mar 10, 2005, at 2:17 PM, Diane Stokes wrote:
> Hi,
>
> I'm running GrADS 1.9b3 on AIX:
>
> Config: v1.9b3 32-bit big-endian readline sdf/xdf netcdf lats printim
>
>
> I'm trying to help someone here read a netcdf file, using the new
> "dtype
> netcdf" entry in the data descriptor file. This data includes a packed
> variable, temp.
>
> The data descriptor file and output from ncdump -c are attached.
>
> When I include the "upack" line in the descriptor file, temp displays
> correctly *except* for the fact that the missing points are not masked
> out. GrADS sees these values as -10.0017.
>
> If I remove the "undef" line, the missing values are seen as -32768 and
> are masked out correctly, but, of course, the wrong values are
> displayed
> for the temperature (because they haven't been unpacked).
>
> Is the problem that GrADS is expecting -32768 but seeing -10.0017? Or
> might it be a roundoff issue, eg, GrADS is expecting -10, but seeing
> -10.0017? Either way, any suggestions?
>
> We could use sdfopen to read one file, but we are templating a time
> series of files. We have to use "dtype netcdf" instead to avoid core
> dumps.
>
> Maskout will work in the meantime, but that is not an ideal solution.
>
>
>
> *ALSO*, I can display the non-packed packed variables version using
> gradsnc. I get, eg:
>
> ga-> d eta_t
> Warning: Scale Factor Attribute Value Not Retrieved ... setting to 1.0
> Warning: Offset Attribute Value Not Retrieved ... setting to 0.0
> Contouring: -1.8 to 0.6 interval 0.3
>
> But with gradshdf, I get:
>
> ga-> d eta_t
> ncattget: ncid 4: Attribute not found
>
> and GrADS exits. (No core dump).
>
> I'll start a new thread with this one if you like, but then you'd get
> another copy of the same ncdump -c output.
>
> Thanks in advance for any help.
> Diane
>
> --
> Diane Stokes
> Environmental Modeling Center
> National Weather Service/NOAA
>
> DSET /ptmp/wx20wn/SIMU_OUT/TS/%y4%m201.ocean_month.nc
> dtype netcdf
> options template
> undef -1.e+34f missing_value
> unpack scale_factor add_offset
> title operational godas time_mean files
> *
>
> XDEF 360 LINEAR 0.5 1.0
> YDEF 200 LEVELS -74.5 -73.5 -72.5 -71.5 -70.5 -69.5 -68.5 -67.5
> -66.5
> -65.5 -64.5 -63.5 -62.5 -61.5 -60.5 -59.5 -58.5 -57.5
> -56.5
> -55.5 -54.5 -53.5 -52.5 -51.5 -50.5 -49.5 -48.5 -47.5
> -46.5
> -45.5 -44.5 -43.5 -42.5 -41.5 -40.5 -39.5 -38.5 -37.5
> -36.5
> -35.5 -34.5 -33.5 -32.5 -31.5 -30.5 -29.50046 -28.50319
> -27.51137 -26.52855 -25.5582 -24.60365 -23.66805 -22.75435
> -21.86525 -21.00319 -20.17032 -19.36844 -18.59905 -17.86328
> -17.16186 -16.4952 -15.86328 -15.26573 -14.70179 -14.17033
> -13.66988 -13.19861 -12.75437 -12.33475 -11.93702 -11.55824
> -11.19526 -10.84474 -10.50324 -10.16717 -9.833385 -9.500055
> -9.166725 -8.833395 -8.500065 -8.166735 -7.833405 -7.500075
> -7.166745 -6.833415 -6.500085 -6.166755 -5.833425 -5.500095
> -5.166765 -4.833435 -4.500105 -4.166775 -3.833445 -3.500115
> -3.166785 -2.833455 -2.500125 -2.166795 -1.833465 -1.500135
> -1.166805 -0.833475 -0.500145 -0.166815 0.166515 0.499845
> 0.833175
> 1.166505 1.499835 1.833165 2.166495 2.499825 2.833155
> 3.166485
> 3.499815 3.833145 4.166475 4.499805 4.833135 5.166465
> 5.499795
> 5.833125 6.166455 6.499785 6.833115 7.166445 7.499775
> 7.833105
> 8.166435 8.499765 8.833095 9.166425 9.499755 9.833085
> 10.16687
> 10.50294 10.84444 11.19496 11.55794 11.93672 12.33445
> 12.75407
> 13.19831 13.66958 14.17003 14.70149 15.26543 15.86298 16.4949
> 17.16156 17.86298 18.59875 19.36814 20.17002 21.00289
> 21.86495
> 22.75405 23.66775 24.60335 25.5579 26.52825 27.51107 28.50289
> 29.50016 30.4997 31.4997 32.4997 33.4997 34.4997 35.4997
> 36.4997
> 37.4997 38.4997 39.4997 40.4997 41.4997 42.4997 43.4997
> 44.4997
> 45.4997 46.4997 47.4997 48.4997 49.4997 50.4997 51.4997
> 52.4997
> 53.4997 54.4997 55.4997 56.4997 57.4997 58.4997 59.4997
> 60.4997
> 61.4997 62.4997 63.4997 64.4997
>
> ZDEF 40 levels 5 15 25 35 45 55 65 75 85 95 105 115 125
> 135 145 155
> 165 175 185 195 205 215 225 238.4779 262.2945 303.0287
> 366.7978 459.091 584.6193 747.187 949.5881 1193.53 1479.588
> 1807.187 2174.619 2579.091 3016.798 3483.029 3972.294
> 4478.478
>
> TDEF 14 LINEAR 16jan0001 1mo
>
> vars 6
> eta_t 0 t,y,x ssh
> temp 40 t,z,y,x temp
> salt 40 t,z,y,x salt
> mld 0 t,y,x mld
> sfc_hflux 0 t,y,x heat flux
> pme 0 t,y,x precip-evap
> endvars
>
> netcdf 00010101.ocean_month {
> dimensions:
> xt_ocean = 360 ;
> yt_ocean = 200 ;
> Time = UNLIMITED ; // (1 currently)
> nv = 2 ;
> zt_ocean = 40 ;
> zt_edges_ocean = 41 ;
> zw_ocean = 40 ;
> zw_edges_ocean = 41 ;
> variables:
> float xt_ocean(xt_ocean) ;
> xt_ocean:long_name = "tcell longitude" ;
> xt_ocean:units = "degrees_E" ;
> xt_ocean:cartesian_axis = "X" ;
> float yt_ocean(yt_ocean) ;
> yt_ocean:long_name = "tcell latitude" ;
> yt_ocean:units = "degrees_N" ;
> yt_ocean:cartesian_axis = "Y" ;
> double Time(Time) ;
> Time:long_name = "Time" ;
> Time:units = "days since 0001-01-01 00:00:00" ;
> Time:cartesian_axis = "T" ;
> Time:calendar_type = "JULIAN" ;
> Time:calendar = "JULIAN" ;
> Time:bounds = "Time_bounds" ;
> float nv(nv) ;
> nv:long_name = "vertex number" ;
> nv:units = "none" ;
> nv:cartesian_axis = "N" ;
> float zt_ocean(zt_ocean) ;
> zt_ocean:long_name = "tcell depth" ;
> zt_ocean:units = "meters" ;
> zt_ocean:cartesian_axis = "Z" ;
> zt_ocean:positive = "down" ;
> zt_ocean:edges = "zt_edges_ocean" ;
> float zt_edges_ocean(zt_edges_ocean) ;
> zt_edges_ocean:long_name = "tcell depth edges" ;
> zt_edges_ocean:units = "meters" ;
> zt_edges_ocean:cartesian_axis = "Z" ;
> zt_edges_ocean:positive = "down" ;
> float zw_ocean(zw_ocean) ;
> zw_ocean:long_name = "ucell depth" ;
> zw_ocean:units = "meters" ;
> zw_ocean:cartesian_axis = "Z" ;
> zw_ocean:positive = "down" ;
> zw_ocean:edges = "zw_edges_ocean" ;
> float zw_edges_ocean(zw_edges_ocean) ;
> zw_edges_ocean:long_name = "ucell depth edges" ;
> zw_edges_ocean:units = "meters" ;
> zw_edges_ocean:cartesian_axis = "Z" ;
> zw_edges_ocean:positive = "down" ;
> float eta_t(Time, yt_ocean, xt_ocean) ;
> eta_t:long_name = "surface height on T cells" ;
> eta_t:units = "meter" ;
> eta_t:valid_range = -10.f, 10.f ;
> eta_t:missing_value = -10.f ;
> eta_t:cell_methods = "time: mean" ;
> eta_t:time_avg_info =
> "average_T1,average_T2,average_DT" ;
> short temp(Time, zt_ocean, yt_ocean, xt_ocean) ;
> temp:long_name = "Potential temperature" ;
> temp:units = "deg_C" ;
> temp:valid_range = -32767s, 32767s ;
> temp:missing_value = -32768s ;
> temp:packing = 4 ;
> temp:scale_factor = 0.001678518f ;
> temp:add_offset = 45.f ;
> temp:cell_methods = "time: mean" ;
> temp:time_avg_info =
> "average_T1,average_T2,average_DT" ;
> short salt(Time, zt_ocean, yt_ocean, xt_ocean) ;
> salt:long_name = "Salinity" ;
> salt:units = "psu" ;
> salt:valid_range = -32767s, 32767s ;
> salt:missing_value = -32768s ;
> salt:packing = 4 ;
> salt:scale_factor = 0.001678518f ;
> salt:add_offset = 45.f ;
> salt:cell_methods = "time: mean" ;
> salt:time_avg_info =
> "average_T1,average_T2,average_DT" ;
> float wt(Time, zw_ocean, yt_ocean, xt_ocean) ;
> wt:long_name = "vertical velocity T-points" ;
> wt:units = "m/sec" ;
> wt:valid_range = -10.f, 10.f ;
> wt:missing_value = -10.f ;
> wt:cell_methods = "time: mean" ;
> wt:time_avg_info = "average_T1,average_T2,average_DT" ;
> float mld(Time, yt_ocean, xt_ocean) ;
> mld:long_name = "mixed layer depth" ;
> mld:units = "m" ;
> mld:valid_range = 0.f, 1000000.f ;
> mld:missing_value = 0.f ;
> mld:cell_methods = "time: mean" ;
> mld:time_avg_info = "average_T1,average_T2,average_DT"
> ;
> float sfc_hflux(Time, yt_ocean, xt_ocean) ;
> sfc_hflux:long_name = "surface heat flux" ;
> sfc_hflux:units = "Watts/m^2" ;
> sfc_hflux:valid_range = -10000.f, 10000.f ;
> sfc_hflux:missing_value = -1.e+10f ;
> sfc_hflux:cell_methods = "time: mean" ;
> sfc_hflux:time_avg_info =
> "average_T1,average_T2,average_DT" ;
> float pme(Time, yt_ocean, xt_ocean) ;
> pme:long_name = "precip-evap (liquid, frozen,
> evaporation)" ;
> pme:units = "m/sec" ;
> pme:valid_range = -10.f, 10.f ;
> pme:missing_value = -1.e+10f ;
> pme:cell_methods = "time: mean" ;
> pme:time_avg_info = "average_T1,average_T2,average_DT"
> ;
> double average_T1(Time) ;
> average_T1:long_name = "Start time for average period"
> ;
> average_T1:units = "days since 0001-01-01 00:00:00" ;
> double average_T2(Time) ;
> average_T2:long_name = "End time for average period" ;
> average_T2:units = "days since 0001-01-01 00:00:00" ;
> double average_DT(Time) ;
> average_DT:long_name = "Length of average period" ;
> average_DT:units = "days" ;
> double Time_bounds(Time, nv) ;
> Time_bounds:long_name = "Time axis boundaries" ;
> Time_bounds:units = "days" ;
>
> // global attributes:
> :filename = "ocean_month.nc" ;
> :title = "indian ocean exp" ;
> data:
>
> xt_ocean = 0.5, 1.5, 2.5, 3.5, 4.5, 5.5, 6.5, 7.5, 8.5, 9.5, 10.5,
> 11.5,
> 12.5, 13.5, 14.5, 15.5, 16.5, 17.5, 18.5, 19.5, 20.5, 21.5, 22.5,
> 23.5,
> 24.5, 25.5, 26.5, 27.5, 28.5, 29.5, 30.5, 31.5, 32.5, 33.5, 34.5,
> 35.5,
> 36.5, 37.5, 38.5, 39.5, 40.5, 41.5, 42.5, 43.5, 44.5, 45.5, 46.5,
> 47.5,
> 48.5, 49.5, 50.5, 51.5, 52.5, 53.5, 54.5, 55.5, 56.5, 57.5, 58.5,
> 59.5,
> 60.5, 61.5, 62.5, 63.5, 64.5, 65.5, 66.5, 67.5, 68.5, 69.5, 70.5,
> 71.5,
> 72.5, 73.5, 74.5, 75.5, 76.5, 77.5, 78.5, 79.5, 80.5, 81.5, 82.5,
> 83.5,
> 84.5, 85.5, 86.5, 87.5, 88.5, 89.5, 90.5, 91.5, 92.5, 93.5, 94.5,
> 95.5,
> 96.5, 97.5, 98.5, 99.5, 100.5, 101.5, 102.5, 103.5, 104.5, 105.5,
> 106.5,
> 107.5, 108.5, 109.5, 110.5, 111.5, 112.5, 113.5, 114.5, 115.5,
> 116.5,
> 117.5, 118.5, 119.5, 120.5, 121.5, 122.5, 123.5, 124.5, 125.5,
> 126.5,
> 127.5, 128.5, 129.5, 130.5, 131.5, 132.5, 133.5, 134.5, 135.5,
> 136.5,
> 137.5, 138.5, 139.5, 140.5, 141.5, 142.5, 143.5, 144.5, 145.5,
> 146.5,
> 147.5, 148.5, 149.5, 150.5, 151.5, 152.5, 153.5, 154.5, 155.5,
> 156.5,
> 157.5, 158.5, 159.5, 160.5, 161.5, 162.5, 163.5, 164.5, 165.5,
> 166.5,
> 167.5, 168.5, 169.5, 170.5, 171.5, 172.5, 173.5, 174.5, 175.5,
> 176.5,
> 177.5, 178.5, 179.5, 180.5, 181.5, 182.5, 183.5, 184.5, 185.5,
> 186.5,
> 187.5, 188.5, 189.5, 190.5, 191.5, 192.5, 193.5, 194.5, 195.5,
> 196.5,
> 197.5, 198.5, 199.5, 200.5, 201.5, 202.5, 203.5, 204.5, 205.5,
> 206.5,
> 207.5, 208.5, 209.5, 210.5, 211.5, 212.5, 213.5, 214.5, 215.5,
> 216.5,
> 217.5, 218.5, 219.5, 220.5, 221.5, 222.5, 223.5, 224.5, 225.5,
> 226.5,
> 227.5, 228.5, 229.5, 230.5, 231.5, 232.5, 233.5, 234.5, 235.5,
> 236.5,
> 237.5, 238.5, 239.5, 240.5, 241.5, 242.5, 243.5, 244.5, 245.5,
> 246.5,
> 247.5, 248.5, 249.5, 250.5, 251.5, 252.5, 253.5, 254.5, 255.5,
> 256.5,
> 257.5, 258.5, 259.5, 260.5, 261.5, 262.5, 263.5, 264.5, 265.5,
> 266.5,
> 267.5, 268.5, 269.5, 270.5, 271.5, 272.5, 273.5, 274.5, 275.5,
> 276.5,
> 277.5, 278.5, 279.5, 280.5, 281.5, 282.5, 283.5, 284.5, 285.5,
> 286.5,
> 287.5, 288.5, 289.5, 290.5, 291.5, 292.5, 293.5, 294.5, 295.5,
> 296.5,
> 297.5, 298.5, 299.5, 300.5, 301.5, 302.5, 303.5, 304.5, 305.5,
> 306.5,
> 307.5, 308.5, 309.5, 310.5, 311.5, 312.5, 313.5, 314.5, 315.5,
> 316.5,
> 317.5, 318.5, 319.5, 320.5, 321.5, 322.5, 323.5, 324.5, 325.5,
> 326.5,
> 327.5, 328.5, 329.5, 330.5, 331.5, 332.5, 333.5, 334.5, 335.5,
> 336.5,
> 337.5, 338.5, 339.5, 340.5, 341.5, 342.5, 343.5, 344.5, 345.5,
> 346.5,
> 347.5, 348.5, 349.5, 350.5, 351.5, 352.5, 353.5, 354.5, 355.5,
> 356.5,
> 357.5, 358.5, 359.5 ;
>
> yt_ocean = -74.5, -73.5, -72.5, -71.5, -70.5, -69.5, -68.5, -67.5,
> -66.5,
> -65.5, -64.5, -63.5, -62.5, -61.5, -60.5, -59.5, -58.5, -57.5,
> -56.5,
> -55.5, -54.5, -53.5, -52.5, -51.5, -50.5, -49.5, -48.5, -47.5,
> -46.5,
> -45.5, -44.5, -43.5, -42.5, -41.5, -40.5, -39.5, -38.5, -37.5,
> -36.5,
> -35.5, -34.5, -33.5, -32.5, -31.5, -30.5, -29.50046, -28.50319,
> -27.51137, -26.52855, -25.5582, -24.60365, -23.66805, -22.75435,
> -21.86525, -21.00319, -20.17032, -19.36844, -18.59905, -17.86328,
> -17.16186, -16.4952, -15.86328, -15.26573, -14.70179, -14.17033,
> -13.66988, -13.19861, -12.75437, -12.33475, -11.93702, -11.55824,
> -11.19526, -10.84474, -10.50324, -10.16717, -9.833385, -9.500055,
> -9.166725, -8.833395, -8.500065, -8.166735, -7.833405, -7.500075,
> -7.166745, -6.833415, -6.500085, -6.166755, -5.833425, -5.500095,
> -5.166765, -4.833435, -4.500105, -4.166775, -3.833445, -3.500115,
> -3.166785, -2.833455, -2.500125, -2.166795, -1.833465, -1.500135,
> -1.166805, -0.833475, -0.500145, -0.166815, 0.166515, 0.499845,
> 0.833175,
> 1.166505, 1.499835, 1.833165, 2.166495, 2.499825, 2.833155,
> 3.166485,
> 3.499815, 3.833145, 4.166475, 4.499805, 4.833135, 5.166465,
> 5.499795,
> 5.833125, 6.166455, 6.499785, 6.833115, 7.166445, 7.499775,
> 7.833105,
> 8.166435, 8.499765, 8.833095, 9.166425, 9.499755, 9.833085,
> 10.16687,
> 10.50294, 10.84444, 11.19496, 11.55794, 11.93672, 12.33445,
> 12.75407,
> 13.19831, 13.66958, 14.17003, 14.70149, 15.26543, 15.86298,
> 16.4949,
> 17.16156, 17.86298, 18.59875, 19.36814, 20.17002, 21.00289,
> 21.86495,
> 22.75405, 23.66775, 24.60335, 25.5579, 26.52825, 27.51107,
> 28.50289,
> 29.50016, 30.4997, 31.4997, 32.4997, 33.4997, 34.4997, 35.4997,
> 36.4997,
> 37.4997, 38.4997, 39.4997, 40.4997, 41.4997, 42.4997, 43.4997,
> 44.4997,
> 45.4997, 46.4997, 47.4997, 48.4997, 49.4997, 50.4997, 51.4997,
> 52.4997,
> 53.4997, 54.4997, 55.4997, 56.4997, 57.4997, 58.4997, 59.4997,
> 60.4997,
> 61.4997, 62.4997, 63.4997, 64.4997 ;
>
> Time = 15.5 ;
>
> nv = 1, 2 ;
>
> zt_ocean = 5, 15, 25, 35, 45, 55, 65, 75, 85, 95, 105, 115, 125, 135,
> 145,
> 155, 165, 175, 185, 195, 205, 215, 225, 238.4779, 262.2945,
> 303.0287,
> 366.7978, 459.091, 584.6193, 747.187, 949.5881, 1193.53, 1479.588,
> 1807.187, 2174.619, 2579.091, 3016.798, 3483.029, 3972.294,
> 4478.478 ;
>
> zt_edges_ocean = -0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110,
> 120, 130,
> 140, 150, 160, 170, 180, 190, 200, 210, 220, 231.7389, 250.3862,
> 282.6616, 334.9132, 412.9444, 521.8552, 665.9031, 848.3875,
> 1071.559,
> 1336.559, 1643.388, 1990.903, 2376.855, 2797.944, 3249.913,
> 3727.662,
> 4225.386, 4736.739 ;
>
> zw_ocean = 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130,
> 140, 150,
> 160, 170, 180, 190, 200, 210, 220, 231.7389, 250.3862, 282.6616,
> 334.9132, 412.9444, 521.8552, 665.9031, 848.3875, 1071.559,
> 1336.559,
> 1643.388, 1990.903, 2376.855, 2797.944, 3249.913, 3727.662,
> 4225.386,
> 4736.739 ;
>
> zw_edges_ocean = 5, 15, 25, 35, 45, 55, 65, 75, 85, 95, 105, 115,
> 125, 135,
> 145, 155, 165, 175, 185, 195, 205, 215, 225, 238.4779, 262.2945,
> 303.0287, 366.7978, 459.091, 584.6193, 747.187, 949.5881, 1193.53,
> 1479.588, 1807.187, 2174.619, 2579.091, 3016.798, 3483.029,
> 3972.294,
> 4478.478, 4736.739 ;
> }
More information about the gradsusr
mailing list