[gradsusr] GrADS query: creating a control file to handle multiple netcdf files when each netcdf file has a uniquely different scale_factor and add_offset

[Jennifer M Adams]

Hi, Ned — 
I’m sure ECMWF has an explanation for why they do it that way, but the different sf/ao values in each file make it impossible to template them together as is — GrADS gets the values from the first file it opens and assumes they are the same for all other files in the templated set. (Querying the metadata every time you open a new data file would be a big performance hit.)  My guess is that those values are not something you can control with a setting in your download. You will have to use GrADS or NCO or CDO or BYO (build your own) to read each file, unpack the data, and then write it out again in a more consistent and user-friendly format. I recommend compressed and chunked netcdf4, which employs a better overall strategy for minimizing file size and I/O speed. I would also encourage you to give your feedback to C3S. 
—Jennifer

> On Jun 7, 2018, at 11:15 AM, Vizy, Edward K <ned at jsg.utexas.edu> wrote:
> 
> Dear gradusers,
>  
> Had a query that I was looking for help with.  I've been downloading upper-level pressure ERA5 data from Copernicus Climate Change Service (C3S) in NETCDF format.  Since the data volume is too large, I've had to break the downloads of the hourly data into monthly increments. Thus, I have files:
>  
> ecmwf.era5.anal.hourly.pl.u.201407.nc   ------ > containing July 2014 1-hourly output
> ecmwf.era5.anal.hourly.pl.u.201408.nc   ------ > containing August 2014 1-hourly output
> ecmwf.era5.anal.hourly.pl.u.201409.nc   ------ > containing September 2014 1-hourly output
>  
> Of course wanting to open this data up in GrADS using a single GrADS control file to evaluate time series, I generated the GrADS control file shown below:
> _____________________________________________
>  
> dset ^./ecmwf.era5.anal.hourly.pl.u.%y4%m2.nc
> options yrev zrev template
> title 
> undef -9.99e+33
> unpack scale_factor add_offset
> dtype netcdf
> xdef 1440 linear 0 0.25
> ydef 721 linear -90 0.25
> zdef 19 levels 1000 975 950 925 900 875 850 825 800 775 750 700 650 600 500 400 300 200 100
> tdef 2208 linear 00Z01JUL2014 60mn
> vars 1
> u=>u  19  t,z,y,x  U component of wind (m/s)
> endvars
> _____________________________________________
>  
> At first glance this control file appeared to have worked.  However, once I started quality controlling by comparing with the directly opened NETCDF files there were differences in the output for the second and third months.  After some digging around, I realized that the C3S service from ERA5 does not implement standard values for the scale_factor and add_offset when processed into NETCDF form.  For example, using ncdump to examine the NETCDF files individually, I find:
>  
> NETCDF File:  ecmwf.era5.anal.hourly.pl.u.201407.nc
>         short u(time, level, latitude, longitude) ;
>                 u:scale_factor = 0.00256569367738034 ;
>                 u:add_offset = 25.6669300437863 ;
>                 u:_FillValue = -32767s ;
>                 u:missing_value = -32767s ;
>                 u:units = "m s**-1" ;
>                 u:long_name = "U component of wind" ;
>                 u:standard_name = "eastward_wind" ;
>  
> NETCDF File: ecmwf.era5.anal.hourly.pl.u.201408.nc
>         short u(time, level, latitude, longitude) ;
>                 u:scale_factor = 0.00344970129404031 ;
>                 u:add_offset = 30.5973199491577 ;
>                 u:_FillValue = -32767s ;
>                 u:missing_value = -32767s ;
>                 u:units = "m s**-1" ;
>                 u:long_name = "U component of wind" ;
>                 u:standard_name = "eastward_wind" ;
>  
> NETCDF File: ecmwf.era5.anal.hourly.pl.u.201409.nc
>         short u(time, level, latitude, longitude) ;
>                 u:scale_factor = 0.002590087296414 ;
>                 u:add_offset = 21.8817387087932 ;
>                 u:_FillValue = -32767s ;
>                 u:missing_value = -32767s ;
>                 u:units = "m s**-1" ;
>                 u:long_name = "U component of wind" ;
>                 u:standard_name = "eastward_wind" ;
>  
> Essentially, when I open my GrADS control file up, it is not reading these values for scale_factor and add_offset from each of the NETCDF files when time is changed into another month, only the first data file at the initial time (i.e., from file ecmwf.era5.anal.hourly.pl.u.201407.nc).
>  
> Does anyone know of a way to account for this change with additional setting/manipulation in the GrADS control file, or will I need to reprocess the data once downloaded from C3S to standardize it to a single convention?  BTW… I’m using GrADS Version 2.1.a2.
>  
> Thanks for your assistance.
>  
> Best regards,
> Ned Vizy
>  
> ______________________________________
> Dr. Edward "Ned" Vizy
> Research Scientist Associate V
> Department of Geological Sciences
> Jackson School of Geosciences
> The University of Texas at Austin
> Austin, TX 78712
> ______________________________________
>  
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://secure-web.cisco.com/1SUK5JvzSHwti5Imi737gTzCw-aVsEXmTD6APwS6hD1hRhzx-ZDdWbAvnn5d9ESpIq9G95WRltgd-CtrhWtcplFtNpEKy1dnBkPn3pLsJqykQylFvhZa_hfx_CG74ivJyYnReVJJ4xzpco_g9vTouOyqpbA9cvNxGkx-Zmb8oHMUYiTDeuIYTp-cRYgT5dAIXO87q89koKJQkZ9Wr-WoGAyWp_nLFFBWVppvyuO_UUEtxgrfkMqSoFipmqKkAdQn7WOgWWO4b2Z__mOrT5ltRsNPD85p-Ccgi2iAK7B1CW3zfMSvPOoMzMA_-0J9PjMrbYeLlc-o85HE4BFcihO7mNew7O-5InqZV27NCy39FTdOcWnI1f4OSm5AcC-Cy8844tdqCzqktty7Frc6J4NuvJ8eqXhtwtjUzjSyipq6zu0Io7XHBRPdNDGATC43dJ6Zr/http%3A%2F%2Fgradsusr.org%2Fmailman%2Flistinfo%2Fgradsusr

--
Jennifer Miletta Adams
Center for Ocean-Land-Atmosphere Studies (COLA)
George Mason University