<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Dear All,</div><div>Jim Potemra brings up a good point which is worth mentioning to the entire forum. You might want to read his message (included below) before reading the rest of my response.</div><div><br></div><div>When reading netcdf files, GrADS doesn't read every time axis value to make sure the increment is the same throughout the file. It assumes a linear time axis and therefore only reads the first two values to get start time and increment. (If you provide all the time axis metadata in a descriptor, GrADS never checks the metadata in the data file at all).</div><div><br></div><div>I encountered this problem with some CMIP5 data -- NCAR was missing some time steps in a model run, so just skipped it in the data and distributed a file that did not have a linear time axis. The missing grids were hidden in time axis values that did not increase linearly -- somewhere in the middle of a file was a delta-T twice as big as all the others. This is very difficult for GrADS to detect. My solution was to break the file into two parts at the point where the missing time occurred and rename the subset files with appropriate date strings. GrADS handled it from there. </div><div><br></div><div>I believe it would be a performance hit to check the time axis values for all times in every file when opening a dataset, especially one being served via opendap. And I am unlikely to take a lot of time writing new code to slow down GrADS to accomodate (what I feel are) sloppy data practices. Gridded model output should be regularly spaced in time. </div><div><br></div><div>It is my opinion that GrADS handles file aggregation elegantly, and all the other methods out there cannot compete, but I am obviously extremely biased. </div><div><br></div><div>--Jennifer</div><div><br></div><div><br></div><div><br></div><div><br></div><div><div>On Jul 10, 2013, at 5:18 PM, James T. Potemra wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
<div bgcolor="#FFFFFF" text="#000000">
Hi Jennifer:<br>
<br>
Your suggestion on using the descriptor file is the better way to
go, and it reminded me of an<br>
issue with accessing aggregated files in GrADS. It seems like GrADS
gets time information from<br>
the start and increment values. If this is true, one could
potentially run into problems with missing<br>
files (times).<br></div></blockquote></div><div><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000"><br>
As example, let's say you have daily files for a week, but one day
(Wednesday) is missing. If you make<br>
an aggregate file from the six valid days using ncrcat (or some
other tool), then load this aggregated<br>
file into GrADS and plot a timeseries, I think you will actually get
a six-day plot from Sunday through <br>
Friday, rather than a seven-day plot from from Sunday through
Saturday with a gap for Wednesday. <br>
Data that appear for "Wednesday" are actually from "Thursday", and
so on.<br>
<br>
This behavior can be especially problematic when reading files via
OPeNDAP, since you can't really be<br>
sure of the contents in the aggregated file unless you check the
time values in the file.<br>
<br>
I could be wrong on this, so I didn't want to post to the listserv,
but feel free to reply there...<br>
<br>
Thanks,<br>
<br>
Jim<br>
<br>
<div class="moz-cite-prefix">On 7/9/13 8:00 AM, Jennifer Adams
wrote:<br>
</div>
<blockquote cite="mid:F4D53013-3452-4B7F-9E68-2D0959C26755@cola.iges.org" type="cite">You can skip the ncrcat step by using a GrADS
descriptor file to aggregate the files together. Put the following
three lines in a text file called air.sig995.xtl in the same
directory with the data files:
<div><br>
</div>
<div>DSET ^air.sig995.%y4.nc</div>
<div>OPTIONS template</div>
<div>TDEF time 24107 linear 1jan1948 1dy</div>
<div><br>
</div>
<div><br>
</div>
<div>Then open this data set with GrADS using the 'xdfopen'
command:</div>
<div><br>
</div>
<div>ga-> xdfopen air.sig995.xtl</div>
<div><br>
</div>
<div>Now all the files appear as a single data set -- there is no
need to concatenate them together. You can begin doing whatever
data analysis you want with the data. If you want to create a
subset of the data in a new file, have a look at the 'sdfwrite'
command. </div>
<div><br>
</div>
<div>--Jennifer</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
<div>
<div>On Jul 9, 2013, at 1:49 PM, James T. Potemra wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
<div bgcolor="#FFFFFF" text="#000000"> Emily:<br>
<br>
The files in the URL below are served via ftp, so this is
not really a GrADS issue. Instead, you can use "wget" to
retrieve all the files, then "ncrcat" to concatenate them
all together. For example,<br>
<br>
wget -r -A.nc <a moz-do-not-send="true" class="moz-txt-link-freetext" href="ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis.dailyavgs/surface/">ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis.dailyavgs/surface/</a><br>
<br>
will retrieve all the files on that page that end with
".nc". Next,<br>
<br>
ncrcat -h air.sig995.*.nc one_big_file.nc<br>
<br>
will cat all the individual files into one big file.
"ncrcat" is part of the netCDF Operator (NCO) toolkit;
more info at <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://nco.sourceforge.net/">http://nco.sourceforge.net/</a><br>
<br>
Jim<br>
<br>
<div class="moz-cite-prefix">On 7/9/13 7:09 AM, Emily
Wilson wrote:<br>
</div>
<blockquote cite="mid:CAGCDMAk+6VDGu0fOYwrt56qp8GoiHTmt2S_yZYqSxXsy5OyT3w@mail.gmail.com" type="cite">
<div dir="ltr">
<div>
<div>Hello All,<br>
<br>
</div>
I am wanting to write a GrADS script in emacs that
downloads a bunch of NetCDF files from a website and
compresses them into one large file. The website is<br>
<a moz-do-not-send="true" href="http://www.esrl.noaa.gov/psd/cgi-bin/db_search/DBListFiles.pl?did=33&tid=38147&vid=668">http://www.esrl.noaa.gov/psd/cgi-bin/db_search/DBListFiles.pl?did=33&tid=38147&vid=668</a>
and I want to download all of the NetCDF files on
this page. What commands can I use to do this
task? If this is not possible would it be possible
to format a script so that from the webpage, each
file is read individually and specific data is
pulled in each file, wrote or saved to a master file
then the same thing is done for the next file and so
on?<br>
</div>
Thanks for the help in advance,<br clear="all">
<div>
<div>
<div>
<div>
<div dir="ltr"><span style="color:rgb(106,168,79)"><b><br>
Emily P. Wilson, Intern</b></span><br>
<span style="color:rgb(61,133,198)">Research
and Conservation Department<br>
Denver Botanic Gardens<br>
1007 York St.<br>
Denver, CO 80206<br>
720-865-3593</span></div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
gradsusr mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:gradsusr@gradsusr.org">gradsusr@gradsusr.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://gradsusr.org/mailman/listinfo/gradsusr">http://gradsusr.org/mailman/listinfo/gradsusr</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>
gradsusr mailing list<br>
<a moz-do-not-send="true" href="mailto:gradsusr@gradsusr.org">gradsusr@gradsusr.org</a><br>
<a class="moz-txt-link-freetext" href="http://gradsusr.org/mailman/listinfo/gradsusr">http://gradsusr.org/mailman/listinfo/gradsusr</a><br>
</blockquote>
</div>
<br>
<div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; -webkit-text-decorations-in-effect: none; text-indent: 0px; -webkit-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; -webkit-text-decorations-in-effect: none; text-indent: 0px; -webkit-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; -webkit-text-decorations-in-effect: none; text-indent: 0px; -webkit-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; ">
<div>--</div>
<div>Jennifer M. Adams</div>
<div>IGES/COLA</div>
<div>4041 Powder Mill Road, Suite 302</div>
<div>Calverton, MD 20705</div>
<div><a moz-do-not-send="true" href="mailto:jma@cola.iges.org">jma@cola.iges.org</a></div>
<div><br class="khtml-block-placeholder">
</div>
<br class="Apple-interchange-newline">
</span></span></span>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
gradsusr mailing list
<a class="moz-txt-link-abbreviated" href="mailto:gradsusr@gradsusr.org">gradsusr@gradsusr.org</a>
<a class="moz-txt-link-freetext" href="http://gradsusr.org/mailman/listinfo/gradsusr">http://gradsusr.org/mailman/listinfo/gradsusr</a>
</pre>
</blockquote>
<br>
</div>
</blockquote></div><br><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><div>--</div><div>Jennifer M. Adams</div><div>IGES/COLA</div><div>4041 Powder Mill Road, Suite 302</div><div>Calverton, MD 20705</div><div><a href="mailto:jma@cola.iges.org">jma@cola.iges.org</a></div><div><br class="khtml-block-placeholder"></div><br class="Apple-interchange-newline"></span></span></span>
</div>
<br></body></html>