juggling with datafiles

[Huddleston, John]

Joe/Chris

You can use the Climate Data Operators (CDO) http://www.mpimet.mpg.de/fileadmin/software/cdo/ to concatenate and manipulate climate data files and use the netCDF Operators (NCO) http://nco.sourceforge.net/  to concatenate and otherwise manipulate netCDF data files. See http://vista.cira.colostate.edu/nco/readme.htm for static Linux, Cygwin, and native Windows executables.


John Huddleston, PhD

-----Original Message-----
From: GRADSUSR at LIST.CINECA.IT [mailto:GRADSUSR at LIST.CINECA.IT] On Behalf Of Joe Pietrowicz
Sent: Thursday, January 07, 2010 10:56 AM
To: GRADSUSR at LIST.CINECA.IT
Subject: Re: juggling with datafiles

Chris,

I've found that if you are working with grib or grib2 files that are all of the same type you can "cat" them into one file, then process them to be used in GrADS. I'm not sure if you can do this with netcdf files, but I don't think so. Or you can use the template feature when making the *.ctl file. This will allow all times to be available with only one file that is open. Then the loop that Charles Seman suggests should work without the open *.ctl.n.

Regards
Joe

----- Original Message -----
From: "Charles Seman" <Charles.Seman at NOAA.GOV>
To: GRADSUSR at LIST.CINECA.IT
Sent: Thursday, January 7, 2010 12:20:14 PM GMT -05:00 US/Canada Eastern
Subject: Re: juggling with datafiles

Chris,

If all of the defined variables are for the same physical domain, 
perhaps you could try swapping the variables in the time loop? Maybe 
something like this (warning: not tested, please check for errors):

n=1
open ctl.n * or whatever the name should be...
define var1 = var.1
close 1
n=n+1
while (n<=48)
open ctl.n
define var2 = var.1
define var2m1 = var2-var1
...
var1 = var2 * update var1 for next time
close 1
n=n+1
endwhile

I hope this helps,
Chuck

Chris Hayes wrote:
>
> Dear group,
>
> I loop through 48 subsequent files and create maps from them.
>
> Several of the weatherdata are cumulative, adding all precipitation 
> from the runtime to the hour of the prediction.
>
> I would like to show maps of precipitation in the time period. E.g. of 
> time #23 by subtracting val.#22 from val.#23.
>
> I need 2 files to be open at the same time to accomplish that.
>
> I am looking for a memory efficient way to open and close files, in a 
> loop for all files.
>
> This setup came to mind:
>
> Open 1
>
> Open 2
>
> Display 2-1
>
> Close 1
>
> Open 3
>
> Display 3-2
>
> Close 2
>
> Open 4
>
> Display 4-3
>
> But I am only allowed to close the most recently opened file.
>
> If I stop closing files I will end up having loads of files open:
>
> Open 1
>
> Open 2
>
> Display val.2- val.1
>
> Open 3
>
> Display val.3- val.2
>
> Open 4
>
> Display val.4- val.3
>
> (repeat until #48)
>
> Close 1-48
>
> when I’m at the 23^rd timestep I have file #1,#2 …. #23 open.
>
> I’m afraid the server will crash having so many files open.
>
> I just need #22 and #23 for that display moment.
>
> Or I have to close all files and re-open the last as # 1: file #22 
> becomes file 1 and file #23 becomes file 2:
>
> Open #22 (1)
>
> Open #23 (2)
>
> Display val.2- val.1
>
> Close #23 (2)
>
> Close #22 (1)
>
> Open #23
>
> Open #24
>
> Display val.2- val.1
>
> Close #24
>
> Close #23
>
> But closing a file (#23 in this case) to immediately open it again 
> seems inefficient.
>
> Is there a smart way to handle this?
>
> I was pondering:
>
> Open #1
>
> Open #2
>
> Display val.2- val.1
>
> Open #3 AS and REPLACING 1
>
> Display val.1- val.2 (effectively: #3-#2)
>
> Open #4 AS and REPLACING 2
>
> Display val.2- val.1 (effectively: #4-#3)
>
> Is that possible?
>
> FYI: I’m using a php script to send commands to opengrads, so I’m 
> quite flexible with the available loops.
>
> Thanks,
>
> Chris
>

-- 

Please note that Charles.Seman at noaa.gov should be considered my NOAA
email address, not cjs at gfdl.noaa.gov.

********************************************************************
 Charles Seman                                Charles.Seman at noaa.gov
 U.S. Department of Commerce / NOAA / OAR
 Geophysical Fluid Dynamics Laboratory         voice: (609) 452-6547
 201 Forrestal Road                              fax: (609) 987-5063
 Princeton, NJ  08540-6649            http://www.gfdl.noaa.gov/~cjs/
********************************************************************

"The contents of this message are mine personally and do not necessarily
reflect any position of the Government or NOAA."