<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Paul<br>
<br>
There have been several posts about "segmentation faults". We have
experienced them a lot on our RedHat systems (8.0, 9.0, Fedora core
2,3,4 and even Centos 4.x). The problem is widespread but rather
elusive. However, it seems the sdfopen with a template option is the
quickest way of "fishing" for a segmentation fault. We had a post last
year from Kilian Hagemann who suggested a memory leak (see below my
email) ...<br>
<br>
I must admit that we too have stayed with the pre-compiled binaries as
compiling from source requires a little more intellect than I posses,
despite the noble attempts by Patrice Dumas to point us all in the
right direction.<br>
<br>
My solution for now: run the scripts in smaller chunks to avoid the seg
fault, and wait for those who know more about C and Linux to solve the
issue. Another thought I had: does the stack limits controlled by the
"ulimit" command have any bearing on this? Perhaps the small stack size
(by default in RedHat) constrains something.<br>
<br>
Regards to all<br>
Warren<br>
<br>
Kilian Hagemann wrote:<br>
******************<br>
<div class="moz-text-plain" wrap="true" graphical-quote="true"
style="font-family: -moz-fixed; font-size: 12px;" lang="x-western">
<pre wrap="">Hi there,
I wrote a little script to check for global max/min values of NCEP data from
1993-2004 when I noticed that when it got further and further my system
started thrashing. Initial memory requirement: About 16MB, after every
further year of averaging an additional ~10MB and towards the end of the
script a whopping 780MB!!!! Smells like memory leak/improper pointer
management to me...
The script is attached, and as long as you have the NCEP netcdf data somewhere
available (change the ncepPath variable in the script) it should run fine.
I built 1.9b3 from source using all the extra components except the dods
interface. But that shouldn't make a difference. Let me know if you need any
further information.
Regards,
--
Kilian Hagemann
Climate Systems Analysis Group
University of Cape Town
Republic of South Africa
Tel(w): ++27 21 650 2748
</pre>
</div>
<div class="moz-text-plain" wrap="true" graphical-quote="true"
style="font-family: -moz-fixed; font-size: 12px;" lang="x-western">
<pre wrap="">
<hr size="4" width="90%">
#!/usr/local/grads/bin/grads -lbc
* Copyright 2004 by Kilian Hagemann
*
* Script to extract relevant NCEP data for SOM analysis to find an optimally representative 12 month period for MM5 wind simulation
* WARNING: The min/max detection for automatic scaling works ONLY with a patched version of grads (setting mfcmn.warnflg to 0 in grads.c in version 1.9b3), otherwise the min() and max() functions output a whole lot of garbage which chokes the script
ncepPath = '<i class="moz-txt-slash"><span class="moz-txt-tag">/</span>obs-a/ncep/ncep.new<span
class="moz-txt-tag">/</span></i>'
startYear = 1993
endYear = 2004
var.1 = uwnd
var.2 = vwnd
var.3 = hgt
var.4 = omega
var.5 = rhum
var.6 = shum
var.7 = air
*air.sig995
*omega.sig995
*pr_wtr.eatm
*pres.sfc
*rhum.sig995
*slp.sig995
*uwnd.sig995
*vwnd.sig995
nvars = 7
* first find the global maximum and minimum values
* uncomment following while loop to find these
nvar = 1
while (nvar <= nvars)
global_max = -9999
global_min = 9999
year = startYear
while (year <= endYear)
'sdfopen 'ncepPath%var.nvar'.'year'.nc'
* the dimension arguments in the following have been manually determined
'd max(max(max('var.nvar',x=7,x=15),y=22,y=28), t=1, t=1460)'
max = subwrd(result, 4)
if (max > global_max)
global_max = max
endif
'd min(min(min('var.nvar',x=7,x=15),y=22,y=28), t=1, t=1460)'
min = subwrd(result, 4)
if (min < global_min)
global_min = min
endif
say ' In year 'year' min/max 'var.nvar' are 'min'/'max' over the selected domain'
'close 1'
year = year + 1
endwhile
say
say 'Global min/max 'var.nvar' are 'global_min'/'global_max' over the selected domain'
say
nvar = nvar + 1
endwhile
'quit'
</pre>
</div>
******************************************************************<br>
<br>
Willis, Paul ENV:EX wrote:
<blockquote cite="mid20060524183556.CAEB02050F@mx2.cineca.it"
type="cite">
<pre wrap="">My posting below was premature. I am still experiencing the same problems
when attempting to use the TEMPLATE option with the DDF. Jennifer and
Diane have been kindly working with me off-listserve but without
resolution.
here is what I know:
My environment: linux RH7.3
GrADS install: I reinstalled with pre-compiled binary exe. no change.
Data file corruption: I downloaded a fresh set of data files. no change.
Hoop's 'sdfopen' merge: see previous thread. While this resolves the
offset and missing value warnings, the environment is unstable and core
dumps (segmentation faults) make it unworkable.
My questions du-jour:
1) Has anyone had compatibility problems running rh7.3?
2) Jennifer has 'grepped' GrADS source code and has not found the source
of this warning:
"-- using -1e+15 insteade data type for "missing_value"
Does this look familiar to anyone. note, I have not messed with the GrADS
source code on my machine.
3) Suggestions for further debugging of either the offset problem when
using the DDF, or with segmentation faults when using the 'sdfopen' merge?
Thanks.
Paul Willis
On Sun, 21 May 2006 01:25:35 +0200, Willis, Paul ENV:EX
<a class="moz-txt-link-rfc2396E" href="mailto:paul.willis@GOV.BC.CA"><paul.willis@GOV.BC.CA></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Jennifer, et al.,
As it turns out, the original descriptor file that I sent was fine, as you
and Diane have determined. Upon my debugging of another problem, I
discovered I had incorrectly defined my $GADDIR environment variable. I
have since tested the descriptor file without problem.
I am a GrADS newbie and have a lot to learn. My apologies that you all
spent some time on what turned out to be pilot error.
By the way, I have successfully followed Hoop's 'sdfopen'. However, yes,
I have experienced a number of core dumps. I am too inexperienced to make
an assessment; could be nothing to do with this method and more to some
other misconfiguration on my part. But I will try the two different
methods over the course of the next two weeks and may have some comments
of value.
Thanks.
-pw
On Sat, 20 May 2006 16:14:58 -0400, Jennifer M. Adams <a class="moz-txt-link-rfc2396E" href="mailto:jma@COLA.IGES.ORG"><jma@COLA.IGES.ORG></a>
wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Dear Paul, et al.,
OK, so you could read this file with sdfopen and do it the way Hoop
suggests. But if templating with sdfopen often leads to core dumps on
your operating system, then it's perfectly reasonable to do it with a
'dtype netcdf' descriptor. User's choice. My greater concern is why
the 'dtype netcdf' interface isn't retrieving the 'add_offset' or the
'missing_value' attributes properly for you.
I downloaded a NetCDF file from CDC -- daily means of SLP for 1999
with the identical ncdump output -- and used your descriptor file
with this data set and it read the slp data on my mac without any
problems at all:
ga-> open /data/netcdf/rean/slp.ctl
Scanning description file: /data/netcdf/rean/slp.ctl
Data file /data/netcdf/rean/slp.%y4.nc is open as file 1
LON set to 0 360
LAT set to -90 90
LEV set to 1 1
Time values set: 1999:1:1:0 1999:1:1:0
ga-> q file
File 1 : NCEP Reanalysis Daily Averages
Descriptor: /data/netcdf/rean/slp.ctl
Binary: /data/netcdf/rean/slp.%y4.nc
Type = Gridded
Xsize = 144 Ysize = 73 Zsize = 1 Tsize = 731
Number of Variables = 1
slp 0 -103 mean Daily Sea Level Pressure [Pa]
ga-> d slp
Contouring: 97000 to 105000 interval 1000
ga->
The attributes of this file are exactly the same, so ... why doesn't
it work for you???
On May 18, 2006, at 8:42 PM, Willis, Paul ENV:EX wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Running GrADS 1.9b4 on linux.
</pre>
</blockquote>
<pre wrap="">Which executable are you using: gradsnc, gradshdf, or gradsdods?
Which linux: RH7.1, RH9, or RHE3?
Jennifer
</pre>
<blockquote type="cite">
<pre wrap="">
Thanks in advance for any guidance.
-paul
*********************
ga-> q dim
Default file number is: 1
X is varying Lon = 0 to 360 X = 1 to 145
Y is varying Lat = -90 to 90 Y = 0.999999 to 73
Z is fixed Lev = 1 Z = 1
T is fixed Time = 00Z01JAN1999 T = 1
ga-> d slp
" -- using -1e+15 insteade data type for "missing_value
" not handled
-- using -1e+15 instead
Warning: Could not retrieve add offset data type -- setting add
offset to
0.0
Contouring: -20000 to -16500
interval 500
*********************
here is my descriptor file
*********************
dset /home/pfwillis/2006/Grads/circulation/Climate/data/slp_daily.%
y4.nc
dtype netcdf
options yrev template
undef -1e15 missing_value
unpack scale_factor add_offset
title NCEP Reanalysis Daily Averages
xdef 144 linear 0 2.5
ydef 73 linear -90 2.5
zdef 1 linear 1 1
tdef 731 linear 00Z01jan1999 24hr
vars 1
slp 0 t,y,x mean Daily Sea Level Pressure [Pa]
endvars
*********************
should an ncdump provide useful information on this, see below.
*********************
netcdf slp_daily.1999 {
dimensions:
lon = 144 ;
lat = 73 ;
time = UNLIMITED ;
// (365 currently)
variables:
float lat(lat) ;
lat:units = "degrees_north" ;
lat:actual_range = 90.f, -90.f ;
lat:long_name = "Latitude" ;
float lon(lon) ;
lon:units = "degrees_east" ;
lon:long_name = "Longitude" ;
lon:actual_range = 0.f, 357.5f ;
double time(time) ;
time:units = "hours since 1-1-1 00:00:0.0" ;
time:long_name = "Time" ;
time:actual_range = 17514144., 17522880. ;
time:delta_t = "0000-00-01 00:00:00" ;
time:avg_period = "0000-00-01 00:00:00" ;
short slp(time, lat, lon) ;
slp:long_name = "mean Daily Sea Level Pressure" ;
slp:valid_range = 87000.f, 115000.f ;
slp:actual_range = 92970.f, 110552.f ;
slp:units = "Pascals" ;
slp:add_offset = 119765.f ;
slp:scale_factor = 1.f ;
slp:missing_value = 32766s ;
slp:precision = 0s ;
slp:least_significant_digit = -1s ;
slp:GRIB_id = 2s ;
slp:GRIB_name = "PRMSL" ;
slp:var_desc = "Sea Level Pressure\n",
"P" ;
slp:dataset = "NCEP Reanalysis Daily Averages\n",
"AJ" ;
slp:level_desc = "Sea Level\n",
"I" ;
slp:statistic = "Mean\n",
"M" ;
slp:parent_stat = "Individual Obs\n",
"I" ;
// global attributes:
:Conventions = "COARDS" ;
:title = "mean daily NMC reanalysis (1999)" ;
:base_date = 1999s, 1s, 1s ;
:history = "created 99/01/04 by Hoop (netCDF2.3)" ;
:description = "Data is from NMC initialized reanalysis\n",
"(4x/day). It consists of most variables
interpolated to\n",
"pressure surfaces from model (sigma) surfaces." ;
:platform = "Model" ;
data:
lat = .......
</pre>
</blockquote>
<pre wrap="">Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Beltsville, MD 20705
<a class="moz-txt-link-abbreviated" href="mailto:jma@cola.iges.org">jma@cola.iges.org</a>
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">
</pre>
</body>
</html>