[gradsusr] Performance issue with large files open (was: Problem with Nomads servers)
Jennifer M Adams
jadams21 at gmu.edu
Tue Feb 12 18:12:41 EST 2019
> On Feb 11, 2019, at 4:48 AM, Brian Gaze <brianw.gaze at googlemail.com> wrote:
>
> One thing I have noticed with 2.x versions of GrADS and OpenGrADS is that they run into performance issues when many large files are opened. An example is when working with aggregated GFS / GEFS data sets.
The number of open files should not affect performance. How many files are you talking about? What kind of aggregation? Can you document the performance change so it can be reproduced?
> I believe this *could* be related to the metabuffer filling up? Therefore I wondered whether it would be possible for you to add a command line switch that sets the metabuffer size at run time?
That is already in GrADS:
-m NNN Sets metafile buffer size to NNN, which must be an integer. Default value is 1000000.
In more recent versions, the metabuffer is dynamically allocated as needed, so the -m argument is not needed and therefore ignored.
—Jennifer
>
> Thanks
>
> BWG
>
>
>
>
> On Mon, 11 Feb 2019 at 08:53, <gradsusr-request at gradsusr.org> wrote:
> Send gradsusr mailing list submissions to
> gradsusr at gradsusr.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://gradsusr.org/mailman/listinfo/gradsusr
> or, via email, send a message with subject or body 'help' to
> gradsusr-request at gradsusr.org
>
> You can reach the person managing the list at
> gradsusr-owner at gradsusr.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gradsusr digest..."
>
>
> Today's Topics:
>
> 1. Re: problens with opendap from nomads (Reginaldo Ventura de S?)
> 2. Re: Problem with Nomads servers (Arlindo da Silva)
> 3. Re: Problem with Nomads servers (Tim Van Damme)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 11 Feb 2019 00:10:59 -0200
> From: Reginaldo Ventura de S? <regis at lamma.ufrj.br>
> To: GrADS Users Forum <gradsusr at gradsusr.org>
> Subject: Re: [gradsusr] problens with opendap from nomads
> Message-ID:
> <CADz=HLORo_GL+-zZ9mZscGAufBei6zEd7msccRZQrbxwhi7jTA at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> >From NCEP Support.
> Hello,
>
> Yesterday was our cut-over date for NOMADS transition to HTTPS. As we had
> advertised in our NWS Service Change Notices (SCN) and NOMADS Listserv
> emails over the last month, users of Opendap would be affected by the
> transition to HTTPS. This is because most versions of GRADS prior to GRADS
> v2.2.0 do not support SSL (downloading from https websites). You will need
> to download/install a more recent version of GRADS in order to regain
> access to NOMADS Opendap data feeds.
>
> GRADS can be downloaded from the COLA website. http://cola.gmu.edu/grads/
>
> It is also possible that your version of curl needs to be updated to
> support SSL.
>
> Please let us know if we can be of further assistance.
>
> On Wed, Feb 6, 2019 at 4:10 PM Mark Sponsler <msponsler at comcast.net> wrote:
>
> > Hi,
> > Yes.
> > It does not work.
> > NOAA converted all their DODS servers to https format yesterday. The
> > latest version of Opengrads does not support https secure connection.
> >
> > I've sent a message to the gentleman that supports opengrads to see if a
> > hot fix is in work.
> >
> > More to follow.
> > Thanks,
> > Mark
> >
> > On February 6, 2019 9:47:43 AM PST, "Reginaldo Ventura de S?" <
> > regis at lamma.ufrj.br> wrote:
> >>
> >>
> >> Dear users,
> >>
> >> Is anyone having trouble to acess nomads using opendap?
> >>
> >>
> >> greetings
> >> []?regis
> >>
> >
> > Thanks,
> > Mark Sponsler
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://gradsusr.org/pipermail/gradsusr/attachments/20190211/bb0bd8a0/attachment-0001.html>
>
> ------------------------------
>
> Message: 2
> Date: Sun, 10 Feb 2019 23:33:47 -0500
> From: Arlindo da Silva <dasilva at alum.mit.edu>
> To: tim at stormcell.be, GrADS Users Forum <gradsusr at gradsusr.org>
> Subject: Re: [gradsusr] Problem with Nomads servers
> Message-ID:
> <CALz7xZcCoN5yXj+kxesDhGuAHqPsT9km95Wu2mBujNJJyY5i6Q at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> All,
>
> OpenGrADS builds for 2.2.1 will be announced soon. They are undergoing
> additional testing. And yes, https:// will be enabled.
>
> Arlindo
>
>
> On Sun, Feb 10, 2019 at 6:00 AM Tim Van Damme <tim at stormcell.be> wrote:
>
> > I am experiencing the same issue with Opengrads for Windows.
> >
> > It would be cool if there would be a simple workaround.
> >
> >
> >
> > Tim
> >
> >
> >
> > *From:* gradsusr [mailto:gradsusr-bounces at gradsusr.org] *On Behalf Of *avi
> > kojokro
> > *Sent:* 10 February 2019 09:39
> > *To:* GrADS Users Forum <gradsusr at gradsusr.org>
> > *Subject:* Re: [gradsusr] Problem with Nomads servers
> >
> >
> >
> > Hi Jennifer
> >
> > thank you for the answer
> > I forgot to mention that I use Grads on windows
> > Is there a solution to this issue?
> >
> >
> >
> > Avi.
> >
> >
> >
> > ??????? ???, 9 ????? 2019 ?-17:50 ??? ?Jennifer M Adams? <?
> > jadams21 at gmu.edu?>:
> >
> > Opengrads builds are not linked with an SSL-enabled version of the curl
> > library, which means that they are unable to access HTTPS opendap URLs.
> > Until Arlindo posts new builds, you will have to use COLA?s version of
> > GrADS, version 2.1.1.b0 or newer.
> > ?Jennifer
> >
> > > On Feb 9, 2019, at 9:43 AM, avi kojokro <kojokroavi at gmail.com> wrote:
> > >
> > >
> > > Hello everyone,
> > > For a few days now, there has been a problem connecting to Nomads servers
> > > Here is a Screenshot of the Grads command board, I tried to enter the
> > server both with the HTTP address and also with the HTTPS address and both
> > cannot log on to the server, The same problem also for the servers of
> > preparafv3.......
> > > Does anyone know what the problem is?.
> > >
> > > the Screenshot: for GFS
> > >
> > >
> > > GX Package Initialization: Size = 11 8.5
> > > Command line history in \Users\OWNER/.grads.log
> > > <://nomads.ncep.noaa.gov:80/dods/gfs_0p25/gfs20190209/gfs_0p25_00z
> > > Scanning self-describing file:
> > http://nomads.ncep.noaa.gov:80/dods/gfs_0p25/gfs20190209/gfs_0p25_00z
> > > curl error details:
> > > Error: nc_open failed to open file
> > http://nomads.ncep.noaa.gov:80/dods/gfs_0p25/gfs20190209/gfs_0p25_00z
> > > NetCDF: I/O failure
> > > gadsdf: Couldn't ingest SDF metadata.
> > > <s://nomads.ncep.noaa.gov:80/dods/gfs_0p25/gfs20190209/gfs_0p25_00z
> > > Scanning self-describing file:
> > https://nomads.ncep.noaa.gov:80/dods/gfs_0p25/gfs20190209/gfs_0p25_00z
> > > curl error details:
> > > Error: nc_open failed to open file
> > https://nomads.ncep.noaa.gov:80/dods/gfs_0p25/gfs20190209/gfs_0p25_00z
> > > NetCDF: I/O failure
> > > gadsdf: Couldn't ingest SDF metadata.
> > > ga->
> > >
> > > the Screenshot: for preparafv3
> > >
> > >
> > > Command line history in \Users\OWNER/.grads.log
> > > <090/dods/gfs_0p25_preparafv3/gfs20190208/gfs_0p25_preparafv3_12z
> > > Scanning self-describing file:
> > https://nomads.ncep.noaa.gov:9090/dods/gfs_0p25_preparafv3/gfs20190208/gfs_0p25_preparafv3_12z
> > > curl error details:
> > > Error: nc_open failed to open file
> > https://nomads.ncep.noaa.gov:9090/dods/gfs_0p25_preparafv3/gfs20190208/gfs_0p25_preparafv3_12z
> > > NetCDF: I/O failure
> > > gadsdf: Couldn't ingest SDF metadata.
> > > ga->
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > gradsusr mailing list
> > > gradsusr at gradsusr.org
> > >
> > http://secure-web.cisco.com/1YIE5nvA4vDI11pJu6xRoRasU3TtHbb6Nb_ipY6ubaBWg-MrWxXs-Ow4mVOT1fmEMkxfc61JM-vXkeleikdHwg3aqcbNnyt7_L60-pTtRlBbh9kWxwK6NddFFapjPLsum26-wvNlFJgkZBRxlD0CGY3IEI4eTOMrI2IOuCPIVDe9FAlVMm3qI5p4QgnJ_o1LB2pPxTAeZdZr55-dqEUDEhcsiKJRsWYVseFYyyfSWcLJUcw-8J_c_U5YZCjFdAbwn_C1CeooqQsIO7klAy8eQv8J5iUC0rd-j93KIgO5vbva2i7fRVtKXTluWYn8-AeRDRRZqzoEk3sG2GdiMfhgKALBLOPDEx9sW1ZXO3LNCoS3SKOmDmp0xOdsF05J23qtNwlsMt23x9VVaGf1MWOgfcMlmaTC9A6LTUN4ArpIpeShINH26j3bbZGIG30lLAXLD/http%3A%2F%2Fgradsusr.org%2Fmailman%2Flistinfo%2Fgradsusr
> >
> > --
> > Jennifer Miletta Adams
> > Center for Ocean-Land-Atmosphere Studies (COLA)
> > George Mason University
> >
> >
> >
> >
> > _______________________________________________
> > gradsusr mailing list
> > gradsusr at gradsusr.org
> > http://gradsusr.org/mailman/listinfo/gradsusr
> >
> > _______________________________________________
> > gradsusr mailing list
> > gradsusr at gradsusr.org
> > http://gradsusr.org/mailman/listinfo/gradsusr
> >
>
>
> --
> Arlindo da Silva
> *dasilva at alum.mit.edu <dasilva at alum.mit.edu>*
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://gradsusr.org/pipermail/gradsusr/attachments/20190210/3ca091a9/attachment-0001.html>
>
> ------------------------------
>
> Message: 3
> Date: Mon, 11 Feb 2019 08:30:17 +0100
> From: "Tim Van Damme" <tim at stormcell.be>
> To: "'Arlindo da Silva'" <dasilva at alum.mit.edu>, "'GrADS Users Forum'"
> <gradsusr at gradsusr.org>
> Subject: Re: [gradsusr] Problem with Nomads servers
> Message-ID: <000c01d4c1db$a442d930$ecc88b90$@stormcell.be>
> Content-Type: text/plain; charset="utf-8"
>
> Hello Arlindo,
>
>
>
> That is fantastic news. Let?s hope we do not need to alter our scripts fort he newer version of Opengrads.
>
> Meanwhile I was wondering? if anyone has a GFS script that shows the different sorts of precipitation on one map... -> i am interested!
>
>
>
>
>
> Kind regards,
>
>
>
> Tim
>
>
>
> From: Arlindo da Silva [mailto:dasilva at alum.mit.edu]
> Sent: 11 February 2019 05:34
> To: tim at stormcell.be; GrADS Users Forum <gradsusr at gradsusr.org>
> Subject: Re: [gradsusr] Problem with Nomads servers
>
>
>
> All,
>
>
>
> OpenGrADS builds for 2.2.1 will be announced soon. They are undergoing additional testing. And yes, https:// will be enabled.
>
>
>
> Arlindo
>
>
>
>
>
> On Sun, Feb 10, 2019 at 6:00 AM Tim Van Damme <tim at stormcell.be <mailto:tim at stormcell.be> > wrote:
>
> I am experiencing the same issue with Opengrads for Windows.
>
> It would be cool if there would be a simple workaround.
>
>
>
> Tim
>
>
>
> From: gradsusr [mailto:gradsusr-bounces at gradsusr.org <mailto:gradsusr-bounces at gradsusr.org> ] On Behalf Of avi kojokro
> Sent: 10 February 2019 09:39
> To: GrADS Users Forum <gradsusr at gradsusr.org <mailto:gradsusr at gradsusr.org> >
> Subject: Re: [gradsusr] Problem with Nomads servers
>
>
>
> Hi Jennifer
>
> thank you for the answer
> I forgot to mention that I use Grads on windows
> Is there a solution to this issue?
>
>
>
> Avi.
>
>
>
> ??????? ???, 9 ????? 2019 ?-17:50 ??? ?Jennifer M Adams? <? <mailto:jadams21 at gmu.edu> jadams21 at gmu.edu?>:
>
> Opengrads builds are not linked with an SSL-enabled version of the curl library, which means that they are unable to access HTTPS opendap URLs.
> Until Arlindo posts new builds, you will have to use COLA?s version of GrADS, version 2.1.1.b0 or newer.
> ?Jennifer
>
> > On Feb 9, 2019, at 9:43 AM, avi kojokro <kojokroavi at gmail.com <mailto:kojokroavi at gmail.com> > wrote:
> >
> >
> > Hello everyone,
> > For a few days now, there has been a problem connecting to Nomads servers
> > Here is a Screenshot of the Grads command board, I tried to enter the server both with the HTTP address and also with the HTTPS address and both cannot log on to the server, The same problem also for the servers of preparafv3.......
> > Does anyone know what the problem is?.
> >
> > the Screenshot: for GFS
> >
> >
> > GX Package Initialization: Size = 11 8.5
> > Command line history in \Users\OWNER/.grads.log
> > <://nomads.ncep.noaa.gov:80/dods/gfs_0p25/gfs20190209/gfs_0p25_00z <http://nomads.ncep.noaa.gov:80/dods/gfs_0p25/gfs20190209/gfs_0p25_00z>
> > Scanning self-describing file: http://nomads.ncep.noaa.gov:80/dods/gfs_0p25/gfs20190209/gfs_0p25_00z
> > curl error details:
> > Error: nc_open failed to open file http://nomads.ncep.noaa.gov:80/dods/gfs_0p25/gfs20190209/gfs_0p25_00z
> > NetCDF: I/O failure
> > gadsdf: Couldn't ingest SDF metadata.
> > <s://nomads.ncep.noaa.gov:80/dods/gfs_0p25/gfs20190209/gfs_0p25_00z <http://nomads.ncep.noaa.gov:80/dods/gfs_0p25/gfs20190209/gfs_0p25_00z>
> > Scanning self-describing file: https://nomads.ncep.noaa.gov:80/dods/gfs_0p25/gfs20190209/gfs_0p25_00z
> > curl error details:
> > Error: nc_open failed to open file https://nomads.ncep.noaa.gov:80/dods/gfs_0p25/gfs20190209/gfs_0p25_00z
> > NetCDF: I/O failure
> > gadsdf: Couldn't ingest SDF metadata.
> > ga->
> >
> > the Screenshot: for preparafv3
> >
> >
> > Command line history in \Users\OWNER/.grads.log
> > <090/dods/gfs_0p25_preparafv3/gfs20190208/gfs_0p25_preparafv3_12z
> > Scanning self-describing file: https://nomads.ncep.noaa.gov:9090/dods/gfs_0p25_preparafv3/gfs20190208/gfs_0p25_preparafv3_12z
> > curl error details:
> > Error: nc_open failed to open file https://nomads.ncep.noaa.gov:9090/dods/gfs_0p25_preparafv3/gfs20190208/gfs_0p25_preparafv3_12z
> > NetCDF: I/O failure
> > gadsdf: Couldn't ingest SDF metadata.
> > ga->
> >
> >
> >
> >
> > _______________________________________________
> > gradsusr mailing list
> > gradsusr at gradsusr.org <mailto:gradsusr at gradsusr.org>
> > http://secure-web.cisco.com/1YIE5nvA4vDI11pJu6xRoRasU3TtHbb6Nb_ipY6ubaBWg-MrWxXs-Ow4mVOT1fmEMkxfc61JM-vXkeleikdHwg3aqcbNnyt7_L60-pTtRlBbh9kWxwK6NddFFapjPLsum26-wvNlFJgkZBRxlD0CGY3IEI4eTOMrI2IOuCPIVDe9FAlVMm3qI5p4QgnJ_o1LB2pPxTAeZdZr55-dqEUDEhcsiKJRsWYVseFYyyfSWcLJUcw-8J_c_U5YZCjFdAbwn_C1CeooqQsIO7klAy8eQv8J5iUC0rd-j93KIgO5vbva2i7fRVtKXTluWYn8-AeRDRRZqzoEk3sG2GdiMfhgKALBLOPDEx9sW1ZXO3LNCoS3SKOmDmp0xOdsF05J23qtNwlsMt23x9VVaGf1MWOgfcMlmaTC9A6LTUN4ArpIpeShINH26j3bbZGIG30lLAXLD/http%3A%2F%2Fgradsusr.org%2Fmailman%2Flistinfo%2Fgradsusr
>
> --
> Jennifer Miletta Adams
> Center for Ocean-Land-Atmosphere Studies (COLA)
> George Mason University
>
>
>
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org <mailto:gradsusr at gradsusr.org>
> http://gradsusr.org/mailman/listinfo/gradsusr
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org <mailto:gradsusr at gradsusr.org>
> http://gradsusr.org/mailman/listinfo/gradsusr
>
>
>
>
>
>
> --
>
> Arlindo da Silva
> dasilva at alum.mit.edu <mailto:dasilva at alum.mit.edu>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://gradsusr.org/pipermail/gradsusr/attachments/20190211/d35673ae/attachment.html>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
>
>
> ------------------------------
>
> End of gradsusr Digest, Vol 108, Issue 11
> *****************************************
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://secure-web.cisco.com/1fmwZYWkbaV2teiDoLN8xD9dvQHoTqO70AzLRkzkwuqfLf2VoFL1XatTrBVNBfdDd8S1aogqesGLtj6HCjZJQqoF1hb6xTr4f4whbRff7pNysWJ8Pb6254bfq7RKIWB9IbzFvi_ratFg5tgQmWOO5xlawxwNhVeZC99t1ttvKu7j7wt5JOcWXlk_WFYzuheOWGDPWqRhHlQUoVCMhH7UdqxYJuzQq6T23sC4hcTYe1xfoU1lx8wTvIXsfBGAjtDrazvLoi91QXLCkgUdtjQzAdkZve3hON_3wwmFrlhFfiCFtA7G7t2M1HuHv-K2FEfvYCqgYh2oTubkBBStUNJZhLmk1ZiuDTOqEQS_jhpdRUxMJeviIFRVXgMzkGdY1P-YBSbz1MAmD4DIouf22o9GqAF0BYP4YfBJ9R7SLTZKUkdh8xa5ZpfF8ooOOhaMLHJau/http%3A%2F%2Fgradsusr.org%2Fmailman%2Flistinfo%2Fgradsusr
--
Jennifer Miletta Adams
Center for Ocean-Land-Atmosphere Studies (COLA)
George Mason University
More information about the gradsusr
mailing list