[gradsusr] GrADS performance vs alternatives?

Tue Apr 12 10:38:41 EDT 2022

Part of the time to create a plot is the time needed to create the *.ctl
and *.idx files.

You can save time creating the *.ctl file by using an existing *.ctl file
and changing
the dset, index and tdef lines of the *.ctl.

If you are using "options template", then alt_gmp is faster than gribmap.,
Alt_gmp is
threaded, so the processing of ithe grib files can be handled in parallel.
You will
have to tune the number of threads because a locally mounted SSD, a NFS file
system or locally mounted hard drive have different bottlenecks.

Wesley

On Tue, Apr 12, 2022 at 8:22 AM Jennifer M Adams <jadams21 at gmu.edu> wrote:

> Follow up in case the performance issues are related to I/O and not the
> graphics buffer.
>
> In general, GrADS I/O is very fast, even for the less friendly file
> formats, and it does take advantage of cacheing when possible. However, you
> can try using 'define' if you are drawing the same grid more than once
> (e.g. a shaded fill with a contour line overlay). Or you can read data into
> memory for a global domain and then draw subregions from the defined grid
> instead.
> --Jennifer
> ------------------------------
> *From:* gradsusr <gradsusr-bounces at gradsusr.org> on behalf of Jennifer M
> Adams <jadams21 at gmu.edu>
> *Sent:* Monday, April 11, 2022 9:22 PM
> *To:* GrADS Users Forum <gradsusr at gradsusr.org>
> *Subject:* Re: [gradsusr] GrADS performance vs alternatives?
>
> The original question didn't mention which version of GrADS is in use. The
> more recent ones have a metafile buffer that gets dynamically allocated as
> you start drawing stuff. The -m option on startup allows you to set a
> maximum buffer size. As you use 'draw' or 'display' commands, each
> graphical element gets added to the metabuffer. Check the memory usage of
> the grads executable while the drawing is going on to see if that memory
> footprint is growing steadily to a point where you're reaching system
> limits. Use 'reinit' or 'undefine' to keep memory use in check. When
> rendering an image, GrADS goes through the metabuffer and draws each line,
> dot, polygon, or letter. If you're drawing a grfill plot of a very high
> resolution grid, that's a lot of little tiny squares to draw and it can
> take a while. Then if you add basemap to mask out land or ocean, that's a
> whole lot of polygons to draw on top of that, and the more you layer on
> with contour line overlays or labels and other cosmetics... you get the
> idea. A rendering time of 1.5 seconds doesn't seem like a lot, but I can
> see where it would scale up. Use maskout() or negative numbers in your 'set
> ccols' command to avoid drawing filler you don't need in your graphic.
> Maybe there's some things you can pre-draw and use as a background or
> foreground image (merged with the -b or -f option with gxprint) so they
> don't have to be rendered multiple times. Or check if you are needlessly
> drawing stuff that will get masked out by another layer on top.
>
> If you're testing without using batch mode, does it take 1.5 seconds to
> draw the graphic in the display window? Watch and see if you can tell what
> takes the longest to draw for each of your plot styles, maybe that will
> shed some light on what's slowing down your image production.
>
> Of course, it may just be a basic truth that your beautiful GrADS plots
> are going to need 1.5 seconds to get ready for prime time. 😄
> --Jennifer
>
>
> ------------------------------
> *From:* gradsusr <gradsusr-bounces at gradsusr.org> on behalf of Mark
> Sponsler <msponsler at comcast.net>
> *Sent:* Monday, April 11, 2022 12:21 PM
> *To:* GrADS Users Forum <gradsusr at gradsusr.org>
> *Subject:* Re: [gradsusr] GrADS performance vs alternatives?
>
> One other consideration is HOW your data is stored. Reading grib files
> versus reading binary files I believe is inherently quicker.
>
> Try coverting your grib2 files to binary using wgrib2 before you start
> running scrpits to produce images.
>
> Just a thought.
> Mark
>
> On April 11, 2022 1:33:15 AM PDT, Davide Sacchetti <
> davide.sacchetti at arpal.liguria.it> wrote:
>
> I see
> there are different ways to produce png outputs, depending also on grads
> version
> eg: gxprint, printim, print: you could investigate which method is faster.
> when I have to produce lots of png I divide the number of png processing
> among the available cpus, but I immagine you already did this
>
> On my knowledge it is difficult to read and render data from grib or
> netcdf faster than grads does, the only chance I imagine is to take
> advantage of parallelization
>
> Davide
>
> *Dott. Davide Sacchetti*
>
>
>
>
> *Dipartimento Stato dell’Ambiente e Tutela dai Rischi Naturali U.O. Clima
> Meteo Idro Viale Brigate Partigiane 2, 16129 Genova (I) tel: +39 0106437535
> mail: davide.sacchetti at arpal.liguria.it <mauro.damonte at arpal.liguria.it>
> web: https://www.arpal.liguria.it/homepage/meteo.html
> <http://secure-web.cisco.com/1gZr8j65sWn3pQixShqA64yx2-ElCh3P2HieHemBe2Gk4wfXBgHys44Avjcj6ZTK1C3CIq764eOP1ntiV6A9ZXZfXzVtBROgZx5LAf017ewplrcVXyW3NeWn_CNFF24alGBreWMBMPcab0Nh5SHId-7FZl5X3rQx6_R8pCAUymoAUfX4SrlMbg-2dycf8dphl8V3m7SWWAQugf1JjfIar0Ol_xc--xJnJs7L624ihdh4IPkxQoaZvPBPKqzL2bSTAjPqSu4fCAKW_eol8TODIjRNNkaxLeNdazFqEy5wta4aHKK_ypzH3FYJcqkRtM5IBkYOXnUSPuvWLUaV_BUI9fbACyDbOZnVABcrLoFXC6X6G4p2aXxf05DkHEMJFnGJkBMxAMqOYtYx5Mn0TTmcQd1K-2Kj2AHGulwDZRZasvmCzhD5hE31ZUuDBTG3RjgPu/http%3A%2F%2Fwww.meteoliguria.it%2F>*
>
>
> *Da: *Brian Gaze <brianw.gaze at googlemail.com>
> *A: *<gradsusr at gradsusr.org>
> *Inviato: *08/04/2022 19:03
> *Oggetto: *[gradsusr] GrADS performance vs alternatives?
>
> Thanks David.
>
> The data is stored locally on high speed SSDs.
>
> Any tips with scripting? My professional background is software
> development so I'm reasonably comfortable working with grads script.
> However, I've not found many tweaks that make a great deal of difference in
> terms of performance.
>
> Thanks
>
> Brian
>
>
> On Fri, 8 Apr 2022 at 17:08, <gradsusr-request at gradsusr.org> wrote:
>
> Send gradsusr mailing list submissions to
>         gradsusr at gradsusr.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://gradsusr.org/mailman/listinfo/gradsusr
> <https://secure-web.cisco.com/163dD1XGeO-hjnmSvUANLOGWcV4Myz0XcxmmeR5AW1l9fMIYZvifQ1cSy3DXLU9gJg7IRXjSZlFt9uWucCXDBEopejhCeEZySbPKE3VK7QLpGr6pP98nvm_QJA4QTBhMlg_8ndf-6_dqqY8nHCdZn94z-KHMSuu9xfcCHGqtmT73Q-S-huFoJ7Dw2rlXf1vIAMCGXHB5zHshe3D4ZmM-LTViFUQQbg4E8CN8aeZZVC_Y1962Gnh6bONjXd0Semezdg8hKlXZV21mqWKEUT5i6q9kbCEtgQ84DzV9j5qhpwgi_T3SDGlB5j34uI3sjuF9lLa8OBG4yQZDwM4hCAfRu9KXP019jxqG2g87KZ9XuxXoQEVXS5jc_EZuA6gIMZj8pnHgl_NOrXLDno5zBLaA78ib-i8RHTPZLvtDZYBdV2mbqnFgENQ9_DkCYCRNfmP66/https%3A%2F%2Furlsand.esvalabs.com%2F%3Fu%3Dhttp%253A%252F%252Fgradsusr.org%252Fmailman%252Flistinfo%252Fgradsusr%26e%3D7312d17f%26h%3D5ee7e1e3%26f%3Dy%26p%3Dn>
> or, via email, send a message with subject or body 'help' to
>         gradsusr-request at gradsusr.org
>
> You can reach the person managing the list at
>         gradsusr-owner at gradsusr.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gradsusr digest..."
>
>
> Today's Topics:
>
>    1. Re: GrADS performance vs alternatives? (Davide Sacchetti)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 7 Apr 2022 18:06:10 +0200
> From: Davide Sacchetti <davide.sacchetti at arpal.liguria.it>
> To: GrADS Users Forum <gradsusr at gradsusr.org>
> Subject: Re: [gradsusr] GrADS performance vs alternatives?
> Message-ID: <51862359-4676 at mbox.arpal.liguria.it>
> Content-Type: text/plain; charset="utf-8"
>
> the speed may depend on cpu but also on where your data is stored (are
> your datasets available locally?) and how your script is written ...
> Upon my knowloedge GrADS IS fast, it may be the fastest sw you can find
> everywhere ...
> Davide
>
>
> Dott. Davide Sacchetti
> Dipartimento Stato dell?Ambiente e Tutela dai Rischi Naturali
> U.O. Clima Meteo Idro
> Viale Brigate Partigiane 2, 16129 Genova (I)
> tel: +39 0106437535
> mail:?davide.sacchetti at arpal.liguria.it? web:?
> https://www.arpal.liguria.it/homepage/meteo.html
> <https://secure-web.cisco.com/1LIZPEjfzGBuSGwKcHKEOukwPXRwTCBIU_CPX8ZtsY9O0P34nXLlUGBlSD7PjeNRIWdDE_51LCqDVo9FNQERjSjwEk8dXp1e_fN00HUbJpWWgTOyFw5HWXD8hX_j_FI0Ue43-KKvchAHGg0qo_1kmJ3YZ1zobRPH2Op9HZISH1mAIHbZxtfyFv-bNXiWQGzXIX5YAzYeF-pT09ko1ZVRIUiXYOQwXQkxxRpdjK8D6DPfYEbT84hIGKwyiEFOlh2CTvcnlosOScgdcB9Jj43YFABpmRqgIwy6BqGZz_XzN-DgLvQQMu7Ch5FsRCI2GqIVZWywxj-33xVHUIcw_ftdOp09IHmFMqUq8bYA9RQvEtdVLPolAwwrANerDWFjkRUwhW_hOdkR9JOyRgYbyHLs_kVJNa-BKiHvQeLqtnmxEHfrca06ypJGZ3b0U1O9RLBYV/https%3A%2F%2Furlsand.esvalabs.com%2F%3Fu%3Dhttps%253A%252F%252Fwww.arpal.liguria.it%252Fhomepage%252Fmeteo.html%26e%3D7312d17f%26h%3D07ef49f6%26f%3Dy%26p%3Dn>
>
>
>
>  Da:   Brian Gaze <brianw.gaze at googlemail.com>
>  A:   <gradsusr at gradsusr.org>
>  Inviato:   07/04/2022 17:07
>  Oggetto:   [gradsusr] GrADS performance vs alternatives?
>
>
>
>                  Questa ? la prima volta che ricevi un'email da questo
> mittente.         Assicurati che sia qualcuno di cui ti fidi.
>
> Hi,
>
>
>
> I've recently started generating ECMWF 50 member ensemble plots using a
> GrADS script for a large latitude - longitude range. Technically it all
> works fine, but the issue I'm having is performance. It is dependent of
> course on CPU speed, but even with a very fast processor each individual
> graph can take approx 1.5 seconds to render and save.?
>
>
> To an extent running instances on multiple cores helps, but generating a
> range of graphs (e.g. 2m temps, 850hPa temps, MSLP etc...) can take a long
> time. Perhaps 40 mins or so for a country the size of the United Kingdom.?
>
>
> Does anyone have tips for how to optimize performance in the script?
> Alternatively, is there software available which runs much faster than
> GrADS script to do the same thing??
>
>
> Thanks
>
>
> BWG
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
>
> https://urlsand.esvalabs.com/?u=http%3A%2F%2Fgradsusr.org%2Fmailman%2Flistinfo%2Fgradsusr&e=7312d17f&h=5ee7e1e3&f=y&p=n
> <https://secure-web.cisco.com/163dD1XGeO-hjnmSvUANLOGWcV4Myz0XcxmmeR5AW1l9fMIYZvifQ1cSy3DXLU9gJg7IRXjSZlFt9uWucCXDBEopejhCeEZySbPKE3VK7QLpGr6pP98nvm_QJA4QTBhMlg_8ndf-6_dqqY8nHCdZn94z-KHMSuu9xfcCHGqtmT73Q-S-huFoJ7Dw2rlXf1vIAMCGXHB5zHshe3D4ZmM-LTViFUQQbg4E8CN8aeZZVC_Y1962Gnh6bONjXd0Semezdg8hKlXZV21mqWKEUT5i6q9kbCEtgQ84DzV9j5qhpwgi_T3SDGlB5j34uI3sjuF9lLa8OBG4yQZDwM4hCAfRu9KXP019jxqG2g87KZ9XuxXoQEVXS5jc_EZuA6gIMZj8pnHgl_NOrXLDno5zBLaA78ib-i8RHTPZLvtDZYBdV2mbqnFgENQ9_DkCYCRNfmP66/https%3A%2F%2Furlsand.esvalabs.com%2F%3Fu%3Dhttp%253A%252F%252Fgradsusr.org%252Fmailman%252Flistinfo%252Fgradsusr%26e%3D7312d17f%26h%3D5ee7e1e3%26f%3Dy%26p%3Dn>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://gradsusr.org/pipermail/gradsusr/attachments/20220407/a20b8590/attachment-0001.html
> <https://secure-web.cisco.com/12AezH2TlXxg4q3edBAH9cssl9qlPgHU928uAsy6oUBjLPuNOMN9gr45aNKl0DC0PObuVL93D0WrlBQND_SO9R6wS5czhAMD4g_3gdlCjduoTxlKUDKdB-v-zk7y26auowee86bl3EVKIeopIFWeTgnFd8ztf4Q1nb0xX-OXh9ZGf3mf04WJdPbDHXFBPVfme9xe9SKeJ0PhzB0eS5_45kY0Jes_FGhmkkZv7m_Ktug6jgPiRXrHhKxj6-9tHjWlCIPtAshlJxPuLmXR9eWJL0TU75jWOWEjCfuBAUwUUPbt-AY-hP-T92X0CpojKoD9vmuYUv42R_wa2dnkrfu6eFqdRGkQfrhbHraasbYbipub0lwyqQJXEQOZ2Sh0QvNYNPIfWak5_nZGQbQWx9PHR0q5xe-e3sehg6tw3AZdnyqjuXLZ8P267gDaTxNtLM0kF/https%3A%2F%2Furlsand.esvalabs.com%2F%3Fu%3Dhttp%253A%252F%252Fgradsusr.org%252Fpipermail%252Fgradsusr%252Fattachments%252F20220407%252Fa20b8590%252Fattachment-0001.html%26e%3D7312d17f%26h%3D296be31e%26f%3Dy%26p%3Dn>
> >
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
> <https://secure-web.cisco.com/163dD1XGeO-hjnmSvUANLOGWcV4Myz0XcxmmeR5AW1l9fMIYZvifQ1cSy3DXLU9gJg7IRXjSZlFt9uWucCXDBEopejhCeEZySbPKE3VK7QLpGr6pP98nvm_QJA4QTBhMlg_8ndf-6_dqqY8nHCdZn94z-KHMSuu9xfcCHGqtmT73Q-S-huFoJ7Dw2rlXf1vIAMCGXHB5zHshe3D4ZmM-LTViFUQQbg4E8CN8aeZZVC_Y1962Gnh6bONjXd0Semezdg8hKlXZV21mqWKEUT5i6q9kbCEtgQ84DzV9j5qhpwgi_T3SDGlB5j34uI3sjuF9lLa8OBG4yQZDwM4hCAfRu9KXP019jxqG2g87KZ9XuxXoQEVXS5jc_EZuA6gIMZj8pnHgl_NOrXLDno5zBLaA78ib-i8RHTPZLvtDZYBdV2mbqnFgENQ9_DkCYCRNfmP66/https%3A%2F%2Furlsand.esvalabs.com%2F%3Fu%3Dhttp%253A%252F%252Fgradsusr.org%252Fmailman%252Flistinfo%252Fgradsusr%26e%3D7312d17f%26h%3D5ee7e1e3%26f%3Dy%26p%3Dn>
>
>
> ------------------------------
>
> End of gradsusr Digest, Vol 145, Issue 2
> ****************************************
>
>
>
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
>
> https://urlsand.esvalabs.com/?u=http%3A%2F%2Fgradsusr.org%2Fmailman%2Flistinfo%2Fgradsusr&e=7312d17f&h=5ee7e1e3&f=y&p=n
>
> --
> Thanks
> Mark Sponsler
> _______________________________________________
> gradsusr mailing list
> gradsusr at gradsusr.org
> http://gradsusr.org/mailman/listinfo/gradsusr
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gradsusr.org/pipermail/gradsusr/attachments/20220412/fd7401a7/attachment-0001.html>