[rrd-users] rrdtool, python and threads

permalink · #1 (**permalink**) 04-22-2008

This is a multi-part message in MIME format.

--===============0145389486==
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_0018_01C8A4CE.A4D3C7A0"

This is a multi-part message in MIME format.

------=_NextPart_000_0018_01C8A4CE.A4D3C7A0
Content-Type: text/plain;
charset="US-ASCII"
Content-Transfer-Encoding: 7bit

Hello!

I'm completely new to rrdtool AND mailing lists, so please bear with me.

Since I don't know whether this has been answered before, and I don't know
how (or if) I can search the mailing list archives, I'm posting a question.

Here's my situation:

I have a series of resources I'm monitoring (nothing to do with traffic
etc). I've already figured out what DS and RRA I need. I'm planning on
updating the relative RRDatabases (1 file per resource) via a python script.
So far, I've created a bash script that calls the same python script over
and over again with different command line arguments each time, so that I
collect data from each resource, i.e. my bash script is like this:

#!/bin/bash

python get_the_resource.py resource1

python get_the_resource.py resource2

python get_the_resource.py resource3

python get_the_resource.py resource4

..

The python script itself is quite I/O intensive: The whole bash script takes
about 15 seconds to run for 6 resources monitored. I've also tried one
single resource (i.e. running one of the above lines on its own) and saw
that it took consistently about 2 seconds, most of which was used up merely
for fetching data from the web.

So I'm guessing that if I make a threaded python script that sends each
request in its own thread, I'll have a more effective program.

Here's where it gets hairy: as I said, I also want to record this data in an
RRDatabase. I've read on the site that if I use librrdtool in threaded
programs, I need to use different functions. However, That library is for C
programs (I think). Does the same hold for python scripts as well? Do I have
to use the *_r functions? Maybe it's not noteworthy, but I should mention
again that each resource has its own rrd (because more resources to be
monitored will be added in the future, and I haven't seen a way to add a DS
after rrdcreate).

Thanks in advance

Yiannis Vavouranakis

------=_NextPart_000_0018_01C8A4CE.A4D3C7A0
Content-Type: text/html;
charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable

<html xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<style>

</style>

</head>

<body lang=3DEL link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

Hello!<o:p></o:p>

<o:p> </o:p>

I’m completely new to rrdtool AND =
mailing
lists, so please bear with me…<o:p></o:p>

<o:p> </o:p>

Since I don’t know whether this has been
answered before, and I don’t know how (or if) I can search the =
mailing
list archives, I’m posting a question. =
<o:p></o:p>

<o:p> </o:p>

Here’s my =
situation:<o:p></o:p>

<o:p> </o:p>

I have a series of resources I’m =
monitoring
(nothing to do with traffic etc). I’ve already figured out what DS =
and
RRA I need. I’m planning on updating the relative RRDatabases (1 =
file per
resource) via a python script. So far, I’ve created a bash script =
that
calls the same python script over and over again with different command =
line
arguments each time, so that I collect data from each resource, i.e. my =
bash
script is like this:<o:p></o:p>

<o:p> </o:p>

#!/bin/bash<o:p></o:p>

python get_the_resource.py =
resource1<o:p></o:p>

python get_the_resource.py =
resource2<o:p></o:p>

python get_the_resource.py =
resource3<o:p></o:p>

python get_the_resource.py =
resource4<o:p></o:p>

…<o:p></o:p>

<o:p> </o:p>

The python script itself is quite I/O =
intensive: The
whole bash script takes about 15 seconds to run for 6 resources =
monitored.
I’ve also tried one single resource (i.e. running one of the above =
lines
on its own) and saw that it took consistently about 2 seconds, most of =
which
was used up merely for fetching data from the =
web.<o:p></o:p>

<o:p> </o:p>

So I’m guessing that if I make a =
threaded
python script that sends each request in its own thread, I’ll have =
a more
effective program.<o:p></o:p>

<o:p> </o:p>

Here’s where it gets hairy: as I said, I =
also
want to record this data in an RRDatabase. I’ve read on the site =
that if
I use librrdtool in threaded programs, I need to use different =
functions.
However, That library is for C programs (I think). Does the same hold =
for
python scripts as well? Do I have to use the *_r functions? Maybe =
it’s
not noteworthy, but I should mention again that each resource has its =
own rrd
(because more resources to be monitored will be added in the future, and =
I
haven’t seen a way to add a DS after =
rrdcreate).<o:p></o:p>

<o:p> </o:p>

Thanks in advance<o:p></o:p>

<o:p> </o:p>

Yiannis =
Vavouranakis<o:p></o:p>

<o:p> </o:p>

</div>

</body>

</html>

------=_NextPart_000_0018_01C8A4CE.A4D3C7A0--

--===============0145389486==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

--===============0145389486==--

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode