Re: [mrtg] Large Master Config Vulnerability

#1 (**permalink**) 04-17-2008

This is a multi-part message in MIME format.

--===============2054609588==
Content-class: urn:content-classes:message
Content-Type: multipart/alternative;
boundary="----_=_NextPart_001_01C8A0D3.5D3EA484"

This is a multi-part message in MIME format.

------_=_NextPart_001_01C8A0D3.5D3EA484
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

There are several things to try, some have already been mentioned. We
have >4500 targets being polled at the moment and so have had to do all
of them.

=20

1) Set lower SNMP timeout and retry options as other people have said

2) Use the Forks: option to create multiple polling threads. This does
not work in Windows, I think. You need to set a Forks level appropriate
for your system, which depends on the available memory and CPU.

3) Run multiple instances of MRTG by having more than one master.cfg
file. We actually do this by using a home-grown scheduler which builds
the master.cfg files on the fly and takes care of multithreading, and
also temporarily disabled a CFG file if the host/device is down.

4) Get a more powerful polling machine! We use a 2x3GHz Xeon with 6Gb
memory.

5) Split your MRTG over multiple servers. You can get an integrated
frontend if you use the distributed MRTG feature in routers2=20

6) Get faster disks. MRTG also bottlenecks on disk IO, so faster disks
can help the processing finish sooner. We installed an Adaptec SAS
array with multiple spindles.

7) Upgrade MRTG and RRDTool to the latest versions. Apparently they can
handle errors and IO better now.

=20

Steve

=20

=20

________________________________

From: mrtg-bounces@lists.oetiker.ch
[mailto:mrtg-bounces@lists.oetiker.ch] On Behalf Of Brad Lodgen
Sent: Friday, 18 April 2008 04:39
To: mrtg@lists.oetiker.ch
Subject: [mrtg] Large Master Config Vulnerability

=20

Hi everyone,

I'm running a master config with hundreds of include lines and thousands
of targets. This type of setup is vulnerable to errors in config files
and/or changes made in the field not being immediately updated within
the configs. If there are a few errors or changes out in the field to
ports causing them to become 'unpollable', it causes the MRTG polling
interval to go over five minutes because it's retrying those interfaces.
At the moment, with only about 30 error lines in my log(equating to
about 15 interfaces/targets), it's causing MRTG to take 7-9 minutes to
complete polling. As this is a very small percentage compared to the
total amount of targets being polled, I'm trying to figure out a way to
get around this, if possible, or at least to minimize the effects.

Is anyone else running a system like this or does anyone have
suggestions to try?

Thanks in advance for any help!
Brad

------_=_NextPart_001_01C8A0D3.5D3EA484
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">

<style>

</style>

</head>

<body lang=3DEN-GB link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

There are several things to try, =
some have
already been mentioned.  We have >4500 targets being polled at =
the moment
and so have had to do all of them.<o:p></o:p>

<o:p> </o:p>

1) Set lower SNMP timeout and retry
options as other people have said<o:p></o:p>

2) Use the Forks: option to create
multiple polling threads.  This does not work in Windows, I =
think.  You need to
set a Forks level appropriate for your system, which depends on the =
available
memory and CPU.<o:p></o:p>

3) Run multiple instances of MRTG =
by
having more than one master.cfg file.  We actually do this by using =
a
home-grown scheduler which builds the master.cfg files on the fly and =
takes
care of multithreading, and also temporarily disabled a CFG file if the =
host/device
is down.<o:p></o:p>

4) Get a more powerful polling =
machine! 
We use a 2x3GHz Xeon with 6Gb memory.<o:p></o:p>

5) Split your MRTG over multiple =
servers. 
You can get an integrated frontend if you use the distributed MRTG =
feature in
routers2 <o:p></o:p>

6) Get faster disks.  MRTG =
also
bottlenecks on disk IO, so faster disks can help the processing finish =
sooner. 
We installed an Adaptec SAS array with multiple =
spindles.<o:p></o:p>

7) Upgrade MRTG and RRDTool to the =
latest
versions.  Apparently they can handle errors and IO better =
now.<o:p></o:p>

<o:p> </o:p>

Steve<o:p></o:p>

<o:p> </o:p>

<o:p> </o:p>

<div style=3D'border:none;border-left:solid blue 1.5pt;padding:0cm 0cm =
0cm 4.0pt'>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</div>

From:</spa=
n>
mrtg-bounces@lists.oetiker.ch [mailto:mrtg-bounces@lists.oetiker.ch] =
On Behalf Of Brad Lodgen 
Sent: Friday, 18 April =
2008 04:39 
To: =
mrtg@lists.oetiker.ch 
Subject: [mrtg] Large =
Master
Config Vulnerability<o:p></o:p>

</div>

<o:p> </o:p>

Hi everyone, 
 
I'm running a master config with hundreds of include lines and thousands =
of
targets. This type of setup is vulnerable to errors in config files =
and/or
changes made in the field not being immediately updated within the =
configs. If
there are a few errors or changes out in the field to ports causing them =
to
become 'unpollable', it causes the MRTG polling interval to go over five
minutes because it's retrying those interfaces. At the moment, with only =
about
30 error lines in my log(equating to about 15 interfaces/targets), it's =
causing
MRTG to take 7-9 minutes to complete polling. As this is a very small
percentage compared to the total amount of targets being polled, I'm =
trying to
figure out a way to get around this, if possible, or at least to =
minimize the
effects. 
 
Is anyone else running a system like this or does anyone have =
suggestions to
try? 
 
Thanks in advance for any help! 
Brad<o:p></o:p>

</div>

</div>

</body>

</html>

------_=_NextPart_001_01C8A0D3.5D3EA484--

--===============2054609588==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
mrtg mailing list
mrtg@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/mrtg

--===============2054609588==--

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode