[rrd-users] Re: bogus spikes with Infiniband traffic

On Mon, Oct 23, 2006 at 03:47:58PM -0400, Andy Riebs wrote:
> Our Voltaire Infiniband switches (and perhaps others?) don't wrap their
> traffic counters back to zero when they hit their maximum value.
> Instead, they simply report "overflow" until a procedure is run to reset
> them to zero.
>
> On one of our mid-sized clusters, a handful of port counters are already
> overflowing after a couple of hours, and more enter that state as the
> day goes on.
>
> To keep useful statistics generally available, we have a procedure that
> resets the counters to zero once a day. Unfortunately, when we reset the
> counters, rrdtool assumes that the counters have wrapped, and infers
> very large traffic bursts through little-used ports.
>
> I've been contemplating adding an `rrdtool reset` command that would
> replace the last_ds value and timestamp with specified values, but would
> prefer to learn that a better way to handle this already exists :)

When you reset your counters, make sure RRDtool knows about this.

Just before the reset, read the counters and feed this known data
to rrdtool.

Then reset the counter on your switch, and tell rrdtool the input
is unknown (rrdtool update yourfile.rrd N:U)

Then either:
-1- As soon as possible, read the counters and feed this known data
to rrdtool.
-2- make sure to know the exact reset time, and perform an update:
"rrdtool update yourfile.rrd <timstamp>:0
(assuming the counters start at zero; they probably will!)

Either way you will loose some data. The interval between the last update
before your reset, upto the interval following the first one after the reset,
will be lost.

When doing this procedure, make sure timestamps are not the same. RRDtool
will complain if you aren't taking care of this.

HTH
--
Alex van den Bogaerdt
http://www.vandenbogaerdt.nl/rrdtool/

--
Unsubscribe mailto:rrd-users-request@list.ee.ethz.ch?subject=unsubscribe
Help mailto:rrd-users-request@list.ee.ethz.ch?subject=help
Archive http://lists.ee.ethz.ch/rrd-users
WebAdmin http://lists.ee.ethz.ch/lsg2.cgi