[AMaViS-user] Scalable amavisd-new installation (3): log aggregation

[part 3 of: "Scalable amavisd-new installation"]

There is also a self-written component I'm really proud of: the log
aggregation system. I need to somehow offer detailed logs to our support
team and to some of our customers. This is impossible to achieve using
daily logfiles & grep as of performance reasons and as of privacy prob-
lems (simply grepping a customers domain could show him log lines he
shouldn't see: BCC etc).

Using the amavis database was also not an option as there are important
information missing (front mx is blocking most email traffic not shown
here) and performance can be bad on some types of searches, even if
assigning large amounts of memory to MySQL and trying to somehow opti-
mize table structure.

The only way of solving this seemed to be to create a central, "real-
time" log aggregation system - and that's what we have done. Syslog of
all servers is sent to a central syslog server, and there

a) written to daily rotated and compressed plain logfiles
b) piped to our aggregation software

Writing this software was not an easy task - and I've had to struggle a
lot with thing's I would have never thought about without seeing them
happening in real time. Log lines happening later very often happen to
appear earlier (multiple cores, network latencies...), SMTP sessions are
stopped / killed in various strange ways / states, the same mail can
pass amavis multiple times etc.

Right now it's running fine, even if there are some little issues I'm
aware of, that still need to be fixed. We are slowly leaking memory,
so we are monitoring the daemon's memory usage and restarting it from
time to time. The reason is known, but as it is running fine I never
forced myself to sit down and fix it - as we have a lot of other pro-
jects with higher priority :-(

I would really like to share this piece of software (as my very own
contribution to the community ;-). I still have to get my employers
permission for doing so - but I'm pretty confident that doing so would
be fine.

Some details about the software:

- daemon is written in PHP5 (DON'T laugh, it's well done and behaves
fine ;-)
- even if written in PHP (CLI) it is handling POSIX signals, writing to
syslog etc
- my first attempt was focussed on raw speed
- later I realized that no one would ever be able to debug this code,
so I've rewritten it from scratch with an object-oriented approach
- each mail is going to build a new object, each line add's itself to
this object - and creates additional sub-objects (recipients etc)
- "completed" mails are stored to a flat "manually partitioned"
database, object is destroyed
- database contains redundant information and many indexes - main goal
is speed - and I can proudly report that it is VERY fast
- no foreign keys, just stupid plain MyISAM
- unhandled mails / log lines are discarded, information is written to
syslog
- disconnects / lost connections are handled, unfinished mails written
to db and cleaned up
- right now there are parser components for postfix and amavis
- there is also a perdition parser, doing a similar job - and also allo-
wing us to still provide SMTP-after-POP to customers using this
service for historical reasons

For sure I've forgotten some important details, but you should have some
idea about the project right now. So please let me know:

- is it worth the effort starting a small OS project for this piece of
software?
- are there similar projects I should look at before doing so?
- are you (all of you!!) going to clean up and perfectionate my code
once opensourced - so that I can lay back and let you do my job?

That's all, folks ;o)

Cheers,
Thomas Gelf


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757...un.com/javaone
_______________________________________________
AMaViS-user mailing list
AMaViS-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/...fo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/