[AMaViS-user] Scalable amavisd-new installation (2): database

[part 2 of: "Scalable amavisd-new installation"]

We are right now preparing a few new virtual servers for our switch to
amavisd-new 2.6. Main reason: database partitioning. Last time we tried
out the cleanup procedures it took four full days (as there has been
some issue with one of the slaves) until finished on the last slave
(it took something like 5-6 hours on our master).

As we have seen that partitioning for amavis was on it's way we decided
to let the database grow and to spend no more time for garbage collection.
This should last unless we are ready for migration to a partitioned setup.
Disks are filling up, 2.6.0 is stable - so the right time is... NOW ;-)

We are running two amavisd-new vServers, using a common replicated MySQL
"cluster" (as we are still really sceptic about MySQL NBD). Partitioning
would solve our garbage collection issues - but for performance and sca-
labilty reasons I would like to run one dedicated (and replicated, for
backup / failover) DB instance for each amavisd-new instance.

It should look like this (planned setup):

------------- --------- ---------
| Filter F1 | <=> | F1DB1 | => | F1DB2 |
------------- --------- ---------

------------- --------- ---------
| Filter F2 | <=> | F2DB1 | => | F2DB2 |
------------- --------- ---------

Instead of (current setup):

------------- --------- ---------
| Filter F1 | <=> | | => | DB2 |
------------- | | ---------
| DB 1 |
------------- | | ---------
| Filter F2 | <=> | | => | DB3 |
------------- --------- ---------

(DB3 is/was there to allow us to create a snapshot and add another DB
server once needed without the need to take one of the two primary DB
servers down)

So why split DB? I would loose the benefit of a centralized DB wouldn't
I? Some reasons making me think that it could be a good idea are:

- As each server has to handle all queries in a replicated setup it
could once be the bottleneck (even if it's running fine today)
- A Master-Master Setup would probably produce a lot more rollbacks
- Some servers for reads and one for writes IMO doesn't scale and does
absolutely make no sense for such a DB scheme
- I could use MySQL 5.1 for F1DB and PostgreSQL 8.3 for F2DB, this would
make things even more complicated - but doing so I would give me a
real-world comparsion I can trust for farther "strategic decisions"

What I'm concerned about:

- Penpals. Not using them right now, so no experience on my side. But
I want to use them in the near future - so what impact would such a
setup have?

- Other concerns: none right now. I would put just maddr, msgs, msgrcpt
and quarantine in this DBs, nothing else. There will be a separate
Master-Master replication for my "config DB", used by Postfix, future
back-end mail servers and Amavis (policy...)

- MySQL (5.1) or PostgreSQL (8.3)? Hehe... Let me know your opinion ;-)

- Official Linux distributions == old packages. We are running Debian
Etch on most of our (v)hosts, and we will add some Lenny this weeks
for some services of our mailfilter cluster. We are used to package
software we need for our servers - but obviously it is easier to
just "apt-get something". Debian-volatile allowed us to stay with
Etch 'til now - but this is not an option anymore. I'm really in-
terested to hear how you are handling this problem (requirement for
new packages / "outdated" distros) in your environments?!


Other questions:

- Is there some better way to achieve a VERY scalable setup I didn't
think about?

Kind regards,
Thomas Gelf


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757...un.com/javaone
_______________________________________________
AMaViS-user mailing list
AMaViS-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/...fo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/