How to archive mails relayed by postfix?

I'm almost beginner on Postfix and Amavis and I just learnt it for a
few weeks. I searched through the google for the workaround for days
with no luck.

The architecture I would like to accomplish may sound silly, but it's
what I would like to have, as follows. Basically the idea is to have
Postfix as a mail gateway for anti-virus, anti-spam AND also mail
archiver for the existing Exchange server on the dumb Windows.

Simple view:
External -> Posfix -> Exchange

Detail view:
External
|
Posfix smtpd
|
Amavisd-new
\___ ClamAV and SpamAssassin
/
|
Postfix qmgr ----- Postfix local
|
Postfix relay
|
Exchange

That means I would like to have two copies of each mail: When a mail
is received by the Postfix, it is scanned through Amavis. Then, the
mail would "tee" into two copies and then deliver to both local and
also relay to the Exchange server.

My current settings follows those guides and FAQs and forum posts
everywhere on google and I have the following works:

External
|
Posfix smtpd
|
Amavisd-new
\___ ClamAV and SpamAssassin
/
|
Postfix qmgr
|
Postfix relay
|
Exchange

(Although currently the SpamAssassin lets all mail pass and cannot
distinguish spam yet.)



Please advise if there is some workarounds or the scenario is totally
stupid. Thanks for all of your help.

~ray.

Ray.SWC@gmail.com wrote:
> The architecture I would like to accomplish may sound silly, but it's
> what I would like to have, as follows. Basically the idea is to have
> Postfix as a mail gateway for anti-virus, anti-spam AND also mail
> archiver for the existing Exchange server on the dumb Windows.
>
You can do all that, though there is one snag that will need a workround.

> Simple view:
> External -> Posfix -> Exchange
>
> Detail view:
> External
> |
> Posfix smtpd
> |
> Amavisd-new
> \___ ClamAV and SpamAssassin
> /
> |
> Postfix qmgr ----- Postfix local
> |
> Postfix relay
> |
> Exchange
>
> That means I would like to have two copies of each mail: When a mail
> is received by the Postfix, it is scanned through Amavis. Then, the
> mail would "tee" into two copies and then deliver to both local and
> also relay to the Exchange server.
>
Simple.

Use "always_bcc" to send a copy of the mail to a special archive
mailbox. You'll need to provide some mechanism to deal with the mail
when it arrives in the mailbox.

I'm currently using procmail and a self developed shell script to store
mail in a set of mbox files in a directory structure: archive/yyyy/mbox
where yyyy is the year when the mail was sent. Mail will be discarded if
the mbox file hits the size defined by "mailbox_size_limit", so the
script monitors the mailbox size and renames when its approaching the
max size, so a set of files (mbox, mbox.1, mbox.2, ....) are built up in
the year directory.

I've recently written a database-based archiving system which has just
been loaded with the last three years' worth of archived mail and should
be in full time use by the end of this week following a minor tweak or
two. It indexes the mail and allows searches on any combination of
address, subject, date range and (last resort) text search of the
message's plain text part. It should be portable as its fairly
database-independent and written in Java. I'm using PostgreSQL as the
database, but anything with a JDBC driver that has a sequence generator
and can handle CLOB fields should work, i.e. I think Derby and MySQL
would be OK too.

> My current settings follows those guides and FAQs and forum posts
> everywhere on google and I have the following works:
>
The problem I mentioned is that "always_bcc" copies every message that
hits qmgr, so when I ran Spamassassin as a Postfix controlled service
two copies of each message got sent to the archive (one as the message
was received, the second as it was re-injected after being inspected by
Spamassassin.

I solved the problem by adapting my mail flow:

ISP --> fetchmail | spamc | sendmail --> Postfix --> the archive
|
v
dovecot --> users

Another approach would be for the archiving system to discard all
messages that don't contain the X-Spam-Status header. Spamassassin adds
this to every message it processes, so this mechanism would only archive
messages that have been looked at by Spamassassin.

My database archiver filters its input anyway to avoid archiving spam.
It discards:
- mail marked as spam
- mail that was retrieved from the archive and returned to the
search user
- mail whose sender domain doesn't exist (this traps some spam that
Spamassassin misses - notably 419 and the better constructed
phishing scams.

> (Although currently the SpamAssassin lets all mail pass and cannot
> distinguish spam yet.)
>
That's done by design. If you want to filter spam out of the stream
rather than using rules in mail clients to put it in a Spam mailbox
you'll have to write the filter yourself. Its not a totally trivial task
because you'll need to work out how to reliably handle false positives.

My to do list includes two enhancements:
- a program that sits downstream of spamc and filters out all messages
that Spamassassin has marked as spam
- a local rule for Spamassassin that forces mail from people in the
archive to be accepted. This should stop my filter from discarding
(very rare) false positives. So far I've only had mail from one(!)
correspondent that was flagged as spam, so the local rule is low
priority for me.

> Please advise if there is some workarounds or the scenario is totally
> stupid. Thanks for all of your help.
>
Sounds like a good plot to me.


--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |

On Jan 8, 9:51 pm, Martin Gregorie <mar...@see.sig.for.address> wrote:
> Ray....@gmail.com wrote:
> > The architecture I would like to accomplish may sound silly, but it's
> > what I would like to have, as follows. Basically the idea is to have
> > Postfix as a mail gateway for anti-virus, anti-spam AND also mail
> > archiver for the existing Exchange server on the dumb Windows.
>
> You can do all that, though there is one snag that will need a workround.
>
>
>
> > Simple view:
> > External -> Posfix -> Exchange
>
> > Detail view:
> > External
> > |
> > Posfix smtpd
> > |
> > Amavisd-new
> > \___ ClamAV and SpamAssassin
> > /
> > |
> > Postfix qmgr ----- Postfix local
> > |
> > Postfix relay
> > |
> > Exchange
>
> > That means I would like to have two copies of each mail: When a mail
> > is received by the Postfix, it is scanned through Amavis. Then, the
> > mail would "tee" into two copies and then deliver to both local and
> > also relay to the Exchange server.
>
> Simple.
>
> Use "always_bcc" to send a copy of the mail to a special archive
> mailbox. You'll need to provide some mechanism to deal with the mail
> when it arrives in the mailbox.
>
> I'm currently using procmail and a self developed shell script to store
> mail in a set of mbox files in a directory structure: archive/yyyy/mbox
> where yyyy is the year when the mail was sent. Mail will be discarded if
> the mbox file hits the size defined by "mailbox_size_limit", so the
> script monitors the mailbox size and renames when its approaching the
> max size, so a set of files (mbox, mbox.1, mbox.2, ....) are built up in
> the year directory.
>
> I've recently written a database-based archiving system which has just
> been loaded with the last three years' worth of archived mail and should
> be in full time use by the end of this week following a minor tweak or
> two. It indexes the mail and allows searches on any combination of
> address, subject, date range and (last resort) text search of the
> message's plain text part. It should be portable as its fairly
> database-independent and written in Java. I'm using PostgreSQL as the
> database, but anything with a JDBC driver that has a sequence generator
> and can handle CLOB fields should work, i.e. I think Derby and MySQL
> would be OK too.
>
> > My current settings follows those guides and FAQs and forum posts
> > everywhere on google and I have the following works:
>
> The problem I mentioned is that "always_bcc" copies every message that
> hits qmgr, so when I ran Spamassassin as a Postfix controlled service
> two copies of each message got sent to the archive (one as the message
> was received, the second as it was re-injected after being inspected by
> Spamassassin.
>
> I solved the problem by adapting my mail flow:
>
> ISP --> fetchmail | spamc | sendmail --> Postfix --> the archive
> |
> v
> dovecot --> users
>
> Another approach would be for the archiving system to discard all
> messages that don't contain the X-Spam-Status header. Spamassassin adds
> this to every message it processes, so this mechanism would only archive
> messages that have been looked at by Spamassassin.
>
> My database archiver filters its input anyway to avoid archiving spam.
> It discards:
> - mail marked as spam
> - mail that was retrieved from the archive and returned to the
> search user
> - mail whose sender domain doesn't exist (this traps some spam that
> Spamassassin misses - notably 419 and the better constructed
> phishing scams.
>
> > (Although currently the SpamAssassin lets all mail pass and cannot
> > distinguish spam yet.)
>
> That's done by design. If you want to filter spam out of the stream
> rather than using rules in mail clients to put it in a Spam mailbox
> you'll have to write the filter yourself. Its not a totally trivial task
> because you'll need to work out how to reliably handle false positives.
>
> My to do list includes two enhancements:
> - a program that sits downstream of spamc and filters out all messages
> that Spamassassin has marked as spam
> - a local rule for Spamassassin that forces mail from people in the
> archive to be accepted. This should stop my filter from discarding
> (very rare) false positives. So far I've only had mail from one(!)
> correspondent that was flagged as spam, so the local rule is low
> priority for me.
>
> > Please advise if there is some workarounds or the scenario is totally
> > stupid. Thanks for all of your help.
>
> Sounds like a good plot to me.
>
> --
> martin@ | Martin Gregorie
> gregorie. | Essex, UK
> org |

Thanks very much for your reply. Yet, I'm sorry not quite understand
your advice actually.

Should I configure "always_bcc" parameter on Postfix main.cf so I can
get *at least* 1 copy of each mail in a particular address? If so, is
there any easy mechanism to distinguish the mails?

Maybe my previous mail is misleading. The whole picture I want is
this:
Have a single domain foobar.com,
1. Have the Exchange server running in the back, it stores user
mailboxes.
2. Have a SMTP gateway in front to do mail filtering (anti-virus +
anti-spam) and also *stores user mailboxes*.

It is very redundant actually, but this what I want to accomplish at
the moment. Up to now, the mail gateway of Postfix, Amavis,
SpamAssassin, ClamAV does work by letting mails passes through, but no
local delivery.

By your advice, I tried setting the *always_bcc* to a special account
called archive ("always_bcc=archive"). Then, use the transport map to
route it to deliver locally. And lastly force it to write to file /var/
mail/archive for that address.

It does the archive job, but it is not the concept of mailboxes. The
best would be Postfix duplicate each mail, deliver one locally and
relay another to the Exchange.

Thanks a lot again as this is the only response I get on the issue.

~ray.

~ray. wrote:
>
> Should I configure "always_bcc" parameter on Postfix main.cf so I can
> get *at least* 1 copy of each mail in a particular address? If so, is
> there any easy mechanism to distinguish the mails?
>
That's what I suggested. These directives are all in main.cf. I think
that "archive" will get two copies of each message, one as it arrived at
Postfix and the other after its been through amavis/Spamassassin. The
mechanism would be to discard any mail that hasn't been marked by
Spamassassin.

> Maybe my previous mail is misleading. The whole picture I want is
> this:
> Have a single domain foobar.com,
> 1. Have the Exchange server running in the back, it stores user
> mailboxes.
> 2. Have a SMTP gateway in front to do mail filtering (anti-virus +
> anti-spam) and also *stores user mailboxes*.
>
So Postfix front-ends Exchange. That's clear. You can do what you like
with the archive mailbox - keep it local to Postfix or ship it on to
Exchange and do your archiving there, but keeping it local gives access
to more tools, such as procmail, and better scripting facilities.

Why are you using Exchange? Are you using its groupware facilities as
well as using it as an MTA? If its purely an MTA you could dump it and
install Dovecot to distribute the mail. It supports both POP3 and IMAP
access.

> It is very redundant actually, but this what I want to accomplish at
> By your advice, I tried setting the *always_bcc* to a special account
> called archive ("always_bcc=archive"). Then, use the transport map to
> route it to deliver locally. And lastly force it to write to file /var/
> mail/archive for that address.
>
If you set up an "archive" user on the Linux system the mail will be
delivered to it with no further action on your part and you'll have a
place to store and process the archive. The archive stream will be
temporarily stored in /var/mail/archive and can be accesses with "mail",
"mutt" or any other Linux mail reader. Alternatively, you can create a
..forward script in the archive user which uses procmail to process each
message as it gets delivered to archive.

Set things up this way and then use mail or mutt to inspect the archive
mail stream and check that you're getting what I said you'd get.

> It does the archive job, but it is not the concept of mailboxes. The
> best would be Postfix duplicate each mail, deliver one locally and
> relay another to the Exchange.
>
I think you'll find that you can use the "relayhost" in main.cf to
forward all mail that can't be delivered locally to Exchange provided
you set up the appropriate virtual user mapping. It may be relatively
difficult to deliver the non-archive mail both locally and to Exchange:
thats a guess because I haven't tried anything like that.

I don't use any delivery maps, but have pointed "relayhost" at my ISP.
The result is that mail sent to my local Linux user logins is delivered
to them, mail sent to nonexistent user logins is rejected and everything
else is sent via the ISP.

The only extra sophistication is that I've used /etc/aliases to redirect
all mail to "root" to my usual login user so I get to read it with all
the other stuff.

> Thanks a lot again as this is the only response I get on the issue.
>
Mail archiving isn't exactly a hot topic - yet - but I have a feeling
that this will change for SMEs as Sarbanes-Oxley type regulations get
more pervasive.


--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |