[AMaViS-user] Pre-release of amavisd-new-2.4.2,

A pre-release of amavisd-new-2.4.2 is available at:

http://www.ijs.si/software/amavisd/a....2-pre1.tar.gz

The 2.4.2 will primarily be a bug fix release,
but adds two interesting new features.


SUMMARY OF CHANGES:

- new feature: "pen pals soft-whitelisting" lowers spam score of received
replies to a message previously sent by a local user to this address;
suggested by Aaron P. Martinez and Gary V;
- new features: added command line options to override certain configuration
settings from a config file, see below;
- documentation bug fixes;
- zoo decoder interface routine can now use utility unzoo(1) or zoo(1);
- changed SMTP status code 550 to 554 when rejecting mail contents;

More details in RELEASE_NOTES.


Here is a section on "pen pals soft-whitelisting" from the RELEASE_NOTES:

- new feature: "pen pals soft-whitelisting" lowers spam score of received
replies to a message previously sent by a local user to this address;

Pre-requisites:
* both the outgoing and the incoming mail must pass through amavisd;
* SQL logging must be enabled (@storage_sql_dsn) and records should
be kept for at least several days (85% of replied mail gets a reply
within a week, 60% within one working day);
* @mynetworks and @local_domains_maps must reflect reality;
* the information about client IP address must be available to amavisd,
i.e. Postfix XFORWARD protocol extension must be enabled, or AM.PDP+milter;
* configuration variable $penpals_bonus_score must be set to a positive
value (such as 1.0, increase after gaining some experience),
zero disables the feature and is a default);
* $sql_clause{'sel_penpals'} must contain a SELECT clause (which by default
it does, unless overridden by an old assignment in amavisd.conf);

How it works:
* SQL logging stores records about all mail messages processed by amavisd,
their sender, recipients, delivery status, mail contents (no changes
there, this feature was introduced with amavisd-new-2.3.0);
* when a message is received, a SQL lookup against a SQL logging database
is performed, looking for previous messages sent in reverse direction,
i.e. from a local user (which is now a recipient of the current mail)
to the address that is now the sender or the message being processed;
A SELECT clause in $sql_clause{'sel_penpals'} is used, which by default
only considers records of previous messages that were actually
delivered (not rejected, discarded or bounced), and were not infected.
SQL lookup returns a timestamp of the most recent such message (if any),
the difference (in seconds) between the current time and the timestamp
is the 'age' as used in the following formula;
* an exponential decay formula calculates score points to be deducted
from the SA score:
weight = 1 / 2^(age/penpals_halflife)
score_boost = -penpals_bonus_score * weight
i.e. penpals_bonus_score is multiplied by 1, 1/2, 1/4, 1/8, 1/16, ...
at age 0, 1*halflife, 2*halflife, 3*halflife, 4*halflife ...
weight is a continuous function of age (actually, in steps of one second);

* configuration variables:
$penpals_bonus_score ... a (positive) score value by which spam score
is lowered when sender is known to have previously received mail
from our local user from this mail system. Zero or undef disables
pen pals lookups in SQL tables msgs and msgrcpt, and is a default.
$penpals_halflife ... exponential decay time constant in seconds,
defaults to 7 days; pen pal bonus is halved for each halflife
period since the last mail sent by a local user to a current
message's sender;
$penpals_threshold ... SA score (prior to boosts) below which
pen pals lookups are not performed to save time, defaults to 1.0;
undef is considered lower than any score;

Caveats / exceptions to "pen pals soft-whitelisting":
* infected messages are exempted from pen pals checks;
* mail with (unadjusted) SA score below $penpals_threshold (1.0 by default)
is exempted from pen pals check to save time and lighten the load on SQL;
* non-delivery notifications have null sender address, so can not match
previous correspondence and can not receive a pen pal bonus;
* sender address matching local domains but coming from outside is
not trusted and is exempted from pen pals checks; this implies that
authenticated local users posting from outside to this MTA do not
benefit from pen pals scheme when sending to local recipient - the
cleanest solution is to separate MSA from MTA (which also avoids the
SpamAssassin SPF & DUL problem with trusted networks);
* messages from a local user to self are exempted from pen pals check;
* outgoing messages (i.e. to non-local recipients) are exempted from
pen pals checks to save some time and simplify reasoning (which reverse
mail transaction to trust?); assuming that local users rarely send
spammy mail, outgoing mail would rarely need help from pen pals checks;
* messages received from mailing list typically use ML bounce or admin
address (possibly VERPed) as the sending address, so they would not be
considered replies to postings to a mailing list from a local user;
* underlying assumption is that a local-domains sender address in mail
coming from inside can be trusted not to be faked; if this is not the case,
an internal user cooperating with a spamer can widen spam tolerance for
another internal user (but it probably does not pay off, too much trouble
for too little effect);
* if a spammer knows or can guess that a local user is frequently sending
mail to some address (e.g. a mailing list unprotected by DK or SPF), he
can gain few bonus score points by using such sending address in his spam;
* there may be multiple MTA+amavisd servers, but all must use the same
logging SQL database;
* forwarding is compatible with the pen pals scheme;

Testing:
* set $penpals_bonus_score initially to a low value such as 1 or 0.5
to avoid surprises;
* set $penpals_threshold to undef to perform pen pals checks regardless
of the score;
* at log level 2 (or higher) search the log for a string "penpals: "
(only shows on incoming mail sent by a non-local sender)

Based on a feature request by Aaron P. Martinez, thanks to Gary V for
suggestions and prompting and to Michael Scheidell and Richard Bishop
for feedback.


Mark


_______________________________________________
AMaViS-user mailing list
AMaViS-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/...fo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/