Re: [AMaViS-user] High memory - high latency and slow processing

Rene wrote:

> Hi All,

> I'm running Amavisd-new with clamav 0.83, Spam Assassin 3.0.2, razor2, pyzor
> 0.40 and dccproc

> My filter server running amavisd-new 2.2.0 have a problem : every once in a
> while (2-3 days), some process get stuck on high memory consumption and my
> server is then very slow on processing the emails.

> Out of all the amavisd-new process I run, a couple of them suddently get
> stuck :

> 10811 amavis 25 0 756m 668m 6416 R 99.9 16.6 10:38.81 amavisd
> 10818 amavis 25 0 756m 729m 6416 R 99.5 18.1 11:12.63 amavisd
> 10824 amavis 25 0 757m 738m 6416 R 97.2 18.3 10:36.59 amavisd
> 10810 amavis 25 0 757m 633m 6416 R 58.6 15.7 6:50.30 amavisd
> 10825 amavis 25 0 756m 625m 6416 R 43.9 15.5 5:46.50 amavisd

> (Notice the ~750 memory consumption)

> after they get stuck, my processing time goes from an average 1.5 seconds to
> 10-15 seconds :

> Aug 15 06:29:10 epalsws004 amavis[10815]: (10815-44) TIMING [total 32371 ms] -
> SMTP EHLO: 3 (0%), SMTP pre-MAIL: 1 (0%), lookup_sql: 5 (0%), SMTP
> pre-DATA-flush: 2 (0%), SMTP DATA: 54 (0%), body_hash: 1 (0%), lookup_sql: 5
> (0%), mime_decode: 12 (
> 0%), get-file-type1: 15 (0%), decompose_part: 1 (0%), parts_decode: 0 (0%),
> AV-scan-1: 67 (0%), spam-wb-list: 2 (0%), SA msg read: 1 (0%), SA parse: 3
> (0%), SA check: 31782 (98%), update_cache: 22 (0%), deal_with_mail_size: 1
> (0%), fwd-connect: 1
> 47 (0%), fwd-mail-from: 1 (0%), fwd-rcpt-to: 1 (0%), write-header: 4 (0%),
> fwd-data: 0 (0%), fwd-data-end: 43 (0%), fwd-rundown: 1 (0%), main_log_entry:
> 17 (0%), update_snmp: 1 (0%), unlink-1-files: 175 (1%), rundown: 4 (0%)


> I chek the process causing problem with lsof and strace. The process don't do
> anything. The email opened is always spam. Even with SA debugging or
> Amavisd-new debugging there is nothing wrong.

> If I try to stop amavis with amavisd stop, all the process will die BUT the
> one with high memory :
> PID 10825: sending SIGKILL in 0 s
> Waiting for the process to terminate: 10810, 10811, 10818, 10824, 10825

> I don't see anything in amavisd-agent :
> sysContact STR
> sysDescr STR amavisd-new-2.2.0 (20041102)
> sysLocation STR
> sysName STR filter.ibm.epals.com
> sysObjectID OID 1.3.6.1.4.1.15312.2.1
> sysServices INT 64
> sysUpTime Timeticks 118041 (0 days, 0:19:40.41)
> CacheAttempts 388 1183/h 100.0 % (CacheAttempts)
> CacheHits 18 55/h 4.6 % (CacheAttempts)
> CacheHitsSpamCheck 18 55/h 4.6 % (CacheAttempts)
> CacheHitsSpamMsgs 17 52/h --- % (ContentSpamMsgs)
> CacheHitsVirusCheck 14 43/h 3.6 % (CacheAttempts)
> CacheMisses 370 1128/h 95.4 % (CacheAttempts)
> ContentBadHdrMsgs 7 21/h 1.8 % (InMsgs)
> ContentCleanMsgs 377 1150/h 97.2 % (InMsgs)
> ContentVirusMsgs 4 12/h 1.0 % (InMsgs)
> InMsgs 388 1183/h 100.0 % (InMsgs)
> InMsgsRecips 495 1510/h 127.6 % (InMsgs)
> OpsDec 374 1141/h 96.4 % (InMsgs)
> OpsDecByArZip 2 6/h 0.5 % (InMsgs)
> OpsDecByArZipAttempt 6 18/h 1.5 % (InMsgs)
> OpsDecByLhaAttempt 4 12/h 1.0 % (InMsgs)
> OpsDecByMimeParser 388 1183/h 100.0 % (InMsgs)
> OpsDecByUUlibAttempt 336 1025/h 86.6 % (InMsgs)
> OpsDecByUnrarAttempt 4 12/h 1.0 % (InMsgs)
> OpsSpamCheck 366 1116/h 94.3 % (InMsgs)
> OpsSqlSelect 990 3019/h 200.0 % (InMsgsRecips)
> OpsVirusCheck 374 1141/h 96.4 % (InMsgs)
> OutForwMsgs 384 1171/h 100.0 % (OutMsgs)
> OutMsgs 384 1171/h 100.0 % (OutMsgs)
> OutMsgsDelivers 384 1171/h 100.0 % (OutMsgs)
> QuarMsgs 4 12/h 100.0 % (QuarMsgs)
> QuarVirusMsgs 4 12/h 100.0 % (QuarMsgs)
> OpsDecType-asc 336 1025/h 86.6 % (InMsgs)
> OpsDecType-html 278 848/h 71.6 % (InMsgs)
> OpsDecType-txt 22 67/h 5.7 % (InMsgs)
> OpsDecType-sgml 8 24/h 2.1 % (InMsgs)
> OpsDecType-exe.exe-ms 4 12/h 1.0 % (InMsgs)
> OpsDecType-dat 2 6/h 0.5 % (InMsgs)
> OpsDecType-zip 2 6/h 0.5 % (InMsgs)
> OpsDecType-gif 1 3/h 0.3 % (InMsgs)
> Worm.SomeFool.P 2 6/h 50.0 % (ContentVirusMsgs)
> Worm.Mytob.T-2 1 3/h 25.0 % (ContentVirusMsgs)
> Exploit.HTML.IFrame 1 3/h 25.0 % (ContentVirusMsgs)
> Worm.Mytob.AF 1 3/h 25.0 % (ContentVirusMsgs)



> If I restart it all and send the email that was stuck back into the filter
> server, it will stuck again...

> I had this problem on 2 filter servers. Now I updated the first one with all
> the newest perl modules, razor2, dcc and spamassassin 3.0.4. The filter
> server is really faster and the memory doesn't have high consumption but it
> still seems to get affected by some stuck email. Latency goest 10 time
> longer..

> I hope I gave enough informations...
> can anybody help ?
> thanks for all your advises.
> Rene Bouchard

Just some thoughts.
Are you using a lot of custom rule sets? Are you still using an old
(no longer provided) bigevel.cf? After a reboot, how much memory does
each process use, and how much free memory is left? Have you run
'spamassassin --lint' to check for errors in your rule sets?

Gary V



-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
_______________________________________________
AMaViS-user mailing list
AMaViS-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/...fo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/