Help! Outbound rate has slowed to a crawl ...

Hi All!

We are running postfix 2.0.16-3.rh8 primarily as an
outbound relay for sending approx 50-100K
messages/day. Our outbound delivery rate is usually
around 1000 messages/minute (at peak times). After
months of steady reliable throughput, our outbound
became very constipated today. Rate has dropped to
~60 messages/min and we have a queue that has crept up
to 36K messages over the course of the day. Other
details:

1) 16050 of the remaining 17646 requests are flagged
as "(delivery temporarily suspended: unknown mail
transport error)" (when grepping thru mailq output).
Restarting postfix reduced this number to 1510 (which
will presumably grow)
2) The vast majority of the remaining messages are now
oscillating between active & deferred status at long
intervals (postfix backoff/retry cycles, I assume)
3) As mentioned, when the active queue is full, the
outbound rate is extremely slow (~60 as noted above)
4) we have 50 smtp daemons (default) delivering
outbound
5) this server receives very little inbound mail (it's
a designated outbound)
6) the backed up mail requests are being sent to
multiple different domains (see qshape output below)
7) the server is not on a ton of blacklists (only one,
according to dnsstuff.com)
8) the box's memory is pretty topped out, but this is
probably to be expected given the other symptoms

Can anyone offer some troubleshooting tips? DNS
lookups seem to be working fine, as some messages are
being delivered (see example at bottom), but I'm not
sure if my periodic nslookups/etc. would reveal any
intermittent problems ...

Thanks!

Andy


[root@euphrates log]# /usr/local/bin/qshape.pl active
| head -20
T 5 10 20 40
80 160 320 640 1280 1280+
TOTAL 14470 3 2 0 0
0 1 1 12703 489 1271
yahoo.com 556 0 0 0 0
0 0 0 553 3 0
aol.com 526 2 0 0 0
0 0 0 524 0 0
earthlink.net 259 0 0 0 0
0 0 0 259 0 0
msn.com 159 0 0 0 0
0 0 0 159 0 0
home.com 143 0 0 0 0
0 0 0 143 0 0
comcast.net 119 0 1 0 0
0 0 0 118 0 0
mindspring.com 117 0 0 0 0
0 0 0 117 0 0
juno.com 113 0 0 0 0
0 0 0 113 0 0
excite.com 80 0 0 0 0
0 0 0 80 0 0
nc.rr.com 77 0 0 0 0
0 0 0 63 14 0
bellsouth.net 77 0 0 0 0
0 0 0 77 0 0
attbi.com 76 0 0 0 0
0 0 0 76 0 0
worldnet.att.net 72 0 0 0 0
0 0 0 72 0 0
usa.net 66 0 0 0 0
0 0 0 66 0 0
nyc.rr.com 65 0 0 0 0
0 0 0 22 40 3
att.net 61 0 0 0 0
0 0 0 61 0 0
cs.com 56 0 0 0 0
0 0 0 56 0 0
sympatico.ca 55 0 0 0 0
0 0 0 55 0 0




Jan 31 21:45:21 euphrates postfix/smtp[17936]:
8A402E022C: to=<xxxxx@xxxx.net>,
relay=mx07.xxxx.net[207.69.200.133], delay=27068,
status=sent (250 1cVQQN4ab3Nl3qB0 Message accepted for
delivery)
Jan 31 21:45:21 euphrates postfix/qmgr[17845]:
8A402E022C: removed




__________________________________
Do you Yahoo!?
Yahoo! Mail - 250MB free storage. Do more. Manage less.
http://info.mail.yahoo.com/mail_250