joe-job sucking up TCP sockets?

My site is getting joe-jobbed, fending it off ok.
Problem is that my linux mail server is taking some of the incoming
SMTP traffic (gateways are really busy) and seems to be struggling
with providing TCP sockets for HTTPD (Webmail), POP3 or SSHD.

Any advice? It takes 2-6 times for HTTPD to actually answer a
request. The HTTPD log confirms that the daemon isn't getting the
request initially. SSHD takes 3-4 attempts to connect.

Box is running at 0.45 usage, plenty of mem, daemons are running
properly, just cannot reach them. Any configuration ideas? I've
tried many IPtables rules, but they haven't helped so far. The
incoming flood is from unique IP addresses, real or spoofed seems to
be mostly real as the victims bounce the message.

isp@derdev.com (Dave Richardson) writes:

> My site is getting joe-jobbed, fending it off ok. Problem is that my
> linux mail server is taking some of the incoming SMTP traffic (gateways
> are really busy) and seems to be struggling with providing TCP sockets
> for HTTPD (Webmail), POP3 or SSHD.

Hmmm.

> Any advice? It takes 2-6 times for HTTPD to actually answer a request.
> The HTTPD log confirms that the daemon isn't getting the request
> initially. SSHD takes 3-4 attempts to connect.

What's in `dmesg`? What does iptraf show? How about netstat output - group
things by status with e.g.:

| zsh, trough 1:08PM piglet/ % netstat -an | awk '/tcp/ {print $NF}' | \
| sort | uniq -c | sort -n
|
| 1 CLOSE_WAIT
| 2 CLOSED
| 6 ESTABLISHED
| 8 LISTEN

see if you can pick up a pattern there.

What sysctls do you have? How about these? :

| net.ipv4.icmp_ratelimit = 100
| net.ipv4.icmp_ignore_bogus_error_responses = 1
| net.ipv4.icmp_echo_ignore_broadcasts = 1
| net.ipv4.icmp_echo_ignore_all = 0
|
| net.ipv4.tcp_syncookies = 1
|
| net.ipv4.tcp_keepalive_intvl = 75
| net.ipv4.tcp_keepalive_probes = 3
| net.ipv4.tcp_keepalive_time = 600

The latter 4 should be particularly useful if you have lots of netstat
output in clogged-up states (CLOSE_WAIT, FIN_WAIT, FIN_WAIT2 in
particular).

> Box is running at 0.45 usage, plenty of mem, daemons are running
> properly, just cannot reach them. Any configuration ideas? I've tried
> many IPtables rules, but they haven't helped so far. The incoming flood
> is from unique IP addresses, real or spoofed seems to be mostly real as
> the victims bounce the message.

Can you not limit your mail-server a bit better? For example, run it out of
xinetd, thus:

| service smtp
| {
| socket_type = stream
| protocol = tcp
| wait = no
| user = mail
| server = /usr/sbin/exim
| server_args = -bs
| instances = 10
| per_source = 3
| cps = 5 10
| max_load = 3.5
| rlimit_cpu = 40
| no_access = spammerip/netblock...
| }

and/or some appropriate options in your config file?

You should really be trying to block these things as early as possible -
maybe put an overall rate-limiter on new incoming connections to port
25/tcp if that's where the problems are coming from; try something like

| iptables -A INPUT -p tcp --dport 25 -m limit \
| --limit 1/sec --limit-burst 5/sec \
| -j ACCEPT
| iptables -A INPUT -p tcp --dport 25 -j LOG --log-prefix "spam-limited"
| iptables -A INPUT -p tcp --dport 25 -j DROP

see if that helps. You could theoretically clone that block for other
port#s, of course.

~Tim
--
13:11:41 up 174 days, 15:31, 4 users, load average: 0.01, 0.05, 0.05
piglet@stirfried.vegetable.org.uk |Arise soul
http://spodzone.org.uk/cesspit/ |Soar above the singing river

> My site is getting joe-jobbed, fending it off ok.
> Problem is that my linux mail server is taking some of the incoming
> SMTP traffic (gateways are really busy) and seems to be struggling
> with providing TCP sockets for HTTPD (Webmail), POP3 or SSHD.
>
> Any advice? It takes 2-6 times for HTTPD to actually answer a
> request. The HTTPD log confirms that the daemon isn't getting the
> request initially. SSHD takes 3-4 attempts to connect.
>
> Box is running at 0.45 usage, plenty of mem, daemons are running
> properly, just cannot reach them. Any configuration ideas? I've
> tried many IPtables rules, but they haven't helped so far. The
> incoming flood is from unique IP addresses, real or spoofed seems to
> be mostly real as the victims bounce the message.

The stalling on TCP connections sounds unusual. How many simultaneous
connections do you see with netstat -n ? Using kernel 2.4.x my 200 MHz
Pentium system has been able to support hundreds at once.

Could you be out of bandwidth? What kind of network connection do you have?
If it's asymmetric, remember that downstream can also impact maximum
upstream throughput.

--
Jem Berkes
http://www.sysdesign.ca/