Solaris 9 and ip filter performance issues.

Hello mailinglist.

I was hoping that someone with ipfilter expertise might be able to give us
suggestions as to how we can improve our current natter box.

Setup:

supermicro (dual 1.09 Ghz, 2G ram) x86 running Solaris 9 Generic_112234-06.
Current ipfilter version is v3.3.22
gcc 3.2.2
It has two nics (iprb0 WAN, iprb1 LAN, both 100Base)
It runs squid on port 8080.

The purpose of this box is to NAT three main internal networks (192.168, 172.16,
10) to the net. Additionally, redirect port 80 to squid on port 8080.

There should be at least a thousand machines on the internal network.

The problem is, in particular in the morning, that we start seeing packet loss,
and high fluctuating latency with pings, to eventualy timeout (presumably when
packet loss is high enough). This appears to only occur on the internal nic. Not
the external nic. Even though the traffic should be similar. (depending on how
well squid cache performs). It doesn't die, or stop performing as far as I know,
it just deteriates.

If I issue "/etc/init.f/ipfboot reload" the problem is "cleared" for a short
while, until it gets worse again.

This can happen much faster if people have virus/trojans, or run p2p, but
perhaps that just highlights that it is a load issue.


The total output (nagios graphs etc) seems to be around 4Mb/s. There is more
room on the external link, not that we see any packet loss on that side, and ftp
transfers made on it from local shell can get a further 1-2Mb/s.


I would assume that the hardware should be able to handle this load?

I have been reading the lists, and various googling to find some tweaks, both
for Solaris, and for ipfilter.

We have tried replacing cable, and changing the switch port.

We also tried upgrading to ipfilter 4.1.3, with pfil. However it totally hangs
the machine within 30 seconds of turning ipf on. No panic, nor core, total hang.
Rebooted twice to confirm.

Rolled back to ipf 3.3.22, but this time had LARGE_NAT on. However that too did
not appear to make much difference.

(I did not uninstall pfil yet, it is still there and running).

We've thought about moving squid and DNS onto a second box if indeed it is just
that the hardware is not keeping up. Any other suggestions? Really wanted to try
a newer ipfil version, but with the lockups this is not possible.



=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Current tweaks and settings are:

/usr/sbin/ndd -set /dev/udp udp_smallest_anon_port 8192
/usr/sbin/ndd -set /dev/tcp tcp_smallest_anon_port 8192
/usr/sbin/ndd -set /dev/tcp tcp_fin_wait_2_flush_interval 67500

set tcp:tcp_conn_hash_size=8192
set ipf:fr_statemax=7000
set ipf:fr_statesize=10009
set ipf:fr_tcpidletimeout=172800
set ipf:fr_tcphalfclosed=7200

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
The ipf.conf:
block out quick on iprb0 proto icmp from any to any icmp-type unreach
block out quick on iprb0 proto icmp from any to any icmp-type echorep
block return-rst in quick proto tcp from any port = 139 to any
block in quick from any port = 139 to any

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
The ipnat.conf:
map iprb0 192.168.0.0/16 -> 0/32 proxy port ftp ftp/tcp
map iprb0 172.16.0.0/16 -> 0/32 proxy port ftp ftp/tcp
map iprb0 10.0.0.0/8 -> 0/32 proxy port ftp ftp/tcp

map iprb0 192.168.0.0/16 -> 0/32 portmap tcp/udp auto
map iprb0 172.16.0.0/16 -> 0/32 portmap tcp/udp auto
map iprb0 10.0.0.0/8 -> 0/32 portmap tcp/udp auto

map iprb0 192.168.0.0/16 -> 0/32
map iprb0 172.16.0.0/16 -> 0/32
map iprb0 10.0.0.0/8 -> 0/32

rdr iprb1 0.0.0.0/0 port 80 -> 192.168.1.197 port 8080 tcp


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Output from misc commands (please ask for any I don't know about!)

last pid: 5540; load averages: 1.46, 1.42, 1.43 18:33:38
34 processes: 33 sleeping, 1 on cpu
CPU states: 58.3% idle, 8.3% user, 33.3% kernel, 0.0% iowait, 0.0% swap
Memory: 2048M real, 1391M free, 507M swap in use, 3143M swap free

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
nat01:~# ipf -V
ipf: IP Filter: v3.3.22 (164)
Kernel: IP Filter: v3.3.22
Running: yes
Log Flags: 0 = none set
Default: pass all, Logging: available
Active list: 1
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

nat01:~# ipnat -l | wc -l
8884
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

nat01:~# ipnat -s
mapped in 1927769 out 2148089
added 93871 expired 85190
inuse 8681
rules 10
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

nat01:~# ipfstat
dropped packets: in 0 out 0
non-data packets: in 0 out 0
no-data packets: in 0 out 0
non-ip packets: in 0 out 0
bad packets: in 0 out 0
copied messages: in 0 out 1104135
input packets: blocked 1970 passed 3980297 nomatch 3120758 counted 0
short 0
output packets: blocked 867 passed 4142474 nomatch 3352257 counted 0 short 0
input packets logged: blocked 0 passed 0
output packets logged: blocked 0 passed 0
packets logged: input 0 output 0
log failures: input 0 output 0
fragment state(in): kept 0 lost 0
fragment state(out): kept 0 lost 0
packet state(in): kept 0 lost 0
packet state(out): kept 0 lost 0
ICMP replies: 0 TCP RSTs sent: 0
Result cache hits(in): 859541 (out): 790228
IN Pullups succeeded: 0 failed: 0
OUT Pullups succeeded: 705 failed: 0
Fastroute successes: 0 failures: 29
TCP cksum fails(in): 0 (out): 0
Packet log flags set: (0)
none
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

nat01:~# netstat -rn

Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ------ ---------
210.172.<ip>.<ip> 210.172.<ip>.<ip> U 1 32 iprb0
192.168.1.0 192.168.1.197 U 30 17 iprb1
192.168.0.0 192.168.1.1 UG 1 1248
172.16.0.0 192.168.1.254 UG 1 12
10.0.0.0 192.168.1.254 UG 1 15
224.0.0.0 192.168.1.197 U 1 0 iprb1
default 210.172.<ip>.<ip> UG 1 17642
127.0.0.1 127.0.0.1 UH 2 284 lo0
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

nat01:~# netstat -i
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue
lo0 8232 loopback localhost 341 0 341 0 0 0
iprb0 1500 natoutside natoutside 2184679 1239 1990641 11246 0 0
iprb1 1500 natinside nat01 2083821 1839 2442424 1095 0 0

IPv4 ipForwarding = 1 ipDefaultTTL = 255
ipInReceives =4303003 ipInHdrErrors = 0
ipInAddrErrors = 0 ipInCksumErrs = 0
ipForwDatagrams =2198042 ipForwProhibits = 4657
ipInUnknownProtos = 0 ipInDiscards = 0
ipInDelivers =2089532 ipOutRequests =2280862
ipOutDiscards = 0 ipOutNoRoutes = 29
ipReasmTimeout = 60 ipReasmReqds = 3
ipReasmOKs = 2 ipReasmFails = 1
ipReasmDuplicates = 0 ipReasmPartDups = 0
ipFragOKs = 0 ipFragFails = 0
ipFragCreates = 0 ipRoutingDiscards = 0
tcpInErrs = 33 udpNoPorts = 1537
udpInCksumErrs = 0 udpInOverflows = 0
rawipInOverflows = 0 ipsecInSucceeded = 0
ipsecInFailed = 0 ipInIPv6 = 0
ipOutIPv6 = 0 ipOutSwitchIPv6 = 15
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

nat01:~# ipnat -slv|head -60
mapped in 2098552 out 2328598
added 102851 expired 94387
inuse 8464
rules 10
table 8047c70 list e40199b0
List of active MAP/Redirect filters:
map iprb0 192.168.0.0/16 -> 0.0.0.0/32 proxy port ftp ftp/tcp
ifp e0bb3a9c space 16777190 nextip 0.0.0.0 pnext 0 flags 0 use 26
map iprb0 172.16.0.0/16 -> 0.0.0.0/32 proxy port ftp ftp/tcp
ifp e0bb3a9c space 16777216 nextip 0.0.0.0 pnext 0 flags 0 use 0
map iprb0 10.0.0.0/8 -> 0.0.0.0/32 proxy port ftp ftp/tcp
ifp e0bb3a9c space 4294967293 nextip 0.0.0.0 pnext 0 flags 0 use 2
map iprb0 192.168.0.0/16 -> 0.0.0.0/32 portmap auto [1024:65535 1 64512]
ifp e0bb3a9c space 4294963279 nextip 0.0.0.0 pnext 1024 flags 13 use 4016
map iprb0 172.16.0.0/16 -> 0.0.0.0/32 portmap auto [1024:65535 1 64512]
ifp e0bb3a9c space 4294967283 nextip 0.0.0.0 pnext 1024 flags 13 use 12
map iprb0 10.0.0.0/8 -> 0.0.0.0/32 portmap auto [1024:65535 1 64512]
ifp e0bb3a9c space 4294966900 nextip 0.0.0.0 pnext 1024 flags 13 use 395
map iprb0 192.168.0.0/16 -> 0.0.0.0/32
ifp e0bb3a9c space 16777210 nextip 0.0.0.0 pnext 0 flags 0 use 6
map iprb0 172.16.0.0/16 -> 0.0.0.0/32
ifp e0bb3a9c space 16777216 nextip 0.0.0.0 pnext 0 flags 0 use 0
map iprb0 10.0.0.0/8 -> 0.0.0.0/32
ifp e0bb3a9c space 4294967295 nextip 0.0.0.0 pnext 0 flags 0 use 0
rdr iprb1 0.0.0.0/0 port 80 -> 192.168.1.197 port 8080 tcp
e0bb361c 63939 1 36895 8047be0 4007

List of active sessions:
MAP 192.168.32.107 1025 <- -> 210.172.<ip>.<ip> 2049 [131.107.1.10 123]
age 1200 use 0 sumd 0x767f/0x767f pr 17 bkt 0 flags 2 bytes 152 pkts 2 727f
RDR 192.168.1.197 8080 <- -> 202.93.87.249 80 [192.168.33.190 2672]
age 1200 use 0 sumd 0xbf56/0xbf56 pr 6 bkt 0 flags 1 bytes 871 pkts 7 a016
[snip] (all bkt appear to be 0)

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-


nat01:/etc/opt/ipf# netstat -naf inet | awk '/\.80/ {print $NF}' | sort | uniq -c
2 CLOSE_WAIT
1048 ESTABLISHED
9 FIN_WAIT_1
60 FIN_WAIT_2
14 LAST_ACK
2 LISTEN
6 SYN_RCVD
4 SYN_SENT
881 TIME_WAIT







--
Jorgen Lundman | <lundman@lundman.net>
Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work)
Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell)
Japan | +81 (0)3 -3375-1767 (home)