Re: Solaris 9 and ip filter performance issues.

So the morning crush is on:

Currently about 12% packet loss. My shell running netstat -i can pause for
almost indefinite periods, but wakes up if I push return..

The switches are Cisco catalysts 2950, and don't seem to be concerned. It is
only the final hop on the internal side that starts having issues.

If it is just cheap/poor hardware that can't keep up with the load, that is
fine, put in a different nic card, or somehow setup to use two nics on the LAN
side (can one do that with ipfilter?). Or perhaps replace the entire box.

It just seemed like a 100base link shouldn't be able to take down this box of
"reasonable" hardware?





Various stats:

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

nat01:~# netstat -i -I iprb1 1
input iprb1 output input (Total) output
packets errs packets errs colls packets errs packets errs colls
729 0 1059 0 0 1729 0 1749 3 0
569 0 810 0 0 1360 0 1366 9 0
566 0 806 0 0 1352 0 1383 10 0
632 0 797 1 0 1237 0 1289 4 0
471 0 591 1 0 1096 1 1083 5 0
685 0 1004 0 0 1631 1 1736 7 0
469 0 629 1 0 1080 0 1107 9 0
454 0 567 0 0 932 0 983 3 0
593 0 781 2 0 1282 0 1307 17 0
689 0 878 0 0 1406 1 1449 5 0
875 0 1257 0 0 1790 2 1973 9 0
620 0 880 0 0 1406 0 1471 7 0
705 0 931 0 0 1631 0 1653 10 0
657 0 852 0 0 1476 0 1529 8 0
692 0 889 0 0 1434 0 1489 8 0

There are contant errors like so, they don't seem to change (noticably) with
load. If the errors grew proportionally with heavy use that could be something?

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

nat01:~# kstat -p iprb:1: 1
iprb:1:iprb1:align_errors 0
iprb:1:iprb1:blocked 59
iprb:1:iprb1:brdcstrcv 105133
iprb:1:iprb1:brdcstxmt 178
iprb:1:iprb1:carrier_errors 0
iprb:1:iprb1:class net
iprb:1:iprb1:collisions 0
iprb:1:iprb1:crtime 79.044499568
iprb:1:iprb1:defer_xmts 0
iprb:1:iprb1:duplex full
iprb:1:iprb1:ex_collisions 0
iprb:1:iprb1:fcs_errors 0
iprb:1:iprb1:first_collisions 0
iprb:1:iprb1:ierrors 4237
iprb:1:iprb1:ifspeed 100000000
iprb:1:iprb1:intr 14450655
iprb:1:iprb1:ipackets 16663831
iprb:1:iprb1:ipackets64 16663831
iprb:1:iprb1:macrcv_errors 0
iprb:1:iprb1:macxmt_errors 0
iprb:1:iprb1:media PHY/MII
iprb:1:iprb1:missed 0
iprb:1:iprb1:multi_collisions 0
iprb:1:iprb1:multircv 0
iprb:1:iprb1:multixmt 98
iprb:1:iprb1:norcvbuf 0
iprb:1:iprb1:noxmtbuf 0
iprb:1:iprb1:obytes 1778147987
iprb:1:iprb1:obytes64 14663049875
iprb:1:iprb1:oerrors 4924
iprb:1:iprb1:oflo 4237
iprb:1:iprb1:opackets 18016065
iprb:1:iprb1:opackets64 18016065
iprb:1:iprb1:promisc off
iprb:1:iprb1:rbytes 2750247318
iprb:1:iprb1:rbytes64 7045214614
iprb:1:iprb1:rcv_badinterp 0
iprb:1:iprb1:runt_errors 0
iprb:1:iprb1:snaptime 59088.180337263
iprb:1:iprb1:sqe_errors 0
iprb:1:iprb1:toolong_errors 0
iprb:1:iprb1:tx_late_collisions 0
iprb:1:iprb1:uflo 4924
iprb:1:iprb1:unknowns 4599
iprb:1:iprb1:xmt_badinterp 5
iprb:1:iprb1:xmtretry 0

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

nat01:~# ipnat -s
mapped in 16170941 out 17899623
added 583640 expired 573521
inuse 10119
rules 10

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

nat01:~# ipfstat
dropped packets: in 0 out 0
non-data packets: in 0 out 0
no-data packets: in 0 out 0
non-ip packets: in 0 out 0
bad packets: in 0 out 0
copied messages: in 0 out 5642011
input packets: blocked 5325 passed 33322713 nomatch 25820346 counted 0
short 5
output packets: blocked 6838 passed 34085676 nomatch 27159146 counted 0
short 5
input packets logged: blocked 0 passed 0
output packets logged: blocked 0 passed 0
packets logged: input 0 output 0
log failures: input 0 output 0
fragment state(in): kept 0 lost 0
fragment state(out): kept 0 lost 0
packet state(in): kept 0 lost 0
packet state(out): kept 0 lost 0
ICMP replies: 0 TCP RSTs sent: 104
Result cache hits(in): 7502377 (out): 6926826
IN Pullups succeeded: 0 failed: 0
OUT Pullups succeeded: 5840 failed: 0
Fastroute successes: 104 failures: 75
TCP cksum fails(in): 0 (out): 0
Packet log flags set: (0)
none

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

nat01:~# netstat -s -P ip

IPv4 ipForwarding = 1 ipDefaultTTL = 255
ipInReceives =33445834 ipInHdrErrors = 0
ipInAddrErrors = 0 ipInCksumErrs = 0
ipForwDatagrams =23528411 ipForwProhibits = 4657
ipInUnknownProtos = 0 ipInDiscards = 0
ipInDelivers =9826661 ipOutRequests =10773455
ipOutDiscards = 0 ipOutNoRoutes = 29
ipReasmTimeout = 60 ipReasmReqds = 6
ipReasmOKs = 5 ipReasmFails = 1
ipReasmDuplicates = 0 ipReasmPartDups = 0
ipFragOKs = 0 ipFragFails = 0
ipFragCreates = 0 ipRoutingDiscards = 0
tcpInErrs = 241 udpNoPorts = 23358
udpInCksumErrs = 0 udpInOverflows = 0
rawipInOverflows = 0 ipsecInSucceeded = 0
ipsecInFailed = 0 ipInIPv6 = 0
ipOutIPv6 = 0 ipOutSwitchIPv6 = 215


-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-


nat01:~# ipnat -slv|head -60
mapped in 16380487 out 18148742
added 590584 expired 581442
inuse 9142
rules 10
table 8047c70 list e40199b0
List of active MAP/Redirect filters:
map iprb0 192.168.0.0/16 -> 0.0.0.0/32 proxy port ftp ftp/tcp
ifp e0bb3a9c space 16777201 nextip 0.0.0.0 pnext 0 flags 0 use 15
map iprb0 172.16.0.0/16 -> 0.0.0.0/32 proxy port ftp ftp/tcp
ifp e0bb3a9c space 16777216 nextip 0.0.0.0 pnext 0 flags 0 use 0
map iprb0 10.0.0.0/8 -> 0.0.0.0/32 proxy port ftp ftp/tcp
ifp e0bb3a9c space 4294967293 nextip 0.0.0.0 pnext 0 flags 0 use 2
map iprb0 192.168.0.0/16 -> 0.0.0.0/32 portmap auto [1024:65535 1 64512]
ifp e0bb3a9c space 4294961742 nextip 0.0.0.0 pnext 1024 flags 13 use 555
3
map iprb0 172.16.0.0/16 -> 0.0.0.0/32 portmap auto [1024:65535 1 64512]
ifp e0bb3a9c space 4294967281 nextip 0.0.0.0 pnext 1024 flags 13 use 14
map iprb0 10.0.0.0/8 -> 0.0.0.0/32 portmap auto [1024:65535 1 64512]
ifp e0bb3a9c space 4294966800 nextip 0.0.0.0 pnext 1024 flags 13 use 495
map iprb0 192.168.0.0/16 -> 0.0.0.0/32
ifp e0bb3a9c space 16777174 nextip 0.0.0.0 pnext 0 flags 0 use 42
map iprb0 172.16.0.0/16 -> 0.0.0.0/32
ifp e0bb3a9c space 16777216 nextip 0.0.0.0 pnext 0 flags 0 use 0
map iprb0 10.0.0.0/8 -> 0.0.0.0/32
ifp e0bb3a9c space 4294967295 nextip 0.0.0.0 pnext 0 flags 0 use 0
rdr iprb1 0.0.0.0/0 port 80 -> 192.168.1.197 port 8080 tcp
e0bb361c 329920 1 36895 8047be0 3021

List of active sessions:
MAP 192.168.33.177 1379 <- -> 210.172.<ip>.<ip> 2403 [210.157.17.127 443]
age 480 use 0 sumd 0x7539/0x7539 pr 6 bkt 0 flags 1 bytes 1435 pkts 13 7
139
MAP 192.168.34.153 1694 <- -> 210.172.<ip>.<ip> 2718 [202.181.98.209 110]
age 1200 use 0 sumd 0x7451/0x7451 pr 6 bkt 0 flags 1 bytes 650 pkts 13 7
051

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

nat01:~# netstat -naf inet | awk '/\.80/ {print $NF}' | sort | uniq -c
2 CLOSE_WAIT
1148 ESTABLISHED
12 FIN_WAIT_1
93 FIN_WAIT_2
52 LAST_ACK
2 LISTEN
2 SYN_RCVD
2 SYN_SENT
896 TIME_WAIT
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-


last pid: 6751; load averages: 1.62, 1.81, 1.62 10:22:13
38 processes: 37 sleeping, 1 on cpu
CPU states: 37.4% idle, 10.9% user, 51.7% kernel, 0.0% iowait, 0.0% swap
Memory: 2048M real, 1188M free, 706M swap in use, 2936M swap free
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

nat01:~# vmstat 1
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s2 -- in sy cs us sy id
0 0 0 3041824 1252080 34 165 16 17 17 0 2 0 7 0 0 851 1442 363 2 12 87
0 0 0 3007084 1217088 295 1050 16 40 40 0 0 0 5 0 0 1853 5356 1723 18 49 34
0 0 0 3007080 1217160 0 0 0 16 16 0 0 0 36 1 0 2628 1597 901 3 39 58
1 0 0 3007080 1217136 69 361 63 281 281 0 0 0 18 0 0 2243 2195 1212 7 43 50
0 0 0 3007080 1217332 15 0 0 0 0 0 0 0 0 1 0 1953 2498 1025 4 40 56
0 0 0 3007080 1217344 8 0 0 20 20 0 0 0 1 0 0 1910 2182 997 7 37 56
0 0 0 3007080 1217364 6 0 8 75 75 0 0 0 10 0 0 2555 3005 1352 10 44 47
0 0 0 3007080 1217392 17 3 4 103 103 0 0 0 79 2 0 1609 4282 1812 12 60 28
0 0 0 3007080 1217480 41 0 40 28 28 0 0 0 7 0 0 2846 2599 1083 7 35 58
0 0 0 3007080 1217504 5 0 0 16 16 0 0 0 2 0 0 3426 2458 923 10 32 58
0 0 0 3007080 1217480 18 1 4 4 4 0 0 0 3 0 0 1469 3005 1133 7 50 43
0 0 0 3007080 1217480 84 1 12 44 44 0 0 0 7 0 0 2368 3829 1389 6 50 44
0 0 0 3007080 1217532 4 0 44 51 51 0 0 0 68 1 0 1817 2590 1191 9 52 39
0 0 0 3007080 1217504 4 0 4 8 8 0 0 0 3 0 0 3225 1083 526 2 29 69
0 0 0 3007080 1217504 25 3 103 0 0 0 0 0 20 0 0 3286 2963 1136 7 39 54
0 0 0 3007080 1217472 26 2 143 16 16 0 0 0 35 0 0 1891 3932 1249 9 47 44
0 0 0 3007080 1217540 9 0 0 0 0 0 0 0 0 0 0 2320 1111 526 3 29 68
0 0 0 3007080 1217540 1 0 0 69 69 0 0 0 36 1 0 2872 2131 981 5 38 58
0 0 0 3007080 1217500 6 0 20 149 149 0 0 0 7 0 0 2619 1841 956 5 33 62
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

nat01:~# mpstat 1
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 85 1 38 243 115 200 15 8 8 24 881 3 5 0 92
1 79 1 24 609 461 163 13 8 9 25 562 1 18 0 80
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 5 0 394 377 182 1186 61 24 32 252 2297 14 21 1 64
1 0 1 76 1171 566 506 126 24 30 217 449 1 88 0 11
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 1 179 289 125 1022 85 21 47 194 3202 16 26 0 58
1 0 1 49 1130 768 514 88 23 30 191 37 0 77 0 23
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 2 2 236 268 132 1205 160 36 42 203 3530 14 26 0 60
1 2 0 15 1151 702 594 129 36 43 195 417 1 76 0 23
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 2 50 253 112 823 78 44 46 147 2750 13 26 1 60
1 0 0 19 921 718 517 66 46 40 146 728 2 70 1 27
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 2 262 112 485 24 13 13 55 1846 9 16 0 75
1 0 1 38 1224 1105 410 23 22 30 68 212 2 38 0 60
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 15 152 369 205 703 63 43 28 59 2579 17 22 9 52
1 0 19 50 1505 1240 534 27 31 36 65 806 2 52 6 40

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-


nat01:~# kstat |egrep 'fail|err|name:|max'
name: cpu_stat0 class: misc
rw_rdfails 145111
rw_wrfails 1343595
name: cpu_stat1 class: misc
rw_rdfails 131393
rw_wrfails 1422651
name: iprb0 class: net
align_errors 0
carrier_errors 0
fcs_errors 0
ierrors 4947
macrcv_errors 0
macxmt_errors 0
oerrors 106051
runt_errors 0
sqe_errors 0
toolong_errors 0
name: iprb1 class: net
align_errors 0
carrier_errors 0
fcs_errors 0
ierrors 5385
macrcv_errors 0
macxmt_errors 0
oerrors 5548
runt_errors 0
sqe_errors 0
toolong_errors 0
(all other "fails", "badcalls" are 0)

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-






--
Jorgen Lundman | <lundman@lundman.net>
Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work)
Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell)
Japan | +81 (0)3 -3375-1767 (home)