Re: ipfstat not clearing the state table - a similar problem?

Wes Zuber wrote:
> Darren, I noticed on another box that it looks like this:
>
> IP states added:
> 446043 TCP
> 735333 UDP
> 21536 ICMP
> 80110891 hits
> 47075066 misses
> 0 maximum
> 0 no memory
> 92 bkts in use
> 92 active
> 756869 expired
> 445951 closed
>
>
> bkts and active pretty much matching up.
>
>
> But on the box that is having the issue (and a bunch more traffic and
> connections)
>
> IP states added:
> 503251 TCP
> 103500 UDP
> 63074 ICMP
> 59277927 hits
> 31067012 misses
> 49128 maximum
> 0 no memory
> 44 bkts in use
> 9040 active
> 166568 expired
> 494217 closed
>
>
> When I check the state table I see about 44 connections.. certainly not
> 9 thousand by any means, if this helps at all.
>
> Almost seems like the count is not decrementing or something once the
> state is actually cleared.
>
> Thanks,
>
> --Wes
>
> On Aug 14, 2006, at 12:03 PM, Darren Reed wrote:
>
>>> Hi there,
>>>
>>> running FreeBSD 6.1 stable with:
>>>
>>> ipf: IP Filter: v4.1.13 (416)
>>> Kernel: IP Filter: v4.1.13
>>> Running: yes
>>> Log Flags: 0x20000000 = block
>>> Default: block all, Logging: available
>>> Active list: 1
>>> Feature mask: 0xa
>>>
>> ..
>>> If we run ipfstat -FS it only clears a few states.. If I run ipfstat -
>>> sl we only see a fraction of the states.
>>>
>>>
>>> On previous versions ipfstat -FS always knocked the state table to
>>> zero then it started building again.
>>
>> I think you mean "ipf -FS". Try "ipf -FS -Fs".
>>
>> Darren

Following complaints of poor performance (substantiated by mrtg network
packet loss and traffic graphs) I think I've just seen what looks like a
very similar issue on one of our firewall routers (FreeBSD
5.4-RELEASE-p22 with ipfilter 4.1.13). It's one of approximately 10
absolutely identical machines (same hardware, same parent build binaries
installed), but is the only one currently exhibiting these problems.
Whilst it is probably the one of the busier ones in terms of
'connections', hence state turnover, it is not at all busy in terms of
traffic.

router:~ $ sysctl -a net.inet.ipf
net.inet.ipf.fr_flags: 0
net.inet.ipf.fr_pass: 134217730
net.inet.ipf.fr_active: 0
net.inet.ipf.fr_tcpidletimeout: 86400
net.inet.ipf.fr_tcphalfclosed: 14400
net.inet.ipf.fr_tcpclosewait: 480
net.inet.ipf.fr_tcplastack: 480
net.inet.ipf.fr_tcptimeout: 480
net.inet.ipf.fr_tcpclosed: 120
net.inet.ipf.fr_udptimeout: 240
net.inet.ipf.fr_udpacktimeout: 24
net.inet.ipf.fr_icmptimeout: 120
net.inet.ipf.fr_defnatage: 1200
net.inet.ipf.fr_ipfrttl: 120
net.inet.ipf.fr_running: 1
net.inet.ipf.fr_statesize: 399979
net.inet.ipf.fr_statemax: 279883
net.inet.ipf.ipf_nattable_sz: 2047
net.inet.ipf.ipf_natrules_sz: 127
net.inet.ipf.ipf_rdrrules_sz: 127
net.inet.ipf.ipf_hostmap_sz: 2047
net.inet.ipf.fr_authsize: 32
net.inet.ipf.fr_authused: 0
net.inet.ipf.fr_defaultauthage: 600
net.inet.ipf.fr_chksrc: 0
net.inet.ipf.fr_minttl: 4

router:~ $ sudo ipfstat -s
IP states added:
52878784 TCP
284828793 UDP
1314093 ICMP
3431784467 hits
810105418 misses
1390510 maximum
0 no memory
5121 bkts in use
278711 active
286138681 expired
52604278 closed
State logging enabled

State table bucket statistics:
5121 in use
1.28% bucket usage
0 minimal length
2 maximal length
1.008 average length

I routinely log the output of 'ipfstat -s' (and many other things) to a
file every couple of minutes to help diagnose resource exhaustion.
Running the 'ipfstat -s' output through a small analysis program shows
that the number of 'active' states is permanently hovering dangerously
near or hitting the maximum, hence the increasing 'maximum' counter.

Just like Wes, we only see a small number of state entries in the table,
which makes it impossible to see with-what and why the table is full:

router: $ sudo ipfstat -R -sl | grep '^[.0-9]\{1,\} ->' | wc -l
5589

this number approximates the number of buckets in use above.

Following Darren's suggestion in answer to Wes, I also tried:

router:analyse $ sudo ipf -Fs
router:analyse $ sudo ipfstat -s
IP states added:
52883307 TCP
284859681 UDP
1314258 ICMP
3432259293 hits
810184346 misses
1390583 maximum
0 no memory
3670 bkts in use
277255 active
286171244 expired
52608747 closed
State logging enabled

State table bucket statistics:
3670 in use
0.92% bucket usage
0 minimal length
2 maximal length
1.007 average length

and

router:analyse $ sudo ipf -FS
router:~ $ sudo ipfstat -s
IP states added:
52883977 TCP
284862978 UDP
1314270 ICMP
3432313434 hits
810194606 misses
1390583 maximum
0 no memory
1268 bkts in use
274830 active
286176199 expired
52610196 closed
State logging enabled

State table bucket statistics:
1268 in use
0.32% bucket usage
0 minimal length
2 maximal length
1.002 average length

both made very little difference to the number of active states -
in fact, they've only affected the 'bkts in use' which is approximately
the number seen in the output of 'ipfstat -R -sl'.

We've been forced to reboot the machine to clear the problem, but
already I notice that there is a considerable disparity between the
entries in the state table and the 'active' states:

router $ sudo ipfstat -s
IP states added:
608735 TCP
3058970 UDP
11518 ICMP
95440825 hits
25601396 misses
0 maximum
0 no memory
30396 bkts in use
39408 active
3052623 expired
587192 closed
State logging enabled

State table bucket statistics:
30396 in use
7.60% bucket usage
0 minimal length
4 maximal length
1.067 average length

$ sudo ipfstat -R -sl | grep -- '^[.0-9]* ->' | wc -l
33488

Doing a quick survey of all my routers I find that a number of them
exhibit this sort of disparity (the active ones), with some of the
differences being quite large:

router active statetbl active-statetbl
2 21 20 1
3 7550 5853 1697
4 133 37 96
5 119888 22560 97328
6 25 25 0
7 37461 30154 7307 <- this is the router discussed above
8 32248 6213 26035
9 20624 8499 12125
10 12593 4570 8023
11 3 3 0
12 150645 34458 116187
13 116973 18377 98596
14 5 5 0
15 6 6 0
16 6 6 0
17 6 6 0

I have been considering whether I need to adjust some of the ipf.fr_*
timeouts downwards, but since the states don't appear in the state table
I'm not very confident that this will make any difference (when I've had
problems which were helped by adjusting timeouts before, the state table
always looked full).

I'm now wondering what the 'invisible' states are (Darren mentioned
'orphans'), how they get created and if there is a way to clear them and
free up resources again.

Thanks in advance for any light shed or wisdom dispensed.

Best wishes,

Simon

--
----------------------------------------------------------------------
Dr Simon A. Boggis Senior Network Analyst
Computing Services, Tel. 020 7882 7078
Queen Mary, University of London, London E1 4NS UK.