Solaris10 IPF and return-rst

I am frustrated trying to get clients to drop TCP connections cleanly
using return-rst.

I am using a Solaris 10 host with its installed ipf package:

uname -a: SunOS 5.10 Generic_118833-36 sun4u sparc SUNW,Ultra-60
ipf -V: v4.0.3 (592)

I'm also testing against a bunch of Solaris8 with the latest ipfilter
4.1.28.

These are the relevant rules

# Split into incoming and outgoing rule groups
block in quick on hme0 all head 100
pass out quick on hme0 all head 200
block in log level local1.info quick proto tcp from any to any head 110 group 100
...
# The rule I want to get working
block return-rst in log level mail.info quick proto tcp from any to any port = 25 group 110
...
# Outgoing rule: free and easy access.
pass out quick from any to any keep state keep frags group 200

It doesn't seems to work: client hangs and ipfilter host gets multiple
SYN packets as the client retries to connect. A RST/ACK packet doesn't
get emitted (verified by "snoop" on client). I know about the outgoing
rule workaround as described in Phil's FAQ, but since I split my incoming
and outgoing rules into separate rule groups, I though the last rule
fixed that. Anyways, if you read on, it gets stranger.

If I modify the egress rule (the last rule) by removing the "keep
frags" modifier,

pass out quick from any to any keep state group 200

connections to Solaris8/ipfilter4.1.28 hosts then closed down properly.

For Solaris10/ipfilter4.0.3, a RST/ACK packets gets returned, but the
client still hangs. Here is a snoop dump showing the SYN sent out,
and the RST/ACK response (client=xx.xx.xx.xx, ipfilter host=yy.yy.yy.yy).

The client doesn't seem to acknowledge the RST/ACK, and sends another
SYN and these packet pairs go back and forth until the client gives up.
Is it possible the RST/ACK response is somehow corrupted? I've tested
using Linux/Solaris/Windows clients and they all hang.

Packet 1: xx.xx.xx.xx -> yy.yy.yy.yy (SYN to port 25/tcp):

ETHER: ----- Ether Header -----
ETHER:
ETHER: Packet 1 arrived at 20:45:25.14298
ETHER: Packet size = 66 bytes
ETHER: Destination = 8:0:20:cf:3d:3e, Sun
ETHER: Source = 8:0:20:8a:a:2f, Sun
ETHER: Ethertype = 0800 (IP)
ETHER:
IP: ----- IP Header -----
IP:
IP: Version = 4
IP: Header length = 20 bytes
IP: Type of service = 0x00
IP: xxx. .... = 0 (precedence)
IP: ...0 .... = normal delay
IP: .... 0... = normal throughput
IP: .... .0.. = normal reliability
IP: .... ..0. = not ECN capable transport
IP: .... ...0 = no ECN congestion experienced
IP: Total length = 52 bytes
IP: Identification = 22482
IP: Flags = 0x4
IP: .1.. .... = do not fragment
IP: ..0. .... = last fragment
IP: Fragment offset = 0 bytes
IP: Time to live = 64 seconds/hops
IP: Protocol = 6 (TCP)
IP: Header checksum = 8844
IP: Source address = xx.xx.xx.xx, xx.xx.xx.xx
IP: Destination address = yy.yy.yy.yy, yy.yy.yy.yy
IP: No options
IP:
TCP: ----- TCP Header -----
TCP:
TCP: Source port = 38953
TCP: Destination port = 25 (SMTP)
TCP: Sequence number = 3749460709
TCP: Acknowledgement number = 0
TCP: Data offset = 32 bytes
TCP: Flags = 0x02
TCP: 0... .... = No ECN congestion window reduced
TCP: .0.. .... = No ECN echo
TCP: ..0. .... = No urgent pointer
TCP: ...0 .... = No acknowledgement
TCP: .... 0... = No push
TCP: .... .0.. = No reset
TCP: .... ..1. = Syn
TCP: .... ...0 = No Fin
TCP: Window = 49640
TCP: Checksum = 0x9fdd
TCP: Urgent pointer = 0
TCP: Options: (12 bytes)
TCP: - Maximum segment size = 1460 bytes
TCP: - No operation
TCP: - Window scale = 0
TCP: - No operation
TCP: - No operation
TCP: - SACK permitted option
TCP:
SMTP: ----- SMTP: -----
SMTP:
SMTP: ""
SMTP:

Packet 2: yy.yy.yy.yy -> xx.xx.xx.xx (RST/ACK response):

ETHER: ----- Ether Header -----
ETHER:
ETHER: Packet 2 arrived at 20:45:25.14329
ETHER: Packet size = 60 bytes
ETHER: Destination = 8:0:20:8a:a:2f, Sun
ETHER: Source = 8:0:20:cf:3d:3e, Sun
ETHER: Ethertype = 0800 (IP)
ETHER:
IP: ----- IP Header -----
IP:
IP: Version = 4
IP: Header length = 20 bytes
IP: Type of service = 0x00
IP: xxx. .... = 0 (precedence)
IP: ...0 .... = normal delay
IP: .... 0... = normal throughput
IP: .... .0.. = normal reliability
IP: .... ..0. = not ECN capable transport
IP: .... ...0 = no ECN congestion experienced
IP: Total length = 40 bytes
IP: Identification = 9010
IP: Flags = 0x4
IP: .1.. .... = do not fragment
IP: ..0. .... = last fragment
IP: Fragment offset = 0 bytes
IP: Time to live = 255 seconds/hops
IP: Protocol = 6 (TCP)
IP: Header checksum = fdef
IP: Source address = yy.yy.yy.yy, yy.yy.yy.yy
IP: Destination address = xx.xx.xx.xx, xx.xx.xx.xx
IP: No options
IP:
TCP: ----- TCP Header -----
TCP:
TCP: Source port = 25
TCP: Destination port = 38953
TCP: Sequence number = 0
TCP: Acknowledgement number = 3749460710
TCP: Data offset = 20 bytes
TCP: Flags = 0x14
TCP: 0... .... = No ECN congestion window reduced
TCP: .0.. .... = No ECN echo
TCP: ..0. .... = No urgent pointer
TCP: ...1 .... = Acknowledgement
TCP: .... 0... = No push
TCP: .... .1.. = Reset
TCP: .... ..0. = No Syn
TCP: .... ...0 = No Fin
TCP: Window = 0
TCP: Checksum = 0x7a37
TCP: Urgent pointer = 0
TCP: No options
TCP:
SMTP: ----- SMTP: -----
SMTP:
SMTP: ""
SMTP:

Joseph Tam <tam@math.ubc.ca>