Re: IPFilter 4.1.1 caused SUN V240 to panic and reboot

Hi Darren,

Looks like the problem moved to different stack. I am now using IPFilter-4.1.3 compiled with Sun Studio 8 ( This a DNS
server running Solaris 9). The following are from the dump analysis:

ts-du-ns2[/var/crash/its-du-ns2]
$>mdb -k unix.2 vmcore.2
Loading modules: [ unix krtld genunix ip ufs_log usba cpc ipc random nfs ]
> $c
mutex_enter+4(30004009180, 0, 780e24e8, 0, 300055dc1ac, 0)
fr_updatestate+0x5c(2a1005d68e8, 30004009080, 780e24e8, 2a1005d694c,
30004009180, 1)
fr_checkstate+0x32c(2a1005d68e8, 2a1005d68d0, 40004702, 3, 30002ec4a18,
30004009080)
fr_check+0x5e4(300055dc198, 30006fe2f80, 0, 1, 2a1005d68e8, 2a1005d6d30)
pfil_precheck+0xd28(e, 30006fe2f80, 2, 30001245ed8, 0, 300055dc198)
pfilmodwput+0x17c(300011f8898, 30006fe2f80, 20, 300055dc1ac, de010800,
30001245ed8)
putnext+0x21c(0, 30006fe2f80, ffffffffffffffff, 0, 579b40000, 8)
ip_wput_ire+0xacc(10000, 300000727b0, 3, ffff, 83aa0201, 30006fe2f80)
ip_wput+0x4d4(30006cc3088, 0, c9ebf91, 300055dc198, 0, 21683a1)
putnext+0x21c(0, 300010e5f80, 20, 0, 0, c0)
udp_wput+0x5b0(38, 14, 38, 300055dc1ac, 300055dc198, c0)
putnext+0x21c(0, 30006f4d680, 20, 30006fe2f80, 0, 98)
strput+0x270(30003a28018, 0, 0, 2a1005d7598, 0, 0)
kstrputmsg+0x36c(300011e58f8, 30004392b88, 0, 0, 0, 0)
sosend_dgram+0x25c(0, 3000135f9e8, 10, 2a1005d78d0, 8000, 2337d78)
sosendmsg+0x3f4(0, 3000135f9e8, 7, 20, 428aa4, 2337d78)
sendit+0x15c(2a1005d78d0, 8000, 300011e58f8, 8000, 2a1005d7900, 21683a1)
sendmsg+0x2e8(18, fef7f820, 8000, fef7f840, fef7f83c, c0)
syscall_trap32+0xa8(18, fef7f820, 8000, fef7f840, fef7f83c, c0)


> $<msgbuf
0x300010ac8a3: mc-us3i1 at root: SAFARI 0x1 0x0 ...
0x300010ac4e3: mc-us3i1 is /memory-controller@1,0
0x300010ac260: power0 at ebus0: offset 0,800
0x30001093ea3: power0 is /pci@1e,600000/isa@7/power@0,800
0x30001093c20: rmc_comm0 at ebus0: offset 0,3e8
0x300010939a3: rmc_comm0 is /pci@1e,600000/isa@7/rmc-comm@0,3e8
0x30001093722: PCI-device: pmu@6, pmubus0
0x300010934a2: PCI-device: gpio@8a, pmugpio0
0x30001093223: pmugpio0 is /pci@1e,600000/pmu@6/gpio@8a
0x30001092fa2: pseudo-device: rmclomv0
0x30001092d23: rmclomv0 is /pseudo/rmclomv@0
0x30001092aa3: Hardware watchdog enabled
0x30001092820: su0 at ebus0: offset 0,3f8
0x300010925a3: su0 is /pci@1e,600000/isa@7/serial@0,3f8
0x30001092320: su1 at ebus0: offset 0,2e8
0x300010920a3: su1 is /pci@1e,600000/isa@7/serial@0,2e8
0x300010dbda0: cpu1: UltraSPARC-IIIi (portid 1 impl 0x16 ver 0x24 clock 1002
MHz)
0x300010dbb20: cpu0: UltraSPARC-IIIi (portid 0 impl 0x16 ver 0x24 clock 1002
MHz)
0x300010db8a0: cpu 0 initialization complete - online
0x300010db622: PCI-device: usb@a, ohci0
0x300010db3a3: ohci0 is /pci@1e,600000/usb@a
0x300010db11f: bge0: BCM570x driver v0.37: type "ether" mac address
00:03:ba:5f:de:01
0x300010daea2: PCI-device: network@2, bge0
0x300010dac23: bge0 is /pci@1f,700000/network@2
0x300010da99f: bge1: BCM570x driver v0.37: type "ether" mac address
00:03:ba:5f:de:02
0x300010da722: PCI-device: network@2,1, bge1
0x300010da4a3: bge1 is /pci@1f,700000/network@2,1
0x300010da21f: bge2: BCM570x driver v0.37: type "ether" mac address
00:03:ba:5f:de:03
0x30001253f22: PCI-device: network@2, bge2
0x30001253ca3: bge2 is /pci@1d,700000/network@2
0x30001253a1f: bge3: BCM570x driver v0.37: type "ether" mac address
00:03:ba:5f:de:04
0x300012537a2: PCI-device: network@2,1, bge3
0x30001253523: bge3 is /pci@1d,700000/network@2,1
0x30001252edf: NOTICE: bge0: link down (initialised)
0x30001252c63: dump on /dev/md/dsk/d20 size 2048 MB
0x300012529df: NOTICE: bge0: link up 1000Mbps Full-Duplex (initialised)
0x30001252622: pseudo-device: devinfo0
0x300012523a3: devinfo0 is /pseudo/devinfo@0
0x30001252122: pseudo-device: fssnap0
0x3000241dd63: fssnap0 is /pseudo/fssnap@0
0x3000241d9a2: pseudo-device: ramdisk1024
0x3000241d723: ramdisk1024 is /pseudo/ramdisk@1024
0x3000241d4a2: pseudo-device: winlock0
0x3000241d223: winlock0 is /pseudo/winlock@0
0x3000241cfa2: pseudo-device: lockstat0
0x3000241cd23: lockstat0 is /pseudo/lockstat@0
0x3000241caa2: pseudo-device: vol0
0x3000241c823: vol0 is /pseudo/vol@0
0x3000241c6e2: pseudo-device: llc10
0x3000241c463: llc10 is /pseudo/llc1@0
0x3000241c1e2: pseudo-device: pm0
0x30002573ee3: pm0 is /pseudo/pm@0
0x30002573c62: pseudo-device: tod0
0x300025739e3: tod0 is /pseudo/tod@0
0x30002573622: pseudo-device: lofi0
0x300025733a3: lofi0 is /pseudo/lofi@0
0x30002573122: pseudo-device: fcp0
0x30002572ea3: fcp0 is /pseudo/fcp@0
0x30002572c21: /kernel/drv/sparcv9/fcsm: undefined symbol
0x30002572861: 'fc_ulp_get_port_login_params'
0x300025725e1: WARNING: mod_load: cannot load module 'fcsm'
0x30002572362: pseudo-device: rsm0
0x300025720e3: rsm0 is /pseudo/rsm@0
0x30002561ca2: pseudo-device: trapstat0
0x30002561a23: trapstat0 is /pseudo/trapstat@0
0x300025618e2: pseudo-device: rmcadm0
0x30002561663: rmcadm0 is /pseudo/rmcadm@0
0x3000256129f: IP Filter: v4.1.3, running.
0x30002560b1f: IP Filter: v4.1.3 detached.
0x30002560edf: IP Filter: v4.1.3, running.
0x300061d4762: pseudo-device: devinfo0
0x300061c53a3: devinfo0 is /pseudo/devinfo@0
0x300061c0620:
panic[cpu0]/thread=30004d517e0:
0x300000824a0: BAD TRAP: type=31 rp=2a1005d6490 addr=0 mmu_fsr=0 occurred in
module "unix"
due to a NULL pointer dereference
0x300061c6820:

0x300061c5da0: named:
0x300061c65a0: trap type = 0x31
0x300061bb360: pid=364, pc=0x1035ba4, sp=0x2a1005d5d31, tstate=0x9980001605,
context=0x8a0
0x300061c6d20: g1-g7: ffffffffffffffd0, 1, 1388, 0, 300055dc1ac, 0, 30004d517e0
0x300061c79a0:
0x300061c03a3: 000002a1005d61c0 unix:die+80 (31, 2a1005d6490, 0, 0, de010800,
30001245ed8)
0x300061c5623: %l0-3: 0000000000000000 0000000001413590 000002a1005d6490
000002a1005d638
8
%l4-7: 0000000000000031 000002a1005d78d0 0000000000000000 00000300070ef4c0
0x300061c74a3: 000002a1005d62a0 unix:trap+874 (2a1005d6490, 0, 10000, 10200, 0,
8)
0x30001252263: %l0-3: 0000000000000001 0000000000000000 0000030004d37488
000000000000003
1
%l4-7: 0000000000000005 0000000000000001 0000000000000000 0000000000000000
0x300061c58a3: 000002a1005d63e0 unix:ktl0+48 (0, 30004d517e0, 3, ffff,
83aa0201, 300055dc1
ac)
0x300061c7723: %l0-3: 0000000000000006 0000000000001400 0000009980001605
000000000102c90
c
%l4-7: 0000000000000000 0000000000000000 0000000000000000 000002a1005d6490
0x30002560763: 000002a1005d6530 ipf:fr_movequeue+30 (30004009180, 0, 780e24e8,
0, 300055dc
1ac, 0)
0x300061c1a23: %l0-3: 0000000000000000 0000000000000002 0000000000000000
00000300040090c
8
%l4-7: 0000030004009080 0000030004009080 0000030006cc3088 000000000000800f
0x300061c5123: 000002a1005d6600 ipf:fr_updatestate+5c (2a1005d68e8,
30004009080, 780e24e8,
2a1005d694c, 30004009180, 1)
0x300061c4c23: %l0-3: 00000000780e24e8 0000030004009080 0000000000000000
000003000400908
0
%l4-7: 0000030004a6a000 00000000904941e7 000002a1005d68f0 0000000000000000
0x30000082363: 000002a1005d66f0 ipf:fr_checkstate+32c (2a1005d68e8,
2a1005d68d0, 40004702,
3, 30002ec4a18, 30004009080)
0x30002572223: %l0-3: 0000000000000000 0000000000000011 000000000000ffff
000000000000000
3
%l4-7: 000002a1005d68e8 0000030006f4d680 00000000014dad2c 00000300055dc1ac
0x3000241c5a3: 000002a1005d67e0 ipf:fr_check+5e4 (300055dc198, 30006fe2f80, 0,
1, 2a1005d6
8e8, 2a1005d6d30)
0x300010db9e3: %l0-3: 0000000000000000 000002a1005d68e8 0000000040000000
00000000780e2fe
0
%l4-7: 000000000000001c 0000030006fe2f80 000000000000001c 0000000000000004
0x300010926e3: 000002a1005d69f0 pfil:pfil_precheck+d28 (e, 30006fe2f80, 2,
30001245ed8, 0,
300055dc198)
0x300025613e3: %l0-3: 0000000000000014 0000000078161558 0000000000000000
0000030006fe2f8
0
%l4-7: 000000000000004c 0000030006fe2fa0 000000000000004c 0000000000000000
0x300025617a3: 000002a1005d6c40 pfil:pfilmodwput+17c (300011f8898, 30006fe2f80,
20, 300055
dc1ac, de010800, 30001245ed8)
0x30003210fa3: %l0-3: 00000300000727b0 0000030001245ed8 0000030003a28018
00000300011f889
8
%l4-7: 0000000000000083 000002a1005d78d0 0000000000000000 00000300070ef4c0
0x30003210463: 000002a1005d6d40 unix:putnext+21c (0, 30006fe2f80,
ffffffffffffffff, 0, 579
b40000, 8)
0x30000083da3: %l0-3: 00000000013fbe10 00000300011f8988 00000300011f8898
000000000000000
0
%l4-7: 0000000000000000 0000000078020930 00000300011f9540 0000030006fe2f80
0x30003211223: 000002a1005d6df0 ip:ip_wput_ire+acc (10000, 300000727b0, 3,
ffff, 83aa0201,
30006fe2f80)
0x30003210d23: %l0-3: 0000030007041540 000003000102e000 0000030006cc3088
00000300055dc19
8
%l4-7: 0000000000000000 0000000000000000 00000300011f9540 000000000000000e
0x300061d5b63: 000002a1005d6ff0 ip:ip_wput+4d4 (30006cc3088, 0, c9ebf91,
300055dc198, 0, 2
1683a1)
0x300003a79e3: %l0-3: 0000030006fe2f80 0000000000000000 000000000c9ebf91
00000000014a940
0
%l4-7: 0000030004392900 0000030002fba7b8 0000030006cc3088 0000030002fba7c8
0x300000839e3: 000002a1005d70a0 unix:putnext+21c (0, 300010e5f80, 20, 0, 0, c0)
0x30003211363: %l0-3: 00000000011fee0c 00000300010e9ee0 0000030004392900
000000000000000
0
%l4-7: 00000300010e5f88 00000000014a9b50 0000030004392678 0000030006fe2f80
0x30003211ae3: 000002a1005d7150 udp:udp_wput+5b0 (38, 14, 38, 300055dc1ac,
300055dc198, c0
)
0x300025608a3: %l0-3: 000000000c9ebf91 0000030006fe2f80 000000000000ffff
000000000000003
5
%l4-7: 0000030004392678 0000030006f4d680 00000000014dad2c 0000030004301498
0x30001092463: 000002a1005d7220 unix:putnext+21c (0, 30006f4d680, 20,
30006fe2f80, 0, 98)
0x3000241dc23: %l0-3: 00000000012d9060 0000030004392768 0000030004392678
000000000000000
0
%l4-7: 0000000000000000 00000000014d9350 0000030004392b88 0000030006f4d680
0x300061d5f23: 000002a1005d72d0 genunix:strput+270 (30003a28018, 0, 0,
2a1005d7598, 0, 0)
0x300061573a3: %l0-3: 0000000000000004 0000030004392b88 0000000000000000
0000030003a2801
8
%l4-7: 0000030006f4d680 000002a1005d78d0 0000000000000000 0000000000000000
0x30002560da3: 000002a1005d74c0 genunix:kstrputmsg+36c (300011e58f8,
30004392b88, 0, 0, 0,
0)
0x3000241d363: %l0-3: 0000030003a28098 00000300070ef4c0 0000030003a28018
000000000000000
4
%l4-7: 0000000000000083 000002a1005d78d0 0000000000000000 00000300070ef4c0
0x30002561f23: 000002a1005d75a0 sockfs:sosend_dgram+25c (0, 3000135f9e8, 10,
2a1005d78d0,
8000, 2337d78)
0x30000c3f3e3: %l0-3: 0000000000000010 000000000144f000 0000000000000030
00000300011e58f
8
%l4-7: 00000300070ef4c0 00000000000003c0 0000000000000000 0000000000000000
0x300003c6963: 000002a1005d7690 sockfs:sosendmsg+3f4 (0, 3000135f9e8, 7, 20,
428aa4, 2337d
78)
0x300010921e3: %l0-3: 0000000000000010 0000000000008000 000003000135f9e8
000000000000000
0
%l4-7: 00000300011e58f8 000002a1005d7a80 000002a1005d78d0 0000000000000000
0x30002573da3: 000002a1005d7750 sockfs:sendit+15c (2a1005d78d0, 8000,
300011e58f8, 8000, 2
a1005d7900, 21683a1)
0x30001252763: %l0-3: 000003000135f9e8 0000000000000000 0000000000000000
000000000000000
0
%l4-7: 00000300041bea80 0000000000000018 0000000000000030 000002a1005d7a80
0x300010dbee3: 000002a1005d7820 sockfs:sendmsg+2e8 (18, fef7f820, 8000,
fef7f840, fef7f83c
, c0)
0x300010db263: %l0-3: 0000000000000001 0000000000008000 00000000004bea90
000000000010000
0
%l4-7: 0000000000008000 000000000140f5b8 0000000000000001 00000000001555c0
0x3000241d0e0:
0x30001253663: syncing file systems...
0x300061bafa3: 4
0x300061bb4a3: done
0x300061bb9a3: dumping to /dev/md/dsk/d20, offset 429588480, content: kernel +
curproc
>


Thanks
Mathew



>>> Darren Reed <darrenr@reed.wattle.id.au> 20/07/04 7:54:47 >>>
In some email I received from Laurent Blume, sie wrote:
> Jeff A. Earickson wrote:
>
> > 2) Sun software engineers will **always** blame ipfilter for the problem
> > when the see it in the stack trace. While the problem may actually be
> > due to ipfilter, don't believe Sun.
> [snip]
>
> Well, since ipfilter is integrated in Solaris 10, it's not like they're
> always going to get away with that -- so you'll be able to believe them
> pretty soon :-)

The problem reported was fixed for 4.1.2.

Darren