IPF on TRU64 NUMA

Hello everyone.

I am currently running a Tru64 cluster and have tested IPF 4.0alpha on this (The server used for testing was a ES47 4 Proc 8gb ram). The cluster is running Tru64 5.1b PK3. The IP Filter component is causing the server to crash. The response from the HP engineer is shown below.

This crash appears to have been caused by the third party vendor that
provided this software (taken from the boot messages):

ipfilter_postconfig:major # = 89
IP Filter: v4.0alpha31 configured
IP Filter: v4.0alpha31 initialized. Default = pass all, Logging = enabled

It was caused by the ipfilter code with a PC of: 0xffffffff8001ab20 and the
lock was at address 0xfffffc006e49a480. I would give you a line number for
that PC, except that I don't have the module on our systems. The panic
itself was because the code was unlocking a lock it didn't own.

On non-NUMA systems running in lockmode 2 (production/performance locking)
this would go undetected aside from side effects. On NUMA systems, the
architecture behind MCS locking pretty much guarantees that you can't unlock
a lock you don't own, hence this panic.

To find the line number (for reporting to the original vendor) have the
customer execute these commands on the system that crashed:

cd /var/adm/crash
dbx -k vmunix.0 vmzcore.0 # Assuming they still have the dumps on their
system

# Will give the functions and line numbers since dbx can see the modules
(dbx) 0xffffffff8001ab20?20i
(dbx) quit

< Inserted Output of above command >

[fr_delstate:2531, 0xffffffff8001aad4] ble t1, 0xffffffff8001aadc
[fr_delstate:2532, 0xffffffff8001aad8] br zero, 0xffffffff8001ab4c
[fr_delstate:2542, 0xffffffff8001aadc] ldah t2, -32765(zero)
[fr_delstate:2542, 0xffffffff8001aae0] ldl t2, -16592(t2)
[fr_delstate:2542, 0xffffffff8001aae4] beq t2, 0xffffffff8001aafc
[fr_delstate:2542, 0xffffffff8001aae8] ldl t3, 32(sp)
[fr_delstate:2542, 0xffffffff8001aaec] beq t3, 0xffffffff8001aafc
[fr_delstate:2543, 0xffffffff8001aaf0] bis zero, s1, a0
[fr_delstate:2543, 0xffffffff8001aaf4] ldl a1, 32(sp)
[fr_delstate:2543, 0xffffffff8001aaf8] bsr ra, ipstate_log(line 3064)
[fr_delstate:2545, 0xffffffff8001aafc] ldq v0, 104(s1)
[fr_delstate:2545, 0xffffffff8001ab00] beq v0, 0xffffffff8001ab1c
[fr_delstate:2546, 0xffffffff8001ab04] ldq t0, 104(s1)
[fr_delstate:2546, 0xffffffff8001ab08] ldl t1, 76(t0)
[fr_delstate:2546, 0xffffffff8001ab0c] subl t1, 0x1, t1
[fr_delstate:2546, 0xffffffff8001ab10] stl t1, 76(t0)
[fr_delstate:2547, 0xffffffff8001ab14] lda a0, 104(s1)
[fr_delstate:2547, 0xffffffff8001ab18] bsr ra, fr_derefrule(line 4205)
[fr_delstate:2550, 0xffffffff8001ab1c] bis zero, s1, a0
[fr_delstate:2550, 0xffffffff8001ab20] bsr ra, 0xffffffff80028f50

< Inserted Output of above command >

The way I arrived at this conclusion was by examining the PC and correlating
it with its appropriate subsystem using crash:

(from the msg command, or p *pmsgbuf in dbx)
mcs_unlock: current lock not found

pc of caller: 0xffffffff8001ab20 <--- loadable address
lock address: 0xfffffc006e49a480
lock class name: unknown_simple_lock
current lock state: 0x800000008001be4e
(cpu=?,pc=0xffffffff8001be4c,free)

crash> set 0
crash> show vm
[...]
### This shows the VM entry containing the given PC
fffffc006bb53a80 ffffffff01345df0 ffffffff80004000 ffffffff8002a000 KERNEL

### Now our job is to find a subsystem whose entry point lies within that
### same region
crash> sysconfig | head -15
Type Name Entry Point
dynamic envmon ffffffff80264210
dynamic dna_xti ffffffff80260000
dynamic dna_base ffffffff801f4000
dynamic x25_relay ffffffff801d63b0
dynamic x25_ip ffffffff801c7590
dynamic x25_access ffffffff800b0040
dynamic dna_dli ffffffff8009e000
dynamic wdd_datalinks ffffffff80088ab0
dynamic wdd_base ffffffff80063130
dynamic ctf_base ffffffff80046000
dynamic wan_utilities ffffffff80041e80
dynamic dna_netman ffffffff80035c70
dynamic ipfilter ffffffff800279b0 <--- same region
dynamic hwautoconfig ffffffff80000000


If you could please give me a suggestion for moving forward I would appreciate it. IP Filter is the only solution that can be used on Tru64.


Shanon Loveridge
Reporting Data Warehouse (RDW - formerly known as RORE)
Phone: 03 9693 4030
http://www.in.telstra.com.au/ism/infomgt/rore.asp