NetBSD Problem Report #44418
From Wolfgang.Stukenbrock@nagler-company.com Wed Jan 19 18:54:24 2011
Return-Path: <Wolfgang.Stukenbrock@nagler-company.com>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by www.NetBSD.org (Postfix) with ESMTP id 994E063B883
for <gnats-bugs@gnats.NetBSD.org>; Wed, 19 Jan 2011 18:54:24 +0000 (UTC)
Message-Id: <20110119185413.BF61E1E80CE@test-s0.nagler-company.com>
Date: Wed, 19 Jan 2011 19:54:13 +0100 (CET)
From: Wolfgang.Stukenbrock@nagler-company.com
Reply-To: Wolfgang.Stukenbrock@nagler-company.com
To: gnats-bugs@gnats.NetBSD.org
Subject: FAST_IPSEC and if_wm kernel panic - may affect the whole network stack
X-Send-Pr-Version: 3.95
>Number: 44418
>Category: kern
>Synopsis: FAST_IPSEC and if_wm kernel panic - may affect the whole network stack
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: kern-bug-people
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Wed Jan 19 18:55:00 +0000 2011
>Closed-Date: Sat Feb 19 13:04:56 +0000 2011
>Last-Modified: Sat Feb 19 14:35:01 +0000 2011
>Originator: Dr. W. Stukenbrock
>Release: NetBSD 5.1 - HEAD also
>Organization:
Dr. Nagler & Company GmbH
>Environment:
System: NetBSD s0g7 5.1 NetBSD 5.1 (NSW-locationGW) #26: Wed Jan 19 17:05:17 CET 2011 ncadmin@s0g7:/usr/src/sys/arch/amd64/compile/NSW-locationGW amd64
Architecture: x86_64
Machine: amd64
>Description:
The system is a supermicro X8SIL (3400 chipset) with a L3406.
Due to problems with the onboard NIC's, there is a dual port NIC (82571EB based) too.
The test uses only the two 82571EB NIC's. No cable installed on the onbaord NIC's.
I've configured a tunnel with ipcomp and esp - the problem happens on outgooing packets.
spdadd -n 172.25.0.0/16 172.16.0.0/16 any -P out ipsec \
ipcomp/tunnel/62.153.101.247-62.153.101.241/use \
esp/tunnel/62.153.101.247-62.153.101.241/require;
Only the outgooing rule shown. There is a corresponding setup on a NetBSD4.0 system at 62.153.101.241.
This system is 172.25.0.1 and 62.153.101.247.
Some trafic (ftp put of a large file) is done from 172.25.0.2 to 172.16.3.1 that goes through the tunnel.
In general it works fine, but .... -> panic
Problem description based on the NetBSD source code:
The function key_checkrequest() in /usr/src/sys/netipsec/key.c removes the sav entry from the
isr structure and a new one is allocated after that. This places NULL for a short time in isr->sav.
This is done regardless the number of packets currently useing this isr! (or this SP where the isr is attached)
In the ipcomp code after compression, the sav in the isr is checked in /usr/src/sys/netipsec/xfrom_ipcomp.c
function ipcomp_output_cb() against a new allocated sav and if they do not match an ASSERTION is triggered.
Some statements later in /usr/src/sys/netipsec/ipsec_output.c function ipsec_process_done() there
are some assertens on isr->sav too and after that the pointer isr->sav is referenced.
Now I've got lots of kernel panics when the isr->sav is referenced, because it is a NULL pointer!
I've tried to figured out the reason, because it was not clear from the source.
I've enabled DIAGNOSTICS and DEBUG - turns on IPSEC_ASSERT.
Now the ASSERT fails sometimes in ipcomp_output_cb() and sometimes in ipsec_process_done() - every time
with a NULL pointer ...
But in DDB I'm always finding the correct valid pointer in the isr structure ... ?!?!?!
I've added some print messages with the actual values in front of the assertion in case of NULL, and
find out that the pointer reads NULL and next time the read reads the valid pointer again.
Strange - is another CPU modifying the data, or is it a cache problem with the L3406 ????
I've search in the whole sources where isr->sav gets modified and found only one place in key_checkrequest().
I've changed the way the the isr->sav is updated - mainly allocate new one first and do some kind of
"atomic" update by assignement of the new pointer to avoid NULL in isr->sav.
The NULL problems has gone!
Hmmmmm .... it is a multiprocessing related problem!
I've checked the SPL-state (by source code analyses) and it is splsoftnet all the time.
I've checked the splsoftnet() implementation on amd64 and find out that it is mapped to the assembler
stub "splraise" in /usr/src/sys/arch/amd64/amd64/spl.S. That one simply changes a value
in "CPUVAR(ILEVEL)". It does nothing in respect to other CPU's - as far as I can see ...
So if an outgooing packet is processed and key_checkrequest() is called, this may be concurrent to the
call to ipcomp_output_cb() by the crypto-stuff-kernel-thread.
That could be the reason for my problem ...
I accedently do not know if the fact, that both CPU's are runnung on SPLSOFTNET at the same time, is correct or not.
The amd64 implementation looks like that this is a valid situation.
(And I think it would be very slow to contact all other cpu's when changing SPL level.)
If it is allowed for any number of CPU's to run in parallel on SPLSOFTNET, then the current implementation
of FAST_IPSEC is broken!
The assumption that there is a valid isr->sav pointer in ipsec_process_done() is void, because
it may have changed by key_checkrequest() to a NULL pointer after the check for NULL done in
ipcomp_output_cb() -> panic.
And this may happen every time if a second packet is forwarded in the tunnel just in that moment when the
ipcomp processing is between the check in ipcomp_output_cb() and the access to the pointer in ipsec_process_done().
There must be a mutex in order to synchronize the access to the structures!
remark: I'm not shure if other parts of the network stack or devices are affected too ...
With the desscribed change to key_checkrequest() above, only the NULL panic in ipsec_process_done()
are gone.
I've still additional crashes in if_wm.c when extracting a mbuf from the send-queue with NULL ...
Seems to be something simular to the NULL crash above, but still no time to go deeper into that.
I've still the problem that sometimes a static added SA with setkey disapears (the outgooing ESP-SA in
all cases up to now - no time to search that one till now, but I think it is related to the broken
MP-sync too). No racoon started. Happens too if only the sshd and the loging shell is running on
the system. (All other processes killed after boot.)
At the moment it looks like the FAST_IPSEC implementation is not MP safe and the whole
network stack runs only stable, because all device interrupts are processed (and serialised) on one CPU only.
At least the wm-driver seems to get into problems with the send-queue if multiple CPU's are gooing to start packets.
This is the reason why I classified this PR as critical.
remark:
This setup runs fine (on other systems of cause with only 2 cores) with NetBSD4.0 and I cannot say where
the main difference in the FAST_IPSEC and or if_wm implementation is.
Also the change of the splxxxx routines from C to assembler should not be the reason for anything.
At the moment I think it is running "stable" with 4.0 due to the slower machine with less cores
and the "other" kernel-internal thread scheduler. So 4.0 may be also affected.
>How-To-Repeat:
Setup a tunnel with ipcomp and esp - ipcomp alone shoulc be good enougth too.
Use a fast machine with at least 4 Cores/Threads - e.g. Xeon L3406 (2 Cores, each 2 Threads)
>Fix:
Still not realy known to me. (sorry)
For the NULL panics in ipsec_process_done() the suggested workaround above for key_checkrequest() seems to help.
But there should be a mutex to synchronise access to the key structures.
But the problem seems to be a much more general one !!!!
The complete multi-CPU synchronisation in FAST_IPSEC needs a review and seems to be instable at the moment.
The whole network stack may be affected too - e.g. when accessing interface structrures.
From my current point of view the whole network stack is affected and the MP-synchronisation needs a review.
>Release-Note:
>Audit-Trail:
From: Wolfgang Stukenbrock <Wolfgang.Stukenbrock@nagler-company.com>
To: gnats-bugs@NetBSD.org
Cc: Wolfgang.Stukenbrock@nagler-company.com
Subject: Re: kern/44418: FAST_IPSEC and if_wm kernel panic - may affect the whole network stack
Date: Thu, 20 Jan 2011 14:15:25 +0100
Hi again,
I've done some further analyses and detect that the major difference
between the 4.0 and 5.x implementation is the softnet_lock mutex.
It looks like someone has missed to add this to then path from the
crypto-callback into the network stack in xform_ah.c, xform_esp.c and
xform_ipcomp.c.
I've added the mutex_enter/mutex_exit after/before the
splsoftnet()/splx() in this tree files and the problem in the FAST_IPSEC
parts seems to be fixed.
By the way: in xform_ah.c there was a missing splx() when reentering a
crypto request. This is fixed in the patches below too.
At the moment I've still a problem in the wm-driver with NULL mbuf in
wm_start(), but I'm gooing to look at that separately.
Perhaps it is related to some obsolete debugging code from my tests ....
I need to reinstall the system to validate this.
Perhaps somewhere else there is some locking on softnet_lock missing too.
If the patches below are integrated in 5.x and HEAD, I think this PR can
be closed.
I think this PR is still "critical", because FAST_IPSEC in 5.x and HEAD
is broken and cannot be used without blowing up the kernel.
Best regards
W. Stukenbrock
Here the three patches that solves the problem:
diff -u -r1.1 xform_ah.c
--- xform_ah.c 2011/01/20 12:10:56 1.1
+++ xform_ah.c 2011/01/20 12:13:53
@@ -54,6 +54,8 @@
#include <sys/kernel.h>
#include <sys/sysctl.h>
+#include <sys/socketvar.h>
+
#include <net/if.h>
#include <netinet/in.h>
@@ -829,6 +831,7 @@
#endif
s = splsoftnet();
+ mutex_enter(softnet_lock);
sav = KEY_ALLOCSA(&tc->tc_dst, tc->tc_proto, tc->tc_spi, sport,
dport);
if (sav == NULL) {
@@ -851,8 +854,11 @@
if (sav->tdb_cryptoid != 0)
sav->tdb_cryptoid = crp->crp_sid;
- if (crp->crp_etype == EAGAIN)
+ if (crp->crp_etype == EAGAIN) {
+ mutex_exit(softnet_lock);
+ splx(s);
return crypto_dispatch(crp);
+ }
AH_STATINC(AH_STAT_NOXFORM);
DPRINTF(("ah_input_cb: crypto error %d\n",
crp->crp_etype));
@@ -959,11 +965,13 @@
IPSEC_COMMON_INPUT_CB(m, sav, skip, protoff, mtag);
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
return error;
bad:
if (sav)
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
if (m != NULL)
m_freem(m);
@@ -1247,6 +1255,7 @@
if (crp->crp_etype == EAGAIN) {
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
return crypto_dispatch(crp);
}
@@ -1293,11 +1302,13 @@
/* NB: m is reclaimed by ipsec_process_done. */
err = ipsec_process_done(m, isr);
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
return err;
bad:
if (sav)
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
if (m)
m_freem(m);
diff -u -r1.1 xform_esp.c
--- xform_esp.c 2011/01/20 12:14:23 1.1
+++ xform_esp.c 2011/01/20 12:32:25
@@ -55,6 +55,8 @@
/*#include <sys/random.h>*/
#include <sys/sysctl.h>
+#include <sys/socketvar.h>
+
#include <net/if.h>
#include <netinet/in.h>
@@ -498,6 +500,7 @@
#endif
s = splsoftnet();
+ mutex_enter(softnet_lock);
sav = KEY_ALLOCSA(&tc->tc_dst, tc->tc_proto, tc->tc_spi, sport,
port);
if (sav == NULL) {
@@ -526,6 +529,7 @@
if (crp->crp_etype == EAGAIN) {
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
return crypto_dispatch(crp);
}
@@ -664,11 +668,13 @@
IPSEC_COMMON_INPUT_CB(m, sav, skip, protoff, mtag);
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
return error;
bad:
if (sav)
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
if (m != NULL)
m_freem(m);
@@ -935,6 +941,7 @@
m = (struct mbuf *) crp->crp_buf;
s = splsoftnet();
+ mutex_enter(softnet_lock);
isr = tc->tc_isr;
sav = KEY_ALLOCSA(&tc->tc_dst, tc->tc_proto, tc->tc_spi, 0, 0);
@@ -957,6 +964,7 @@
if (crp->crp_etype == EAGAIN) {
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
return crypto_dispatch(crp);
}
@@ -1003,11 +1011,13 @@
/* NB: m is reclaimed by ipsec_process_done. */
err = ipsec_process_done(m, isr);
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
return err;
bad:
if (sav)
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
if (m)
m_freem(m);
diff -u -r1.2 xform_ipcomp.c
--- xform_ipcomp.c 2010/05/04 13:53:23 1.2
+++ xform_ipcomp.c 2011/01/20 12:12:15
@@ -46,6 +46,8 @@
#include <sys/protosw.h>
#include <sys/sysctl.h>
+#include <sys/socketvar.h>
+
#include <netinet/in.h>
#include <netinet/in_systm.h>
#include <netinet/ip.h>
@@ -258,6 +260,7 @@
#endif
s = splsoftnet();
+ mutex_enter(softnet_lock);
sav = KEY_ALLOCSA(&tc->tc_dst, tc->tc_proto, tc->tc_spi, sport,
dport);
if (sav == NULL) {
@@ -281,6 +284,7 @@
if (crp->crp_etype == EAGAIN) {
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
return crypto_dispatch(crp);
}
@@ -338,11 +342,13 @@
IPSEC_COMMON_INPUT_CB(m, sav, skip, protoff, NULL);
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
return error;
bad:
if (sav)
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
if (m)
m_freem(m);
@@ -513,6 +519,7 @@
rlen = crp->crp_ilen - skip;
s = splsoftnet();
+ mutex_enter(softnet_lock);
isr = tc->tc_isr;
sav = KEY_ALLOCSA(&tc->tc_dst, tc->tc_proto, tc->tc_spi, 0, 0);
@@ -532,6 +539,7 @@
if (crp->crp_etype == EAGAIN) {
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
return crypto_dispatch(crp);
}
@@ -627,11 +635,13 @@
/* NB: m is reclaimed by ipsec_process_done. */
error = ipsec_process_done(m, isr);
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
return error;
bad:
if (sav)
KEY_FREESAV(&sav);
+ mutex_exit(softnet_lock);
splx(s);
if (m)
m_freem(m);
gnats-admin@NetBSD.org wrote:
> Thank you very much for your problem report.
> It has the internal identification `kern/44418'.
> The individual assigned to look at your
> report is: kern-bug-people.
>
>
>>Category: kern
>>Responsible: kern-bug-people
>>Synopsis: FAST_IPSEC and if_wm kernel panic - may affect the whole network stack
>>Arrival-Date: Wed Jan 19 18:55:00 +0000 2011
>>
>
--
Dr. Nagler & Company GmbH
Hauptstraße 9
92253 Schnaittenbach
Tel. +49 9622/71 97-42
Fax +49 9622/71 97-50
Wolfgang.Stukenbrock@nagler-company.com
http://www.nagler-company.com
Hauptsitz: Schnaittenbach
Handelregister: Amberg HRB
Gerichtsstand: Amberg
Steuernummer: 201/118/51825
USt.-ID-Nummer: DE 273143997
Geschäftsführer: Dr. Martin Nagler, Dr. Dr. Karl-Kuno Kunze
From: Wolfgang Stukenbrock <Wolfgang.Stukenbrock@nagler-company.com>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@NetBSD.org, gnats-admin@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: kern/44418: FAST_IPSEC and if_wm kernel panic - may affect the whole network stack
Date: Fri, 21 Jan 2011 14:40:30 +0100
Hi once again ...
sorry but according to my latest test I need to correct the last patch
aproch --- sorry
I've found my problen inside of the wm driver now.
The main problem is, that the network code is not able to handle
processing of more than one CPU at a time.
The introduction of the softnet_lock mutex does not realy help a lot
here ....
There is a very big problem with the device drivers!!!!
The network code does the following:
1. lock softnet_lock
2. lock KERNEL_LOCK
3. increase spl-level if required
The driver interrupt does the following:
1. enter a higher SPL-level
2. lock KERNEL_LOCK
3. call driver interrupt-routine
This is bad, because we now cannot lock the softnet_lock anymore - order
problem.
Next problem here is, that I've got the following calling sequence (in DDB):
db{0}> trace
breakpoint() at netbsd:breakpoint+0x5
panic() at netbsd:panic+0x24d
lockdebug_abort() at netbsd:lockdebug_abort+0x42
mutex_vector_enter() at netbsd:mutex_vector_enter+0x229
wm_intr() at netbsd:wm_intr+0x6ba
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
Xintr_ioapic_level5() at netbsd:Xintr_ioapic_level5+0xf7
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
crypto_getreq() at netbsd:crypto_getreq+0x25
esp_input() at netbsd:esp_input+0x2cf
ipsec_common_input() at netbsd:ipsec_common_input+0x643
ipsec4_common_input() at netbsd:ipsec4_common_input+0xc3
ip_input() at netbsd:ip_input+0x4a3
ipintr() at netbsd:ipintr+0x6a
softint_dispatch() at netbsd:softint_dispatch+0xc3
DDB lost frame for netbsd:Xsoftintr+0x50, trying 0xffff80005382bd70
Xsoftintr() at netbsd:Xsoftintr+0x50
--- interrupt ---
0:
db{0}>
It is possible, that we handle driver interrupts while we are processing
network-packets! And we have locked the softnet_lock in ipintr() and it
is still locked.
(Output comes from one of my aproches to avoid concurrent access of
ipsec-callback and the driver interrupts for the if_snd queue).
The current implementation of the mutex is a little bit problematic from
my point of view, because it cannot be used to synchronise critical
sections with interrupts - at least not if we are holding the lock while
we are processing pending interrupts ...
So currently the only way to avoid concurrent access in any
network/driver-data is to hold the global KERNEL_LOCK.
Not very smart! This is a performance issue ...
I've modified my previous patch, so that the KERNEL_LOCK is locked too
prior reinjection the packet into the network code.
I'm still not 100% shure if no changes to the interfaces drivers are
required.
I'm currently useing only the wm driver and there I've located the
following problem in wm_tick():
It updates some event-counters - OK should be safe as long as the
register access on the HW is MP-safe. Is it ??? I'm not shure ...
But it also calls wm_tbi_check_link() for some chips and that one
may call wm_start() and that accesses if_snd.
Now I'm not shure if the KERNEL_LOCK is hold while processing wm_tick().
A short look at it leads me to the oppinion that it is not!
And if not, there is unsynchronised MP-access to the queue !!!!!
For me, I've added KERNEL_LOCK's to my sources, but I won't issue a PR
for that, because I'm simply not shure if it is realy required.
(same for wm_82547_txfifo_stall() and wm_watchdog())
I think the maintainer of the wm-driver should think a little big about
it and if the synchronisation is not OK, then all drivers should have a
review.
Here the corrected patch for the xfrom_xxx.c files of the FAST_IPSEC
implementation:
--- xform_ah.c 2011/01/20 12:10:56 1.1
+++ xform_ah.c 2011/01/21 13:03:35
@@ -54,6 +54,8 @@
#include <sys/kernel.h>
#include <sys/sysctl.h>
+#include <sys/socketvar.h>
+
#include <net/if.h>
#include <netinet/in.h>
@@ -828,6 +830,8 @@
}
#endif
+ mutex_enter(softnet_lock);
+ KERNEL_LOCK(1,NULL);
s = splsoftnet();
sav = KEY_ALLOCSA(&tc->tc_dst, tc->tc_proto, tc->tc_spi, sport,
dport);
@@ -851,8 +855,12 @@
if (sav->tdb_cryptoid != 0)
sav->tdb_cryptoid = crp->crp_sid;
- if (crp->crp_etype == EAGAIN)
+ if (crp->crp_etype == EAGAIN) {
+ splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
return crypto_dispatch(crp);
+ }
AH_STATINC(AH_STAT_NOXFORM);
DPRINTF(("ah_input_cb: crypto error %d\n",
crp->crp_etype));
@@ -960,11 +968,15 @@
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
return error;
bad:
if (sav)
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
if (m != NULL)
m_freem(m);
if (tc != NULL)
@@ -1228,6 +1240,8 @@
ptr = (tc + 1);
m = (struct mbuf *) crp->crp_buf;
+ mutex_enter(softnet_lock);
+ KERNEL_LOCK(1,NULL);
s = splsoftnet();
isr = tc->tc_isr;
@@ -1248,6 +1262,8 @@
if (crp->crp_etype == EAGAIN) {
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
return crypto_dispatch(crp);
}
@@ -1294,11 +1310,15 @@
err = ipsec_process_done(m, isr);
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
return err;
bad:
if (sav)
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
if (m)
m_freem(m);
free(tc, M_XDATA);
diff -u -r1.1 xform_esp.c
--- xform_esp.c 2011/01/20 12:14:23 1.1
+++ xform_esp.c 2011/01/21 13:04:27
@@ -55,6 +55,8 @@
/*#include <sys/random.h>*/
#include <sys/sysctl.h>
+#include <sys/socketvar.h>
+
#include <net/if.h>
#include <netinet/in.h>
@@ -497,6 +499,8 @@
}
#endif
+ mutex_enter(softnet_lock);
+ KERNEL_LOCK(1,NULL);
s = splsoftnet();
sav = KEY_ALLOCSA(&tc->tc_dst, tc->tc_proto, tc->tc_spi, sport,
dport);
@@ -527,6 +531,8 @@
if (crp->crp_etype == EAGAIN) {
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
return crypto_dispatch(crp);
}
@@ -665,11 +671,15 @@
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
return error;
bad:
if (sav)
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
if (m != NULL)
m_freem(m);
if (tc != NULL)
@@ -934,6 +944,8 @@
IPSEC_ASSERT(tc != NULL, ("esp_output_cb: null opaque data
area!"));
m = (struct mbuf *) crp->crp_buf;
+ mutex_enter(softnet_lock);
+ KERNEL_LOCK(1,NULL);
s = splsoftnet();
isr = tc->tc_isr;
@@ -958,6 +970,8 @@
if (crp->crp_etype == EAGAIN) {
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
return crypto_dispatch(crp);
}
@@ -1004,11 +1018,15 @@
err = ipsec_process_done(m, isr);
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
return err;
bad:
if (sav)
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
if (m)
m_freem(m);
free(tc, M_XDATA);
--- xform_ipcomp.c 2010/05/04 13:25:55 1.1
+++ xform_ipcomp.c 2011/01/21 13:05:37
@@ -46,6 +46,8 @@
#include <sys/protosw.h>
#include <sys/sysctl.h>
+#include <sys/socketvar.h>
+
#include <netinet/in.h>
#include <netinet/in_systm.h>
#include <netinet/ip.h>
@@ -257,6 +259,8 @@
}
#endif
+ mutex_enter(softnet_lock);
+ KERNEL_LOCK(1,NULL);
s = splsoftnet();
sav = KEY_ALLOCSA(&tc->tc_dst, tc->tc_proto, tc->tc_spi, sport,
dport);
@@ -282,6 +286,8 @@
if (crp->crp_etype == EAGAIN) {
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
return crypto_dispatch(crp);
}
@@ -299,6 +305,9 @@
}
IPCOMP_STATINC(IPCOMP_STAT_HIST + sav->alg_comp);
+ /* Update the counters */
+ IPCOMP_STATADD(IPCOMP_STAT_IBYTES, m->m_pkthdr.len - skip);
+
clen = crp->crp_olen; /* Length of data after
processing */
/* Release the crypto descriptors */
@@ -336,11 +345,15 @@
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
return error;
bad:
if (sav)
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
if (m)
m_freem(m);
if (tc != NULL)
@@ -509,6 +522,8 @@
skip = tc->tc_skip;
rlen = crp->crp_ilen - skip;
+ mutex_enter(softnet_lock);
+ KERNEL_LOCK(1,NULL);
s = splsoftnet();
isr = tc->tc_isr;
@@ -530,6 +545,8 @@
if (crp->crp_etype == EAGAIN) {
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
return crypto_dispatch(crp);
}
IPCOMP_STATINC(IPCOMP_STAT_NOXFORM);
@@ -625,11 +642,15 @@
error = ipsec_process_done(m, isr);
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
return error;
bad:
if (sav)
KEY_FREESAV(&sav);
splx(s);
+ KERNEL_UNLOCK_ONE(NULL);
+ mutex_exit(softnet_lock);
if (m)
m_freem(m);
free(tc, M_XDATA);
From: "Matthias Drochner" <drochner@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/44418 CVS commit: src/sys/netipsec
Date: Thu, 10 Feb 2011 20:24:27 +0000
Module Name: src
Committed By: drochner
Date: Thu Feb 10 20:24:27 UTC 2011
Modified Files:
src/sys/netipsec: ipsec_output.c xform_ah.c xform_esp.c xform_ipcomp.c
Log Message:
-in opencrypto callbacks (which run in a kernel thread), pull softnet_lock
everywhere splsoftnet() was used before, to fix MP concurrency problems
-pull KERNEL_LOCK where ip(6)_output() is called, as this is what
the network stack (unfortunately) expects, in particular to avoid
races for packets in the interface send queues
From Wolfgang Stukenbrock per PR kern/44418, with the application
of KERNEL_LOCK to what I think are the essential points, tested
on a dual-core i386.
To generate a diff of this commit:
cvs rdiff -u -r1.29 -r1.30 src/sys/netipsec/ipsec_output.c
cvs rdiff -u -r1.26 -r1.27 src/sys/netipsec/xform_ah.c
cvs rdiff -u -r1.22 -r1.23 src/sys/netipsec/xform_esp.c
cvs rdiff -u -r1.20 -r1.21 src/sys/netipsec/xform_ipcomp.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->feedback
State-Changed-By: drochner@NetBSD.org
State-Changed-When: Thu, 10 Feb 2011 21:28:08 +0000
State-Changed-Why:
added acquisition of softnet_lock as suggested in the PR, added
KERNEL_LOCK differently, only for ip_output.
For me, a dual-CPU box is solid.
From: Wolfgang Stukenbrock <Wolfgang.Stukenbrock@nagler-company.com>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@NetBSD.org, netbsd-bugs@NetBSD.org, gnats-admin@NetBSD.org,
drochner@NetBSD.org, Wolfgang.Stukenbrock@nagler-company.com
Subject: Re: kern/44418 (FAST_IPSEC and if_wm kernel panic - may affect the whole network stack)
Date: Fri, 11 Feb 2011 15:03:18 +0100
Hi,
I've had a look at it again and with the aproach to aquire the
KENREL_LOOK in ipsec_reinject_ipstack() this way of dooing it seems to
be OK.
But I've just recognised another very small whole in this patch.
(Also included in my original posting - sorry)
When the softnet_lock is aquired, the spl-level should have been already
raised to splnet/splsoftnet!
Otherwise we may hold the lock and an Interrupt on splnet/splsoftnet
could be taken.
In that interrupt we may try to lock the softnet_lock on the same CPU
again - resulting in panic ...
This whole is of cause very small, so the change to stumble over it is
very very small.
Same problem when freeing the lock again.
The order of splx()/splsoftnet() and mutex_enter()/mutex_exit() should
be changed.
If that is corrected too, I think the PR can be closed.
Best regards.
W. Stukenbrock
drochner@NetBSD.org wrote:
> Synopsis: FAST_IPSEC and if_wm kernel panic - may affect the whole network stack
>
> State-Changed-From-To: open->feedback
> State-Changed-By: drochner@NetBSD.org
> State-Changed-When: Thu, 10 Feb 2011 21:28:08 +0000
> State-Changed-Why:
> added acquisition of softnet_lock as suggested in the PR, added
> KERNEL_LOCK differently, only for ip_output.
> For me, a dual-CPU box is solid.
>
>
>
>
--
Dr. Nagler & Company GmbH
Hauptstraße 9
92253 Schnaittenbach
Tel. +49 9622/71 97-42
Fax +49 9622/71 97-50
Wolfgang.Stukenbrock@nagler-company.com
http://www.nagler-company.com
Hauptsitz: Schnaittenbach
Handelregister: Amberg HRB
Gerichtsstand: Amberg
Steuernummer: 201/118/51825
USt.-ID-Nummer: DE 273143997
Geschäftsführer: Dr. Martin Nagler, Dr. Dr. Karl-Kuno Kunze
From: Matthias Drochner <M.Drochner@fz-juelich.de>
To: Wolfgang Stukenbrock <Wolfgang.Stukenbrock@nagler-company.com>
Cc: <gnats-bugs@NetBSD.org>, <kern-bug-people@NetBSD.org>,
<netbsd-bugs@NetBSD.org>, <gnats-admin@NetBSD.org>,
<drochner@NetBSD.org>
Subject: Re: kern/44418 (FAST_IPSEC and if_wm kernel panic - may affect the
whole network stack)
Date: Fri, 11 Feb 2011 18:39:54 +0100
Wolfgang.Stukenbrock@nagler-company.com said:
> When the softnet_lock is aquired, the spl-level should have been
> already raised to splnet/splsoftnet!
I'm not sure this is a problem here because softnet_lock is
an adaptive mutex.
best regards
Matthias
---------------------------------------------------------------------------=
---------------------
---------------------------------------------------------------------------=
---------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDirig Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
---------------------------------------------------------------------------=
---------------------
---------------------------------------------------------------------------=
---------------------
From: Wolfgang Stukenbrock <Wolfgang.Stukenbrock@nagler-company.com>
To: M.Drochner@fz-juelich.de
Cc: Wolfgang Stukenbrock <Wolfgang.Stukenbrock@nagler-company.com>,
gnats-bugs@NetBSD.org, kern-bug-people@NetBSD.org,
netbsd-bugs@NetBSD.org, gnats-admin@NetBSD.org, drochner@NetBSD.org
Subject: Re: kern/44418 (FAST_IPSEC and if_wm kernel panic - may affect the whole network stack)
Date: Mon, 14 Feb 2011 09:16:41 +0100
I think it is.
During testing I've got a panic when a CPU tries to get the mutex twice.
And if the spl-level does not lock out network interrupts, this may
happen. Even it the whole is very small ...
best regards
W. Stukenbrock
Matthias Drochner wrote:
> Wolfgang.Stukenbrock@nagler-company.com said:
>
>>When the softnet_lock is aquired, the spl-level should have been
>>already raised to splnet/splsoftnet!
>>
>
> I'm not sure this is a problem here because softnet_lock is
> an adaptive mutex.
>
> best regards
> Matthias
>
>
>
> ------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDirig Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
> Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
> ------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------
>
From: Matthias Drochner <M.Drochner@fz-juelich.de>
To: Wolfgang Stukenbrock <Wolfgang.Stukenbrock@nagler-company.com>
Cc: <gnats-bugs@NetBSD.org>, <kern-bug-people@NetBSD.org>,
<netbsd-bugs@NetBSD.org>, <gnats-admin@NetBSD.org>
Subject: Re: kern/44418 (FAST_IPSEC and if_wm kernel panic - may affect the
whole network stack)
Date: Mon, 14 Feb 2011 14:54:45 +0100
Wolfgang.Stukenbrock@nagler-company.com said:
> During testing I've got a panic when a CPU tries to get the mutex
> twice.
OK, so I've reversed the locking order.
> And if the spl-level does not lock out network interrupts, this may
> happen. Even it the whole is very small ...
This is still somewhat strange... The mutex is an adaptive one,
which cannot be taken in interrupt handlers at all. Only in
the softint handler. As I understand it, it is also OK for an
adaptive mutex to be attempted to be taken a second time,
even if held by the same CPU (just not by the same thread).
I can only suspect that the panic might be related to the
limited thread context of the softint handler.
If this is the case it would be an unnecessary limitation.
Can you post the exact panic message and traceback,
just to help to understand the issue?
best regards
Matthias
---------------------------------------------------------------------------=
---------------------
---------------------------------------------------------------------------=
---------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDirig Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
---------------------------------------------------------------------------=
---------------------
---------------------------------------------------------------------------=
---------------------
From: Wolfgang Stukenbrock <Wolfgang.Stukenbrock@nagler-company.com>
To: M.Drochner@fz-juelich.de
Cc: Wolfgang Stukenbrock <Wolfgang.Stukenbrock@nagler-company.com>,
gnats-bugs@NetBSD.org, kern-bug-people@NetBSD.org,
netbsd-bugs@NetBSD.org, gnats-admin@NetBSD.org
Subject: Re: kern/44418 (FAST_IPSEC and if_wm kernel panic - may affect the whole network stack)
Date: Mon, 14 Feb 2011 17:38:09 +0100
Hi,
sorry I cannot post it, because I haven't kept it.
I cannot say if it was a real interrupt or a soft interrupt.
The stack in DDB was some like this:
- opencrypto ret-thread starts processing
- I've added a lock to softnet_lock there, but does not increase SPL-Level
- An interrupt was there next and it panics in mutex_enter(softnet_lock).
I think it was ipintr(), but I'm not shure anymore - sorry
It had something to do with incomming packet processing.
And the panic was triggered by an ASSERT as far as I remember - not a
"normal" crash due to bad pointers or so...
After adding splsoftnet() (and getting the KERNEL_LOCK) to the path in
opencrypto ret-thread, the problem disapears. That's why I remember that
I must be carefull with mutex_enter() calls and softnet_lock may not be
locked mutliple times on different paths on the same CPU as the
KERNEL_LOCK may be.
I haven't digged deeper into that - simply assumed that the
mutex_enter/exit() stuff behaves like a posix-mutex - will also fail if
locked twice in the "correct" setup.
Best regards
W. Stukenbrock
Matthias Drochner wrote:
> Wolfgang.Stukenbrock@nagler-company.com said:
>
>>During testing I've got a panic when a CPU tries to get the mutex
>>twice.
>>
>
> OK, so I've reversed the locking order.
>
>
>>And if the spl-level does not lock out network interrupts, this may
>>happen. Even it the whole is very small ...
>>
>
> This is still somewhat strange... The mutex is an adaptive one,
> which cannot be taken in interrupt handlers at all. Only in
> the softint handler. As I understand it, it is also OK for an
> adaptive mutex to be attempted to be taken a second time,
> even if held by the same CPU (just not by the same thread).
> I can only suspect that the panic might be related to the
> limited thread context of the softint handler.
> If this is the case it would be an unnecessary limitation.
>
> Can you post the exact panic message and traceback,
> just to help to understand the issue?
>
> best regards
> Matthias
>
>
>
> ------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDirig Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
> Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
> ------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------
>
--
Dr. Nagler & Company GmbH
Hauptstraße 9
92253 Schnaittenbach
Tel. +49 9622/71 97-42
Fax +49 9622/71 97-50
Wolfgang.Stukenbrock@nagler-company.com
http://www.nagler-company.com
Hauptsitz: Schnaittenbach
Handelregister: Amberg HRB
Gerichtsstand: Amberg
Steuernummer: 201/118/51825
USt.-ID-Nummer: DE 273143997
Geschäftsführer: Dr. Martin Nagler, Dr. Dr. Karl-Kuno Kunze
From: Mindaugas Rasiukevicius <rmind@netbsd.org>
To: M.Drochner@fz-juelich.de
Cc: Wolfgang Stukenbrock <Wolfgang.Stukenbrock@nagler-company.com>,
<gnats-bugs@NetBSD.org>, <netbsd-bugs@NetBSD.org>, <gnats-admin@NetBSD.org>
Subject: Re: kern/44418 (FAST_IPSEC and if_wm kernel panic - may affect the
whole network stack)
Date: Fri, 18 Feb 2011 22:22:08 +0000
Matthias Drochner <M.Drochner@fz-juelich.de> wrote:
> This is still somewhat strange... The mutex is an adaptive one,
> which cannot be taken in interrupt handlers at all. Only in
> the softint handler. As I understand it, it is also OK for an
> adaptive mutex to be attempted to be taken a second time,
> even if held by the same CPU (just not by the same thread).
> I can only suspect that the panic might be related to the
> limited thread context of the softint handler.
> If this is the case it would be an unnecessary limitation.
Not sure what do you mean by "second time", but that sounds very wrong.
There can be one lock owner at a time (or lock is not held) - that is a
fundamental of mutual exclusion. It does not behave like kernel-lock,
which gets released/acquired around sleeps. Also, Recursive locking is
deliberately prohibited in our mutex(9) implementation (even in such case,
it would be the same lock owner).
Talking about limited softintr thread context - it may sleep on adaptive
locks, but otherwise blocking is prohibited, since it is very expensive.
Anyway, thanks for working on IPsec code, it really needs a major revamp.
> best regards
> Matthias
--
Mindaugas
From: Matthias Drochner <M.Drochner@fz-juelich.de>
To: Mindaugas Rasiukevicius <rmind@netbsd.org>
Cc: <M.Drochner@fz-juelich.de>, Wolfgang Stukenbrock
<Wolfgang.Stukenbrock@nagler-company.com>, <gnats-bugs@netbsd.org>,
<netbsd-bugs@netbsd.org>, <gnats-admin@netbsd.org>
Subject: Re: kern/44418 (FAST_IPSEC and if_wm kernel panic - may affect the
whole network stack)
Date: Fri, 18 Feb 2011 23:32:14 +0100
rmind@netbsd.org said:
> Not sure what do you mean by "second time"
> [...]
> Recursive locking is
> deliberately prohibited in our mutex(9) implementation
> [...]
> Talking about limited softintr thread context
What I did suspect here is that some kernel code takes softnet_lock
without a splsoftnet() before, and a softnet handler gets to run
and happens to borrow the same thread context. In that case,
the diagnostic check could fire even if the softnet handler
doesn't do anything wrong.
Can this happen?
best regards
Matthias
---------------------------------------------------------------------------=
---------------------
---------------------------------------------------------------------------=
---------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDirig Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
---------------------------------------------------------------------------=
---------------------
---------------------------------------------------------------------------=
---------------------
From: Mindaugas Rasiukevicius <rmind@netbsd.org>
To: <M.Drochner@fz-juelich.de>
Cc: Wolfgang Stukenbrock <Wolfgang.Stukenbrock@nagler-company.com>,
<gnats-bugs@netbsd.org>, <netbsd-bugs@netbsd.org>, <gnats-admin@netbsd.org>
Subject: Re: kern/44418 (FAST_IPSEC and if_wm kernel panic - may affect the
whole network stack)
Date: Sat, 19 Feb 2011 03:52:18 +0000
Matthias Drochner <M.Drochner@fz-juelich.de> wrote:
> > Not sure what do you mean by "second time"
> > [...]
> > Recursive locking is
> > deliberately prohibited in our mutex(9) implementation
> > [...]
> > Talking about limited softintr thread context
>
> What I did suspect here is that some kernel code takes softnet_lock
> without a splsoftnet() before, and a softnet handler gets to run
> and happens to borrow the same thread context. In that case,
> the diagnostic check could fire even if the softnet handler
> doesn't do anything wrong.
> Can this happen?
While soft-interrupt borrows a current thread context - it has a separate
identity, that is, own 'curlwp' value. Therefore, softint would block on
the lock, return execution to the interrupted LWP, and when the lock gets
released - soft-interrupt would continue. In other words - that is not a
condition of recursive lock attempt.
--
Mindaugas
From: Matthias Drochner <M.Drochner@fz-juelich.de>
To: Mindaugas Rasiukevicius <rmind@netbsd.org>
Cc: Wolfgang Stukenbrock <Wolfgang.Stukenbrock@nagler-company.com>,
<gnats-bugs@netbsd.org>, <netbsd-bugs@netbsd.org>,
<gnats-admin@netbsd.org>
Subject: Re: kern/44418 (FAST_IPSEC and if_wm kernel panic - may affect the
whole network stack)
Date: Sat, 19 Feb 2011 13:58:35 +0100
rmind@netbsd.org said:
> While soft-interrupt borrows a current thread context - it has a
> separate identity, that is, own 'curlwp' value
Thanks for the clarification. (I see - it is IS_LWP on i386.)
> In other words - that is not a condition of recursive lock attempt
So the order of splsoftnet() and mutex_enter(softnet_lock)
cannot explain a possible recursive locking attempt, right?
I'm afraid we will not find out where the panic Wolfgang
reported could have come from without more information.
Since the current code works at least for Wolfgang and me
I'll close the PR now.
---------------------------------------------------------------------------=
---------------------
---------------------------------------------------------------------------=
---------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDirig Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
---------------------------------------------------------------------------=
---------------------
---------------------------------------------------------------------------=
---------------------
State-Changed-From-To: feedback->closed
State-Changed-By: drochner@NetBSD.org
State-Changed-When: Sat, 19 Feb 2011 13:04:56 +0000
State-Changed-Why:
while it is not clear whether the panic observed is really related
to the locking order, the current code seems to be stable
From: matthew green <mrg@eterna.com.au>
To: M.Drochner@fz-juelich.de
Cc: Wolfgang Stukenbrock <Wolfgang.Stukenbrock@nagler-company.com>,
gnats-bugs@netbsd.org, netbsd-bugs@netbsd.org,
gnats-admin@netbsd.org, Mindaugas Rasiukevicius <rmind@netbsd.org>
Subject: re: kern/44418 (FAST_IPSEC and if_wm kernel panic - may affect the whole network stack)
Date: Sun, 20 Feb 2011 01:34:35 +1100
> rmind@netbsd.org said:
> > Not sure what do you mean by "second time"
> > [...]
> > Recursive locking is
> > deliberately prohibited in our mutex(9) implementation
> > [...]
> > Talking about limited softintr thread context
>
> What I did suspect here is that some kernel code takes softnet_lock
> without a splsoftnet() before, and a softnet handler gets to run
> and happens to borrow the same thread context. In that case,
> the diagnostic check could fire even if the softnet handler
> doesn't do anything wrong.
> Can this happen?
in sparc smp land, where we've only recently started being stable
again, the last 2 crashes i've seen were softnet_lock issues.
i don't have the details handy, but it invovled slowtimo processing
and softnet_lock being re-taken on the same CPU.
i've wondered the same thing r.e. splsoftnet.
.mrg.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home