NetBSD Problem Report #51467
From mlelstv@tazz.1st.de Sun Sep 11 15:14:17 2016
Return-Path: <mlelstv@tazz.1st.de>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
by mollari.NetBSD.org (Postfix) with ESMTPS id 637307A106
for <gnats-bugs@gnats.NetBSD.org>; Sun, 11 Sep 2016 15:14:17 +0000 (UTC)
Message-Id: <20160911151353.AB49F26A08@tazz.1st.de>
Date: Sun, 11 Sep 2016 17:13:53 +0200 (CEST)
From: mlelstv@serpens.de
Reply-To: mlelstv@serpens.de
To: gnats-bugs@NetBSD.org
Subject: detaching USB network interface panics
X-Send-Pr-Version: 3.95
>Number: 51467
>Category: kern
>Synopsis: detaching USB network interface panics
>Confidential: no
>Severity: critical
>Priority: medium
>Responsible: kern-bug-people
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sun Sep 11 15:15:00 +0000 2016
>Closed-Date: Mon Dec 05 17:24:56 +0000 2016
>Last-Modified: Mon Dec 05 17:24:56 +0000 2016
>Originator: Michael van Elst
>Release: NetBSD 7.99.36
>Organization:
>Environment:
System: NetBSD tazz 7.99.36 NetBSD 7.99.36 (TAZZ) #7: Sun Sep 11 16:12:19 CEST 2016 mlelstv@gossam:/home/netbsd-current/obj.amd64/home/netbsd-current/src/sys/arch/amd64/compile/TAZZ amd64
Architecture: x86_64
Machine: amd64
>Description:
Detaching a configured run(4) interface resulted in a panic.
DDB shows:
find_pfxlist_reachable_router.isra.6() at find_pfxlist_reachable_router.isra.6+0x11
pfxlist_onlink_check() at pfxlist_onlink_check+0x132
nd6_purge() at nd6_purge+0x103
in6_ifdetach() at in6_ifdetach+0x1b
udp6_purgeif_wrapper() at udp6_purgeif_wrapper+0x39
if_detach() at if_detach+0x264
run_detach() at run_detach+0x75
config_detach() at config_detach+0xf8
usb_disconnect_port() at usb_disconnect_port+0x18b
uhub_explore() at uhub_explore+0x1fe
uhub_explore() at uhub_explore+0xac
usb_discover() at usb_discover+0x6f
usb_event_thread() at usb_event_thread+0x238
GDB shows more detail:
#8 0xffffffff80566be7 in find_pfxlist_reachable_router (pr=<optimized out>)
at /home/netbsd-current/src/sys/netinet6/nd6_rtr.c:1416
#9 0xffffffff80567a07 in pfxlist_onlink_check () at /home/netbsd-current/src/sys/netinet6/nd6_rtr.c:1576
#10 0xffffffff8056800c in prelist_remove (pr=<optimized out>)
at /home/netbsd-current/src/sys/netinet6/nd6_rtr.c:1058
#11 0xffffffff805610d6 in nd6_purge (ifp=ifp@entry=0xffff8000070f3008, ext=0xfffffe811e16aab8, ext@entry=0x0)
at /home/netbsd-current/src/sys/netinet6/nd6.c:866
#12 0xffffffff8054cf7f in in6_ifdetach (ifp=ifp@entry=0xffff8000070f3008)
at /home/netbsd-current/src/sys/netinet6/in6_ifattach.c:815
0xffffffff80566bd6 <find_pfxlist_reachable_router>: push %rbp
0xffffffff80566bd7 <find_pfxlist_reachable_router+1>: mov %rsp,%rbp
0xffffffff80566bda <find_pfxlist_reachable_router+4>: push %rbx
0xffffffff80566bdb <find_pfxlist_reachable_router+5>: sub $0x8,%rsp
0xffffffff80566bdf <find_pfxlist_reachable_router+9>: mov %rdi,%rbx
0xffffffff80566be2 <find_pfxlist_reachable_router+12>: test %rdi,%rdi
0xffffffff80566be5 <find_pfxlist_reachable_router+15>: je 0xffffffff80566c38 <find_pfxlist_reachable_router+98>
=> 0xffffffff80566be7 <find_pfxlist_reachable_router+17>: mov 0x10(%rbx),%rdi
0xffffffff80566beb <find_pfxlist_reachable_router+21>: mov 0x30(%rdi),%rsi
0xffffffff80566bef <find_pfxlist_reachable_router+25>: testb $0x1,0x4c(%rsi)
rbx 0x2587e94bac0e70d2 2704386612477718738
which is garbage.
>How-To-Repeat:
Detach a USB network interface that has IPv6 configured.
>Fix:
>Release-Note:
>Audit-Trail:
From: Ryota Ozaki <ozaki-r@netbsd.org>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/51467: detaching USB network interface panics
Date: Mon, 12 Sep 2016 12:59:59 +0900
On Mon, Sep 12, 2016 at 12:15 AM, <mlelstv@serpens.de> wrote:
>>Number: 51467
>>Category: kern
>>Synopsis: detaching USB network interface panics
>>Confidential: no
>>Severity: critical
>>Priority: medium
>>Responsible: kern-bug-people
>>State: open
>>Class: sw-bug
>>Submitter-Id: net
>>Arrival-Date: Sun Sep 11 15:15:00 +0000 2016
>>Originator: Michael van Elst
>>Release: NetBSD 7.99.36
>>Organization:
>
>>Environment:
>
>
> System: NetBSD tazz 7.99.36 NetBSD 7.99.36 (TAZZ) #7: Sun Sep 11 16:12:19 CEST 2016 mlelstv@gossam:/home/netbsd-current/obj.amd64/home/netbsd-current/src/sys/arch/amd64/compile/TAZZ amd64
> Architecture: x86_64
> Machine: amd64
>>Description:
> Detaching a configured run(4) interface resulted in a panic.
>
> DDB shows:
>
> find_pfxlist_reachable_router.isra.6() at find_pfxlist_reachable_router.isra.6+0x11
> pfxlist_onlink_check() at pfxlist_onlink_check+0x132
> nd6_purge() at nd6_purge+0x103
> in6_ifdetach() at in6_ifdetach+0x1b
> udp6_purgeif_wrapper() at udp6_purgeif_wrapper+0x39
> if_detach() at if_detach+0x264
> run_detach() at run_detach+0x75
> config_detach() at config_detach+0xf8
> usb_disconnect_port() at usb_disconnect_port+0x18b
> uhub_explore() at uhub_explore+0x1fe
> uhub_explore() at uhub_explore+0xac
> usb_discover() at usb_discover+0x6f
> usb_event_thread() at usb_event_thread+0x238
>
> GDB shows more detail:
>
> #8 0xffffffff80566be7 in find_pfxlist_reachable_router (pr=<optimized out>)
> at /home/netbsd-current/src/sys/netinet6/nd6_rtr.c:1416
> #9 0xffffffff80567a07 in pfxlist_onlink_check () at /home/netbsd-current/src/sys/netinet6/nd6_rtr.c:1576
> #10 0xffffffff8056800c in prelist_remove (pr=<optimized out>)
> at /home/netbsd-current/src/sys/netinet6/nd6_rtr.c:1058
> #11 0xffffffff805610d6 in nd6_purge (ifp=ifp@entry=0xffff8000070f3008, ext=0xfffffe811e16aab8, ext@entry=0x0)
> at /home/netbsd-current/src/sys/netinet6/nd6.c:866
> #12 0xffffffff8054cf7f in in6_ifdetach (ifp=ifp@entry=0xffff8000070f3008)
> at /home/netbsd-current/src/sys/netinet6/in6_ifattach.c:815
>
> 0xffffffff80566bd6 <find_pfxlist_reachable_router>: push %rbp
> 0xffffffff80566bd7 <find_pfxlist_reachable_router+1>: mov %rsp,%rbp
> 0xffffffff80566bda <find_pfxlist_reachable_router+4>: push %rbx
> 0xffffffff80566bdb <find_pfxlist_reachable_router+5>: sub $0x8,%rsp
> 0xffffffff80566bdf <find_pfxlist_reachable_router+9>: mov %rdi,%rbx
> 0xffffffff80566be2 <find_pfxlist_reachable_router+12>: test %rdi,%rdi
> 0xffffffff80566be5 <find_pfxlist_reachable_router+15>: je 0xffffffff80566c38 <find_pfxlist_reachable_router+98>
> => 0xffffffff80566be7 <find_pfxlist_reachable_router+17>: mov 0x10(%rbx),%rdi
> 0xffffffff80566beb <find_pfxlist_reachable_router+21>: mov 0x30(%rdi),%rsi
> 0xffffffff80566bef <find_pfxlist_reachable_router+25>: testb $0x1,0x4c(%rsi)
>
> rbx 0x2587e94bac0e70d2 2704386612477718738
>
> which is garbage.
>
>
>>How-To-Repeat:
> Detach a USB network interface that has IPv6 configured.
Do you know when it worked lastly?
If degraded by recent changes, one suspect is a change at 8/16(*).
Then could you try a kernel at 8/15?
(*) http://www.nerv.org/netbsd/?q=id:20160816T103157Z.479577018086b726daa7c1600fe5219e96a677d6
ozaki-r
From: Michael van Elst <mlelstv@serpens.de>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/51467 detaching USB network interface panics
Date: Sat, 22 Oct 2016 18:01:06 +0200
The problem seems to be unrelated to USB. With -current 7.99.40 I just
got the following panic during shutdown:
fatal protection fault in supervisor mode
trap type 4 code 0 rip ffffffff8042d087 cs 8 rflags 10282 cr2 ffff800090099000 ilevel 6 rsp fffffe810ec96960
curlwp 0xfffffe8216546220 pid 1759.1 lowest kstack 0xfffffe810ec932c0
kernel: protection fault trap, code=0
Stopped in pid 1759.1 (halt) at
netbsd:find_pfxlist_reachable_router.isra.6+0x11
: movq 10(%rbx),%rdi
db{0}> bt
find_pfxlist_reachable_router.isra.6() at
netbsd:find_pfxlist_reachable_router.isra.6+0x11
pfxlist_onlink_check() at netbsd:pfxlist_onlink_check+0x132
nd6_purge() at netbsd:nd6_purge+0x103
in6_ifdetach() at netbsd:in6_ifdetach+0x1b
udp6_purgeif_wrapper() at netbsd:udp6_purgeif_wrapper+0x39
if_detach() at netbsd:if_detach+0x264
bge_detach() at netbsd:bge_detach+0x6e
config_detach() at netbsd:config_detach+0xf8
config_detach_all() at netbsd:config_detach_all+0x97
cpu_reboot() at netbsd:cpu_reboot+0x174
sys_reboot() at netbsd:sys_reboot+0x75
syscall() at netbsd:syscall+0x164
--- syscall (number 208) ---
7d86fda3e18a:
--
Michael van Elst
Internet: mlelstv@serpens.de
"A potential Snark may lurk in every tree."
State-Changed-From-To: open->closed
State-Changed-By: roy@NetBSD.org
State-Changed-When: Mon, 07 Nov 2016 13:44:25 +0000
State-Changed-Why:
mlelstv indicated this issue was fixed in nd6.c 1.210 Add missing pserialize_read_exit
State-Changed-From-To: closed->open
State-Changed-By: mlelstv@NetBSD.org
State-Changed-When: Sun, 20 Nov 2016 12:10:47 +0000
State-Changed-Why:
the bug is still there
From: Michael van Elst <mlelstv@serpens.de>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/51467 detaching USB network interface panics
Date: Sun, 20 Nov 2016 13:09:41 +0100
The problem re-occured today with a 7.99.42 kernel.
Greetings,
--
Michael van Elst
Internet: mlelstv@serpens.de
"A potential Snark may lurk in every tree."
From: Michael van Elst <mlelstv@serpens.de>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/51467 detaching USB network interface panics
Date: Sun, 27 Nov 2016 10:45:27 +0100
nd6_purge() contains a hack:
/*
* Because if_detach() does *not* release prefixes
* while purging addresses the reference count will
* still be above zero. We therefore reset it to
* make sure that the prefix really gets purged.
*/
pr->ndpr_refcnt = 0;
followed by a call to prelist_remove(pr). This will free the prefix list
although it is still referenced by the interface.
It will also finally call pfxlist_onlink_check().
/*
* Changes on the prefix status might affect address status as well.
* Make sure that all addresses derived from an attached prefix are
* attached, and that all addresses derived from a detached prefix are
* detached. Note, however, that a manually configured address should
* always be attached.
* The precise detection logic is same as the one for prefixes.
*/
Here pfxlist_onlink_check uses the stale ia->ia6_ndpr reference
when calling find_pfxlist_reachable_router().
If the kernel is compiled with DEBUG (implies KMEM_POISON), this
leads to an immediate crash.
Greetings,
--
Michael van Elst
Internet: mlelstv@serpens.de
"A potential Snark may lurk in every tree."
From: Ryota Ozaki <ozaki-r@netbsd.org>
To: Michael van Elst <mlelstv@serpens.de>
Cc: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>, kern-bug-people@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/51467 detaching USB network interface panics
Date: Mon, 28 Nov 2016 18:12:58 +0900
On Sun, Nov 27, 2016 at 6:50 PM, Michael van Elst <mlelstv@serpens.de> wrote:
> The following reply was made to PR kern/51467; it has been noted by GNATS.
>
> From: Michael van Elst <mlelstv@serpens.de>
> To: gnats-bugs@netbsd.org
> Cc:
> Subject: Re: kern/51467 detaching USB network interface panics
> Date: Sun, 27 Nov 2016 10:45:27 +0100
>
> nd6_purge() contains a hack:
>
> /*
> * Because if_detach() does *not* release prefixes
> * while purging addresses the reference count will
> * still be above zero. We therefore reset it to
> * make sure that the prefix really gets purged.
> */
> pr->ndpr_refcnt = 0;
>
> followed by a call to prelist_remove(pr). This will free the prefix list
> although it is still referenced by the interface.
>
> It will also finally call pfxlist_onlink_check().
>
> /*
> * Changes on the prefix status might affect address status as well.
> * Make sure that all addresses derived from an attached prefix are
> * attached, and that all addresses derived from a detached prefix are
> * detached. Note, however, that a manually configured address should
> * always be attached.
> * The precise detection logic is same as the one for prefixes.
> */
>
> Here pfxlist_onlink_check uses the stale ia->ia6_ndpr reference
> when calling find_pfxlist_reachable_router().
>
> If the kernel is compiled with DEBUG (implies KMEM_POISON), this
> leads to an immediate crash.
Thank you for the investigation. I can reproduce it now on an ATF test (t_ra)
by enabling KMEM_POISON by hand (it's never enabled on rump kernels).
The following patch fixes the panic. It restores the original behavior
of in6_purgeif changed by in6.c,v 1.203 and in6_ifattach.c,v 1.99.
Thanks,
ozaki-r
diff --git a/sys/netinet6/in6_ifattach.c b/sys/netinet6/in6_ifattach.c
index 2065d18..0eb8ddb 100644
--- a/sys/netinet6/in6_ifattach.c
+++ b/sys/netinet6/in6_ifattach.c
@@ -809,15 +809,15 @@ void
in6_ifdetach(struct ifnet *ifp)
{
+ /* nuke any of IPv6 addresses we have */
+ if_purgeaddrs(ifp, AF_INET6, in6_purgeaddr);
+
/* remove ip6_mrouter stuff */
ip6_mrouter_detach(ifp);
/* remove neighbor management table */
nd6_purge(ifp, NULL);
- /* nuke any of IPv6 addresses we have */
- if_purgeaddrs(ifp, AF_INET6, in6_purgeaddr);
-
/* cleanup multicast address kludge table, if there is any */
in6_purgemkludge(ifp);
From: Michael van Elst <mlelstv@serpens.de>
To: Ryota Ozaki <ozaki-r@netbsd.org>
Cc: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>,
kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org
Subject: Re: kern/51467 detaching USB network interface panics
Date: Mon, 28 Nov 2016 11:10:47 +0100
On Mon, Nov 28, 2016 at 06:12:58PM +0900, Ryota Ozaki wrote:
> On Sun, Nov 27, 2016 at 6:50 PM, Michael van Elst <mlelstv@serpens.de> wrote:
> > nd6_purge() contains a hack:
> >
> > /*
> > * Because if_detach() does *not* release prefixes
> > * while purging addresses the reference count will
> > * still be above zero. We therefore reset it to
> > * make sure that the prefix really gets purged.
> > */
> > pr->ndpr_refcnt = 0;
Do you think that hack is really necessary?
Greetings,
--
Michael van Elst
Internet: mlelstv@serpens.de
"A potential Snark may lurk in every tree."
From: Ryota Ozaki <ozaki-r@netbsd.org>
To: Michael van Elst <mlelstv@serpens.de>
Cc: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>, kern-bug-people@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/51467 detaching USB network interface panics
Date: Mon, 28 Nov 2016 23:05:17 +0900
On Mon, Nov 28, 2016 at 7:10 PM, Michael van Elst <mlelstv@serpens.de> wrote:
> On Mon, Nov 28, 2016 at 06:12:58PM +0900, Ryota Ozaki wrote:
>> On Sun, Nov 27, 2016 at 6:50 PM, Michael van Elst <mlelstv@serpens.de> wrote:
>> > nd6_purge() contains a hack:
>> >
>> > /*
>> > * Because if_detach() does *not* release prefixes
>> > * while purging addresses the reference count will
>> > * still be above zero. We therefore reset it to
>> > * make sure that the prefix really gets purged.
>> > */
>> > pr->ndpr_refcnt = 0;
>
> Do you think that hack is really necessary?
Maybe not, but I'm not sure for now.
I'll investigate it when I work on MP-ifying the default route list
and the prefix list.
ozaki-r
From: "Ryota Ozaki" <ozaki-r@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/51467 CVS commit: src/sys/netinet6
Date: Wed, 30 Nov 2016 02:08:57 +0000
Module Name: src
Committed By: ozaki-r
Date: Wed Nov 30 02:08:57 UTC 2016
Modified Files:
src/sys/netinet6: in6_ifattach.c
Log Message:
Fix panic on destroying an interface with IPv6 addresses obtained with RA
nd6_purge depends on that IPv6 addresses are purged. If addresses remain,
pfxlist_onlink_check called from nd6_purge dereferences a dangling pointer
(ia->ia6_ndpr) that is freed before calling pfxlist_onlink_check. Fix it by
removing addresses before calling nd6_purge, which is the original behavior
that was changed by in6.c,v 1.203 and in6_ifattach.c,v 1.99.
Note that it seems the issue occurs because of a hack that forcibly destroys
prefix list entries of a given interface in nd6_purge. We should tackle the
hack in the future.
Fix PR kern/51467
To generate a diff of this commit:
cvs rdiff -u -r1.106 -r1.107 src/sys/netinet6/in6_ifattach.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->closed
State-Changed-By: maya@NetBSD.org
State-Changed-When: Mon, 05 Dec 2016 17:24:56 +0000
State-Changed-Why:
Confirmed fixed, thanks ozaki-r
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home