NetBSD Problem Report #49007

From www@NetBSD.org  Thu Jul 17 10:09:42 2014
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 4B12DA655B
	for <gnats-bugs@gnats.NetBSD.org>; Thu, 17 Jul 2014 10:09:42 +0000 (UTC)
Message-Id: <20140717100940.D41BAA6561@mollari.NetBSD.org>
Date: Thu, 17 Jul 2014 10:09:40 +0000 (UTC)
From: ozaki-r@netbsd.org
Reply-To: ozaki-r@netbsd.org
To: gnats-bugs@NetBSD.org
Subject: if_vioif panic when ifconfig up/down under load
X-Send-Pr-Version: www-1.0

>Number:         49007
>Category:       kern
>Synopsis:       if_vioif panic when ifconfig up/down under load
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Jul 17 10:10:00 +0000 2014
>Closed-Date:    Fri Aug 08 02:18:31 +0000 2014
>Last-Modified:  Fri Aug 08 02:18:31 +0000 2014
>Originator:     Ryota Ozaki
>Release:        current (and 6)
>Organization:
>Environment:
NetBSD kvm 6.99.30 NetBSD 6.99.30 (GENERIC) #0: Sun Jan 26 19:39:39 UTC 2014  builds@b8.netbsd.org:/home/builds/ab/HEAD/amd64/201401261800Z-obj/home/builds/ab/HEAD/src/sys/arch/amd64/compile/GENERIC amd64
and
NetBSD kvm 6.99.47 NetBSD 6.99.47 (KVM) #27: Thu Jul 17 17:38:39 JST 2014  ozaki-r@hidden:/hidden/

Linux fedora20 3.14.7-200.fc20.x86_64 #1 SMP Wed Jun 11 22:38:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
qemu-kvm-2.1.0-0.3.rc1.fc20.x86_64
>Description:
vioif_populate_rx_mbufs is not MPSAFE, however, it may run in parallel
on different CPUs and cause panic.

It is called via vioif_ioctl and vioif_rx_softint. vioif_ioctl runs with the
big kernel lock while vioif_rx_softint runs without it because
SOFTINT_MPSAFE is specified. OTOH, vioif_ioctl can run on an arbitrary
CPU while vioif_rx_softint runs only on CPU#0. Eventually,
vioif_populate_rx_mbufs can run on different CPUs simultaneously
and cause panic. Even worse, it kills qemu sometimes.
>How-To-Repeat:
Attach two virtio NICs to a VM (qemu) and run NetBSD (current or 6)
on it.

[VM]
Create a bridge and a tap and attach vioif0, vioif1 and tap0 to
bridge0. Assign some IP to tap0 and run an iperf server on it.

[outside]
Run an iperf server on one side of a VM NIC. Run two iperf clients;
one sends packets to the iperf server in the VM and the other
sends packets to the iperf server outside via the bridge in the VM.

[VM]
ifconfig vioif0 up/down repeatedly.

A simpler configuration may reproduce the problem, but the above
configuration can reproduce it quickly.

>Fix:
cvs -q diff -u sys/dev/pci/if_vioif.c                                                                                                                [src]
Index: sys/dev/pci/if_vioif.c
===================================================================
RCS file: /cvs/cvsroot/src/sys/dev/pci/if_vioif.c,v
retrieving revision 1.4
diff -u -r1.4 if_vioif.c
--- sys/dev/pci/if_vioif.c      9 May 2013 12:23:17 -0000       1.4
+++ sys/dev/pci/if_vioif.c      17 Jul 2014 09:10:13 -0000
@@ -557,7 +557,7 @@
                }
        }

-       sc->sc_rx_softint = softint_establish(SOFTINT_NET|SOFTINT_MPSAFE,
+       sc->sc_rx_softint = softint_establish(SOFTINT_NET,
                                              vioif_rx_softint, sc);
        if (sc->sc_rx_softint == NULL) {
                aprint_error_dev(self, "cannot establish softint\n");

>Release-Note:

>Audit-Trail:
From: "Ryota Ozaki" <ozaki-r@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/49007 CVS commit: src/sys/dev/pci
Date: Fri, 18 Jul 2014 02:10:55 +0000

 Module Name:	src
 Committed By:	ozaki-r
 Date:		Fri Jul 18 02:10:55 UTC 2014

 Modified Files:
 	src/sys/dev/pci: if_vioif.c

 Log Message:
 Don't set SOFTINT_MPSAFE to vioif_rx_softint

 vioif_rx_softint calls vioif_populate_rx_mbufs that is not MPSAFE.
 vioif_populate_rx_mbufs is also called via vioif_ioctl and so can
 be called by two LWPs simultaneously, resulting in kernel panic.

 PR kern/49007


 To generate a diff of this commit:
 cvs rdiff -u -r1.4 -r1.5 src/sys/dev/pci/if_vioif.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->pending-pullups
State-Changed-By: ozaki-r@NetBSD.org
State-Changed-When: Tue, 22 Jul 2014 01:17:57 +0000
State-Changed-Why:
Ticket #1102


From: "SAITOH Masanobu" <msaitoh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/49007 CVS commit: [netbsd-6] src/sys/dev/pci
Date: Thu, 7 Aug 2014 09:31:09 +0000

 Module Name:	src
 Committed By:	msaitoh
 Date:		Thu Aug  7 09:31:09 UTC 2014

 Modified Files:
 	src/sys/dev/pci [netbsd-6]: if_vioif.c

 Log Message:
 Pull up following revision(s) (requested by ozaki-r in ticket #1102):
 	sys/dev/pci/if_vioif.c: revision 1.5
 Don't set SOFTINT_MPSAFE to vioif_rx_softint
 vioif_rx_softint calls vioif_populate_rx_mbufs that is not MPSAFE.
 vioif_populate_rx_mbufs is also called via vioif_ioctl and so can
 be called by two LWPs simultaneously, resulting in kernel panic.
 PR kern/49007


 To generate a diff of this commit:
 cvs rdiff -u -r1.2.8.1 -r1.2.8.2 src/sys/dev/pci/if_vioif.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "SAITOH Masanobu" <msaitoh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/49007 CVS commit: [netbsd-6-1] src/sys/dev/pci
Date: Thu, 7 Aug 2014 09:34:21 +0000

 Module Name:	src
 Committed By:	msaitoh
 Date:		Thu Aug  7 09:34:21 UTC 2014

 Modified Files:
 	src/sys/dev/pci [netbsd-6-1]: if_vioif.c

 Log Message:
 Pull up following revision(s) (requested by ozaki-r in ticket #1102):
 	sys/dev/pci/if_vioif.c: revision 1.5
 Don't set SOFTINT_MPSAFE to vioif_rx_softint
 vioif_rx_softint calls vioif_populate_rx_mbufs that is not MPSAFE.
 vioif_populate_rx_mbufs is also called via vioif_ioctl and so can
 be called by two LWPs simultaneously, resulting in kernel panic.
 PR kern/49007


 To generate a diff of this commit:
 cvs rdiff -u -r1.2.8.1 -r1.2.8.1.2.1 src/sys/dev/pci/if_vioif.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "SAITOH Masanobu" <msaitoh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/49007 CVS commit: [netbsd-6-0] src/sys/dev/pci
Date: Thu, 7 Aug 2014 09:35:22 +0000

 Module Name:	src
 Committed By:	msaitoh
 Date:		Thu Aug  7 09:35:22 UTC 2014

 Modified Files:
 	src/sys/dev/pci [netbsd-6-0]: if_vioif.c

 Log Message:
 Pull up following revision(s) (requested by ozaki-r in ticket #1102):
 	sys/dev/pci/if_vioif.c: revision 1.5
 Don't set SOFTINT_MPSAFE to vioif_rx_softint
 vioif_rx_softint calls vioif_populate_rx_mbufs that is not MPSAFE.
 vioif_populate_rx_mbufs is also called via vioif_ioctl and so can
 be called by two LWPs simultaneously, resulting in kernel panic.
 PR kern/49007


 To generate a diff of this commit:
 cvs rdiff -u -r1.2.14.1 -r1.2.14.2 src/sys/dev/pci/if_vioif.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: pending-pullups->closed
State-Changed-By: ozaki-r@NetBSD.org
State-Changed-When: Fri, 08 Aug 2014 02:18:31 +0000
State-Changed-Why:
pullup done


>Unformatted:

Home	PR Database Search