NetBSD Problem Report #48105

From www@NetBSD.org  Mon Aug  5 19:21:46 2013
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id E275570FCD
	for <gnats-bugs@gnats.NetBSD.org>; Mon,  5 Aug 2013 19:21:46 +0000 (UTC)
Message-Id: <20130805192145.6FE5370FCE@mollari.NetBSD.org>
Date: Mon,  5 Aug 2013 19:21:45 +0000 (UTC)
From: rhansen@bbn.com
Reply-To: rhansen@bbn.com
To: gnats-bugs@NetBSD.org
Subject: panic at shutdown from NULL pointer dereference caused by virtio_detach()
X-Send-Pr-Version: www-1.0

>Number:         48105
>Category:       kern
>Synopsis:       panic at shutdown from NULL pointer dereference caused by virtio_detach()
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    tsutsui
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Aug 05 19:25:00 +0000 2013
>Closed-Date:    Mon Oct 07 04:09:24 +0000 2013
>Last-Modified:  Mon Oct 07 04:09:24 +0000 2013
>Originator:     Richard Hansen
>Release:        6.0.1
>Organization:
BBN
>Environment:
NetBSD netbsd-vm.bbn.com 6.0.1 NetBSD 6.0.1 (GENERIC) i386
>Description:
NetBSD virtual machine guests that have the 0x10031af4 PCI device (vendor ID 0x1af4 = Qumranet, device ID 0x1003 = Virtio) panic on shutdown due to a NULL pointer dereference in intr_disestablish() at src/sys/arch/x86/x86/intr.c line 901.

Console output when it panics:

uvm_fault(0xc4fc5438, 0, 1) -> 0xe
fatal page fault in supervisor mode
trap type 6 code 0 eip c04c915a cs 8 eflags 10246 cr2 24 ilevel 0
Skipping crash dump on recursive panic
panic: trap
cpu0: Begin traceback...
printf_nolog(c0ba9fab,dcfb7aa4,dcfb7aa4,c04c915a,8,10246,24,0,c07869a3,c4fca040)
 at netbsd:printf_nolog
trap_tss() at netbsd:trap_tss
--- trap via task gate ---
netbsd:cpu_lock:
cpu0: End traceback...
rebooting...

Going up the call stack, virtio_detach() passes the NULL pointer to pci_intr_disestablish() at src/sys/dev/pci/virtio.c line 202.

sc->sc_ih is NULL in virtio_detach() because it is never set in virtio_attach().  virtio_attach() hits an error case and returns early at src/sys/dev/pci/virtio.c line 158.  sc->sc_ih isn't set until line 173.

Evidence of the early return from virtio_attach() is in dmesg:

$ dmesg | grep virtio1
virtio1 at pci0 dev 6 function 0
virtio1: Virtio Console Device (rev. 0x00)
virtio1: no matching child driver; not configured

$ sudo pcictl /dev/pci0 list -n | grep 6:0
000:06:0: 0x10031af4 (0x078000000)
>How-To-Repeat:
1. On an amd64 Ubuntu 13.04 host system, install the following
   packages (and their dependencies):
     * libvirt-bin
     * ubuntu-virt
2. Set up libvirt (see
   <https://help.ubuntu.com/community/KVM/Installation>)
3. Using virt-manager, create a new i386 guest machine.  Select the
   defaults everywhere except on the last page of the wizard:
     a. expand Advanced options
     b. change Architecture from x86_64 to i686
4. Install i386 NetBSD 6.0.1 on the guest
5. Run 'sudo shutdown -p now'
6. Watch it panic and reboot rather than power off

To confirm that the problem only happens when the 0x10031af4 PCI
device is installed in the guest VM:

1. Using virt-manager, edit the guest VM settings
2. Remove the "Controller Virtio Serial" PCI device
3. Boot the VM
4. Run 'sudo shutdown -p now'
5. The VM will power off as expected

>Fix:
Only call pci_intr_disestablish() from virtio_detach() if the arguments are non-NULL?

>Release-Note:

>Audit-Trail:
From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@NetBSD.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: Re: kern/48105: panic at shutdown from NULL pointer dereference caused
	 by virtio_detach()
Date: Wed, 7 Aug 2013 21:22:44 +0900

 > >Synopsis:       panic at shutdown from NULL pointer dereference caused by virtio_detach()
  :
 > >Environment:
 > NetBSD netbsd-vm.bbn.com 6.0.1 NetBSD 6.0.1 (GENERIC) i386

 Probably already fixed in 6.1 and 6.0.2?
 http://www.nerv.org/~ryo/netbsd/netbsd/?q=id:20130509T122318Z.39bd3df7a6f2722fa3b984d9ee3b65ee2622b9e4
 http://releng.netbsd.org/cgi-bin/req-6.cgi?show=890

 ---
 Izumi Tsutsui

From: Richard Hansen <rhansen@bbn.com>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/48105: panic at shutdown from NULL pointer dereference caused
 by virtio_detach()
Date: Wed, 07 Aug 2013 14:02:32 -0400

 On 2013-08-07 08:25, Izumi Tsutsui wrote:
 >  Probably already fixed in 6.1 and 6.0.2?

 Sadly, no.  I just tried 6.1 and the problem still exists.


 The instructions above for reproducing the problem are incorrect.  Here
 are the correct instructions:

   1. On an amd64 Ubuntu 13.04 host system, install the following
      packages (and their dependencies):
        * libvirt-bin
        * ubuntu-virt
        * python-spice-client-gtk
   2. Set up libvirt (see
      <https://help.ubuntu.com/community/KVM/Installation>)
   3. Using virt-manager, create a new i386 guest machine.  Select the
      defaults everywhere except on the last page of the wizard:
        a. expand Advanced options
        b. change Architecture from x86_64 to i686
   4. Install i386 NetBSD 6.1 on the guest
   5. Shut down the guest
   6. Using virt-manager, modify the VM settings to add a "Controller
      Virtio Serial" PCI device.  You can't manually add that device via
      the 'Add Hardware' button for some reason, so you have to follow
      these steps:
        a. select 'Display VNC'
        b. change Type from VNC to Spice
        c. click Apply
        d. when it asks "You are switching graphics type to spice, would
           you like to add Spice agent channels?" click Yes
        e. change Type back to VNC
        f. click Apply
        g. when it says "You are switching graphics type to vnc, would
           you like to remove Spice agent channels?" click Yes
   7. Boot the guest VM
   8. As root, run 'shutdown -p now'
   9. Watch it panic and reboot rather than power off

 To confirm that the problem goes away when the 0x10031af4 PCI
 device is removed from the guest VM:

   1. Using virt-manager, edit the guest VM settings
   2. Select 'Controller Virtio Serial'
   3. Click Remove.  When it asks for confirmation hit Yes.
   4. Boot the VM
   5. As root, run 'shutdown -p now'
   6. The VM will power off as expected


 -Richard

From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@NetBSD.org
Cc: rhansen@bbn.com, tsutsui@ceres.dti.ne.jp
Subject: Re: kern/48105: panic at shutdown from NULL pointer dereference causedby
	 virtio_detach()
Date: Fri, 9 Aug 2013 20:12:56 +0900

 Ah, I'm a bit stupid.
 Your first analysis seems correct and it's enough to add a check
 sc->sc_ih != NULL in a detach function.
 Could you confirm the following patch?

 Index: virtio.c
 ===================================================================
 RCS file: /cvsroot/src/sys/dev/pci/virtio.c,v
 retrieving revision 1.3
 diff -u -p -r1.3 virtio.c
 --- virtio.c	2 Nov 2011 23:05:52 -0000	1.3
 +++ virtio.c	9 Aug 2013 11:12:18 -0000
 @@ -199,8 +199,10 @@ virtio_detach(device_t self, int flags)
  	}
  	KASSERT(sc->sc_child == 0 || sc->sc_child == (void*)1);
  	KASSERT(sc->sc_vqs == 0);
 -	pci_intr_disestablish(sc->sc_pc, sc->sc_ih);
 -	sc->sc_ih = 0;
 +	if (sc->sc_ih != NULL) {
 +		pci_intr_disestablish(sc->sc_pc, sc->sc_ih);
 +		sc->sc_ih = NULL;
 +	}
  	if (sc->sc_iosize)
  		bus_space_unmap(sc->sc_iot, sc->sc_ioh, sc->sc_iosize);
  	sc->sc_iosize = 0;

 ---

 Thanks,

 ---
 Izumi Tsutsui

From: Richard Hansen <rhansen@bbn.com>
To: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/48105: panic at shutdown from NULL pointer dereference causedby
 virtio_detach()
Date: Fri, 09 Aug 2013 20:23:48 -0400

 On 2013-08-09 07:12, Izumi Tsutsui wrote:
 > Your first analysis seems correct and it's enough to add a check
 > sc->sc_ih != NULL in a detach function.
 > Could you confirm the following patch?

 Yes, it works.

 Thanks,
 Richard

From: "Izumi Tsutsui" <tsutsui@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/48105 CVS commit: src/sys/dev/pci
Date: Sat, 10 Aug 2013 18:01:31 +0000

 Module Name:	src
 Committed By:	tsutsui
 Date:		Sat Aug 10 18:01:31 UTC 2013

 Modified Files:
 	src/sys/dev/pci: virtio.c

 Log Message:
 Make sure to check if the driver has a valid intr handler in virtio_detach().

 Fixes a panic during shutdown on KVM under ubuntu 13.04 with virtio,
 as reported in PR kern/48105 by Richard Hansen.

 Should be pulled up to netbsd-6 branches.


 To generate a diff of this commit:
 cvs rdiff -u -r1.3 -r1.4 src/sys/dev/pci/virtio.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

Responsible-Changed-From-To: kern-bug-people->tsutsui
Responsible-Changed-By: tsutsui@NetBSD.org
Responsible-Changed-When: Sat, 10 Aug 2013 18:13:11 +0000
Responsible-Changed-Why:
I'll send a pullup request later.


State-Changed-From-To: open->pending-pullups
State-Changed-By: tsutsui@NetBSD.org
State-Changed-When: Mon, 12 Aug 2013 15:38:06 +0000
State-Changed-Why:
pullup-6 #931


From: "Manuel Bouyer" <bouyer@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/48105 CVS commit: [netbsd-6] src/sys/dev/pci
Date: Sat, 7 Sep 2013 16:01:03 +0000

 Module Name:	src
 Committed By:	bouyer
 Date:		Sat Sep  7 16:01:03 UTC 2013

 Modified Files:
 	src/sys/dev/pci [netbsd-6]: virtio.c

 Log Message:
 Pull up following revision(s) (requested by tsutsui in ticket #931):
 	sys/dev/pci/virtio.c: revision 1.4
 Make sure to check if the driver has a valid intr handler in virtio_detach().
 Fixes a panic during shutdown on KVM under ubuntu 13.04 with virtio,
 as reported in PR kern/48105 by Richard Hansen.
 Should be pulled up to netbsd-6 branches.


 To generate a diff of this commit:
 cvs rdiff -u -r1.3 -r1.3.8.1 src/sys/dev/pci/virtio.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Manuel Bouyer" <bouyer@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/48105 CVS commit: [netbsd-6-0] src/sys/dev/pci
Date: Sat, 7 Sep 2013 16:01:25 +0000

 Module Name:	src
 Committed By:	bouyer
 Date:		Sat Sep  7 16:01:25 UTC 2013

 Modified Files:
 	src/sys/dev/pci [netbsd-6-0]: virtio.c

 Log Message:
 Pull up following revision(s) (requested by tsutsui in ticket #931):
 	sys/dev/pci/virtio.c: revision 1.4
 Make sure to check if the driver has a valid intr handler in virtio_detach().
 Fixes a panic during shutdown on KVM under ubuntu 13.04 with virtio,
 as reported in PR kern/48105 by Richard Hansen.
 Should be pulled up to netbsd-6 branches.


 To generate a diff of this commit:
 cvs rdiff -u -r1.3 -r1.3.14.1 src/sys/dev/pci/virtio.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Manuel Bouyer" <bouyer@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/48105 CVS commit: [netbsd-6-1] src/sys/dev/pci
Date: Sat, 7 Sep 2013 16:01:28 +0000

 Module Name:	src
 Committed By:	bouyer
 Date:		Sat Sep  7 16:01:28 UTC 2013

 Modified Files:
 	src/sys/dev/pci [netbsd-6-1]: virtio.c

 Log Message:
 Pull up following revision(s) (requested by tsutsui in ticket #931):
 	sys/dev/pci/virtio.c: revision 1.4
 Make sure to check if the driver has a valid intr handler in virtio_detach().
 Fixes a panic during shutdown on KVM under ubuntu 13.04 with virtio,
 as reported in PR kern/48105 by Richard Hansen.
 Should be pulled up to netbsd-6 branches.


 To generate a diff of this commit:
 cvs rdiff -u -r1.3 -r1.3.22.1 src/sys/dev/pci/virtio.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: pending-pullups->closed
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Mon, 07 Oct 2013 04:09:24 +0000
State-Changed-Why:
pullups completed last month


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.