NetBSD Problem Report #44069

From gson@gson.org  Mon Nov  8 16:38:39 2010
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by www.NetBSD.org (Postfix) with ESMTP id 5F38C63BAC2
	for <gnats-bugs@gnats.NetBSD.org>; Mon,  8 Nov 2010 16:38:39 +0000 (UTC)
Message-Id: <20101108163835.5DFEB75E92@guava.gson.org>
Date: Mon,  8 Nov 2010 18:38:35 +0200 (EET)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@gnats.NetBSD.org
Cc: toshii@NetBSD.org
Subject: NetBSD-current hangs during boot under Linux kvm emulation
X-Send-Pr-Version: 3.95

>Number:         44069
>Category:       kern
>Synopsis:       NetBSD-current hangs during boot under Linux kvm emulation
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Nov 08 16:40:00 +0000 2010
>Closed-Date:    Thu Oct 06 14:53:36 +0000 2016
>Last-Modified:  Thu Oct 06 14:53:36 +0000 2016
>Originator:     Andreas Gustafsson
>Release:        NetBSD-current >= 2009.11.04.14.39.17
>Organization:
>Environment:
System: NetBSD
Architecture: i386
Machine: i386
>Description:

Attempting to boot NetBSD-current/i386 in a Linux KVM virtual machine
fails.  The kernel locks up after printing the line

  attimer0: attached to pcppi0

On some Linux systems (e.g., Ubuntu 10.04), qemu will use KVM
virtualization by default, so this problem also affects booting NetBSD
under qemu on those systems unless the qemu command line option
"--no-kvm" is used.

Booting versions of -current with CVS source date older than
2009.11.04.14.39.17 works (the exact date was identified using
automated binary serach).  The problem was introduced, or perhaps
exposed, by following commit by toshii:

Index: mpacpi.c
===================================================================
RCS file: /cvsroot/src/sys/arch/x86/x86/mpacpi.c,v
retrieving revision 1.78
retrieving revision 1.79
diff -u -r1.78 -r1.79
--- mpacpi.c    16 Sep 2009 10:47:54 -0000      1.78
+++ mpacpi.c    4 Nov 2009 14:39:17 -0000       1.79
@@ -1069,7 +1069,7 @@
                arg.Type = ACPI_TYPE_INTEGER;
                arg.Integer.Value = 1;  /* I/O APIC (0 = PIC, 2 = IOSAPIC) */
                rv = AcpiEvaluateObject(NULL, "\\_PIC", &arglist, NULL);
-               if (ACPI_FAILURE(rv)) {
+               if (ACPI_FAILURE(rv) && rv != AE_NOT_FOUND) {
                        if (mp_verbose)
                                printf("mpacpi: switch to APIC mode failed\n");
                        return 0;


>How-To-Repeat:

On a Linux system (e.g., Ubuntu 10.04 amd64), attempt to boot a NetBSD-current/i386
disk image using

  kvm -hda disk_image_file

Alternatively, you can install anita from http://www.gson.org/netbsd/anita/
on the Linux system, and then use it to attempt to boot a recent releng snapshot,
e.g.,

  anita boot http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/201011080000Z/i386/

>Fix:

Reverting the above commit should fix (or perhaps hide) the problem,
but I don't understand the code in question and can't tell if that
would break something else.

>Release-Note:

>Audit-Trail:
From: Jukka Ruohonen <jruohonen@iki.fi>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/44069: NetBSD-current hangs during boot under Linux kvm emulation
Date: Mon, 8 Nov 2010 20:10:36 +0200

 zOn Mon, Nov 08, 2010 at 04:40:01PM +0000, Andreas Gustafsson wrote:
 > Booting versions of -current with CVS source date older than
 > 2009.11.04.14.39.17 works (the exact date was identified using
 > automated binary serach).  The problem was introduced, or perhaps
 > exposed, by following commit by toshii:
 > 
 > Index: mpacpi.c
 > ===================================================================
 > RCS file: /cvsroot/src/sys/arch/x86/x86/mpacpi.c,v
 > retrieving revision 1.78
 > retrieving revision 1.79
 > diff -u -r1.78 -r1.79
 > --- mpacpi.c    16 Sep 2009 10:47:54 -0000      1.78
 > +++ mpacpi.c    4 Nov 2009 14:39:17 -0000       1.79
 > @@ -1069,7 +1069,7 @@
 >                 arg.Type = ACPI_TYPE_INTEGER;
 >                 arg.Integer.Value = 1;  /* I/O APIC (0 = PIC, 2 = IOSAPIC) */
 >                 rv = AcpiEvaluateObject(NULL, "\\_PIC", &arglist, NULL);
 > -               if (ACPI_FAILURE(rv)) {
 > +               if (ACPI_FAILURE(rv) && rv != AE_NOT_FOUND) {
 >                         if (mp_verbose)
 >                                 printf("mpacpi: switch to APIC mode failed\n");
 >                         return 0;

 Interestingly, the commit message tells the exact opposite story:

 "revision 1.79
  date: 2009/11/04 14:39:17;  author: toshii;  state: Exp;  lines: +3 -3

  Don't return an error if the _PIC method isn't found.
  It's an optional method and not found in kvm/qemu.
  ----------------------------"

 To my understanding, the check added by toshii is right. If the _PIC method
 is not available, PIC is assumed by default. The specification is quote
 clear about this:

 "The \_PIC optional method is used to report to the BIOS the current
  interrupt model used by the OS.  This control method returns nothing. The
  argument passed into the method signifies the interrupt model OSPM has
  chosen, PIC mode, APIC mode, or SAPIC mode.  Notice that calling this method
  is optional for OSPM.  If the method is never called, the BIOS must assume
  PIC mode.  It is important that the BIOS save the value passed in by OSPM
  for later use during wake operations."

 Moreover, the check (namely 'if (ACPI_FAILURE(rv) && rv != AE_NOT_FOUND)')
 is exactly the same as in Linux.

 A quick conclusion: I doubt anyone dares to change this highly error-prone
 code path based on the supplied information alone. It works on real hardware.

 Has NetBSD worked on KVM before? Can you post the full dmesg?

 - Jukka.

From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org,
    gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org,
    gson@gson.org (Andreas Gustafsson)
Subject: Re: kern/44069: NetBSD-current hangs during boot under Linux kvm emulation
Date: Mon, 8 Nov 2010 21:31:22 +0200

 Jukka,

 Thanks for taking the time to analyze this issue.

 >  > The problem was introduced, or perhaps
 >  > exposed, by following commit by toshii:
 [...]
 >  Interestingly, the commit message tells the exact opposite story:
 >  
 >  "revision 1.79
 >   date: 2009/11/04 14:39:17;  author: toshii;  state: Exp;  lines: +3 -3
 >  
 >   Don't return an error if the _PIC method isn't found.
 >   It's an optional method and not found in kvm/qemu.
 >   ----------------------------"

 That's not quite the opposite story - it's consistent with the problem
 being exposed by rather than caused by toshii's commit.  For example,
 the hang could happen later in the function, in code that was never
 executed under KVM before the commit, because the function returned
 prematurely when _PIC wasn't found.

 >  Has NetBSD worked on KVM before? 

 As I said in the bug report, versions from before the commit do work,
 or at the very least they boot.  Here's a summary of the source dates
 I have tested:

    2009.08.23.20.57.40 ok
    2009.10.25.23.42.58 ok
    2009.10.29.14.32.18 ok
    2009.11.02.17.17.34 ok
    2009.11.04.13.29.45 ok
    2009.11.04.13.32.39 ok
    2009.11.04.14.39.17 fails
    2009.11.04.15.37.39 fails
    2009.11.04.18.11.11 fails
    2009.11.06.15.41.58 fails
    2009.11.07.19.54.17 fails
    2009.11.15.23.29.18 fails
    2009.11.23.05.01.12 fails
    2009.12.31.15.42.46 fails
    2010.04.25.22.50.11 fails
    2010.07.07.11.42.18 fails

 > Can you post the full dmesg?

 Here's the console output from the 2009.11.04.13.32.39 build booting in kvm.

 Loading /stand/i386/5.99.21/modules/ffs/ffs.kmod |/-\|/-\|/-\|/-\|/-\|/-\|/ 
 Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
     2006, 2007, 2008, 2009
     The NetBSD Foundation, Inc.  All rights reserved.
 Copyright (c) 1982, 1986, 1989, 1991, 1993
     The Regents of the University of California.  All rights reserved.

 NetBSD 5.99.21 (GENERIC) #0: Mon Nov  8 19:13:08  2010
 	gson@guesthouse:/bracket/work/2009.11.04.13.32.39/obj/sys/arch/i386/compile/GENERIC
 total memory = 383 MB
 avail memory = 366 MB
 mainbus0 (root)
 cpu0 at mainbus0 apid 0: Intel 686-class, 2400MHz, id 0x623
 ioapic0 at mainbus0 apid 1
 acpi0 at mainbus0: Intel ACPICA 20090730
 pckbc1 at acpi0 (KBD, PNP0303) (kbd port): io 0x60,0x64 irq 1
 pckbc2 at acpi0 (MOU, PNP0F13) (aux port): irq 12
 FDC0 (PNP0700) at acpi0 not configured
 LPT (PNP0400) at acpi0 not configured
 COM1 (PNP0501) at acpi0 not configured
 hpet0 at acpi0 (HPET, PNP0103-0): mem 0xfed00000-0xfed003ff
 apm0 at acpi0: Power Management spec V1.2
 pckbd0 at pckbc1 (kbd slot)
 pckbc1: using irq 1 for kbd slot
 wskbd0 at pckbd0 mux 1
 pms0 at pckbc1 (aux slot)
 pckbc1: using irq 12 for aux slot
 wsmouse0 at pms0 mux 0
 pci0 at mainbus0 bus 0: configuration mode 1
 pchb0 at pci0 dev 0 function 0: vendor 0x8086 product 0x1237 (rev. 0x02)
 pcib0 at pci0 dev 1 function 0: vendor 0x8086 product 0x7000 (rev. 0x00)
 piixide0 at pci0 dev 1 function 1: Intel 82371SB IDE Interface (PIIX3) (rev. 0x00)
 piixide0: primary channel interrupting at irq 14
 atabus0 at piixide0 channel 0
 piixide0: secondary channel interrupting at irq 15
 atabus1 at piixide0 channel 1
 piixpm0 at pci0 dev 1 function 3
 piixpm0: vendor 0x8086 product 0x7113 (rev. 0x03)
 piixpm0: 24-bit timer
 piixpm0: interrupting at irq 9
 iic0 at piixpm0: I2C bus
 vga1 at pci0 dev 2 function 0: vendor 0x1013 product 0x00b8 (rev. 0x00)
 wsdisplay0 at vga1 kbdmux 1
 drm at vga1 not configured
 re0 at pci0 dev 3 function 0: RealTek 8139C+ 10/100BaseTX (rev. 0x20)
 re0: interrupting at irq 11
 re0: Ethernet address 52:54:00:12:34:56
 rlphy0 at re0 phy 0: Realtek internal PHY
 rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 isa0 at pcib0
 lpt0 at isa0 port 0x378-0x37b irq 7
 com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
 com0: console
 attimer0 at isa0 port 0x40-0x43
 pcppi0 at isa0 port 0x61
 midi0 at pcppi0: PC speaker
 sysbeep0 at pcppi0
 isapnp0 at isa0 port 0x279
 npx0 at isa0 port 0xf0-0xff
 fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
 attimer0: attached to pcppi0
 wd0 at atabus0 drive 0: <QEMU HARDDISK>
 wd0: 512 MB, 1040 cyl, 16 head, 63 sec, 512 bytes/sect x 1048576 sectors
 atapibus0 at atabus1: 2 targets
 cd0 at atapibus0 drive 0: <QEMU DVD-ROM, QM00003, 0.12.3> cdrom removable
 Kernelized RAIDframe activated
 pad0: outputs: 44100Hz, 16-bit, stereo
 audio0 at pad0: half duplex, playback, capture
 boot device: wd0
 root on wd0a dumps on wd0b
 root file system type: ffs
 Mon Nov  8 19:23:50 UTC 2010
 Starting root file system check:
 /dev/rwd0a: file system is clean; not checking
 swapctl: setting dump device to /dev/wd0b
 swapctl: adding /dev/wd0b as swap device at priority 0
 Starting file system checks:
 /dev/rwd0a: file system is mounted read-write on /; not checking
 Setting tty flags.
 Setting sysctl variables:
 ddb.onpanic: 1 -> 0
 Starting network.
 /etc/rc: WARNING: $hostname not set.
 IPv6 mode: host
 Configuring network interfaces:.
 Adding interface aliases:.
 Building databases: dev, utmp, utmpx, services done
 Starting syslogd.
 Mounting all filesystems...
 Clearing temporary files.
 Creating a.out runtime link editor directory cache.
 Checking quotas: done.
 swapctl: setting dump device to /dev/wd0b
 Starting virecover.
 Checking for core dump...
 savecore - - - no core dump
 Starting local daemons:.
 Updating motd.
 postfix: rebuilding /etc/mail/aliases (missing /etc/mail/aliases.db)
 newaliases: warning: valid_hostname: empty hostname
 newaliases: fatal: unable to use my own hostname
 Nov  8 19:23:54  postfix/sendmail[316]: fatal: unable to use my own hostname
 /etc/rc.d/postfix reported failure status 1
 Starting inetd.
 Starting cron.
 The following components reported failures:
     /etc/rc.d/postfix
 See /var/run/rc.log for more information.
 Mon Nov  8 19:23:54 UTC 2010

 NetBSD/i386 (Amnesiac) (console)

 login:

 -- 
 Andreas Gustafsson, gson@gson.org

From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org,
    gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org,
    gson@gson.org (Andreas Gustafsson)
Subject: Re: kern/44069: NetBSD-current hangs during boot under Linux kvm emulation
Date: Mon, 8 Nov 2010 22:52:07 +0200

 For what it's worth, "kvm -no-kvm-irqchip" looks like a viable work-around:
 with that option, a recent -current (2010.11.08.13.05.49) boots.
 -- 
 Andreas Gustafsson, gson@gson.org

From: Jukka Ruohonen <jruohonen@iki.fi>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/44069: NetBSD-current hangs during boot under Linux kvm emulation
Date: Tue, 9 Nov 2010 03:54:34 +0200

 On Mon, Nov 08, 2010 at 07:35:02PM +0000, Andreas Gustafsson wrote:
 >  >  Has NetBSD worked on KVM before? 
 >  
 >  As I said in the bug report, versions from before the commit do work,
 >  or at the very least they boot.  Here's a summary of the source dates

 What I meant to ask is: has this revision worked on KVM before? That is,
 whether this regression is ours or theirs.

 - Jukka.

From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org,
    gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org,
    gson@gson.org (Andreas Gustafsson)
Subject: Re: kern/44069: NetBSD-current hangs during boot under Linux kvm emulation
Date: Tue, 9 Nov 2010 11:36:39 +0200

 Jukka Ruohonen wrote:
 >  What I meant to ask is: has this revision worked on KVM before? 

 All my test results were obtained using the same version of KVM
 (qemu-kvm 0.12.3 and the kvm.ko in the Ubuntu 2.6.32-25-server
 kernel); the NetBSD versions tested were all recently built from
 date-based checkouts of historic sources.

 I don't know whether or not the versions of NetBSD that failed to boot
 with this version of KVM would boot with older versions of KVM, and I
 don't have an easy way to test that.

 > That is, whether this regression is ours or theirs.

 There is a regression on our part, in the sense that a newer version
 of NetBSD fails where an older version works, everything else being
 equal.  Lacking an easy way of testing old KVM versions, I can't say
 whether or not there is also a regression on KVM's part.
 -- 
 Andreas Gustafsson, gson@gson.org

From: "Andreas Gustafsson" <gson@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/44069 CVS commit: pkgsrc/misc/py-anita
Date: Thu, 25 Nov 2010 14:44:20 +0000

 Module Name:	pkgsrc
 Committed By:	gson
 Date:		Thu Nov 25 14:44:20 UTC 2010

 Modified Files:
 	pkgsrc/misc/py-anita: Makefile distinfo

 Log Message:
 Update misc/py-anita from 1.15 to 1.17.  Changes:

 Update the discussion of Linux KVM virtualization compatibility in the
 BUGS section, noting the --qemu-args -no-kvm-irqchip workaround and
 including a reference to PR 44069.

 New command line option "--sets".

 Add a rudimentary INSTALL file.

 The "misc" set is now installed by default, as it is required for
 the ATF XML DTD, XSL files, and style sheets.

 The "test" command now exports the raw and XML format ATF output
 to the host, along with some auxiliary files needed for further
 processing of the XML output.

 Add support for FreeBSD hosts, from Kurt Lidl.


 To generate a diff of this commit:
 cvs rdiff -u -r1.12 -r1.13 pkgsrc/misc/py-anita/Makefile
 cvs rdiff -u -r1.10 -r1.11 pkgsrc/misc/py-anita/distinfo

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: Jukka Ruohonen <jruohonen@iki.fi>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/44069
Date: Thu, 26 Apr 2012 11:57:57 +0300

 For cross-references: see also PR kern/46380: -no-kvm option to qemu works
 also there.

From: "Jukka Ruohonen" <jruoho@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/44069 CVS commit: src/sys/arch/x86/x86
Date: Thu, 26 Apr 2012 10:15:43 +0000

 Module Name:	src
 Committed By:	jruoho
 Date:		Thu Apr 26 10:15:42 UTC 2012

 Modified Files:
 	src/sys/arch/x86/x86: mpacpi.c

 Log Message:
 Based on PR kern/44069, revert revision 1.79.

 XXX: The actual problem related to Qemu/KVM is yet to be determined.


 To generate a diff of this commit:
 cvs rdiff -u -r1.92 -r1.93 src/sys/arch/x86/x86/mpacpi.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: Emmanuel Kasper <emmanuel@libera.cc>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Subject: Re: kern/44069
Date: Tue, 18 Sep 2012 22:31:18 +0200

 I suffer from this bug as well:
 http://mail-index.netbsd.org/netbsd-users/2012/09/15/msg011423.html


 maybe would it be interesting to see if this bug is specific to a KVM
 release ?

 it seems to be specially the 0.12.3 version
 as your ubuntu 10.04 has probably 0.12.3 according to
 http://packages.ubuntu.com/lucid/qemu-kvmand my hosted NetBSD VPS also
 0.12.3 according to cpuctl

 cpuctl identify 0 | grep QEMU
 cpu0: "QEMU Virtual CPU version 0.12.3"

 there is no changelog for qemu 0.13 but
 http://www.linux-kvm.com/content/qemu-kvm-0130-released

 reads:

 hpet: many enhancements
 ...

State-Changed-From-To: open->feedback
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Sun, 02 Oct 2016 18:59:22 +0000
State-Changed-Why:
Is this problem still current?


From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org,
    netbsd-bugs@netbsd.org,
    gnats-admin@netbsd.org,
    dholland@NetBSD.org
Subject: Re: kern/44069 (NetBSD-current hangs during boot under Linux kvm emulation)
Date: Mon, 3 Oct 2016 20:41:31 +0300

 dholland@NetBSD.org wrote:
 > Is this problem still current?

 It's complicated.  The boot no longer hangs since jruoho reverted
 mpacpi.c 1.79 in 2012, but cause of he problem remains unidentified
 and may well have been hidden rather than fixed by the reversion.

 I suppose I could try reapplying mpacpi.c 1.79 in -current and
 retesting on a recent KVM, but that will have to wait a bit
 because my build machine is now busy bisecting another, more
 recent bug...
 -- 
 Andreas Gustafsson, gson@gson.org

From: Andreas Gustafsson <gson@gson.org>
To: dholland@NetBSD.org
Cc: gnats-bugs@NetBSD.org,
    kern-bug-people@netbsd.org,
    gnats-admin@netbsd.org
Subject: Re: kern/44069 (NetBSD-current hangs during boot under Linux kvm emulation)
Date: Thu, 6 Oct 2016 17:46:04 +0300

 On Monday, I wrote:
 >  It's complicated.  The boot no longer hangs since jruoho reverted
 >  mpacpi.c 1.79 in 2012, but cause of he problem remains unidentified
 >  and may well have been hidden rather than fixed by the reversion.

 I had a closer look and found that although jruoho did revert mpacpi.c
 1.79 in 1.93, he reapplied the change the following day in 1.94.  So
 the code in case is now again the same as when the problem first
 appeared, yet the problem no longer occurs.  Therefore I must conclude
 that the problem has been fixed, whatever it was.
 -- 
 Andreas Gustafsson, gson@gson.org

State-Changed-From-To: feedback->closed
State-Changed-By: gson@NetBSD.org
State-Changed-When: Thu, 06 Oct 2016 14:53:36 +0000
State-Changed-Why:
Problem no longer occurs with current NetBSD and KVM versions.


>Unformatted:

Home	PR Database Search