NetBSD Problem Report #53980

From gson@gson.org  Thu Feb 14 15:47:02 2019
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 890787A154
	for <gnats-bugs@gnats.NetBSD.org>; Thu, 14 Feb 2019 15:47:02 +0000 (UTC)
Message-Id: <20190214140621.02F789892F2@guava.gson.org>
Date: Thu, 14 Feb 2019 16:06:21 +0200 (EET)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@NetBSD.org
Subject: INSTALL_XEN3PAE_DOMU from -current panics
X-Send-Pr-Version: 3.95

>Number:         53980
>Category:       port-xen
>Synopsis:       INSTALL_XEN3PAE_DOMU from -current panics
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    port-xen-maintainer
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Feb 14 15:50:00 +0000 2019
>Closed-Date:    Sat Apr 27 14:26:05 +0000 2019
>Last-Modified:  Sun Apr 28 12:45:01 +0000 2019
>Originator:     Andreas Gustafsson
>Release:        NetBSD-current, source date 2019.02.14.08.18.26
>Organization:
>Environment:
System: NetBSD
Architecture: i386
Machine: i386
>Description:

Attempting to install a -current i386 domU under xen411 as part of a
suite of automated tests of NetBSD/Xen, the INSTALL_XEN3PAE_DOMU
kernel paniced:

[   1.0000000] NetBSD 8.99.34 (INSTALL_XEN3PAE_DOMU) #0: Thu Feb 14 13:18:51 EET 2019
[   1.0000000]  root@guido.araneus.fi:/tmp/bracket/build/2019.02.14.08.18.26-i386/obj/sys/arch/i386/compile/INSTALL_XEN3PAE_DOMU
[   1.0000000] total memory = 32768 KB
[   1.0000000] avail memory = 18856 KB
[   1.0000000] running cgd selftest aes-xts-256 aes-xts-512 done
[   1.0000000] mainbus0 (root)
[   1.0000000] hypervisor0 at mainbus0: Xen version 4.11.0nb1
[   1.0000000] vcpu0 at hypervisor0
[   1.0000000] vcpu0: Intel(R) Xeon(R) CPU           L5630  @ 2.13GHz, id 0x206c2
[   1.0000000] vcpu0: package 1, core 0, smt 0
[   1.0000000] xenbus0 at hypervisor0: Xen Virtual Bus Interface
[   1.0000000] xencons0 at hypervisor0: Xen Virtual Console Driver
[   1.0000030] xenbus: can't get state for device/suspend/event-channel (2)
[   1.0000030] xbd0 at xenbus0 id 51712: Xen Virtual Block Device Interface
[   1.0100745] xbd1 at xenbus0 id 51728: Xen Virtual Block Device Interface
[   1.0200695] balloon0 at xenbus0 id 0: Xen Balloon driver
[   1.0200695] balloon0: current reservation: 32768 KiB
[   1.0200695] panic: kernel diagnostic assertion "bp != NULL" failed: file "/tmp/bracket/build/2019.02.14.08.18.26-i386/src/sys/arch/xen/xen/xbd_xenbus.c", line 706 
[   1.0200695] cpu0: Begin traceback...
[   1.0200695] vpanic(c0554a6c,c347ce28,c347ce64,c01393d1,c0554a6c,c0554a06,c0599d1a,c055aac4,2c2,1) at netbsd:vpanic+0x13f
[   1.0200695] kern_assert(c0554a6c,c0554a06,c0599d1a,c055aac4,2c2,1,0,c0ee2160,0,c3507000) at netbsd:kern_assert+0x23
[   1.0200695] xbd_handler(c0ee2008,c0d0e000,6,c0e5b2a8,c347cebc,c011a89f,c0e5b2a8,c347cf14,c0b19bc0,f9b46430) at netbsd:xbd_handler+0x3c1
[   1.0200695] xen_intr_biglock_wrapper(c0e5b2a8,c347cf14,c0b19bc0,f9b46430,cc,f9b8a970,0,80000000,0,c0c259b8) at netbsd:xen_intr_biglock_wrapper+0x1f
[   1.0200695] evtchn_do_event(6,c347cf14,c347cf58,1,c347cef0,0,c0b0b0c0,c0d0e000,c0d0e000,c0d0e800) at netbsd:evtchn_do_event+0x10f
[   1.0200695] do_hypervisor_callback(c347cf14,0,c0110011,c0b00031,c0e60011,c3470011,c0b0b0c0,c0e3ad40,c347cf74,deadf00d) at netbsd:do_hypervisor_callback+0x15c
[   1.0200695] Xhypervisor_pvhvm_callback(c0e3cd20,0,c0e3cd20,c0e3cd20,c0102102,c0e3cd20,c03ad7c0,c0e3cd20,0,c0102031) at netbsd:Xhypervisor_pvhvm_callback+0x67
[   1.0200695] idle_loop(c0e3cd20,d0e000,c0b22200,0,c010007a,0,0,0,0,0) at netbsd:idle_loop+0xbf
[   1.0200695] cpu0: End traceback...
[   1.0200695] fatal breakpoint trap in supervisor mode
[   1.0200695] trap type 1 code 0 eip 0xc0105634 cs 0x9 eflags 0x202 cr2 0 ilevel 0x8 esp 0xc347ce0c
[   1.0200695] curlwp 0xc0e3cd20 pid 0 lid 2 lowest kstack 0xc347a2c0
Stopped in pid 0.2 (system) at  netbsd:breakpoint+0x4:  popl    %ebp
breakpoint(c05548fc,c0c65f80,c0554a6c,c347ce28,c0ee2008,c055aac4,c0599d1a,c347ce
1c,c0525203,c0554a6c) at netbsd:breakpoint+0x4
vpanic(c0554a6c,c347ce28,c347ce64,c01393d1,c0554a6c,c0554a06,c0599d1a,c055aac4,2
c2,1) at netbsd:vpanic+0x13f
kern_assert(c0554a6c,c0554a06,c0599d1a,c055aac4,2c2,1,0,c0ee2160,0,c3507000) at n
etbsd:kern_assert+0x23
xbd_handler(c0ee2008,c0d0e000,6,c0e5b2a8,c347cebc,c011a89f,c0e5b2a8,c347cf14,c0b
19bc0,f9b46430) at netbsd:xbd_handler+0x3c1
xen_intr_biglock_wrapper(c0e5b2a8,c347cf14,c0b19bc0,f9b46430,cc,f9b8a970,0,80000
000,0,c0c259b8) at netbsd:xen_intr_biglock_wrapper+0x1f
evtchn_do_event(6,c347cf14,c347cf58,1,c347cef0,0,c0b0b0c0,c0d0e000,c0d0e000,c0d0
e800) at netbsd:evtchn_do_event+0x10f
do_hypervisor_callback(c347cf14,0,c0110011,c0b00031,c0e60011,c3470011,c0b0b0c0,c
0e3ad40,c347cf74,deadf00d) at netbsd:do_hypervisor_callback+0x15c
Xhypervisor_pvhvm_callback(c0e3cd20,0,c0e3cd20,c0e3cd20,c0102102,c0e3cd20,c03ad7
c0,c0e3cd20,0,c0102031) at netbsd:Xhypervisor_pvhvm_callback+0x67
idle_loop(c0e3cd20,d0e000,c0b22200,0,c010007a,0,0,0,0,0) at netbsd:idle_loop+0xb
f
ds          c3470011
es          c0550011    static_ltree+0x91
fs          31
gs          c3470011
edi         c347ce28
esi         c0554a6c    ostype+0xb04
ebp         c347cde8
ebx         104
edx         1
ecx         6
eax         0
eip         c0105634    breakpoint+0x4
cs          9
eflags      202
esp         c347cde8
ss          11
netbsd:breakpoint+0x4:  popl    %ebp
db{0}> 

>How-To-Repeat:

>Fix:

>Release-Note:

>Audit-Trail:
From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@NetBSD.org
Cc: cherry@NetBSD.org
Subject: Re: kern/53980: INSTALL_XEN3PAE_DOMU from -current panics
Date: Mon, 25 Feb 2019 22:41:51 +0200

 Today's current (src and pkgsrc from 2019.02.25.13.19.14) still panics
 the same way.
 -- 
 Andreas Gustafsson, gson@gson.org

From: Manuel Bouyer <bouyer@antioche.eu.org>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
        Andreas Gustafsson <gson@gson.org>
Subject: Re: kern/53980: INSTALL_XEN3PAE_DOMU from -current panics
Date: Mon, 25 Feb 2019 22:12:38 +0100

 On Mon, Feb 25, 2019 at 08:45:01PM +0000, Andreas Gustafsson wrote:
 > The following reply was made to PR kern/53980; it has been noted by GNATS.
 > 
 > From: Andreas Gustafsson <gson@gson.org>
 > To: gnats-bugs@NetBSD.org
 > Cc: cherry@NetBSD.org
 > Subject: Re: kern/53980: INSTALL_XEN3PAE_DOMU from -current panics
 > Date: Mon, 25 Feb 2019 22:41:51 +0200
 > 
 >  Today's current (src and pkgsrc from 2019.02.25.13.19.14) still panics
 >  the same way.

 FWIW, I'm not seeing it on my testbed:
 http://www-soc.lip6.fr/~bouyer/NetBSD-tests/xen/HEAD/http://www-soc.lip6.fr/~bouyer/NetBSD-tests/xen/HEAD/

 so it could be a race condition, or some other host-dependant issue ...

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

From: Manuel Bouyer <bouyer@antioche.eu.org>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/53980: INSTALL_XEN3PAE_DOMU from -current panics
Date: Fri, 26 Apr 2019 10:48:41 +0200

 On Thu, Feb 14, 2019 at 03:50:00PM +0000, Andreas Gustafsson wrote:
 > >Number:         53980
 > >Category:       kern
 > >Synopsis:       INSTALL_XEN3PAE_DOMU from -current panics
 > >Confidential:   no
 > >Severity:       serious
 > >Priority:       high
 > >Responsible:    kern-bug-people
 > >State:          open
 > >Class:          sw-bug
 > >Submitter-Id:   net
 > >Arrival-Date:   Thu Feb 14 15:50:00 +0000 2019
 > >Originator:     Andreas Gustafsson
 > >Release:        NetBSD-current, source date 2019.02.14.08.18.26
 > >Organization:
 > >Environment:
 > System: NetBSD
 > Architecture: i386
 > Machine: i386
 > >Description:
 > 
 > Attempting to install a -current i386 domU under xen411 as part of a
 > suite of automated tests of NetBSD/Xen, the INSTALL_XEN3PAE_DOMU
 > kernel paniced:

 What is your dom0 ?

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

From: Andreas Gustafsson <gson@gson.org>
To: Manuel Bouyer <bouyer@antioche.eu.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/53980: INSTALL_XEN3PAE_DOMU from -current panics
Date: Fri, 26 Apr 2019 12:11:53 +0300

 Manuel Bouyer wrote:
 >  What is your dom0 ?

 NetBSD-current/amd64 built from sources dated 2019.02.14.08.18.26,
 the same source date as the crashing i386PAE domU.

 Full logs including the dom0 console output are at:

  http://www.gson.org/netbsd/bugs/xen/results/2019-02-14/index.html

 -- 
 Andreas Gustafsson, gson@gson.org

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@netbsd.org
Cc: Andreas Gustafsson <gson@gson.org>
Subject: Re: kern/53980: INSTALL_XEN3PAE_DOMU from -current panics
Date: Fri, 26 Apr 2019 11:19:37 +0200

 On Fri, Apr 26, 2019 at 09:15:01AM +0000, Andreas Gustafsson wrote:
 >  NetBSD-current/amd64 built from sources dated 2019.02.14.08.18.26,
 >  the same source date as the crashing i386PAE domU.

 You need a newer dom0, see port-xen/54099.

 Martin

State-Changed-From-To: open->feedback
State-Changed-By: bouyer@NetBSD.org
State-Changed-When: Fri, 26 Apr 2019 09:33:57 +0000
State-Changed-Why:
Please try a kernel including
http://mail-index.netbsd.org/source-changes/2019/04/07/msg104860.html


Responsible-Changed-From-To: kern-bug-people->port-xen-maintainer
Responsible-Changed-By: bouyer@NetBSD.org
Responsible-Changed-When: Fri, 26 Apr 2019 09:36:12 +0000
Responsible-Changed-Why:
port-xen problem


From: Manuel Bouyer <bouyer@antioche.eu.org>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/53980: INSTALL_XEN3PAE_DOMU from -current panics
Date: Fri, 26 Apr 2019 11:33:07 +0200

 On Fri, Apr 26, 2019 at 12:11:53PM +0300, Andreas Gustafsson wrote:
 > Manuel Bouyer wrote:
 > >  What is your dom0 ?
 > 
 > NetBSD-current/amd64 built from sources dated 2019.02.14.08.18.26,
 > the same source date as the crashing i386PAE domU.

 So please try a newer dom0.
 It may be fixed by this commit:
 http://mail-index.netbsd.org/source-changes/2019/04/07/msg104860.html

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

From: Andreas Gustafsson <gson@gson.org>
To: Martin Husemann <martin@duskware.de>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/53980: INSTALL_XEN3PAE_DOMU from -current panics
Date: Sat, 27 Apr 2019 17:21:29 +0300

 Martin Husemann wrote:
 > You need a newer dom0, see port-xen/54099.

 With dom0 and domU both from 2019.04.27.02.00.12, the
 INSTALL_XEN3PAE_DOMU kernel boots:

   http://www.gson.org/netbsd/bugs/xen/results/2019-04-27/data-411-current-2019.04.27.02.00.12-amd64/current-2019.04.27.02.00.12-i386.log

 Installation still fails, but that's a different bug.
 -- 
 Andreas Gustafsson, gson@gson.org

State-Changed-From-To: feedback->closed
State-Changed-By: gson@NetBSD.org
State-Changed-When: Sat, 27 Apr 2019 14:26:05 +0000
State-Changed-Why:
The kernel now boots - thanks.


From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/53980: INSTALL_XEN3PAE_DOMU from -current panics
Date: Sun, 28 Apr 2019 15:43:46 +0300

 Earlier, I wrote:
 > Installation still fails, but that's a different bug.

 After increasing the amount of memory for the i386 domU from
 32M to 512M, installation also works:

   http://www.gson.org/netbsd/bugs/xen/results/2019-04-28/index.html

 -- 
 Andreas Gustafsson, gson@gson.org

>Unformatted:

Home	PR Database Search