NetBSD Problem Report #53487

From www@NetBSD.org  Tue Jul 31 14:21:40 2018
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 3D7D17A1CC
	for <gnats-bugs@gnats.NetBSD.org>; Tue, 31 Jul 2018 14:21:40 +0000 (UTC)
Message-Id: <20180731142138.AB3EC7A1F0@mollari.NetBSD.org>
Date: Tue, 31 Jul 2018 14:21:38 +0000 (UTC)
From: kiers@xs4all.nl
Reply-To: kiers@xs4all.nl
To: gnats-bugs@NetBSD.org
Subject: xen dom0 crash when reboot of domU
X-Send-Pr-Version: www-1.0

>Number:         53487
>Category:       port-xen
>Synopsis:       xen dom0 crash when reboot of domU
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    jdolecek
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Jul 31 14:25:00 +0000 2018
>Closed-Date:    Wed Aug 15 13:56:45 +0000 2018
>Last-Modified:  Wed Aug 15 13:56:45 +0000 2018
>Originator:     Bert Kiers
>Release:        NetBSD 8.0, Xen 4.11.0nb0
>Organization:
>Environment:
NetBSD esmeralda.quipama.co 8.0 NetBSD 8.0 (XEN3_DOM0) #0: Thu Jul 26 01:09:53 CEST 2018  kiers@yvresse.boppelans.net:/tmp/obj330/sys/arch/amd64/compile/XEN3_DOM0 amd64
>Description:
When I reboot a Linux domU, dom0 crashes:

fatal page fault in supervisor mode
trap type 6 code 0 rip 0xffffffff80239626 cs 0xe030 rflags 0x10206 cr2 0xffffa0012f71f000 ilevel 0x6 rsp 0xffffa0012f0bf
b88
curlwp 0xffffa0000e7326a0 pid 388.1 lowest kstack 0xffffa0012f0bc2c0
panic: trap
cpu0: Begin traceback...
vpanic() at netbsd:vpanic+0x140
snprintf() at netbsd:snprintf
trap() at netbsd:trap+0x953
--- trap (number 6) ---
xennetback_evthandler() at netbsd:xennetback_evthandler+0x4e
Xresume_xenev6() at netbsd:Xresume_xenev6+0x47
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
sys_ioctl() at netbsd:sys_ioctl+0x101
syscall() at netbsd:syscall+0x9c
--- syscall (number 54) ---
753aff0fedfa:
cpu0: End traceback...

dumping to dev 0,1 (offset=67110591, size=0): not possible
rebooting...
(XEN) Hardware Dom0 shutdown: rebooting machine
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.

This is 100% repeatable.  

This is not new in NetBSD8.0 or in Xen 4.11.  ISTR that I did not see the 'fatal page fault' in older Xen or NetBSD - just poof reboot.

No difference if I type 'reboot' in the domU or 'xl shutdown' in dom0

This happens with Debian 9.5 and Ubuntu 17.10 as domU, not with NetBSD as domU, nor with FreeBSD as domU.

See http://netbsd.itsx.net/xenshutdown/ for 'dmesg', 'xl dmesg', /boot.cfg, domU config.

Otherise the system is stable with FreeBSD, NetBSD, Linux domUs for weeks.
>How-To-Repeat:
reboot Linux domU
>Fix:

>Release-Note:

>Audit-Trail:

Responsible-Changed-From-To: kern-bug-people->jdolecek
Responsible-Changed-By: jdolecek@NetBSD.org
Responsible-Changed-When: Tue, 31 Jul 2018 20:24:05 +0000
Responsible-Changed-Why:
I'll look at this. This looks very similar as port-xen/48125 which I fixed
in June 2018. Fix for that was pulled up and is part of NetBSD 8.0, so this
might be another problem in the code.


From: Dennis Lindroos <denafcm@gmail.com>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-xen/53487
Date: Sat, 11 Aug 2018 13:11:25 +0300

 Hi guys,

 After upgrading my Xen DOM0 to NetBSD/amd64 8.0 myself i've got
 exactly the same fatal page fault with a Linux domU (Arch Linux,
 kernel 4.17.2..) and i tracked down this PR.

 I hacked the code in xennetback_xenbus.c in the function
 xennetback_xenbus_destroy() i noticed that with my domU, the value of
 xni_status was DISCONNECTING, thus I added matching that too:
 if (xneti->xni_status == CONNECTED || xneti->xni_status == DISCONNECTING) {

 Now on domU shutdown the vif interface is removed or atleast there are
 no more panic traps. :)
 Didn't produce anything in "xl dmesg" even with the xen-debug hypervisor.

 Xen version is 4.8.3 and tested on two different amd64-arch machines..

 TIA,
 Dennis.

State-Changed-From-To: open->feedback
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Sat, 11 Aug 2018 10:35:20 +0000
State-Changed-Why:
Committed variant of the fix of Dennis. Can you confirm it fixes your problem?


From: "Jaromir Dolecek" <jdolecek@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/53487 CVS commit: src/sys/arch/xen/xen
Date: Sat, 11 Aug 2018 10:34:25 +0000

 Module Name:	src
 Committed By:	jdolecek
 Date:		Sat Aug 11 10:34:25 UTC 2018

 Modified Files:
 	src/sys/arch/xen/xen: xennetback_xenbus.c

 Log Message:
 disestablish interrupt handler regardless of the state whenever it was
 setup, i.e. also when the instance is DISCONNECTING already

 fixes PR port-xen/53487 by Bert Kiers; fix found and reported by
 Dennis Lindroos

 XXX pullup netbsd-8


 To generate a diff of this commit:
 cvs rdiff -u -r1.67 -r1.68 src/sys/arch/xen/xen/xennetback_xenbus.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: Bert Kiers <kiers@original.xs4all.nl>
To: gnats-bugs@NetBSD.org
Cc: jdolecek@NetBSD.org, netbsd-bugs@netbsd.org, gnats-admin@netbsd.org,
	kiers@xs4all.nl
Subject: Re: port-xen/53487 (xen dom0 crash when reboot of domU)
Date: Mon, 13 Aug 2018 15:37:15 +0200

 On Sat, Aug 11, 2018 at 10:35:20AM +0000, jdolecek@NetBSD.org wrote:
 > Synopsis: xen dom0 crash when reboot of domU
 > 
 > State-Changed-From-To: open->feedback
 > State-Changed-By: jdolecek@NetBSD.org
 > State-Changed-When: Sat, 11 Aug 2018 10:35:20 +0000
 > State-Changed-Why:
 > Committed variant of the fix of Dennis. Can you confirm it fixes your problem?

 I mistakenly tested with a fresh 8.0_STABLE en of course that did not help.
 I need a little more time.  Now compiling current.

 (This is in /netbsd-XEN3_DOM0, right?)


 -- 
 B*E*R*T

From: =?UTF-8?B?SmFyb23DrXIgRG9sZcSNZWs=?= <jaromir.dolecek@gmail.com>
To: Bert Kiers <kiers@original.xs4all.nl>
Cc: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>, kiers@xs4all.nl
Subject: Re: port-xen/53487 (xen dom0 crash when reboot of domU)
Date: Mon, 13 Aug 2018 17:50:14 +0200

 2018-08-13 15:37 GMT+02:00 Bert Kiers <kiers@original.xs4all.nl>:
 > (This is in /netbsd-XEN3_DOM0, right?)

 Yes, amd64 XEN3_DOM0.

From: Bert Kiers <kiers@original.xs4all.nl>
To: gnats-bugs@NetBSD.org
Cc: jdolecek@NetBSD.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
	kiers@xs4all.nl, Dennis Lindroos <denafcm@gmail.com>
Subject: Re: port-xen/53487 (xen dom0 crash when reboot of domU)
Date: Tue, 14 Aug 2018 11:53:01 +0200

 On Mon, Aug 13, 2018 at 03:55:01PM +0000, Jarom?r Dole?ek wrote:
 > The following reply was made to PR port-xen/53487; it has been noted by GNATS.
 > 
 > From: =?UTF-8?B?SmFyb23DrXIgRG9sZcSNZWs=?= <jaromir.dolecek@gmail.com>
 > To: Bert Kiers <kiers@original.xs4all.nl>
 > Cc: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>, kiers@xs4all.nl
 > Subject: Re: port-xen/53487 (xen dom0 crash when reboot of domU)
 > Date: Mon, 13 Aug 2018 17:50:14 +0200
 > 
 >  2018-08-13 15:37 GMT+02:00 Bert Kiers <kiers@original.xs4all.nl>:
 >  > (This is in /netbsd-XEN3_DOM0, right?)
 >  
 >  Yes, amd64 XEN3_DOM0.

 Fixed! Thank you!


 -- 
 B*E*R*T

State-Changed-From-To: feedback->pending-pullups
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Tue, 14 Aug 2018 17:21:40 +0000
State-Changed-Why:
Fix confirmed. Requested pullup to netbsd-8, ticket #973


From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/53487 CVS commit: [netbsd-8] src/sys/arch/xen/xen
Date: Wed, 15 Aug 2018 12:04:02 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Wed Aug 15 12:04:02 UTC 2018

 Modified Files:
 	src/sys/arch/xen/xen [netbsd-8]: xennetback_xenbus.c

 Log Message:
 Pull up following revision(s) (requested by dolecek in ticket #973):

 	sys/arch/xen/xen/xennetback_xenbus.c: revision 1.68 (patch)

 disestablish interrupt handler regardless of the state whenever it was
 setup, i.e. also when the instance is DISCONNECTING already
 fixes PR port-xen/53487 by Bert Kiers; fix found and reported by
 Dennis Lindroos

 XXX pullup netbsd-8


 To generate a diff of this commit:
 cvs rdiff -u -r1.58.8.2 -r1.58.8.3 src/sys/arch/xen/xen/xennetback_xenbus.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: pending-pullups->closed
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Wed, 15 Aug 2018 13:56:45 +0000
State-Changed-Why:
Pullup to netbsd-8 done. Thanks for report.


>Unformatted:

Home	PR Database Search