NetBSD Problem Report #56969

From bernd@niob.bersie.home  Wed Aug 17 15:46:35 2022
Return-Path: <bernd@niob.bersie.home>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 6814A1A921F
	for <gnats-bugs@gnats.NetBSD.org>; Wed, 17 Aug 2022 15:46:35 +0000 (UTC)
Message-Id: <20220817140143.04528300965@niob.bersie.home>
Date: Wed, 17 Aug 2022 16:01:42 +0200 (CEST)
From: bernd@niob.bersie.home
Reply-To: bernd@niob.bersie.home
To: gnats-bugs@NetBSD.org
Subject: Kernel panic on host when qemu-nvmm virtual machine exits
X-Send-Pr-Version: 3.95

>Number:         56969
>Category:       kern
>Synopsis:       Kernel panic on host when qemu-nvmm virtual machine exits
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Aug 17 15:50:00 +0000 2022
>Last-Modified:  Thu Aug 18 01:00:01 +0000 2022
>Originator:     bernd.sieker@posteo.net
>Release:        NetBSD 9.3
>Organization:
>Environment:
System: NetBSD niob.bersie.home 9.3 NetBSD 9.3 (NIOB_DEBUG) #1: Wed Aug 17 12:07:51 CEST 2022 bernd@niob.bersie.home:/usr/src/sys/arch/amd64/compile/NIOB_DEBUG amd64
SunFire X2270 M2, Dual Xeon X5675, 56 GB ECC RAM
Architecture: x86_64
Machine: amd64
>Description:
I have recently upgraded a 9.2_STABLE system to 9.3 RELEASE, built and installed from local source copy using build.sh.
I also reinstalled all packages from pkgsrc-2022Q2, including qemu 7.0.0.
The virtual machine also runs NetBSD 9.3 RELEASE, also completely upgraded and reinstalled.
Whenever a virtual machine quits (either shutting down the NetBSD guest with "shutdown -p" or killing the qemu process with TERM signal, the host machine kernel panics and the machine reboots.

I have built a kernel with DEBUG and LOCKDEBUG enabled, here are the last lines extracted from the crashdump using dmesg:

[  1260.922078] panic: kernel diagnostic assertion "semcnt >= 0" failed: file "../../../../kern/kern_uidinfo.c", line 241
[  1260.922078] cWpuA9R:N IBNeGg:i nS PtLr aNcOeTb aLcOkW.E.R.E
[  1260.922078] D ON SYSCALL 2 675736328 EXIT ff844ed0 7
[  1260.922078] WARNING: SPL NOT LOWERED ON SYSCALL 2 675736328 EXIT ff844ed0 7
[   126.000000] 0 7v8p4a]n iWcA(R)N IaNtG : SPL NOT LOWERED ON SYSCALL 2 675736328 EXIT ff844ed0 7
[  1260.922078] + 0WxA1R6N0I
[  1260.922078] NG: SPL NOT LOWERED ON SYSCALL 0 675736328 EXIT ff844ed0 7
[  1260.922078] ugen_get_alt_index() at netbsd:ugen_get_alt_index
[  1260.922078] chgsemcnt() at netbsd:chgsemcnt+0x56
[  1260.922078] ksem_release() at netbsd:ksem_release+0x6a
[  1260.932083] ksem_close_fop() at netbsd:ksem_close_fop+0x49
[  1260.932083] closef() at netbsd:closef+0x6d
[  1260.932083] fd_close() at netbsd:fd_close+0x2b1
[  1260.932083] sys__ksem_destroy() at netbsd:sys__ksem_destroy+0x9c
[  1260.932083] syscall() at netbsd:syscall+0x196
[  1260.932083] --- syscall (number 255) ---
[  1260.932083] 7be3fd84384a:
[  1260.932083] cpu9: End traceback...

Possibly relevant kernel options include:

options         SVS             # Separate Virtual Space
makeoptions     SPECTRE_V2_GCC_MITIGATION=1     # GCC Spectre variant 2
                                                # migitation
options         SPECTRE_V2_GCC_MITIGATION
[...]
# Diagnostic/debugging support options
options         DIAGNOSTIC      # inexpensive kernel consistency checks
                                # XXX to be commented out on release branch
options         DEBUG           # expensive debugging checks/support
options         LOCKDEBUG       # expensive locking checks/support
[...]
makeoptions     COPTS="-O2 -fno-omit-frame-pointer"
options         DDB             # in-kernel debugger
options         DDB_COMMANDONENTER="bt" # execute command when ddb is entered
options         DDB_ONPANIC=1   # see also sysctl(7): `ddb.onpanic'
options         DDB_HISTORY_SIZE=512    # enable history editing in DDB
#options        KGDB            # remote debugger
#options        KGDB_DEVNAME="\"com\"",KGDB_DEVADDR=0x3f8,KGDB_DEVRATE=9600
makeoptions     DEBUG="-g"      # compile full symbol table for CTF
[...]

KUBSAN, KASAN, KLEAK and KCOV are not enabled.

>How-To-Repeat:
Start a virtual machine using qemu with nvmm acceleration, stop the VM. A kernel panic ensues.
>Fix:
Unknown

>Audit-Trail:
From: Taylor R Campbell <riastradh@NetBSD.org>
To: bernd@niob.bersie.home
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/56969: Kernel panic on host when qemu-nvmm virtual machine exits
Date: Wed, 17 Aug 2022 16:03:27 +0000

 Possible duplicate of <https://gnats.netbsd.org/55509> (which doesn't
 have the panic details so it's hard to search) and of
 <https://syzkaller.appspot.com/bug?extid=9d04b3ef2ca180ef9b06>, which
 were both fixed in sys/kern/uipc_sem.c rev. 1.60 -- but that never got
 pulled up to netbsd-9.

 https://mail-index.netbsd.org/source-changes/2020/12/14/msg125176.html

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: Taylor R Campbell <riastradh@netbsd.org>
Subject: Re: kern/56969: Kernel panic on host when qemu-nvmm virtual machine
 exits
Date: Thu, 18 Aug 2022 00:57:27 +0000

 On Wed, Aug 17, 2022 at 04:05:01PM +0000, Taylor R Campbell wrote:
  >  Possible duplicate of <https://gnats.netbsd.org/55509> (which doesn't
  >  have the panic details so it's hard to search) and of
  >  <https://syzkaller.appspot.com/bug?extid=9d04b3ef2ca180ef9b06>, which
  >  were both fixed in sys/kern/uipc_sem.c rev. 1.60 -- but that never got
  >  pulled up to netbsd-9.
  >  
  >  https://mail-index.netbsd.org/source-changes/2020/12/14/msg125176.html

 The submitter's mail isn't working so they may not be seeing this. (FYI)

 -- 
 David A. Holland
 dholland@netbsd.org

Home	PR Database Search