NetBSD Problem Report #56700

From reinoud@gorilla.13thmonkey.org  Mon Feb  7 20:29:14 2022
Return-Path: <reinoud@gorilla.13thmonkey.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 7C45E1A9239
	for <gnats-bugs@gnats.NetBSD.org>; Mon,  7 Feb 2022 20:29:14 +0000 (UTC)
Message-Id: <20220207202910.738542FF0772@gorilla.13thmonkey.org>
Date: Mon,  7 Feb 2022 21:29:10 +0100 (CET)
From: reinoud@13thmonkey.org
Reply-To: reinoud@13thmonkey.org
To: gnats-bugs@NetBSD.org
Subject: panic on dkctl makewedges on vnd
X-Send-Pr-Version: 3.95

>Number:         56700
>Category:       kern
>Synopsis:       panic on dkctl makewedges on vnd
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          feedback
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Feb 07 20:30:01 +0000 2022
>Closed-Date:    
>Last-Modified:  Mon Mar 28 11:44:09 +0000 2022
>Originator:     Reinoud Zandijk
>Release:        NetBSD 9.99.93
>Organization:
NetBSD

>Environment:


System: NetBSD gorilla.13thmonkey.org 9.99.93 NetBSD 9.99.93 (GENERIC) #0: Thu Feb 3 10:36:28 CET 2022 reinoud@gorilla.13thmonkey.org:/tmp/obj/sys/arch/amd64/compile/GENERIC amd64
Architecture: x86_64
Machine: amd64

>Description:

When trying to figure out an issue with wedge enumeration in a Qemu instance I
vnconfig'd its drive and seeing that it didn't produce wedges there too I
issued `dkctl vnd0 makewedges' and got the following crash. IIRC it wasn't
mounted but was accessed.

[ 24940.730314] dk6 at vnd0 (f346c70d-e85c-452f-9b00-3fe14c3f0413) deleted
... vnconfig'ing another disk and issued the command

[ 25425.651260] Mutex error: mutex_destroy,392: assertion failed: !MUTEX_OWNED(owner)

[ 25425.651260] lock address : 0xffff849ea62a60c0
[ 25425.651260] current cpu  :                  0
[ 25425.651260] current lwp  : 0xffff849aabc8d640
[ 25425.651260] owner field  : 0xffff849aabc8d640 wait/spin:                0/0

[ 25425.651260] panic: lock error: Mutex: mutex_destroy,392: assertion failed: !MUTEX_OWNED(owner): lock 0xffff849ea62a60c0 cpu 0 lwp 0xffff849aabc8d640

traceback:
...
#10 0xffffffff80221a35 in breakpoint ()
#11 0xffffffff80e09a98 in vpanic (fmt=fmt@entry=0xffffffff814cc360 "lock error: %s: %s,%zu: %s: lock %p cpu %d lwp %p", ap=ap@entry=0xffff9084b5cb16c8) at /usr/sources/cvs.netbsd.org/src-clean/sys/kern/subr_prf.c:288
#12 0xffffffff80e09b67 in panic (fmt=fmt@entry=0xffffffff814cc360 "lock error: %s: %s,%zu: %s: lock %p cpu %d lwp %p") at /usr/sources/cvs.netbsd.org/src-clean/sys/kern/subr_prf.c:209
#13 0xffffffff80dff686 in lockdebug_abort (func=func@entry=0xffffffff8136cbf8 <__func__.5> "mutex_destroy", line=line@entry=392, lock=lock@entry=0xffff849ea62a60c0, ops=0xffffffff818885f0 <mutex_adaptive_lockops>, 
    msg=msg@entry=0xffffffff814c40f0 "assertion failed: !MUTEX_OWNED(owner)") at /usr/sources/cvs.netbsd.org/src-clean/sys/sys/cpu.h:108
#14 0xffffffff80db9434 in mutex_abort (func=func@entry=0xffffffff8136cbf8 <__func__.5> "mutex_destroy", line=line@entry=392, mtx=mtx@entry=0xffff849ea62a60c0, 
    msg=msg@entry=0xffffffff814c40f0 "assertion failed: !MUTEX_OWNED(owner)") at /usr/sources/cvs.netbsd.org/src-clean/sys/kern/kern_mutex.c:336
#15 0xffffffff80db95bb in mutex_destroy (mtx=mtx@entry=0xffff849ea62a60c0) at /usr/sources/cvs.netbsd.org/src-clean/sys/kern/kern_mutex.c:392
#16 0xffffffff80df54c7 in disk_destroy (diskp=diskp@entry=0xffff849ea62a6048) at /usr/sources/cvs.netbsd.org/src-clean/sys/kern/subr_disk.c:260
#17 0xffffffff80ea9b35 in vnd_detach (self=0xffff84a12e02e180, flags=<optimized out>) at /usr/sources/cvs.netbsd.org/src-clean/sys/dev/vnd.c:292
#18 0xffffffff80deca1f in config_detach (dev=dev@entry=0xffff84a12e02e180, flags=flags@entry=2) at /usr/sources/cvs.netbsd.org/src-clean/sys/kern/subr_autoconf.c:1970
#19 0xffffffff80ea92f1 in vnd_destroy (dev=0xffff84a12e02e180) at /usr/sources/cvs.netbsd.org/src-clean/sys/dev/vnd.c:318
#20 vndclose (dev=<optimized out>, flags=<optimized out>, mode=<optimized out>, l=<optimized out>) at /usr/sources/cvs.netbsd.org/src-clean/sys/dev/vnd.c:468
#21 0xffffffff80df4613 in bdev_close (dev=3587, flag=1, devtype=24576, l=0xffff849aabc8d640) at /usr/sources/cvs.netbsd.org/src-clean/sys/kern/subr_devsw.c:729
#22 0xffffffff80e915df in spec_close (v=<optimized out>) at ./machine/cpu.h:72
#23 0xffffffff80e83414 in VOP_CLOSE (vp=vp@entry=0xffff84a3b05587c0, fflag=fflag@entry=1, cred=cred@entry=0xffffffffffffffff) at /usr/sources/cvs.netbsd.org/src-clean/sys/kern/vnode_if.c:563
#24 0xffffffff80e7b1f3 in vn_close (vp=0xffff84a3b05587c0, flags=flags@entry=1, cred=cred@entry=0xffffffffffffffff) at /usr/sources/cvs.netbsd.org/src-clean/sys/kern/vfs_vnops.c:480
#25 0xffffffff80e9cfdd in dk_close_parent (mode=1, vp=<optimized out>) at /usr/sources/cvs.netbsd.org/src-clean/sys/dev/dkwedge/dk.c:1122
#26 dkwedge_read (pdk=pdk@entry=0xffff849ea62a6048, vp=vp@entry=0xffff84a29032b380, blkno=blkno@entry=0, tbuf=0xffff9084bb2c6600, len=len@entry=16384) at /usr/sources/cvs.netbsd.org/src-clean/sys/dev/dkwedge/dk.c:1057
#27 0xffffffff80e9d1c2 in dkwedge_discover_apple (pdk=0xffff849ea62a6048, vp=0xffff84a29032b380) at /usr/sources/cvs.netbsd.org/src-clean/sys/dev/dkwedge/dkwedge_apple.c:148
#28 0xffffffff80e9ce31 in dkwedge_discover (pdk=0xffff849ea62a6048) at /usr/sources/cvs.netbsd.org/src-clean/sys/dev/dkwedge/dk.c:977
#29 0xffffffff80df5bda in disk_ioctl (dk=<optimized out>, dev=<optimized out>, cmd=<optimized out>, data=<optimized out>, flag=<optimized out>, l=<optimized out>)
    at /usr/sources/cvs.netbsd.org/src-clean/sys/kern/subr_disk.c:637
#30 0xffffffff80ea9ec5 in vndioctl (dev=10499, cmd=1074029699, data=0xffff9084b5cb1ee0, flag=3, l=0xffff849aabc8d640) at /usr/sources/cvs.netbsd.org/src-clean/sys/dev/vnd.c:1253
#31 0xffffffff80df4d0d in cdev_ioctl (dev=10499, cmd=1074029699, data=0xffff9084b5cb1ee0, flag=3, l=0xffff849aabc8d640) at /usr/sources/cvs.netbsd.org/src-clean/sys/kern/subr_devsw.c:935
#32 0xffffffff80e83b09 in VOP_IOCTL (vp=vp@entry=0xffff849aa352ad00, command=command@entry=1074029699, data=data@entry=0xffff9084b5cb1ee0, fflag=<optimized out>, cred=<optimized out>)
    at /usr/sources/cvs.netbsd.org/src-clean/sys/kern/vnode_if.c:883
#33 0xffffffff80e7a7af in vn_ioctl (fp=0xffff84a456966a40, com=1074029699, data=0xffff9084b5cb1ee0) at /usr/sources/cvs.netbsd.org/src-clean/sys/kern/vfs_vnops.c:865
#34 0xffffffff80e1b981 in sys_ioctl (l=<optimized out>, uap=0xffff9084b5cb2000, retval=<optimized out>) at /usr/sources/cvs.netbsd.org/src-clean/sys/kern/sys_generic.c:673
#35 0xffffffff80573fbe in sy_call (rval=0xffff9084b5cb1fb0, uap=0xffff9084b5cb2000, l=0xffff849aabc8d640, sy=0xffffffff81883fb0 <sysent+1296>) at /usr/sources/cvs.netbsd.org/src-clean/sys/sys/syscallvar.h:65
#36 sy_invoke (code=54, rval=0xffff9084b5cb1fb0, uap=0xffff9084b5cb2000, l=0xffff849aabc8d640, sy=0xffffffff81883fb0 <sysent+1296>) at /usr/sources/cvs.netbsd.org/src-clean/sys/sys/syscallvar.h:94
#37 syscall (frame=0xffff9084b5cb2000) at /usr/sources/cvs.netbsd.org/src-clean/sys/arch/x86/x86/syscall.c:138
#38 0xffffffff8020b25d in handle_syscall ()


#17 0xffffffff80ea9b35 in vnd_detach (self=0xffff84a12e02e180, flags=<optimized out>) at /usr/sources/cvs.netbsd.org/src-clean/sys/dev/vnd.c:292
292		disk_destroy(&sc->sc_dkdev);

290		pmf_device_deregister(self);
291		bufq_free(sc->sc_tab);
292		disk_destroy(&sc->sc_dkdev);
293	

(gdb) print sc->sc_dkdev
$1 = {dk_link = {tqe_next = 0x0, tqe_prev = 0x0}, dk_name = 0xffff84a12e02e1c4 "vnd0", dk_info = 0x0, dk_geom = {dg_secperunit = 0, dg_secsize = 0, dg_nsectors = 0, dg_ntracks = 0, dg_ncylinders = 0, dg_secpercyl = 0, 
    dg_pcylinders = 0, dg_sparespertrack = 0, dg_sparespercyl = 0, dg_acylinders = 0}, dk_bopenmask = 0, dk_copenmask = 8, dk_openmask = 8, dk_state = 0, dk_blkshift = 0, dk_byteshift = 9, dk_stats = 0x0, 
  dk_driver = 0xffffffff81374580 <vnddkdriver>, dk_rawlock = {u = {mtxa_owner = 18446608398574736960, s = {mtxs_dummy = 64 '@', mtxs_ipl = {_ipl = 214 '\326'}, mtxs_lock = 200 '\310', mtxs_unused = 171 '\253'}}}, 
  dk_rawopens = 0, dk_rawvp = 0x0, dk_openlock = {u = {mtxa_owner = 18446744073709551600, s = {mtxs_dummy = 240 '�', mtxs_ipl = {_ipl = 255 '\377'}, mtxs_lock = 255 '\377', mtxs_unused = 255 '\377'}}}, dk_nwedges = 0, 
  dk_wedges = {lh_first = 0x0}, dk_labelsector = 1, dk_label = 0x0, dk_cpulabel = 0x0}


#16 0xffffffff80df54c7 in disk_destroy (diskp=diskp@entry=0xffff849ea62a6048) at /usr/sources/cvs.netbsd.org/src-clean/sys/kern/subr_disk.c:260
260		mutex_destroy(&diskp->dk_rawlock);
(gdb) list
255	void
256	disk_destroy(struct disk *diskp)
257	{
258	
259		mutex_destroy(&diskp->dk_openlock);
260		mutex_destroy(&diskp->dk_rawlock);
261	}

(gdb) print diskp->dk_rawlock
$2 = {u = {mtxa_owner = 18446608398574736960, s = {mtxs_dummy = 64 '@', mtxs_ipl = {_ipl = 214 '\326'}, mtxs_lock = 200 '\310', mtxs_unused = 171 '\253'}}}


#15 0xffffffff80db95bb in mutex_destroy (mtx=mtx@entry=0xffff849ea62a60c0) at /usr/sources/cvs.netbsd.org/src-clean/sys/kern/kern_mutex.c:392
392			MUTEX_ASSERT(mtx, !MUTEX_OWNED(owner));
(gdb) list
387	mutex_destroy(kmutex_t *mtx)
388	{
389		uintptr_t owner = mtx->mtx_owner;
390	
391		if (MUTEX_ADAPTIVE_P(owner)) {
392			MUTEX_ASSERT(mtx, !MUTEX_OWNED(owner));
393			MUTEX_ASSERT(mtx, !MUTEX_HAS_WAITERS(mtx));
394		} else {
395			MUTEX_ASSERT(mtx, !MUTEX_SPINBIT_LOCKED_P(mtx));
396		}

The disc is a file on FFS.

> disklabel vnd0
# /dev/rvnd0:
type: vnd
disk: vnd
label: fictitious
flags:
bytes/sector: 512
sectors/track: 32
tracks/cylinder: 64
sectors/cylinder: 2048
cylinders: 8097
total sectors: 16582656
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0		# microseconds
track-to-track seek: 0	# microseconds
drivedata: 0 

5 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
 c:  16582593        63     unused      0     0        # (Cyl.      0*-   8096)
 d:  16582656         0     unused      0     0        # (Cyl.      0 -   8096)
 e:  16582593        63     4.2BSD      0     0     0  # (Cyl.      0*-   8096)



>How-To-Repeat:
I tried to repeat it but after issuing a `dkctl vnd0 listwedges' it gives that
there are no wedges.


>Fix:
Unknown


>Release-Note:

>Audit-Trail:
From: mlelstv@serpens.de (Michael van Elst)
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/56700: panic on dkctl makewedges on vnd
Date: Tue, 8 Feb 2022 06:35:45 -0000 (UTC)

 reinoud@13thmonkey.org writes:

 >When trying to figure out an issue with wedge enumeration in a Qemu instance I
 >vnconfig'd its drive and seeing that it didn't produce wedges there too I
 >issued `dkctl vnd0 makewedges' and got the following crash. IIRC it wasn't
 >mounted but was accessed.

 This happens when the device doesn't exist yet, i.e. the vnconfig wasn't done
 yet or failed. vndioctl() already filters some ioctls, but not those
 handled by disk_ioctl.

 Let me prepare a patch.

From: "Michael van Elst" <mlelstv@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/56700 CVS commit: src/sys/dev
Date: Mon, 28 Mar 2022 11:16:59 +0000

 Module Name:	src
 Committed By:	mlelstv
 Date:		Mon Mar 28 11:16:59 UTC 2022

 Modified Files:
 	src/sys/dev: vnd.c

 Log Message:
 Check INITED state by default for all ioctls but VNDIOCSET. Avoids crashes
 with disk_ioctls on default unit, which is not INITED.
 Fixes PR 56700.


 To generate a diff of this commit:
 cvs rdiff -u -r1.283 -r1.284 src/sys/dev/vnd.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->feedback
State-Changed-By: mlelstv@NetBSD.org
State-Changed-When: Mon, 28 Mar 2022 11:44:09 +0000
State-Changed-Why:
Fix committed, please verify.


>Unformatted:



 Sources are from Jan 30th at around 20:59

Home	PR Database Search