NetBSD Problem Report #57450

From chuq@chuq.com  Sat Jun  3 23:59:42 2023
Return-Path: <chuq@chuq.com>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 0C1CB1A923C
	for <gnats-bugs@gnats.NetBSD.org>; Sat,  3 Jun 2023 23:59:42 +0000 (UTC)
Message-Id: <20230603235937.F190C3CFFA26@spathi.chuq.com>
Date: Sat,  3 Jun 2023 16:59:37 -0700 (PDT)
From: chuq@chuq.com
Reply-To: chuq@chuq.com
To: gnats-bugs@NetBSD.org
Subject: unplugging hung USB disk triggers panic via _vstate_assert
X-Send-Pr-Version: 3.95

>Number:         57450
>Category:       kern
>Synopsis:       unplugging hung USB disk triggers panic via _vstate_assert
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    hannken
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Jun 04 00:00:01 +0000 2023
>Closed-Date:    Fri Jun 23 09:49:44 +0000 2023
>Last-Modified:  Fri Jun 23 09:49:44 +0000 2023
>Originator:     Chuck Silvers
>Release:        NetBSD 10.99.4 as of 2023/04/23 12:09:30
>Organization:
NetBSD
>Environment:
Architecture: x86_64
Machine: amd64
>Description:

When a USB disk that is hung by running "usbdevs" (as described in PR 57379)
is unplugged, and the disk contains a file system that is mounted,
the kernel panics:

[ 2835.4017665] sd1d: error writing fsbn 3340727744 of 3340727744-3340727871 (sd1 bn 3340727744; cn 1631214 tn 46 sn 0)
[ 2835.4120744] sd1(umass1:0:0:0): generic HBA error
[ 2835.4169086] sd1: cache synchronization failed
[ 2835.4589019] vnode 0xffffa6c15bcc5800 flags 0xc010<MPSAFE,ONWORKLST,DEADCHECK>
[ 2835.4657376]         tag VT_UFS(1) type VBLK(3) mount 0xffffa6c153a73000 typedata 0xffffa6c848db9100
[ 2835.4745269]         usecount 881 writecount 0 holdcount 49
[ 2835.4794098]         size 0 writesize 0 numoutput 0
[ 2835.4833161]         data 0xffffa6c15a295938 lock 0xffffa6c15bcc59c0
[ 2835.4891756]         state RECLAIMING key(0xffffa6c153a73000 8) 36 61 00 00 00 00 00 00
[ 2835.4969883]         lrulisthd 0xffffffff818b61d0
[ 2835.5008946]         tag VT_UFS, ino 24886, on dev 4, 0 flags 0x0, nlink 1
[ 2835.5067541]         mode 060640, owner 0, group 5, size 0
[ 2835.5116370] panic: state is RECLAIMING, usecount 881, expected ACTIVE at genfs_lock:400
[ 2835.5204262] cpu1: Begin traceback...
[ 2835.5243325] vpanic() at netbsd:vpanic+0x183
[ 2835.5301920] vnpanic() at netbsd:vnpanic+0x4a
[ 2835.5360515] _vstate_assert() at netbsd:_vstate_assert+0x177
[ 2835.5428875] genfs_lock() at netbsd:genfs_lock+0x48
[ 2835.5487471] VOP_LOCK() at netbsd:VOP_LOCK+0xa8
[ 2835.5536300] vn_lock() at netbsd:vn_lock+0xb0
[ 2835.5594894] ffs_vfs_fsync() at netbsd:ffs_vfs_fsync+0xc7
[ 2835.5653489] spec_fsync() at netbsd:spec_fsync+0x4a
[ 2835.5712084] ffs_spec_fsync() at netbsd:ffs_spec_fsync+0x19
[ 2835.5770679] VOP_FSYNC() at netbsd:VOP_FSYNC+0x47
[ 2835.5829274] vinvalbuf() at netbsd:vinvalbuf+0x65
[ 2835.5887869] vcache_reclaim() at netbsd:vcache_reclaim+0x16f
[ 2835.5946463] vgone() at netbsd:vgone+0xce
[ 2835.5995293] vrevoke() at netbsd:vrevoke+0xfa
[ 2835.6044122] genfs_revoke() at netbsd:genfs_revoke+0x13
[ 2835.6112482] VOP_REVOKE() at netbsd:VOP_REVOKE+0x3b
[ 2835.6161311] vdevgone() at netbsd:vdevgone+0x64
[ 2835.6219906] dkwedge_detach() at netbsd:dkwedge_detach+0x18f
[ 2835.6278501] config_detach() at netbsd:config_detach+0x1bc
[ 2835.6346863] dkwedge_delall1() at netbsd:dkwedge_delall1+0xc6
[ 2835.6415222] dkwedge_delall() at netbsd:dkwedge_delall+0xb
[ 2835.6473817] sddetach() at netbsd:sddetach+0x126
[ 2835.6522646] config_detach() at netbsd:config_detach+0x1bc
[ 2835.6581241] scsipi_target_detach() at netbsd:scsipi_target_detach+0xf3
[ 2835.6659368] scsibusdetach() at netbsd:scsibusdetach+0x5b
[ 2835.6717963] config_detach() at netbsd:config_detach+0x1bc
[ 2835.6786323] umass_detach() at netbsd:umass_detach+0x9a
[ 2835.6844918] config_detach() at netbsd:config_detach+0x1bc
[ 2835.6903513] usb_disconnect_port() at netbsd:usb_disconnect_port+0xc3
[ 2835.6971874] uhub_explore() at netbsd:uhub_explore+0x373
[ 2835.7040235] usb_discover() at netbsd:usb_discover+0x4f
[ 2835.7098829] usb_event_thread() at netbsd:usb_event_thread+0x7a
[ 2835.7157424] cpu1: End traceback...


This panic happens when the file in the file system is being written to
when the I/O hang is triggered.  If the file is only being read
then the system hangs instead, but I'll report that separately
since it seems like a separate problem.


>How-To-Repeat:

Run "usbdevs" while writing to a file in a file system stored in a USB disk,
then disconnect the USB disk from the host.

>Fix:

TBD

>Release-Note:

>Audit-Trail:

Responsible-Changed-From-To: kern-bug-people->hannken
Responsible-Changed-By: hannken@NetBSD.org
Responsible-Changed-When: Mon, 12 Jun 2023 10:55:49 +0000
Responsible-Changed-Why:
Take -- I broke it with revision 1,376 of sys/ufs/ffs/ffs_vfsops.c

It is not ok to unlock/relock the device node here, VOP_FSYNC must
never drop the lock as vcache_reclaim() for example depends on
this behaviour.


From: "Juergen Hannken-Illjes" <hannken@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/57450 CVS commit: src
Date: Thu, 15 Jun 2023 09:13:37 +0000

 Module Name:	src
 Committed By:	hannken
 Date:		Thu Jun 15 09:13:36 UTC 2023

 Modified Files:
 	src/share/man/man9: vnodeops.9
 	src/sys/kern: vnode_if.src

 Log Message:
 VOP_IOCTL() is a wrapper around spec_ioctl() aka Xdev_ioctl() and
 protected with spec_io_enter()/spec_io_exit() so there is no need
 to force specific vnode locking.

 Set locking requirement to '= = =' (unchanged, locked or unlocked).

 PR kern/57450 (unplugging hung USB disk triggers panic via _vstate_assert)


 To generate a diff of this commit:
 cvs rdiff -u -r1.100 -r1.101 src/share/man/man9/vnodeops.9
 cvs rdiff -u -r1.84 -r1.85 src/sys/kern/vnode_if.src

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Juergen Hannken-Illjes" <hannken@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/57450 CVS commit: src/sys
Date: Thu, 15 Jun 2023 09:15:55 +0000

 Module Name:	src
 Committed By:	hannken
 Date:		Thu Jun 15 09:15:54 UTC 2023

 Modified Files:
 	src/sys/dev/raidframe: rf_netbsdkintf.c
 	src/sys/ufs/ffs: ffs_vfsops.c

 Log Message:
 Undo unlock/relock for VOP_IOCTL().

 PR kern/57450 (unplugging hung USB disk triggers panic via _vstate_assert)


 To generate a diff of this commit:
 cvs rdiff -u -r1.411 -r1.412 src/sys/dev/raidframe/rf_netbsdkintf.c
 cvs rdiff -u -r1.380 -r1.381 src/sys/ufs/ffs/ffs_vfsops.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/57450 CVS commit: [netbsd-10] src/sys
Date: Wed, 21 Jun 2023 16:55:02 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Wed Jun 21 16:55:02 UTC 2023

 Modified Files:
 	src/sys/dev/raidframe [netbsd-10]: rf_netbsdkintf.c
 	src/sys/ufs/ffs [netbsd-10]: ffs_vfsops.c

 Log Message:
 Pull up following revision(s) (requested by hannken in ticket #197):

 	sys/ufs/ffs/ffs_vfsops.c: revision 1.381
 	sys/dev/raidframe/rf_netbsdkintf.c: revision 1.412

 Undo unlock/relock for VOP_IOCTL().
 PR kern/57450 (unplugging hung USB disk triggers panic via _vstate_assert)


 To generate a diff of this commit:
 cvs rdiff -u -r1.410 -r1.410.4.1 src/sys/dev/raidframe/rf_netbsdkintf.c
 cvs rdiff -u -r1.378.2.1 -r1.378.2.2 src/sys/ufs/ffs/ffs_vfsops.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->closed
State-Changed-By: hannken@NetBSD.org
State-Changed-When: Fri, 23 Jun 2023 09:49:44 +0000
State-Changed-Why:
Fix committed and pullup done.


>Unformatted:

Home	PR Database Search