NetBSD Problem Report #60795

From www@netbsd.org  Sat Sep 26 06:18:21 2026
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits)
	 client-signature RSA-PSS (2048 bits))
	(Client CN "mail.netbsd.org", Issuer "YR1" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id E69781A923C
	for <gnats-bugs@gnats.NetBSD.org>; Sat, 26 Sep 2026 06:18:20 +0000 (UTC)
Message-Id: <20260926061819.91DD11A923E@mollari.NetBSD.org>
Date: Sat, 26 Sep 2026 06:18:19 +0000 (UTC)
From: clare@csel.org
Reply-To: clare@csel.org
To: gnats-bugs@NetBSD.org
Subject: LFS userland cleaner and umount can enter deadlock
X-Send-Pr-Version: www-1.0
X-From4GNATS: "clare@csel.org via gnats" <gnats-admin@NetBSD.org>

>Number:         60795
>Category:       kern
>Synopsis:       LFS userland cleaner and umount can enter deadlock
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Sep 26 06:20:00 +0000 2026
>Last-Modified:  Sat Sep 26 12:50:02 +0000 2026
>Originator:     Shinichi Doyashiki
>Release:        11.99.8 around 2026-09-26 14:00 JST
>Organization:
at home
>Environment:
etBSD kanade2.hq.csel.org 11.99.8 NetBSD 11.99.8 (KANADE2_12) #21: Sat Sep 26 14:10:54 JST 2026  clare@kanade2.hq.csel.org:/export/netbsd/debuglfs/src/sys/arch/amd64/compile/KANADE2_12 amd64

>Description:
LFS deadlock was observed while cycling mount/dd/rm/umount test.

db{0}> ps
PID     LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
(snip)
19034 19034 3   1         0   ffff9bff037be800       sshd-session mutex
21479 21479 3   1   1000000   ffff9bff0365e000             umount rwlock
22182 22182 3   1         0   ffff9bff0365e400       lfs_cleanerd fstchg
(snip)
0     21074 3   1       240   ffff9bff01c77000         lfs_writer mutex
(snip)
0    >  102 1   3       201   ffff9bfef726bc00             idle/3
(snip)
0    >   96 1   2       201   ffff9bfef717b400             idle/2
(snip)
0    >   25 1   1       201   ffff9c062fa54c00             idle/1
(snip)
0        12 3   0       200   ffff9c0630ab1800             lfsswq mutex
0        11 3   0       200   ffff9c0630ab1400             lfscwq lfscwq
(snip)
0    >    2 1   0       201   ffff9c0633b1d000             idle/0
0         0 3   0       200   ffffffff814648c0            swapper uvm

the umount process

db{0}> bt /a ffff9bff0365e000
trace: pid 21479 lid 21479 at 0xffffcc094754cc00
sleepq_block() at netbsd:sleepq_block+0xea
turnstile_block() at netbsd:turnstile_block+0x24b
rw_vector_enter() at netbsd:rw_vector_enter+0x161
lfs_prelock() at netbsd:lfs_prelock+0x86
lfs_seglock() at netbsd:lfs_seglock+0x16
lfs_segwrite() at netbsd:lfs_segwrite+0x71
lfs_flushfiles() at netbsd:lfs_flushfiles+0x1a4
lfs_unmount() at netbsd:lfs_unmount+0x27
VFS_UNMOUNT() at netbsd:VFS_UNMOUNT+0x22
dounmount() at netbsd:dounmount+0xa9
sys_unmount() at netbsd:sys_unmount+0x101
syscall() at netbsd:syscall+0x9d
--- syscall (number 22) ---
netbsd:syscall+0x9d:

the lfs_cleanerd (userland) process

db{0}> bt /a ffff9bff0365e400
trace: pid 22182 lid 22182 at 0xffffcc09486d4950
sleepq_block() at netbsd:sleepq_block+0xea
cv_wait() at netbsd:cv_wait+0xca
fstrans_start() at netbsd:fstrans_start+0x14c
VOP_BMAP() at netbsd:VOP_BMAP+0x48
lfs_strategy() at netbsd:lfs_strategy+0x3b3
VOP_STRATEGY() at netbsd:VOP_STRATEGY+0x64
bio_doread() at netbsd:bio_doread+0x5f
bread() at netbsd:bread+0x18
ino_func_setclean() at netbsd:ino_func_setclean+0x2f7
lfs_parse_pseg() at netbsd:lfs_parse_pseg+0x87f
lfs_rewrite_segments() at netbsd:lfs_rewrite_segments+0x3f6
lfs_fcntl() at netbsd:lfs_fcntl+0x15b3
VOP_FCNTL() at netbsd:VOP_FCNTL+0x6e
sys_fcntl() at netbsd:sys_fcntl+0x461
syscall() at netbsd:syscall+0x9d
--- syscall (number 92) ---
netbsd:syscall+0x9d:

the lfs_writer thread

db{0}> bt /a ffff9bff01c77000
trace: pid 0 lid 21074 at 0xffffcc094a0d4e60
sleepq_block() at netbsd:sleepq_block+0xea
turnstile_block() at netbsd:turnstile_block+0x24b
mutex_vector_enter() at netbsd:mutex_vector_enter+0x13f
cv_timedwait() at netbsd:cv_timedwait+0xde
lfs_writerd() at netbsd:lfs_writerd+0x3ad

db{0}> show locks
PID     LID CPU       STRUCT LWP *              OWNER BITS LOCK
19034 19034   1   ffff9bff037be800   ffff9bff0365e000    2 netbsd:vfs_suspend_lock
21479 21479   1   ffff9bff0365e000                 20    3 ffff9bff04a112b8
0     21074   1   ffff9bff01c77000   ffff9bff0365e000    2 netbsd:lfs_lock
0        12   0   ffff9c0630ab1800   ffff9bff0365e000    2 netbsd:lfs_lock

db{0}> call fstrans_dump(0)
Fstrans locks by lwp:
[ 1371.1778959] Fstrans state by mount:
[ 1371.1778959] /testlfs         owner 0xffff9bff0365e000 state suspended
0

>How-To-Repeat:
#!/bin/sh
#
# LFS umount while cleaner test 2
#
DISK=/dev/dk4
RDISK=/dev/rdk4

newfs_lfs -f4k -b32k $RDISK
mkdir -p /testlfs
count=0
while true; do
  count=`expr $count + 1`
  echo "===> MOUNT $count"
  if mount_lfs -u $DISK /testlfs; then
    echo "===> GOOD mount"
  else
    echo "===> BAD mount"
    exit 1
  fi
  echo "===> try file access"
  {
    cd /testlfs
    dd if=/dev/zero of=testfile1.bin bs=1m count=1
    #dd if=/dev/zero of=testfile2.bin bs=1m count=1
    #dd if=/dev/zero of=testfile3.bin bs=1m count=1
    #dd if=/dev/zero of=testfile4.bin bs=1m count=1
    #dd if=/dev/zero of=testfile5.bin bs=1m count=1
    #dd if=/dev/zero of=testfile6.bin bs=1m count=1
    #dd if=/dev/zero of=testfile7.bin bs=1m count=1
    #dd if=/dev/zero of=testfile8.bin bs=1m count=1
    rm testfile*.bin
    cd ..
  }
  echo "===> try umount"
  if umount /testlfs; then
    echo "===> GOOD umount"
  else
    echo "===> BAD umount"
    exit 1
  fi
  if fsck_lfs -nf $RDISK; then
    echo "===> GOOD fsck_lfs"
  else
    echo "===> BAD fsck_lfs"
    exit 1
  fi
done

>Fix:
unknown

>Audit-Trail:
From: Taylor R Campbell <riastradh@NetBSD.org>
To: Shinichi Doyashiki <clare@csel.org>
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org,
	"J. Hannken-Illjes" <hannken@mailbox.org>,
	Konrad Schroder <perseant@hhhh.org>
Subject: Re: kern/60795: LFS userland cleaner and umount can enter deadlock
Date: Sat, 26 Sep 2026 12:46:03 +0000

 > Date: Sat, 26 Sep 2026 06:18:19 +0000 (UTC)
 > From: clare@csel.org
 > 
 > the umount process
 > 
 > db{0}> bt /a ffff9bff0365e000
 > trace: pid 21479 lid 21479 at 0xffffcc094754cc00
 > sleepq_block() at netbsd:sleepq_block+0xea
 > turnstile_block() at netbsd:turnstile_block+0x24b
 > rw_vector_enter() at netbsd:rw_vector_enter+0x161
 > lfs_prelock() at netbsd:lfs_prelock+0x86
 > lfs_seglock() at netbsd:lfs_seglock+0x16
 > [...]
 > 
 > the lfs_cleanerd (userland) process
 > 
 > db{0}> bt /a ffff9bff0365e400
 > trace: pid 22182 lid 22182 at 0xffffcc09486d4950
 > sleepq_block() at netbsd:sleepq_block+0xea
 > cv_wait() at netbsd:cv_wait+0xca
 > fstrans_start() at netbsd:fstrans_start+0x14c
 > [...]
 > bread() at netbsd:bread+0x18
 > [...]
 > lfs_rewrite_segments() at netbsd:lfs_rewrite_segments+0x3f6
 > lfs_fcntl() at netbsd:lfs_fcntl+0x15b3

 - unmount suspends fs, waits for seglock.
 - lfs_rewrite_segments takes seglock, tries block I/O which waits for
   concurrent suspension to resume.
 *deadlock*

 Perhaps lfs_fcntl, or lfs_rewrite_segments, should start a lazy
 fstrans on the file system before trying to do any work on it, so it
 isn't blocked by a concurrent suspension for unmount?

 	fstrans_start_lazy(ap->a_vp->v_mount);
 	...
 	fstrans_done(ap->a_vp->v_mount);

 Maybe the fstrans should be started before testing IMNT_SHUTDOWN?

    1897 	/* Avoid locking a draining lock */
    1898 	if (ap->a_vp->v_mount->mnt_iflag & IMNT_UNMOUNT) {
    1899 		return ESHUTDOWN;
    1900 	}

 https://nxr.NetBSD.org/xref/src/sys/ufs/lfs/lfs_vnops.c#1897

 Side note: How does one ensure v_mount is stable at this point in the
 face of concurrent revoke?  Seems like we need to do something like

 	for (;;) {
 		mp = atomic_load_consume(&v->v_mount);
 		fstrans_start_lazy(mp);
 		if (__predict_true(mp == atomic_load_consume(&v->v_mount)))
 			break;
 		fstrans_done(mp);
 	}
 	vn_lock(vp, LK_SHARED|LK_WAIT);
 	/*
 	 * mp may be lfs or deadfs if the vnode was concurrently
 	 * revoked; check for concurrent revoke before proceeding.
 	 */
 	mutex_enter(vp->v_interlock);
 	error = vdead_check(vp, 0);
 	mutex_exit(vp->v_interlock);
 	if (error)
 		goto out;

 	... check IMNT_SHUTDOWN, do the rest of lfs_fcntl ...

 out:	VOP_UNLOCK(vp);
 	fstrans_done(mp);

 ...feeling some deja vu about this question, looks like I was
 wondering something similar a couple years ago too:

 https://mail-index.NetBSD.org/tech-kern/2024/10/01/msg029758.html

 Actually, maybe it doesn't need to be a lazy fstrans at all; maybe the
 ordinary fstrans in vn_lock is enough.  But then maybe we can't hold
 vn_lock across the whole fcntl operation for other reasons -- that
 leads to a lot of code paths that might try to take vnode locks.

Home	PR Database Search