NetBSD Problem Report #60820
From www@netbsd.org Mon Sep 28 21:37:11 2026
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
client-signature RSA-PSS (2048 bits) client-digest SHA256)
(Client CN "mail.netbsd.org", Issuer "YR1" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id A6C931A923B
for <gnats-bugs@gnats.NetBSD.org>; Mon, 28 Sep 2026 21:37:11 +0000 (UTC)
Message-Id: <20260928213709.E72521A923C@mollari.NetBSD.org>
Date: Mon, 28 Sep 2026 21:37:09 +0000 (UTC)
From: clare@csel.org
Reply-To: clare@csel.org
To: gnats-bugs@NetBSD.org
Subject: LFS in-kernel cleaner can enter deadlock with build.sh
X-Send-Pr-Version: www-1.0
X-From4GNATS: "clare@csel.org via gnats" <gnats-admin@NetBSD.org>
>Number: 60820
>Category: kern
>Synopsis: LFS in-kernel cleaner can enter deadlock with build.sh
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: perseant
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Mon Sep 28 21:40:00 +0000 2026
>Last-Modified: Sat Oct 31 17:15:01 +0000 2026
>Originator: Shinichi Doyashiki
>Release: 11.99.8 around 2026-09-28 04:00 JST
>Organization:
at home
>Environment:
NetBSD kanade2.hq.csel.org 11.99.8 NetBSD 11.99.8 (KANADE2_12) #27: Mon Sep 28 04:34:19 JST 2026 clare@kanade2.hq.csel.org:/export/netbsd/debuglfs/src/sys/arch/amd64/compile/KANADE2_12 amd64
>Description:
LFS in-kernel cleaner can enter deadlock with build.sh
reproducibility is currently unknown
db{0}> ps
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
3229 3229 3 0 0 ffff89ac014a4800 find prelockcv
10122 10122 3 1 0 ffff89ac5c118400 rm prelockcv
0 21402 3 2 200 ffff89a80059f400 lfs_cleaner vnode
0 8556 3 3 200 ffff89abe64d2000 lfs_writer lfswrite
0 169 3 3 240 ffff89a7f8561c00 ioflush prelockcv
0 12 3 0 200 ffff89af30ab7800 lfsswq lfsswq
0 11 3 2 200 ffff89af30ab7400 lfscwq lfscwq
find process
db{0}> bt /a ffff89ac014a4800
trace: pid 3229 lid 3229 at 0xffff9c894c8ccd20
sleepq_block() at netbsd:sleepq_block+0xea
cv_wait() at netbsd:cv_wait+0xca
lfs_prelock() at netbsd:lfs_prelock+0x59
lfs_itimes() at netbsd:lfs_itimes+0x3c6
lfs_close() at netbsd:lfs_close+0x81
VOP_CLOSE() at netbsd:VOP_CLOSE+0xa0
vn_close() at netbsd:vn_close+0x34
closef() at netbsd:closef+0xa1
fd_close() at netbsd:fd_close+0xf3
sys_close() at netbsd:sys_close+0x22
syscall() at netbsd:syscall+0x9d
--- syscall (number 6) ---
netbsd:syscall+0x9d:
rm process
db{0}> bt /a ffff89ac5c118400
trace: pid 10122 lid 10122 at 0xffff9c895e5fbc00
sleepq_block() at netbsd:sleepq_block+0xea
cv_wait() at netbsd:cv_wait+0xca
lfs_prelock() at netbsd:lfs_prelock+0x59
lfs_vfree() at netbsd:lfs_vfree+0xc2
lfs_reclaim() at netbsd:lfs_reclaim+0x275
VOP_RECLAIM() at netbsd:VOP_RECLAIM+0x60
vcache_reclaim() at netbsd:vcache_reclaim+0x19e
vrelel() at netbsd:vrelel+0x24d
lfs_remove() at netbsd:lfs_remove+0xca
VOP_REMOVE() at netbsd:VOP_REMOVE+0xe7
do_sys_unlinkat() at netbsd:do_sys_unlinkat+0x2bc
syscall() at netbsd:syscall+0x9d
--- syscall (number 10) ---
netbsd:syscall+0x9d:
lfs_cleaner thread
db{0}> bt /a ffff89a80059f400
trace: pid 0 lid 21402 at 0xffff9c8949164a80
sleepq_block() at netbsd:sleepq_block+0xea
cv_wait() at netbsd:cv_wait+0xca
vstate_assert_wait_stable() at netbsd:vstate_assert_wait_stable+0x4b
vcache_vget() at netbsd:vcache_vget+0x34
vcache_get() at netbsd:vcache_get+0x31b
lfs_vget() at netbsd:lfs_vget+0x23
finfo_func_checkempty() at netbsd:finfo_func_checkempty+0xe2
lfs_parse_pseg() at netbsd:lfs_parse_pseg+0x39f
lfs_checkempty() at netbsd:lfs_checkempty+0xee
lfs_markclean() at netbsd:lfs_markclean+0x3e
lfs_auto_segclean() at netbsd:lfs_auto_segclean+0x24b
lfs_segunlock() at netbsd:lfs_segunlock+0x781
lfs_segwrite() at netbsd:lfs_segwrite+0x531
lfs_cleanerd() at netbsd:lfs_cleanerd+0x89e
db{0}> bt /a ffff89abe64d2000
trace: pid 0 lid 8556 at 0xffff9c895448bf50
sleepq_block() at netbsd:sleepq_block+0x233
cv_timedwait() at netbsd:cv_timedwait+0xd4
lfs_writerd() at netbsd:lfs_writerd+0x3ad
ioflush
db{0}> bt /a ffff89a7f8561c00
trace: pid 0 lid 169 at 0xffff9c8946d5ce20
sleepq_block() at netbsd:sleepq_block+0xea
cv_wait() at netbsd:cv_wait+0xca
lfs_prelock() at netbsd:lfs_prelock+0x59
lfs_seglock() at netbsd:lfs_seglock+0x16
lfs_segwrite() at netbsd:lfs_segwrite+0x71
lfs_sync() at netbsd:lfs_sync+0xc6
VFS_SYNC() at netbsd:VFS_SYNC+0x22
sched_sync() at netbsd:sched_sync+0xd6
db{0}> bt /a ffff89af30ab7800
trace: pid 0 lid 12 at 0xffff9c8936382f60
sleepq_block() at netbsd:sleepq_block+0xea
cv_wait() at netbsd:cv_wait+0xca
workqueue_worker() at netbsd:workqueue_worker+0x90
db{0}> bt /a ffff89af30ab7400
trace: pid 0 lid 11 at 0xffff9c893637bf60
sleepq_block() at netbsd:sleepq_block+0xea
cv_wait() at netbsd:cv_wait+0xca
workqueue_worker() at netbsd:workqueue_worker+0x90
db{0}> call fstrans_dump(0)
Fstrans locks by lwp:
[ 88245.5130885] 3229.322 (/lfs) shared 1 cow 0 alias 0
[ 88245.5130885] 10122.10 (/lfs) shared 2 cow 0 alias 0
[ 88245.5130885] 0.21402 (/lfs) shared 1 cow 0 alias 0
[ 88245.5130885] 0.169 (/lfs) shared 1 cow 0 alias 0
[ 88245.5130885] Fstrans state by mount:
0
db{0}> show all locks
PID LID CPU STRUCT LWP * OWNER BITS LOCK
>How-To-Repeat:
$ cat /run-build.sh
#!/bin/sh
TMPDIR=/tmp
export TMPDIR
OBJMACHINE=yes
export OBJMACHINE
testdir=/lfs/netbsd/current
while true; do
if fsck_lfs -nf /lfs; then
echo "===> FSCK OK";
else
echo "===> FSCK ERROR";
exit 1
fi
if mount /lfs; then
echo "===> MOUNT OK";
else
echo "===> MOUNT ERROR";
exit 1
fi
rm ${testdir}/build.log
for machine in amd64; do
(
cd ${testdir}/src || exit
tooldir=${testdir}/tools.${machine}
destdir=${testdir}/destdir.${machine}
objdir=${testdir}/objdir.${machine}
releasedir=${testdir}/release.test
rm -rf ${tooldir}
rm -rf ${destdir}
rm -rf ${objdir}
rm -rf ${releasedir}
mkdir -p ${objdir}
./build.sh -m ${machine} \
-j 5 \
-U \
-O ${objdir} \
-D ${destdir} \
-R ${releasedir} \
-T ${tooldir} \
release
) 2>&1 | tee ${testdir}/build.log
done
sync
if umount /lfs; then
echo "===> UMOUNT OK";
else
echo "===> UMOUNT ERROR";
exit 1
fi
done
>Fix:
unknown
>Release-Note:
>Audit-Trail:
From: Shinichi Doyashiki <clare@csel.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/60820: LFS in-kernel cleaner can enter deadlock with build.sh
Date: Wed, 07 Oct 2026 03:32:59 +0900
A deadlock was being observed, likely the reproduced.
the kernel version was 11.99.9 around 2026-10-06 05:44 JST.
db{0}> ps
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
1117 1117 3 3 1000000 ffff8509fda7dc00 rm prelockcv
0 1078 3 3 200 ffff8509f8b4c800 lfs_cleaner vnode
0 2227 3 2 200 ffff8509fcbfbc00 lfs_writer lfswrite
0 170 3 0 200 ffff8509f8891c00 ioflush prelockcv
db{0}> show all locks
[Locks tracked through LWPs]
****** LWP 1117.1117 (rm) @ 0xffff8509fda7dc00, l_stat=3
*** Locks held:
* Lock 0 (initialized at netbsd:vcache_alloc+0x6b)
lock address : ffff850a41e1bf00
type : sleep/adaptive
initialized : netbsd:vcache_alloc+0x6b
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
relevant cpu : 3 last held: 2
relevant lwp : 0xffff8509fda7dc00 last held: 0xffff8509fda7dc00
last locked* : netbsd:genfs_lock+0x89
unlocked : netbsd:genfs_unlock+0x14
owner/count : 0xffff8509fda7dc00 flags : 0x0000000000000004
Turnstile: no active turnstile for this lock.
*** Locks wanted: none
****** LWP 1443.1443 (ntpd) @ 0xffff8509fbba4000, l_stat=3
*** Locks held: none
*** Locks wanted:
* Lock 0 (initialized at netbsd:module_hook_init+0x17)
lock address : netbsd:module_hook
type : sleep/adaptive
initialized : netbsd:module_hook_init+0x17
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 0
relevant cpu : 3 last held: 0
relevant lwp : 0xffff8509fbba4000 last held: 000000000000000000
last locked : 0
unlocked* : 0
owner field : 000000000000000000 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
****** LWP 389.389 (rpcbind) @ 0xffff8509faf96400, l_stat=3
*** Locks held: none
*** Locks wanted:
* Lock 0 (initialized at netbsd:module_hook_init+0x17)
lock address : netbsd:module_hook
type : sleep/adaptive
initialized : netbsd:module_hook_init+0x17
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 0
relevant cpu : 1 last held: 0
relevant lwp : 0xffff8509faf96400 last held: 000000000000000000
last locked : 0
unlocked* : 0
owner field : 000000000000000000 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
****** LWP 692.692 (dhcpcd) @ 0xffff8509f921e800, l_stat=3
*** Locks held: none
*** Locks wanted:
* Lock 0 (initialized at netbsd:module_hook_init+0x17)
lock address : netbsd:module_hook
type : sleep/adaptive
initialized : netbsd:module_hook_init+0x17
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 0
relevant cpu : 3 last held: 0
relevant lwp : 0xffff8509f921e800 last held: 000000000000000000
last locked : 0
unlocked* : 0
owner field : 000000000000000000 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
****** LWP 679.679 (dhcpcd) @ 0xffff8509f98c7c00, l_stat=3
*** Locks held: none
*** Locks wanted:
* Lock 0 (initialized at netbsd:module_hook_init+0x17)
lock address : netbsd:module_hook
type : sleep/adaptive
initialized : netbsd:module_hook_init+0x17
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 0
relevant cpu : 2 last held: 0
relevant lwp : 0xffff8509f98c7c00 last held: 000000000000000000
last locked : 0
unlocked* : 0
owner field : 000000000000000000 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
****** LWP 0.1078 (lfs_cleaner) @ 0xffff8509f8b4c800, l_stat=3
*** Locks held:
* Lock 0 (initialized at netbsd:lfs_mountfs+0x537)
lock address : ffff850a07f822b8
type : sleep/adaptive
initialized : netbsd:lfs_mountfs+0x537
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
relevant cpu : 3 last held: 3
relevant lwp : 0xffff8509f8b4c800 last held: 0xffff8509f8b4c800
last locked* : netbsd:lfs_prelock+0xaa
unlocked : netbsd:lfs_preunlock+0x77
owner/count : 0xffff8509f8b4c800 flags : 0x0000000000000004
Turnstile: no active turnstile for this lock.
*** Locks wanted: none
****** LWP 0.28 (softclk/1) @ 0xffff8509f715f800, l_stat=1
*** Locks held: none
*** Locks wanted:
* Lock 0 (initialized at netbsd:module_hook_init+0x17)
lock address : netbsd:module_hook
type : sleep/adaptive
initialized : netbsd:module_hook_init+0x17
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 0
relevant cpu : 1 last held: 0
relevant lwp : 0xffff8509f715f800 last held: 000000000000000000
last locked : 0
unlocked* : 0
owner field : 000000000000000000 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
****** LWP 0.13 (iflnkst) @ 0xffff851130aa9c00, l_stat=3
*** Locks held: none
*** Locks wanted:
* Lock 0 (initialized at netbsd:module_hook_init+0x17)
lock address : netbsd:module_hook
type : sleep/adaptive
initialized : netbsd:module_hook_init+0x17
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 0
relevant cpu : 2 last held: 0
relevant lwp : 0xffff851130aa9c00 last held: 000000000000000000
last locked : 0
unlocked* : 0
owner field : 000000000000000000 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
****** LWP 0.3 (softnet/0) @ 0xffff851133b13400, l_stat=1
*** Locks held: none
*** Locks wanted:
* Lock 0 (initialized at netbsd:module_hook_init+0x17)
lock address : netbsd:module_hook
type : sleep/adaptive
initialized : netbsd:module_hook_init+0x17
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 0
relevant cpu : 0 last held: 0
relevant lwp : 0xffff851133b13400 last held: 000000000000000000
last locked : 0
unlocked* : 0
owner field : 000000000000000000 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
****** LWP 0.0 (swapper) @ 0xffffffff814648c0, l_stat=3
*** Locks held: none
*** Locks wanted:
* Lock 0 (initialized at netbsd:module_hook_init+0x17)
lock address : netbsd:module_hook
type : sleep/adaptive
initialized : netbsd:module_hook_init+0x17
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 0
relevant cpu : 2 last held: 0
relevant lwp : 0xffffffff814648c0 last held: 000000000000000000
last locked : 0
unlocked* : 0
owner field : 000000000000000000 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
[Locks tracked through CPUs]
******* Locks held on cpu0:
* Lock 0 (initialized at netbsd:com_attach_subr+0x18c)
lock address : ffff851134b807b0
type : spin
initialized : netbsd:com_attach_subr+0x18c
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
relevant cpu : 0 last held: 0
relevant lwp : 0xffff851133b13000 last held: 0xffff851133b13000
last locked* : netbsd:comintr+0x52
unlocked : netbsd:comintr+0x891
owner field : 0x0000000000010800 wait/spin: 0/1
db{0}> show proc /a ffff8509f8b4c800
lwp_t ffff8509f8b4c800
system: pid 0 proc ffffffff81465040 vmspace/map ffffffff8155a2e0 flags 20002
lwp 1078 [lfs_cleaner] ffff8509f8b4c800 pcb ffff948946d4a000
stat 3 flags 200 cpu 3 pri 123 ref 0
wmesg vnode wchan ffff850a39e3d580
lwp 2227 [lfs_writer] ffff8509fcbfbc00 pcb ffff948947625000
stat 3 flags 200 cpu 2 pri 123 ref 0
wmesg lfswrite wchan ffffffff814f3e50
lwp 170 [ioflush] ffff8509f8891c00 pcb ffff948946d58000
stat 3 flags 200 cpu 0 pri 124 ref 0
wmesg prelockcv wchan ffff850a07f82228
lwp 12 [lfsswq] ffff851130aa9800 pcb ffff94893637e000
stat 3 flags 200 cpu 1 pri 123 ref 0
wmesg lfsswq wchan ffff851133b34448
lwp 11 [lfscwq] ffff851130aa9400 pcb ffff948936377000
stat 3 flags 200 cpu 2 pri 123 ref 0
wmesg lfscwq wchan ffff851133b34308
the cleaner thread
db{0}> trace /a ffff8509f8b4c800
trace: pid 0 lid 1078 at 0xffff948946d4ea80
sleepq_block() at netbsd:sleepq_block+0xea
cv_wait() at netbsd:cv_wait+0xca
vstate_assert_wait_stable() at netbsd:vstate_assert_wait_stable+0x4b
vcache_vget() at netbsd:vcache_vget+0x34
vcache_get() at netbsd:vcache_get+0x31b
lfs_vget() at netbsd:lfs_vget+0x23
finfo_func_checkempty() at netbsd:finfo_func_checkempty+0xf4
lfs_parse_pseg() at netbsd:lfs_parse_pseg+0x3f4
lfs_checkempty() at netbsd:lfs_checkempty+0xfc
lfs_markclean() at netbsd:lfs_markclean+0x4c
lfs_auto_segclean() at netbsd:lfs_auto_segclean+0x259
lfs_segunlock() at netbsd:lfs_segunlock+0x727
lfs_segwrite() at netbsd:lfs_segwrite+0x53f
lfs_cleanerd() at netbsd:lfs_cleanerd+0x8d0
db{0}> trace /a ffff8509fcbfbc00
trace: pid 0 lid 2227 at 0xffff948947629f50
sleepq_block() at netbsd:sleepq_block+0x233
cv_timedwait() at netbsd:cv_timedwait+0xd4
lfs_writerd() at netbsd:lfs_writerd+0x3bb
db{0}> trace /a ffff8509f8891c00
trace: pid 0 lid 170 at 0xffff948946d5ce20
sleepq_block() at netbsd:sleepq_block+0xea
cv_wait() at netbsd:cv_wait+0xca
lfs_prelock() at netbsd:lfs_prelock+0x59
lfs_seglock() at netbsd:lfs_seglock+0x16
lfs_segwrite() at netbsd:lfs_segwrite+0x7f
lfs_sync() at netbsd:lfs_sync+0xc6
VFS_SYNC() at netbsd:VFS_SYNC+0x22
sched_sync() at netbsd:sched_sync+0xe4
the rm process
db{0}> show proc /a ffff8509fda7dc00
lwp_t ffff8509fda7dc00
rm: pid 1117 proc ffff8509f93477c0 vmspace/map ffff8509fb3b9900 flags 4000
lwp 1117 ffff8509fda7dc00 pcb ffff948947544000
stat 3 flags 1000000 cpu 3 pri 63 ref 0
wmesg prelockcv wchan ffff850a07f82228
db{0}> trace /a ffff8509fda7dc00
trace: pid 1117 lid 1117 at 0xffff948947548c20
sleepq_block() at netbsd:sleepq_block+0xea
cv_wait() at netbsd:cv_wait+0xca
lfs_prelock() at netbsd:lfs_prelock+0x59
lfs_vfree() at netbsd:lfs_vfree+0xd4
lfs_reclaim() at netbsd:lfs_reclaim+0x275
VOP_RECLAIM() at netbsd:VOP_RECLAIM+0x60
vcache_reclaim() at netbsd:vcache_reclaim+0x1cc
vrelel() at netbsd:vrelel+0x251
lfs_rmdir() at netbsd:lfs_rmdir+0xc3
VOP_RMDIR() at netbsd:VOP_RMDIR+0xe0
do_sys_unlinkat() at netbsd:do_sys_unlinkat+0x2eb
syscall() at netbsd:syscall+0xaf
--- syscall (number 137) ---
netbsd:syscall+0xaf:
db{0}> call fstrans_dump(0)
Fstrans locks by lwp:
[ 71281.6153000] 1117.111 (/lfs) shared 2 cow 0 alias 0
[ 71281.6153000] 0.1078 (/lfs) shared 1 cow 0 alias 0
[ 71281.6153000] 0.170 (/lfs) shared 1 cow 0 alias 0
[ 71281.6153000] Fstrans state by mount:
0
Responsible-Changed-From-To: kern-bug-people->perseant
Responsible-Changed-By: perseant@NetBSD.org
Responsible-Changed-When: Sat, 24 Oct 2026 00:07:19 +0000
Responsible-Changed-Why:
take
From: Konrad Schroder <perseant@hhhh.org>
To: gnats-bugs@netbsd.org, kern-bug-people@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, clare@csel.org
Cc:
Subject: Re: kern/60820: LFS in-kernel cleaner can enter deadlock with
build.sh
Date: Fri, 23 Oct 2026 17:21:43 -0700
On 10/6/2026 11:35 AM, Shinichi Doyashiki via gnats wrote:
> the cleaner thread
>
> vstate_assert_wait_stable() at netbsd:vstate_assert_wait_stable+0x4b
> vcache_vget() at netbsd:vcache_vget+0x34
>
> the rm process
>
> lfs_prelock() at netbsd:lfs_prelock+0x59
> lfs_vfree() at netbsd:lfs_vfree+0xd4
> lfs_reclaim() at netbsd:lfs_reclaim+0x275
> VOP_RECLAIM() at netbsd:VOP_RECLAIM+0x60
> vcache_reclaim() at netbsd:vcache_reclaim+0x1cc
The essence of the deadlock is here: the cleaner, holding the segment
lock, has to vget the vnode to check its version number, so waits for
the vnode state to reach VS_RECLAIMED; while VOP_RECLAIM, waiting for
the segment lock, is holding that vnode in VS_RECLAIMING, so vcache_vget
will never return.
One possibility might be simply not to check that segments are actually
clean before releasing them. If all is well with LFS, we would trust
the accounting and would not need to check; but of course there are
still bugs, so this check is turned on if compiled with DEBUG.
Another possibility might be to pass a flag to vcache_get() that would
allow it to return if the vnode state is VS_RECLAIMING (or maybe,
specify allowable states?), explicitly avoiding this use case. This
seems a bit hackish but would avoid the deadlock.
--
Konrad Schroder
perseant@hhhh.org
From: "Konrad Schroder" <perseant@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/60820 CVS commit: src/sys/ufs/lfs
Date: Fri, 30 Oct 2026 01:03:04 +0000
Module Name: src
Committed By: perseant
Date: Fri Oct 30 01:03:04 UTC 2026
Modified Files:
src/sys/ufs/lfs: lfs.h lfs_accessors.h lfs_extern.h lfs_segment.c
lfs_subr.c lfs_syscalls.c lfs_vfsops.c
Log Message:
Reclaim empty segments in a separate thread, rather than doing so at
segment unlock. This allows segment reclaim to be done without the segment
lock. Should address PR 60820.
To generate a diff of this commit:
cvs rdiff -u -r1.220 -r1.221 src/sys/ufs/lfs/lfs.h
cvs rdiff -u -r1.60 -r1.61 src/sys/ufs/lfs/lfs_accessors.h
cvs rdiff -u -r1.132 -r1.133 src/sys/ufs/lfs/lfs_extern.h
cvs rdiff -u -r1.317 -r1.318 src/sys/ufs/lfs/lfs_segment.c
cvs rdiff -u -r1.114 -r1.115 src/sys/ufs/lfs/lfs_subr.c
cvs rdiff -u -r1.183 -r1.184 src/sys/ufs/lfs/lfs_syscalls.c
cvs rdiff -u -r1.407 -r1.408 src/sys/ufs/lfs/lfs_vfsops.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Shinichi Doyashiki <clare@csel.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/60820: LFS in-kernel cleaner can enter deadlock with build.sh
Date: Sat, 31 Oct 2026 14:18:29 +0900
After updating the kernel and running the test script, I observed a new deadlock.
db{0}> ps
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
2599 2599 3 1 1000000 ffff91cff881f400 umount rwlock
18209 18209 3 0 0 ffff91cffbd28800 sh fstchg
0 18260 3 3 240 ffff91cff881fc00 lfs_writer fstchg
0 18580 3 1 200 ffff91cff8ca7400 lfs_reclaim fstchg
0 12 3 2 200 ffff91d730aa9800 lfsswq lfsswq
0 11 3 1 200 ffff91d730aa9400 lfscwq lfscwq
db{0}> show proc /a ffff91cff881f400
lwp_t ffff91cff881f400
umount: pid 2599 proc ffff91cff8613740 vmspace/map ffff91d03fbf4580 flags 4000
lwp 2599 ffff91cff881f400 pcb ffff94894719e000
stat 3 flags 1000000 cpu 1 pri 63 ref 0
wmesg rwlock wchan ffff91d01f6342b8
db{0}> bt /a ffff91cff881f400
trace: pid 2599 lid 2599 at 0xffff9489471a2c30
sleepq_block() at netbsd:sleepq_block+0xea
turnstile_block() at netbsd:turnstile_block+0x2d0
rw_enter() at netbsd:rw_enter+0x17c
lfs_prelock() at netbsd:lfs_prelock+0xaa
lfs_seglock() at netbsd:lfs_seglock+0x16
lfs_segwrite() at netbsd:lfs_segwrite+0x7f
lfs_sync() at netbsd:lfs_sync+0xc6
VFS_SYNC() at netbsd:VFS_SYNC+0x22
dounmount() at netbsd:dounmount+0x158
sys_unmount() at netbsd:sys_unmount+0x157
syscall() at netbsd:syscall+0xaf
--- syscall (number 22) ---
netbsd:syscall+0xaf:
db{0}> show proc /a ffff91cff881fc00
lwp_t ffff91cff881fc00
system: pid 0 proc ffffffff81465040 vmspace/map ffffffff8155a2e0 flags 20002
lwp 18260 [lfs_writer] ffff91cff881fc00 pcb ffff9489473f9000
stat 3 flags 240 cpu 3 pri 123 ref 0
wmesg fstchg wchan ffffffff815765b0
lwp 18580 [lfs_reclaim] ffff91cff8ca7400 pcb ffff948947627000
stat 3 flags 200 cpu 1 pri 123 ref 0
wmesg fstchg wchan ffffffff815765b0
lwp 11 [lfscwq] ffff91d730aa9400 pcb ffff948936377000
stat 3 flags 200 cpu 1 pri 123 ref 0
wmesg lfscwq wchan ffff91d733b34308
db{0}> bt /a ffff91cff881fc00
trace: pid 0 lid 18260 at 0xffff9489473fdf10
sleepq_block() at netbsd:sleepq_block+0xea
cv_wait() at netbsd:cv_wait+0xca
fstrans_start() at netbsd:fstrans_start+0x14c
_mountlist_iterator_next() at netbsd:_mountlist_iterator_next+0x110
lfs_writerd() at netbsd:lfs_writerd+0x13f
db{0}> bt /a ffff91cff8ca7400
trace: pid 0 lid 18580 at 0xffff94894762bbf0
sleepq_block() at netbsd:sleepq_block+0xea
cv_wait() at netbsd:cv_wait+0xca
fstrans_start() at netbsd:fstrans_start+0x14c
vfs_busy() at netbsd:vfs_busy+0x1c
vcache_get() at netbsd:vcache_get+0xfe
lfs_vget() at netbsd:lfs_vget+0x23
finfo_func_checkempty() at netbsd:finfo_func_checkempty+0xf4
lfs_parse_pseg() at netbsd:lfs_parse_pseg+0x3f4
lfs_checkempty() at netbsd:lfs_checkempty+0xfc
lfs_markclean() at netbsd:lfs_markclean+0x51
lfs_auto_segclean() at netbsd:lfs_auto_segclean+0x29e
lfs_reclaimd() at netbsd:lfs_reclaimd+0x50
db{0}> show all locks
[Locks tracked through LWPs]
****** LWP 2599.2599 (umount) @ 0xffff91cff881f400, l_stat=3
*** Locks held:
* Lock 0 (initialized at netbsd:fstrans_init+0x1b)
lock address : netbsd:vfs_suspend_lock
type : sleep/adaptive
initialized : netbsd:fstrans_init+0x1b
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
relevant cpu : 1 last held: 2
relevant lwp : 0xffff91cff881f400 last held: 0xffff91cff881f400
last locked* : netbsd:vfs_suspend+0xdb
unlocked : netbsd:vrevoke+0xc8
owner field : 0xffff91cff881f400 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
* Lock 1 (initialized at netbsd:vfs_mountalloc+0x76)
lock address : ffff91d4d4c1c1c0
type : sleep/adaptive
initialized : netbsd:vfs_mountalloc+0x76
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
relevant cpu : 1 last held: 2
relevant lwp : 0xffff91cff881f400 last held: 0xffff91cff881f400
last locked* : netbsd:dounmount+0x65
unlocked : netbsd:do_sys_sync+0x32
owner field : 0xffff91cff881f400 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
*** Locks wanted:
* Lock 0 (initialized at netbsd:config_init+0x3a)
lock address : netbsd:config_misc_lock
type : sleep/adaptive
initialized : netbsd:config_init+0x3a
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 0
relevant cpu : 1 last held: 2
relevant lwp : 0xffff91cff881f400 last held: 000000000000000000
last locked : netbsd:device_lookup_acquire+0x22
unlocked* : netbsd:device_lookup_acquire+0xe6
owner field : 000000000000000000 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
*** Locks held:
* Lock 0 (initialized at netbsd:lfs_mountfs+0x536)
lock address : ffff91d01f6342b8
type : sleep/adaptive
initialized : netbsd:lfs_mountfs+0x536
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 1
relevant cpu : 1 last held: 1
relevant lwp : 0xffff91cff8ca7400 last held: 0xffff91cff8ca7400
last locked* : netbsd:lfs_reclaimd+0x48
unlocked : netbsd:lfs_preunlock+0x77
owner/count : 0xffff91cff8ca7400 flags : 0x0000000000000007
Turnstile:
=> 0 waiting readers:
=> 1 waiting writers: 0xffff91cff881f400
*** Locks wanted: none
db{0}> call fstrans_dump(0)
Fstrans locks by lwp:
[ 19612.1666798] Fstrans state by mount:
[ 19612.1666798] /lfs owner 0xffff91cff881f400 state suspended
0
db{0}> call fstrans_dump(1)
Fstrans locks by lwp:
[ 19612.1666798] 2599.259 @0xffff91d041837080 (/var/shm) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91d04094ea00 (/export) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91d040e86400 (/tmp) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91d03d0f2840 (/dev/pts) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91d4d3cd79c0 (/) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d2740 (/lfs) - 0 cow 0 alias 0
[ 19612.1666798] 29277.29 @0xffff91d04094e880 (/) - 0 cow 0 alias 0
[ 19612.1666798] 3636.363 @0xffff91cff93d2200 (/) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cffb526940 (/dev/pts) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d23c0 (/tmp) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d2340 (/export) - 0 cow 0 alias 0
[ 19612.1666798] 18833.18 @0xffff91cff93d2800 (/dev/pts) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff93d2580 (/tmp) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cffb526a40 (/) - 0 cow 0 alias 0
[ 19612.1666798] 18682.18 @0xffff91cff93d2480 (/) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff93d2080 (/dev/pts) - 0 cow 0 alias 0
[ 19612.1666798] 18209.18 @0xffff91d044320340 (/lfs) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff93d2780 (/export) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cffb526840 (/dev/pts) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cffb526140 (/) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cffb526b40 (/tmp) - 0 cow 0 alias 0
[ 19612.1666798] 18515.18 @0xffff91cff93d2400 (/dev/pts) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff93d2280 (/tmp) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff93d2180 (/) - 0 cow 0 alias 0
[ 19612.1666798] 17835.17 @0xffff91cffb526f40 (/) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cffb526cc0 (/dev/pts) - 0 cow 0 alias 0
[ 19612.1666798] 19386.19 @0xffff91cffb526c40 (/dev/pts) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cffb526040 (/) - 0 cow 0 alias 0
[ 19612.1666798] 17822.17 @0xffff91cffb526d40 (/dev/pts) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cffb5266c0 (/) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d27c0 (/tmp) - 0 cow 0 alias 0
[ 19612.1666798] 1956.195 @0xffff91cff93d2a80 (/lfs) gone - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cffb526ac0 (/tmp) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d24c0 (/dev/pts) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d22c0 (/) - 0 cow 0 alias 0
[ 19612.1666798] 845.845 @0xffff91cffb5269c0 (/) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cffb5260c0 (/dev/pts) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d2840 (/tmp) - 0 cow 0 alias 0
[ 19612.1666798] 1981.198 @0xffff91cff86d2440 (/) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d2240 (/dev/pts) - 0 cow 0 alias 0
[ 19612.1666798] 1992.199 @0xffff91cff86d2bc0 (/) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d2540 <dead> - 0 cow 0 alias 0
[ 19612.1666798] 1196.119 @0xffff91cffb5262c0 (/) - 0 cow 0 alias 0
[ 19612.1666798] 811.811 @0xffff91cff93d2d00 (/) - 0 cow 0 alias 0
[ 19612.1666798] 912.912 @0xffff91cff86d2a40 (/) - 0 cow 0 alias 0
[ 19612.1666798] 1695.169 @0xffff91cffb526440 (/) - 0 cow 0 alias 0
[ 19612.1666798] 1208.120 @0xffff91cffb526740 (/) - 0 cow 0 alias 0
[ 19612.1666798] 553.553 @0xffff91cff86d2b40 (/) - 0 cow 0 alias 0
[ 19612.1666798] 552.552 @0xffff91cff93d2d80 (/) - 0 cow 0 alias 0
[ 19612.1666798] 1315.131 @0xffff91cff86d2c40 (/) - 0 cow 0 alias 0
[ 19612.1666798] 500.500 @0xffff91cffb526340 (/) - 0 cow 0 alias 0
[ 19612.1666798] 348.348 @0xffff91cff93d2e00 (/) - 0 cow 0 alias 0
[ 19612.1666798] 830.325 @0xffff91cffb5267c0 (/export) - 0 cow 0 alias 0
[ 19612.1666798] 830.324 @0xffff91cffb526bc0 (/export) - 0 cow 0 alias 0
[ 19612.1666798] 830.830 @0xffff91cffb5264c0 (/) - 0 cow 0 alias 0
[ 19612.1666798] 823.823 @0xffff91cff93d2f80 (/) - 0 cow 0 alias 0
[ 19612.1666798] 390.390 @0xffff91cff86d2cc0 (/) - 0 cow 0 alias 0
[ 19612.1666798] 934.1004 @0xffff91cff86d2ac0 (/) - 0 cow 0 alias 0
[ 19612.1666798] 934.1020 @0xffff91cff93d2880 (/) - 0 cow 0 alias 0
[ 19612.1666798] 934.1096 @0xffff91cff93d2e80 (/) - 0 cow 0 alias 0
[ 19612.1666798] 934.568 @0xffff91cff86d2640 (/) - 0 cow 0 alias 0
[ 19612.1666798] 934.934 @0xffff91cff93d2f00 (/) - 0 cow 0 alias 0
[ 19612.1666798] 702.702 @0xffff91cff93d2b80 (/) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d29c0 <dead> - 0 cow 0 alias 0
[ 19612.1666798] 695.695 @0xffff91cff93d2600 (/) - 0 cow 0 alias 0
[ 19612.1666798] 699.699 @0xffff91cff93d2900 (/) - 0 cow 0 alias 0
[ 19612.1666798] 692.692 @0xffff91cff86d26c0 (/) - 0 cow 0 alias 0
[ 19612.1666798] 679.679 @0xffff91cff93d2700 (/) - 0 cow 0 alias 0
[ 19612.1666798] 1.1 @0xffff91d04094e180 (/lfs) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d2140 (/) - 0 cow 0 alias 0
[ 19612.1666798] 0.19847 @0xffff91d0443203c0 (/lfs) - 0 cow 0 alias 0
[ 19612.1666798] 0.18260 @0xffff91cff93d2100 (/lfs) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d2f40 (/export) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d2ec0 (/var/shm) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d2e40 (/dev/pts) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d2dc0 (/tmp) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d2940 (/) - 0 cow 0 alias 0
[ 19612.1666798] 0.18580 @0xffff91cffb5261c0 (/lfs) - 0 cow 0 alias 0
[ 19612.1666798] 0.169 @0xffff91cffb526240 (/lfs) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cffb5265c0 (/dev/pts) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cffb526540 (/tmp) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff93d2680 (/var/shm) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d2d40 (/export) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d25c0 <dead> - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d2040 (/) - 0 cow 0 alias 0
[ 19612.1666798] 0.168 @0xffff91cff93d2b00 (/export) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff93d2980 (/) - 0 cow 0 alias 0
[ 19612.1666798] 0.0 @0xffff91cff86d21c0 (/) - 0 cow 0 alias 0
[ 19612.1666798] @0xffff91cff86d20c0 <dead> - 0 cow 0 alias 0
[ 19612.1666798] Fstrans state by mount:
[ 19612.1666798] / owner 0x0 state normal
[ 19612.1666798] /tmp owner 0x0 state normal
[ 19612.1666798] /dev/pts owner 0x0 state normal
[ 19612.1666798] /var/shm owner 0x0 state normal
[ 19612.1666798] /export owner 0x0 state normal
[ 19612.1666798] /lfs owner 0xffff91cff881f400 state suspended
0
From: Shinichi Doyashiki <clare@csel.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/60820: LFS in-kernel cleaner can enter deadlock with build.sh
Date: Sun, 01 Nov 2026 02:09:56 +0900
another issue was observed in LFS mount (lfs_reclaimd).
# mount /lfs
[ 224.8750501] Reader / writer lock error: rw_vector_enter,301: locking against myself
[ 224.8850397] lock address : ffff8e1302ad62b8
[ 224.8850397] type : sleep/adaptive
[ 224.8850397] initialized : netbsd:lfs_mountfs+0x536
[ 224.8950340] shared holds : 0 exclusive: 1
[ 224.9050289] shares wanted: 1 exclusive: 1
[ 224.9050289] relevant cpu : 2 last held: 1
[ 224.9150230] relevant lwp : 0xffff8e12fc311800 last held: 0xffff8e12fc311800
[ 224.9250178] last locked* : netbsd:lfs_reclaimd+0x48
[ 224.9250178] unlocked : netbsd:lfs_mountfs+0x12df
[ 224.9350128] owner/count : 0xffff8e12fc311800 flags : 0x0000000000000005
[ 224.9350128] Turnstile:
[ 224.9450076] => 1 waiting readers: 0xffff8e12f9040000
[ 224.9450076] => 0 waiting writers:
[ 224.9450076] panic: LOCKDEBUG: Reader / writer lock error: rw_vector_enter,301: locking against myself
[ 224.9550056] cpu2: Begin traceback...
[ 224.9650025] vpanic() at netbsd:vpanic+0x171
[ 224.9650025] panic() at netbsd:panic+0x4a
[ 224.9749924] lockdebug_abort1() at netbsd:lockdebug_abort1+0xdd
[ 224.9849873] rw_enter() at netbsd:rw_enter+0x34d
[ 224.9849873] lfs_prelock() at netbsd:lfs_prelock+0xaa
[ 224.9949822] lfs_vfree() at netbsd:lfs_vfree+0xd4
[ 224.9949822] lfs_reclaim() at netbsd:lfs_reclaim+0x275
[ 225.0049771] VOP_RECLAIM() at netbsd:VOP_RECLAIM+0x60
[ 225.0049771] vcache_reclaim() at netbsd:vcache_reclaim+0x1c1
[ 225.0149720] vrelel() at netbsd:vrelel+0x251
[ 225.0249669] finfo_func_checkempty() at netbsd:finfo_func_checkempty+0x225
[ 225.0249669] lfs_parse_pseg() at netbsd:lfs_parse_pseg+0x3f4
[ 225.0349618] lfs_checkempty() at netbsd:lfs_checkempty+0xfc
[ 225.0449567] lfs_markclean() at netbsd:lfs_markclean+0x51
[ 225.0449567] lfs_auto_segclean() at netbsd:lfs_auto_segclean+0x29e
[ 225.0549517] lfs_reclaimd() at netbsd:lfs_reclaimd+0x50
[ 225.0549517] cpu2: End traceback...
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home