NetBSD Problem Report #59663
From www@netbsd.org Sun Sep 21 07:26:30 2025
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits)
client-signature RSA-PSS (2048 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id A19091A923F
for <gnats-bugs@gnats.NetBSD.org>; Sun, 21 Sep 2025 07:26:30 +0000 (UTC)
Message-Id: <20250921072628.C084D1A9241@mollari.NetBSD.org>
Date: Sun, 21 Sep 2025 07:26:28 +0000 (UTC)
From: hpaluch@seznam.cz
Reply-To: hpaluch@seznam.cz
To: gnats-bugs@NetBSD.org
Subject: ffs_snapshot_read -> uvm_fault (or pool page empty) doing dump/restore with snapshot
X-Send-Pr-Version: www-1.0
>Number: 59663
>Notify-List: riastradh
>Category: kern
>Synopsis: ffs_snapshot_read -> uvm_fault (or pool page empty) doing dump/restore with snapshot
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: hannken
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sun Sep 21 07:30:03 +0000 2025
>Closed-Date: Wed Sep 02 18:52:13 +0000 2026
>Last-Modified: Wed Sep 02 18:52:13 +0000 2026
>Originator: Henryk Paluch
>Release: 11.0_BETA 20250916040529Z
>Organization:
N/A
>Environment:
NetBSD nbsd11-4zotac.example.com 11.0_BETA NetBSD 11.0_BETA (GENERIC) #0: Tue Sep 16 04:05:29 UTC 2025 mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/amd64/compile/GENERIC amd64
>Description:
Having NetBSD 11.0_BETA guest (Host is Linux KVM, openSUSE LEAP 15.6).
1. NetBSD installed on Virtio-BLK disk ld0 64GB
2. target ld1 is 500GB SATA SSD attached to host via USB JMicron adapter, exposed as Virtio-BLK /dev/sdb block device from host
When I bulk copy lot of data (46GB) using dump/restore with snapshot I will after while get panic:
uvm_fault(0xffffffff81976c20, 0x0, 1) -> e
[ 883.0589643] fatal page fault in supervisor mode
[ 883.0589643] trap type 6 code 0 rip 0xffffffff80e261f3 cs 0x8 rflags 0x10202 cr2 0x9 ilevel 0 rsp 0xffff9d04c49d4c10
[ 883.0589643] curlwp 0xffff8b60ae5fe400 pid 0.1215 lowest kstack 0xffff9d04c49d02c0
kernel: page fault trap, code=0
Stopped in pid 0.1215 (system) at netbsd:pool_get+0x2b9: cmpq %r15,8(%
rax)
pool_get() at netbsd:pool_get+0x2b9
allocbuf() at netbsd:allocbuf+0x113
getblk() at netbsd:getblk+0x18c
bio_doread() at netbsd:bio_doread+0x1d
breadn() at netbsd:breadn+0x8b
ffs_snapshot_read() at netbsd:ffs_snapshot_read+0x1b2
VOP_READ() at netbsd:VOP_READ+0x42
vn_rdwr() at netbsd:vn_rdwr+0xf1
fss_bs_io() at netbsd:fss_bs_io+0x89
fss_bs_thread() at netbsd:fss_bs_thread+0x50f
ds 1
es e400
fs 4bf0
gs 5ff8
rdi ffffffff819835e0 bmempools+0x7e0
rsi 2
rbp ffff9d04c49d4c70
rbx ffffffff81983530 bmempools+0x730
rdx ffffffff
rcx 0
--db_more-- rax 1
r8 0
r9 ffff9d04cbea0030
r10 fffffffe
r11 0
r12 2
r13 ffffffff819835e0 bmempools+0x7e0
r14 ffff9d04cbea0c00
r15 ffff9d04cbea0000
rip ffffffff80e261f3 pool_get+0x2b9
cs 8
rflags 10202
rsp ffff9d04c49d4c10
ss 10
netbsd:pool_get+0x2b9: cmpq %r15,8(%rax)
db{0}> ?
Because there is ffs_snapshot_read() right before panic I suspect that it somehow exhaust all buffers.
Below is output from: vmstat -M netbsd.2.core -N netbsd.2 -m
Memory resource pool statistics
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
amappl 128 1171 0 130 36 1 35 36 0 inf 0
anonpl 40 76445 0 576 755 0 755 755 0 inf 0
biopl 280 54387 0 42890 1176 354 822 971 0 inf 0
buf16k 16896 183710 0 180200 50961 49790 1171 35313 0 inf 0
buf1k 1536 2 0 2 2 1 1 2 0 inf 1
buf2k 2048 372 0 344 187 172 15 173 0 inf 1
buf32k 32768 16466 0 4355 6893 837 6056 6064 0 inf 0
buf4k 4096 280578 0 173240 280579 173240 107339 226999 0 inf 1
buf512b 1024 3 0 3 1 0 1 1 0 inf 1
buf64k 65536 6 0 2 7 2 5 7 0 inf 1
buf8k 8704 1143 0 745 75 18 57 65 0 inf 0
bufpl 280 256490 0 133102 16813 5064 11749 16813 0 inf 0
dirhash 376 2262 0 411 186 0 186 186 0 inf 0
dirhashblk 2048 7684 0 1147 3269 0 3269 3269 0 inf 0
ehcixfer 376 4 0 3 1 0 1 1 0 inf 0
execargs 262144 243 0 243 8 7 1 4 0 16 1
ffsdino2 264 980776 0 155749 55861 0 55861 55861 0 inf 859
ffsino 280 980333 0 155306 59851 0 59851 59851 0 inf 920
file 128 275 0 81 8 0 8 8 0 inf 0
filedesc 832 119 0 86 28 10 18 28 0 inf 0
fstlwp 128 149 0 59 4 0 4 4 0 inf 0
in4pcbpl 256 13 0 7 1 0 1 1 0 inf 0
in6pcbpl 320 11 0 8 1 0 1 1 0 inf 0
inmltpl 56 2 0 0 1 0 1 1 0 inf 0
kcredpl 192 85 0 22 4 0 4 4 0 inf 0
kmem-00016 16 2209 0 301 8 0 8 8 0 inf 0
kmem-00032 32 3652 0 440 26 0 26 26 0 inf 0
kmem-00064 128 9172 0 6193 292 170 122 292 0 inf 0
kmem-00128 192 1907040 0 250751 80096 2 80094 80096 0 inf 1223
kmem-00192 256 362516 0 50703 20293 804 19489 20251 0 inf 0
kmem-00256 320 430 0 89 30 0 30 30 0 inf 0
kmem-00320 384 743 0 76 72 0 72 72 0 inf 0
kmem-00384 448 669 0 106 73 3 70 73 0 inf 0
kmem-00448 512 157 0 56 18 0 18 18 0 inf 0
kmem-00512 576 113 0 58 14 1 13 14 0 inf 0
kmem-00768 832 344 0 66 83 3 80 83 0 inf 0
kmem-01024 1088 723 0 47 238 0 238 238 0 inf 0
kmem-02048 2048 710 0 231 259 19 240 253 0 inf 0
kmem-04096 4096 121 0 30 116 25 91 115 0 inf 0
ksiginfo 136 359 0 260 5 1 4 4 0 inf 0
kva-12288 12288 526 0 7 26 0 26 26 0 inf 0
kva-16384 16384 11 0 8 1 0 1 1 0 inf 0
kva-20480 20480 17 0 10 2 1 1 2 0 inf 0
kva-24576 24576 1 0 1 1 1 0 1 0 inf 0
kva-28672 28672 10 0 0 2 0 2 2 0 inf 0
kva-32768 32768 3 0 2 1 0 1 1 0 inf 0
kva-36864 36864 77 0 0 11 0 11 11 0 inf 0
kva-65536 65536 2 0 0 1 0 1 1 0 inf 0
kva-8192 8192 27 0 7 1 0 1 1 0 inf 0
llentrypl 280 1 0 0 1 0 1 1 0 inf 0
lwppl 1024 182 0 15 45 0 45 45 0 inf 0
mbpl 520 1032 0 579 73 6 67 68 0 inf 2
mclpl 2048 1043 0 592 242 12 230 234 0 261831 4
msdosnopl 208 3 0 2 3 2 1 1 0 inf 0
namecache 192 885618 0 75876 38560 0 38560 38560 0 inf 0
pcache 2688 92 0 4 88 0 88 88 0 inf 0
pcachecpu 128 651 0 0 21 0 21 21 0 inf 0
pcglarge 1088 740 0 629 56 19 37 46 0 inf 0
pcgnormal 320 200584 0 197957 3795 1976 1819 3020 0 inf 1193
pdict16 80 262 0 216 1 0 1 1 0 inf 0
pdict32 96 11 0 1 1 0 1 1 0 inf 0
pdppl 4096 122 0 87 115 80 35 114 0 inf 0
phpool-128 72 26 0 0 1 0 1 1 0 inf 0
phpool-256 88 8 0 0 1 0 1 1 0 inf 0
phpool-64 64 313108 0 175253 4007 577 3430 4007 0 inf 0
piperd 256 19 0 13 2 1 1 2 0 inf 0
pipewr 256 52 0 45 4 1 3 4 0 inf 0
pmappl 512 122 0 87 15 2 13 15 0 inf 0
procpl 896 50 0 16 12 1 11 12 0 inf 0
proparay 56 117 0 0 2 0 2 2 0 inf 0
propdict 56 447 0 107 5 0 5 5 0 inf 0
propnmbr 64 47 0 3 1 0 1 1 0 inf 0
propstng 72 742 0 558 4 0 4 4 0 inf 0
pvpage 4096 831 0 40 819 28 791 813 0 inf 0
rtentpl 320 28 0 2 3 0 3 3 0 inf 0
scxspl 264 30 0 30 1 0 1 1 1 inf 1
sigacts 3088 113 0 79 111 77 34 110 0 inf 0
socket 592 141 0 21 23 1 22 23 0 inf 0
synpl 320 1 0 1 1 1 0 1 0 inf 0
tcpcbpl 832 65 0 62 1 0 1 1 0 inf 0
thplthrd 72 3 0 2 1 0 1 1 0 inf 0
tmpfs_dirent 56 5 0 1 1 0 1 1 0 inf 0
tmpfs_node 232 7 0 1 1 0 1 1 0 inf 0
uarea 24576 49 0 15 47 13 34 46 0 inf 0
uareasys 24576 141 0 8 141 8 133 141 0 inf 0
ufsdir 272 32 0 24 1 0 1 1 0 inf 0
uhcixfer 376 3 0 2 1 0 1 1 0 inf 0
uhcixfer 376 3 0 2 1 0 1 1 0 inf 0
uhcixfer 376 3 0 2 1 0 1 1 0 inf 0
vcachepl 640 923969 0 86108 139653 0 139653 139653 0 inf 9
vmembt 64 16546 0 4964 237 0 237 237 0 inf 0
vmmpepl 192 27152 0 16682 523 24 499 499 0 inf 0
wapbldealloc 40 8 0 8 1 1 0 1 0 inf 0
wapblentrypl 48 84 0 84 2 1 1 1 0 inf 1
wapblinopl 40 176 0 175 1 0 1 1 0 inf 0
Totals 7506830 0 1718506 767003 233357 533646
In use 2459547K, total allocated 2576816K; utilization 95.4%
>How-To-Repeat:
Have disk with lot of data (46GB in my case). Clone one disk to another using dump/restore with snapshot:
newfs -O 2 /dev/dk8
mkdir /mnt/target
mount -o async /dev/dk8 /mnt/target
df -h / /mnt/target
Filesystem Size Used Avail %Cap Mounted on
/dev/dk1 58G 46G 9.5G 82% /
/dev/dk8 197G 4.0K 187G 0% /mnt/target
# commands below will trigger panic after a while (around 20GB copied):
cd /mnt/target
dump -0uaX -f - / | restore rf -
Note: my NetBSD VM has lot of RAM (16GB) and cores (8) - I guess that lowering RAM will trigger panic sooner - need to test it.
Note: the system is otherwise idle while doing dump/restore (so snapshot device should not fill).
>Fix:
N/A
Even using SCSI emulation for target disk will cause similar panic - this time not uvm_fault
but rather "pool_get: [buf16k] pool page empty" - but again FFS snapshot is involved in stacktrace:
[ 2161.615264] panic: kernel diagnostic assertion "(ph->ph_nmissing < pp->pr_itemsperpage)" failed: file "/usr/src/sys/kern/subr_pool.c", l[0/1368]
pool_get: [buf16k] pool page empty
[ 2161.625407] cpu1: Begin traceback...
[ 2161.625407] vpanic() at netbsd:vpanic+0x171
[ 2161.625407] kern_assert() at netbsd:kern_assert+0x4b
[ 2161.625407] pool_get() at netbsd:pool_get+0x7c9
[ 2161.625407] allocbuf() at netbsd:allocbuf+0x113
[ 2161.625407] getblk() at netbsd:getblk+0x18c
[ 2161.625407] bio_doread() at netbsd:bio_doread+0x1d
[ 2161.625407] breadn() at netbsd:breadn+0x8b
[ 2161.625407] ffs_snapshot_read() at netbsd:ffs_snapshot_read+0x1b2
[ 2161.625407] VOP_READ() at netbsd:VOP_READ+0x42
[ 2161.625407] vn_rdwr() at netbsd:vn_rdwr+0xf1
[ 2161.625407] fss_bs_io() at netbsd:fss_bs_io+0x89
[ 2161.625407] fss_bs_thread() at netbsd:fss_bs_thread+0x50f
[ 2161.625407] cpu1: End traceback...
>Release-Note:
>Audit-Trail:
From: "Henryk Paluch" <hpaluch@seznam.cz>
To: <gnats-bugs@NetBSD.org>
Cc:
Subject: Subject: Re: kern/59663
Date: Tue, 23 Sep 2025 19:50:35 +0200 (CEST)
I found reliable way how to crash above "NetBSD 11.0_BETA (GENERIC) #0: Tu=
e Sep 16 04:05:29 UTC 2025" in *few seconds*.
Just these 2 commands are enough (reproducible on both VMD and bare metal)=
:
nbsd11-test3# fssconfig -cx fss0 / /root/backup
nbsd11-test3# dd if=3D/dev/fss0 of=3D/dev/null bs=3D1024k
After few seconds:
[ 58.8640367] panic: kernel diagnostic assertion "(i * BITMAP_SIZE) < pp-=
>pr_itemsperpage" failed: file "/usr/src/sys/kern/subr_pool.c", line 450 =
[ 58.8640367] cpu0: Begin traceback...
[ 58.8741603] vpanic() at netbsd:vpanic+0x171
[ 58.8741603] kern_assert() at netbsd:kern_assert+0x4b
[ 58.8741603] pool_get() at netbsd:pool_get+0x46b
[ 58.8741603] allocbuf() at netbsd:allocbuf+0x113
[ 58.8741603] getblk() at netbsd:getblk+0x18c
[ 58.8741603] bio_doread() at netbsd:bio_doread+0x1d
[ 58.8842815] breadn() at netbsd:breadn+0x24
[ 58.8842815] ffs_snapshot_read() at netbsd:ffs_snapshot_read+0x1b2
[ 58.8842815] VOP_READ() at netbsd:VOP_READ+0x42
[ 58.8842815] vn_rdwr() at netbsd:vn_rdwr+0xf1
[ 58.8842815] fss_bs_io() at netbsd:fss_bs_io+0x89
[ 58.8842815] fss_bs_thread() at netbsd:fss_bs_thread+0x50f
[ 58.8842815] cpu0: End traceback...
[ 58.8842815] fatal breakpoint trap in supervisor mode
[ 58.8842815] trap type 1 code 0 rip 0xffffffff8023541d cs 0x8 rflags 0x2=
02 cr2 0xffff8f026ade5000 ilevel 0 rsp 0xffff8f0277ef7b70
[ 58.8955114] curlwp 0xffff8889b54f7800 pid 0.1426 lowest kstack 0xffff8f=
0277ef32c0
Stopped in pid 0.1426 (system) at netbsd:breakpoint+0x5: leave
breakpoint() at netbsd:breakpoint+0x5
vpanic() at netbsd:vpanic+0x171
kern_assert() at netbsd:kern_assert+0x4b
pool_get() at netbsd:pool_get+0x46b
allocbuf() at netbsd:allocbuf+0x113
getblk() at netbsd:getblk+0x18c
bio_doread() at netbsd:bio_doread+0x1d
breadn() at netbsd:breadn+0x24
ffs_snapshot_read() at netbsd:ffs_snapshot_read+0x1b2
VOP_READ() at netbsd:VOP_READ+0x42
vn_rdwr() at netbsd:vn_rdwr+0xf1
fss_bs_io() at netbsd:fss_bs_io+0x89
fss_bs_thread() at netbsd:fss_bs_thread+0x50f
ds 0
es 0
fs 180
gs 7b20
rdi 0
rsi 3f8
rbp ffff8f0277ef7b70
rbx ffffffff8142a670 ostype+0x7867e
rdx 1
--db_more--rcx ffffffffffffff
rax 800000000000000
r8 0
r9 0
r10 0
r11 0
r12 ffff8f0277ef7bb8
r13 104
r14 1
r15 ffff8f02670b0000
rip ffffffff8023541d breakpoint+0x5
cs 8
rflags 202
rsp ffff8f0277ef7b70
ss 10
netbsd:breakpoint+0x5: leave
db{0}> bt
breakpoint() at netbsd:breakpoint+0x5
vpanic() at netbsd:vpanic+0x171
kern_assert() at netbsd:kern_assert+0x4b
pool_get() at netbsd:pool_get+0x46b
allocbuf() at netbsd:allocbuf+0x113
getblk() at netbsd:getblk+0x18c
bio_doread() at netbsd:bio_doread+0x1d
breadn() at netbsd:breadn+0x24
ffs_snapshot_read() at netbsd:ffs_snapshot_read+0x1b2
VOP_READ() at netbsd:VOP_READ+0x42
vn_rdwr() at netbsd:vn_rdwr+0xf1
fss_bs_io() at netbsd:fss_bs_io+0x89
fss_bs_thread() at netbsd:fss_bs_thread+0x50f
From: "Henryk Paluch" <hpaluch@seznam.cz>
To: <gnats-bugs@NetBSD.org>
Cc:
Subject: Re: kern/59663
Date: Tue, 23 Sep 2025 20:04:20 +0200 (CEST)
I found reliable way how to crash above "NetBSD 11.0_BETA (GENERIC) #0: Tu=
e Sep 16 04:05:29 UTC 2025" in *few seconds*.
Just these 2 commands are enough (reproducible on both VM and bare metal):=
nbsd11-test3# fssconfig -cx fss0 / /root/backup
nbsd11-test3# dd if=3D/dev/fss0 of=3D/dev/null bs=3D1024k
After few seconds:
[ 58.8640367] panic: kernel diagnostic assertion "(i * BITMAP_SIZE) < pp->=
pr_itemsperpage" failed: file "/usr/src/sys/kern/subr_pool.c", line 450
[ 58.8640367] cpu0: Begin traceback...
[ 58.8741603] vpanic() at netbsd:vpanic+0x171
[ 58.8741603] kern_assert() at netbsd:kern_assert+0x4b
[ 58.8741603] pool_get() at netbsd:pool_get+0x46b
[ 58.8741603] allocbuf() at netbsd:allocbuf+0x113
[ 58.8741603] getblk() at netbsd:getblk+0x18c
[ 58.8741603] bio_doread() at netbsd:bio_doread+0x1d
[ 58.8842815] breadn() at netbsd:breadn+0x24
[ 58.8842815] ffs_snapshot_read() at netbsd:ffs_snapshot_read+0x1b2
[ 58.8842815] VOP_READ() at netbsd:VOP_READ+0x42
[ 58.8842815] vn_rdwr() at netbsd:vn_rdwr+0xf1
[ 58.8842815] fss_bs_io() at netbsd:fss_bs_io+0x89
[ 58.8842815] fss_bs_thread() at netbsd:fss_bs_thread+0x50f
[ 58.8842815] cpu0: End traceback...
[ 58.8842815] fatal breakpoint trap in supervisor mode
[ 58.8842815] trap type 1 code 0 rip 0xffffffff8023541d cs 0x8 rflags 0x20=
2 cr2 0xffff8f026ade5000 ilevel 0 rsp 0xffff8f0277ef7b70
[ 58.8955114] curlwp 0xffff8889b54f7800 pid 0.1426 lowest kstack 0xffff8f0=
277ef32c0
Stopped in pid 0.1426 (system) at netbsd:breakpoint+0x5: leave
breakpoint() at netbsd:breakpoint+0x5
vpanic() at netbsd:vpanic+0x171
kern_assert() at netbsd:kern_assert+0x4b
pool_get() at netbsd:pool_get+0x46b
allocbuf() at netbsd:allocbuf+0x113
getblk() at netbsd:getblk+0x18c
bio_doread() at netbsd:bio_doread+0x1d
breadn() at netbsd:breadn+0x24
ffs_snapshot_read() at netbsd:ffs_snapshot_read+0x1b2
VOP_READ() at netbsd:VOP_READ+0x42
vn_rdwr() at netbsd:vn_rdwr+0xf1
fss_bs_io() at netbsd:fss_bs_io+0x89
fss_bs_thread() at netbsd:fss_bs_thread+0x50f
ds 0
es 0
fs 180
gs 7b20
rdi 0
rsi 3f8
rbp ffff8f0277ef7b70
rbx ffffffff8142a670 ostype+0x7867e
rdx 1
--db_more--rcx ffffffffffffff
rax 800000000000000
r8 0
r9 0
r10 0
r11 0
r12 ffff8f0277ef7bb8
r13 104
r14 1
r15 ffff8f02670b0000
rip ffffffff8023541d breakpoint+0x5
cs 8
rflags 202
rsp ffff8f0277ef7b70
ss 10
netbsd:breakpoint+0x5: leave
db{0}> bt
breakpoint() at netbsd:breakpoint+0x5
vpanic() at netbsd:vpanic+0x171
kern_assert() at netbsd:kern_assert+0x4b
pool_get() at netbsd:pool_get+0x46b
allocbuf() at netbsd:allocbuf+0x113
getblk() at netbsd:getblk+0x18c
bio_doread() at netbsd:bio_doread+0x1d
breadn() at netbsd:breadn+0x24
ffs_snapshot_read() at netbsd:ffs_snapshot_read+0x1b2
VOP_READ() at netbsd:VOP_READ+0x42
vn_rdwr() at netbsd:vn_rdwr+0xf1
fss_bs_io() at netbsd:fss_bs_io+0x89
fss_bs_thread() at netbsd:fss_bs_thread+0x50f
From: Henryk Paluch <hpaluch@seznam.cz>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/59663: ffs_snapshot_read -> uvm_fault (or pool page empty)
doing dump/restore with snapshot
Date: Thu, 25 Sep 2025 16:40:31 +0200
Building my own kernel with netbsd.gdb I was able to get full backtrace
with source lines. Just enabled:
options DEBUG # expensive debugging checks/support
options LOCKDEBUG # expensive locking checks/support
Source code is:
https://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-11/20250916040529Z/source/sets/syssrc.tgz
MD5: MD5 (syssrc.tgz) = 35f104087fa2eab6ef7804f411c529cf
uname -a: NetBSD nbsd-crash3.example.com 11.0_BETA NetBSD 11.0_BETA (HP)
#0: Wed Sep 24 20:47:28 CEST 2025
root@nbsd-crash3.example.com:/usr/src/sys/arch/amd64/compile/HP amd64
Commands quickly leading to panic (few seconds):
fssconfig -cx fss0 / /root/backup
dd if=/dev/fss0 of=/dev/null bs=1024k
[ 34.9028584] panic: kernel diagnostic assertion "(i * BITMAP_SIZE) <
pp->pr_itemsperpage" failed: file "../../../../kern/subr_pool.c", line 450
[ 34.9129630] cpu1: Begin traceback...
[ 34.9129630] vpanic() at netbsd:vpanic+0x171
[ 34.9129630] kern_assert() at netbsd:kern_assert+0x4b
[ 34.9129630] pool_get() at netbsd:pool_get+0x496
[ 34.9129630] allocbuf() at netbsd:allocbuf+0x113
[ 34.9129630] getblk() at netbsd:getblk+0x18c
[ 34.9129630] bio_doread() at netbsd:bio_doread+0x1d
[ 34.9129630] breadn() at netbsd:breadn+0x24
[ 34.9230632] ffs_snapshot_read() at netbsd:ffs_snapshot_read+0x1b2
[ 34.9230632] VOP_READ() at netbsd:VOP_READ+0x42
[ 34.9230632] vn_rdwr() at netbsd:vn_rdwr+0xf1
[ 34.9230632] fss_bs_io() at netbsd:fss_bs_io+0x89
[ 34.9230632] fss_bs_thread() at netbsd:fss_bs_thread+0x50f
[ 34.9230632] cpu1: End traceback...
[ 34.9230632] fatal breakpoint trap in supervisor mode
[ 34.9230632] trap type 1 code 0 rip 0xffffffff8023541d cs 0x8 rflags
0x202 cr2 0x764ef540bce0 ilevel 0 rsp 0xffffcf0154c99b70
[ 34.9331650] curlwp 0xffff8591f760f000 pid 0.1320 lowest kstack
0xffffcf0154c952c0
Stacktrace from gdb:
Reading symbols from /usr/src/sys/arch/amd64/compile/HP/netbsd.gdb...
+target kvm netbsd.0.core
0xffffffff80239b95 in cpu_reboot (howto=howto@entry=256,
bootstr=bootstr@entry=0x0) at ../../../../arch/amd64/amd64/machdep.c:709
709 dumpsys();
+bt
#0 0xffffffff80239b95 in cpu_reboot (howto=howto@entry=256,
bootstr=bootstr@entry=0x0) at ../../../../arch/amd64/amd64/machdep.c:709
#1 0xffffffff80df1efe in kern_reboot (howto=howto@entry=256,
bootstr=bootstr@entry=0x0) at ../../../../kern/kern_reboot.c:91
#2 0xffffffff80b6c1d4 in db_sync_cmd (addr=<optimized out>,
have_addr=<optimized out>, count=<optimized out>, modif=<optimized out>)
at ../../../../ddb/db_command.c:1651
#3 0xffffffff80b6c9c4 in db_command
(last_cmdp=last_cmdp@entry=0xffffffff81a74220 <db_last_command>) at
../../../../ddb/db_command.c:970
#4 0xffffffff80b6ce53 in db_command_loop () at
../../../../ddb/db_command.c:629
#5 0xffffffff80b7119d in db_trap (type=type@entry=1, code=code@entry=0)
at ../../../../ddb/db_trap.c:91
#6 0xffffffff80236a1b in kdb_trap (type=type@entry=1,
code=code@entry=0, regs=regs@entry=0xffffcf0154c99a80) at
../../../../arch/amd64/amd64/db_interface.c:251
#7 0xffffffff8023bf96 in trap (frame=0xffffcf0154c99a80) at
../../../../arch/amd64/amd64/trap.c:314
#8 0xffffffff80234ad4 in alltraps ()
#9 0xffffffff8023541d in breakpoint ()
#10 0xffffffff80e3a289 in vpanic (fmt=0xffffffff8162d8f8 "kernel
%sassertion \"%s\" failed: file \"%s\", line %d ",
ap=ap@entry=0xffffcf0154c99bb8) at ../../../../kern/subr_prf.c:286
#11 0xffffffff8100c76e in kern_assert (fmt=fmt@entry=0xffffffff8162d8f8
"kernel %sassertion \"%s\" failed: file \"%s\", line %d ") at
../../../../../../lib/libkern/kern_assert.c:51
#12 0xffffffff80e34f33 in pr_item_bitmap_get (ph=0xffffcf0144320000,
pp=0xffffffff81b9c790 <bmempools+1840>) at ../../../../kern/subr_pool.c:450
#13 pool_get (pp=pp@entry=0xffffffff81b9c790 <bmempools+1840>,
flags=flags@entry=2) at ../../../../kern/subr_pool.c:1217
#14 0xffffffff80e8fca1 in buf_alloc (size=<optimized out>) at
../../../../kern/vfs_bio.c:652
#15 allocbuf (bp=bp@entry=0xffff8591dc3211b0, size=size@entry=16384,
preserve=preserve@entry=0) at ../../../../kern/vfs_bio.c:1337
#16 0xffffffff80e90cb3 in getblk (vp=vp@entry=0xffff8591f190d840,
blkno=16516, size=size@entry=16384, slpflag=slpflag@entry=0,
slptimeo=slptimeo@entry=0) at ../../../../kern/vfs_bio.c:1262
#17 0xffffffff80e90f05 in bio_doread (vp=vp@entry=0xffff8591f190d840,
blkno=<optimized out>, size=size@entry=16384, async=async@entry=0) at
../../../../kern/vfs_bio.c:692
#18 0xffffffff80e9115e in breadn (vp=vp@entry=0xffff8591f190d840,
blkno=<optimized out>, size=size@entry=16384,
rablks=rablks@entry=0xffffcf0154c99dd8,
rasizes=rasizes@entry=0xffffcf0154c99dcc, nrablks=nrablks@entry=1,
flags=flags@entry=0, bpp=bpp@entry=0xffffcf0154c99dd0) at
../../../../kern/vfs_bio.c:783
#19 0xffffffff80d2bef4 in ffs_snapshot_read (vp=0xffff8591f190d840,
uio=0xffffcf0154c99e90, ioflag=<optimized out>) at
../../../../ufs/ffs/ffs_snapshot.c:2104
#20 0xffffffff80ebe5b2 in VOP_READ (vp=0xffff8591f190d840,
uio=<optimized out>, ioflag=<optimized out>, cred=<optimized out>) at
../../../../kern/vnode_if.c:785
#21 0xffffffff80eb5737 in vn_rdwr (rw=rw@entry=UIO_READ,
vp=0xffff8591f190d840, base=base@entry=0xffff8591dba51000,
len=len@entry=2048, offset=offset@entry=270600192,
segflg=segflg@entry=UIO_SYSSPACE, ioflg=ioflg@entry=129,
cred=0xffff8591f8725040, aresid=0xffffcf0154c99fb8, l=0x0) at
../../../../kern/vfs_vnops.c:558
#22 0xffffffff80edd2c6 in fss_bs_io (sc=sc@entry=0xffff8591f7380ac0,
rw=rw@entry=FSS_READ, cl=cl@entry=0, off=270600192, len=2048,
data=0xffff8591dba51000, resid=resid@entry=0xffffcf0154c99fb8) at
../../../../dev/fss.c:1100
#23 0xffffffff80eddb06 in fss_bs_thread (arg=0xffff8591f7380ac0) at
../../../../dev/fss.c:1198
#24 0xffffffff80210327 in lwp_trampoline ()
#25 0x0000000000000000 in ?? ()
+q
Please note that quite often I get various uvm_traps when accessing pool
which often prevents me to dump core at all.
Responsible-Changed-From-To: kern-bug-people->hannken
Responsible-Changed-By: riastradh@NetBSD.org
Responsible-Changed-When: Sat, 18 Jul 2026 19:58:48 +0000
Responsible-Changed-Why:
Can you take a look at this?
From: Henryk Paluch <hpaluch@seznam.cz>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/59663 (ffs_snapshot_read -> uvm_fault (or pool page empty)
doing dump/restore with snapshot)
Date: Mon, 20 Jul 2026 22:24:51 +0200
Quick re-test with early results:
* 11.0_RC1 - last known to crash
* 11.0_RC2 to RC4 - unable to find ISO to download and test
* 11.0_RC5 - does not crash
* 11.0_RC6 - does not crash
* HEAD - 20260720014948Z - does not crash
Tested commands (should crash in few seconds if vulnerable):
fssconfig -cx fss0 / /root/backup
dd if=/dev/fss0 of=/dev/null bs=1024k
From: Henryk Paluch <hpaluch@seznam.cz>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/59663 (ffs_snapshot_read -> uvm_fault (or pool page empty)
doing dump/restore with snapshot)
Date: Wed, 22 Jul 2026 21:03:15 +0200
I performed manual kernel bisection (building kernel from CVS for each
tag and booting on target machine where fssconfig & dd tests were run
Example build for RC3 (on machine called nb11rc6-fss-src - kernel builder)
cd /usr/src/
cvs -q -z2 -d anoncvs@anoncvs.NetBSD.org:/cvsroot update -r
netbsd-11-0-RC3 -dP
rm -rf /usr/src/sys/arch/amd64/compile/HP
cd /usr/src/sys/arch/amd64/conf
config HP # see notes below
cd ../compile/HP
make depend && make -j 1C
gzip -9v netbsd.gdb netbsd netbsd-HP.debug netbsd.map
scp *.gz tgt:kernel-11.0_RC3 # copy to TARGET machine where crashes are
tested, machine called nb11rc6-fss-tgt
My config (called HP) has these changes from GENERIC (enabled all debug
options):
options DIAGNOSTIC # inexpensive kernel consistency checks
options DEBUG # expensive debugging checks/support
options LOCKDEBUG # expensive locking checks/support
Both build machine and target machine is official
NetBSD-11.0_RC6-amd64-dvd.iso, testing Target just boots different RC
kernels. Both machines are VMs under Linux KVM. Target is using Host
snapshot to ensure that after crash it is rolled back to consistent state.
Results are following (and confusing):
* 11.0_RC1 quick crash (matches this report)
* 11.0_RC2 skipped (not part of bisection)
* 11.0_RC3 quick crash (matches this report)
* 11.0_RC4 quick crash (matches this report) - LAST ONE
here happened something in sources that significantly reduced pool
corruption panics
* 11.0_RC5 nearly stable - pool panics are very rare (but still exist)
* 11.0_RC6 no panic occurred so far, but kernel freezes after few
minutes of dd (only local console shows keystrokes, but everything else
is frozen) - is seems that logout from other console triggers it(?)
Below is "systat vmstat" session when it last updated on RC6:
3 users Load 0.97 0.39 0.15 Wed Jul 22 18:34:59
Proc:r d s Csw Traps SysCal Intr Soft Fault PAGING
SWAPPING
1 1 55568 3113 47 291 in out
in out
ops
67.8% Sy 0.0% Us 0.0% Ni 0.0% In 32.2% Id pages
| | | | | | | | | | |
==================================
forks
fkppw
Anon 22024 1% zero 147 Interrupts
fksvm
Exec 10772 % wired TLB shootdown
pwait
File 31468 1% inact 100 cpu0 timer
relck
Meta 297308 -1% bufs 148439 ioapic0 pin 4
rlkok
(kB) real swaponly free ioapic0 pin 1
noram
Active 64264 1535108 46 msix0 vec 1
ndcpy
Namei Sys-cache Proc-cache msix1 vec 0
fltcp
Calls hits % hits % ioapic0 pin 22
zfod
6 6 100 1 msix2 vec 1
cow
512 fmin
Disks: ld0 cd0 fss0
682 ftarg
seeks
itarg
xfers 2 36002
flnan
bytes 34K 72004K
pdfre
%busy 50.0
pdscn
Note: that Meta percent (-1%) is quite suspicious...
I'm now a bit confused at what direction should I go to get closer to
cause of these problems. Any clues?
From: Henryk Paluch <hpaluch@seznam.cz>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/59663 (ffs_snapshot_read -> uvm_fault (or pool page empty)
doing dump/restore with snapshot)
Date: Thu, 23 Jul 2026 22:03:41 +0200
I'm now able to reliably freeze 11.0_RC6 with this simple setup.
Running on serial console as root:
fssconfig -cx fss0 / /root/backup
dd if=/dev/fss0 of=/dev/null bs=1024k
At same time running on local (graphical) wscons console:
- login as root
- logout - typically logout freezes whole system
After logout everything is frozen and idle. I was able
to send break on serial console and dump all locks:
db{0}> show all locks/t
[Locks tracked through LWPs]
****** LWP 1089.1089 (dd) @ 0xffff80f8b86db400, l_stat=3
*** Locks held:
* Lock 0 (initialized at netbsd:vcache_alloc+0x6b)
lock address : ffff80f8b4fe1480
type : sleep/adaptive
initialized : netbsd:vcache_alloc+0x6b
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
relevant cpu : 0 last held: 0
relevant lwp : 0xffff80f8b86db400 last held: 0xffff80f8b86db400
last locked* : netbsd:genfs_lock+0x89
unlocked : netbsd:genfs_unlock+0x14
owner/count : 0xffff80f8b86db400 flags : 0x0000000000000004
Turnstile: no active turnstile for this lock.
*** Locks wanted: none
*** Traceback:
trace: pid 1089 lid 1089 at 0xffffa980c2c1fc20
sleepq_block() at netbsd:sleepq_block+0xee
cv_timedwait() at netbsd:cv_timedwait+0xd4
bbusy() at netbsd:bbusy+0xe3
getblk() at netbsd:getblk+0x68
bio_doread() at netbsd:bio_doread+0x1d
bread() at netbsd:bread+0x18
spec_read() at netbsd:spec_read+0x1e6
VOP_READ() at netbsd:VOP_READ+0x42
vn_read() at netbsd:vn_read+0x136
dofileread() at netbsd:dofileread+0x79
sys_read() at netbsd:sys_read+0x49
syscall() at netbsd:syscall+0x9d
--- syscall (number 3) ---
netbsd:syscall+0x9d:
****** LWP 850.850 (login) @ 0xffff80f8b822f000, l_stat=3
*** Locks held:
* Lock 0 (initialized at netbsd:fork1+0x1e5)
lock address : ffff80f8b84e2710
type : sleep/adaptive
initialized : netbsd:fork1+0x1e5
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
relevant cpu : 0 last held: 0
relevant lwp : 0xffff80f8b822f000 last held: 0xffff80f8b822f000
last locked* : netbsd:exit1+0x11c
unlocked : netbsd:execve_runproc+0x95d
owner/count : 0xffff80f8b822f000 flags : 0x0000000000000004
Turnstile: no active turnstile for this lock.
* Lock 1 (initialized at netbsd:fstrans_init+0x1b)
lock address : netbsd:vfs_suspend_lock
type : sleep/adaptive
initialized : netbsd:fstrans_init+0x1b
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
relevant cpu : 0 last held: 0
relevant lwp : 0xffff80f8b822f000 last held: 0xffff80f8b822f000
last locked* : netbsd:vfs_suspend+0xd2
unlocked : netbsd:ffs_snapshot+0x67c
owner field : 0xffff80f8b822f000 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
*** Locks wanted: none
*** Traceback:
trace: pid 850 lid 850 at 0xffffa980c2c26cd0
sleepq_block() at netbsd:sleepq_block+0xee
cv_wait_sig() at netbsd:cv_wait_sig+0xd2
fstrans_setstate() at netbsd:fstrans_setstate+0x10c
genfs_suspendctl() at netbsd:genfs_suspendctl+0x32
VFS_SUSPENDCTL() at netbsd:VFS_SUSPENDCTL+0x25
vfs_suspend() at netbsd:vfs_suspend+0x8d
vrevoke_suspend_next() at netbsd:vrevoke_suspend_next+0x2a
vrevoke() at netbsd:vrevoke+0x44
genfs_revoke() at netbsd:genfs_revoke+0x13
VOP_REVOKE() at netbsd:VOP_REVOKE+0x3b
exit1() at netbsd:exit1+0x7bd
sys_exit() at netbsd:sys_exit+0x3b
syscall() at netbsd:syscall+0x9d
--- syscall (number 1) ---
netbsd:syscall+0x9d:
****** LWP 309.309 (dhcpcd) @ 0xffff80f8b822f800, l_stat=3
*** Locks held: none
*** Locks wanted:
* Lock 0 (initialized at netbsd:module_hook_init+0x17)
lock address : netbsd:module_hook
type : sleep/adaptive
initialized : netbsd:module_hook_init+0x17
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 0
relevant cpu : 1 last held: 0
relevant lwp : 0xffff80f8b822f800 last held: 000000000000000000
last locked : 0
unlocked* : 0
owner field : 000000000000000000 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
*** Traceback:
trace: pid 309 lid 309 at 0xffffa980c3a24d10
sleepq_block() at netbsd:sleepq_block+0xee
sel_do_scan() at netbsd:sel_do_scan+0x5a2
pollcommon() at netbsd:pollcommon+0xbb
sys___pollts50() at netbsd:sys___pollts50+0x68
syscall() at netbsd:syscall+0x9d
--- syscall (number 437) ---
netbsd:syscall+0x9d:
****** LWP 308.308 (dhcpcd) @ 0xffff80f8b86bfc00, l_stat=3
*** Locks held: none
*** Locks wanted:
* Lock 0 (initialized at netbsd:module_hook_init+0x17)
lock address : netbsd:module_hook
type : sleep/adaptive
initialized : netbsd:module_hook_init+0x17
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 0
relevant cpu : 0 last held: 0
relevant lwp : 0xffff80f8b86bfc00 last held: 000000000000000000
last locked : 0
unlocked* : 0
owner field : 000000000000000000 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
*** Traceback:
trace: pid 308 lid 308 at 0xffffa980c2c64d10
sleepq_block() at netbsd:sleepq_block+0x230
sel_do_scan() at netbsd:sel_do_scan+0x5a2
pollcommon() at netbsd:pollcommon+0xbb
sys___pollts50() at netbsd:sys___pollts50+0x68
syscall() at netbsd:syscall+0x9d
--- syscall (number 437) ---
netbsd:syscall+0x9d:
****** LWP 0.26 (softclk/1) @ 0xffff80f87d1ba000, l_stat=1
*** Locks held: none
*** Locks wanted:
* Lock 0 (initialized at netbsd:module_hook_init+0x17)
lock address : netbsd:module_hook
type : sleep/adaptive
initialized : netbsd:module_hook_init+0x17
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 0
relevant cpu : 1 last held: 0
relevant lwp : 0xffff80f87d1ba000 last held: 000000000000000000
last locked : 0
unlocked* : 0
owner field : 000000000000000000 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
*** Traceback:
trace: pid 0 lid 26 at 0xffff80f87d2bf840
cpu_intr_p() at netbsd:cpu_intr_p+0x36
Bad user frame pointer: 0xffff80f87d2bf800
****** LWP 0.11 (iflnkst) @ 0xffff80f8b88f0400, l_stat=3
*** Locks held: none
*** Locks wanted:
* Lock 0 (initialized at netbsd:module_hook_init+0x17)
lock address : netbsd:module_hook
type : sleep/adaptive
initialized : netbsd:module_hook_init+0x17
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 0
relevant cpu : 1 last held: 0
relevant lwp : 0xffff80f8b88f0400 last held: 000000000000000000
last locked : 0
unlocked* : 0
owner field : 000000000000000000 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
*** Traceback:
trace: pid 0 lid 11 at 0xffffa980b2604f60
sleepq_block() at netbsd:sleepq_block+0xee
cv_wait() at netbsd:cv_wait+0xca
workqueue_worker() at netbsd:workqueue_worker+0x97
****** LWP 0.0 (swapper) @ 0xffffffff81a8be80, l_stat=3
*** Locks held: none
*** Locks wanted:
* Lock 0 (initialized at netbsd:module_hook_init+0x17)
lock address : netbsd:module_hook
type : sleep/adaptive
initialized : netbsd:module_hook_init+0x17
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 0
relevant cpu : 0 last held: 0
relevant lwp : 0xffffffff81a8be80 last held: 000000000000000000
last locked : 0
unlocked* : 0
owner field : 000000000000000000 wait/spin: 0/0
Turnstile: no active turnstile for this lock.
*** Traceback:
trace: pid 0 lid 0 at 0xffffffff81f5de90
sleepq_block() at netbsd:sleepq_block+0x230
uvm_scheduler() at netbsd:uvm_scheduler+0x6f
main() at netbsd:main+0x782
[Locks tracked through CPUs]
******* Locks held on cpu0:
* Lock 0 (initialized at netbsd:com_attach_subr+0xba)
lock address : ffff80f87d3323a0
type : spin
initialized : netbsd:com_attach_subr+0xba
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
relevant cpu : 0 last held: 0
relevant lwp : 0xffff80f8b8c5d000 last held: 0xffff80f8b8c5d000
last locked* : netbsd:comintr+0x4b
unlocked : netbsd:comsoft+0x1b9
owner field : 0x0000000000010800 wait/spin: 0/1
trace: pid 0 lid 2 at 0xffffa980b2178db8
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x840
intr_wrapper() at netbsd:intr_wrapper+0x4b
Xhandle_ioapic_edge1() at netbsd:Xhandle_ioapic_edge1+0x75
--- interrupt ---
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0x19a
idle_loop() at netbsd:idle_loop+0x13d
To me most suspicious is that:
1. dd is sleeping with:
trace: pid 1089 lid 1089 at 0xffffa980c2c1fc20
sleepq_block() at netbsd:sleepq_block+0xee
cv_timedwait() at netbsd:cv_timedwait+0xd4
bbusy() at netbsd:bbusy+0xe3
getblk() at netbsd:getblk+0x68
bio_doread() at netbsd:bio_doread+0x1d
bread() at netbsd:bread+0x18
spec_read() at netbsd:spec_read+0x1e6
VOP_READ() at netbsd:VOP_READ+0x42
vn_read() at netbsd:vn_read+0x136
dofileread() at netbsd:dofileread+0x79
sys_read() at netbsd:sys_read+0x49
syscall() at netbsd:syscall+0x9d
--- syscall (number 3) ---
netbsd:syscall+0x9d:
2. and logout (sys_exit) is also sleeping:
trace: pid 850 lid 850 at 0xffffa980c2c26cd0
sleepq_block() at netbsd:sleepq_block+0xee
cv_wait_sig() at netbsd:cv_wait_sig+0xd2
fstrans_setstate() at netbsd:fstrans_setstate+0x10c
genfs_suspendctl() at netbsd:genfs_suspendctl+0x32
VFS_SUSPENDCTL() at netbsd:VFS_SUSPENDCTL+0x25
vfs_suspend() at netbsd:vfs_suspend+0x8d
vrevoke_suspend_next() at netbsd:vrevoke_suspend_next+0x2a
vrevoke() at netbsd:vrevoke+0x44
genfs_revoke() at netbsd:genfs_revoke+0x13
VOP_REVOKE() at netbsd:VOP_REVOKE+0x3b
exit1() at netbsd:exit1+0x7bd
sys_exit() at netbsd:sys_exit+0x3b
syscall() at netbsd:syscall+0x9d
--- syscall (number 1) ---
netbsd:syscall+0x9d:
In other words basically all processes are sleeping/waiting but there is
nothing that would wake them up...
From: "J. Hannken-Illjes" <hannken@mailbox.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/59663 (ffs_snapshot_read -> uvm_fault (or pool page empty)
doing dump/restore with snapshot)
Date: Sat, 25 Jul 2026 19:43:44 +0200
This PR reports two different problems:
1) Memory corruption around (buffer) pools with combination of
fssconfig and dd. Bisecting -current this problem got fixed
with rev. 1.167 of sys/uvm/uvm_km.c on 2026/06/27 09:39:57.
This revision got pulled up to -10 and -11, so for this
problem the PR should be closed.
2) Deadlock on logout involving vrevoke and vfs_suspend. This
problem looks similar to a problem reported on tech-kern@ on
Sat, 28 Jun 2025, thread starting here:
https://mail-index.netbsd.org/tech-kern/2025/06/29/msg030580.html
I'm not aware of an open PR for this problem so would propose
to open a new one with the information/analysis of this PR.
--
J. Hannken-Illjes - hannken@mailbox.org
From: "Juergen Hannken-Illjes" <hannken@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/59663 CVS commit: src/sys/dev
Date: Wed, 29 Jul 2026 17:39:23 +0000
Module Name: src
Committed By: hannken
Date: Wed Jul 29 17:39:23 UTC 2026
Modified Files:
src/sys/dev: fss.c
Log Message:
Bracket I/O from or to the backing store with fstrans_start_lazy()
so the fss device doesn't block before the file system is suspended.
Should fix some deadlocks with revokation of ttys.
PR kern/59663 (ffs_snapshot_read -> uvm_fault (or pool page empty)
To generate a diff of this commit:
cvs rdiff -u -r1.115 -r1.116 src/sys/dev/fss.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Taylor R Campbell <riastradh@NetBSD.org>
To: "Juergen Hannken-Illjes" <hannken@netbsd.org>
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org,
Henryk Paluch <hpaluch@seznam.cz>
Subject: Re: PR/59663 CVS commit: src/sys/dev
Date: Fri, 31 Jul 2026 15:26:53 +0000
> Module Name: src
> Committed By: hannken
> Date: Wed Jul 29 17:39:23 UTC 2026
>
> Modified Files:
> src/sys/dev: fss.c
>
> Log Message:
> Bracket I/O from or to the backing store with fstrans_start_lazy()
> so the fss device doesn't block before the file system is suspended.
>
> Should fix some deadlocks with revokation of ttys.
>
> PR kern/59663 (ffs_snapshot_read -> uvm_fault (or pool page empty)
Do you hypothesize that this will fix the deadlocks reported in
https://mail-index.netbsd.org/tech-kern/2025/06/29/msg030580.html
and/or this PR?
I'm trying to figure out how it could help, but I don't see any direct
connection to the fss I/O path in the stack traces in question:
- In this PR, dd(1) on /dev/fssN hasn't reached it yet -- it's still
waiting to busy the buffer in specfs:
cv_timedwait() at netbsd:cv_timedwait+0xd4
bbusy() at netbsd:bbusy+0xe3
getblk() at netbsd:getblk+0x68
bio_doread() at netbsd:bio_doread+0x1d
bread() at netbsd:bread+0x18
spec_read() at netbsd:spec_read+0x1e6
VOP_READ() at netbsd:VOP_READ+0x42
vn_read() at netbsd:vn_read+0x136
dofileread() at netbsd:dofileread+0x79
sys_read() at netbsd:sys_read+0x49
syscall() at netbsd:syscall+0x9d
I'm not sure who has the buffer busy; it's probably _not_ the
process that's exiting and revoking its controlling terminal because
I don't see anything in that path that could hold any buffers busy
for any reason:
cv_wait_sig() at netbsd:cv_wait_sig+0xd2
fstrans_setstate() at netbsd:fstrans_setstate+0x10c
genfs_suspendctl() at netbsd:genfs_suspendctl+0x32
VFS_SUSPENDCTL() at netbsd:VFS_SUSPENDCTL+0x25
vfs_suspend() at netbsd:vfs_suspend+0x8d
vrevoke_suspend_next() at netbsd:vrevoke_suspend_next+0x2a
vrevoke() at netbsd:vrevoke+0x44
genfs_revoke() at netbsd:genfs_revoke+0x13
VOP_REVOKE() at netbsd:VOP_REVOKE+0x3b
exit1() at netbsd:exit1+0x7bd
sys_exit() at netbsd:sys_exit+0x3b
Having stack traces of other threads might help to find the cycle.
- In https://mail-index.netbsd.org/tech-kern/2025/06/29/msg030580.html
buhrow never followed up with stack traces, but csh was waiting in
either open, close, or revoke (most likely revoke, I'd guess) on
specfs_iocv for another thread in the middle of spec_io_enter/exit
or spec_node_close. Also not sure how that path could hold any
buffers busy.
I note that the VCHR path for spec_read/write drops the vnode lock and
enters spec_io_enter/exit transaction instead while it works (or fails
if the vnode was revoked between VOP_UNLOCK and spec_io_enter), in
order to avoid blocking indefinitely with the vnode lock held on,
e.g., tty devices:
1123 VOP_UNLOCK(vp);
1124 error = spec_io_enter(vp, &sn, &dev);
1125 if (error)
1126 goto out;
1127 error = cdev_read(dev, uio, ap->a_ioflag);
1128 spec_io_exit(vp, sn);
1129 out: /* XXX What if the caller held an exclusive lock? */
1130 vn_lock(vp, LK_SHARED | LK_RETRY);
1131 return error;
https://nxr.netbsd.org/xref/src/sys/miscfs/specfs/spec_vnops.c?r=1.219#1108
I wonder whether the VBLK case should do the same, instead of blocking
indefinitely on bbusy while a snapshot is taken and the file system is
being synced, involving bwrite?
But that's probably not it, because if we reach spec_read, we should
already be in an fstrans(9) transaction, so any concurrent
snapshot-taking has to wait for us to complete anyway -- including
waiting for bbusy.
Side note: I wonder whether we need a path for revoking a VBLK-type
vnode to interrupt concurrent bbusy and make it fail.
From: "J. Hannken-Illjes" <hannken@mailbox.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: PR/59663 CVS commit: src/sys/dev
Date: Fri, 31 Jul 2026 17:52:31 +0200
> On 31. Jul 2026, at 17:30, Taylor R Campbell via gnats =
<gnats-admin@NetBSD.org> wrote:
>=20
> The following reply was made to PR kern/59663; it has been noted by =
GNATS.
>=20
> From: Taylor R Campbell <riastradh@NetBSD.org>
> To: "Juergen Hannken-Illjes" <hannken@netbsd.org>
> Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org,
> Henryk Paluch <hpaluch@seznam.cz>
> Subject: Re: PR/59663 CVS commit: src/sys/dev
> Date: Fri, 31 Jul 2026 15:26:53 +0000
>=20
>> Module Name: src
>> Committed By: hannken
>> Date: Wed Jul 29 17:39:23 UTC 2026
>>=20
>> Modified Files:
>> src/sys/dev: fss.c
>>=20
>> Log Message:
>> Bracket I/O from or to the backing store with fstrans_start_lazy()
>> so the fss device doesn't block before the file system is suspended.
>>=20
>> Should fix some deadlocks with revokation of ttys.
>>=20
>> PR kern/59663 (ffs_snapshot_read -> uvm_fault (or pool page empty)
>=20
> Do you hypothesize that this will fix the deadlocks reported in
> https://mail-index.netbsd.org/tech-kern/2025/06/29/msg030580.html
> and/or this PR?
>=20
> I'm trying to figure out how it could help, but I don't see any direct
> connection to the fss I/O path in the stack traces in question:
>=20
> - In this PR, dd(1) on /dev/fssN hasn't reached it yet -- it's still
> waiting to busy the buffer in specfs:
>=20
> cv_timedwait() at netbsd:cv_timedwait+0xd4
> bbusy() at netbsd:bbusy+0xe3
> getblk() at netbsd:getblk+0x68
> bio_doread() at netbsd:bio_doread+0x1d
> bread() at netbsd:bread+0x18
> spec_read() at netbsd:spec_read+0x1e6
> VOP_READ() at netbsd:VOP_READ+0x42
> vn_read() at netbsd:vn_read+0x136
> dofileread() at netbsd:dofileread+0x79
> sys_read() at netbsd:sys_read+0x49
> syscall() at netbsd:syscall+0x9d
>=20
> I'm not sure who has the buffer busy; it's probably _not_ the
> process that's exiting and revoking its controlling terminal because
> I don't see anything in that path that could hold any buffers busy
> for any reason:
>=20
> cv_wait_sig() at netbsd:cv_wait_sig+0xd2
> fstrans_setstate() at netbsd:fstrans_setstate+0x10c
> genfs_suspendctl() at netbsd:genfs_suspendctl+0x32
> VFS_SUSPENDCTL() at netbsd:VFS_SUSPENDCTL+0x25
> vfs_suspend() at netbsd:vfs_suspend+0x8d
> vrevoke_suspend_next() at netbsd:vrevoke_suspend_next+0x2a
> vrevoke() at netbsd:vrevoke+0x44
> genfs_revoke() at netbsd:genfs_revoke+0x13
> VOP_REVOKE() at netbsd:VOP_REVOKE+0x3b
> exit1() at netbsd:exit1+0x7bd
> sys_exit() at netbsd:sys_exit+0x3b
>=20
> Having stack traces of other threads might help to find the cycle.
>=20
> - In https://mail-index.netbsd.org/tech-kern/2025/06/29/msg030580.html
> buhrow never followed up with stack traces, but csh was waiting in
> either open, close, or revoke (most likely revoke, I'd guess) on
> specfs_iocv for another thread in the middle of spec_io_enter/exit
> or spec_node_close. Also not sure how that path could hold any
> buffers busy.
<snip>
I was able to reproduce this deadllock on my test machine and I'm sure
the buffer is busy on the fss kthread waitiung for fss_bs_io -> vn_lock
and here waiting for the end of the suspension of the file system:
sleepq_block at sys/kern/kern_sleepq.c:392
cv_wait at sys/kern/kern_condvar.c:182
_fstrans_start at sys/kern/vfs_trans.c:554
fstrans_start at sys/kern/vfs_trans.c:567
vop_pre at sys/kern/vnode_if.c:86
VOP_LOCK at sys/kern/vnode_if.c:1629
vn_lock at sys/kern/vfs_vnops.c:1384
fss_bs_io at sys/dev/fss.c:1098
fss_bs_thread at sys/dev/fss.c:1198
The recipe from Henryk works well, use old style ptys /dev/ptyXX, take a
peristent snapshot on root fs, run dd if=3D/dev/fssX and continously
login/logout with ssh to force the revoke of ths pty -> deadlock.
--
J. Hannken-Illjes - hannken@mailbox.org
State-Changed-From-To: open->needs-pullups
State-Changed-By: riastradh@NetBSD.org
State-Changed-When: Fri, 31 Jul 2026 16:03:31 +0000
State-Changed-Why:
Great, sounds like the mechanism of the deadlock is understood now, so
let's get this pulled up to 9, 10, and 11.
Sounds like ptyfs will avoid this problem for ssh, but without the
patch, revoke -- whether done explicitly by user programs, or in the
kernel when logging out of a non-pty tty -- will still trigger the
deadlock.
From: "J. Hannken-Illjes" <hannken@mailbox.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/59663 (ffs_snapshot_read -> uvm_fault (or pool page empty)
doing dump/restore with snapshot)
Date: Fri, 31 Jul 2026 18:10:44 +0200
> On 31. Jul 2026, at 18:03, riastradh@netbsd.org <riastradh@NetBSD.org> =
wrote:
>=20
> Synopsis: ffs_snapshot_read -> uvm_fault (or pool page empty) doing =
dump/restore with snapshot
>=20
> State-Changed-From-To: open->needs-pullups
> State-Changed-By: riastradh@NetBSD.org
> State-Changed-When: Fri, 31 Jul 2026 16:03:31 +0000
> State-Changed-Why:
> Great, sounds like the mechanism of the deadlock is understood now, so
> let's get this pulled up to 9, 10, and 11.
>=20
> Sounds like ptyfs will avoid this problem for ssh, but without the
> patch, revoke -- whether done explicitly by user programs, or in the
> kernel when logging out of a non-pty tty -- will still trigger the
> deadlock.
Pullups already requested as:
9 #2042
10 #1320
11 #409
--
J. Hannken-Illjes - hannken@mailbox.org
State-Changed-From-To: needs-pullups->pending-pullups
State-Changed-By: riastradh@NetBSD.org
State-Changed-When: Fri, 31 Jul 2026 19:20:53 +0000
State-Changed-Why:
fixed in HEAD, pullups requested to 11, 10, and 9 as noted above
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/59663 CVS commit: [netbsd-11] src/sys/dev
Date: Sun, 2 Aug 2026 13:43:18 +0000
Module Name: src
Committed By: martin
Date: Sun Aug 2 13:43:18 UTC 2026
Modified Files:
src/sys/dev [netbsd-11]: fss.c
Log Message:
Pull up following revision(s) (requested by hannken in ticket #409):
sys/dev/fss.c: revision 1.116
Bracket I/O from or to the backing store with fstrans_start_lazy()
so the fss device doesn't block before the file system is suspended.
Should fix some deadlocks with revokation of ttys.
PR kern/59663 (ffs_snapshot_read -> uvm_fault (or pool page empty)
To generate a diff of this commit:
cvs rdiff -u -r1.114 -r1.114.8.1 src/sys/dev/fss.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/59663 CVS commit: [netbsd-10] src/sys/dev
Date: Sun, 2 Aug 2026 13:44:42 +0000
Module Name: src
Committed By: martin
Date: Sun Aug 2 13:44:42 UTC 2026
Modified Files:
src/sys/dev [netbsd-10]: fss.c
Log Message:
Pull up following revision(s) (requested by hannken in ticket #1320):
sys/dev/fss.c: revision 1.116
Bracket I/O from or to the backing store with fstrans_start_lazy()
so the fss device doesn't block before the file system is suspended.
Should fix some deadlocks with revokation of ttys.
PR kern/59663 (ffs_snapshot_read -> uvm_fault (or pool page empty)
To generate a diff of this commit:
cvs rdiff -u -r1.113.4.1 -r1.113.4.2 src/sys/dev/fss.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/59663 CVS commit: [netbsd-9] src/sys/dev
Date: Sun, 2 Aug 2026 13:46:01 +0000
Module Name: src
Committed By: martin
Date: Sun Aug 2 13:46:01 UTC 2026
Modified Files:
src/sys/dev [netbsd-9]: fss.c
Log Message:
Pull up following revision(s) (requested by hannken in ticket #2042):
sys/dev/fss.c: revision 1.116
Bracket I/O from or to the backing store with fstrans_start_lazy()
so the fss device doesn't block before the file system is suspended.
Should fix some deadlocks with revokation of ttys.
PR kern/59663 (ffs_snapshot_read -> uvm_fault (or pool page empty)
To generate a diff of this commit:
cvs rdiff -u -r1.107.4.1 -r1.107.4.2 src/sys/dev/fss.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: pending-pullups->closed
State-Changed-By: maya@NetBSD.org
State-Changed-When: Wed, 02 Sep 2026 18:52:13 +0000
State-Changed-Why:
Pullups done
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home