NetBSD Problem Report #60531
From www@netbsd.org Sun Aug 2 14:39:25 2026
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
client-signature RSA-PSS (2048 bits) client-digest SHA256)
(Client CN "mail.netbsd.org", Issuer "YR2" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 122C01A923B
for <gnats-bugs@gnats.NetBSD.org>; Sun, 2 Aug 2026 14:39:25 +0000 (UTC)
Message-Id: <20260802143923.8DB6E1A923C@mollari.NetBSD.org>
Date: Sun, 2 Aug 2026 14:39:23 +0000 (UTC)
From: campbell+netbsd@mumble.net
Reply-To: campbell+netbsd@mumble.net
To: gnats-bugs@NetBSD.org
Subject: looping "vflushbuf: dirty" message
X-Send-Pr-Version: www-1.0
X-From4GNATS: "campbell+netbsd@mumble.net via gnats" <gnats-admin@NetBSD.org>
>Number: 60531
>Notify-List: hannken
>Category: kern
>Synopsis: looping "vflushbuf: dirty" message
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: kern-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sun Aug 02 14:40:00 +0000 2026
>Last-Modified: Mon Aug 17 15:55:01 +0000 2026
>Originator: Taylor R Campbell
>Release: 10.1
>Organization:
The NetBSD Foundation, Inc.
>Environment:
NetBSD franklin.NetBSD.org 10.1_STABLE NetBSD 10.1_STABLE (FRANKLIN) #0: Tue Jan 28 19:43:34 UTC 2025 spz@franklin.NetBSD.org:/home/netbsd/10/amd64/obj/sys/arch/amd64/compile/FRANKLIN amd64
>Description:
Sometimes, under heavy load (possibly snapshot-related), the
kernel starts to spew this to the console:
vflushbuf: dirty: vnode 0xffffea44c2bd54c0 flags 0x4010<MPSAFE,ONWORKLST>
tag VT_UFS(1) type VBLK(3) mount 0xffffea4ab61a2000 typedata 0xffffea4ab5f92560
usecount 123652 writecount 0 holdcount 6035
size 0 writesize 0 numoutput 0
data 0xffffea44c2602558 lock 0xffffea44c2bd5680
state LOADED key(0xffffea4ab61a2000 8) dd 23 23 00 00 00 00 00
lrulisthd 0xffffffff80e88390
tag VT_UFS, ino 2302941, on dev 4, 0 flags 0x0, nlink 1
mode 060640, owner 0, group 5, size 0
or
vflushbuf: dirty: vnode 0xffffcca82a4784c0 flags 0x4010<MPSAFE,ONWORKLST>
tag VT_UFS(1) type VBLK(3) mount 0xffffcca82c32b000 typedata 0xffffcca82c13cf00
usecount 30 writecount 0 holdcount 32
size 0 writesize 0 numoutput 0
data 0xffffcca82ab9a668 lock 0xffffcca82a478680
state LOADED key(0xffffcca82c32b000 8) dd 23 23 00 00 00 00 00
lrulisthd 0xffffffff80e88390
tag VT_UFS, ino 2302941, on dev 4, 0 flags 0x0, nlink 1
mode 060640, owner 0, group 5, size 0
An incomprehensibly terse message like this is scary and bad
enough for warning fatigue even if it is otherwise harmless,
but it also sometimes renders the machine unusable by spewing
constantly to the console.
The inode and mounts in question here are:
$ ls -li /dev/sd0a /dev/dk0
2302941 brw-r----- 1 root operator 168, 0 Oct 31 2015 /dev/dk0
2301566 brw-r----- 1 root operator 4, 0 Oct 31 2015 /dev/sd0a
$ mount | grep -e dk0 -e sd0a
/dev/sd0a on / type ffs (log, noatime, nodevmtime, local)
/dev/dk0 on /storage type ffs (local)
The messages have manifested on different machines over time,
but these are the only ones I have handy to review. From the
sample I have, and from discussion with hannken@, I suspect
it's always VBLK devices.
When this last happened, there was an active snapshot of /
being dumped, and heavy I/O to /storage to write dumps from
this host and from various other hosts.
There's a comment atop vflushbuf saying it assumes that holding
the vnode lock (which it doesn't assert!) guarantees that new
dirty buffers won't be queued, but I suspect there may be a
path to queueing new dirty buffers _for VBLK vnodes_:
330 /*
331 * Flush all dirty buffers from a vnode.
332 * Called with the underlying vnode locked, which should prevent new dirty
333 * buffers from being queued.
334 */
335 int
336 vflushbuf(struct vnode *vp, int flags)
337 {
...
349 loop:
350 mutex_enter(&bufcache_lock);
351 for (bp = LIST_FIRST(&vp->v_dirtyblkhd); bp; bp = nbp) {
352 KASSERT(bp->b_vp == vp);
...
372 }
373 mutex_exit(&bufcache_lock);
374
375 if (!sync)
376 return 0;
377
378 mutex_enter(vp->v_interlock);
379 while (vp->v_numoutput != 0)
380 cv_wait(&vp->v_cv, vp->v_interlock);
381 dirty = !LIST_EMPTY(&vp->v_dirtyblkhd);
382 mutex_exit(vp->v_interlock);
383
384 if (dirty) {
385 vprint("vflushbuf: dirty", vp);
386 goto loop;
387 }
>How-To-Repeat:
not sure, maybe just heavy I/O and running sync(8)
>Fix:
Yes, please!
Possible workaround: print the message only if the vnode type
is not VBLK.
>Release-Note:
>Audit-Trail:
From: Taylor R Campbell <riastradh@NetBSD.org>
To: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Cc: hannken@NetBSD.org
Subject: Re: kern/60531: looping "vflushbuf: dirty" message
Date: Sun, 2 Aug 2026 15:25:08 +0000
I added KASSERT(VOP_ISLOCKED(vp)) to vflushbuf and to the path in
reassignbuf that puts a buffer on vp->v_dirtyblkhd, and immediately
hit:
# mount -u -o rw,noatime,nodevmtime,log /
[ 20.8447068] panic: kernel diagnostic assertion "VOP_ISLOCKED(vp)" failed=
: file "/home/riastradh/netbsd/current/src/sys/kern/vfs_subr.c", line 526
[ 20.8611358] cpu0: Begin traceback...
[ 20.8611358] trace fp ffffc000c64ff680
[ 20.8786823] fp ffffc000c64ff6b0 vpanic() at ffffc0000061f864 netbsd:vpan=
ic+0x1bc
[ 20.8877138] fp ffffc000c64ff710 kern_assert() at ffffc000008c68b8 netbsd=
:kern_assert+0x58
[ 20.9087004] fp ffffc000c64ff7a0 reassignbuf() at ffffc0000068639c netbsd=
:reassignbuf+0x2bc
[ 20.9297182] fp ffffc000c64ff7d0 bdwrite() at ffffc000006743e8 netbsd:bdw=
rite+0x128
[ 20.9507110] fp ffffc000c64ff800 ffs_nodealloccg() at ffffc00000536178 ne=
tbsd:ffs_nodealloccg+0x730
[ 20.9827495] fp ffffc000c64ff8a0 ffs_hashalloc() at ffffc00000535278 netb=
sd:ffs_hashalloc+0x38
[ 21.0036885] fp ffffc000c64ff900 ffs_valloc() at ffffc00000538400 netbsd:=
ffs_valloc+0xe0
[ 21.0248940] fp ffffc000c64ff970 ffs_newvnode() at ffffc00000546598 netbs=
d:ffs_newvnode+0x88
[ 21.0457100] fp ffffc000c64ffa00 vcache_new() at ffffc00000696738 netbsd:=
vcache_new+0x94
[ 21.0667119] fp ffffc000c64ffa80 wapbl_log_position() at ffffc0000054ac3c=
netbsd:wapbl_log_position+0x1f8
[ 21.0987335] fp ffffc000c64ffca0 ffs_wapbl_start() at ffffc0000054b9b4 ne=
tbsd:ffs_wapbl_start+0x1e8
[ 21.1197893] fp ffffc000c64ffd00 ffs_mount() at ffffc00000548564 netbsd:f=
fs_mount+0x564
[ 21.1407173] fp ffffc000c64ffd60 do_sys_mount() at ffffc0000068a4c4 netbs=
d:do_sys_mount+0x584
[ 21.1617312] fp ffffc000c64ffe10 sys___mount50() at ffffc0000068a88c netb=
sd:sys___mount50+0x28
[ 21.1826953] fp ffffc000c64ffe20 syscall() at ffffc000000be2c4 netbsd:sys=
call+0x104
So I think vflushbuf's assumption that buffers are added to
v_dirtyblkhd only while the vnode lock is held may not be valid!
Now, this path involves creating a new vnode, so that may not be
relevant here.
From: Taylor R Campbell <riastradh@NetBSD.org>
To: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Cc: hannken@NetBSD.org, dholland@NetBSD.org
Subject: Re: kern/60531: looping "vflushbuf: dirty" message
Date: Sun, 2 Aug 2026 19:43:58 +0000
> Date: Sun, 2 Aug 2026 15:25:08 +0000
> From: Taylor R Campbell <riastradh@NetBSD.org>
>=20
> I added KASSERT(VOP_ISLOCKED(vp)) to vflushbuf and to the path in
> reassignbuf that puts a buffer on vp->v_dirtyblkhd, and immediately
> hit:
>=20
> # mount -u -o rw,noatime,nodevmtime,log /
> [ 20.8447068] panic: kernel diagnostic assertion "VOP_ISLOCKED(vp)" fail=
ed: file "/home/riastradh/netbsd/current/src/sys/kern/vfs_subr.c", line 526
> [ 20.8611358] cpu0: Begin traceback...
> [ 20.8611358] trace fp ffffc000c64ff680
> ...
> [ 20.9297182] fp ffffc000c64ff7d0 bdwrite() at ffffc000006743e8 netbsd:b=
dwrite+0x128
> [ 20.9507110] fp ffffc000c64ff800 ffs_nodealloccg() at ffffc00000536178 =
netbsd:ffs_nodealloccg+0x730
> ...
> [ 21.0457100] fp ffffc000c64ffa00 vcache_new() at ffffc00000696738 netbs=
d:vcache_new+0x94
> ...
> So I think vflushbuf's assumption that buffers are added to
> v_dirtyblkhd only while the vnode lock is held may not be valid!
>=20
> Now, this path involves creating a new vnode, so that may not be
> relevant here.
dholland and I took a closer look at this path, and it turns out the
assertion failure is not about the vnode being created -- it's about
the block device the file system is mounted from:
1261 static daddr_t
1262 ffs_nodealloccg(struct inode *ip, u_int cg, daddr_t ipref, int mode=
, int realsize,
1263 int flags)
1264 {
...
1291 error =3D bread(ip->i_devvp, FFS_FSBTODB(fs, cgtod(fs, cg)),
1292 (int)fs->fs_cgsize, B_MODIFY, &bp);
1293 if (error)
1294 goto fail;
...
1416 if (ibp !=3D NULL) {
1417 bwrite(ibp);
1418 bwrite(bp);
1419 } else
=3D> 1420 bdwrite(bp);
1421 return ((ino_t)(cg * fs->fs_ipg + ipref));
https://nxr.netbsd.org/xref/src/sys/ufs/ffs/ffs_alloc.c?r=3D1.175#1252
Here ip->i_devvp is presumably the block device the file system is
mounted from (why do we even have ip->i_devvp instead of just using
ump->um_devvp?):
2182 int
2183 ffs_loadvnode(struct mount *mp, struct vnode *vp,
2184 const void *key, size_t key_len, const void **new_key)
2185 {
...
2212 ip->i_devvp =3D ump->um_devvp;
https://nxr.netbsd.org/xref/src/sys/ufs/ffs/ffs_vfsops.c?r=3D1.385#2212
2234 int
2235 ffs_newvnode(struct mount *mp, struct vnode *dvp, struct vnode *vp,
2236 struct vattr *vap, kauth_cred_t cred, void *extra,
2237 size_t *key_len, const void **new_key)
2238 {
...
2335 ip->i_devvp =3D ump->um_devvp;
https://nxr.netbsd.org/xref/src/sys/ufs/ffs/ffs_vfsops.c?r=3D1.385#2335
I suspect there a are a lot of paths like this: bread/bdwrite on the
block device to edit file system metadata.
What are the bad consequences of calling reassignbuf without the vnode
lock held? Well, vflushbuf can only guarantee completion in bounded
time if either:
(a) other threads are blocked from adding new dirty bufs,
or
(b) vflushbuf can distinguish the bufs that were already on the queue
when it started from those that were added after it started, e.g.
with another queue or a flush generation counter.
Without one of those two options, v_numoutput might never reach zero.
On the other hand, once it _has_ reached zero once, we don't really
need to loop at all: the printf is only evidence that the system
_could_ have waited indefinitely in vflushbuf because new bwrites kept
appearing as we waited -- but it _didn't_, because v_numoutput finally
went down to zero.
So I think we should do some combination of:
1. delete the `goto loop' part
2. put the vprint under #ifdef DEBUG
3. vprint only once per call to vflushbuf, or at most twice -- once
when we notice a potential problem, once when we've finally broken
out of the loop
4. apply a global rate limit to the vprint
5. put a flush generation number on struct buf and wait for that to
drain (i.e., wait until there are no dirty bufs queued with the old
generation number) instead of waiting for v_numoutput to drain
Curiously, this printf was put under #ifdef DIAGNOSTIC in the CSRG BSD
tree back in 1992 when vflushbuf was deleted and open-coded in
ffs_fsync:
int
ffs_fsync(ap)
- struct vop_fsync_args *ap;
+ struct vop_fsync_args /* {
+ struct vnode *a_vp;
+ struct ucred *a_cred;
+ int a_waitfor;
+ struct proc *a_p;
+ } */ *ap;
{
...
- vflushbuf(ap->a_vp, ap->a_waitfor =3D=3D MNT_WAIT ? B_SYNC : 0);
+ /*
+ * Flush all dirty buffers associated with a vnode.
+ */
+loop:
+ s =3D splbio();
+ for (bp =3D vp->v_dirtyblkhd; bp; bp =3D nbp) {
+ nbp =3D bp->b_blockf;
+ if ((bp->b_flags & B_BUSY))
+ continue;
+ if ((bp->b_flags & B_DELWRI) =3D=3D 0)
+ panic("ffs_fsync: not dirty");
+ bremfree(bp);
+ bp->b_flags |=3D B_BUSY;
+ splx(s);
+ /*
+ * Wait for I/O associated with indirect blocks to complete,
+ * since there is no way to quickly wait for them below.
+ */
+ if (bp->b_vp =3D=3D vp || ap->a_waitfor =3D=3D MNT_NOWAIT)
+ (void) bawrite(bp);
+ else
+ (void) bwrite(bp);
+ goto loop;
+ }
+ if (ap->a_waitfor =3D=3D MNT_WAIT) {
+ while (vp->v_numoutput) {
+ vp->v_flag |=3D VBWAIT;
+ sleep((caddr_t)&vp->v_numoutput, PRIBIO + 1);
+ }
+#ifdef DIAGNOSTIC
+ if (vp->v_dirtyblkhd) {
+ vprint("ffs_fsync: dirty", vp);
+ goto loop;
+ }
+#endif
+ }
https://github.com/robohack/ucb-csrg-bsd/commit/fce3da350147b2a731b529826ce=
af7f89fb33311
From: Taylor R Campbell <riastradh@NetBSD.org>
To: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Cc: hannken@NetBSD.org, dholland@NetBSD.org
Subject: Re: kern/60531: looping "vflushbuf: dirty" message
Date: Sun, 2 Aug 2026 20:20:48 +0000
Found another path to reassignbuf called without the vnode lock, this
time for a VREG vnode instead of VBLK -- this one is definitely
snapshot-related (presumably the vnode is the snapshot backing store,
since it's marked VV_SYSTEM), and could cause the vflushbuf warning
message and/or almost unbounded running time for the vflushbuf loop
(really, bounded by the file system size; once the whole fs is backed
up, there won't be any more writes to it):
# mount
/dev/dk1 on / type ffs (log, local)
ptyfs on /dev/pts type ptyfs (local)
tmpfs on /tmp type tmpfs (local)
tmpfs on /var/run type tmpfs (local)
tmpfs on /var/shm type tmpfs (local)
/dev/vio9p0 on /mnt type puffs|9p
/dev/ld4a on /mnt type ffs (local)
# dump -X -0ant -f - / | gzip -nc >/mnt/var/tmp/dump.0
[20:10:11 UTC] DUMP: Found /dev/rdk1 on / in /etc/fstab
[20:10:11 UTC] DUMP: Date of this level 0 dump: Sun Aug 2 20:10:11 2026
[20:10:11 UTC] DUMP: Date of last level 0 dump: the epoch
[20:10:11 UTC] DUMP: Dumping a snapshot of /dev/rdk1 (/) to standard outp=
ut
[20:10:11 UTC] DUMP: Label: none
[20:10:13 UTC] DUMP: mapping (Pass I) [regular files]
[20:10:19 UTC] DUMP: mapping (Pass II) [directories]
[20:10:19 UTC] DUMP: estimated 3140383 tape blocks.
[20:10:19 UTC] DUMP: Volume 1 started at: Sun Aug 2 20:10:19 2026
[20:10:19 UTC] DUMP: dumping (Pass III) [directories]
[ 479.4390328] panic: kernel diagnostic assertion "vp->v_type =3D=3D VBLK |=
| VOP_ISLOCKED(vp)" failed: file "/home/riastradh/netbsd/current/src/sys/ke=
rn/vfs_subr.c", line 562 vp=3D0xffff00007cf02800 state=3D3
[ 479.4577654] cpu0: Begin traceback...
[ 479.4577654] trace fp ffffc000c63c78f0
[ 479.4790797] fp ffffc000c63c7920 vpanic() at ffffc0000061f864 netbsd:vpan=
ic+0x1bc
[ 479.5110599] fp ffffc000c63c7980 kern_assert() at ffffc000008c6958 netbsd=
:kern_assert+0x58
[ 479.5425878] fp ffffc000c63c7a10 reassignbuf() at ffffc00000686434 netbsd=
:reassignbuf+0x2d4
[ 479.5740262] fp ffffc000c63c7a40 bdwrite() at ffffc000006743e8 netbsd:bdw=
rite+0x128
[ 479.5961369] fp ffffc000c63c7a70 ffs_balloc() at ffffc0000053c240 netbsd:=
ffs_balloc+0x15e0
[ 479.6380279] fp ffffc000c63c7bd0 wrsnapblk() at ffffc000005401c4 netbsd:w=
rsnapblk+0x44
[ 479.6800518] fp ffffc000c63c7c10 ffs_copyonwrite() at ffffc00000540b50 ne=
tbsd:ffs_copyonwrite+0x1a4
[ 479.7140520] fp ffffc000c63c7ca0 fscow_run() at ffffc0000069236c netbsd:f=
scow_run+0xbc
[ 479.7349917] fp ffffc000c63c7ce0 bread() at ffffc00000676278 netbsd:bread=
+0x58
[ 479.7680479] fp ffffc000c63c7d10 ffs_update() at ffffc0000053ed94 netbsd:=
ffs_update+0x128
[ 479.8000553] fp ffffc000c63c7d70 ffs_sync() at ffffc00000548ae8 netbsd:ff=
s_sync+0x1e8
[ 479.8209946] fp ffffc000c63c7df0 VFS_SYNC() at ffffc000006874e4 netbsd:VF=
S_SYNC+0x24
[ 479.8540963] fp ffffc000c63c7e20 sched_sync() at ffffc00000687670 netbsd:=
sched_sync+0x110
[ 479.8860508] tf ffffc000c63c7ed0 el0_trap() at ffffc000000c0ff0 netbsd:el=
1_trap_exit+0x68
[ 479.8960548] cpu0: End traceback...
Stopped in pid 0.124 (system) at netbsd:cpu_Debugger+0xc: ldp
x29, x30, [sp],#16
db{0}> show vnode ffff00007cf02800
OBJECT 0xffff00007cf02800: locked=3D0, pgops=3D0xffffc00000aebb48, npages=
=3D0, refs=3D1
vnode 0xffff00007cf02800 flags 0x12<SYSTEM,MPSAFE>
tag VT_UFS(1) type VREG(1) mount 0xffff00007daf2000 typedata 0x0
usecount 1 writecount 0 holdcount 410
size f9c00050 writesize f9c00050 numoutput 0
data 0xffff00007d0d94c0 lock 0xffff00007cf02a80
state LOADED key(0xffff00007daf2000 8) 67 1f 00 00 00 00 00 00
lrulisthd 0xffffc00001265e10
db{0}> call vprint(0,ffff00007cf02800)
vnode 0xffff00007cf02800 flags 0x12<SYSTEM,MPSAFE>
[ 479.8960548] tag VT_UFS(1) type VREG(1) mount 0xffff00007daf2000 typedat=
a 0x0
[ 479.8960548] usecount 1 writecount 0 holdcount 410
[ 479.8960548] size f9c00050 writesize f9c00050 numoutput 0
[ 479.8960548] data 0xffff00007d0d94c0 lock 0xffff00007cf02a80
[ 479.8960548] state LOADED key(0xffff00007daf2000 8) 67 1f 00 00 00 00 00=
00
[ 479.8960548] lrulisthd 0xffffc00001265e10
[ 479.8960548] tag VT_UFS, ino 8039, on dev 168, 1 flags 0x6, nlink 0
[ 479.8960548] mode 0100600, owner 0, group 0, size 4190109776
0
From: Taylor R Campbell <riastradh@NetBSD.org>
To: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Cc: hannken@NetBSD.org, dholland@NetBSD.org
Subject: Re: kern/60531: looping "vflushbuf: dirty" message
Date: Mon, 3 Aug 2026 00:38:22 +0000
This is a multi-part message in MIME format.
--=_QJyEfqlAfncoM8fB56hLDQdX5vxg3caH
I prepared a pair of patches:
1. WORKAROUND: pr60531-vflushbufdirty-ratelimit.patch works around the
problem of overwhelming the console by putting the message under
#ifdef DEBUG, and limiting it to once per call, and globally
rate-limiting the output under bufcache_lock.
It doesn't fix the underlying problem that the time vflushbuf can
take is unbounded even if each write completes in bounded time --
it's just a low-risk change for quick pullup, because this is
interfering with backups on TNF infrastructure, and probably others
too.
2. FIX: pr60531-vflushbufdirty-flushgen.patch fixes the problem that
the time vflushbuf can take is unbounded, not by making the invalid
assumption that reassignbuf(bp, vp) hold vp's vnode lock, but
rather by using a global flush generation counter:
- For dirty buffers, reassignbuf(bp, vp) sets bp->b_flushgen to the
global bufcache_flushgen.
- vflushbuf(vp) grabs the current bufcache_flushgen as f,
increments bufcache_flushgen, and then waits until all buffers bp
on vp->v_dirtyblkhd with bp->b_flushgen <= f have completed.
This should properly fix the underlying problem.
(Haven't yet thought about whether vinvalbuf or vtruncbuf need to
consider flushgen too.)
--=_QJyEfqlAfncoM8fB56hLDQdX5vxg3caH
Content-Type: text/plain; charset="ISO-8859-1"; name="pr60531-vflushbufdirty-ratelimit"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment; filename="pr60531-vflushbufdirty-ratelimit.patch"
# HG changeset patch
# User Taylor R Campbell <riastradh@NetBSD.org>
# Date 1785716340 0
# Mon Aug 03 00:19:00 2026 +0000
# Branch trunk
# Node ID 2c597c41062d49e186d14e507ecf9c07b940937f
# Parent 1f2d855671ed4d4eee9ff4420a357756f04e271a
# EXP-Topic riastradh-pr60531-vflushbufdirty
vflushbuf: Print `vflushbuf: dirty' warning only once per call.
And rate-limit it globally.
This can happen when vflushbuf is competing with an onslaught of
concurrent I/O, which might be happening indefinitely, so we are
likely to have to print a _lot_ of warnings, potentially overwhelming
the console, until the underlying problem is fixed by teaching
vflushbuf to wait only for the writes that began before have
completed, not also all writes that have begun since.
This stop-gap measure intended to be low-risk for pullup to release
branches.
PR kern/60531: looping "vflushbuf: dirty" message
diff -r 1f2d855671ed -r 2c597c41062d sys/kern/vfs_subr.c
--- a/sys/kern/vfs_subr.c Sat Aug 01 05:12:21 2026 +0000
+++ b/sys/kern/vfs_subr.c Mon Aug 03 00:19:00 2026 +0000
@@ -338,6 +338,9 @@ vflushbuf(struct vnode *vp, int flags)
struct buf *bp, *nbp;
int error, pflags;
bool dirty, sync;
+#ifdef DEBUG
+ bool warned =3D false;
+#endif
=20
sync =3D (flags & FSYNC_WAIT) !=3D 0;
pflags =3D PGO_CLEANIT | PGO_ALLPAGES |
@@ -382,7 +385,18 @@ loop:
mutex_exit(vp->v_interlock);
=20
if (dirty) {
- vprint("vflushbuf: dirty", vp);
+#ifdef DEBUG
+ if (!warned) {
+ static struct timeval vflushbuf_warntime;
+ const struct timeval interval =3D {1,0};
+
+ mutex_enter(&bufcache_lock);
+ if (ratecheck(&vflushbuf_warntime, &interval))
+ vprint("vflushbuf: dirty", vp);
+ mutex_exit(&bufcache_lock);
+ warned =3D true;
+ }
+#endif
goto loop;
}
=20
--=_QJyEfqlAfncoM8fB56hLDQdX5vxg3caH
Content-Type: text/plain; charset="ISO-8859-1"; name="pr60531-vflushbufdirty-flushgen"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment; filename="pr60531-vflushbufdirty-flushgen.patch"
# HG changeset patch
# User Taylor R Campbell <riastradh@NetBSD.org>
# Date 1785715442 0
# Mon Aug 03 00:04:02 2026 +0000
# Branch trunk
# Node ID 1946a089abf1ad98fd95ac56d7030395dc4213a3
# Parent 1f2d855671ed4d4eee9ff4420a357756f04e271a
# EXP-Topic riastradh-pr60531-vflushbufdirty
vfs: Fix `vflushbuf: dirty'.
Currently, vflushbuf(vp) requires the caller to hold vp's vnode lock,
and assumes that reassignbuf(..., vp) is only done while vp's vnode
lock is held.
Under this assumption, vp->v_dirtyblkhd can never have dirty blocks
added to it once vflushbuf starts, so the set of block writes that
vflushbuf must wait for is limited to those dirty blocks that were
already queued before it began. Thus, it can safely wait until the
vnode's number of active block writes (v_numoutput) has gone to zero,
which it will in bounded time as long as each block write
independently completes in bounded time.
Unfortunately, this assumption is invalid, in at least two
circumstances:
- vp is a block device vnode on which another file system is mounted,
and the file system uses getblk/bread/bwrite on vp to edit file
system metadata. These paths do not take vp's vnode lock. Perhaps
these could take vp's vnode lock because they will generally only
hold _another file system's_ vnode lock, but these paths are
extensively scattered through file system logic and confidently
finding them and adjusting them to take the vnode lock -- and
proving that it's safe to do so -- seems likely no small task.
- vp is a snapshot backing store file on ffs (SF_SNAPSHOT). Data
copied on write for the snapshot is written with getblk/bwrite.
This path does not take vp's vnode lock, and taking it might be
troublesome because the cow logic runs while holding other vnode
locks, so it is a high risk of deadlocks.
To avoid this assumption, we instead cache a global flush generation
number in each dirty block as it is queued in reassignbuf. Each call
to vflushbuf advances the flush generation. This way, vflushbuf can
detect which dirty blocks were already there when it started, and
which ones are new, and wait only until the ones that were already
there are done.
Since the buffercache(9) path never has B_PHYS set, I repurposed two
existing members of struct buf (b_proc, b_saveaddr) that are used
only for B_PHYS buffers, to make room for for a uint64_t flush
generation number used only for !B_PHYS buffers. Using a 64-bit
generation number guarantees that it will never wrap around in our
lifetimes, even if we issue sync(8) sequentially once per nanosecond,
and it completes in that time too.
PR kern/60531: looping "vflushbuf: dirty" message
diff -r 1f2d855671ed -r 1946a089abf1 sys/kern/vfs_subr.c
--- a/sys/kern/vfs_subr.c Sat Aug 01 05:12:21 2026 +0000
+++ b/sys/kern/vfs_subr.c Mon Aug 03 00:04:02 2026 +0000
@@ -171,6 +171,8 @@ const int vttoif_tab[9] =3D {
(bp)->b_vnbufs.le_next =3D NOLIST; \
}
=20
+static uint64_t bufcache_flushgen =3D 1;
+
int doforce =3D 1; /* 1 =3D> permit forcible unmounting */
=20
/*
@@ -329,8 +331,7 @@ restart:
=20
/*
* Flush all dirty buffers from a vnode.
- * Called with the underlying vnode locked, which should prevent new dirty
- * buffers from being queued.
+ * Called with the underlying vnode locked.
*/
int
vflushbuf(struct vnode *vp, int flags)
@@ -338,7 +339,11 @@ vflushbuf(struct vnode *vp, int flags)
struct buf *bp, *nbp;
int error, pflags;
bool dirty, sync;
+ uint64_t flushgen;
=20
+ KASSERT(VOP_ISLOCKED(vp));
+
+ flushgen =3D 0;
sync =3D (flags & FSYNC_WAIT) !=3D 0;
pflags =3D PGO_CLEANIT | PGO_ALLPAGES |
(sync ? PGO_SYNCIO : 0) |
@@ -346,12 +351,20 @@ vflushbuf(struct vnode *vp, int flags)
rw_enter(vp->v_uobj.vmobjlock, RW_WRITER);
(void) VOP_PUTPAGES(vp, 0, 0, pflags);
=20
+ /*
+ * Ensure every dirty block associated with this vnode has
+ * begun to be written out (BC_BUSY, I/O in progress) -- any
+ * delayed writes must delay no longer.
+ */
loop:
mutex_enter(&bufcache_lock);
+ if (flushgen =3D=3D 0)
+ flushgen =3D bufcache_flushgen++;
for (bp =3D LIST_FIRST(&vp->v_dirtyblkhd); bp; bp =3D nbp) {
KASSERT(bp->b_vp =3D=3D vp);
+ KASSERT((bp->b_flags & B_PHYS) =3D=3D 0);
nbp =3D LIST_NEXT(bp, b_vnbufs);
- if ((bp->b_cflags & BC_BUSY))
+ if ((bp->b_cflags & BC_BUSY) !=3D 0 || bp->b_flushgen > flushgen)
continue;
if ((bp->b_oflags & BO_DELWRI) =3D=3D 0)
panic("vflushbuf: not dirty, bp %p", bp);
@@ -375,17 +388,23 @@ loop:
if (!sync)
return 0;
=20
+ /*
+ * Wait until all pending writes issued before we incremented
+ * bufcache_flushgen have completed.
+ */
mutex_enter(vp->v_interlock);
- while (vp->v_numoutput !=3D 0)
- cv_wait(&vp->v_cv, vp->v_interlock);
- dirty =3D !LIST_EMPTY(&vp->v_dirtyblkhd);
+ do {
+ dirty =3D false;
+ for (bp =3D LIST_FIRST(&vp->v_dirtyblkhd); bp; bp =3D nbp) {
+ if (bp->b_flushgen > flushgen)
+ continue;
+ dirty =3D true;
+ cv_wait(&vp->v_cv, vp->v_interlock);
+ break;
+ }
+ } while (dirty);
mutex_exit(vp->v_interlock);
=20
- if (dirty) {
- vprint("vflushbuf: dirty", vp);
- goto loop;
- }
-
return 0;
}
=20
@@ -497,6 +516,7 @@ reassignbuf(struct buf *bp, struct vnode
KASSERT(bp->b_objlock =3D=3D vp->v_interlock);
KASSERT(mutex_owned(vp->v_interlock));
KASSERT((bp->b_cflags & BC_BUSY) !=3D 0);
+ KASSERT((bp->b_flags & B_PHYS) =3D=3D 0);
=20
/*
* Delete from old vnode list, if on one.
@@ -535,6 +555,7 @@ reassignbuf(struct buf *bp, struct vnode
(vp->v_mount->mnt_flag & MNT_ASYNC) =3D=3D 0)
vn_syncer_add_to_worklist(vp, delayx);
}
+ bp->b_flushgen =3D bufcache_flushgen;
}
bufinsvn(bp, listheadp);
}
diff -r 1f2d855671ed -r 1946a089abf1 sys/sys/buf.h
--- a/sys/sys/buf.h Sat Aug 01 05:12:21 2026 +0000
+++ b/sys/sys/buf.h Mon Aug 03 00:04:02 2026 +0000
@@ -135,8 +135,15 @@ struct buf {
(partition relative) */
daddr_t b_rawblkno; /* b: raw physical block number
(volume relative) */
- struct proc *b_proc; /* b: proc if BB_PHYS */
- void *b_saveaddr; /* b: saved b_data for physio */
+ union {
+ /* B_PHYS */
+ struct {
+ struct proc *b_proc; /* b: proc if B_PHYS */
+ void *b_saveaddr; /* b: saved b_data for physio */
+ };
+ /* !B_PHYS */
+ uint64_t b_flushgen; /* b: flush generation */
+ };
struct cpu_info *b_ci; /* b: originating CPU */
=20
/*
--=_QJyEfqlAfncoM8fB56hLDQdX5vxg3caH--
From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/60531 CVS commit: src/sys/kern
Date: Thu, 6 Aug 2026 22:28:24 +0000
Module Name: src
Committed By: riastradh
Date: Thu Aug 6 22:28:23 UTC 2026
Modified Files:
src/sys/kern: vfs_subr.c
Log Message:
vflushbuf: Print `vflushbuf: dirty' warning only once per call.
And do it only under DEBUG, and rate-limit it globally.
This can happen when vflushbuf is competing with an onslaught of
concurrent I/O on a snapshot or block device (which is done without
holding the vnode lock, despite what the comment says). That
concurrent I/O might be happening indefinitely, so we are likely to
have to print a _lot_ of warnings, potentially overwhelming the
console, until the underlying problem is fixed by teaching vflushbuf
to wait only for the writes that began before have completed, not
also all writes that have begun since.
This stop-gap measure intended to be low-risk for pullup to release
branches.
PR kern/60531: looping "vflushbuf: dirty" message
To generate a diff of this commit:
cvs rdiff -u -r1.503 -r1.504 src/sys/kern/vfs_subr.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "J. Hannken-Illjes" <hannken@netbsd.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/60531: looping "vflushbuf: dirty" message
Date: Fri, 7 Aug 2026 16:39:33 +0000
--ItY42kjenHJt4ZkN
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
The FIX will not work, it is not ok to replace 'vp->v_numoutput == 0' with
'LIST_EMPTY(&vp->v_dirtyblkhd)'.
Buffers are removed from v_dirtyblkhd through reassignbuf() when bwrite()
initiates the write while v_numoutput gets decremented through
biodone()->vwakekup() when the I/O has completed.
With your patch VOP_FSYNC(vp, FSYNC_WAIT) would return before all buffers
are written to disk.
The first part should be sufficient to prevent looping here, an updfated
patch is attached.
--
J. Hannken-Illjes - hannken@netbsd.org
--ItY42kjenHJt4ZkN
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment; filename=pr60531.patch
diff -r 983d69604667 sys/kern/vfs_subr.c
--- sys/kern/vfs_subr.c Fri Aug 07 13:18:45 2026 +0200
+++ sys/kern/vfs_subr.c Fri Aug 07 18:28:00 2026 +0200
@@ -171,6 +171,8 @@ const int vttoif_tab[9] = {
(bp)->b_vnbufs.le_next = NOLIST; \
}
+static uint64_t bufcache_flushgen = 1;
+
int doforce = 1; /* 1 => permit forcible unmounting */
/*
@@ -329,19 +331,19 @@ restart:
/*
* Flush all dirty buffers from a vnode.
- * Called with the underlying vnode locked, which should prevent new dirty
- * buffers from being queued.
+ * Called with the underlying vnode locked.
*/
int
vflushbuf(struct vnode *vp, int flags)
{
struct buf *bp, *nbp;
int error, pflags;
- bool dirty, sync;
-#ifdef DEBUG
- bool warned = false;
-#endif
+ bool sync;
+ uint64_t flushgen;
+ KASSERT(VOP_ISLOCKED(vp));
+
+ flushgen = 0;
sync = (flags & FSYNC_WAIT) != 0;
pflags = PGO_CLEANIT | PGO_ALLPAGES |
(sync ? PGO_SYNCIO : 0) |
@@ -349,12 +351,20 @@ vflushbuf(struct vnode *vp, int flags)
rw_enter(vp->v_uobj.vmobjlock, RW_WRITER);
(void) VOP_PUTPAGES(vp, 0, 0, pflags);
+ /*
+ * Ensure every dirty block associated with this vnode has
+ * begun to be written out (BC_BUSY, I/O in progress) -- any
+ * delayed writes must delay no longer.
+ */
loop:
mutex_enter(&bufcache_lock);
+ if (flushgen == 0)
+ flushgen = bufcache_flushgen++;
for (bp = LIST_FIRST(&vp->v_dirtyblkhd); bp; bp = nbp) {
KASSERT(bp->b_vp == vp);
+ KASSERT((bp->b_flags & B_PHYS) == 0);
nbp = LIST_NEXT(bp, b_vnbufs);
- if ((bp->b_cflags & BC_BUSY))
+ if ((bp->b_cflags & BC_BUSY) != 0 || bp->b_flushgen > flushgen)
continue;
if ((bp->b_oflags & BO_DELWRI) == 0)
panic("vflushbuf: not dirty, bp %p", bp);
@@ -378,28 +388,17 @@ loop:
if (!sync)
return 0;
+ /*
+ * Wait until all pending writes issued before we incremented
+ * bufcache_flushgen have completed.
+ */
mutex_enter(vp->v_interlock);
while (vp->v_numoutput != 0)
cv_wait(&vp->v_cv, vp->v_interlock);
- dirty = !LIST_EMPTY(&vp->v_dirtyblkhd);
+ KASSERT((vp->v_type == VBLK && spec_node_getmountedfs(vp)) ||
+ LIST_EMPTY(&vp->v_dirtyblkhd));
mutex_exit(vp->v_interlock);
- if (dirty) {
-#ifdef DEBUG
- if (!warned) {
- static struct timeval vflushbuf_warntime;
- const struct timeval interval = {60,0};
-
- mutex_enter(&bufcache_lock);
- if (ratecheck(&vflushbuf_warntime, &interval))
- vprint("vflushbuf: dirty", vp);
- mutex_exit(&bufcache_lock);
- warned = true;
- }
-#endif
- goto loop;
- }
-
return 0;
}
@@ -511,6 +510,7 @@ reassignbuf(struct buf *bp, struct vnode
KASSERT(bp->b_objlock == vp->v_interlock);
KASSERT(mutex_owned(vp->v_interlock));
KASSERT((bp->b_cflags & BC_BUSY) != 0);
+ KASSERT((bp->b_flags & B_PHYS) == 0);
/*
* Delete from old vnode list, if on one.
@@ -549,6 +549,7 @@ reassignbuf(struct buf *bp, struct vnode
(vp->v_mount->mnt_flag & MNT_ASYNC) == 0)
vn_syncer_add_to_worklist(vp, delayx);
}
+ bp->b_flushgen = bufcache_flushgen;
}
bufinsvn(bp, listheadp);
}
diff -r 983d69604667 sys/sys/buf.h
--- sys/sys/buf.h Fri Aug 07 13:18:45 2026 +0200
+++ sys/sys/buf.h Fri Aug 07 18:28:00 2026 +0200
@@ -135,8 +135,15 @@ struct buf {
(partition relative) */
daddr_t b_rawblkno; /* b: raw physical block number
(volume relative) */
- struct proc *b_proc; /* b: proc if BB_PHYS */
- void *b_saveaddr; /* b: saved b_data for physio */
+ union {
+ /* B_PHYS */
+ struct {
+ struct proc *b_proc; /* b: proc if B_PHYS */
+ void *b_saveaddr; /* b: saved b_data for physio */
+ };
+ /* !B_PHYS */
+ uint64_t b_flushgen; /* b: flush generation */
+ };
struct cpu_info *b_ci; /* b: originating CPU */
/*
--ItY42kjenHJt4ZkN--
From: Taylor R Campbell <riastradh@NetBSD.org>
To: "J. Hannken-Illjes" <hannken@NetBSD.org>
Cc: gnats-bugs@NetBSD.org, kern-bug-people@NetBSD.org, gnats-admin@NetBSD.org,
netbsd-bugs@NetBSD.org, dholland@NetBSD.org
Subject: Re: kern/60531: looping "vflushbuf: dirty" message
Date: Fri, 7 Aug 2026 17:29:13 +0000
> Date: Fri, 7 Aug 2026 16:39:33 +0000
> From: "J. Hannken-Illjes" <hannken@netbsd.org>
>
> The FIX will not work, it is not ok to replace 'vp->v_numoutput == 0' with
> 'LIST_EMPTY(&vp->v_dirtyblkhd)'.
>
> Buffers are removed from v_dirtyblkhd through reassignbuf() when bwrite()
> initiates the write while v_numoutput gets decremented through
> biodone()->vwakekup() when the I/O has completed.
>
> With your patch VOP_FSYNC(vp, FSYNC_WAIT) would return before all buffers
> are written to disk.
Thanks, that would be bad!
> The first part should be sufficient to prevent looping here, an updfated
> patch is attached.
> [...]
> + /*
> + * Wait until all pending writes issued before we incremented
> + * bufcache_flushgen have completed.
> + */
> mutex_enter(vp->v_interlock);
> while (vp->v_numoutput != 0)
> cv_wait(&vp->v_cv, vp->v_interlock);
The trouble with waiting for vp->v_numoutput is that if there's
concurrent I/O on a block device or a snapshot, it can keep adding
more writes to v_numoutput so that vflushbuf(9) never completes.
So I don't think this change accomplishes much: we add some more
bookkeeping around flushgen, but we may continue waiting even if we've
long since passed that flushgen.
In fact this change may even create a deadlock, because:
(a) vflushbuf holds the vnode lock while it waits, and
(b) it does not issue bwrite for any buffers past flushgen,
so nothing else can flush those buffers concurrently either with
vflushbuf.
So perhaps we need to wait until:
(a) v_dirtyblkhd has no entries before flushgen,
AND
(b) v_cleanblkhd has no entries before flushgen that are still being
written out.
Not sure exactly how to detect (b). Is a buffercache(9) buf bp still
being written out iff
(bp->b_flags & B_READ) == 0 &&
(bp->b_oflags & BO_DONE) == 0?
I'm not really happy about the prospect of iterating over v_cleanblkhd
but if the buffers in question -- those currently being written out --
aren't on v_dirtyblkhd, I don't see another way to tell when they've
all passed the flushgen short of either iterating over v_cleanblkhd,
or adding a new queue like v_writingblkhd and dealing with all the
consequences of that.
> + KASSERT((vp->v_type == VBLK && spec_node_getmountedfs(vp)) ||
> + LIST_EMPTY(&vp->v_dirtyblkhd));
Won't this assertion also fire if vp is a VREG snapshot, since the
snapshot write path also uses reassignbuf without holding the vnode
lock, as I noted earlier?
https://mail-index.NetBSD.org/netbsd-bugs/2026/08/02/msg094240.html
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/60531 CVS commit: [netbsd-11] src/sys/kern
Date: Mon, 17 Aug 2026 15:47:55 +0000
Module Name: src
Committed By: martin
Date: Mon Aug 17 15:47:55 UTC 2026
Modified Files:
src/sys/kern [netbsd-11]: vfs_subr.c
Log Message:
Pull up following revision(s) (requested by riastradh in ticket #415):
sys/kern/vfs_subr.c: revision 1.504
vflushbuf: Print `vflushbuf: dirty' warning only once per call.
And do it only under DEBUG, and rate-limit it globally.
This can happen when vflushbuf is competing with an onslaught of
concurrent I/O on a snapshot or block device (which is done without
holding the vnode lock, despite what the comment says). That
concurrent I/O might be happening indefinitely, so we are likely to
have to print a _lot_ of warnings, potentially overwhelming the
console, until the underlying problem is fixed by teaching vflushbuf
to wait only for the writes that began before have completed, not
also all writes that have begun since.
This stop-gap measure intended to be low-risk for pullup to release
branches.
PR kern/60531: looping "vflushbuf: dirty" message
To generate a diff of this commit:
cvs rdiff -u -r1.502.2.1 -r1.502.2.2 src/sys/kern/vfs_subr.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/60531 CVS commit: [netbsd-10] src/sys/kern
Date: Mon, 17 Aug 2026 15:49:23 +0000
Module Name: src
Committed By: martin
Date: Mon Aug 17 15:49:23 UTC 2026
Modified Files:
src/sys/kern [netbsd-10]: vfs_subr.c
Log Message:
Pull up following revision(s) (requested by riastradh in ticket #1325):
sys/kern/vfs_subr.c: revision 1.504
vflushbuf: Print `vflushbuf: dirty' warning only once per call.
And do it only under DEBUG, and rate-limit it globally.
This can happen when vflushbuf is competing with an onslaught of
concurrent I/O on a snapshot or block device (which is done without
holding the vnode lock, despite what the comment says). That
concurrent I/O might be happening indefinitely, so we are likely to
have to print a _lot_ of warnings, potentially overwhelming the
console, until the underlying problem is fixed by teaching vflushbuf
to wait only for the writes that began before have completed, not
also all writes that have begun since.
This stop-gap measure intended to be low-risk for pullup to release
branches.
PR kern/60531: looping "vflushbuf: dirty" message
To generate a diff of this commit:
cvs rdiff -u -r1.496.2.1 -r1.496.2.2 src/sys/kern/vfs_subr.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/60531 CVS commit: [netbsd-9] src/sys/kern
Date: Mon, 17 Aug 2026 15:51:04 +0000
Module Name: src
Committed By: martin
Date: Mon Aug 17 15:51:04 UTC 2026
Modified Files:
src/sys/kern [netbsd-9]: vfs_subr.c
Log Message:
Pull up following revision(s) (requested by riastradh in ticket #2045):
sys/kern/vfs_subr.c: revision 1.504
vflushbuf: Print `vflushbuf: dirty' warning only once per call.
And do it only under DEBUG, and rate-limit it globally.
This can happen when vflushbuf is competing with an onslaught of
concurrent I/O on a snapshot or block device (which is done without
holding the vnode lock, despite what the comment says). That
concurrent I/O might be happening indefinitely, so we are likely to
have to print a _lot_ of warnings, potentially overwhelming the
console, until the underlying problem is fixed by teaching vflushbuf
to wait only for the writes that began before have completed, not
also all writes that have begun since.
This stop-gap measure intended to be low-risk for pullup to release
branches.
PR kern/60531: looping "vflushbuf: dirty" message
To generate a diff of this commit:
cvs rdiff -u -r1.471 -r1.471.4.1 src/sys/kern/vfs_subr.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home