NetBSD Problem Report #59885

From www@netbsd.org  Sat Jan  3 21:45:26 2026
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits)
	 client-signature RSA-PSS (2048 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id E70BA1A923F
	for <gnats-bugs@gnats.NetBSD.org>; Sat,  3 Jan 2026 21:45:25 +0000 (UTC)
Message-Id: <20260103214524.50D311A9241@mollari.NetBSD.org>
Date: Sat,  3 Jan 2026 21:45:24 +0000 (UTC)
From: campbell+netbsd@mumble.net
Reply-To: campbell+netbsd@mumble.net
To: gnats-bugs@NetBSD.org
Subject: zfs: unlink/rm is slow to delete last link because it always zil_commits
X-Send-Pr-Version: www-1.0

>Number:         59885
>Category:       kern
>Synopsis:       zfs: unlink/rm is slow to delete last link because it always zil_commits
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          needs-pullups
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Jan 03 21:50:00 +0000 2026
>Closed-Date:    
>Last-Modified:  Sun Mar 22 13:20:01 +0000 2026
>Originator:     Taylor R Campbell
>Release:        current, 11, 10, 9, ...
>Organization:
The NetZFS Slowdowndation, Inc.
>Environment:
>Description:

	We have a local change to zfs to commit the zil whenever a zfs
	vnode is reclaimed, which usually happens when deleting the
	last link to a file:

	/*
	 * Operation zfs_znode.c::zfs_zget_cleaner() depends on this
	 * zil_commit() as a barrier to guarantee the znode cannot
	 * get freed before its log entries are resolved.
	 */
	if (zfsvfs->z_log)
		zil_commit(zfsvfs->z_log, zp->z_id);

	This is required because the logic to write data for file the
	to the log (zfs_get_data), queued up by past writes to the
	file, relies on acquiring a reference to the vnode as indexed
	by the zfs object id (equivalent of inode number) and using the
	struct znode before it is freed.

	That logic might run after the vnode has gone through
	VOP_RECLAIM, but NetBSD's vnode life cycle treats reclamation
	as final and forbids acquiring new references or even looking
	up the vnode by its object id via vcache(9), and
	zfs_netbsd_reclaim unconditionally frees the struct znode with
	zfs_znode_free immediately afterward (all paths out of
	zfs_zinactive go, either directly or via zfs_rmnode and
	sometimes then via zfs_znode_delete, through zfs_znode_free):

	if (zp->z_sa_hdl == NULL)
		zfs_znode_free(zp);
	else
		zfs_zinactive(zp);

	Committing the zil first avoids this trouble.  But committing
	the zil is costly (requires writing all pending transactions to
	disk and flushing the disk cache and updating root pointers and
	so on), much costlier than just logging a file operation like
	unlink.

	And removing the last link to a file causes its vnode to be
	reclaimed synchronously, essentially every rm(1) or equivalent
	(in the absence of multiple hard links to a file) triggers this
	logic, making it very slow.

>How-To-Repeat:

	rm -rf /large/directory/tree

>Fix:

	It was a huge improvement to the reliability and
	maintainability of NetBSD's vnode life cycle that we began to
	forbid reviving vnodes from the dead about a decade ago, so
	reversing that decision is a non-starter.

	But perhaps we can add a reference count to the znode itself,
	when it is pending in a log transaction for zfs_get_data later,
	so that it is only freed after both zfs_netbsd_reclaim _and_
	zfs_get_data are done with it.  Note that the only use that
	zfs_get_data makes of the _vnode_ is to release a reference
	(which we have made into a no-op because it is taking that
	`reference' only during reclamation when acquiring new vnode
	references is forbidden).

	This will also require making sure the object id is not
	recycled too early -- possibly by some combination of
	ZFS_OBJ_HOLD_ENTER and ZFS_TEARDOWN_INACTIVE_ENTER/EXIT_READ,
	not sure.

>Release-Note:

>Audit-Trail:
From: "YAMAMOTO Takashi" <yamt@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/59885 CVS commit: src/sys/kern
Date: Sun, 22 Mar 2026 13:03:26 +0000

 Module Name:	src
 Committed By:	yamt
 Date:		Sun Mar 22 13:03:26 UTC 2026

 Modified Files:
 	src/sys/kern: vfs_mount.c

 Log Message:
 dounmount/vfs_insmntque: allow vcache_get during VFS_UNMOUNT

 we currently have assertions to prevent file systems from
 populating its vnode cache during VFS_UNMOUNT. this commit
 relaxes the assertions a bit to allow vcache_get during
 VFS_UNMOUNT. although VFS_UNMOUNT should still eventually
 drain the vnode cache for the mount, this commit allows it
 to populate its vnode cache temporarily.

 this is for zfs, which sometimes need to access znode when
 committing the log. (zfs_get_data)  a forthcoming zfs change
 will depend on this change.

 PR/59885
 https://gnats.netbsd.org/59885
 discussed on tech-kern.
 https://mail-index.netbsd.org/tech-kern/2026/02/20/msg030817.html


 To generate a diff of this commit:
 cvs rdiff -u -r1.110 -r1.111 src/sys/kern/vfs_mount.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->needs-pullups
State-Changed-By: yamt@NetBSD.org
State-Changed-When: Sun, 22 Mar 2026 13:15:19 +0000
State-Changed-Why:
pullup-11. i haven't checked earlier branches.


From: "YAMAMOTO Takashi" <yamt@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/59885 CVS commit: src/external/cddl/osnet/dist/uts/common/fs/zfs
Date: Sun, 22 Mar 2026 13:10:57 +0000

 Module Name:	src
 Committed By:	yamt
 Date:		Sun Mar 22 13:10:57 UTC 2026

 Modified Files:
 	src/external/cddl/osnet/dist/uts/common/fs/zfs: zfs_vfsops.c
 	    zfs_vnops.c zfs_znode.c
 	src/external/cddl/osnet/dist/uts/common/fs/zfs/sys: zfs_znode.h

 Log Message:
 zfs: fix "slow rm" issue

 * stop commiting zil in zfs_netbsd_reclaim and other operations
   in vnode reclaim path.

 * retire zfs_zget_cleaner/VN_RELE_CLEANER.
   instead, just use normal zfs_zget and vrele_async.

 note that these two changes depend on each other:

 * zfs_zget_cleaner relies on zil_commit in zfs_netbsd_reclaim to
   ensure that the znode referenced by TX_WRITE itx is always in-core.

 * otoh, zfs_zget_clear makes zil_commit in the vnode reclaim path
   possible. that is, zfs_netbsd_reclaim (VOP_RECLAIM) is called with
   the vnode in VS_RECLAIMING state, which would make vcache_vget
   block.
   if the vnode being reclaimed happened to have TX_WRITE itx on the
   zil, it deadlocks.

 an alternative would be to make the upper layer (vfs_vnode.c) retain
 unlinked vnodes for a while. (a bit longer than the 5 sec txg commit
 interval should be enough.) eg. by making zfs_netbsd_inactive report
 a_recycle = 0. but i guess it's better to remove
 zfs_zget_cleaner/VN_RELE_CLEANER to to keep the code less diverged
 from the upstream zfs.

 also, this commit makes zfs_umount retry vflush a bit.
 it's necessary because, for some reasons, during unmount, zil_close
 commits the log, which can load some referenced vnodes back to the
 cache. i don't understand why zil_close needs to commit the log
 when we are syncing txg for unmount anyway. although it might be
 possible to avoid the zil commit at all, probably this change
 is less invasive than that. this logic is partly from J. Hannken-Illjes.

 PR/59885
 https://gnats.netbsd.org/59885
 discussed on tech-kern.
 https://mail-index.netbsd.org/tech-kern/2026/02/20/msg030817.html


 To generate a diff of this commit:
 cvs rdiff -u -r1.32 -r1.33 \
     src/external/cddl/osnet/dist/uts/common/fs/zfs/zfs_vfsops.c
 cvs rdiff -u -r1.91 -r1.92 \
     src/external/cddl/osnet/dist/uts/common/fs/zfs/zfs_vnops.c
 cvs rdiff -u -r1.35 -r1.36 \
     src/external/cddl/osnet/dist/uts/common/fs/zfs/zfs_znode.c
 cvs rdiff -u -r1.9 -r1.10 \
     src/external/cddl/osnet/dist/uts/common/fs/zfs/sys/zfs_znode.h

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

>Unformatted:

Home	PR Database Search