NetBSD Problem Report #60584

From tsutsui@ceres.dti.ne.jp  Fri Aug 14 05:27:11 2026
Return-Path: <tsutsui@ceres.dti.ne.jp>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits)
	 client-signature RSA-PSS (2048 bits))
	(Client CN "mail.netbsd.org", Issuer "YR2" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 59FEC1A923B
	for <gnats-bugs@gnats.NetBSD.org>; Fri, 14 Aug 2026 05:27:11 +0000 (UTC)
Message-Id: <202608140527.67E5R5SJ012727@ceres.dti.ne.jp>
Date: Fri, 14 Aug 2026 14:27:05 +0900 (JST)
From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
Reply-To: tsutsui@ceres.dti.ne.jp
To: gnats-bugs@NetBSD.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: vfs_bio: getnewbuf() can deadlock when fresh allocation is refused and no buffer is reclaimable
X-Send-Pr-Version: 3.95
X-From4GNATS: "Izumi Tsutsui via gnats" <gnats-admin@NetBSD.org>

>Number:         60584
>Category:       kern
>Synopsis:       vfs_bio: getnewbuf() can deadlock when fresh allocation is refused and no buffer is reclaimable
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    tsutsui
>State:          needs-pullups
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Aug 14 05:30:01 +0000 2026
>Closed-Date:    
>Last-Modified:  Thu Oct 01 14:25:01 +0000 2026
>Originator:     Izumi Tsutsui
>Release:        NetBSD 11.99.7
>Organization:
>Environment:
NetBSD libero 11.99.7 NetBSD 11.99.7 (GENERIC) #24: Thu Aug 13 09:21:55 JST 2026  tsutsui@mirage:/s/cvs/src/sys/arch/news68k/compile/GENERIC news68k
Architecture: m68k
Machine: news68k, Sony NWS-1750 (MC68030 16MB RAM, 8KB/page)

>Description:

getnewbuf() can deadlock when buf_lotsfree() decides not to allocate
a new buffer and there is no buffer which can actually be reclaimed
from BQ_AGE or BQ_LRU.

getnewbuf() first uses buf_lotsfree() to decide whether to try a fresh
allocation.  If fresh allocation is not attempted or fails, it tries
to recycle a buffer from BQ_AGE or BQ_LRU.  If no suitable buffer is
available there, it sleeps on needbuffer_cv.

In this case, there is no guarantee that a reclaimable buffer will
eventually become available.  Buffers may instead be busy or locked,
for example by WAPBL.  A thread which could release such a buffer
may itself need another buffer before it can make progress.

Then a buffer allocation policy decision can turn into a deadlock.
getnewbuf() waits for a buffer to become available while the operation
which could make one available is itself waiting for another buffer.

I observed this on NetBSD/news68k on 16MB NWS-1750 under heavy memory
pressure with WAPBL enabled.  makemandb and the pagedaemon were both
sleeping in getnewbuf() on needbuffer_cv, while multiple user
processes were waiting for physical memory.

As a diagnostic experiment in DDB, I temporarily raised
bufmem_lowater so that buf_lotsfree() would permit fresh allocation,
woke the needbuffer_cv waiters, and continued execution.

After that, the machine immediately resumed normal operation.
The pagedaemon started making progress again, free memory increased,
and the affected user processes continued running.

This appears to demonstrate that getnewbuf() has no forward-progress
path when fresh allocation is refused and no buffer can actually be
reclaimed.

I have also previously observed system hangs after decreasing
vm.bufcache at runtime on low-memory machines.  The vm.bufcache sysctl
handler changes the buffer cache watermarks and attempts to drain the
cache, but it can return successfully even if the cache cannot be
drained below the new high-water mark.

This may provide an easier way to trigger the same problem, but
I have not yet verified that the older vm.bufcache hang is caused
by this same bug.

>How-To-Repeat:

The exact workload required to reproduce the problem has not been
minimized.

The problem can occur when buf_lotsfree() rejects fresh allocation,
no suitable buffer is available from BQ_AGE or BQ_LRU, and releasing
the buffers needed for forward progress requires another buffer
allocation.

>Fix:

Yes, please.

getnewbuf() appears to need a forward-progress mechanism for the case
where fresh allocation is refused but no buffer can actually be reclaimed.

The appropriate fix might be to reconsider fresh allocation after
reclamation fails, change the allocation policy, reserve resources
needed for forward progress, or change the handling of locked buffers.

I am not sure which approach is appropriate, and would prefer to leave
the decision to people more familiar with the buffer cache and UVM.

---
Izumi Tsutsui

>Release-Note:

>Audit-Trail:
From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@netbsd.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: Re: kern/60584: vfs_bio: getnewbuf() can deadlock when fresh allocation
	 is refused and no buffer is reclaimable
Date: Fri, 21 Aug 2026 04:42:09 +0900

 I wrote:

 > >Number:         60584
 > >Category:       kern
 > >Synopsis:       vfs_bio: getnewbuf() can deadlock when fresh allocation is refused and no buffer is reclaimable

 I looked further into the history of buf_lotsfree().

 The probabilistic allocation policy was introduced in vfs_bio.c
 r1.114 to control buffer cache growth and avoid thrashing between
 the high and low water marks.

  https://github.com/NetBSD/src/commit/3e8ad532aa3cba09a96fa5ddfa876318a5133107

 The next r1.115 made BQ_AGE take precedence over fresh allocation,

  https://github.com/NetBSD/src/commit/249b0cda71b6c885c87eee8f0e7d3ad9ca203ef1

 and r1.136 adjusted the probability curve to start at the low water mark.

  https://github.com/NetBSD/src/commit/249b0cda71b6c885c87eee8f0e7d3ad9ca203ef1


 >From these changes, my understanding is that buf_lotsfree() is mainly
 intended to control buffer cache growth.  A false return seems to mean
 that recycling an existing buffer is preferable to growing the cache.

 The current getnewbuf() path can be summarized roughly as follows:

 ```
         if (!from_bufq && buf_lotsfree()) {
                 /*
                  * Try a fresh allocation according to the existing
                  * probabilistic growth policy.
                  */
                 ...
         }

         /*
          * Look for a buffer which can actually be recycled from
          * BQ_AGE or BQ_LRU.
          */
         ...

         if (bp != NULL) {
                 /*
                  * Recycle the buffer as before.
                  */
                 ...
         } else {
                 /*
                  * At this point getnewbuf() has not only preferred
                  * recycling as a policy decision, but has also found
                  * that there is in fact no buffer which can be recycled.
                  */

                 /*
                  * Sleep on needbuffer_cv,
                  * but no forward path to wake up this in case of this PR
                  */
                 ...
         }
 ```

 The problem reported in this PR seems to occur in the last
 "Sleep on needbuffer_cv" case as noted above.

 There is no reusable buffer, but getnewbuf() still waits for one to
 become available.  If releasing such a buffer itself requires another
 buffer allocation, there is no way to make progress.

 I think one possible way to avoid this would be to add another
 PR_NOWAIT allocation attempt before sleeping, while bufmem is still
 below bufmem_hiwater.

 The resulting path would look roughly like this:

 ```
         if (!from_bufq && buf_lotsfree()) {
                 /*
                  * Existing probabilistic fresh allocation.
                  */
                 ...
         }

         /*
          * Existing BQ_AGE/BQ_LRU scan.
          */
         ...

         if (bp != NULL) {
                 /*
                  * Existing recycle path.
                  */
                 ...
         } else {
 -->             /*
 -->              * No buffer can actually be recycled.
 -->              */
 -->             if (!from_bufq && bufmem < bufmem_hiwater) {
 -->                     /*
 -->                      * Try one fresh allocation for forward
 -->                      * progress.
 -->                      */
 -->                     ...
 -->             }

                 /*
                  * Sleep on needbuffer_cv only if that also fails.
                  */
                 ...
         }
 ```

 This would leave the existing buf_lotsfree() policy in place for the
 normal case.  The additional allocation would only be attempted after
 getnewbuf() has actually found that recycling is not possible, and it
 would not allow the cache to grow past bufmem_hiwater.

 I have not tested this change yet, and I may be missing some reason
 why a fresh allocation should not be attempted at this point.

 It would also not address the case where bufmem is already at or
 above the high water mark.

 Comments from people more familiar with the buffer cache and UVM
 would be appreciated.

 Thanks,
 ---
 Izumi Tsutsui

From: "Izumi Tsutsui" <tsutsui@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/60584 CVS commit: src/sys/kern
Date: Mon, 21 Sep 2026 04:24:36 +0000

 Module Name:	src
 Committed By:	tsutsui
 Date:		Mon Sep 21 04:24:36 UTC 2026

 Modified Files:
 	src/sys/kern: vfs_bio.c

 Log Message:
 vfs_bio: retry fresh buffer allocation if recycling fails

 getnewbuf() uses buf_lotsfree() to choose whether to
 allocate a fresh buffer before trying BQ_AGE and BQ_LRU.

 If fresh allocation is skipped and no buffer can actually be
 recycled, it sleeps on needbuffer_cv even when the buffer cache
 is still below its high water mark.

 In that situation, a thread which could release a buffer may
 itself need another buffer, resulting in a deadlock.

 To avoid this, if recycling finds no usable buffer and bufmem is
 still below bufmem_hiwater, make one more PR_NOWAIT fresh allocation
 attempt before sleeping.  See PR kern/60584 for details.

 Worth to pullup to netbsd-11.


 To generate a diff of this commit:
 cvs rdiff -u -r1.308 -r1.309 src/sys/kern/vfs_bio.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

Responsible-Changed-From-To: kern-bug-people->tsutsui
Responsible-Changed-By: tsutsui@NetBSD.org
Responsible-Changed-When: Mon, 21 Sep 2026 07:22:46 +0000
Responsible-Changed-Why:
Committed by me


State-Changed-From-To: open->needs-pullups
State-Changed-By: tsutsui@NetBSD.org
State-Changed-When: Mon, 21 Sep 2026 07:22:46 +0000
State-Changed-Why:
netbsd-11 is also affected


From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/60584 CVS commit: [netbsd-11] src/sys/kern
Date: Thu, 1 Oct 2026 14:22:46 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Thu Oct  1 14:22:46 UTC 2026

 Modified Files:
 	src/sys/kern [netbsd-11]: vfs_bio.c

 Log Message:
 Pull up following revision(s) (requested by tsutsui in ticket #493):

 	sys/kern/vfs_bio.c: revision 1.309

 vfs_bio: retry fresh buffer allocation if recycling fails

 getnewbuf() uses buf_lotsfree() to choose whether to
 allocate a fresh buffer before trying BQ_AGE and BQ_LRU.

 If fresh allocation is skipped and no buffer can actually be
 recycled, it sleeps on needbuffer_cv even when the buffer cache
 is still below its high water mark.

 In that situation, a thread which could release a buffer may
 itself need another buffer, resulting in a deadlock.

 To avoid this, if recycling finds no usable buffer and bufmem is
 still below bufmem_hiwater, make one more PR_NOWAIT fresh allocation
 attempt before sleeping.  See PR kern/60584 for details.


 To generate a diff of this commit:
 cvs rdiff -u -r1.306 -r1.306.2.1 src/sys/kern/vfs_bio.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

>Unformatted:

Home	PR Database Search