NetBSD Problem Report #58964

From www@netbsd.org  Mon Jan  6 15:24:51 2025
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits)
	 client-signature RSA-PSS (2048 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id D50321A9238
	for <gnats-bugs@gnats.NetBSD.org>; Mon,  6 Jan 2025 15:24:51 +0000 (UTC)
Message-Id: <20250106152450.2FF641A923B@mollari.NetBSD.org>
Date: Mon,  6 Jan 2025 15:24:50 +0000 (UTC)
From: campbell+netbsd@mumble.net
Reply-To: campbell+netbsd@mumble.net
To: gnats-bugs@NetBSD.org
Subject: uvm: missing wakeup on uvmexp.free
X-Send-Pr-Version: www-1.0

>Number:         58964
>Category:       kern
>Synopsis:       uvm: missing wakeup on uvmexp.free
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    chs
>State:          needs-pullups
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Jan 06 15:25:06 +0000 2025
>Closed-Date:    
>Last-Modified:  Wed Aug 26 08:35:01 +0000 2026
>Originator:     Taylor R Campbell
>Release:        10.1
>Organization:
The NetBageSleepDaemon Foundation
>Environment:
>Description:
I found my bulk build machine hanging today, with lots of processes waiting on flt_pmfail2, flt_pmfail1, flt_noram5, plpg, tstile, and various zfs condvars.

I entered ddb.  Sampling of output:

db{0}> ps
...
24652 24652 3   0         0   ffff80ace8f2c480                 sh tstile
10978 10978 3   1       180   ffff80abc3a61a00                 sh wait
14020 14020 3   0       180   ffff80aba5153540                 sh wait
8120   8120 3   1         0   ffff80abb5c2b0c0                 sh flt_pmfail2
3599   3599 3   0         0   ffff80ad78b6f540                 sh flt_pmfail1
13915 13915 3   0       180   ffff80abb4992980                 sh pipe_rd
7909   7909 3  10         0   ffff80abb7e9e580          pkg_admin flt_noram5
19573 19573 3  19         0   ffff80b168ecc8c0                 sh flt_noram5
...
0     13053 3  22       200   ffff80b1a9ca3340    zio_write_issue zio_data_buf_983
0     10086 3  23       200   ffff80ac379152c0    zio_write_issue plpg
0     24427 3   8       200   ffff80af3710e040    zio_write_issue zio_data_buf_983
...
0      1630 3  11       200   ffff80b0945d0a80    zio_write_issue zio_data_buf_983
0     16826 3  19       200   ffff80b15d39d740    zio_write_issue plpg
0     17874 3   2       200   ffff80ab83eb81c0    zio_write_issue zio_data_buf_983                                                                
...
0       891 3   2       240   ffff80abb8d30bc0            ioflush tstile
0       890 3  23       200   ffff80abb8d30780           pgdaemon &tx->tx_quiesce_
...
db{0}> show all tstiles
  PID   LID          COMMAND      WAITING-FOR     WAIT-CHANNEL
23365 23365             sync                0 ffff80a95d5dbf80
24652 24652               sh                0 ffff80a95d5dbf80
    0   891           system                0 ffff80aed792bd40

I then woke everything waiting on uvmexp.free and the system began to make progress again:

db{0}> call wakeup(uvmexp+0x10)
7
db{0}> continue

Nothing stuck on flt_pmfail2, flt_pmfail1, flt_noram5, plpg, or tstile, according to `ps -Alww | grep ...'.
>How-To-Repeat:
no idea
>Fix:
Yes, please!

>Release-Note:

>Audit-Trail:

Responsible-Changed-From-To: kern-bug-people->chs
Responsible-Changed-By: riastradh@NetBSD.org
Responsible-Changed-When: Mon, 06 Jan 2025 15:26:14 +0000
Responsible-Changed-Why:
Can I trouble you to take a look?


From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/58964 CVS commit: src/sys
Date: Sun, 23 Aug 2026 22:08:41 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Sun Aug 23 22:08:41 UTC 2026

 Modified Files:
 	src/sys/arch/aarch64/aarch64: pmap.c
 	src/sys/arch/aarch64/include: asan.h
 	src/sys/arch/amd64/amd64: gdt.c
 	src/sys/arch/arm/include: asan.h
 	src/sys/arch/i386/i386: gdt.c
 	src/sys/arch/m68k/m68k: pmap_68k.c pmap_motorola.c
 	src/sys/arch/powerpc/oea: pmap.c
 	src/sys/arch/sparc/sparc: pmap.c
 	src/sys/arch/sparc64/sparc64: pmap.c
 	src/sys/rump/librump/rumpkern: vm.c
 	src/sys/uvm: uvm_amap.c uvm_aobj.c uvm_bio.c uvm_fault.c uvm_glue.c
 	    uvm_km.c uvm_loan.c uvm_object.c uvm_pdaemon.c uvm_pdaemon.h
 	    uvm_pglist.c uvm_vnode.c
 	src/sys/uvm/pmap: pmap_segtab.c

 Log Message:
 uvm: Fix missed wakeups and reduce lock contention a little bit.

 This addresses two problems under heavy load:

 (a) uvm_wait would sometimes miss wakeups, causing various processes
     system to hang but then recover from `call wakeup(uvmexp+0x10)'
     in ddb:

     PR kern/58964: uvm: missing wakeup on uvmexp.free

 (b) Contention on uvmpd_lock would lead to so much time spent at
     IPL_SOFTBIO softint context in uvm_pageout_done spinning for
     uvmpd_lock that it would trip heartbeat panics:

     PR kern/60029: panic: cpu0: softints stuck for 16 seconds

 There are three intertwined parts to this, which I tried to split
 into separate commits, but I eventually decided it wasn't worth the
 trouble:

 1. New ticket idiom for uvm_wait is needed to avoid missing a wakeup
    in the face of multiprocessing or kernel preemption:

 top:	ticket = uvm_wait_prepare();
 	pg = uvm_pagealloc(...);
 	if (pg == NULL) {
 		uvm_wait("foo", ticket);
 		goto top;
 	}

    Without this, a wakeup by the page daemon that arrives after
    uvm_pagealloc has failed and before we have called uvm_wait will
    be lost, so we might hang indefinitely.

    Internally, the ticket is a 64-bit generation number, so it can't
    possibly overflow, that is advanced on any wakeup(&uvmexp.free).
    For LP32 platforms, we manage it with a seqlock/Lamport-type
    algorithm, so that reading it is cheap and unlocked.

    This may also help reduce contention on uvmpd_lock under heavy
    paging load by having uvm_wait skip the lock if there has been
    concurrent paging since uvm_wait_prepare.

 2. Split uvmpd_lock into three different locks for different
    purposes:

    - uvmpd_kick_lock to coordinate threads or interrupt handlers
      wanting the page daemon to work with the page daemon wanting to
      sleep (poor beast, woken by any thread or interrupt handler,
      hence IPL_VM),

    - uvmpd_waiter_lock to coordinate the page daemon thread or paging
      completion soft interrupt handler threads waiting for memory to
      be freed up (thread/softint, hence IPL_SOFTBIO), and

    - uvmpd_pool_drain_lock to coordinate the page daemon thread with
      the pool drain thread (thread-to-thread, hence IPL_NONE).

    This way, anyone trying to kick the page daemon, and the page
    daemon's internal coordination with the pool drainer, won't incur
    any contention on the same lock that uvm_pageout_done is trying to
    take on paging completion to notify threads that they can try
    allocation again.

 3. Use Dekker-synchronized atomic_swap fast paths to reduce time
    spent spinning on these locks when they are under contention.

    This way, if multiple callers (such as uvm_pageout_done and a
    thread trying to allocate memory) want to wake the page daemon,
    only one of them needs to take the lock most of the time.

    And if there's a lot of pageout activity because many threads want
    to allocate memory, uvm_pageout_done need only take the lock once
    to trigger a wakeup until the threads that have been waiting have
    noticed it.

 Additionally, move one call to uvm_availmem out from under any locks,
 to match the other calls to it -- this should help reduce the time
 anyone spends holding the locks, and thus the time everyone else
 spends waiting for it.

 Empirically, this seems to enable a heavily loaded bulk build to
 complete when before the softbio completions would starve softclk
 leading to heartbeat panics.  Even without the heartbeat panics, we
 measured over 8sec running time for softint_dispatch for softbio, and
 extremely heavy contention on uvmpd_lock in uvm_pageout_done and
 uvm_wait.

 This isn't really a solution to the contention, however -- it just
 mitigates the contention.  To scale to more CPUs in parallel, we'll
 probably need to devise a per-CPU or per-NUMA-cluster page daemon
 system, with each parallel daemon splitting up the work in some
 reasonable way that doesn't add more contention.

 Note: Before, all access to uvm_pagedaemon_waiters was serialized by
 uvmpd_lock.  Now, all _writes_ are serialized by uvmpd_lock, but
 _reads_ are allowed unlocked.  So the writes have to be issued with
 atomic_store_*, but there is no need for atomic_r/m/w_*; hence we can
 increment it with

 	atomic_store_relaxed(&uvm_pagedaemon_waiters,
 	    atomic_load_relaxed(&uvm_pagedaemon_waiters) + 1);

 rather than needing the often costlier

 	atomic_inc_uint(&uvm_pagedaemon_waiters);


 To generate a diff of this commit:
 cvs rdiff -u -r1.158 -r1.159 src/sys/arch/aarch64/aarch64/pmap.c
 cvs rdiff -u -r1.19 -r1.20 src/sys/arch/aarch64/include/asan.h
 cvs rdiff -u -r1.48 -r1.49 src/sys/arch/amd64/amd64/gdt.c
 cvs rdiff -u -r1.8 -r1.9 src/sys/arch/arm/include/asan.h
 cvs rdiff -u -r1.74 -r1.75 src/sys/arch/i386/i386/gdt.c
 cvs rdiff -u -r1.78 -r1.79 src/sys/arch/m68k/m68k/pmap_68k.c
 cvs rdiff -u -r1.110 -r1.111 src/sys/arch/m68k/m68k/pmap_motorola.c
 cvs rdiff -u -r1.124 -r1.125 src/sys/arch/powerpc/oea/pmap.c
 cvs rdiff -u -r1.379 -r1.380 src/sys/arch/sparc/sparc/pmap.c
 cvs rdiff -u -r1.320 -r1.321 src/sys/arch/sparc64/sparc64/pmap.c
 cvs rdiff -u -r1.199 -r1.200 src/sys/rump/librump/rumpkern/vm.c
 cvs rdiff -u -r1.129 -r1.130 src/sys/uvm/uvm_amap.c
 cvs rdiff -u -r1.157 -r1.158 src/sys/uvm/uvm_aobj.c
 cvs rdiff -u -r1.128 -r1.129 src/sys/uvm/uvm_bio.c
 cvs rdiff -u -r1.239 -r1.240 src/sys/uvm/uvm_fault.c
 cvs rdiff -u -r1.183 -r1.184 src/sys/uvm/uvm_glue.c
 cvs rdiff -u -r1.168 -r1.169 src/sys/uvm/uvm_km.c
 cvs rdiff -u -r1.104 -r1.105 src/sys/uvm/uvm_loan.c
 cvs rdiff -u -r1.25 -r1.26 src/sys/uvm/uvm_object.c
 cvs rdiff -u -r1.140 -r1.141 src/sys/uvm/uvm_pdaemon.c
 cvs rdiff -u -r1.20 -r1.21 src/sys/uvm/uvm_pdaemon.h
 cvs rdiff -u -r1.92 -r1.93 src/sys/uvm/uvm_pglist.c
 cvs rdiff -u -r1.122 -r1.123 src/sys/uvm/uvm_vnode.c
 cvs rdiff -u -r1.37 -r1.38 src/sys/uvm/pmap/pmap_segtab.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->needs-pullups
State-Changed-By: riastradh@NetBSD.org
State-Changed-When: Sun, 23 Aug 2026 23:16:42 +0000
State-Changed-Why:
Likely fixed in HEAD; let's get some experience with it for a few
weeks before requesting pullup-11.  Might be worth adapting to 10,
probably not to 9.


From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/58964 CVS commit: src/sys/rump/librump/rumpkern
Date: Sun, 23 Aug 2026 23:56:54 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Sun Aug 23 23:56:54 UTC 2026

 Modified Files:
 	src/sys/rump/librump/rumpkern: vm.c

 Log Message:
 rump uvm: Allow uvm_wait_prepare before uvm_init.

 Probably not worthwhile to figure out how to restructure rump
 initialization.

 Fixes total failure of rump after changes for:

 PR kern/58964: uvm: missing wakeup on uvmexp.free
 PR kern/60029: panic: cpu0: softints stuck for 16 seconds


 To generate a diff of this commit:
 cvs rdiff -u -r1.200 -r1.201 src/sys/rump/librump/rumpkern/vm.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/58964 CVS commit: src/sys/rump/librump/rumpkern
Date: Mon, 24 Aug 2026 20:31:32 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Mon Aug 24 20:31:32 UTC 2026

 Modified Files:
 	src/sys/rump/librump/rumpkern: vm.c

 Log Message:
 rump_hypermalloc: Don't get a uvm_wait ticket if we won't uvm_wait.

 This path may be taken with a spin lock held, which means we can't
 take pdaemonmtx (IPL_NONE).

 PR kern/60636: rump on sparc64 totally broken still with vm.c rev 1.201

 Followup for:

 PR kern/58964: uvm: missing wakeup on uvmexp.free
 PR kern/60029: panic: cpu0: softints stuck for 16 seconds


 To generate a diff of this commit:
 cvs rdiff -u -r1.201 -r1.202 src/sys/rump/librump/rumpkern/vm.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/58964 CVS commit: src/sys/rump/librump/rumpkern
Date: Mon, 24 Aug 2026 20:47:01 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Mon Aug 24 20:47:01 UTC 2026

 Modified Files:
 	src/sys/rump/librump/rumpkern: vm.c

 Log Message:
 rump: Handle uvm_wait tickets like the real kernel does.

 This should avoid trouble in any path that calls uvm_wait_prepare
 with spin locks held even though it's not going to sleep in uvm_wait,
 of which there are probably many more than just rump_hypermalloc --
 uvm_wait_prepare is supposed to be cheap enough not to be worth
 conditionalizing.  (Perhaps we should measure that, though.)

 PR kern/60636: rump on sparc64 totally broken still with vm.c rev 1.201

 Followup for:

 PR kern/58964: uvm: missing wakeup on uvmexp.free
 PR kern/60029: panic: cpu0: softints stuck for 16 seconds


 To generate a diff of this commit:
 cvs rdiff -u -r1.202 -r1.203 src/sys/rump/librump/rumpkern/vm.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Juergen Hannken-Illjes" <hannken@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/58964 CVS commit: src/sys/uvm
Date: Wed, 26 Aug 2026 08:31:42 +0000

 Module Name:	src
 Committed By:	hannken
 Date:		Wed Aug 26 08:31:42 UTC 2026

 Modified Files:
 	src/sys/uvm: uvm_fault.c

 Log Message:
 Kernels ALL/i386 and ALL/amd64 warn "'ticket' may be used uninitialized"
 at line 244, uvmfault_amapcopy(): "uvm_wait("fltamapcopy", ticket);".

 Restructure to reflect the logic better and keep GCC quiet.

 Ok: Taylor R Campbell

 PR kern/58964: uvm: missing wakeup on uvmexp.free
 PR kern/60029: panic: cpu0: softints stuck for 16 seconds


 To generate a diff of this commit:
 cvs rdiff -u -r1.240 -r1.241 src/sys/uvm/uvm_fault.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

>Unformatted:

Home	PR Database Search