NetBSD Problem Report #56764
From www@netbsd.org Mon Mar 21 09:34:45 2022
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 88FE61A921F
for <gnats-bugs@gnats.NetBSD.org>; Mon, 21 Mar 2022 09:34:45 +0000 (UTC)
Message-Id: <20220321093444.096081A9239@mollari.NetBSD.org>
Date: Mon, 21 Mar 2022 09:34:43 +0000 (UTC)
From: prlw1@cam.ac.uk
Reply-To: prlw1@cam.ac.uk
To: gnats-bugs@NetBSD.org
Subject: panic assertion "uvmexp.swpgonly > 0"
X-Send-Pr-Version: www-1.0
>Number: 56764
>Category: kern
>Synopsis: panic assertion "uvmexp.swpgonly > 0"
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: kern-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Mon Mar 21 09:35:00 +0000 2022
>Last-Modified: Fri Apr 03 12:25:01 +0000 2026
>Originator: Patrick Welche
>Release: NetBSD-9.99.95/amd64 20 Mar 2022
>Organization:
>Environment:
>Description:
During "shutdown -p now" of a pbulk run that was swapping heavily (has
something changed with libreoffice?), the following panic happened
(leaving disks needing fsck -y)
(gdb) print panicstr
$1 = 0xffffffff8151cc80 <scratchstr> "kernel diagnostic assertion \"uvmexp.swpgonly > 0\" failed: file \"../../../../uvm/uvm_swap.c\", line 1830 "
(gdb) bt
#0 0xffffffff802229c5 in cpu_reboot (howto=howto@entry=260,
bootstr=bootstr@entry=0x0) at ../../../../arch/amd64/amd64/machdep.c:720
#1 0xffffffff80aca704 in kern_reboot (howto=howto@entry=260,
bootstr=bootstr@entry=0x0) at ../../../../kern/kern_reboot.c:73
#2 0xffffffff80b0db72 in vpanic (
fmt=0xffffffff80fce6f0 "kernel %sassertion \"%s\" failed: file \"%s\", line %d ", ap=ap@entry=0xffffab04b0be4ad8) at ../../../../kern/subr_prf.c:290
#3 0xffffffff80c81c8f in kern_assert (
fmt=fmt@entry=0xffffffff80fce6f0 "kernel %sassertion \"%s\" failed: file \"%s\", line %d ") at ../../../../../../lib/libkern/kern_assert.c:51
#4 0xffffffff80a8793d in uvm_swap_get (page=<optimized out>,
page@entry=0xffffab0020000d00, swslot=<optimized out>, flags=flags@entry=2)
at ../../../../uvm/uvm_swap.c:1830
#5 0xffffffff80a6798e in uvmfault_anonget (ufi=ufi@entry=0x0,
amap=amap@entry=0xffff9665d0e40198, anon=anon@entry=0xffff9665db0cdad8)
at ../../../../uvm/uvm_fault.c:392
#6 0xffffffff80a61bb8 in uvm_anon_pagein (amap=amap@entry=0xffff9665d0e40198,
anon=0xffff9665db0cdad8) at ../../../../uvm/uvm_anon.c:287
#7 0xffffffff80a60ad2 in amap_swap_off (startslot=1, endslot=122095319)
at ../../../../uvm/uvm_amap.c:1402
#8 0xffffffff80a8513b in swap_off (l=l@entry=0xffff9665e3dde480,
sdp=0xffff9665d12ed100) at ../../../../uvm/uvm_swap.c:1082
#9 0xffffffff80a86af7 in sys_swapctl (l=0xffff9665e3dde480,
uap=0xffffab04b0be4f00, retval=<optimized out>)
at ../../../../uvm/uvm_swap.c:748
#10 0xffffffff803fd09e in sy_call (rval=0xffffab04b0be4eb0,
uap=0xffffab04b0be4f00, l=0xffff9665e3dde480,
sy=0xffffffff814639e8 <sysent+6504>) at ../../../../sys/syscallvar.h:65
#11 sy_invoke (code=271, rval=0xffffab04b0be4eb0, uap=0xffffab04b0be4f00,
l=0xffff9665e3dde480, sy=0xffffffff814639e8 <sysent+6504>)
at ../../../../sys/syscallvar.h:94
#12 syscall (frame=0xffffab04b0be4f00)
at ../../../../arch/x86/x86/syscall.c:138
#13 0xffffffff8020867d in handle_syscall ()
#14 0x0000000000000002 in ?? ()
#15 0x00007f7fffffd7f0 in ?? ()
#16 0x0000000000000000 in ?? ()
(gdb) print uvmexp
$5 = {pagesize = 4096, pagemask = 4095, pageshift = 12, npages = 16245534,
free = 5948528, paging = 0, wired = 4978, ncolors = 16, colormask = 15,
zeropages = 0, reserve_pagedaemon = 1, reserve_kernel = 80,
anonpages = 12304, filepages = 6062858, execpages = 16584, freemin = 4096,
freetarg = 5461, wiredmax = 5415178, nswapdev = 1, swpages = 122095317,
swpgavail = 0, swpginuse = 2, swpgonly = 0, nswget = 1903783,
faults = 1647202138, traps = 1648322817, intrs = 4570866, swtch = 73563035,
softs = 8174162, syscalls = -1962293868, pageins = 68644, _unused1 = 0,
_unused2 = 0, pgswapin = 0, pgswapout = 5382281, forks = 2573596,
forks_ppwait = 1630951, forks_sharevm = 1630951, pga_zerohit = 0,
pga_zeromiss = 0, zeroaborts = 0, colorhit = 1333019305,
colormiss = 139555180, cpuhit = 1156296487, cpumiss = 316277998,
fltnoram = 11842172, fltnoanon = 0, fltpgwait = 53, fltpgrele = 0,
fltrelck = 452245, fltrelckok = 452073, fltanget = 59468194,
fltanretry = 68735, fltamcopy = 157083835, fltnamap = 55460761,
fltnomap = 1168414334, fltlget = 326455861, fltget = 415713,
flt_anon = 35398387, flt_acow = 24037224, flt_obj = 245215384,
flt_prcopy = 81224287, flt_przero = 1245736078, pdwoke = 3499331,
pdrevs = 1612, _unused3 = 0, pdfreed = 31670364, pdscans = 62863648,
pdanscan = 229129, pdobscan = 31441586, pdreact = 35679, pdbusy = 0,
pdpageouts = 337035, pdpending = 5045166, pddeact = 66951433, pdreanon = 0,
pdrefile = 0, pdreexec = 0, bootpages = 507705}
>How-To-Repeat:
>Fix:
>Audit-Trail:
From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@netbsd.org
Cc: Chuck Silvers <chuq@chuq.com>
Subject: Re: kern/56764: panic assertion "uvmexp.swpgonly > 0"
Date: Mon, 25 Aug 2025 13:55:38 +0200
I haven't seen this particular panic yet, but I have problems with
this counter since at least 2017.
My latest one was
panic: kernel diagnostic assertion "uvmexp.swpgonly < uvmexp.swpginuse" failed: file "/usr/src/sys/uvm/uvm_pdaemon.c", line 795
The backtrace is
(gdb) bt
#0 0xffffffff80239b85 in cpu_reboot (howto=howto@entry=260, bootstr=bootstr@entry=0x0) at /usr/src/sys/arch/amd64/amd64/machdep.c:709
#1 0xffffffff80de5a95 in kern_reboot (howto=howto@entry=260, bootstr=bootstr@entry=0x0) at /usr/src/sys/kern/kern_reboot.c:91
#2 0xffffffff80e2ba63 in vpanic (fmt=0xffffffff81427c00 "kernel %sassertion \"%s\" failed: file \"%s\", line %d ", ap=ap@entry=0xffffc1a47baa2e98) at /usr/src/sys/kern/subr_prf.c:288
#3 0xffffffff80ffb6fe in kern_assert (fmt=fmt@entry=0xffffffff81427c00 "kernel %sassertion \"%s\" failed: file \"%s\", line %d ") at /usr/src/sys/lib/libkern/kern_assert.c:51
#4 0xffffffff80d9532c in uvmpd_scan_queue () at /usr/src/sys/uvm/uvm_pdaemon.c:795
#5 uvmpd_scan () at /usr/src/sys/uvm/uvm_pdaemon.c:917
#6 uvm_pageout (arg=<optimized out>) at /usr/src/sys/uvm/uvm_pdaemon.c:318
#7 0xffffffff80210327 in lwp_trampoline ()
#8 0x0000000000000000 in ?? ()
Chuck Silvers took a number of looks at this over the years, and his
best guess is that the counters are not maintained correctly, but we
haven't found the root cause of this.
Thomas
From: "Thomas Klausner" <wiz@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/56764 CVS commit: src/sys/uvm
Date: Sun, 4 Jan 2026 00:41:15 +0000
Module Name: src
Committed By: wiz
Date: Sun Jan 4 00:41:14 UTC 2026
Modified Files:
src/sys/uvm: uvm_pdaemon.c uvm_swap.c
Log Message:
Add more debugging.
To help understand PR 56764 better.
Ok riastradh@
To generate a diff of this commit:
cvs rdiff -u -r1.134 -r1.135 src/sys/uvm/uvm_pdaemon.c
cvs rdiff -u -r1.209 -r1.210 src/sys/uvm/uvm_swap.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Thomas Klausner <wiz@netbsd.org>
To: NetBSD bugtracking <gnats-bugs@netbsd.org>
Cc:
Subject: Re: kern/56764
Date: Tue, 10 Feb 2026 07:51:00 +0100
The newly added assertions have been triggered today.
[ 56284.8616186] panic: kernel diagnostic assertion "uvmexp.swpgonly < uvmexp.swpginuse" failed: file "/usr/src/sys/uvm/uvm_pdaemon.c", line 795 swpgonly 18576318 swpginuse 18570395
Kernel backtrace:
(gdb) target kvm netbsd.25.core
0xffffffff80239eca in cpu_reboot (howto=howto@entry=260, bootstr=bootstr@entry=0x0) at /usr/src/sys/arch/amd64/amd64/machdep.c:709
709 dumpsys();
(gdb) bt
#0 0xffffffff80239eca in cpu_reboot (howto=howto@entry=260, bootstr=bootstr@entry=0x0) at /usr/src/sys/arch/amd64/amd64/machdep.c:709
#1 0xffffffff80e2be7f in kern_reboot (howto=howto@entry=260, bootstr=bootstr@entry=0x0) at /usr/src/sys/kern/kern_reboot.c:91
#2 0xffffffff80e77155 in vpanic (fmt=0xffffffff816ee020 "kernel %sassertion \"%s\" failed: file \"%s\", line %d swpgonly %d swpginuse %d", ap=ap@entry=0xffffad247baabe98) at /usr/src/sys/kern/subr_prf.c:288
#3 0xffffffff81051bbe in kern_assert (fmt=fmt@entry=0xffffffff816ee020 "kernel %sassertion \"%s\" failed: file \"%s\", line %d swpgonly %d swpginuse %d") at /usr/src/sys/lib/libkern/kern_assert.c:51
#4 0xffffffff80dd86a3 in uvmpd_scan_queue () at /usr/src/sys/uvm/uvm_pdaemon.c:795
#5 uvmpd_scan () at /usr/src/sys/uvm/uvm_pdaemon.c:919
#6 uvm_pageout (arg=<optimized out>) at /usr/src/sys/uvm/uvm_pdaemon.c:318
#7 0xffffffff80210327 in lwp_trampoline ()
#8 0x0000000000000000 in ?? ()
(gdb) fr 4
#4 0xffffffff80dd86a3 in uvmpd_scan_queue () at /usr/src/sys/uvm/uvm_pdaemon.c:795
795 KASSERTMSG(uvmexp.swpgonly < uvmexp.swpginuse,
(gdb) l
790 } else {
791 slot = uao_find_swslot(uobj, pageidx);
792 }
793 if (slot > 0) {
794 /* this page is now only in swap. */
795 KASSERTMSG(uvmexp.swpgonly < uvmexp.swpginuse,
796 "swpgonly %d swpginuse %d",
797 uvmexp.swpgonly, uvmexp.swpginuse);
798 atomic_inc_uint(&uvmexp.swpgonly);
799 }
(gdb) p slot
$1 = <optimized out>
(gdb) p uvmexp
$2 = {pagesize = 4096, pagemask = 4095, pageshift = 12, npages = 32315696, free = 792, paging = 0, wired = 4979, ncolors = 32, colormask = 31, zeropages = 0, reserve_pagedaemon = 1, reserve_kernel = 160,
anonpages = 22220503, filepages = 5515267, execpages = 96251, freemin = 8192, freetarg = 10922, wiredmax = 10771898, nswapdev = 3, swpages = 133993646, swpgavail = 133993646, swpginuse = 18418907,
swpgonly = 18576494, nswget = 839794, faults = 431457751, traps = 444959363, intrs = 123789873, swtch = 801648567, softs = 196734403, syscalls = 1279015017, pageins = 317835, _unused1 = 0, _unused2 = 0, pgswapin = 0,
pgswapout = 57374711, forks = 2716424, forks_ppwait = 1178253, forks_sharevm = 1178253, pga_zerohit = 0, pga_zeromiss = 0, zeroaborts = 0, colorhit = -1069245636, colormiss = 155300626, cpuhit = -2103165643,
cpumiss = 1189220633, fltnoram = 125291247, fltnoanon = 0, fltpgwait = 2919, fltpgrele = 0, fltrelck = 1752063358, fltrelckok = 1752006885, fltanget = 157777663, fltanretry = 386737, fltamcopy = 481707478,
fltnamap = 144737577, fltnomap = 978014010, fltlget = -2138418117, fltget = 1751675474, flt_anon = 94785099, flt_acow = 62991413, flt_obj = 1814723619, flt_prcopy = 298511667, flt_przero = -1979758914,
pdwoke = 30586934, pdrevs = 1259, _unused3 = 0, pdfreed = 91987649, pdscans = 120124040, pdanscan = 690052, pdobscan = 91298403, pdreact = 431879, pdbusy = 0, pdpageouts = 3586174, pdpending = 53788537,
pddeact = 224044232, pdreanon = 0, pdrefile = 0, pdreexec = 0, bootpages = 1009928}
(gdb) p uvmexp.swpgonly
$3 = 18576494
(gdb) p uvmexp.swpginuse
$4 = 18418907
I don't understand why the numbers here are both different than in the
panic message itself.
Also, some (unrelated?) values are negative, like:
colorhit = -1069245636
which doesn't make sense to me either.
Core dump is available on request.
Cheers,
Thomas
From: Thomas Klausner <wiz@netbsd.org>
To: NetBSD bugtracking <gnats-bugs@netbsd.org>
Cc:
Subject: Re: kern/56764
Date: Sat, 21 Feb 2026 14:19:35 +0100
And again today:
panic: kernel diagnostic assertion "uvmexp.swpgonly < uvmexp.swpginuse" failed: file "/usr/src/sys/uvm/uvm_pdaemon.c", line 795 swpgonly 26000749 swpginuse 25888760
(gdb) bt
#0 0xffffffff80239eca in cpu_reboot (howto=howto@entry=260, bootstr=bootstr@entry=0x0) at /usr/src/sys/arch/amd64/amd64/machdep.c:709
#1 0xffffffff80e2bfbf in kern_reboot (howto=howto@entry=260, bootstr=bootstr@entry=0x0) at /usr/src/sys/kern/kern_reboot.c:91
#2 0xffffffff80e77295 in vpanic (fmt=0xffffffff816ee020 "kernel %sassertion \"%s\" failed: file \"%s\", line %d swpgonly %d swpginuse %d", ap=ap@entry=0xffffde247baa4e98)
at /usr/src/sys/kern/subr_prf.c:288
#3 0xffffffff81051cfe in kern_assert (fmt=fmt@entry=0xffffffff816ee020 "kernel %sassertion \"%s\" failed: file \"%s\", line %d swpgonly %d swpginuse %d")
at /usr/src/sys/lib/libkern/kern_assert.c:51
#4 0xffffffff80dd86a3 in uvmpd_scan_queue () at /usr/src/sys/uvm/uvm_pdaemon.c:795
#5 uvmpd_scan () at /usr/src/sys/uvm/uvm_pdaemon.c:919
#6 uvm_pageout (arg=<optimized out>) at /usr/src/sys/uvm/uvm_pdaemon.c:318
#7 0xffffffff80210327 in lwp_trampoline ()
#8 0x0000000000000000 in ?? ()
(gdb) p uvmexp
$1 = {pagesize = 4096, pagemask = 4095, pageshift = 12, npages = 32315697, free = 8039, paging = 16, wired = 4913, ncolors = 32, colormask = 31, zeropages = 0,
reserve_pagedaemon = 1, reserve_kernel = 160, anonpages = 21282242, filepages = 4459464, execpages = 959813, freemin = 8192, freetarg = 10922, wiredmax = 10771899,
nswapdev = 3, swpages = 133993646, swpgavail = 133993646, swpginuse = 25789384, swpgonly = 25706596, nswget = 29166374, faults = 2039508928, traps = -2132403363,
intrs = 502227833, swtch = -13256194, softs = 754970585, syscalls = -364678381, pageins = 2547602, _unused1 = 0, _unused2 = 0, pgswapin = 0, pgswapout = 529198543,
forks = 11607662, forks_ppwait = 3708387, forks_sharevm = 3708387, pga_zerohit = 0, pga_zeromiss = 0, zeroaborts = 0, colorhit = 578435610, colormiss = 694700272,
cpuhit = 686404484, cpumiss = 586731398, fltnoram = 849845748, fltnoanon = 0, fltpgwait = 119329, fltpgrele = 0, fltrelck = -1500510270, fltrelckok = -1500820650,
fltanget = 1101511225, fltanretry = 2987802, fltamcopy = 1912608152, fltnamap = 969584305, fltnomap = 252048530, fltlget = 31900090, fltget = -1503503167,
flt_anon = 689900560, flt_acow = 411604461, flt_obj = -1188934567, flt_prcopy = 964935297, flt_przero = 303214857, pdwoke = 173361105, pdrevs = 37988, _unused3 = 0,
pdfreed = 775174807, pdscans = 986789583, pdanscan = 4021869, pdobscan = 771157076, pdreact = 3977394, pdbusy = 32, pdpageouts = 33088753, pdpending = 496109791,
pddeact = 1571316217, pdreanon = 0, pdrefile = 0, pdreexec = 0, bootpages = 1009928}
again the numbers are different than in the assert, to wit:
assert: swpgonly 26000749 swpginuse 25888760
struct uvmpexp: swpgonly 25706596 swpginuse 25789384
so they are fine in the struct but not in the assert.
But the assert is for exactly these:
795 KASSERTMSG(uvmexp.swpgonly < uvmexp.swpginuse,
796 "swpgonly %d swpginuse %d",
797 uvmexp.swpgonly, uvmexp.swpginuse);
so is something else modifying the values without a proper lock?
Thomas
From: Thomas Klausner <wiz@gatalith.at>
To: NetBSD bugtracking <gnats-bugs@netbsd.org>
Cc:
Subject: Re: kern/56764
Date: Sat, 21 Feb 2026 16:49:59 +0100
--ykk54ve36zcfxrfd
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Hi!
I'm trying to find causes why I get
kernel diagnostic assertion "uvmexp.swpgonly < uvmexp.swpginuse" failed
quite regularly.
void uvm_aio_aiodone_pages(struct vm_page **pgs, int npages, bool
write, int error) in uvm_pager.c v1.131 ends with this:
if (!swap) {
uvm_page_unbusy(pgs, npages);
rw_exit(slock);
} else {
#if defined(VMSWAP)
KASSERT(write);
/* these pages are now only in swap. */
if (error != ENOMEM) {
atomic_add_int(&uvmexp.swpgonly, npages);
}
if (error) {
if (error != ENOMEM)
uvm_swap_markbad(swslot, npages);
else
uvm_swap_free(swslot, npages);
}
atomic_dec_uint(&uvmexp.pdpending);
#endif /* defined(VMSWAP) */
}
In GENERIC on my system, VMSWAP is 1.
So, if 'swap != 0' and 'error != ENOMEM', uvmexp.swpgonly is increased
by npages. In case error is not 0 and not ENOMEM, it is then decreased
again in uvm_swap_markbad().
In uvm_swap_markbad() the change is protected by
mutex_enter(&uvm_swap_data_lock), but here in uvm_aio_aiodone_pages()
it isn't.
1. Should this be protected by mutex_enter(&uvm_swap_data_lock) in
uvm_aio_aiodone_pages()?
2. How is it ensured that uvmexp.swpgonly < uvmexp.swpginuse here when
error = 0? uvm_aio_aiodone_pages() does not increase swpginuse.
I see a couple other places where changing swpgonly is not protected
by the mutex.
The other suspect are the two places where swpginuse is reduced,
neither of them compares against swpgonly.
I'll try the attached diff (I've had to make the lock available in
uvm_swap.h for the three places outside of uvm_swap.c that modify
swpgonly.)
Thomas
--ykk54ve36zcfxrfd
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="uvm.diff"
Index: uvm_km.c
===================================================================
RCS file: /cvsroot/src/sys/uvm/uvm_km.c,v
retrieving revision 1.166
diff -u -r1.166 uvm_km.c
--- uvm_km.c 7 Dec 2024 23:19:07 -0000 1.166
+++ uvm_km.c 21 Feb 2026 15:32:36 -0000
@@ -491,8 +491,10 @@
rw_exit(uobj->vmobjlock);
if (swpgonlydelta > 0) {
+ mutex_enter(&uvm_swap_data_lock);
KASSERT(uvmexp.swpgonly >= swpgonlydelta);
atomic_add_int(&uvmexp.swpgonly, -swpgonlydelta);
+ mutex_exit(&uvm_swap_data_lock);
}
}
Index: uvm_pager.c
===================================================================
RCS file: /cvsroot/src/sys/uvm/uvm_pager.c,v
retrieving revision 1.131
diff -u -r1.131 uvm_pager.c
--- uvm_pager.c 15 Mar 2024 07:09:37 -0000 1.131
+++ uvm_pager.c 21 Feb 2026 15:32:36 -0000
@@ -483,7 +483,12 @@
/* these pages are now only in swap. */
if (error != ENOMEM) {
+ mutex_enter(&uvm_swap_data_lock);
+ KASSERTMSG(uvmexp.swpginuse >= uvmexp.swpgonly + npages,
+ "swpginuse %d swpgonly %d npages %d",
+ uvmexp.swpginuse, uvmexp.swpgonly, npages);
atomic_add_int(&uvmexp.swpgonly, npages);
+ mutex_enter(&uvm_swap_data_lock);
}
if (error) {
if (error != ENOMEM)
Index: uvm_pdaemon.c
===================================================================
RCS file: /cvsroot/src/sys/uvm/uvm_pdaemon.c,v
retrieving revision 1.135
diff -u -r1.135 uvm_pdaemon.c
--- uvm_pdaemon.c 4 Jan 2026 00:41:14 -0000 1.135
+++ uvm_pdaemon.c 21 Feb 2026 15:32:37 -0000
@@ -792,10 +792,12 @@
}
if (slot > 0) {
/* this page is now only in swap. */
+ mutex_enter(&uvm_swap_data_lock);
KASSERTMSG(uvmexp.swpgonly < uvmexp.swpginuse,
"swpgonly %d swpginuse %d",
uvmexp.swpgonly, uvmexp.swpginuse);
atomic_inc_uint(&uvmexp.swpgonly);
+ mutex_exit(&uvm_swap_data_lock);
}
rw_exit(slock);
continue;
Index: uvm_swap.c
===================================================================
RCS file: /cvsroot/src/sys/uvm/uvm_swap.c,v
retrieving revision 1.215
diff -u -r1.215 uvm_swap.c
--- uvm_swap.c 13 Feb 2026 19:16:41 -0000 1.215
+++ uvm_swap.c 21 Feb 2026 15:32:37 -0000
@@ -201,7 +201,7 @@
static struct swap_priority swap_priority;
/* locks */
-static kmutex_t uvm_swap_data_lock __cacheline_aligned;
+kmutex_t uvm_swap_data_lock __cacheline_aligned;
static krwlock_t swap_syscall_lock;
bool uvm_swap_init_done = false;
@@ -1246,6 +1246,9 @@
KASSERTMSG(uvmexp.swpginuse >= sdp->swd_npgbad,
"swpginuse %d sdp->swd_npgbad %d",
uvmexp.swpginuse, sdp->swd_npgbad);
+ KASSERTMSG(uvmexp.swpginuse >= uvmexp.swpgonly + sdp->swd_npgbad,
+ "swpginuse %d swpgonly %d sdp->swd_npgbad %d",
+ uvmexp.swpginuse, uvmexp.swpgonly, sdp->swd_npgbad);
uvmexp.swpginuse -= sdp->swd_npgbad;
if (swaplist_find(sdp->swd_vp, true) == NULL)
@@ -1932,6 +1935,9 @@
sdp->swd_npginuse -= nslots;
KASSERTMSG(uvmexp.swpginuse >= nslots, "swpginuse %d nslots %d",
uvmexp.swpginuse, nslots);
+ KASSERTMSG(uvmexp.swpginuse >= uvmexp.swpgonly + nslots,
+ "swpginuse %d swpgonly %d nslots %d",
+ uvmexp.swpginuse, uvmexp.swpgonly, nslots);
uvmexp.swpginuse -= nslots;
mutex_exit(&uvm_swap_data_lock);
}
@@ -1977,8 +1983,10 @@
* this page is no longer only in swap.
*/
+ mutex_enter(&uvm_swap_data_lock);
KASSERT(uvmexp.swpgonly > 0);
atomic_dec_uint(&uvmexp.swpgonly);
+ mutex_exit(&uvm_swap_data_lock);
}
return error;
}
Index: uvm_swap.h
===================================================================
RCS file: /cvsroot/src/sys/uvm/uvm_swap.h,v
retrieving revision 1.29
diff -u -r1.29 uvm_swap.h
--- uvm_swap.h 15 Mar 2024 22:15:21 -0000 1.29
+++ uvm_swap.h 21 Feb 2026 15:32:37 -0000
@@ -56,6 +56,7 @@
int uvm_swap_stats(char *, int,
void (*)(void *, const struct swapent *), size_t, register_t *);
+extern kmutex_t uvm_swap_data_lock;
#else /* defined(VMSWAP) */
#define uvm_swapisfull() true
--ykk54ve36zcfxrfd--
From: Thomas Klausner <wiz@netbsd.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/56764
Date: Sat, 21 Feb 2026 17:23:25 +0100
--blvg5djcc6d5falj
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
The first diff causes a 'locking against myself' error in
uvm_aio_aiodone_pages, so I removed the lock access in uvm_pager.c and
am trying this version now.
Thomas
--blvg5djcc6d5falj
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="uvm.2.diff"
Index: uvm_km.c
===================================================================
RCS file: /cvsroot/src/sys/uvm/uvm_km.c,v
retrieving revision 1.166
diff -u -r1.166 uvm_km.c
--- uvm_km.c 7 Dec 2024 23:19:07 -0000 1.166
+++ uvm_km.c 21 Feb 2026 16:21:45 -0000
@@ -491,8 +491,10 @@
rw_exit(uobj->vmobjlock);
if (swpgonlydelta > 0) {
+ mutex_enter(&uvm_swap_data_lock);
KASSERT(uvmexp.swpgonly >= swpgonlydelta);
atomic_add_int(&uvmexp.swpgonly, -swpgonlydelta);
+ mutex_exit(&uvm_swap_data_lock);
}
}
Index: uvm_pager.c
===================================================================
RCS file: /cvsroot/src/sys/uvm/uvm_pager.c,v
retrieving revision 1.131
diff -u -r1.131 uvm_pager.c
--- uvm_pager.c 15 Mar 2024 07:09:37 -0000 1.131
+++ uvm_pager.c 21 Feb 2026 16:21:45 -0000
@@ -483,6 +483,9 @@
/* these pages are now only in swap. */
if (error != ENOMEM) {
+ KASSERTMSG(uvmexp.swpginuse >= uvmexp.swpgonly + npages,
+ "swpginuse %d swpgonly %d npages %d",
+ uvmexp.swpginuse, uvmexp.swpgonly, npages);
atomic_add_int(&uvmexp.swpgonly, npages);
}
if (error) {
Index: uvm_pdaemon.c
===================================================================
RCS file: /cvsroot/src/sys/uvm/uvm_pdaemon.c,v
retrieving revision 1.135
diff -u -r1.135 uvm_pdaemon.c
--- uvm_pdaemon.c 4 Jan 2026 00:41:14 -0000 1.135
+++ uvm_pdaemon.c 21 Feb 2026 16:21:45 -0000
@@ -792,10 +792,12 @@
}
if (slot > 0) {
/* this page is now only in swap. */
+ mutex_enter(&uvm_swap_data_lock);
KASSERTMSG(uvmexp.swpgonly < uvmexp.swpginuse,
"swpgonly %d swpginuse %d",
uvmexp.swpgonly, uvmexp.swpginuse);
atomic_inc_uint(&uvmexp.swpgonly);
+ mutex_exit(&uvm_swap_data_lock);
}
rw_exit(slock);
continue;
Index: uvm_swap.c
===================================================================
RCS file: /cvsroot/src/sys/uvm/uvm_swap.c,v
retrieving revision 1.215
diff -u -r1.215 uvm_swap.c
--- uvm_swap.c 13 Feb 2026 19:16:41 -0000 1.215
+++ uvm_swap.c 21 Feb 2026 16:21:45 -0000
@@ -201,7 +201,7 @@
static struct swap_priority swap_priority;
/* locks */
-static kmutex_t uvm_swap_data_lock __cacheline_aligned;
+kmutex_t uvm_swap_data_lock __cacheline_aligned;
static krwlock_t swap_syscall_lock;
bool uvm_swap_init_done = false;
@@ -1246,6 +1246,9 @@
KASSERTMSG(uvmexp.swpginuse >= sdp->swd_npgbad,
"swpginuse %d sdp->swd_npgbad %d",
uvmexp.swpginuse, sdp->swd_npgbad);
+ KASSERTMSG(uvmexp.swpginuse >= uvmexp.swpgonly + sdp->swd_npgbad,
+ "swpginuse %d swpgonly %d sdp->swd_npgbad %d",
+ uvmexp.swpginuse, uvmexp.swpgonly, sdp->swd_npgbad);
uvmexp.swpginuse -= sdp->swd_npgbad;
if (swaplist_find(sdp->swd_vp, true) == NULL)
@@ -1932,6 +1935,9 @@
sdp->swd_npginuse -= nslots;
KASSERTMSG(uvmexp.swpginuse >= nslots, "swpginuse %d nslots %d",
uvmexp.swpginuse, nslots);
+ KASSERTMSG(uvmexp.swpginuse >= uvmexp.swpgonly + nslots,
+ "swpginuse %d swpgonly %d nslots %d",
+ uvmexp.swpginuse, uvmexp.swpgonly, nslots);
uvmexp.swpginuse -= nslots;
mutex_exit(&uvm_swap_data_lock);
}
@@ -1977,8 +1983,10 @@
* this page is no longer only in swap.
*/
+ mutex_enter(&uvm_swap_data_lock);
KASSERT(uvmexp.swpgonly > 0);
atomic_dec_uint(&uvmexp.swpgonly);
+ mutex_exit(&uvm_swap_data_lock);
}
return error;
}
Index: uvm_swap.h
===================================================================
RCS file: /cvsroot/src/sys/uvm/uvm_swap.h,v
retrieving revision 1.29
diff -u -r1.29 uvm_swap.h
--- uvm_swap.h 15 Mar 2024 22:15:21 -0000 1.29
+++ uvm_swap.h 21 Feb 2026 16:21:45 -0000
@@ -56,6 +56,7 @@
int uvm_swap_stats(char *, int,
void (*)(void *, const struct swapent *), size_t, register_t *);
+extern kmutex_t uvm_swap_data_lock;
#else /* defined(VMSWAP) */
#define uvm_swapisfull() true
--blvg5djcc6d5falj--
From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@netbsd.org, netbsd-bugs@netbsd.org, wiz@netbsd.org
Cc:
Subject: Re: kern/56764
Date: Sun, 22 Feb 2026 00:10:53 +0700
Date: Sat, 21 Feb 2026 13:20:02 +0000 (UTC)
From: "Thomas Klausner via gnats" <gnats-admin@NetBSD.org>
Message-ID: <20260221132002.315021A923D@mollari.NetBSD.org>
| so is something else modifying the values without a proper lock?
I know nothing about the UVM system, so nothing I say should be treated
as being correct, but will perhaps inspire someone who knows more to
correct me.
But I just took a look, and I can see no evidence at all that anything in
uvmexp expects to take any kind of lock before modifications.
Some values (like uvmexp.swpgonly) are only ever modified via atomic
ops (increment/decrement/add) so in theory should be safe, but it isn't
listed as being volatile, so accesses to it could be cached for who
knows how long (in a local register) while the actual value is being
modified by some other thread. Others (like uvmexp.swpginuse) are
simply modified by regular C += and -= operations.
All of this looks very unsafe to me, for as much as the uvmexp values are
just FYIs, it would all be OK, and many of the values are, though just
being int (for almost all of them) many are very likely to overflow on
a busy system, and I see that the many of the values in your gdb dump of
the struct are negative which almost certainly means they overflowed).
But some of the values (like these two) aren't quite the same, they're
actually tracking the state of the swap system. But the way things are
done, I don't know that it makes sense to ever compare them, despite the
logic of their uses implying that should be possible (one would expect that
the number of swap pages allocated to pages that are only resident on the
swap device (/dev/drum) must be less than the total number of swap pages
allocated, or at least not greater, as there would often be some pages
resident in both ram and on the swap device, which should make a difference).
WRT your newer message:
gnats-admin@NetBSD.org (wiz@ really) said:
| In uvm_swap_markbad() the change is protected by
| mutex_enter(&uvm_swap_data_lock),
Not really, that lock is held, but its purpose is to protect the swapdev
lists, it prevents manipulations of those lists while held, it isn't
intended to apply to uvmexp, which as I said above, I don't think anything
is.
I think I'd just remove the KASSERT that is firing for you, and look for
some other way to try and understand the original problem from the PR.
kre
ps: once again, I really have no idea what I am typing about!
From: Thomas Klausner <wiz@netbsd.org>
To: Robert Elz <kre@munnari.oz.au>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/56764
Date: Sat, 21 Feb 2026 18:30:55 +0100
Hi!
Thanks for looking at this.
On Sun, Feb 22, 2026 at 12:10:53AM +0100, Robert Elz wrote:
> But I just took a look, and I can see no evidence at all that anything in
> uvmexp expects to take any kind of lock before modifications.
I thought this was the case because of functions like uvm_swapisfull, in full:
bool
uvm_swapisfull(void)
{
int swpgonly;
bool rv;
if (uvmexp.swpages == 0) {
return true;
}
mutex_enter(&uvm_swap_data_lock);
KASSERT(uvmexp.swpgonly <= uvmexp.swpages);
swpgonly = (int)((uint64_t)uvmexp.swpgonly * 100 /
uvm_swapisfull_factor);
rv = (swpgonly >= uvmexp.swpgavail);
mutex_exit(&uvm_swap_data_lock);
return (rv);
}
It only accesses uvmexp members and still grabs the lock.
> Some values (like uvmexp.swpgonly) are only ever modified via atomic
> ops (increment/decrement/add) so in theory should be safe, but it isn't
> listed as being volatile, so accesses to it could be cached for who
> knows how long (in a local register) while the actual value is being
> modified by some other thread. Others (like uvmexp.swpginuse) are
> simply modified by regular C += and -= operations.
If you look closely you'll find that swpginuse is only modified when
uvm_swap_data_lock is held.
That was the other reason why I thought the same should apply to
swpgonly.
> I think I'd just remove the KASSERT that is firing for you, and look for
> some other way to try and understand the original problem from the PR.
I've hijacked this PR a bit - it was about
kernel diagnostic assertion "uvmexp.swpgonly > 0" failed
while I was hitting
kernel diagnostic assertion "uvmexp.swpgonly < uvmexp.swpginuse" failed
the swpgonly > 0 definitely sounds sane while the other one should
(according to Chuck) always be true too.
Thomas
From: Thomas Klausner <wiz@netbsd.org>
To: NetBSD bugtracking <gnats-bugs@netbsd.org>
Cc:
Subject: Re: kern/56764
Date: Sat, 21 Feb 2026 18:55:17 +0100
My second diff just showed where the assumption is violated:
panic: kernel diagnostic assertion "uvmexp.swpginuse >= uvmexp.swpgonly + nslots" failed: file "/usr/src/sys/uvm/uvm_swap.c", line 1938 swpginuse 18158179 swpgonly 18158179 nslots 1
i.e. swpginuse is reduced before swpgonly is reduced.
I think I'll just remove the assertion that swpginuse > swpgonly
locally - it's just not consistently true.
Thomas
From: Robert Elz <kre@munnari.OZ.AU>
To: Thomas Klausner <wiz@netbsd.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/56764
Date: Sun, 22 Feb 2026 01:14:26 +0700
Date: Sat, 21 Feb 2026 18:30:55 +0100
From: Thomas Klausner <wiz@netbsd.org>
Message-ID: <aZnqBQcezBYRM5zw@exadelic.gatalith.at>
| It only accesses uvmexp members and still grabs the lock.
OK, like I said, I am guessing a lot, but if
| If you look closely you'll find that swpginuse is only modified when
| uvm_swap_data_lock is held.
that's true, then good.
| I've hijacked this PR a bit - it was about
| kernel diagnostic assertion "uvmexp.swpgonly > 0" failed
Yes. I saw.
| while I was hitting
|
| kernel diagnostic assertion "uvmexp.swpgonly < uvmexp.swpginuse" failed
But that's an assertion you just added in Jan, right?
| the swpgonly > 0 definitely sounds sane while the other one should
| (according to Chuck) always be true too.
Logically, I agree, but with new compilers and the current C rules,
I'm not sure the code can guarantee that.
Since the addr of uvmexp.swpgonly is passed to the atomic functions,
the compiler must assume that it might be being modified, which of
course it is. But that means it needs to be read from the struct
on any reference after one of those operations (there's just the one
in uvmpd_scan_queue() which is the KASSERT() in question).
But that's not true of uvmexp.swpginuse ... uvmpd_scan_queue() is just one
big loop, runs until it has nothing left to do, over and over. The compiler
might read uvmexp.swpginuse, stick in a register, and as nothing in that
func is touching it, nor passing its addr to any other funcs, and it isn't
volatile, the compiler is entitled (by modern C rules) to assume that nothing
is changing it. Maybe that doesn't apply to extern structs, beats me.
But I also see this code in uvmpd_scan_queue() (not that I expect this
is related to your issue):
if ((p->flags & PG_SWAPBACKED) == 0) {
KASSERT(uobj != NULL);
(void) (uobj->pgops->pgo_put)(uobj, p->offset,
p->offset + PAGE_SIZE, PGO_CLEANIT|PGO_FREE);
continue;
}
That's going around the big loop again - but I see no attempt to
rw_exit(slock);
slock is a local var in uvmpd_scan_queue() so whatever the function is
(uobj->pgops->pgo_put) it cannot be exiting the locked region.
Every other "continue;" in that section of code (before and after that one
- from after the slock is obtained) has that rw_exit(slock) just before the
continue, so there may well be lots of locked stuff hanging around, if
that particular piece of code ever runs. Very unlikely to be related to
your issue(s) but it looks broken to me.
kre
From: Thomas Klausner <wiz@netbsd.org>
To: Robert Elz <kre@munnari.oz.au>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/56764
Date: Sat, 21 Feb 2026 20:44:32 +0100
On Sun, Feb 22, 2026 at 01:14:26AM +0100, Robert Elz wrote:
> | while I was hitting
> |
> | kernel diagnostic assertion "uvmexp.swpgonly < uvmexp.swpginuse" failed
>
> But that's an assertion you just added in Jan, right?
No - I just added details for the values, the assertion has been there
(and hitting me) for a long time.
Thomas
From: "Thomas Klausner" <wiz@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/56764 CVS commit: src/sys/uvm
Date: Sun, 22 Mar 2026 12:00:08 +0000
Module Name: src
Committed By: wiz
Date: Sun Mar 22 12:00:08 UTC 2026
Modified Files:
src/sys/uvm: uvm_pdaemon.c
Log Message:
Disable a kassertmsg.
This triggers for me quite reliably over years now, and has been
tracked in PR 56764, with no resolution.
It seems the asserted inequality just is not correct.
To generate a diff of this commit:
cvs rdiff -u -r1.136 -r1.137 src/sys/uvm/uvm_pdaemon.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "YAMAMOTO Takashi" <yamt@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/56764 CVS commit: src/sys/uvm
Date: Mon, 23 Mar 2026 12:44:50 +0000
Module Name: src
Committed By: yamt
Date: Mon Mar 23 12:44:50 UTC 2026
Modified Files:
src/sys/uvm: uvm_pdaemon.c
Log Message:
uvmpd_scan_queue: remove ENABLE_UNRELIABLE_CHECK_PR_56764 block
while this condition is true in most of times, we can't
assert it here because these counters are not always
updated in-sync.
for example, consider a removal of a large tmpfs file which is
mostly swapped out. because uao_dropswap_range() batches swpgonly
updates, swpgonly can be temporarily larger than swpginuse.
the original symptom reported in PR/56764 ("uvmexp.swpgonly > 0")
looks like a different issue though.
https://gnats.netbsd.org/56764
To generate a diff of this commit:
cvs rdiff -u -r1.137 -r1.138 src/sys/uvm/uvm_pdaemon.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/56764 CVS commit: [netbsd-11] src/sys/uvm
Date: Fri, 3 Apr 2026 12:22:23 +0000
Module Name: src
Committed By: martin
Date: Fri Apr 3 12:22:23 UTC 2026
Modified Files:
src/sys/uvm [netbsd-11]: uvm_pdaemon.c uvm_swap.c
Log Message:
Pull up following revision(s) (requested by yamt in ticket #242):
sys/uvm/uvm_pdaemon.c: revision 1.135
sys/uvm/uvm_pdaemon.c: revision 1.137
sys/uvm/uvm_pdaemon.c: revision 1.138
sys/uvm/uvm_swap.c: revision 1.210
Add more debugging.
To help understand PR 56764 better.
Ok riastradh@
Disable a kassertmsg.
This triggers for me quite reliably over years now, and has been
tracked in PR 56764, with no resolution.
It seems the asserted inequality just is not correct.
uvmpd_scan_queue: remove ENABLE_UNRELIABLE_CHECK_PR_56764 block
while this condition is true in most of times, we can't
assert it here because these counters are not always
updated in-sync.
for example, consider a removal of a large tmpfs file which is
mostly swapped out. because uao_dropswap_range() batches swpgonly
updates, swpgonly can be temporarily larger than swpginuse.
the original symptom reported in PR/56764 ("uvmexp.swpgonly > 0")
looks like a different issue though.
To generate a diff of this commit:
cvs rdiff -u -r1.134 -r1.134.8.1 src/sys/uvm/uvm_pdaemon.c
cvs rdiff -u -r1.209 -r1.209.2.1 src/sys/uvm/uvm_swap.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home