NetBSD Problem Report #60958

From www@netbsd.org  Tue Nov  3 17:25:23 2026
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "mail.netbsd.org", Issuer "YR1" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id E48281A923B
	for <gnats-bugs@gnats.NetBSD.org>; Tue,  3 Nov 2026 17:25:22 +0000 (UTC)
Message-Id: <20261103172521.565F81A923C@mollari.NetBSD.org>
Date: Tue,  3 Nov 2026 17:25:21 +0000 (UTC)
From: el@horse64.org
Reply-To: el@horse64.org
To: gnats-bugs@NetBSD.org
Subject: (Probaby) avoidable stability problems due to default memory management behavior on systems with a lot of RAM, ZFS, and no swap
X-Send-Pr-Version: www-1.0
X-From4GNATS: "el@horse64.org via gnats" <gnats-admin@NetBSD.org>

>Number:         60958
>Notify-List:    riastradh
>Category:       misc
>Synopsis:       (Probaby) avoidable stability problems due to default memory management behavior on systems with a lot of RAM, ZFS, and no swap
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    misc-bug-people
>State:          feedback
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Tue Nov 03 17:30:01 +0000 2026
>Closed-Date:    
>Last-Modified:  Sat Nov 07 19:10:01 +0000 2026
>Originator:     Ellie
>Release:        11.0
>Organization:
none, really!
>Environment:
NetBSD ruby-b-s-delly 11.0_STABLE NetBSD 11.0_STABLE (GENERIC) #0: Fri Oct 30 19:02:53 UTC 2026  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/amd64/compile/GENERIC amd64

>Description:
After some discussions in chat, I was encouraged to report that I'm running into (probaby) avoidable stability problems when using NetBSD as a desktop. And these seem to be due to default memory management behavior.

Preconditions for these issues:

- I'm testing this on a system with 32GB RAM.

- I don't have swap: based on my past usage of Linux, I assumed with that amount I wouldn't strictly speaking need swap. Since I like avoiding unnecessary SSD disk writes, I went without one. I also have an encrypted / and encrypted /home so I wrongly assumed a swap would also be unencrypted - that seems to be untrue for NetBSD, but I didn't know that yet.

- I'm also using ZFS for /home, inside a cgd volume.

- The main offender leading to chaos seems to be a moderately used firefox with less than 30 tabs.

So what happens under these conditions seems to be roughly these chain of events:

1. Firefox on NetBSD seems fundamentally unable to see its own memory usage. You can see this when you open up about:memory in Firefox, it lists a lot of warnings and error messages. Perhaps in part due to that, it keeps growing.

2. Many JS heavy websites seem to leak memory, or at least to generate so many garbage objects that browsers struggle to collect garbage after them.

3. With under 30 tabs and little interaction, if one of these tabs is e.g. Matrix Element or Discord, Firefox will continue to use more memory on NetBSD, including 30GB+ RAM and beyond.

4. When memory runs out for the first and second time, no kernel memory killer kicks in. Instead, ZFS evicts a lot of its cache, trying desperately to keep the system running.

5. Then, with memory in "top" still reported to not be fully used up, ZFS will evict so much of its own memory that it seems to start to break down. At this point, a lot of programs start to freeze, and it's impossible to launch any new programs. I had a root shell open, I typed "dmesg" and it just hung.

6. Memory continues to get used up and if anything is getting killed I would be flying blind, since tail -f /var/log/messages seems to get stuck too and I can't even invoke "dmesg" anymore. If the OOM killer ever does something, it's too late.

Sorry for the wall of text, and I hope you're having a nice day! NetBSD is cool.

Regards,

Ellie
>How-To-Repeat:
1. Install NetBSD desktop. Don't configure any swap partitions.

2. Launch X or Wayland or whatever you like.

3. Install and launch Firefox. Open up a bunch of tabs, make sure to include some heavy JS apps that deal with a continuous intake of data like Matrix's Element or Disord's web UI.

4. Now you just have to wait. It might take a few hours or even more than a day, depending on how much memory your system has.
>Fix:
There might be multiple relatively doable ways of handling this better:

- For example, there could be a configurable user space OOM daemon similar to https://gihtub.com/rfjakob/earlyoom which could have more easily configurable limits, or a list of preferred processes to kill (e.g. browser stuff) and a list of processes to avoid (X, etc.).

- Or the kernel OOM could be configured by default to trigger earlier on systems without swap.

- The OOM should also probably try to do something on systems that have plenty of RAM, let's say 4GB plus, once ZFS starts evicting things, which seems to be at less than 10% free. Because letting ZFS take the fall only tends to provoke other problems.

- But at the end of the day I'm probably not smart enough to suggest the best solution. But I hope you'll have some ideas.

I hope you'll feel inspired to look into some of these things, since I think it would make a huge difference for NetBSD desktop use.

(As a slight tangent, for example I first thought the kernel was freezing up due to other problems and had major stability issues due to some driver bug.

Only because I'm I guess a comparatively somewhat experienced Unix user, did I eventually track down after 10+ freezes what was even going on. Thanks to the #NetBSD chatters too, they are awesome! I could easily see other new users give up earlier.)

>Release-Note:

>Audit-Trail:

State-Changed-From-To: open->feedback
State-Changed-By: riastradh@NetBSD.org
State-Changed-When: Tue, 03 Nov 2026 17:49:14 +0000
State-Changed-Why:
There's a lot of speculation and analysis here, and not a lot of
specific evidence.  Can you share some more details like:

1. exactly what observations led you to say `When memory runs out for
   the first and second time [...] ZFS evicts a lot of cache'?

2. exactly what observations led you to say `ZFS will evict so much of
   its own memory that it seems to start to break down'?

3. output of top(1) at various stages?

4. output of the following commands at various stages?
   - echo ps | crash
   - echo ps/w | crash
   - echo show all locks | crash
   - echo show all tstiles | crash

5. if you kill firefox, do the symptoms continue?

6. if you can't start crash, try switching to the first virtual
   terminal ttyE0 with C-A-F1, drop into ddb with C-A-ESC, and take a
   photo of the first page of output of the same commands (ps, ps/w,
   show all locks, show all tstiles) -- don't go wild recording
   everything, mostly just want to see a few wchans to find what
   processes are waiting on

7. if you drop into ddb and type `call wakeup(uvmexp+0x10)' and then
   `continue', do the symptoms continue?

(For context: I have been running zfs (on cgd) and firefox with
hundreds of tabs on laptops with 32-64GB of RAM for years, and have not
encountered these symptoms.)


From: Ellie <el@horse64.org>
To: gnats-bugs@netbsd.org, misc-bug-people@netbsd.org,
 netbsd-bugs@netbsd.org, gnats-admin@netbsd.org, riastradh@NetBSD.org
Cc: 
Subject: Re: misc/60958 ((Probaby) avoidable stability problems due to default
 memory management behavior on systems with a lot of RAM, ZFS, and no swap)
Date: Tue, 3 Nov 2026 19:42:47 +0000

 Happy to elaborate, and sorry if some of my speculation is wrong.

 For what it's worth, I was in the #NetBSD chat at the time and shared 
 details about the process, so some of this is based on the input of 
 others. I'm really not that terribly smart...

 Let me provide more details where I can:

 On 11/3/26 17:49, riastradh@NetBSD.org wrote:
 > Synopsis: (Probaby) avoidable stability problems due to default memory management behavior on systems with a lot of RAM, ZFS, and no swap
 > 
 > State-Changed-From-To: open->feedback
 > State-Changed-By: riastradh@NetBSD.org
 > State-Changed-When: Tue, 03 Nov 2026 17:49:14 +0000
 > State-Changed-Why:
 > There's a lot of speculation and analysis here, and not a lot of
 > specific evidence.  Can you share some more details like:
 > 
 > 1. exactly what observations led you to say `When memory runs out for
 >     the first and second time [...] ZFS evicts a lot of cache'?

 I had "top" open and was talking about the process in the chat, while 
 waiting for the OOM to confirm my hypothesis and observing what was 
 going on as best as I could.

 During the unfolding of events, I observed the only major process taking 
 up a lot of memory (Firefox) continuing to grow, and free memory (shown 
 in the top right of "top") going to zero then suddenly jumping to 8GB.

 Another such jump happened from zero free memory back to around 4GB. At 
 the same time, "Swap / Pools" went down - I didn't check if by the exact 
 same amount, but from around 12G to less than 1G at the end.

 > 
 > 2. exactly what observations led you to say `ZFS will evict so much of
 >     its own memory that it seems to start to break down'?

 This was my best guess because after the second jump from zero memory to 
 around 4GB free, I could no longer launch any processes:

 https://horse64.org/100_7111.JPG

 For example I had a terminal open, I typed "dmesg" which I still saw as 
 a word appearing in the terminal, but after pressing enter, I just get a 
 blank line and nothing ever happened. See in the photo!

 Firefox's window stopped redrawing, but generally X remained responsive 
 at first and I saw "top" continuing to run and update the remaining 
 memory count.

 That this is related to ZFS causing trouble for the I/O subsystem after 
 evicting so much memory was a mere guess and I could be 100% wrong.

 > 
 > 3. output of top(1) at various stages?

 You can see the output of "top" and "dmesg" being stuck on this photo 
 that I took at that point:

 https://horse64.org/100_7111.JPG

 Firefox had multiple processes listed that took over 30GB at that point.

 > 
 > 4. output of the following commands at various stages?
 >     - echo ps | crash
 >     - echo ps/w | crash
 >     - echo show all locks | crash
 >     - echo show all tstiles | crash

 I assume that once "dmesg" no longer works, I wouldn't be able to run 
 these anymore.

 I could collect those at earlier stages before memory runs out, perhaps 
 before & after the first possibly-ZFS-eviction, would that help?
 > 
 > 5. if you kill firefox, do the symptoms continue?

 The system stays perfectly alive without Firefox, for many hours. Once 
 "dmesg" hangs I wouldn't know how to kill Firefox. Before that, there 
 aren't really any symptoms. The first big perhaps-ZFS-eviction had no 
 noticeable effect other than the suspicious change in numbers in "top".

 > 
 > 6. if you can't start crash, try switching to the first virtual
 >     terminal ttyE0 with C-A-F1, drop into ddb with C-A-ESC, and take a
 >     photo of the first page of output of the same commands (ps, ps/w,
 >     show all locks, show all tstiles) -- don't go wild recording
 >     everything, mostly just want to see a few wchans to find what
 >     processes are waiting on

 I couldn't start anything at that point, SSH login doesn't even work 
 anymore once "dmesg" also no longer does. I highly doubt the TTY login 
 would still work at that stage.


 > 
 > 7. if you drop into ddb and type `call wakeup(uvmexp+0x10)' and then
 >     `continue', do the symptoms continue?
 > 
 > (For context: I have been running zfs (on cgd) and firefox with
 > hundreds of tabs on laptops with 32-64GB of RAM for years, and have not
 > encountered these symptoms.)

 I wonder, had you ever firefox take up all your memory? Perhaps you just 
 didn't let the right web apps sit in idle, or something.

 Also, people in chat were guessing that no swap might make a big 
 difference in how recoverable this situation.

 Regards,

 Ellie

From: Ellie <el@horse64.org>
To: gnats-bugs@netbsd.org, misc-bug-people@netbsd.org,
 gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Cc: riastradh@netbsd.org
Subject: Re: misc/60958 ((Probaby) avoidable stability problems due to default
 memory management behavior on systems with a lot of RAM, ZFS, and no swap)
Date: Tue, 3 Nov 2026 20:01:40 +0000

 On 11/3/26 19:45, Ellie via gnats wrote:
 > Subject: Re: misc/60958 ((Probaby) avoidable stability problems due to default
 >   memory management behavior on systems with a lot of RAM, ZFS, and no swap)
 > Date: Tue, 3 Nov 2026 19:42:47 +0000
 >   
 >   On 11/3/26 17:49, riastradh@NetBSD.org wrote:
 >   > Synopsis: (Probaby) avoidable stability problems due to default memory management behavior on systems with a lot of RAM, ZFS, and no swap
 >   >
 >   > State-Changed-From-To: open->feedback
 >   > State-Changed-By: riastradh@NetBSD.org
 >   > State-Changed-When: Tue, 03 Nov 2026 17:49:14 +0000
 >   > State-Changed-Why:

 >   Another such jump happened from zero free memory back to around 4GB. At
 >   the same time, "Swap / Pools" went down - I didn't check if by the exact
 >   same amount, but from around 12G to less than 1G at the end.

 Oops, a correction: I may have mixed that up, see photo! It did go down 
 but not as much.

 Also, my photo link was wrong: https://horse64.org/temp/100_7111.JPG

 Regards,

 Ellie

From: Ellie <el@horse64.org>
To: gnats-bugs@netbsd.org, misc-bug-people@netbsd.org,
 gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Cc: riastradh@netbsd.org
Subject: Re: misc/60958 ((Probaby) avoidable stability problems due to default
 memory management behavior on systems with a lot of RAM, ZFS, and no swap)
Date: Tue, 3 Nov 2026 20:10:29 +0000

 On 11/3/26 20:05, Ellie via gnats wrote:
 > Subject: Re: misc/60958 ((Probaby) avoidable stability problems due to default
 >   memory management behavior on systems with a lot of RAM, ZFS, and no swap)
 > Date: Tue, 3 Nov 2026 20:01:40 +0000
 > 
 Here, a screenshot of the normal output of "top" when not nearly OOM:

 https://horse64.org/temp/memorynormal.png

 Perhaps that is useful as a reference, for whatever numbers that I 
 barely understand myself (e.g. for the pool number, whatever that one 
 truly means and whether it's connected to ZFS at all, or not).

 Regards,

 Ellie

From: Taylor R Campbell <riastradh@NetBSD.org>
To: Ellie <el@horse64.org>
Cc: gnats-bugs@netbsd.org, misc-bug-people@netbsd.org,
	netbsd-bugs@netbsd.org, gnats-admin@netbsd.org
Subject: Re: misc/60958 ((Probaby) avoidable stability problems due to default memory management behavior on systems with a lot of RAM, ZFS, and no swap)
Date: Tue, 3 Nov 2026 20:20:31 +0000

 > Date: Tue, 3 Nov 2026 19:42:47 +0000
 > From: Ellie <el@horse64.org>
 > 
 > During the unfolding of events, I observed the only major process taking 
 > up a lot of memory (Firefox) continuing to grow, and free memory (shown 
 > in the top right of "top") going to zero then suddenly jumping to 8GB.

 Which columns were you counting?

 SIZE is the virtual address space size, and doesn't mean much:
 processes can map large swaths of virtual address space without ever
 needing backing pages allocated for it.

 RES is the resident set size, i.e., amount of RAM actually used as
 backing pages in the process's address space.  This means more than
 SIZE for RAM usage -- but it may be double-counted, e.g. if two
 processes have the same file (such as the multi-gigabyte libxul.so)
 mapped into their address space and in use.  So the RES column doesn't
 sum to the systemwide RAM in use.

 > Another such jump happened from zero free memory back to around 4GB. At 
 > the same time, "Swap / Pools" went down - I didn't check if by the exact 
 > same amount, but from around 12G to less than 1G at the end.

 FYI, you should expect to only rarely have substantial amounts of
 `Free' memory in top(1) -- e.g., shortly after memory-intensive
 processes have exited or large cached files have been deleted, but
 before the pages have been reused to cache file data.

 It is almost always better for the system to use RAM to cache file
 data indefinitely than to discard the file data promptly to `free' the
 RAM:

 - If the system discards the file data, and someone asks for the file
   data again, the system has to read it back from disk, which is
   costly.

 - If the system keeps the file data, and someone asks for more RAM,
   the system can just ditch the file data and recycle the RAM
   immediately, which is almost free (except possibly for zeroing the
   pages).

 So the system does not aim to keep the `Free' memory high; it only
 aims to free RAM on demand, generally by finding inactive cached file
 data to discard.

 > https://horse64.org/100_7111.JPG

 404?

 > For example I had a terminal open, I typed "dmesg" which I still saw as 
 > a word appearing in the terminal, but after pressing enter, I just get a 
 > blank line and nothing ever happened. See in the photo!

 Does ^T respond in that case?

 > > 4. output of the following commands at various stages?
 > >     - echo ps | crash
 > >     - echo ps/w | crash
 > >     - echo show all locks | crash
 > >     - echo show all tstiles | crash
 > 
 > I assume that once "dmesg" no longer works, I wouldn't be able to run 
 > these anymore.
 > 
 > I could collect those at earlier stages before memory runs out, perhaps 
 > before & after the first possibly-ZFS-eviction, would that help?

 If you notice anything hanging, you could try that.  If it doesn't
 work, the equivalent commands are available in ddb, which runs in the
 kernel even if userland is stuck.

 > > 6. if you can't start crash, try switching to the first virtual
 > >     terminal ttyE0 with C-A-F1, drop into ddb with C-A-ESC, and take a
 > >     photo of the first page of output of the same commands (ps, ps/w,
 > >     show all locks, show all tstiles) -- don't go wild recording
 > >     everything, mostly just want to see a few wchans to find what
 > >     processes are waiting on
 > 
 > I couldn't start anything at that point, SSH login doesn't even work 
 > anymore once "dmesg" also no longer does. I highly doubt the TTY login 
 > would still work at that stage.

 You don't need to interact with the login process.  The logic of
 switching virtual terminals mostly happens in the kernel, and the
 logic of entering ddb with C-A-ESC is entirely in the kernel.

From: Ellie <el@horse64.org>
To: Taylor R Campbell <riastradh@NetBSD.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: misc/60958 ((Probaby) avoidable stability problems due to default
 memory management behavior on systems with a lot of RAM, ZFS, and no swap)
Date: Wed, 4 Nov 2026 01:38:33 +0000

 On 11/3/26 20:20, Taylor R Campbell wrote:
 > Which columns were you counting?
 > 
 > [...]

 It is increasingly clear to me now I might have counted the wrong ones, 
 and generally the numbers I gave are very confused.

 I apologize!

 - The actually available memory seems to be around 10GB. I found this 
 out via userspace memtest: https://horse64.org/temp/memtest.png This 
 seems a bit regrettably low and explains why I run into OOMs relatively 
 fast.

 - Nevertheless, OOM seems to mean system lockup. This seems to be 
 reflected here as if this is common behavior for ZFS: 
 https://wiki.netbsd.org/zfs/#index5h1 "4GB systems are known to lock up."

 - I still feel like this would be improvable by stopping user space 
 programs early enough...? Via some earlyoom-like service.

 > 
 > FYI, you should expect to only rarely have substantial amounts of
 > `Free' memory in top(1) -- e.g., shortly after memory-intensive
 > processes have exited or large cached files have been deleted, but
 > before the pages have been reused to cache file data.
 > 

 Seems like for me the free amount is typically 4+ digits megabytes.

 Once the "free" goes below 4 digits, eventually things seem to South.

 But I wouldn't know how to interpret this number well, clearly.

 > [...]> 
 > You don't need to interact with the login process.  The logic of
 > switching virtual terminals mostly happens in the kernel, and the
 > logic of entering ddb with C-A-ESC is entirely in the kernel.

 I did try this now, after running out of the 10GB again and having X 
 crash or get killed:

 Sadly, it didn't work. I was staring at the X startup messages, shown to 
 me after X quit, but with no prompt. Nothing happened when pressing any 
 of CTRL+ALT+Escape or CTRL+ALT+F1. I tried multiple times.



 Sorry again for my previous emails containing a lot of unconfirmed and 
 in many parts wrong numbers. I still hope that it inspires some ideas.

 Regards,

 Ellie


From: Ellie <el@horse64.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: misc/60958 ((Probaby) avoidable stability problems due to default
 memory management behavior on systems with a lot of RAM, ZFS, and no swap)
Date: Wed, 4 Nov 2026 23:08:29 +0000

 On 11/4/26 01:40, Ellie via gnats wrote:
 >   - The actually available memory seems to be around 10GB. 
 Turns out this is just the mlock limit that the memtester is hitting, 
 and I have actually no idea what is going on. Other than I have some 
 sort of stability problem, and all the numbers I thought were related 
 are likely unrelated. :-| Sorry for wasting everyone's time.

 I'll continue memory testing some more, but so far nothing showed up and 
 just a week ago Linux had no stability problems.

 Sorry for being so clueless and misinterpreting everything so far!

 Sucks that doing the CTRL+ALT combos don't seem to work for me either 
 once it locked up, for whatever reason it does so.

 Regards,

 Ellie

From: Ellie <el@horse64.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: misc/60958: (Probaby) avoidable stability problems due to default
 memory management behavior on systems with a lot of RAM, ZFS, and no swap
Date: Sat, 7 Nov 2026 20:05:50 +0000

 On 11/3/26 17:30, gnats-admin@netbsd.org wrote:
 > Thank you very much for your problem report.
 > It has the internal identification `misc/60958'.
 > The individual assigned to look at your
 > report is: misc-bug-people.
 > 
 >> Category:       misc
 >> Responsible:    misc-bug-people
 >> Synopsis:       (Probaby) avoidable stability problems due to default memory management behavior on systems with a lot of RAM, ZFS, and no swap
 >> Arrival-Date:   Tue Nov 03 17:30:01 +0000 2026
 > 

 Dear bugs team,

 I've finally found the source of the freezes and it seems to be some 
 incompatibility with the USB enclosure I used for the system disk: 
 https://mail-index.netbsd.org/netbsd-users/2026/11/07/msg035236.html

 Sorry again for running down the completely wrong road at first, and I 
 would suggest to perhaps just close this ticket given the initial 
 premise seems completely disjointed from what the problem turned out to be.

 If I should file a new ticket for this incompatibility let me know, 
 although I'm honestly unsure if it's important enough to anybody to look 
 into.

 Regards,

 Ellie

>Unformatted:

Home	PR Database Search