NetBSD Problem Report #54988

From mlh@goathill.org  Wed Feb 19 18:26:12 2020
Return-Path: <mlh@goathill.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id C8F621A9213
	for <gnats-bugs@gnats.NetBSD.org>; Wed, 19 Feb 2020 18:26:12 +0000 (UTC)
Message-Id: <20200219182610.1C58F12C47@chopper.goathill.org>
Date: Wed, 19 Feb 2020 13:26:10 -0500 (EST)
From: mlh@goathill.org
Reply-To: mlh@goathill.org
To: gnats-bugs@NetBSD.org
Subject: possible memory leaks/swap problems
X-Send-Pr-Version: 3.95

>Number:         54988
>Category:       port-amd64
>Synopsis:       system freezes shortly after physical memory is exhausted.
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    ad
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Feb 19 18:30:00 +0000 2020
>Last-Modified:  Fri Jun 12 13:20:02 +0000 2020
>Originator:     mlh
>Release:        NetBSD 9.99.46 Thu Feb 13 13:00:32 EST 2020
>Organization:
none
>Environment:
System: NetBSD tiamat 9.99.46 NetBSD 9.99.46 (HDMIAUDIO) #0: Thu Feb 13 13:00:32 EST 2020 mlh@tiamat:/opt/obj/amd64/opt/src/sys/arch/amd64/compile/HDMIAUDIO amd64
Architecture: x86_64
Machine: amd64
amd64 GENERIC + options HDAUDIO_ENABLE_HDMI
>Description:
When doing simple build operations such as a csv update + building
a std distribution, physical memory is exhausted fairly quickly and
appears to not be returned and when physical mem is exhausted, the
system simply freezes as though something is wrong with swap operations.
I have tried using several different swap partitions and from different
disks but it doesn't appear to change the result. I have 4G of physical
memory and I can't even finish building a std distribution without
having to cycle power to reboot in order to contiune. I can cause the
problem to occur within a few minutes by using pkg_chk to load a list
of prebuilt binary pkgsrc packages. It freezes before even 300 packages
are loaded. It might be a problem with disk cache operations but
not sure. Sometimes phy mem appears to be exhausted by application
but it can happen when application use is only around 20% and the
remainder is disk cache, as with loading binary packages. I can stop the
task before phy mem is exhausted but I see no evidence that any
appreciable amount of memory consumed is ever returned. I can stop when
vmstat fre shows down to around 20000 and until I reboot, free memory
stays pretty much at that point. I can even shut down X and most else
and still very little memory is available.
-current as of the last couple of weeks (that I have tried) behaves this way.
Build from late Jan worked fine.

>How-To-Repeat:
$ cd /usr/src
$ cvs up
$ /usr/src/build.sh -j2 -x -r -U -u -m amd64 -D /opt/obj/amd64/build -M /opt/obj/amd64 -T /opt/obj/amd64/tools -R /opt/build/9 tools release
>Fix:

>Release-Note:

>Audit-Trail:
From: mlh@goathill.org (MLH)
To: gnats-bugs@netbsd.org
Cc: mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Wed, 19 Feb 2020 16:56:27 -0500 (EST)

 gnats-admin@netbsd.org wrote:
 > Thank you very much for your problem report.
 > It has the internal identification `port-amd64/54988'.
 > The individual assigned to look at your
 > report is: port-amd64-maintainer. 
 > 
 > >Category:       port-amd64
 > >Responsible:    port-amd64-maintainer
 > >Synopsis:       system freezes shortly after physical memory is exhausted.
 > >Arrival-Date:   Wed Feb 19 18:30:00 +0000 2020

 I noticed that the main consumers appear to be:

 $ vmstat -m
 Memory resource pool statistics
 Name        Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
 ...
 ataspl       160  1264935    0  1264935     3     0     3     3     0   inf    3
 ...
 execargs    262144 1775776   0  1775776     8     0     8     8     0    16    8
 ...
 wapblinopl    40  2723208    0  2721719    15     0    15    15     0   inf    0

From: Joerg Sonnenberger <joerg@bec.de>
To: gnats-bugs@netbsd.org
Cc: port-amd64-maintainer@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org, mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Wed, 19 Feb 2020 23:33:22 +0100

 On Wed, Feb 19, 2020 at 10:00:02PM +0000, MLH wrote:
 > The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 > 
 > From: mlh@goathill.org (MLH)
 > To: gnats-bugs@netbsd.org
 > Cc: mlh@goathill.org
 > Subject: Re: port-amd64/54988: possible memory leaks/swap problems
 > Date: Wed, 19 Feb 2020 16:56:27 -0500 (EST)
 > 
 >  gnats-admin@netbsd.org wrote:
 >  > Thank you very much for your problem report.
 >  > It has the internal identification `port-amd64/54988'.
 >  > The individual assigned to look at your
 >  > report is: port-amd64-maintainer. 
 >  > 
 >  > >Category:       port-amd64
 >  > >Responsible:    port-amd64-maintainer
 >  > >Synopsis:       system freezes shortly after physical memory is exhausted.
 >  > >Arrival-Date:   Wed Feb 19 18:30:00 +0000 2020
 >  
 >  I noticed that the main consumers appear to be:
 >  
 >  $ vmstat -m
 >  Memory resource pool statistics
 >  Name        Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
 >  ...
 >  ataspl       160  1264935    0  1264935     3     0     3     3     0   inf    3
 >  ...
 >  execargs    262144 1775776   0  1775776     8     0     8     8     0    16    8
 >  ...
 >  wapblinopl    40  2723208    0  2721719    15     0    15    15     0   inf    0

 If requests ~= releases, it is no memory leak.

 Joerg

From: mlh@goathill.org (MLH)
To: Joerg Sonnenberger <joerg@bec.de>
Cc: gnats-bugs@netbsd.org, port-amd64-maintainer@netbsd.org, 
 gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Wed, 19 Feb 2020 23:13:41 -0500 (EST)

 Joerg Sonnenberger wrote:
 > On Wed, Feb 19, 2020 at 10:00:02PM +0000, MLH wrote:
 > > The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 > > 
 > > From: mlh@goathill.org (MLH)
 > > To: gnats-bugs@netbsd.org
 > > Cc: mlh@goathill.org
 > > Subject: Re: port-amd64/54988: possible memory leaks/swap problems
 > > Date: Wed, 19 Feb 2020 16:56:27 -0500 (EST)
 > > 
 > >  gnats-admin@netbsd.org wrote:
 > >  > Thank you very much for your problem report.
 > >  > It has the internal identification `port-amd64/54988'.
 > >  > The individual assigned to look at your
 > >  > report is: port-amd64-maintainer. 
 > >  > 
 > >  > >Category:       port-amd64
 > >  > >Responsible:    port-amd64-maintainer
 > >  > >Synopsis:       system freezes shortly after physical memory is exhausted.
 > >  > >Arrival-Date:   Wed Feb 19 18:30:00 +0000 2020
 > >  
 > >  I noticed that the main consumers appear to be:
 > >  
 > >  $ vmstat -m
 > >  Memory resource pool statistics
 > >  Name        Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
 > >  ...
 > >  ataspl       160  1264935    0  1264935     3     0     3     3     0   inf    3
 > >  ...
 > >  execargs    262144 1775776   0  1775776     8     0     8     8     0    16    8
 > >  ...
 > >  wapblinopl    40  2723208    0  2721719    15     0    15    15     0   inf    0
 > 
 > If requests ~= releases, it is no memory leak.

 Any idea on what is preventing the memory from being released and
 why it just locks up?

From: Joerg Sonnenberger <joerg@bec.de>
To: gnats-bugs@netbsd.org
Cc: port-amd64-maintainer@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org, mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Thu, 20 Feb 2020 08:59:14 +0100

 On Thu, Feb 20, 2020 at 04:15:01AM +0000, MLH wrote:
 > The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 > 
 > From: mlh@goathill.org (MLH)
 > To: Joerg Sonnenberger <joerg@bec.de>
 > Cc: gnats-bugs@netbsd.org, port-amd64-maintainer@netbsd.org, 
 >  gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, mlh@goathill.org
 > Subject: Re: port-amd64/54988: possible memory leaks/swap problems
 > Date: Wed, 19 Feb 2020 23:13:41 -0500 (EST)
 > 
 >  Joerg Sonnenberger wrote:
 >  > On Wed, Feb 19, 2020 at 10:00:02PM +0000, MLH wrote:
 >  > > The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 >  > > 
 >  > > From: mlh@goathill.org (MLH)
 >  > > To: gnats-bugs@netbsd.org
 >  > > Cc: mlh@goathill.org
 >  > > Subject: Re: port-amd64/54988: possible memory leaks/swap problems
 >  > > Date: Wed, 19 Feb 2020 16:56:27 -0500 (EST)
 >  > > 
 >  > >  gnats-admin@netbsd.org wrote:
 >  > >  > Thank you very much for your problem report.
 >  > >  > It has the internal identification `port-amd64/54988'.
 >  > >  > The individual assigned to look at your
 >  > >  > report is: port-amd64-maintainer. 
 >  > >  > 
 >  > >  > >Category:       port-amd64
 >  > >  > >Responsible:    port-amd64-maintainer
 >  > >  > >Synopsis:       system freezes shortly after physical memory is exhausted.
 >  > >  > >Arrival-Date:   Wed Feb 19 18:30:00 +0000 2020
 >  > >  
 >  > >  I noticed that the main consumers appear to be:
 >  > >  
 >  > >  $ vmstat -m
 >  > >  Memory resource pool statistics
 >  > >  Name        Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
 >  > >  ...
 >  > >  ataspl       160  1264935    0  1264935     3     0     3     3     0   inf    3
 >  > >  ...
 >  > >  execargs    262144 1775776   0  1775776     8     0     8     8     0    16    8
 >  > >  ...
 >  > >  wapblinopl    40  2723208    0  2721719    15     0    15    15     0   inf    0
 >  > 
 >  > If requests ~= releases, it is no memory leak.
 >  
 >  Any idea on what is preventing the memory from being released and
 >  why it just locks up?

 I don't know, I'm just saying that you are looking at the wrong pools.
 Look for those with high Npage, those are actually big.

 Joerg

From: mlh@goathill.org (MLH)
To: Joerg Sonnenberger <joerg@bec.de>
Cc: gnats-bugs@netbsd.org, port-amd64-maintainer@netbsd.org, 
 gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Fri, 21 Feb 2020 08:09:59 -0500 (EST)

 Joerg Sonnenberger wrote:
 > On Thu, Feb 20, 2020 at 04:15:01AM +0000, MLH wrote:
 > >  
 > >  Any idea on what is preventing the memory from being released and
 > >  why it just locks up?
 > 
 > I don't know, I'm just saying that you are looking at the wrong pools.
 > Look for those with high Npage, those are actually big.

 The problem continues to appear to be a filesystem-related issue
 as I can do compute-bound jobs with no problem. Physical memory is
 recovered as it normally does with no issue. Large or intensive
 filesystem writes appear to cause the system to seize and it doesn't
 even appear to require physical memory to be exhausted as it just
 seized twice with vmstat showing over 2G of available physical
 memory, and physical memory isn't showing to be recovered after
 intensive fs writes when I stop it before the system seizes.

 Have any filesystem-related changes been done recently?

From: mlh@goathill.org (MLH)
To: gnats-bugs@netbsd.org
Cc: port-amd64-maintainer@netbsd.org, gnats-admin@netbsd.org, 
 netbsd-bugs@netbsd.org, mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Sat, 22 Feb 2020 10:40:11 -0500 (EST)

 MLH wrote:
 > From: mlh@goathill.org (MLH)
 > 
 >  Joerg Sonnenberger wrote:
 >  > On Thu, Feb 20, 2020 at 04:15:01AM +0000, MLH wrote:
 >  > >  
 >  > >  Any idea on what is preventing the memory from being released and
 >  > >  why it just locks up?
 >  > 
 >  > I don't know, I'm just saying that you are looking at the wrong pools.
 >  > Look for those with high Npage, those are actually big.
 >  
 >  The problem continues to appear to be a filesystem-related issue
 >  as I can do compute-bound jobs with no problem. Physical memory is
 >  recovered as it normally does with no issue. Large or intensive
 >  filesystem writes appear to cause the system to seize and it doesn't
 >  even appear to require physical memory to be exhausted as it just
 >  seized twice with vmstat showing over 2G of available physical
 >  memory, and physical memory isn't showing to be recovered after
 >  intensive fs writes when I stop it before the system seizes.
 >  
 >  Have any filesystem-related changes been done recently?

 Such as (from CHANGES) :

         uvm: More precisely track clean/dirty pages, and change how they are
                 indexed, speeding up fsync() on large files by orders of
                 magnitude.  Original work done by yamt@. [ad 20200115]

 as all was fine just before this change and then with kernels from
 the last of Jan on, I am seeing the problems.

Responsible-Changed-From-To: port-amd64-maintainer->ad
Responsible-Changed-By: ad@NetBSD.org
Responsible-Changed-When: Sat, 22 Feb 2020 18:19:31 +0000
Responsible-Changed-Why:
will take a look


From: Andrew Doran <ad@netbsd.org>
To: MLH <mlh@goathill.org>
Cc: gnats-bugs@netbsd.org, port-amd64-maintainer@netbsd.org,
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Sat, 22 Feb 2020 18:17:24 +0000

 Very interesting, seems like there are a couple of low memory scenarios
 where the system is hanging up.  The last time I did a stress test on this
 scenario was early January and everything was fine.  I'll try another.

 Andrew

From: mlh@goathill.org (MLH)
To: gnats-bugs@netbsd.org
Cc: ad@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
 mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Sat, 22 Feb 2020 13:39:58 -0500 (EST)

 Andrew Doran wrote:
 > The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 > 
 > From: Andrew Doran <ad@netbsd.org>
 > 
 >  Very interesting, seems like there are a couple of low memory scenarios
 >  where the system is hanging up.  The last time I did a stress test on this
 >  scenario was early January and everything was fine.  I'll try another.
 >  
 >  Andrew

 Yes, please test.  A kernel from about the second week in January
 worked fine for me also and that was the last one which did fine.
 I can trigger a freeze in all sorts of ways with 9.99.4[3-6] all
 of which appear to involve either large files or low physical
 memory.

From: Andrew Doran <ad@netbsd.org>
To: MLH <mlh@goathill.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Sun, 23 Feb 2020 21:55:48 +0000

 Stress testing under low memory conditions I'm not able to reproduce this,
 however the machine is headless.  I'll try more I/O intensive stuff when I
 get a chance.

 If you are still able to run vmstat, then the output of "vmstat -s" say
 every 10 seconds over the course of a minute would be very useful.  If you
 have "top -t" running, it would be interesting to observe if pgdaemon is
 consuming a lot of CPU time.

 Andrew

From: Andrew Doran <ad@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Sun, 23 Feb 2020 22:02:08 +0000

 What is the file system / disk configuration on this machine?  Are you
 running any processes that consume a lot of wired memory by chance?

 Andrew

From: mlh@goathill.org (MLH)
To: gnats-bugs@netbsd.org
Cc: ad@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
 mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Sun, 23 Feb 2020 17:30:23 -0500 (EST)

 Andrew Doran wrote:
 > The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 > 
 >  What is the file system / disk configuration on this machine?  Are you
 >  running any processes that consume a lot of wired memory by chance?

 $ df
 Filesystem    1K-blocks       Used      Avail %Cap Mounted on
 /dev/wd0a      71714288   38691140   29437434  56% /
 /dev/wd1f     946611748  285748620  613532544  31% /opt
 kernfs                1          1          0 100% /kern
 ptyfs                 1          1          0 100% /dev/pts
 procfs                4          4          0 100% /proc
 tmpfs           1044332          0    1044332   0% /var/shm

 Result is the same without /opt

 / is a completely new installation of NetBSD 9.99.46 GENERIC with
 HDMI audio compile option with my previous /etc environment merged
 with it but works the same with a GENERIC kernel. I repartitioned
 the drive and reformatted wd0a before installing.

 [     5.259157] wd0 at atabus2 drive 0
 [     5.259157] wd0: <ST380011A>
 [     5.259157] wd0: drive supports 16-sector PIO transfers, LBA48 addressing
 [     5.259157] wd0: 76319 MB, 155061 cyl, 16 head, 63 sec, 512 bytes/sect x 156301488 sectors
 [     5.319178] wd0: 32-bit data port
 [     5.319178] wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
 [     5.319178] wd0(jmide0:1:0): using PIO mode 4, Ultra-DMA mode 2 (Ultra/33) (using DMA)
 [     5.319178] wd1 at atabus3 drive 0
 [     5.319178] wd1: <ST1000DM003-1ER162>
 [     5.319178] wd1: drive supports 16-sector PIO transfers, LBA48 addressing
 [     5.319178] wd1: 931 GB, 1938021 cyl, 16 head, 63 sec, 512 bytes/sect x 1953525168 sectors
 [     5.379200] wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133), WRITE DMA FUA, NCQ (32 tags)
 [     5.379200] wd1(ahcisata1:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA), NCQ (31 tags)

 I also can have wd2/3 in a raid configuration mounted but the
 behavior is similar without it except that when mounted, the system
 runs out of phymem more quickly

 [     5.379200] wd2 at atabus5 drive 0
 [     5.379200] wd2: <ST4000DM004-2CV104>
 [     5.379200] wd2: drive supports 16-sector PIO transfers, LBA48 addressing
 [     5.379200] wd2: 3726 GB, 7752021 cyl, 16 head, 63 sec, 512 bytes/sect x 7814037168 sectors
 [     5.399207] wd2: GPT GUID: f10e8bb0-de35-4b54-b365-6571da54d265
 [     5.399207] dk0 at wd2: "boot0", 100000 blocks at 34, type: ffs
 [     5.399207] dk1 at wd2: "swap0", 10000000 blocks at 100034, type: swap
 [     5.399207] dk2 at wd2: "raidw0", 7803937101 blocks at 10100034, type: raidframe
 [     5.409211] wd2: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133), WRITE DMA FUA, NCQ (32 tags)
 [     5.409211] wd2(ahcisata1:4:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA), NCQ (31 tags)
 [     5.409211] wd3 at atabus6 drive 0
 [     5.409211] wd3: <ST4000DM004-2CV104>
 [     5.409211] wd3: drive supports 16-sector PIO transfers, LBA48 addressing
 [     5.409211] wd3: 3726 GB, 7752021 cyl, 16 head, 63 sec, 512 bytes/sect x 7814037168 sectors
 [     5.429218] wd3: GPT GUID: b3d4301a-cbfe-414e-ac61-16571d17d548
 [     5.429218] dk3 at wd3: "boot1", 100000 blocks at 34, type: ffs
 [     5.429218] dk4 at wd3: "swap1", 10000000 blocks at 100034, type: swap
 [     5.429218] dk5 at wd3: "raidw1", 7803937101 blocks at 10100034, type: raidframe

 I also get similar results when booting off of dk6(raid0) without
 wd0/1 mounted.  The raid0 has the same install environment as wd0a
 but it also failed the same way right after it was pristeenly
 installed, without any of my older environment and with a GENERIC
 kernel.

From: mlh@goathill.org (MLH)
To: gnats-bugs@netbsd.org
Cc: ad@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
 mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Thu, 27 Feb 2020 20:01:36 -0500 (EST)

 Andrew Doran wrote:
 > The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 > 
 > From: Andrew Doran <ad@netbsd.org>
 > To: MLH <mlh@goathill.org>
 > Cc: gnats-bugs@netbsd.org
 > Subject: Re: port-amd64/54988: possible memory leaks/swap problems
 > Date: Sun, 23 Feb 2020 21:55:48 +0000
 > 
 >  Stress testing under low memory conditions I'm not able to reproduce this,
 >  however the machine is headless.  I'll try more I/O intensive stuff when I
 >  get a chance.
 >  
 >  If you are still able to run vmstat, then the output of "vmstat -s" say
 >  every 10 seconds over the course of a minute would be very useful.  If you
 >  have "top -t" running, it would be interesting to observe if pgdaemon is
 >  consuming a lot of CPU time.

 Unfortunately I saw there were updates to uvm so rebuilt everything
 and now it crashes towards the end of X starting up, right after
 Initializing extension GLX.

 [    57.167] (II) Initializing extension X-Resource
 [    57.167] (II) Initializing extension XVideo
 [    57.168] (II) Initializing extension XVideo-MotionCompensation
 [    57.168] (II) Initializing extension GLX
 crashes here

 (gdb) target kvm netbsd.1.core  
 _kvm_kvatop(0)
 (gdb) bt
 _kvm_kvatop(0)

 So I am having to boot an earlier kernel off of a thumbdrive to
 bring the box up from NetBSD 9.99.41 Thu Jan 23 13:54:41 EST 2020
 Still has the freeze that looks like a uvm issue.

 This is what normally follows Initializing extension GLX :

 [    75.828] (II) AIGLX: Loaded and initialized r600 
 [    75.828] (II) GLX: Initialized DRI2 GL provider for screen 0
 [    75.828] (II) Initializing extension XFree86-VidModeExtension
 [    75.829] (II) Initializing extension XFree86-DGA
 [    75.829] (II) Initializing extension XFree86-DRI
 [    75.829] (II) Initializing extension DRI2
 [    75.881] (II) RADEON(0): Setting screen physical size to 1016 x 317
 [    75.881] (II) RADEON(0): Allocate new frame buffer 3840x1200
 [    75.926] (II) RADEON(0): VRAM usage limit set to 1858971K
 [    78.235] (II) Using input driver 'mouse' for 'Mouse0'
 [    78.235] (**) Option "CorePointer"
 [    78.235] (**) Mouse0: always reports core events
 [    78.277] (**) Option "Protocol" "wsmouse" 
 [    78.277] (**) Option "Device" "/dev/wsmouse" 
 [    78.277] (**) Mouse0: Protocol: "wsmouse"
 [    78.277] (**) Mouse0: always reports core events
 [    78.278] (==) Mouse0: Emulate3Buttons, Emulate3Timeout: 50
 [    78.278] (**) Option "ZAxisMapping" "4 5 6 7"
 [    78.278] (**) Mouse0: ZAxisMapping: buttons 4, 5, 6 and 7
 [    78.278] (**) Mouse0: Buttons: 11
 [    78.278] (II) XINPUT: Adding extended input device "Mouse0" (type: MOUSE, id
 [    78.278] (**) Mouse0: (accel) keeping acceleration scheme 1
 [    78.278] (**) Mouse0: (accel) acceleration profile 0
 [    78.278] (**) Mouse0: (accel) acceleration factor: 2.000
 [    78.278] (**) Mouse0: (accel) acceleration threshold: 4
 [    78.278] (II) Using input driver 'kbd' for 'Keyboard0'
 [    78.278] (**) Option "CoreKeyboard"
 [    78.278] (**) Keyboard0: always reports core events
 [    78.278] (**) Keyboard0: always reports core events
 [    78.278] (**) Option "Protocol" "standard"
 [    78.278] (**) Option "XkbRules" "base"
 [    78.278] (**) Option "XkbModel" "pc105"
 [    78.279] (**) Option "XkbLayout" "us"
 [    78.279] (**) Option "XkbOptions" "ctrl:swapcaps"
 [    78.279] (II) XINPUT: Adding extended input device "Keyboard0" (type: KEYBOA
 [    92.114] (II) RADEON(0): Allocate new frame buffer 1920x1200
 [    92.115] (II) RADEON(0): VRAM usage limit set to 1867071K
 [    92.154] (II) RADEON(0): Allocate new frame buffer 3840x1200
 [    92.159] (II) RADEON(0): VRAM usage limit set to 1858971K


From: mlh@goathill.org (MLH)
To: gnats-bugs@netbsd.org
Cc: port-amd64-maintainer@netbsd.org, gnats-admin@netbsd.org, 
 netbsd-bugs@netbsd.org, mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Wed, 4 Mar 2020 16:30:04 -0500 (EST)

 MLH wrote:
 > The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 >  
 >  Any idea on what is preventing the memory from being released and
 >  why it just locks up?

 With the changes to -current as of NetBSD 9.99.48 Sun Mar  1 17:38:46
 EST 2020, the complete lockups I was seeing when the 4G of phymem
 was exhausted as well as the other issues seems to have been somewhat
 resolved.

 The largest improvements were last week when the big uvm changes
 were checked in but I haven't seen a lockup since the last couple
 of changes made on Sat or Sun. Those appear to have finally made
 this system stable enough to use again, though there still appear
 to be memory alloc/dealloc and swap issues because I have to reboot
 at least once a day when phymem is exhausted and the system becomes
 unusable due to severe swapping. This is when basically nothing is
 still running and disc cache is only taking up a tiny bit of phymem
 and only a tiny bit of swap appears to be in use.

From: Andrew Doran <ad@netbsd.org>
To: MLH <mlh@goathill.org>
Cc: gnats-bugs@netbsd.org, port-amd64-maintainer@netbsd.org,
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Wed, 4 Mar 2020 21:58:13 +0000

 On Wed, Mar 04, 2020 at 04:30:04PM -0500, MLH wrote:

 > MLH wrote:
 > > The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 > >  
 > >  Any idea on what is preventing the memory from being released and
 > >  why it just locks up?
 > 
 > With the changes to -current as of NetBSD 9.99.48 Sun Mar  1 17:38:46
 > EST 2020, the complete lockups I was seeing when the 4G of phymem
 > was exhausted as well as the other issues seems to have been somewhat
 > resolved.
 > 
 > The largest improvements were last week when the big uvm changes
 > were checked in but I haven't seen a lockup since the last couple
 > of changes made on Sat or Sun. Those appear to have finally made
 > this system stable enough to use again, though there still appear
 > to be memory alloc/dealloc and swap issues because I have to reboot
 > at least once a day when phymem is exhausted and the system becomes
 > unusable due to severe swapping. This is when basically nothing is
 > still running and disc cache is only taking up a tiny bit of phymem
 > and only a tiny bit of swap appears to be in use.

 Thanks for reporting.  Not to put a downer on things, but the UVM jumbo
 commit was largely a mechnical one to change lock types. Any corrective
 effect is by chance.  Still investigation to do here.

 Cheers,
 Andrew

From: mlh@goathill.org (MLH)
To: gnats-bugs@netbsd.org
Cc: port-amd64-maintainer@netbsd.org,
	gnats-admin@netbsd.goathill.org, port-amd64-maintainer@netbsd.org,
	netbsd-bugs@netbsd.org, mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Tue, 10 Mar 2020 12:54:39 -0400 (EDT)

 mlh wrote:
 > MLH wrote:
 > > The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 > >  
 > >  Any idea on what is preventing the memory from being released and
 > >  why it just locks up?
 ...
 > though there still appear
 > to be memory alloc/dealloc and swap issues because I have to reboot
 > at least once a day when phymem is exhausted and the system becomes
 > unusable due to severe swapping. This is when basically nothing is
 > still running and disc cache is only taking up a tiny bit of phymem
 > and only a tiny bit of swap appears to be in use.

 Example - reboot, run one console without X, cvs up -current and
 build a -current distribution. Results in basically a nonworking
 system since there is only a few tens of Mbytes of ram available
 for the system to run in. Less than 25 basic processes running
 after the build and none of them are really active.

 NetBSD 9.99.48 Sat Mar  7 11:19:02 EST 2020 amd64

 [     1.000000] total memory = 4079 MB
 [     1.000000] avail memory = 3933 MB
 [     1.000000] sysctl_createv: sysctl_locate(maxtypenum) returned 2
 [     1.000000] pool redzone disabled for 'buf4k'
 [     1.000000] pool redzone disabled for 'buf64k'
 [     1.000000] timecounter: Timecounters tick every 10.000 msec
 [     1.000000] Kernelized RAIDframe activated
 [     1.000000] running cgd selftest aes-xts-256 aes-xts-512 done
 [     1.000000] timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
 [     1.000003] Gigabyte Technology Co., Ltd. H61M-S2-B3 ( )
 [     1.000003] mainbus0 (root)
 [     1.000003] ACPI: RSDP 0x00000000000F6EA0 000014 (v00 GBT   )
 [     1.000003] ACPI: RSDT 0x00000000DF7D3040 00004C (v01 GBT    GBTUACPI 42302E31 GBTU 01010101)
 [     1.000003] ACPI: FACP 0x00000000DF7D3100 000074 (v01 GBT    GBTUACPI 42302E31 GBTU 01010101)
 [     1.000003] ACPI: DSDT 0x00000000DF7D31C0 0049F2 (v01 GBT    GBTUACPI 00001000 MSFT 04000000)
 [     1.000003] ACPI: FACS 0x00000000DF7D0000 000040
 [     1.000003] ACPI: MSDM 0x00000000DF7D7D00 000055 (v03 GBT    GBTUACPI 42302E31 GBTU 01010101)
 [     1.000003] ACPI: HPET 0x00000000DF7D7DC0 000038 (v01 GBT    GBTUACPI 42302E31 GBTU 00000098)
 [     1.000003] ACPI: MCFG 0x00000000DF7D7E40 00003C (v01 GBT    GBTUACPI 42302E31 GBTU 01010101)
 [     1.000003] ACPI: ASPT 0x00000000DF7D7F00 000034 (v07 GBT    PerfTune 312E3042 UTBG 01010101)
 [     1.000003] ACPI: SSPT 0x00000000DF7D7F40 002270 (v01 GBT    SsptHead 312E3042 UTBG 01010101)
 [     1.000003] ACPI: EUDS 0x00000000DF7DA1B0 0000C0 (v01 GBT             00000000      00000000)
 [     1.000003] ACPI: TAMG 0x00000000DF7DA270 000382 (v01 GBT    GBT   B0 5455312E BG?? 45240101)
 [     1.000003] ACPI: APIC 0x00000000DF7D7C00 0000BC (v01 GBT    GBTUACPI 42302E31 GBTU 01010101)
 [     1.000003] ACPI: SSDT 0x00000000DF7DA600 001EC8 (v01 INTEL  PPM RCM  80000001 INTL 20061109)
 [     1.000003] ACPI: 2 ACPI AML tables successfully acquired and loaded
 [     1.000003] ioapic0 at mainbus0 apid 2: pa 0xfec00000, version 0x20, 24 pins
 [     1.000003] cpu0 at mainbus0 apid 0
 [     1.000003] cpu0: Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz, id 0x206a7
 [     1.000003] cpu0: node 0, package 0, core 0, smt 0
 [     1.000003] cpu1 at mainbus0 apid 2
 [     1.000003] cpu1: Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz, id 0x206a7
 [     1.000003] cpu1: node 0, package 0, core 1, smt 0
 [     1.000003] cpu2 at mainbus0 apid 1
 [     1.000003] cpu2: Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz, id 0x206a7
 [     1.000003] cpu2: node 0, package 0, core 0, smt 1
 [     1.000003] cpu3 at mainbus0 apid 3
 [     1.000003] cpu3: Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz, id 0x206a7
 [     1.000003] cpu3: node 0, package 0, core 1, smt 1

 $ vmstat
  procs    memory      page                       disks   faults      cpu
  r b      avm    fre  flt  re  pi   po   fr   sr w0 w1   in   sy  cs us sy id
  0 0    89556 3809396 1605  0   0    0    0    0 68 37  146 3059 360  0  1 99
 $ cd /usr/xsrc
 $ cvs up
 ...
 $ cd /usr/src
 $ cvs up
 ...
 $ /usr/src/build.sh -j2 -x -r -U -u -m amd64 -D /opt/obj/amd64/build -M /opt/obj/amd64 -T /opt/obj/amd64/tools -R /opt/build/9 tools release install-image
 ...

 finish - taking almost 6 hours, about twice as long as typical
 likely due to thrashing

 $ vmstat 
  procs    memory      page                       disks   faults      cpu
  r b      avm    fre  flt  re  pi   po   fr   sr w0 w1   in   sy  cs us sy id
  1 0   535932  17812 16003  7   4   12  330  375  7 46  255 15441 1525 13 4 84

 So the system is down to less than 20M of phymem and never recovers
 much more than maybe 10M of it. Have to reboot to reclaim a workable
 system, so I have to reboot 2-3 times a day to do much of any work.

From: Lars Reichardt <lars@paradoxon.info>
To: MLH <mlh@goathill.org>, gnats-bugs@netbsd.org
Cc: port-amd64-maintainer@netbsd.org, gnats-admin@netbsd.goathill.org,
 netbsd-bugs@netbsd.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Tue, 10 Mar 2020 18:29:43 +0100

 On 10.03.2020 17:54, MLH wrote:
 > mlh wrote:
 >> MLH wrote:
 >>> The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 >>>   
 >>>   Any idea on what is preventing the memory from being released and
 >>>   why it just locks up?
 > ...
 >> though there still appear
 >> to be memory alloc/dealloc and swap issues because I have to reboot
 >> at least once a day when phymem is exhausted and the system becomes
 >> unusable due to severe swapping. This is when basically nothing is
 >> still running and disc cache is only taking up a tiny bit of phymem
 >> and only a tiny bit of swap appears to be in use.
 > Example - reboot, run one console without X, cvs up -current and
 > build a -current distribution. Results in basically a nonworking
 > system since there is only a few tens of Mbytes of ram available
 > for the system to run in. Less than 25 basic processes running
 > after the build and none of them are really active.
 >
 > NetBSD 9.99.48 Sat Mar  7 11:19:02 EST 2020 amd64
 >
 > [     1.000000] total memory = 4079 MB
 > [     1.000000] avail memory = 3933 MB
 > [     1.000000] sysctl_createv: sysctl_locate(maxtypenum) returned 2
 > [     1.000000] pool redzone disabled for 'buf4k'
 > [     1.000000] pool redzone disabled for 'buf64k'
 > [     1.000000] timecounter: Timecounters tick every 10.000 msec
 > [     1.000000] Kernelized RAIDframe activated
 > [     1.000000] running cgd selftest aes-xts-256 aes-xts-512 done
 > [     1.000000] timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
 > [     1.000003] Gigabyte Technology Co., Ltd. H61M-S2-B3 ( )
 > [     1.000003] mainbus0 (root)
 > [     1.000003] ACPI: RSDP 0x00000000000F6EA0 000014 (v00 GBT   )
 > [     1.000003] ACPI: RSDT 0x00000000DF7D3040 00004C (v01 GBT    GBTUACPI 42302E31 GBTU 01010101)
 > [     1.000003] ACPI: FACP 0x00000000DF7D3100 000074 (v01 GBT    GBTUACPI 42302E31 GBTU 01010101)
 > [     1.000003] ACPI: DSDT 0x00000000DF7D31C0 0049F2 (v01 GBT    GBTUACPI 00001000 MSFT 04000000)
 > [     1.000003] ACPI: FACS 0x00000000DF7D0000 000040
 > [     1.000003] ACPI: MSDM 0x00000000DF7D7D00 000055 (v03 GBT    GBTUACPI 42302E31 GBTU 01010101)
 > [     1.000003] ACPI: HPET 0x00000000DF7D7DC0 000038 (v01 GBT    GBTUACPI 42302E31 GBTU 00000098)
 > [     1.000003] ACPI: MCFG 0x00000000DF7D7E40 00003C (v01 GBT    GBTUACPI 42302E31 GBTU 01010101)
 > [     1.000003] ACPI: ASPT 0x00000000DF7D7F00 000034 (v07 GBT    PerfTune 312E3042 UTBG 01010101)
 > [     1.000003] ACPI: SSPT 0x00000000DF7D7F40 002270 (v01 GBT    SsptHead 312E3042 UTBG 01010101)
 > [     1.000003] ACPI: EUDS 0x00000000DF7DA1B0 0000C0 (v01 GBT             00000000      00000000)
 > [     1.000003] ACPI: TAMG 0x00000000DF7DA270 000382 (v01 GBT    GBT   B0 5455312E BG?? 45240101)
 > [     1.000003] ACPI: APIC 0x00000000DF7D7C00 0000BC (v01 GBT    GBTUACPI 42302E31 GBTU 01010101)
 > [     1.000003] ACPI: SSDT 0x00000000DF7DA600 001EC8 (v01 INTEL  PPM RCM  80000001 INTL 20061109)
 > [     1.000003] ACPI: 2 ACPI AML tables successfully acquired and loaded
 > [     1.000003] ioapic0 at mainbus0 apid 2: pa 0xfec00000, version 0x20, 24 pins
 > [     1.000003] cpu0 at mainbus0 apid 0
 > [     1.000003] cpu0: Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz, id 0x206a7
 > [     1.000003] cpu0: node 0, package 0, core 0, smt 0
 > [     1.000003] cpu1 at mainbus0 apid 2
 > [     1.000003] cpu1: Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz, id 0x206a7
 > [     1.000003] cpu1: node 0, package 0, core 1, smt 0
 > [     1.000003] cpu2 at mainbus0 apid 1
 > [     1.000003] cpu2: Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz, id 0x206a7
 > [     1.000003] cpu2: node 0, package 0, core 0, smt 1
 > [     1.000003] cpu3 at mainbus0 apid 3
 > [     1.000003] cpu3: Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz, id 0x206a7
 > [     1.000003] cpu3: node 0, package 0, core 1, smt 1
 >
 > $ vmstat
 >   procs    memory      page                       disks   faults      cpu
 >   r b      avm    fre  flt  re  pi   po   fr   sr w0 w1   in   sy  cs us sy id
 >   0 0    89556 3809396 1605  0   0    0    0    0 68 37  146 3059 360  0  1 99
 > $ cd /usr/xsrc
 > $ cvs up
 > ...
 > $ cd /usr/src
 > $ cvs up
 > ...
 > $ /usr/src/build.sh -j2 -x -r -U -u -m amd64 -D /opt/obj/amd64/build -M /opt/obj/amd64 -T /opt/obj/amd64/tools -R /opt/build/9 tools release install-image
 > ...
 >
 > finish - taking almost 6 hours, about twice as long as typical
 > likely due to thrashing
 >
 > $ vmstat
 >   procs    memory      page                       disks   faults      cpu
 >   r b      avm    fre  flt  re  pi   po   fr   sr w0 w1   in   sy  cs us sy id
 >   1 0   535932  17812 16003  7   4   12  330  375  7 46  255 15441 1525 13 4 84
 >
 > So the system is down to less than 20M of phymem and never recovers
 > much more than maybe 10M of it. Have to reboot to reclaim a workable
 > system, so I have to reboot 2-3 times a day to do much of any work.
 >
 what does vmstat -mvW show when memory gets low the whole output?

 I'm interested especially in the kva pools.

 Lars

 -- 
 You will continue to suffer
 if you have an emotional reaction to everything that is said to you.
 True power is sitting back and observing everything with logic.
 If words control you that means everyone else can control you.
 Breathe and allow things to pass.

 --- Bruce Lee

From: mlh@goathill.org (MLH)
To: gnats-bugs@netbsd.org
Cc: ad@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
 mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Fri, 20 Mar 2020 19:07:09 -0400 (EDT)

 Andrew Doran wrote:
 > The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 > 
 > From: Andrew Doran <ad@netbsd.org>
 > To: gnats-bugs@netbsd.org
 > Cc: 
 > Subject: Re: port-amd64/54988: possible memory leaks/swap problems
 > Date: Sun, 23 Feb 2020 22:02:08 +0000
 > 
 >  What is the file system / disk configuration on this machine?  Are you
 >  running any processes that consume a lot of wired memory by chance?
 >  
 >  Andrew

 With the changes made yesterday as referenced by Andrew in:
 Re: Another pmap panic:

 > I suggest updaing to the latest, delivered yesterday, which has
 > fixes for every problem I have encountered or seen mentioned
 > including this one, and survives low memory stress testing for me:

 I think this can be closed as it appears to have fixed these issues.

 Thank You Andrew

From: mlh@goathill.org (MLH)
To: mlh@goathill.org
Cc: gnats-bugs@netbsd.org, ad@netbsd.org, gnats-admin@netbsd.org, 
 netbsd-bugs@netbsd.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Mon, 23 Mar 2020 10:06:12 -0400 (EDT)

 mlh wrote:
 > Andrew Doran wrote:
 > > The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 > > 
 > > From: Andrew Doran <ad@netbsd.org>
 > > To: gnats-bugs@netbsd.org
 > > Cc: 
 > > Subject: Re: port-amd64/54988: possible memory leaks/swap problems
 ... 
 > With the changes made yesterday as referenced by Andrew in:
 > Re: Another pmap panic:
 > 
 > > I suggest updaing to the latest, delivered yesterday, which has
 > > fixes for every problem I have encountered or seen mentioned
 > > including this one, and survives low memory stress testing for me:
 > 
 > I think this can be closed as it appears to have fixed these issues.

 I take that back. While the status is much better, there appears
 to still be a big memory leak as of yesterday.

 I booted NetBSD 9.99.51 : Sun Mar 22 22:42:31 and did a distribution
 build of -current. 4G of phymem, started the build with over 3.5G
 available, the build did finish without crashing but only about 6M
 of phymem is left available.

  r b      avm    fre  flt  re  pi   po   fr   sr w0 w1   in   sy  cs us sy id
  1 1   157916   6052 5540 103   5    6  362  600  6 127 660 10541 2276 3 2 96

 The box is barely usable running X and takes about 6 seconds to
 just switch focus to another window and about 15 seconds to launch
 another xterm. kmem-00192 still looks suspicious.

 Name        Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
 kmem-00192   256 12942910    2     4093 808677    0 808677 808677   0   inf    0

From: mlh@goathill.org (MLH)
To: gnats-bugs@netbsd.org
Cc: ad@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
 mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Tue, 24 Mar 2020 08:21:41 -0400 (EDT)

 Lars Reichardt wrote:
 > The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 > 
 > From: Lars Reichardt <lars@paradoxon.info>
 > To: MLH <mlh@goathill.org>, gnats-bugs@netbsd.org
 > Cc: port-amd64-maintainer@netbsd.org, gnats-admin@netbsd.goathill.org,
 >  netbsd-bugs@netbsd.org
 > Subject: Re: port-amd64/54988: possible memory leaks/swap problems
 > Date: Tue, 10 Mar 2020 18:29:43 +0100
 > 
 >  On 10.03.2020 17:54, MLH wrote:
 >  > mlh wrote:
 >  >> MLH wrote:
 >  >>> The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 >  >>>   
 >  >>>   Any idea on what is preventing the memory from being released and
 >  >>>   why it just locks up?
 >  > ...
 >  >> though there still appear
 >  >> to be memory alloc/dealloc and swap issues because I have to reboot
 >  >> at least once a day when phymem is exhausted and the system becomes
 >  >> unusable due to severe swapping. This is when basically nothing is
 >  >> still running and disc cache is only taking up a tiny bit of phymem
 >  >> and only a tiny bit of swap appears to be in use.
 ...
 >  > So the system is down to less than 20M of phymem and never recovers
 >  > much more than maybe 10M of it. Have to reboot to reclaim a workable
 >  > system, so I have to reboot 2-3 times a day to do much of any work.
 >  >
 >  what does vmstat -mvW show when memory gets low the whole output?
 >  
 >  I'm interested especially in the kva pools.
 >  
 >  Lars

 Ok. I will check that next time. As of NetBSD 9.99.51 Sun Mar 22,
 the system just grinds to a halt when it runs out of phymem. Still
 seems that almost no swap is being used but at least it doesn't
 crash. Yesterday it got down to less than 1M of phymem and took me
 about an hour to log in from my phone over wifi in order to do a
 clean shutdown as I couldn't get a console up. I had managed to
 shut X down but was getting out of memory errors. Even on shutdown
 many things couldn't cleanly shut down due to out of memory errors.

 I still suspect something in radeondrmkms is leaking memory.

From: mlh@goathill.org (MLH)
To: gnats-bugs@netbsd.org
Cc: ad@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
 mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Tue, 24 Mar 2020 17:16:30 -0400 (EDT)

 MLH wrote:
 >  >  > So the system is down to less than 20M of phymem and never recovers
 >  >  > much more than maybe 10M of it. Have to reboot to reclaim a workable
 >  >  > system, so I have to reboot 2-3 times a day to do much of any work.
 >  >  >
 >  >  what does vmstat -mvW show when memory gets low the whole output?
 >  >  
 >  >  I'm interested especially in the kva pools.
 >  >  
 >  >  Lars
 >  
 >  Ok. I will check that next time. As of NetBSD 9.99.51 Sun Mar 22,

 Two operations, build a kernel then build a -current distribution
 but didn't finish the second. It ground to a halt so I stopped it.

 Took three sets of vmstats before building the kernel, before
 building the distribution, and after stopping it and rebooting.

 Before building the kernel
 $ vmstat
  procs    memory      page                       disks   faults      cpu
  r b      avm    fre  flt  re  pi   po   fr   sr w0 w1   in   sy  cs us sy id
  2 1   823440 587248  348   0   0    0   34   46  2 20  127 2511 574  0  0 100
 $ vmstat -mvW 
 Memory resource pool statistics
 Name               Size    Requests    Fail    Releases    InUse    Avail      Pgreq      Pgrel   Npage PageSz  Hiwat Minpg  Maxpg Idle   Flags   Util
 ah_tdb_crypto       192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 aio_jobs_pool       136           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 aio_lio_pool         48           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 amappl               88       10306       0        6799     3507     4278        175          2     173   4096    175     0    inf    0 0x10040  43.6%
 anonpl               40      246829       0      193244    53585   104985       1777        207    1570   4096   1777     0    inf    1 0x01040  33.3%
 ataspl              160     2501711       0     2501711        0       50          4          2       2   4096      3     0    inf    2 0x10040   0.0%
 biopl               304       29139       0       28899      240        7       1558       1539      19   4096   1558     0    inf    0 0x10040  93.8%
 brtpl                64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 buf16k            16392       11283       0       10529      754        2       3208       2956     252  65536   1505     1      1    0 0x10040  74.8%
 buf1k              1032          59       0          59        0        3         19         18       1   4096     19     1      1    1 0x10040   0.0%
 buf2k              2056       36539       0       35314     1225        1      36540      35314    1226   4096  17763     1      1    1 0x10040  50.2%
 buf32k            32776       38146       0       28418     9728        1      38147      28418    9729  65536   9729     1      1    1 0x10040  50.0%
 buf4k              4096      137128       0       62989    74139        1     137129      62989   74140   4096  88754     1      1    1 0x10000 100.0%
 buf512b             520        1186       0        1185        1        6        138        137       1   4096    137     1      1    0 0x10040  12.7%
 buf64k            65536         240       0         232        8        1        241        232       9  65536    241     1      1    1 0x10000  88.9%
 buf8k              8200        1947       0        1483      464        5        170        103      67  65536     80     1      1    0 0x10040  86.7%
 bufpl               304       95792       0        7819    87973     7096       7369         56    7313   4096   7369     0    inf   53 0x10040  89.3%
 carp                 32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 carp6                32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 ccdbuf              336           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cd9660nopl          208           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptdesc           128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptkop            384           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptop             320           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 csepl               232           0       0           0        0      323         19          0      19   4096     19    19    inf   19 0x10040   0.0%
 cwdi                 64         247       0         157       90       99          3          0       3   4096      3     0    inf    0 0x10040  46.9%
 dbregs              144           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 dirhepl              48           0       0           0        0        0          0          0       0   4096      0     0    313    0 0x00040   ---
 dirhpl              304           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 efsinopl            248           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ehcixfer            384           7       0           3        4        6          1          0       1   4096      1     0    inf    0 0x10040  37.5%
 ehcixfer            384          18       0          14        4        6          2          1       1   4096      2     0    inf    0 0x10040  37.5%
 esp_tdb_crypto      128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 execargs         262144        2335       0        2335        0        2          8          6       2 262144      4     0     16    2 0x10c00   0.0%
 ext2fsinopl         256           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 extent               48           9       0           8        1       83          1          0       1   4096      1     0    inf    0 0x00040   1.2%
 fcrpl               184          31       0          30        1       83          4          0       4   4096      4     4    inf    3 0x10040   1.1%
 fdfile               64        3261       0        1048     2213      496         43          0      43   4096     43     0    inf    0 0x11040  80.4%
 ffsdino1            136           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ffsdino2            264      251731       0       42894   208837       38      14321        396   13925   4096  14320     0    inf    2 0x10040  96.7%
 ffsino              256      251571       0       42734   208837       43      13427        372   13055   4096  13425     0    inf    2 0x10000 100.0%
 file                128        2201       0         624     1577      345         64          2      62   4096     64     0    inf    0 0x10040  79.5%
 filedesc            832         243       0         153       90       38         38          6      32   4096     38     0    inf    0 0x10040  57.1%
 icmp                 32           2       0           2        0      126          1          0       1   4096      1     0    inf    1 0x00040   0.0%
 icmp6                32           6       0           6        0        0          2          2       0   4096      1     0    inf    0 0x00040   ---
 igmppl               40           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 in6pcbpl            280        3247       0        3234       13        1          3          2       1   4096      3     0    inf    0 0x10040  88.9%
 inmltpl              56           2       0           0        2       70          1          0       1   4096      1     0    inf    0 0x00040   2.7%
 inpcbpl             240        3215       0        3192       23        9          9          7       2   4096      9     0    inf    0 0x10040  67.4%
 ipcomp_tdb_cryp     128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ipfrenpl             64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 kcpuset              64         377       0         175      202      176          6          0       6   4096      6     0    inf    0 0x10040  52.6%
 kcredpl             192         573       0         177      396      129         25          0      25   4096     25     0    inf    0 0x10040  74.2%
 kmem-00008            8           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10400   ---
 kmem-00016           16       10441       0        6306     4135     5849         40          1      39   4096     40     0    inf    0 0x10400  41.4%
 kmem-00032           32        8014       0        4258     3756     3924         60          0      60   4096     60     0    inf    0 0x10400  48.9%
 kmem-00064          128       12205       0        7821     4384     2343        377        160     217   4096    377     0    inf    1 0x10040  63.1%
 kmem-00128          192        9384       0        2341     7043      895        379          1     378   4096    379     0    inf    0 0x10040  87.3%
 kmem-00192          256     4088775       0         182  4088593       15     255538          0  255538   4096 255538     0    inf    0 0x10000 100.0%
 kmem-00256          320         958       0         423      535      209         70          8      62   4096     70     0    inf    0 0x10040  67.4%
 kmem-00320          384        1160       0         346      814      206        107          5     102   4096    107     0    inf    0 0x10040  74.8%
 kmem-00384          448         519       0         134      385       65         50          0      50   4096     50     0    inf    0 0x10040  84.2%
 kmem-00448          512         369       0         149      220       68         39          3      36   4096     39     0    inf    0 0x10000  76.4%
 kmem-00512          576         279       0         119      160       50         31          1      30   4096     31     0    inf    0 0x10040  75.0%
 kmem-00768          832        1034       0         367      667      121        240         43     197   4096    240     0    inf    0 0x10040  68.8%
 kmem-01024         1088        2493       0        1499      994      233        700        291     409   4096    700     0    inf    0 0x10040  64.6%
 kmem-02048         2112        1202       0         774      428        0       1192        764     428   4096   1129     0    inf    0 0x10040  51.6%
 kmem-04096         4096         256       0         102      154        0        250         96     154   4096    193     0    inf    0 0x10000 100.0%
 ksiginfo            136         207       0         147       60       27          5          2       3   4096      3     0    inf    0 0x10040  66.4%
 ktrace              128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 kva-12288         12288         115       0          70       45       18          4          1       3 262144      4     0    inf    0 0x10e00  70.3%
 kva-16384         16384          81       0          59       22       10          4          2       2 262144      4     0    inf    0 0x10e00  68.8%
 kva-20480         20480          83       0          55       28        8          7          4       3 262144      7     0    inf    0 0x10e00  72.9%
 kva-24576         24576          30       0          19       11        9          4          2       2 262144      3     0    inf    0 0x10e00  51.6%
 kva-28672         28672          13       0          11        2        7          3          2       1 262144      2     0    inf    0 0x10e00  21.9%
 kva-32768         32768           8       0           6        2        6          1          0       1 262144      1     0    inf    0 0x10e00  25.0%
 kva-36864         36864          17       0           6       11        3          3          1       2 262144      3     0    inf    0 0x10e00  77.3%
 kva-4096           4096           0       0           0        0        0          0          0       0 262144      0     0    inf    0 0x10e00   ---
 kva-40960         40960           3       0           2        1        5          1          0       1 262144      1     0    inf    0 0x10e00  15.6%
 kva-49152         49152           5       0           4        1        4          1          0       1 262144      1     0    inf    0 0x10e00  18.8%
 kva-65536         65536           4       0           4        0        0          1          1       0 262144      1     0    inf    0 0x10e00   ---
 kva-8192           8192         105       0          58       47       17          3          1       2 262144      3     0    inf    0 0x10e00  73.4%
 l2cap_pdu            56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 l2cap_req           128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsdinopl           264           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsinoextpl         200           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsinopl            224           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfslbnpool           32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 llentrypl           280           1       0           0        1       13          1          0       1   4096      1     0    inf    0 0x10040   6.8%
 lockf               120         101       0          82       19       14          3          2       1   4096      3     0    inf    0 0x10040  55.7%
 lwppl              1088         306       0         127      179       49         97         21      76   4096     97     0    inf    6 0x10040  62.6%
 mbpl                520         806       0         415      391      162         83          4      79   4096     83     3    inf    2 0x10040  62.8%
 mclpl              2112         598       0         280      318        8        473        147     326   4096    426     8 130541    8 0x10040  50.3%
 mqmsgpl            1088           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 msdosfhpl            56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 msdosnopl           208          63       0          63        0        0          4          4       0   4096      4     0    inf    0 0x10040   ---
 mutex                64      235184       0       40771   194413    18905       3416         30    3386   4096   3416     0    inf   57 0x10040  89.7%
 nchentry            192      228451       0       31717   196734    12174       9948          0    9948   4096   9948     0    inf    0 0x10040  92.7%
 nfsnodepl           280           3       0           1        2       12          1          0       1   4096      1     0    inf    0 0x10040  13.7%
 nfsreqcachepl       104           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 nfsrvdescpl         256           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 nfsvapl             184           3       0           1        2       19          1          0       1   4096      1     0    inf    0 0x10040   9.0%
 npfcn4pl            144           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npfcn6pl            192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npfnatpl             96           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npftblpl             48           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 pcache             2752          86       0           4       82        0         82          0      82   4096     82     0    inf    0 0x10040  67.2%
 pcachecpu            64         267       0           0      267       48          5          0       5   4096      5     0    inf    0 0x10040  83.4%
 pcglarge           1088        6709       0        6306      403        2       1643       1508     135   4096   1516     0    inf    0 0x10040  79.3%
 pcgnormal           320       50648       0       42378     8270     2698       1462        548     914   4096   1380     0    inf    0 0x10040  70.7%
 pdict128            192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 pdict16              80         301       0         252       49        1          2          1       1   4096      2     0    inf    0 0x10040  95.7%
 pdict32              96          15       0           2       13       29          1          0       1   4096      1     0    inf    0 0x10040  30.5%
 pdppl              4096         181       0          87       94        0        179         85      94   4096    167     0    inf    0 0x10000 100.0%
 pewpl                32           0       0           0        0      126          1          0       1   4096      1     1      1    1 0x00040   0.0%
 phpool-1024         184           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-128           72          60       0           0       60       52          2          0       2   4096      2     0    inf    0 0x10040  52.7%
 phpool-2048         312           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 phpool-256           88          40       0           1       39        6          1          0       1   4096      1     0    inf    0 0x10040  83.8%
 phpool-4096         568           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-512          120           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-64            64      407275       0       63981   343294       56       5493         43    5450   4096   5450     0    inf    0 0x10040  98.4%
 phpool-64            64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 piperd              320         163       0          82       81       63         12          0      12   4096     12     0    inf    0 0x10040  52.7%
 pipewr              320         186       0         108       78       78         14          1      13   4096     14     0    inf    0 0x10040  46.9%
 plimitpl            240          61       0          39       22       26          4          1       3   4096      4     0    inf    0 0x10040  43.0%
 pmappl              512         181       0          87       94       74         21          0      21   4096     21     0    inf    0 0x10000  56.0%
 pnbufpl            1032         189       0         142       47        1         37         21      16   4096     30     0    inf    0 0x10040  74.0%
 procpl              896         127       0          39       88       28         30          1      29   4096     30     0    inf    0 0x10040  66.4%
 proparay             56         130       0          13      117       27          2          0       2   4096      2     0    inf    0 0x00040  80.0%
 propdata             48           1       0           0        1       83          1          0       1   4096      1     0    inf    0 0x00040   1.2%
 propdict             56         507       0         156      351        9          6          1       5   4096      6     0    inf    0 0x00040  96.0%
 propnmbr             64          47       0          11       36       27          1          0       1   4096      1     0    inf    0 0x10040  56.2%
 propstng             48         860       0         290      570       18          8          1       7   4096      8     0    inf    0 0x00040  95.4%
 pstatspl            456         128       0          40       88       32         15          0      15   4096     15     0    inf    0 0x10040  65.3%
 ptimerpl            328         101       0          84       17        7          4          2       2   4096      4     0    inf    0 0x10040  68.1%
 ptimerspl           312         101       0          84       17        9          3          1       2   4096      3     0    inf    0 0x10000  64.7%
 puffpnpl            248           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 puffprkl            120           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 puffvapl            184           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 pvpl                128      213753       0      110985   102768    21232       6789       2789    4000   4096   6789     0    inf    0 0x11040  80.3%
 ractx                40      139076       0       20272   118804      174       1178          0    1178   4096   1178     0    inf    1 0x00040  98.5%
 radixnode           192       85018       0       43075    41943     7092       3340       1005    2335   4096   3340     0    inf    1 0x11040  84.2%
 raidpsspl           200           0       0           0        0       20          1          0       1   4096      1     1      2    1 0x10040   0.0%
 rf_alloclist_pl     264     1962391       0     1962390        1      104         13          6       7   4096      7     5     18    6 0x10040   0.9%
 rf_asm_pl           496     1962390       0     1962390        0       96         24         12      12   4096     12     8     24   12 0x10040   0.0%
 rf_asmhdr_pl         32      915977       0      915977        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_asmhle_pl         24           0       0           0        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_callbackfpl       32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_callbackvpl       32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_dagh_pl          136     1962390       0     1962390        0       87          6          3       3   4096      3     2      5    3 0x10040   0.0%
 rf_daglist_pl       336     1962390       0     1962390        0       72         14          8       6   4096      6     3     11    6 0x10040   0.0%
 rf_dagnode_pl       680     8187823       0     8187823        0      252        107         65      42   4096     47    22     86   42 0x10000   0.0%
 rf_dagpcache_pl     720           0       0           0        0       10          2          0       2   4096      2     2     26    2 0x10040   0.0%
 rf_dqd_pl           208     2300653       0     2300653        0      114         12          6       6   4096      7     4     14    6 0x10040   0.0%
 rf_fss_pl            48           0       0           0        0       84          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_funclist_pl       24     1962390       0     1962390        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_mcpair_pl         48           0       0           0        0       84          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_pda_pl            64     3924780       0     3924780        0      126          2          0       2   4096      2     2      4    2 0x10040   0.0%
 rf_rad_pl           488      915977       0      915977        0       40          6          1       5   4096      5     4     16    5 0x10040   0.0%
 rf_reconbuffer_     112           0       0           0        0       36          1          0       1   4096      1     1      2    1 0x10040   0.0%
 rf_revent_pl         32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_stripelock_p      56     1962390       0     1962390        0       72          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_vfple_pl          24           0       0           0        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_vple_pl           24          30       0           0       30      138          1          0       1   4096      1     1      2    0 0x00040  17.6%
 rfcomm_credit        32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 rndctx               24          29       0          20        9      159          1          0       1   4096      1     0    inf    0 0x00040   5.3%
 rndsample           544          65       0          40       25       10          6          1       5   4096      6     0    586    0 0x10040  66.4%
 rndtemp             520           5       0           4        1        6          2          1       1   4096      1     0    inf    0 0x10040  12.7%
 rtentpl             328          28       0           2       26       10          3          0       3   4096      3     0    inf    0 0x10040  69.4%
 rttmrpl              72           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 rwlock               64      257353       0       45032   212321     7045       3519         37    3482   4096   3519     0    inf    0 0x10040  95.3%
 sackholepl           40          17       0          17        0      101          2          1       1   4096      1     0    inf    1 0x00040   0.0%
 scxspl              264       29890       0       29890        0       30          2          0       2   4096      2     2    inf    2 0x10040   0.0%
 sigacts            3096         180       0          90       90        0        178         88      90   4096    170     0    inf    0 0x10040  75.6%
 smbfsnopl           176           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 smbrqpl             296           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 smbt2pl             232           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 socket              616         492       0         157      335       43         75         12      63   4096     75     0    inf    0 0x10040  80.0%
 swp vnd             312           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 swp vnx              40           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 synpl               320          29       0          29        0        0          1          1       0   4096      1     0    inf    0 0x10040   ---
 tcpcbpl             840         829       0         815       14        2         33         29       4   4096     33     0    inf    0 0x10040  71.8%
 tcpipqepl            64         508       0         508        0        0          1          1       0   4096      1     0    inf    0 0x10040   ---
 thplthrd             80           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 tmpfs_dirent         56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 tmpfs_node          224           1       0           0        1       17          1          0       1   4096      1     0    inf    0 0x10040   5.5%
 tstile              128         298       0         116      182      128         10          0      10   4096     10     0    inf    0 0x10040  56.9%
 uaoeltpl            104           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 uarea             20480         208       0         116       92        0        206        114      92  20480    200     0    inf    0 0x10c00 100.0%
 uareasys          20480          93       0           3       90        0         93          3      90  20480     93     0    inf    0 0x10c00 100.0%
 ufsdir              272           9       0           7        2       13          2          1       1   4096      1     0    inf    0 0x10000  13.3%
 ufsdq                88           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 vcachepl            640      267595       0       58727   208868     1444      35804        752   35052   4096  35804     0    inf  240 0x10040  93.1%
 vmembt               64       23267       0        8983    14284     2411        265          0     265   4096    265     0    inf    0 0x10040  84.2%
 vmmpepl             192       23808       0       12766    11042    11113       1107         52    1055   4096   1107     0    inf    0 0x10040  49.1%
 vmsppl              360         176       0          86       90       64         16          2      14   4096     16     0    inf    0 0x10040  56.5%
 wapbldealloc         40        1022       0        1022        0      101          5          4       1   4096      4     0    inf    1 0x00040   0.0%
 wapblentrypl         48        3749       0        3749        0       84          1          0       1   4096      1     0    inf    1 0x00040   0.0%
 wapblinopl           40      140898       0      140845       53      149          2          0       2   4096      2     0    inf    0 0x00040  25.9%
 Totals                     37908998       0    31685278  6223720   221228     589460     141572  447888

 In use 2041819K, total allocated 2400228K; utilization 85.1%
 -------
 before building the distribution
 $ vmstat
  procs    memory      page                       disks   faults      cpu
  r b      avm    fre  flt  re  pi   po   fr   sr w0 w1   in   sy  cs us sy id
  1 0   494576 781856  867   1   0    2   44   62  3 23  143 2886 674  1  0 99
 $ vmstat -mvW > no2
 Memory resource pool statistics
 Name               Size    Requests    Fail    Releases    InUse    Avail      Pgreq      Pgrel   Npage PageSz  Hiwat Minpg  Maxpg Idle   Flags   Util
 ah_tdb_crypto       192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 aio_jobs_pool       136           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 aio_lio_pool         48           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 amappl               88       13244       0        6900     6344     1441        175          2     173   4096    175     0    inf    0 0x10040  78.8%
 anonpl               40      555865       0      202748   353117       80       3705        208    3497   4096   3497     0    inf    0 0x01040  98.6%
 ataspl              160     3840941       0     3840941        0       75          5          2       3   4096      3     0    inf    3 0x10040   0.0%
 biopl               304       49975       0       43487     6488     8111       2662       1539    1123   4096   1558     0    inf  623 0x10040  42.9%
 brtpl                64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 buf16k            16392       11747       0       11033      714        3       3236       2997     239  65536   1505     1      1    1 0x10040  74.7%
 buf1k              1032          59       0          59        0        3         19         18       1   4096     19     1      1    1 0x10040   0.0%
 buf2k              2056       36566       0       36299      267        1      36567      36299     268   4096  17763     1      1    1 0x10040  50.0%
 buf32k            32776       42414       0       34726     7688        1      42415      34726    7689  65536   9818     1      1    1 0x10040  50.0%
 buf4k              4096      147134       0       71769    75365        1     147135      71769   75366   4096  88754     1      1    1 0x10000 100.0%
 buf512b             520        1252       0        1250        2        5        138        137       1   4096    137     1      1    0 0x10040  25.4%
 buf64k            65536         240       0         232        8        1        241        232       9  65536    241     1      1    1 0x10000  88.9%
 buf8k              8200        2079       0        1917      162      104        171        133      38  65536     80     1      1    0 0x10040  53.3%
 bufpl               304       95954       0        7819    88135     6245       7369        109    7260   4096   7369     0    inf    0 0x10040  90.1%
 carp                 32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 carp6                32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 ccdbuf              336           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cd9660nopl          208           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptdesc           128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptkop            384           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptop             320           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 csepl               232           0       0           0        0      323         19          0      19   4096     19    19    inf   19 0x10040   0.0%
 cwdi                 64         363       0         259      104      148          4          0       4   4096      4     0    inf    1 0x10040  40.6%
 dbregs              144           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 dirhepl              48           0       0           0        0        0          0          0       0   4096      0     0    313    0 0x00040   ---
 dirhpl              304           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 efsinopl            248           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ehcixfer            384           7       0           3        4        6          1          0       1   4096      1     0    inf    0 0x10040  37.5%
 ehcixfer            384          18       0          14        4        6          2          1       1   4096      2     0    inf    0 0x10040  37.5%
 esp_tdb_crypto      128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 execargs         262144       24495       0       24495        0        1         10          9       1 262144      4     0     16    1 0x10c00   0.0%
 ext2fsinopl         256           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 extent               48           9       0           8        1       83          1          0       1   4096      1     0    inf    0 0x00040   1.2%
 fcrpl               184          31       0          30        1       83          4          0       4   4096      4     4    inf    3 0x10040   1.1%
 fdfile               64        3623       0        1353     2270      439         43          0      43   4096     43     0    inf    0 0x11040  82.5%
 ffsdino1            136           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ffsdino2            264      251767       0       76625   175142    33733      14321        396   13925   4096  14320     0    inf   16 0x10040  81.1%
 ffsino              256      251607       0       76465   175142    33738      13427        372   13055   4096  13425     0    inf   14 0x10000  83.8%
 file                128        2408       0         684     1724      198         64          2      62   4096     64     0    inf    0 0x10040  86.9%
 filedesc            832         359       0         255      104       92         55          6      49   4096     49     0    inf   13 0x10040  43.1%
 icmp                 32           2       0           2        0        0          1          1       0   4096      1     0    inf    0 0x00040   ---
 icmp6                32           6       0           6        0        0          2          2       0   4096      1     0    inf    0 0x00040   ---
 igmppl               40           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 in6pcbpl            280        3706       0        3693       13        1          3          2       1   4096      3     0    inf    0 0x10040  88.9%
 inmltpl              56           2       0           0        2       70          1          0       1   4096      1     0    inf    0 0x00040   2.7%
 inpcbpl             240        3607       0        3584       23        9          9          7       2   4096      9     0    inf    0 0x10040  67.4%
 ipcomp_tdb_cryp     128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ipfrenpl             64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 kcpuset              64         575       0         175      400       41          7          0       7   4096      7     0    inf    0 0x10040  89.3%
 kcredpl             192         601       0         200      401      124         25          0      25   4096     25     0    inf    1 0x10040  75.2%
 kmem-00008            8           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10400   ---
 kmem-00016           16       12577       0        6527     6050     3934         40          1      39   4096     40     0    inf    0 0x10400  60.6%
 kmem-00032           32        9464       0        4553     4911     2769         60          0      60   4096     60     0    inf    0 0x10400  63.9%
 kmem-00064          128       13384       0        8163     5221     1475        377        161     216   4096    377     0    inf    0 0x10040  75.5%
 kmem-00128          192       11378       0        3200     8178       12        391          1     390   4096    390     0    inf    0 0x10040  98.3%
 kmem-00192          256     6067336       1         222  6067114        6     379195          0  379195   4096 379195     0    inf    0 0x10000 100.0%
 kmem-00256          320        1364       0         567      797        7         75          8      67   4096     70     0    inf    0 0x10040  92.9%
 kmem-00320          384        1628       0         479     1149        1        120          5     115   4096    115     0    inf    0 0x10040  93.7%
 kmem-00384          448         900       0         228      672        3         75          0      75   4096     75     0    inf    0 0x10040  98.0%
 kmem-00448          512         696       0         244      452        4         60          3      57   4096     57     0    inf    0 0x10000  99.1%
 kmem-00512          576         589       0         202      387        5         57          1      56   4096     56     0    inf    0 0x10040  97.2%
 kmem-00768          832        1727       0         471     1256        0        357         43     314   4096    314     0    inf    0 0x10040  81.2%
 kmem-01024         1088        4666       0        1654     3012        0       1295        291    1004   4096   1004     0    inf    0 0x10040  79.7%
 kmem-02048         2112        2499       0         815     1684        0       2448        764    1684   4096   1684     0    inf    0 0x10040  51.6%
 kmem-04096         4096         476       0         187      289        0        385         96     289   4096    289     0    inf    0 0x10000 100.0%
 ksiginfo            136         326       0         238       88       28          6          2       4   4096      4     0    inf    0 0x10040  73.0%
 ktrace              128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 kva-12288         12288         270       0         165      105       21          7          1       6 262144      6     0    inf    1 0x10e00  82.0%
 kva-16384         16384         215       0         147       68       28          8          2       6 262144      6     0    inf    1 0x10e00  70.8%
 kva-20480         20480         228       0         143       85       35         14          4      10 262144     10     0    inf    2 0x10e00  66.4%
 kva-24576         24576         111       0          41       70        0          9          2       7 262144      7     0    inf    0 0x10e00  93.8%
 kva-28672         28672          30       0          28        2       16          4          2       2 262144      2     0    inf    1 0x10e00  10.9%
 kva-32768         32768          20       0          18        2       14          2          0       2 262144      2     0    inf    1 0x10e00  12.5%
 kva-36864         36864          19       0           8       11        3          3          1       2 262144      3     0    inf    0 0x10e00  77.3%
 kva-4096           4096           0       0           0        0        0          0          0       0 262144      0     0    inf    0 0x10e00   ---
 kva-40960         40960           4       0           3        1        5          1          0       1 262144      1     0    inf    0 0x10e00  15.6%
 kva-49152         49152           5       0           5        0        5          1          0       1 262144      1     0    inf    1 0x10e00   0.0%
 kva-65536         65536           4       0           4        0        0          1          1       0 262144      1     0    inf    0 0x10e00   ---
 kva-8192           8192         260       0         143      117       11          5          1       4 262144      4     0    inf    0 0x10e00  91.4%
 l2cap_pdu            56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 l2cap_req           128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsdinopl           264           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsinoextpl         200           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsinopl            224           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfslbnpool           32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 llentrypl           280           1       0           0        1       13          1          0       1   4096      1     0    inf    0 0x10040   6.8%
 lockf               120         111       0          96       15       18          3          2       1   4096      3     0    inf    0 0x10040  43.9%
 lwppl              1088         417       0         223      194       91        116         21      95   4096     97     0    inf   18 0x10040  54.2%
 mbpl                520         992       0         586      406      133         83          6      77   4096     83     3    inf    3 0x10040  66.9%
 mclpl              2112         772       0         402      370       18        541        153     388   4096    426     8 130541   18 0x10040  49.2%
 mqmsgpl            1088           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 msdosfhpl            56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 msdosnopl           208          63       0          63        0        0          4          4       0   4096      4     0    inf    0 0x10040   ---
 mutex                64      250040       0       74478   175562    34165       3416         87    3329   4096   3416     0    inf    1 0x10040  82.4%
 nchentry            192      240642       0       31879   208763      166       9949          0    9949   4096   9949     0    inf    0 0x10040  98.4%
 nfsnodepl           280           3       0           1        2       12          1          0       1   4096      1     0    inf    0 0x10040  13.7%
 nfsreqcachepl       104           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 nfsrvdescpl         256           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 nfsvapl             184           3       0           1        2       19          1          0       1   4096      1     0    inf    0 0x10040   9.0%
 npfcn4pl            144           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npfcn6pl            192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npfnatpl             96           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npftblpl             48           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 pcache             2752          86       0           4       82        0         82          0      82   4096     82     0    inf    0 0x10040  67.2%
 pcachecpu            64         267       0           0      267       48          5          0       5   4096      5     0    inf    0 0x10040  83.4%
 pcglarge           1088       14696       0        9686     5010        0       3178       1508    1670   4096   1670     0    inf    0 0x10040  79.7%
 pcgnormal           320       99658       0       97237     2421     2943       4687       4240     447   4096   4139     0    inf  245 0x10040  42.3%
 pdict128            192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 pdict16              80         301       0         252       49        1          2          1       1   4096      2     0    inf    0 0x10040  95.7%
 pdict32              96          15       0           2       13       29          1          0       1   4096      1     0    inf    0 0x10040  30.5%
 pdppl              4096         305       0         203      102       91        278         85     193   4096    193     0    inf   91 0x10000  52.8%
 pewpl                32           0       0           0        0      126          1          0       1   4096      1     1      1    1 0x00040   0.0%
 phpool-1024         184           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-128           72          60       0           0       60       52          2          0       2   4096      2     0    inf    0 0x10040  52.7%
 phpool-2048         312           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 phpool-256           88          40       0           1       39        6          1          0       1   4096      1     0    inf    0 0x10040  83.8%
 phpool-4096         568           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-512          120           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-64            64      541370       0       72809   468561       33       7481         43    7438   4096   7438     0    inf    0 0x10040  98.4%
 phpool-64            64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 piperd              320         272       0         210       62       94         13          0      13   4096     13     0    inf    2 0x10040  37.3%
 pipewr              320         296       0         235       61       95         14          1      13   4096     14     0    inf    0 0x10040  36.7%
 plimitpl            240         147       0         132       15       97          8          1       7   4096      7     0    inf    4 0x10040  12.6%
 pmappl              512         305       0         203      102       98         25          0      25   4096     25     0    inf    2 0x10000  51.0%
 pnbufpl            1032         310       0         283       27       66         52         21      31   4096     31     0    inf   22 0x10040  21.9%
 procpl              896         236       0         141       95       97         49          1      48   4096     48     0    inf   14 0x10040  43.3%
 proparay             56         130       0          13      117       27          2          0       2   4096      2     0    inf    0 0x00040  80.0%
 propdata             48           1       0           0        1       83          1          0       1   4096      1     0    inf    0 0x00040   1.2%
 propdict             56         507       0         156      351        9          6          1       5   4096      6     0    inf    0 0x00040  96.0%
 propnmbr             64          47       0          11       36       27          1          0       1   4096      1     0    inf    0 0x10040  56.2%
 propstng             48         860       0         290      570       18          8          1       7   4096      8     0    inf    0 0x00040  95.4%
 pstatspl            456         237       0         142       95       97         24          0      24   4096     24     0    inf    6 0x10040  44.1%
 ptimerpl            328         113       0          92       21        3          4          2       2   4096      4     0    inf    0 0x10040  84.1%
 ptimerspl           312         113       0          92       21        5          3          1       2   4096      3     0    inf    0 0x10000  80.0%
 puffpnpl            248           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 puffprkl            120           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 puffvapl            184           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 pvpl                128      254945       0      223567    31378   107254       7261       2789    4472   4096   6789     0    inf  314 0x11040  21.9%
 ractx                40      147873       0       66718    81155    16714       1178        209     969   4096   1178     0    inf    0 0x00040  81.8%
 radixnode           192       88133       0       68192    19941    26826       3340       1113    2227   4096   3340     0    inf   54 0x11040  42.0%
 raidpsspl           200           0       0           0        0       20          1          0       1   4096      1     1      2    1 0x10040   0.0%
 rf_alloclist_pl     264     2725865       0     2725864        1      104         17         10       7   4096      7     5     18    6 0x10040   0.9%
 rf_asm_pl           496     2725864       0     2725864        0       96         32         20      12   4096     12     8     24   12 0x10040   0.0%
 rf_asmhdr_pl         32     1326198       0     1326198        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_asmhle_pl         24           0       0           0        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_callbackfpl       32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_callbackvpl       32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_dagh_pl          136     2725864       0     2725864        0       87          8          5       3   4096      3     2      5    3 0x10040   0.0%
 rf_daglist_pl       336     2725864       0     2725864        0       72         17         11       6   4096      6     3     11    6 0x10040   0.0%
 rf_dagnode_pl       680    11740276       0    11740276        0      282        157        110      47   4096     47    22     86   47 0x10000   0.0%
 rf_dagpcache_pl     720           0       0           0        0       10          2          0       2   4096      2     2     26    2 0x10040   0.0%
 rf_dqd_pl           208     3562684       0     3562684        0      133         19         12       7   4096      7     4     14    7 0x10040   0.0%
 rf_fss_pl            48           0       0           0        0       84          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_funclist_pl       24     2725864       0     2725864        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_mcpair_pl         48           0       0           0        0       84          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_pda_pl            64     5451728       0     5451728        0      126          2          0       2   4096      2     2      4    2 0x10040   0.0%
 rf_rad_pl           488     1326198       0     1326198        0       40          7          2       5   4096      5     4     16    5 0x10040   0.0%
 rf_reconbuffer_     112           0       0           0        0       36          1          0       1   4096      1     1      2    1 0x10040   0.0%
 rf_revent_pl         32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_stripelock_p      56     2725864       0     2725864        0       72          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_vfple_pl          24           0       0           0        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_vple_pl           24          30       0           0       30      138          1          0       1   4096      1     1      2    0 0x00040  17.6%
 rfcomm_credit        32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 rndctx               24          32       0          24        8      160          1          0       1   4096      1     0    inf    0 0x00040   4.7%
 rndsample           544          83       0          55       28        7          6          1       5   4096      6     0    586    0 0x10040  74.4%
 rndtemp             520           8       0           8        0        7          2          1       1   4096      1     0    inf    1 0x10040   0.0%
 rtentpl             328          28       0           2       26       10          3          0       3   4096      3     0    inf    0 0x10040  69.4%
 rttmrpl              72           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 rwlock               64      260546       0       78348   182198    37168       3519         37    3482   4096   3519     0    inf    0 0x10040  81.8%
 sackholepl           40          23       0          23        0        0          2          2       0   4096      1     0    inf    0 0x00040   ---
 scxspl              264       29890       0       29890        0       30          2          0       2   4096      2     2    inf    2 0x10040   0.0%
 sigacts            3096         352       0         181      171       24        283         88     195   4096    195     0    inf   24 0x10040  66.3%
 smbfsnopl           176           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 smbrqpl             296           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 smbt2pl             232           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 socket              616         514       0         177      337       41         75         12      63   4096     75     0    inf    0 0x10040  80.4%
 swp vnd             312           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 swp vnx              40           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 synpl               320          29       0          29        0        0          1          1       0   4096      1     0    inf    0 0x10040   ---
 tcpcbpl             840         840       0         826       14        2         33         29       4   4096     33     0    inf    0 0x10040  71.8%
 tcpipqepl            64         514       0         514        0        0          2          2       0   4096      1     0    inf    0 0x10040   ---
 thplthrd             80           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 tmpfs_dirent         56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 tmpfs_node          224           1       0           0        1       17          1          0       1   4096      1     0    inf    0 0x10040   5.5%
 tstile              128         463       0         198      265       45         10          0      10   4096     10     0    inf    0 0x10040  82.8%
 uaoeltpl            104          79       0          36       43       33          2          0       2   4096      2     0    inf    0 0x10040  54.6%
 uarea             20480         368       0         198      170       20        304        114     190  20480    200     0    inf   20 0x10c00  89.5%
 uareasys          20480          93       0           3       90        0         93          3      90  20480     93     0    inf    0 0x10c00 100.0%
 ufsdir              272          15       0          11        4       11          2          1       1   4096      1     0    inf    0 0x10000  26.6%
 ufsdq                88           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 vcachepl            640      267835       0       92613   175222    33692      35804        985   34819   4096  35804     0    inf  177 0x10040  78.6%
 vmembt               64       37812       0       16940    20872     5084        412          0     412   4096    412     0    inf    0 0x10040  79.2%
 vmmpepl             192       37948       0       18322    19626     2529       1107         52    1055   4096   1107     0    inf    3 0x10040  87.2%
 vmsppl              360         340       0         174      166       32         20          2      18   4096     18     0    inf    0 0x10040  81.1%
 wapbldealloc         40       55995       0       55995        0      101         10          9       1   4096      6     0    inf    1 0x00040   0.0%
 wapblentrypl         48        4389       0        4389        0       84          1          0       1   4096      1     0    inf    1 0x00040   0.0%
 wapblinopl           40      908209       0      908156       53      149          2          0       2   4096      2     0    inf    0 0x00040  25.9%
 Totals                     54483904       1    46093266  8390638   364760     740626     162158  578468

 In use 2453135K, total allocated 2800216K; utilization 87.6%
 -------
 After stopping the distribution build

 $ vmstat
  procs    memory      page                       disks   faults      cpu
  r b      avm    fre  flt  re  pi   po   fr   sr w0 w1   in   sy  cs us sy id
  0 4  1114236   6520 3327  57  36   41  139  290 17 51  278 5451 1106 2  1 97
 $ vmstat -mvW
 Memory resource pool statistics
 Name               Size    Requests    Fail    Releases    InUse    Avail      Pgreq      Pgrel   Npage PageSz  Hiwat Minpg  Maxpg Idle   Flags   Util
 ah_tdb_crypto       192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 aio_jobs_pool       136           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 aio_lio_pool         48           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 amappl               88       21768       0       16613     5155     3125        210         26     184   4096    208     0    inf    0 0x10040  60.2%
 anonpl               40     1841955       0     1561284   280671        8       9372       6593    2779   4096   6821     0    inf    0 0x01040  98.6%
 ataspl              160    11443306       0    11443305        1       49         16         14       2   4096      3     0    inf    1 0x10040   2.0%
 biopl               304       76780       0       76638      142        1       3377       3366      11   4096   1558     0    inf    0 0x10040  95.8%
 brtpl                64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 buf16k            16392       12701       0       12556      145       92       3285       3206      79  65536   1505     1      1    0 0x10040  45.9%
 buf1k              1032          59       0          59        0        3         19         18       1   4096     19     1      1    1 0x10040   0.0%
 buf2k              2056       36687       0       36639       48        1      36688      36639      49   4096  17763     1      1    1 0x10040  49.2%
 buf32k            32776       46547       0       44012     2535        1      46548      44012    2536  65536   9993     1      1    1 0x10040  50.0%
 buf4k              4096      150428       0      140934     9494        1     150429     140934    9495   4096  88754     1      1    1 0x10000 100.0%
 buf512b             520        1570       0        1569        1        6        138        137       1   4096    137     1      1    0 0x10040  12.7%
 buf64k            65536         240       0         232        8        1        241        232       9  65536    241     1      1    1 0x10000  88.9%
 buf8k              8200        2233       0        2154       79      110        171        144      27  65536     80     1      1    0 0x10040  36.6%
 bufpl               304       97802       0       85412    12390    35372       7369       3695    3674   4096   7369     0    inf  183 0x10040  25.0%
 carp                 32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 carp6                32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 ccdbuf              336           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cd9660nopl          208           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptdesc           128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptkop            384           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptop             320           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 csepl               232           0       0           0        0      323         19          0      19   4096     19    19    inf   19 0x10040   0.0%
 cwdi                 64        1021       0         918      103       86          5          2       3   4096      4     0    inf    0 0x10040  53.6%
 dbregs              144           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 dirhepl              48           0       0           0        0        0          0          0       0   4096      0     0    313    0 0x00040   ---
 dirhpl              304           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 efsinopl            248           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ehcixfer            384           7       0           3        4        6          1          0       1   4096      1     0    inf    0 0x10040  37.5%
 ehcixfer            384          23       0          19        4        6          2          1       1   4096      2     0    inf    0 0x10040  37.5%
 esp_tdb_crypto      128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 execargs         262144      432267       0      432267        0        1         46         45       1 262144      4     0     16    1 0x10c00   0.0%
 ext2fsinopl         256           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 extent               48           9       0           8        1       83          1          0       1   4096      1     0    inf    0 0x00040   1.2%
 fcrpl               184          31       0          30        1       83          4          0       4   4096      4     4    inf    3 0x10040   1.1%
 fdfile               64        6281       0        4099     2182      527         43          0      43   4096     43     0    inf    0 0x11040  79.3%
 ffsdino1            136           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ffsdino2            264      320936       0      115170   205766     4204      14396        398   13998   4096  14320     0    inf    0 0x10040  94.7%
 ffsino              256      316048       0      110282   205766     4234      13497        372   13125   4096  13425     0    inf    0 0x10000  98.0%
 file                128        3840       0        2223     1617      305         65          3      62   4096     64     0    inf    0 0x10040  81.5%
 filedesc            832        1025       0         922      103       49         74         36      38   4096     53     0    inf    0 0x10040  55.1%
 icmp                 32           2       0           2        0        0          1          1       0   4096      1     0    inf    0 0x00040   ---
 icmp6                32           8       0           8        0        0          3          3       0   4096      1     0    inf    0 0x00040   ---
 igmppl               40           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 in6pcbpl            280        4328       0        4315       13        1          3          2       1   4096      3     0    inf    0 0x10040  88.9%
 inmltpl              56           2       0           0        2       70          1          0       1   4096      1     0    inf    0 0x00040   2.7%
 inpcbpl             240        4128       0        4106       22       10          9          7       2   4096      9     0    inf    0 0x10040  64.5%
 ipcomp_tdb_cryp     128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ipfrenpl             64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 kcpuset              64        1033       0         813      220      221          8          1       7   4096      8     0    inf    0 0x10040  49.1%
 kcredpl             192         868       0         484      384      120         25          1      24   4096     25     0    inf    0 0x10040  75.0%
 kmem-00008            8           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10400   ---
 kmem-00016           16       20492       0       15691     4801     5183         40          1      39   4096     40     0    inf    0 0x10400  48.1%
 kmem-00032           32       17486       0       13197     4289     3391         60          0      60   4096     60     0    inf    0 0x10400  55.8%
 kmem-00064          128       22091       0       17137     4954     1649        377        164     213   4096    377     0    inf    0 0x10040  72.7%
 kmem-00128          192       31475       0       23722     7753      836        417          8     409   4096    414     0    inf    0 0x10040  88.9%
 kmem-00192          256    13022181      10        3403 13018778        6     813674          0  813674   4096 813674     0    inf    0 0x10000 100.0%
 kmem-00256          320        3300       0        2702      598      194        120         54      66   4096    111     0    inf    0 0x10040  70.8%
 kmem-00320          384        4077       0        3206      871      229        191         81     110   4096    183     0    inf    0 0x10040  74.2%
 kmem-00384          448        4621       0        4211      410       94        159        103      56   4096    142     0    inf    0 0x10040  80.1%
 kmem-00448          512        4278       0        4002      276       28        166        128      38   4096    132     0    inf    0 0x10000  90.8%
 kmem-00512          576        1910       0        1747      163       82        137        102      35   4096    118     0    inf    0 0x10040  65.5%
 kmem-00768          832        5234       0        4432      802       34        803        594     209   4096    649     0    inf    0 0x10040  77.9%
 kmem-01024         1088       17119       0       14256     2863       23       3050       2088     962   4096   1665     0    inf    7 0x10040  79.1%
 kmem-02048         2112        7454       0        6099     1355        0       5321       3966    1355   4096   3120     0    inf    0 0x10040  51.6%
 kmem-04096         4096        1144       0        1042      102        1        568        465     103   4096    335     0    inf    1 0x10000  99.0%
 ksiginfo            136        1906       0        1873       33       25         11          9       2   4096      4     0    inf    0 0x10040  54.8%
 ktrace              128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 kva-12288         12288         461       0         443       18       45          8          5       3 262144      6     0    inf    0 0x10e00  28.1%
 kva-16384         16384         400       0         396        4       28          9          7       2 262144      6     0    inf    0 0x10e00  12.5%
 kva-20480         20480         393       0         370       23       13         16         13       3 262144     10     0    inf    0 0x10e00  59.9%
 kva-24576         24576         249       0         241        8        2         12         11       1 262144     10     0    inf    0 0x10e00  75.0%
 kva-28672         28672          73       0          73        0        0          9          9       0 262144      5     0    inf    0 0x10e00   ---
 kva-32768         32768          69       0          68        1        7          3          2       1 262144      2     0    inf    0 0x10e00  12.5%
 kva-36864         36864          23       0          12       11        3          3          1       2 262144      3     0    inf    0 0x10e00  77.3%
 kva-4096           4096           0       0           0        0        0          0          0       0 262144      0     0    inf    0 0x10e00   ---
 kva-40960         40960           5       0           4        1        5          1          0       1 262144      1     0    inf    0 0x10e00  15.6%
 kva-49152         49152           5       0           5        0        0          1          1       0 262144      1     0    inf    0 0x10e00   ---
 kva-65536         65536           4       0           4        0        0          1          1       0 262144      1     0    inf    0 0x10e00   ---
 kva-8192           8192         408       0         374       34       30          5          3       2 262144      4     0    inf    0 0x10e00  53.1%
 l2cap_pdu            56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 l2cap_req           128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsdinopl           264           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsinoextpl         200           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsinopl            224           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfslbnpool           32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 llentrypl           280           1       0           0        1       13          1          0       1   4096      1     0    inf    0 0x10040   6.8%
 lockf               120         300       0         288       12       21          5          4       1   4096      3     0    inf    0 0x10040  35.2%
 lwppl              1088         909       0         713      196       29        130         55      75   4096     98     0    inf    0 0x10040  69.4%
 mbpl                520        3781       0        3445      336       35         93         40      53   4096     83     3    inf    1 0x10040  80.5%
 mclpl              2112        3267       0        2959      308       16       1030        706     324   4096    426     8 130541   16 0x10040  49.0%
 mqmsgpl            1088           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 msdosfhpl            56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 msdosnopl           208          63       0          63        0        0          4          4       0   4096      4     0    inf    0 0x10040   ---
 mutex                64      312788       0      106590   206198     4285       3428         87    3341   4096   3416     0    inf    0 0x10040  96.4%
 nchentry            192      241413       0       32835   208578      351       9949          0    9949   4096   9949     0    inf    0 0x10040  98.3%
 nfsnodepl           280           3       0           1        2       12          1          0       1   4096      1     0    inf    0 0x10040  13.7%
 nfsreqcachepl       104           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 nfsrvdescpl         256           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 nfsvapl             184           3       0           1        2       19          1          0       1   4096      1     0    inf    0 0x10040   9.0%
 npfcn4pl            144           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npfcn6pl            192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npfnatpl             96           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npftblpl             48           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 pcache             2752          86       0           4       82        0         82          0      82   4096     82     0    inf    0 0x10040  67.2%
 pcachecpu            64         267       0           0      267       48          5          0       5   4096      5     0    inf    0 0x10040  83.4%
 pcglarge           1088       40125       0       40108       17       25       8394       8380      14   4096   3420     0    inf    8 0x10040  32.3%
 pcgnormal           320      156663       0      156468      195       45       6339       6319      20   4096   4139     0    inf    0 0x10040  76.2%
 pdict128            192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 pdict16              80         321       0         272       49        1          3          2       1   4096      2     0    inf    0 0x10040  95.7%
 pdict32              96          15       0           2       13       29          1          0       1   4096      1     0    inf    0 0x10040  30.5%
 pdppl              4096         950       0         847      103        0        478        375     103   4096    211     0    inf    0 0x10000 100.0%
 pewpl                32           0       0           0        0      126          1          0       1   4096      1     1      1    1 0x00040   0.0%
 phpool-1024         184           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-128           72          60       0           0       60       52          2          0       2   4096      2     0    inf    0 0x10040  52.7%
 phpool-2048         312           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 phpool-256           88          40       0           1       39        6          1          0       1   4096      1     0    inf    0 0x10040  83.8%
 phpool-4096         568           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-512          120           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-64            64      980595       0      143772   836823        6      13326         43   13283   4096  13283     0    inf    0 0x10040  98.4%
 phpool-64            64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 piperd              320         665       0         600       65       79         15          3      12   4096     15     0    inf    0 0x10040  42.3%
 pipewr              320         755       0         692       63       93         16          3      13   4096     15     0    inf    0 0x10040  37.9%
 plimitpl            240         542       0         524       18       30         12          9       3   4096      7     0    inf    0 0x10040  35.2%
 pmappl              512         950       0         847      103       89         33          9      24   4096     27     0    inf    0 0x10000  53.6%
 pnbufpl            1032        1384       0        1372       12        0        105        101       4   4096     33     0    inf    0 0x10040  75.6%
 procpl              896         715       0         613      102       26         60         28      32   4096     50     0    inf    0 0x10040  69.7%
 proparay             56         134       0          17      117       27          2          0       2   4096      2     0    inf    0 0x00040  80.0%
 propdata             48           1       0           0        1       83          1          0       1   4096      1     0    inf    0 0x00040   1.2%
 propdict             56         523       0         172      351        9          6          1       5   4096      6     0    inf    0 0x00040  96.0%
 propnmbr             64          49       0          13       36       27          1          0       1   4096      1     0    inf    0 0x10040  56.2%
 propstng             48         892       0         322      570       18          8          1       7   4096      8     0    inf    0 0x00040  95.4%
 pstatspl            456         720       0         618      102       42         29         11      18   4096     25     0    inf    0 0x10040  63.1%
 ptimerpl            328         117       0          96       21        3          4          2       2   4096      4     0    inf    0 0x10040  84.1%
 ptimerspl           312         117       0          96       21        5          3          1       2   4096      3     0    inf    0 0x10000  80.0%
 puffpnpl            248           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 puffprkl            120           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 puffvapl            184           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 pvpl                128      300499       0      288687    11812    69005       7261       4654    2607   4096   6789     0    inf  329 0x11040  14.2%
 ractx                40      230136       0      138235    91901        9       1478        568     910   4096   1178     0    inf    0 0x00040  98.6%
 radixnode           192      145216       0      140681     4535    18565       3340       2240    1100   4096   3340     0    inf   25 0x11040  19.3%
 raidpsspl           200           0       0           0        0       20          1          0       1   4096      1     1      2    1 0x10040   0.0%
 rf_alloclist_pl     264     5193794       0     5193793        1       74         68         63       5   4096      7     5     18    4 0x10040   1.3%
 rf_asm_pl           496     5193793       0     5193793        0       88        161        150      11   4096     12     8     24   11 0x10040   0.0%
 rf_asmhdr_pl         32     2170796       0     2170796        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_asmhle_pl         24           0       0           0        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_callbackfpl       32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_callbackvpl       32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_dagh_pl          136     5193793       0     5193793        0       58         17         15       2   4096      3     2      5    2 0x10040   0.0%
 rf_daglist_pl       336     5193793       0     5193793        0       60        108        103       5   4096      6     3     11    5 0x10040   0.0%
 rf_dagnode_pl       680    22834855       0    22834855        0      222        872        835      37   4096     47    22     86   37 0x10000   0.0%
 rf_dagpcache_pl     720           0       0           0        0       10          2          0       2   4096      2     2     26    2 0x10040   0.0%
 rf_dqd_pl           208     7253476       0     7253476        0      114        116        110       6   4096      7     4     14    6 0x10040   0.0%
 rf_fss_pl            48           0       0           0        0       84          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_funclist_pl       24     5193793       0     5193793        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_mcpair_pl         48           0       0           0        0       84          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_pda_pl            64    10387586       0    10387586        0      126          2          0       2   4096      2     2      4    2 0x10040   0.0%
 rf_rad_pl           488     2170796       0     2170796        0       40         44         39       5   4096      5     4     16    5 0x10040   0.0%
 rf_reconbuffer_     112           0       0           0        0       36          1          0       1   4096      1     1      2    1 0x10040   0.0%
 rf_revent_pl         32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_stripelock_p      56     5193793       0     5193793        0       72          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_vfple_pl          24           0       0           0        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_vple_pl           24          30       0           0       30      138          1          0       1   4096      1     1      2    0 0x00040  17.6%
 rfcomm_credit        32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 rndctx               24          37       0          29        8      160          1          0       1   4096      1     0    inf    0 0x00040   4.7%
 rndsample           544         206       0         187       19       16          6          1       5   4096      6     0    586    0 0x10040  50.5%
 rndtemp             520          13       0          13        0        0          3          3       0   4096      1     0    inf    0 0x10040   ---
 rtentpl             328          28       0           2       26       10          3          0       3   4096      3     0    inf    0 0x10040  69.4%
 rttmrpl              72           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 rwlock               64      351347       0      140127   211220     8146       3519         37    3482   4096   3519     0    inf    0 0x10040  94.8%
 sackholepl           40          23       0          23        0        0          2          2       0   4096      1     0    inf    0 0x00040   ---
 scxspl              264       29890       0       29890        0       30          2          0       2   4096      2     2    inf    2 0x10040   0.0%
 sigacts            3096         952       0         849      103        0        426        323     103   4096    210     0    inf    0 0x10040  75.6%
 smbfsnopl           176           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 smbrqpl             296           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 smbt2pl             232           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 socket              616         740       0         414      326       52         75         12      63   4096     75     0    inf    0 0x10040  77.8%
 swp vnd             312           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 swp vnx              40           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 synpl               320          29       0          29        0        0          1          1       0   4096      1     0    inf    0 0x10040   ---
 tcpcbpl             840         848       0         835       13        3         33         29       4   4096     33     0    inf    0 0x10040  66.7%
 tcpipqepl            64         515       0         515        0        0          3          3       0   4096      1     0    inf    0 0x10040   ---
 thplthrd             80           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 tmpfs_dirent         56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 tmpfs_node          224           1       0           0        1       17          1          0       1   4096      1     0    inf    0 0x10040   5.5%
 tstile              128         951       0         755      196      114         10          0      10   4096     10     0    inf    0 0x10040  61.2%
 uaoeltpl            104         226       0         179       47       29          3          1       2   4096      3     0    inf    0 0x10040  59.7%
 uarea             20480         850       0         744      106        0        415        309     106  20480    202     0    inf    0 0x10c00 100.0%
 uareasys          20480          93       0           3       90        0         93          3      90  20480     93     0    inf    0 0x10c00 100.0%
 ufsdir              272          70       0          69        1       14         12         11       1   4096      1     0    inf    0 0x10000   6.6%
 ufsdq                88           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 vcachepl            640      345989       0      140179   205810     4130      35991       1001   34990   4096  35804     0    inf    0 0x10040  91.9%
 vmembt               64       62704       0       34209    28495      926        467          0     467   4096    467     0    inf    0 0x10040  95.3%
 vmmpepl             192       62637       0       48291    14346     8229       1250        175    1075   4096   1194     0    inf    0 0x10040  62.6%
 vmsppl              360         941       0         838      103       84         23          6      17   4096     20     0    inf    0 0x10040  53.3%
 wapbldealloc         40       70640       0       70640        0        0         16         16       0   4096      6     0    inf    0 0x00040   ---
 wapblentrypl         48        6985       0        6985        0       84          4          3       1   4096      1     0    inf    1 0x00040   0.0%
 wapblinopl           40     2332333       0     2332276       57      145          2          0       2   4096      2     0    inf    0 0x00040  27.8%
 Totals                    109665202      10    94066088 15599114   178659    1210640     274601  936039

 In use 3770733K, total allocated 3908712K; utilization 96.5%


From: Andrew Doran <ad@netbsd.org>
To: MLH <mlh@goathill.org>
Cc: gnats-bugs@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Tue, 24 Mar 2020 21:23:35 +0000

 > kmem-00192          256     4088775       0         182  4088593       15     255538          0  255538   4096 255538     0    inf    0 0x10000 100.0%

 That's about a gigabyte of kernel memory leaked.  I have a radeon card here. 
 I'll try to reproduce it.

 Andrew

From: mlh@goathill.org (MLH)
To: gnats-bugs@netbsd.org
Cc: ad@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
 mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Tue, 24 Mar 2020 17:34:08 -0400 (EDT)

 Andrew Doran wrote:
 > The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 > 
 > From: Andrew Doran <ad@netbsd.org>
 > To: MLH <mlh@goathill.org>
 > Cc: gnats-bugs@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
 > Subject: Re: port-amd64/54988: possible memory leaks/swap problems
 > Date: Tue, 24 Mar 2020 21:23:35 +0000
 > 
 >  > kmem-00192          256     4088775       0         182  4088593       15     255538          0  255538   4096 255538     0    inf    0 0x10000 100.0%
 >  
 >  That's about a gigabyte of kernel memory leaked.  I have a radeon card here. 
 >  I'll try to reproduce it.

 That's much less than I've seen recently. The really good thing is
 that now the system isn't locking up or crashing. It just grinds
 to a halt but with about 20 minutes of patient work :^) I can stop
 X, and then either get a console up to do some housecleaning and
 reboot or ssh in from my phone to reboot. That's a major improvement
 over the last few weeks!  Yay!! Thanks!

 I say maybe radeon because the more the display changes, the more
 memory is leaked. I can just move the mouse around and activate
 different windows and watch it change fairly rapidly. If I do jobs
 that don't change the display much, the leak rate decreases.

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Fri, 27 Mar 2020 00:13:56 +0000

 On Tue, Mar 24, 2020 at 09:25:02PM +0000, Andrew Doran wrote:
  >> kmem-00192          256     4088775       0         182  4088593       15     255538          0  255538   4096 255538     0    inf    0 0x10000 100.0%
  >  
  > That's about a gigabyte of kernel memory leaked.  I have a radeon card here.
  > I'll try to reproduce it.

 I have been seeing kmem-192 leaks on a machine with a radeon as well,
 but nowhere as severe as described in this PR (takes weeks to burn
 through a couple gigs).

 FWIW.

 -- 
 David A. Holland
 dholland@netbsd.org

From: mlh@goathill.org (MLH)
To: gnats-bugs@netbsd.org
Cc: ad@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
 mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Thu, 26 Mar 2020 21:42:48 -0400 (EDT)

 David Holland wrote:
 > The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 > 
 > From: David Holland <dholland-bugs@netbsd.org>
 > To: gnats-bugs@netbsd.org
 > Cc: 
 > Subject: Re: port-amd64/54988: possible memory leaks/swap problems
 > Date: Fri, 27 Mar 2020 00:13:56 +0000
 > 
 >  On Tue, Mar 24, 2020 at 09:25:02PM +0000, Andrew Doran wrote:
 >   >> kmem-00192          256     4088775       0         182  4088593       15     255538          0  255538   4096 255538     0    inf    0 0x10000 100.0%
 >   >  
 >   > That's about a gigabyte of kernel memory leaked.  I have a radeon card here.
 >   > I'll try to reproduce it.
 >  
 >  I have been seeing kmem-192 leaks on a machine with a radeon as well,
 >  but nowhere as severe as described in this PR (takes weeks to burn
 >  through a couple gigs).

 It can take mine as little as about four hours to burn through 4G.
 The last time I checked, X didn't have to be running for it to
 happen, but I haven't tried that in about a week.

 [     1.027936] radeon0 at pci1 dev 0 function 0: ATI Technologies Radeon HD 6450 (rev. 0x00)
 [     8.893648] radeon0: info: VRAM: 2048M 0x0000000000000000 - 0x000000007FFFFFFF (2048M used)
 [     8.893648] radeon0: info: GTT: 1024M 0x0000000080000000 - 0x00000000BFFFFFFF
 [     8.893648] kern info: [drm] radeon: 2048M of VRAM memory ready
 [     8.893648] kern info: [drm] radeon: 1024M of GTT memory ready.
 [     8.959977] kern info: [drm] radeon: dpm initialized
 [     9.009997] radeon0: info: WB enabled
 [     9.009997] radeon0: info: fence driver on ring 0 use gpu addr 0x0000000080000c00 and cpu addr 0x0xffffbfcb7bd1fc00
 [     9.009997] radeon0: info: fence driver on ring 3 use gpu addr 0x0000000080000c0c and cpu addr 0x0xffffbfcb7bd1fc0c
 [     9.020001] radeon0: info: fence driver on ring 5 use gpu addr 0x0000000000072118 and cpu addr 0x0xffffc300780b2118
 [     9.020001] radeon0: info: radeon: MSI limited to 32-bit
 [     9.020001] radeon0: info: radeon: using MSI.
 [     9.020001] radeon0: interrupting at msi3 vec 0 (radeon0)
 [     9.020001] kern info: [drm] radeon: irq initialized.
 [    10.030413] radeondrmkmsfb0 at radeon0
 [    10.030413] radeondrmkmsfb0: framebuffer at 0xffffc300786e3000, size 1920x1200, depth 32, stride 7680
 [    10.650666] wsdisplay0 at radeondrmkmsfb0 kbdmux 1: console (default, vt100 emulation), using wskbd0
 [    68.046323] kern error: [drm:(/usr/src/sys/external/bsd/drm2/dist/drm/radeon/radeon_btc_dpm.c:2319)btc_dpm_set_power_state] *ERROR* rv770_restrict_performance_levels_before_switch failed

From: Andrew Doran <ad@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Sun, 5 Apr 2020 20:58:48 +0000

 Tried it recently with a Radeon card, including with OpenGL (Quake II) but
 no conclusive repro.  Will try again in the near future.

 Andrew

From: mlh@goathill.org (MLH)
To: Andrew Doran <ad@netbsd.org>
Cc: gnats-bugs@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
 mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Sun, 5 Apr 2020 19:17:26 -0400 (EDT)

 Andrew Doran wrote:
 > Tried it recently with a Radeon card, including with OpenGL (Quake II) but
 > no conclusive repro.  Will try again in the near future.

 Thanks

 Another thing I noticed is that it seems to lose memory the fastest
 when text is scrolling in an xterm at high speed, such as building
 sets, or building/installing pkgsrc binaries. If I hide the window
 during those operations, the loss rate slows pretty dramatically.

From: Andrew Doran <ad@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Tue, 7 Apr 2020 22:12:21 +0000

 On Sun, Apr 05, 2020 at 11:20:01PM +0000, MLH wrote:

 >  Another thing I noticed is that it seems to lose memory the fastest
 >  when text is scrolling in an xterm at high speed, such as building
 >  sets, or building/installing pkgsrc binaries. If I hide the window
 >  during those operations, the loss rate slows pretty dramatically.

 I tried compiling some stuff with the output going to an xterm and sure
 enough it starts to leak out of kmem-192:

 $ vmstat -m | grep kmem-00192
 kmem-00192   192  2911902   28        0 138662    0 138662 138662   0   inf    0
 $ vmstat -C | grep kmem-00192 
 kmem-00192     182    15    0   39   2939175    3245745   9.4    35909883  91.0

 It wasn't leaking before that, so it's a good repro.  Looking with dtrace
 there are many allocations happenning in the DRM code which is probably the
 first place I'd look given that it's very X specific.  Will need to think
 about it some more.

 Andrew

From: Jason Thorpe <thorpej@me.com>
To: Andrew Doran <ad@netbsd.org>
Cc: gnats-bugs@netbsd.org,
 gnats-admin@netbsd.org,
 netbsd-bugs@netbsd.org,
 mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Tue, 7 Apr 2020 15:20:09 -0700

 > On Apr 7, 2020, at 3:12 PM, Andrew Doran <ad@netbsd.org> wrote:
 >=20
 > It wasn't leaking before that, so it's a good repro.  Looking with =
 dtrace
 > there are many allocations happenning in the DRM code which is =
 probably the
 > first place I'd look given that it's very X specific.  Will need to =
 think
 > about it some more.

 Write a dtrace script that builds a stack trace histogram for any =
 allocation from that bucket?

 -- thorpej

From: mlh@goathill.org (MLH)
To: Jason Thorpe <thorpej@me.com>
Cc: Andrew Doran <ad@netbsd.org>, gnats-bugs@netbsd.org, 
 gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Wed, 8 Apr 2020 13:35:31 -0400 (EDT)

 Jason Thorpe wrote:
 > 
 > > On Apr 7, 2020, at 3:12 PM, Andrew Doran <ad@netbsd.org> wrote:
 > > 
 > > It wasn't leaking before that, so it's a good repro.  Looking with dtrace
 > > there are many allocations happenning in the DRM code which is probably the
 > > first place I'd look given that it's very X specific.  Will need to think
 > > about it some more.
 > 
 > Write a dtrace script that builds a stack trace histogram for any allocation from that bucket?

 A bit more than I can figure out how to do. How do you even specify
 to watch kmem-00192?

From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: ad@netbsd.org, thorpej@me.com
Cc: gnats-bugs@netbsd.org, tsutsui@ceres.dti.ne.jp
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Mon, 13 Apr 2020 01:10:19 +0900

 thorpej@ wrote:

 > > On Apr 7, 2020, at 3:12 PM, Andrew Doran <ad@netbsd.org> wrote:
 > > 
 > > It wasn't leaking before that, so it's a good repro.  Looking with dtrace
 > > there are many allocations happenning in the DRM code which is probably the
 > > first place I'd look given that it's very X specific.  Will need to think
 > > about it some more.
 > 
 > Write a dtrace script that builds a stack trace histogram for any allocation from that bucket?

 In a private discussion, I was told the following dtrace script
 (for kmem-96 on NetBSD/i386 9.0 GENERIC):

 ---
 dtrace -n 'fbt::kmem_intr_alloc:entry /80 < (arg0 + 0) && (arg0 + 0) <= 84/ { @["alloc", stack()] = count() } fbt::kmem_intr_free:entry /80 < (arg1 + 0) && (arg1 + 0) <= 84/ { @["free", stack()] = count() } tick-10s { printa(@) }'
 ---

 On NetBSD/amd64 9.0 GENERIC, it looks kmem-160
 (not kmem-192 in -current, probably some DIAGNOSTIC?):

 ---
 dtrace -n 'fbt::kmem_intr_alloc:entry /128 < (arg0 + 0) && (arg0 + 0) <= 148/ { @["alloc", stack()] = count() } fbt::kmem_intr_free:entry /128 < (arg1 + 0) && (arg1 + 0) <= 148/ { @["free", stack()] = count() } tick-10s { printa(@) }'
 ---

 Note 'modload dtrace_profile' is necessary for "tick-10s".

 According to results of these dtrace, it looks the following kmalloc() in
 src/sys/external/bsd/drm2/dist/drm/radeon/radeon_fence.c:radeon_fence_emit():

  https://nxr.netbsd.org/xref/src/sys/external/bsd/drm2/dist/drm/radeon/radeon_fence.c?r=1.15#141
 ---
     134 int radeon_fence_emit(struct radeon_device *rdev,
     135 		      struct radeon_fence **fence,
     136 		      int ring)
     137 {
     138 	u64 seq = ++rdev->fence_drv[ring].sync_seq[ring];
     139 
     140 	/* we are protected by the ring emission mutex */
     141 	*fence = kmalloc(sizeof(struct radeon_fence), GFP_KERNEL);
     142 	if ((*fence) == NULL) {
     143 		return -ENOMEM;
     144 	}
 ---

 I was also told the the fence memory would be free'ed from RCU GC thread
 (fence_put() -> fence_release() -> fence_free() -> fence_free_cb() ?)
 but it's too complicated for me to investegate.

 ---
 Izumi Tsutsui

From: mlh@goathill.org (MLH)
To: gnats-bugs@netbsd.org
Cc: ad@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
 mlh@goathill.org
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Fri, 12 Jun 2020 09:19:55 -0400 (EDT)

 Andrew Doran wrote:
 > The following reply was made to PR port-amd64/54988; it has been noted by GNATS.
 > 
 > From: Andrew Doran <ad@netbsd.org>
 > 
 >  On Sun, Apr 05, 2020 at 11:20:01PM +0000, MLH wrote:
 >  
 >  >  Another thing I noticed is that it seems to lose memory the fastest
 >  >  when text is scrolling in an xterm at high speed, such as building
 >  >  sets, or building/installing pkgsrc binaries. If I hide the window
 >  >  during those operations, the loss rate slows pretty dramatically.
 >  
 >  I tried compiling some stuff with the output going to an xterm and sure
 >  enough it starts to leak out of kmem-192:

 I found another source of memory leakage this morning at 4am when
 "daily" started up. I watched vmstat as memory went from ~70% used
 (with essentially nothing running) to ~98% used and was never
 recovered and had to reboot. I had started suspecting this as every
 morning I usually have to reboot the box to make it usable.

 This with NetBSD 9.99.64 Fri Jun  5

>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.46 2020/01/03 16:35:01 leot Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2020 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.