NetBSD Problem Report #59252

From www@netbsd.org  Sat Apr  5 11:13:28 2025
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id CC79B1A923E
	for <gnats-bugs@gnats.NetBSD.org>; Sat,  5 Apr 2025 11:13:28 +0000 (UTC)
Message-Id: <20250405111327.3165F1A9241@mollari.NetBSD.org>
Date: Sat,  5 Apr 2025 11:13:27 +0000 (UTC)
From: campbell+netbsd@mumble.net
Reply-To: campbell+netbsd@mumble.net
To: gnats-bugs@NetBSD.org
Subject: tests/rump/rumpkern/t_sp:sigsafe: out of memory
X-Send-Pr-Version: www-1.0

>Number:         59252
>Category:       misc
>Synopsis:       tests/rump/rumpkern/t_sp:sigsafe: out of memory
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    misc-bug-people
>State:          needs-pullups
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Apr 05 11:15:00 +0000 2025
>Closed-Date:    
>Last-Modified:  Sun Apr 06 03:17:32 +0000 2025
>Originator:     Taylor R Campbell
>Release:        current
>Organization:
The rump t_sp Foundaleak
>Environment:
>Description:
The `sigsafe' test in tests/rump/rumpkern/t_sp has been intermittently failing for a long time.  I put in various diagnostic measures (e.g., PR bin/58112: daemonized rump conceals output on crash) to no avail, until I noticed this:

Pre-test stderr:
[1]   Killed                  ${Atf_Check} "${@}"
rump.halt: init failed: Connection refused
t_sp: ERROR: The test case cleanup returned a non-ok exit code, but this is not allowed

https://releng.netbsd.org/b5reports/i386/2025/2025.04.04.21.52.19/test.html#rump_rumpkern_t_sp_sigsafe

On review of the test logs, I see that a sampling of t_sp:sigsafe failures all show the uvm oom killer:

    sigsafe: [ 7629.0398811] UVM: pid 25563.25563 (h_sigcli), uid 0 killed: out of swap
[ 7629.0427354] UVM: pid 17924.17924 (sh), uid 0 killed: out of swap
[ 7629.0526265] UVM: pid 17976.26594 (rump_server), uid 0 killed: out of swap
[ 7629.0526265] UVM: pid 27133.27133 (atf-check), uid 0 killed: out of swap
[5.689429s] Failed: atf-check failed; see the output of the test for details

https://releng.netbsd.org/b5reports/i386/2025/2025.04.04.21.52.19/test.log

    sigsafe: [ 7541.7773590] UVM: pid 2566.14124 (rump_server), uid 0 killed: out of swap
[5.739012s] Failed: atf-check failed; see the output of the test for details

https://releng.netbsd.org/b5reports/i386/2025/2025.04.04.20.40.58/test.log

    sigsafe: [ 7636.1123173] UVM: pid 29410.16674 (rump_server), uid 0 killed: out of swap
[ 7636.1123173] UVM: pid 26178.26178 (sh), uid 0 killed: out of swap
[5.838166s] Failed: atf-check failed; see the output of the test for details

https://releng.netbsd.org/b5reports/i386/2025/2025.04.01.16.35.46/test.log
>How-To-Repeat:
On a low-memory machine:

atf-run /usr/tests/rump/rumpkern/t_sp:sigsafe | atf-report
>Fix:
Yes, please!

>Release-Note:

>Audit-Trail:
From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/59252 CVS commit: src/sys/rump/librump/rumpkern
Date: Sun, 6 Apr 2025 01:13:55 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Sun Apr  6 01:13:55 UTC 2025

 Modified Files:
 	src/sys/rump/librump/rumpkern: lwproc.c

 Log Message:
 rump: Nix leaked struct lwp on every rump_server syscall.

 This leak was introduced by lwproc.c rev. 1.51 back in 2020, when the
 kmem_zalloc(sizeof(*l), KM_SLEEP) was factored out of callers and
 into lwproc_makelwp:

 -static void
 -lwproc_makelwp(struct proc *p, struct lwp *l, bool doswitch, bool procmake)
 +static struct lwp *
 +lwproc_makelwp(struct proc *p, bool doswitch, bool procmake)
  {
 +	struct lwp *l = kmem_zalloc(sizeof(*l), KM_SLEEP);
 ...
 @@ -435,8 +431,12 @@ rump_lwproc_newlwp(pid_t pid)
  		kmem_free(l, sizeof(*l));
  		return EBUSY;
  	}
 +	mutex_exit(p->p_lock);
  	mutex_exit(&proc_lock);
 -	lwproc_makelwp(p, l, true, false);
 +
 +	/* XXX what holds proc? */
 +
 +	lwproc_makelwp(p, true, false);

 Unfortunately, the kmem_zalloc in rump_lwproc_newlwp was not deleted.
 So it just leaked.

 And this routine is called on _every_ syscall handled by rump_server
 (via struct rumpuser_hyperup::hyp_lwproc_newlwp):

     674 static void
     675 serv_handlesyscall(struct spclient *spc, struct rsp_hdr *rhdr, uint8_t *data)
     676 {
     677 	register_t retval[2] = {0, 0};
     678 	int rv, sysnum;
     679
     680 	sysnum = (int)rhdr->rsp_sysnum;
     681 	DPRINTF(("rump_sp: handling syscall %d from client %d\n",
     682 	    sysnum, spc->spc_pid));
     683
  => 684 	if (__predict_false((rv = lwproc_newlwp(spc->spc_pid)) != 0)) {
     685 		retval[0] = -1;
     686 		send_syscall_resp(spc, rhdr->rsp_reqno, rv, retval);
     687 		return;
     688 	}
     689 	spc->spc_syscallreq = rhdr->rsp_reqno;
     690 	rv = rumpsyscall(sysnum, data, retval);

 https://nxr.netbsd.org/xref/src/lib/librumpuser/rumpuser_sp.c?r=1.77#684

 So this leak would grow fairly quickly in processes issuing rump
 syscalls to rump_servers, which t_sp:sigsafe's helper h_sigcli does
 as fast as it can in a loop for 5sec -- which is just long enough for
 the i386 releng testbed with about 128 MB of RAM to run out of memory
 in roughly half of the test runs, but not long enough on my laptop
 with 64 GB of RAM to ever reproduce the problem.

 Found by:

 (a) running the test for 20sec rather than 5sec to amplify the leak;
 (b) counting bytes allocated by return addresses of rumpuser_malloc;
 (c) chasing that to rump_hypermalloc (culprit) vs uvm_km_alloc;
 (d) chasing that to pgctor, uvm_map, uvm_km_kmem_alloc (culprit),
     vmapbuf;
 (e) then to pool_page_alloc, then to pool_allocator_alloc, then to
     pool_grow, then to pool_get, then to pool_cache_get, then to
     kmem_zalloc;
 (f) finally to the call to kmem_zalloc in rump_lwproc_newlwp, which
     at last yielded to my stare.

 This instrumentation was extremely ad hoc -- I just created a table
 of 4096 entries for each routine, and populated by linear scan with
 atomics:

 struct {
 	void *volatile ra;
 	volatile unsigned long n;
 } pool_cache_get_bytes;

 void *
 kmem_zalloc(size_t size, km_flags_t flags)
 {
 	...
 	void *const ra = __builtin_return_address(0);
 	size_t i;
 	for (i = 0; i < __arraycount(kmem_zalloc_bytes); i++) {
 		if (pool_cache_get_bytes[i].ra == ra ||
 		    (pool_cache_get_bytes[i].ra == NULL &&
 			atomic_cas_ptr(&pool_cache_get_bytes[i].ra, NULL,
 			    ra) == NULL)) {
 			atomic_add_long(&pool_cache_get_bytes[i].n, size);
 			break;
 		}
 	}
 	...
 }

 Would be nice to systematize this.  Would also be nice to bring back
 malloc tags for accounting purposes so you don't need to match up the
 kmem_zalloc return addresses with the kmem_free return addresses to
 find leaks -- I got lucky here because there were very few return
 addresses to piece through.

 PR misc/59252: tests/rump/rumpkern/t_sp:sigsafe: out of memory


 To generate a diff of this commit:
 cvs rdiff -u -r1.58 -r1.59 src/sys/rump/librump/rumpkern/lwproc.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/59252 CVS commit: src/sys/rump/librump/rumpkern
Date: Sun, 6 Apr 2025 01:58:23 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Sun Apr  6 01:58:23 UTC 2025

 Modified Files:
 	src/sys/rump/librump/rumpkern: intr.c

 Log Message:
 rump: Set up the softint CPU bouncer only once, not once per CPU.

 This is used for softint_schedule_cpu to schedule softints on other
 CPUs.  There is one global lock, and one global condition variable,
 and really there only needs to be one thread to handle the requests.
 Without this change, we reinitialize the lock and condition variable,
 leaking them (and possibly leave some threads confused if they are
 currently taking the lock or waiting on the condition variable as
 they get reinitialized).

 Leak found by lsan while investigating:

 PR misc/59252: tests/rump/rumpkern/t_sp:sigsafe: out of memory


 To generate a diff of this commit:
 cvs rdiff -u -r1.56 -r1.57 src/sys/rump/librump/rumpkern/intr.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->needs-pullups
State-Changed-By: riastradh@NetBSD.org
State-Changed-When: Sun, 06 Apr 2025 03:17:32 +0000
State-Changed-Why:
fixed in HEAD, needs pullup-9 and pullup-10


>Unformatted:

Home	PR Database Search