NetBSD Problem Report #59037

From riastradh@netbsd.org  Wed Jan 29 16:12:25 2025
Return-Path: <riastradh@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits)
	 client-signature RSA-PSS (2048 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id AB3F51A923A
	for <gnats-bugs@gnats.NetBSD.org>; Wed, 29 Jan 2025 16:12:25 +0000 (UTC)
Message-Id: <20250129161223.C49151A923D@mollari.NetBSD.org>
Date: Wed, 29 Jan 2025 16:12:23 +0000 (UTC)
From: Taylor R Campbell <riastradh@NetBSD.org>
Reply-To: Taylor R Campbell <riastradh@NetBSD.org>
To: gnats-bugs@NetBSD.org
Subject: deadlock in posix_spawn
X-Send-Pr-Version: 3.95

>Number:         59037
>Category:       kern
>Synopsis:       deadlock in posix_spawn
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Jan 29 16:15:00 +0000 2025
>Closed-Date:    Sun Apr 13 00:24:49 +0000 2025
>Last-Modified:  Fri May 09 13:05:02 +0000 2025
>Originator:     Taylor R Campbell
>Release:        NetBSD 10.0_STABLE
>Organization:
The NetPOSIXD Spawndation
>Environment:
Architecture: x86_64
Machine: amd64
NetBSD 10.0_STABLE (SHADOW) #0: Mon Oct 28 06:14:44 UTC 2024  spz@franklin.NetBSD.org:/home/netbsd/10/amd64/obj/sys/arch/amd64/compile/SHADOW
>Description:

	Various processes stuck in tstile, vfork, wait, pipe_rd,
	select, poll, &c.:

db{0}> ps /l
PID     LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
19214 19214 3  22         0   ffffd919f99c1a40             screen tstile
15719 15719 3   0         0   ffffd9171a2fa280               cron tstile
16226 16226 3   0         0   ffffd912be5c4bc0               cron vfork
23194 23194 3   0         0   ffffd9a699e51040               cron tstile
9594   9594 3   0         0   ffffd91219955a80               cron vfork
...
23011 23011 3   9       180   ffffd90b14a8b600                 sh pipe_rd
8879   8879 3  16       180   ffffd908c7a44040                 sh pipe_rd
20035 20035 3   9         0   ffffd918a06a3700                 sh tstile
27233 27233 3  13         0   ffffd9de7d301040                 sh tstile
23703 23703 3  22       180   ffffd903d350e780                 sh wait
7301   7301 3  11         0   ffffd90860df9500                gcc vfork
...

	These ones look interesting:

4075   4075 3   5         0   ffffd903f5336080          winebuild tstile
18591 18591 3   1       180   ffffd9884718ea80                 sh wait
27904 27904 3  22       180   ffffd903d8719340                 sh wait
28380 28380 3  19       180   ffffd90536102780          winebuild wait

      The threads waiting in tstile are waiting on exec_lock, except
      for the winebuild child process:

db{0}> show all tstiles
  PID   LID          COMMAND      WAITING-FOR     WAIT-CHANNEL
19214 19214           screen                0 netbsd:exec_lock
15719 15719             cron                0 netbsd:exec_lock
23194 23194             cron                0 netbsd:exec_lock
...
23662 23662                R                0 netbsd:exec_lock
13798 13798              ghc                0 netbsd:exec_lock
 9896  9896              ghc                0 netbsd:exec_lock
 6801  6801              ghc                0 netbsd:exec_lock
  466   466              ghc                0 netbsd:exec_lock
 2432  2432              ghc                0 netbsd:exec_lock
23789 23789              ghc                0 netbsd:exec_lock
 4075  4075        winebuild                0 ffffd903f7ac6a90

	The winebuild child process appears to be waiting for
	rw_enter(&p->p_reflock, RW_WRITER):

db{0}> bt/t 0t4075
trace: pid 4075 lid 4075 at 0xffff9284f9ca8df0
sleepq_block() at netbsd:sleepq_block+0x13a
turnstile_block() at netbsd:turnstile_block+0x3d2
rw_enter() at netbsd:rw_enter+0x18b
spawn_return() at netbsd:spawn_return+0x67

(gdb) print 0x67
$7 = 103
(gdb) disas spawn_return+103
...
   0xffffffff80ce3018 <+98>:    callq  0xffffffff8023fbc0 <rw_enter>
   0xffffffff80ce301d <+103>:   mov    %gs:0xac0,%rax
...

   2356 	rw_enter(&p->p_reflock, RW_WRITER);
https://nxr.netbsd.org/xref/src/sys/kern/kern_exec.c?r=1.518.4.1#2356

	The lock it's waiting for appears to be its own p_reflock:

(gdb) print (long)&((struct proc *)0)->p_pid - (long)&((struct proc *)0)->p_reflock
$8 = -612

db{0}> x/ld ffffd903f7ac6a90-0t612
ffffd903f7ac682c:       4075

	It is held by a single reader with a writer waiting -- 0x23 is:

		(1 << RW_READ_COUNT_SHIFT=5)
		| RW_HAS_WAITERS=1
		| RW_WRITE_WANTED=2

db{0}> x/Lx ffffd903f7ac6a90
ffffd903f7ac6a90:       23

	The winebuild parent process is waiting for the child:

db{0}> bt/t 0t28380
trace: pid 28380 lid 28380 at 0xffff92867f69cc10
sleepq_block() at netbsd:sleepq_block+0x13a
cv_wait_sig() at netbsd:cv_wait_sig+0xbd
do_sys_waitid() at netbsd:do_sys_waitid+0x1ac
do_sys_wait() at netbsd:do_sys_wait+0x90
sys___wait450() at netbsd:sys___wait450+0x42
syscall() at netbsd:syscall+0x211
--- syscall (number 449) ---
netbsd:syscall+0x211:

	In this path, though, I don't think it can hold exec_lock or
	the child's p_reflock.

	exec_lock is also held by a single reader, with writers
	waiting:

db{0}> x/Lx exec_lock
netbsd:exec_lock:       23

	Not sure who holds it.  It could be the winebuild child that
	holds exec_lock, via this sketchy path:

   2345 	struct posix_spawnattr *attrs = spawn_data->sed_attrs;
   2346 	if ((!attrs || (attrs->sa_flags
   2347 		& (POSIX_SPAWN_RETURNERROR|POSIX_SPAWN_SETPGROUP)) == 0)
   2348 	    && rw_tryenter(&exec_lock, RW_READER)) {
   2349 		parent_is_waiting = false;
   2350 		mutex_enter(&spawn_data->sed_mtx_child);
   2351 		cv_signal(&spawn_data->sed_cv_child_ready);
   2352 		mutex_exit(&spawn_data->sed_mtx_child);
   2353 	}
https://nxr.netbsd.org/xref/src/sys/kern/kern_exec.c?r=1.518.4.1#2345

	But it could also be any other process in the middle of
	execve_loadvm.

	The culprit can't fixed by kern_exec.c rev. 1.522, pullup-10
	#1000, which added a missing rw_exit(&exec_lock) -- the path it
	added that to takes a write lock, not a read lock.

	It is interesting that there is a writer waiting.  Not many
	paths take a write lock on exec_lock.  Looks like this might be
	the R process:

db{0}> bt/t 0t23662
trace: pid 23662 lid 23662 at 0xffff9284efb0bc60
sleepq_block() at netbsd:sleepq_block+0x13a
turnstile_block() at netbsd:turnstile_block+0x3d2
rw_enter() at netbsd:rw_enter+0x18b
sigexit() at netbsd:sigexit+0x320
postsig() at netbsd:postsig+0x2c3
lwp_userret() at netbsd:lwp_userret+0x182
syscall() at netbsd:syscall+0x272
--- syscall (number 23662 via SYS_syscall) ---
netbsd:syscall+0x272:

(gdb) info line *(sigexit+0x320)
Line 2338 of "/home/riastradh/netbsd/10/src/sys/kern/kern_sig.c"
   starts at address 0xffffffff80d0680f <sigexit+800>
   and ends at 0xffffffff80d06826 <sigexit+823>.

(gdb) print 0x320
$9 = 800
(gdb) disas sigexit+800
...
   0xffffffff80d0680a <+795>:   callq  0xffffffff8023fbc0 <rw_enter>
   0xffffffff80d0680f <+800>:   mov    0x190(%rbx),%rsi

   2336 #ifdef PAX_SEGVGUARD
   2337 		rw_enter(&exec_lock, RW_WRITER);
   2338 		pax_segvguard(l, p->p_textvp, p->p_comm, true);
   2339 		rw_exit(&exec_lock);
   2340 #endif /* PAX_SEGVGUARD */
https://nxr.netbsd.org/xref/src/sys/kern/kern_sig.c?r=1.404#2336

	That is, it appears the R process is crashing and taking a core
	dump.  Unfortunately, I see no reason for it to be related to
	the winebuild process.

	I skimmed the stack traces of various other processes but I
	didn't find anything obvious.

>How-To-Repeat:

	run bulk builds repeatedly (maybe especially wine)

>Fix:

	Yes, please!

>Release-Note:

>Audit-Trail:

State-Changed-From-To: open->closed
State-Changed-By: riastradh@NetBSD.org
State-Changed-When: Sun, 13 Apr 2025 00:24:49 +0000
State-Changed-Why:
I took a closer look at the crash dump, and it turns out there is ps(1)
process on the CPU in the middle of kcopy, presumably about to fault
again in a loop just like PR kern/59175: posix_spawn hang, hanging
other process too (https://gnats.NetBSD.org/59175).  So, I think I'll
close this as duplicate since there's better diagnostics and discussion
over at PR 59175.


From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/59037 CVS commit: src/sys
Date: Sun, 27 Apr 2025 17:40:56 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Sun Apr 27 17:40:56 UTC 2025

 Modified Files:
 	src/sys/kern: kern_exec.c
 	src/sys/uvm: uvm_extern.h uvm_map.c

 Log Message:
 posix_spawn(2): Allocate a new vmspace at process creation time.

 This allocates a new vmspace for the process at the time the new
 process is created, rather than sharing some other vmspace temporarily.
 This eliminates any risk of anything bad happening due to temporary
 sharing, since there isn't any sharing.

 Resolves a race to where:

 1. we set up the child to share proc0.p_vmspace at first,

 2. another process tries to read the new child's psstrings via
    kern.proc_args.<childpid>.argv or similar with the child's
    p_reflock held and gets stuck in a uvm fault loop because
    proc0.p_vmspace doesn't have the child's psstrings address
    (inherited from the parent) mapped,

 3. the child is waiting for p_reflock before it can replace its
    p_vmspace or psstrings.

 By allocating the vmspace up front, with no mappings in it, we avoid
 exposing the child in this scenario.  Minor possible downside is that
 sysctl kern.proc_args.<childpid>.argv might spuriously fail with
 EFAULT during this time (rather than fail with EBUSY as it does if
 p_reflock is held concurrently) but that's not a particularly big
 deal.

 Patch and first paragraph of commit message written by chs@; minor
 tweaks to comments -- and any mistakes in the analysis -- by me.

 PR kern/59037: deadlock in posix_spawn
 PR kern/59175: posix_spawn hang, hanging other process too


 To generate a diff of this commit:
 cvs rdiff -u -r1.527 -r1.528 src/sys/kern/kern_exec.c
 cvs rdiff -u -r1.233 -r1.234 src/sys/uvm/uvm_extern.h
 cvs rdiff -u -r1.426 -r1.427 src/sys/uvm/uvm_map.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/59037 CVS commit: [netbsd-9] src/sys
Date: Fri, 9 May 2025 13:04:27 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Fri May  9 13:04:27 UTC 2025

 Modified Files:
 	src/sys/kern [netbsd-9]: kern_exec.c
 	src/sys/uvm [netbsd-9]: uvm_extern.h uvm_map.c

 Log Message:
 Pull up following revision(s) (requested by riastradh in ticket #1947):

 	sys/uvm/uvm_extern.h: revision 1.234 (via patch)
 	sys/kern/kern_exec.c: revision 1.528 (via patch)
 	sys/uvm/uvm_map.c: revision 1.427 (via patch)

 posix_spawn(2): Allocate a new vmspace at process creation time.

 This allocates a new vmspace for the process at the time the new
 process is created, rather than sharing some other vmspace temporarily.

 This eliminates any risk of anything bad happening due to temporary
 sharing, since there isn't any sharing.

 Resolves a race to where:
 1. we set up the child to share proc0.p_vmspace at first,
 2. another process tries to read the new child's psstrings via
    kern.proc_args.<childpid>.argv or similar with the child's
    p_reflock held and gets stuck in a uvm fault loop because
    proc0.p_vmspace doesn't have the child's psstrings address
    (inherited from the parent) mapped,
 3. the child is waiting for p_reflock before it can replace its
    p_vmspace or psstrings.

 By allocating the vmspace up front, with no mappings in it, we avoid
 exposing the child in this scenario.  Minor possible downside is that
 sysctl kern.proc_args.<childpid>.argv might spuriously fail with
 EFAULT during this time (rather than fail with EBUSY as it does if
 p_reflock is held concurrently) but that's not a particularly big
 deal.

 Patch and first paragraph of commit message written by chs@; minor
 tweaks to comments -- and any mistakes in the analysis -- by me.

 PR kern/59037: deadlock in posix_spawn
 PR kern/59175: posix_spawn hang, hanging other process too


 To generate a diff of this commit:
 cvs rdiff -u -r1.478.2.2 -r1.478.2.3 src/sys/kern/kern_exec.c
 cvs rdiff -u -r1.213 -r1.213.6.1 src/sys/uvm/uvm_extern.h
 cvs rdiff -u -r1.362.2.6 -r1.362.2.7 src/sys/uvm/uvm_map.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/59037 CVS commit: [netbsd-10] src/sys
Date: Fri, 9 May 2025 12:59:59 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Fri May  9 12:59:59 UTC 2025

 Modified Files:
 	src/sys/kern [netbsd-10]: kern_exec.c
 	src/sys/uvm [netbsd-10]: uvm_extern.h uvm_map.c

 Log Message:
 Pull up following revision(s) (requested by riastradh in ticket #1109):

 	sys/uvm/uvm_extern.h: revision 1.234
 	sys/kern/kern_exec.c: revision 1.528
 	sys/uvm/uvm_map.c: revision 1.427

 posix_spawn(2): Allocate a new vmspace at process creation time.

 This allocates a new vmspace for the process at the time the new
 process is created, rather than sharing some other vmspace temporarily.

 This eliminates any risk of anything bad happening due to temporary
 sharing, since there isn't any sharing.

 Resolves a race to where:
 1. we set up the child to share proc0.p_vmspace at first,
 2. another process tries to read the new child's psstrings via
    kern.proc_args.<childpid>.argv or similar with the child's
    p_reflock held and gets stuck in a uvm fault loop because
    proc0.p_vmspace doesn't have the child's psstrings address
    (inherited from the parent) mapped,
 3. the child is waiting for p_reflock before it can replace its
    p_vmspace or psstrings.

 By allocating the vmspace up front, with no mappings in it, we avoid
 exposing the child in this scenario.  Minor possible downside is that
 sysctl kern.proc_args.<childpid>.argv might spuriously fail with
 EFAULT during this time (rather than fail with EBUSY as it does if
 p_reflock is held concurrently) but that's not a particularly big
 deal.

 Patch and first paragraph of commit message written by chs@; minor
 tweaks to comments -- and any mistakes in the analysis -- by me.

 PR kern/59037: deadlock in posix_spawn
 PR kern/59175: posix_spawn hang, hanging other process too


 To generate a diff of this commit:
 cvs rdiff -u -r1.518.4.1 -r1.518.4.2 src/sys/kern/kern_exec.c
 cvs rdiff -u -r1.232 -r1.232.12.1 src/sys/uvm/uvm_extern.h
 cvs rdiff -u -r1.403.2.3 -r1.403.2.4 src/sys/uvm/uvm_map.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.