NetBSD Problem Report #58425

From www@netbsd.org  Sat Jul 13 19:53:17 2024
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id CE65C1A923A
	for <gnats-bugs@gnats.NetBSD.org>; Sat, 13 Jul 2024 19:53:16 +0000 (UTC)
Message-Id: <20240713195315.A2E021A923B@mollari.NetBSD.org>
Date: Sat, 13 Jul 2024 19:53:15 +0000 (UTC)
From: campbell+netbsd@mumble.net
Reply-To: campbell+netbsd@mumble.net
To: gnats-bugs@NetBSD.org
Subject: negation of pid or pgid is UB for INT_MIN
X-Send-Pr-Version: www-1.0

>Number:         58425
>Category:       kern
>Synopsis:       negation of pid or pgid is UB for INT_MIN
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kre
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Jul 13 19:55:00 +0000 2024
>Closed-Date:    Wed Aug 07 11:47:30 +0000 2024
>Last-Modified:  Wed Aug 07 11:47:30 +0000 2024
>Originator:     Taylor R Campbell
>Release:        current, 10, 9, ...
>Organization:
The -PGID Foundation
>Environment:
>Description:
Certain syscalls such as kill(2) and ktrace(2) that take arguments of type pid_t accept a negative value and take its absolute value to be a process group id.

When the negative value is INT_MIN, however, negation is undefined.

We need to systematically avoid these paths, and it looks like we currently don't.  This could be a theoretical issue or it could be a practical issue, but the UB is a bug either way.
>How-To-Repeat:
code inspection
>Fix:
Yes, please!

>Release-Note:

>Audit-Trail:

Responsible-Changed-From-To: kern-bug-people->kre
Responsible-Changed-By: kre@NetBSD.org
Responsible-Changed-When: Sun, 14 Jul 2024 03:27:30 +0000
Responsible-Changed-Why:
I am looking into this PR


From: Robert Elz <kre@munnari.OZ.AU>
To: netbsd-bugs@netbsd.org, gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/58425: negation of pid or pgid is UB for INT_MIN
Date: Sun, 14 Jul 2024 10:52:19 +0700

     Date:        Sat, 13 Jul 2024 19:55:00 +0000 (UTC)
     From:        campbell+netbsd@mumble.net
     Message-ID:  <20240713195500.A81291A923C@mollari.NetBSD.org>

   | This could be a theoretical issue or it could be a practical issue,

 I believe that unless we have some architecture which traps when
 executing -INT_MIN rather than just generating INT_MIN as the result
 (which is conceivable, but I doubt anything supported will do that)
 then I suspect this is really a non-issue for practical purposes.
 Even if the architecture were to generate a random value (which UB
 allows) it would not be truly harmful ... an application is allowed
 to send any random value to these sys calls, and things must work,
 that the random value was generated by this operation, rather than
 some other way really doesn't matter.

 It is possible for an application to do kill(INT_MIN, sig) and pass
 through INT_MIN as the pid, which being negative, is treated as a
 pgrp, negated.   But assuming -INT_MIN actually becomes -INT_MIN
 or something else, not a trap, that would be harmless, it simply results
 in a search for something different -- if it remains INT_MIN, the normal
 case for most 2's complement architectures, or anything else negative,
 it cannot be found, as all pids (and hence all pgrp ids) are positive
 (or 0 for the kernel) a negative value (any negative value) can never
 match (that's how this hack was invented).   So the result would have
 just been ESRCH anyway I believe.

 I am however in the process of validating the value, and not performing
 the undefined operation.   Note, that as (ktrace & ptrace excepted, where
 the pid arg is actually defined to be an int) generally dealing with pid_t
 values, which we cannot assume are int ... it could be short (and once
 was, even earlier I think it might have been char, not that the name
 "pid_t" existed then, but that was a long long time ago) which would
 never give a problem, it gets promoted to int for both the -pid
 operation (and so could never be INT_MIN) and to pass as a parameter.
 Alternatively, pid_t might be wider than an int (could be long)
 in which case the value might be < INT_MIN, so I am making (most of)
 the tests check that the value is > INT_MIN not just != INT_MIN.

 My test build looks to have succeeded (on amd64) - currently building
 the sets - I will do an i386 build, and then commit the changes, and
 after a few days, see about pullups to -10 and -9.

 The only system place outside the kernel I could see was in the killpg()
 function (compat function, for old BSD code, posix defines the kill(-pid)
 method) - that one will be fixed as well.

 Random applications or other libraries which might accept a pid, and
 negate it, will need to wait for some other time (if there are any).

 kre

 ps: Taylor, if you feel we need ATF test cases for this, by all means,
 go ahead.


From: matthew green <mrg@eterna23.net>
To: Robert Elz <kre@munnari.OZ.AU>
Cc: netbsd-bugs@netbsd.org, gnats-bugs@netbsd.org
Subject: re: kern/58425: negation of pid or pgid is UB for INT_MIN
Date: Sun, 14 Jul 2024 14:57:10 +1000

 >   | This could be a theoretical issue or it could be a practical issue,
 >
 > I believe that unless we have some architecture which traps when
 > executing -INT_MIN rather than just generating INT_MIN as the result

 this may not be relevant, but it triggered me. :-)

 i recently was dealing with a trap related to exactly this.

 i was running atf on m68k virt68k qemu and it crashed the emulator
 itself.  turns out -INT_MAX/-1 traps on x86.

 i have this uncommited/unsubmitted patch for qemu:

    https://www.netbsd.org/~mrg/patch-target_m68k_op_helper.c


 .mrg.

From: "Robert Elz" <kre@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/58425 CVS commit: src
Date: Sun, 14 Jul 2024 05:10:40 +0000

 Module Name:	src
 Committed By:	kre
 Date:		Sun Jul 14 05:10:40 UTC 2024

 Modified Files:
 	src/lib/libc/compat-43: killpg.c
 	src/sys/kern: kern_descrip.c kern_ktrace.c kern_proc.c sys_sig.c tty.c

 Log Message:
 PR kern/58425 --  Disallow INT_MIN as a (negative) pid arg.

 Since -INT_MIN is undefined, and the point of negative pid args is
 to negate them, and use the result as a pgrp id instead, we need
 to avoid accidentally negating INT_MIN.

 Since pid_t is just an integral type, of unspecified width, when
 testing pid_t value test for <= INT_MIN (or > INT_MIN sometimes)
 rather than == INT_MIN.   When testing int values, just == INT_MIN
 is all that is needed, < INT_MIN cannot occur.

 XXX pullup -9, -10


 To generate a diff of this commit:
 cvs rdiff -u -r1.9 -r1.10 src/lib/libc/compat-43/killpg.c
 cvs rdiff -u -r1.262 -r1.263 src/sys/kern/kern_descrip.c
 cvs rdiff -u -r1.184 -r1.185 src/sys/kern/kern_ktrace.c
 cvs rdiff -u -r1.275 -r1.276 src/sys/kern/kern_proc.c
 cvs rdiff -u -r1.57 -r1.58 src/sys/kern/sys_sig.c
 cvs rdiff -u -r1.312 -r1.313 src/sys/kern/tty.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: Robert Elz <kre@munnari.OZ.AU>
To: matthew green <mrg@eterna23.net>
Cc: netbsd-bugs@netbsd.org, gnats-bugs@netbsd.org
Subject: Re: kern/58425: negation of pid or pgid is UB for INT_MIN
Date: Sun, 14 Jul 2024 13:26:53 +0700

     Date:        Sun, 14 Jul 2024 14:57:10 +1000
     From:        matthew green <mrg@eterna23.net>
     Message-ID:  <22479.1720933030@splode.eterna23.net>

   | i recently was dealing with a trap related to exactly this.
   |
   | i was running atf on m68k virt68k qemu and it crashed the emulator
   | itself.  turns out -INT_MAX/-1 traps on x86.

 I think you mean INT_MIN / -1 (INT_MIN != -INT_MAX - which is the whole
 issue).

 That would be a different problem than the pid one, which is just
 taking -INT_MIN, no division, by anything, ever performed (I'm not
 sure what a half of a pid_t would represent).   Further, it should
 be fixed in the kernel now.

   | i have this uncommited/unsubmitted patch for qemu:
   |    https://www.netbsd.org/~mrg/patch-target_m68k_op_helper.c

 Looks reasonable to me.   Though I know nothing about qemu

 kre

State-Changed-From-To: open->needs-pullups
State-Changed-By: kre@NetBSD.org
State-Changed-When: Sun, 14 Jul 2024 12:05:53 +0000
State-Changed-Why:
Needs pullups to -9 and -10


From: Taylor R Campbell <riastradh@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: kre@NetBSD.org, gnats-admin@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: kern/58425: negation of pid or pgid is UB for INT_MIN
Date: Sun, 14 Jul 2024 13:55:11 +0000

 > Date: Sun, 14 Jul 2024 10:52:19 +0700
 > From: Robert Elz <kre@munnari.OZ.AU>
 > 
 >      Date:        Sat, 13 Jul 2024 19:55:00 +0000 (UTC)
 >      From:        campbell+netbsd@mumble.net
 >      Message-ID:  <20240713195500.A81291A923C@mollari.NetBSD.org>
 >  
 >    | This could be a theoretical issue or it could be a practical issue,
 >  
 > I believe that unless we have some architecture which traps when
 > executing -INT_MIN rather than just generating INT_MIN as the result
 > (which is conceivable, but I doubt anything supported will do that)
 > then I suspect this is really a non-issue for practical purposes.
 > Even if the architecture were to generate a random value (which UB
 > allows) it would not be truly harmful ... an application is allowed
 > to send any random value to these sys calls, and things must work,
 > that the random value was generated by this operation, rather than
 > some other way really doesn't matter.

 The issue isn't so much that the architecture's SUB or NEG instruction
 might trap or produce wild behaviour -- although in principle it
 could, and not even in a pathological architecture design, just like
 signed division INT_MIN/-1 traps on x86.

 The more likely risk is that a _compiler_ may draw an inference from
 an unprotected -pgid expression -- it may infer that along any control
 flow paths that unconditionally pass through that expression, pgid
 must not be INT_MIN, because if it were then the behaviour would be
 undefined and we all know C programmers never write programs that
 could lead to UB on the possible inputs.

 So, for example, a compiler might optimize (a)

 	*p = -pgid;
 	return (pgid == INT_MIN);

 into

 	*p = -pgid;
 	return false;

 or it might optimize (b)

 	return (-pgid == pgid);

 into

 	return (pgid == 0);

 where you might expect it (or require it, with -fwrapv) to be
 equivalent instead to

 	return (pgid == 0 || pgid == INT_MIN);

 I haven't seen (a) in practice but (b) is easy to exhibit:

 https://godbolt.org/z/rzxjf38aY

 > I am however in the process of validating the value, and not performing
 > the undefined operation.   Note, that as (ktrace excepted, where the
 > pid arg is actually defined to be an int) generally dealing with pid_t
 > values, which we cannot assume are int ... it could be short (and once
 > was, even earlier I think it might have been char, not that the name
 > "pid_t" existed then, but that was a long long time ago) which would
 > never give a problem, it gets promoted to int for both the -pid
 > operation (and so could never be INT_MIN) and to pass as a parameter.
 > Alternatively, pid_t might be wider than an int (could be long)
 > in which case the value might be < INT_MIN, so I am making (most of)
 > the tests check that the value is > INT_MIN not just != INT_MIN.

 Thanks!  Maybe we should use -PID_MAX as the limit instead of INT_MIN,
 to make it more resilient to (unlikely but possible) changes of the
 type and easier to audit?

 ...Except PID_MAX apparently isn't actually the max, once we cross
 ~16k processes.  So maybe we should have a different name for the
 _real_ maximum pid.

 > ps: Taylor, if you feel we need ATF test cases for this, by all means,
 > go ahead.

 I added one for kill(INT_MIN, 0).  We should add more tests for ktrace
 and for FIOSETOWN on a tty (uses tty.c path) and a pipe or socket
 (uses kern_proc.c fsetown path).

From: Jason Thorpe <thorpej@me.com>
To: gnats-bugs@netbsd.org
Cc: kre@netbsd.org,
 gnats-admin@netbsd.org,
 netbsd-bugs@netbsd.org,
 campbell+netbsd@mumble.net,
 mrg@netbsd.org
Subject: Re: kern/58425: negation of pid or pgid is UB for INT_MIN
Date: Sun, 14 Jul 2024 07:33:56 -0700

 > On Jul 13, 2024, at 10:00=E2=80=AFPM, matthew green <mrg@eterna23.net> =
 wrote:
 >=20
 > i was running atf on m68k virt68k qemu and it crashed the emulator
 > itself.  turns out -INT_MAX/-1 traps on x86.
 >=20
 > i have this uncommited/unsubmitted patch for qemu:
 >=20
 >    https://www.netbsd.org/~mrg/patch-target_m68k_op_helper.c

 I have an outstanding open issue with qemu about this.  I don=E2=80=99t =
 know why it=E2=80=99s stalled.  I haven=E2=80=99t had much time to spend =
 on qemu stuff (long enough for my mailing list sub to time out), but =
 I=E2=80=99ll make a point of circling back around to check on it.  Mark =
 had a patch for it, even.

 -- thorpej

From: Jason Thorpe <thorpej@me.com>
To: gnats-bugs@netbsd.org
Cc: kre@netbsd.org,
 gnats-admin@netbsd.org,
 netbsd-bugs@netbsd.org,
 campbell+netbsd@mumble.net,
 mrg@netbsd.org
Subject: Re: kern/58425: negation of pid or pgid is UB for INT_MIN
Date: Sun, 14 Jul 2024 07:39:00 -0700

 > On Jul 14, 2024, at 7:33=E2=80=AFAM, Jason Thorpe <thorpej@me.com> =
 wrote:
 >=20
 > I have an outstanding open issue with qemu about this.  I don=E2=80=99t =
 know why it=E2=80=99s stalled.  I haven=E2=80=99t had much time to spend =
 on qemu stuff (long enough for my mailing list sub to time out), but =
 I=E2=80=99ll make a point of circling back around to check on it.  Mark =
 had a patch for it, even.

 FWIW: https://gitlab.com/qemu-project/qemu/-/issues/2078

 -- thorpej

From: matthew green <mrg@eterna23.net>
To: Jason Thorpe <thorpej@me.com>
Cc: kre@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
    campbell+netbsd@mumble.net, gnats-bugs@netbsd.org
Subject: re: kern/58425: negation of pid or pgid is UB for INT_MIN
Date: Mon, 15 Jul 2024 08:19:41 +1000

 > FWIW: https://gitlab.com/qemu-project/qemu/-/issues/2078

 thanks.

 the patch in this issue isn't complete -- there are at least 3
 instances of the bug that my patch fixes.

 i'll share my version there and see what happens.


 .mrg.

From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/58425 CVS commit: src/tests/lib/libc/sys
Date: Mon, 15 Jul 2024 06:19:07 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Mon Jul 15 06:19:07 UTC 2024

 Modified Files:
 	src/tests/lib/libc/sys: t_kill.c

 Log Message:
 tests/lib/libc/sys/t_kill: Test kill(INT_MIN, ...) fails with ESRCH.

 PR kern/58425


 To generate a diff of this commit:
 cvs rdiff -u -r1.1 -r1.2 src/tests/lib/libc/sys/t_kill.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: needs-pullups->pending-pullups
State-Changed-By: kre@NetBSD.org
State-Changed-When: Sun, 04 Aug 2024 02:00:14 +0000
State-Changed-Why:
[pullup-10 #773]
[pullup-9 #1859]


From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/58425 CVS commit: [netbsd-10] src
Date: Wed, 7 Aug 2024 10:04:48 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Wed Aug  7 10:04:48 UTC 2024

 Modified Files:
 	src/lib/libc/compat-43 [netbsd-10]: killpg.c
 	src/sys/kern [netbsd-10]: kern_descrip.c kern_ktrace.c kern_proc.c
 	    sys_sig.c tty.c
 	src/tests/lib/libc/sys [netbsd-10]: t_kill.c

 Log Message:
 Pull up following revision(s) (requested by kre in ticket #773):

 	sys/kern/kern_proc.c: revision 1.276
 	sys/kern/kern_ktrace.c: revision 1.185
 	sys/kern/sys_sig.c: revision 1.58
 	sys/kern/kern_descrip.c: revision 1.263
 	lib/libc/compat-43/killpg.c: revision 1.10
 	sys/kern/tty.c: revision 1.313
 	tests/lib/libc/sys/t_kill.c: revision 1.2

 PR kern/58425 --  Disallow INT_MIN as a (negative) pid arg.

 Since -INT_MIN is undefined, and to point of negative pid args is
 to negate them, and use the result as a pgrp id instead, we need
 to avoid accidentally negating INT_MIN.

 Since pid_t is just an integral type, of unspecified width, when
 testing pid_t value test for <= INT_MIN (or > INT_MIN sometimes)
 rather than == INT_MIN.   When testing int values, just == INT_MIN
 is all that is needed, < INT_MIN cannot occur.

 tests/lib/libc/sys/t_kill: Test kill(INT_MIN, ...) fails with ESRCH.
 PR kern/58425


 To generate a diff of this commit:
 cvs rdiff -u -r1.9 -r1.9.14.1 src/lib/libc/compat-43/killpg.c
 cvs rdiff -u -r1.251.10.1 -r1.251.10.2 src/sys/kern/kern_descrip.c
 cvs rdiff -u -r1.182 -r1.182.4.1 src/sys/kern/kern_ktrace.c
 cvs rdiff -u -r1.269 -r1.269.2.1 src/sys/kern/kern_proc.c
 cvs rdiff -u -r1.56 -r1.56.4.1 src/sys/kern/sys_sig.c
 cvs rdiff -u -r1.307 -r1.307.2.1 src/sys/kern/tty.c
 cvs rdiff -u -r1.1 -r1.1.56.1 src/tests/lib/libc/sys/t_kill.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/58425 CVS commit: [netbsd-9] src
Date: Wed, 7 Aug 2024 10:11:45 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Wed Aug  7 10:11:45 UTC 2024

 Modified Files:
 	src/lib/libc/compat-43 [netbsd-9]: killpg.c
 	src/sys/kern [netbsd-9]: kern_descrip.c kern_ktrace.c kern_proc.c
 	    sys_sig.c tty.c
 	src/tests/lib/libc/sys [netbsd-9]: t_kill.c

 Log Message:
 Pull up following revision(s) (requested by kre in ticket #1859):

 	sys/kern/kern_proc.c: revision 1.276 (via patch)
 	sys/kern/kern_ktrace.c: revision 1.185 (via patch)
 	sys/kern/sys_sig.c: revision 1.58 (via patch)
 	sys/kern/kern_descrip.c: revision 1.263 (via patch)
 	lib/libc/compat-43/killpg.c: revision 1.10
 	sys/kern/tty.c: revision 1.313 (via patch)
 	tests/lib/libc/sys/t_kill.c: revision 1.2

 PR kern/58425 --  Disallow INT_MIN as a (negative) pid arg.
 Since -INT_MIN is undefined, and to point of negative pid args is
 to negate them, and use the result as a pgrp id instead, we need
 to avoid accidentally negating INT_MIN.

 Since pid_t is just an integral type, of unspecified width, when
 testing pid_t value test for <= INT_MIN (or > INT_MIN sometimes)
 rather than == INT_MIN.   When testing int values, just == INT_MIN
 is all that is needed, < INT_MIN cannot occur.

 tests/lib/libc/sys/t_kill: Test kill(INT_MIN, ...) fails with ESRCH.
 PR kern/58425


 To generate a diff of this commit:
 cvs rdiff -u -r1.9 -r1.9.6.1 src/lib/libc/compat-43/killpg.c
 cvs rdiff -u -r1.243 -r1.243.4.1 src/sys/kern/kern_descrip.c
 cvs rdiff -u -r1.173 -r1.173.4.1 src/sys/kern/kern_ktrace.c
 cvs rdiff -u -r1.233.2.1 -r1.233.2.2 src/sys/kern/kern_proc.c
 cvs rdiff -u -r1.47.4.3 -r1.47.4.4 src/sys/kern/sys_sig.c
 cvs rdiff -u -r1.281 -r1.281.4.1 src/sys/kern/tty.c
 cvs rdiff -u -r1.1 -r1.1.44.1 src/tests/lib/libc/sys/t_kill.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: pending-pullups->closed
State-Changed-By: riastradh@NetBSD.org
State-Changed-When: Wed, 07 Aug 2024 11:47:30 +0000
State-Changed-Why:
fixed in HEAD, pulled up to 9 and 10
(although more automatic tests would be nice)


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.