NetBSD Problem Report #58630
From he@smistad.uninett.no Fri Aug 23 17:21:21 2024
Return-Path: <he@smistad.uninett.no>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits)
client-signature RSA-PSS (2048 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 7E5541A923F
for <gnats-bugs@gnats.NetBSD.org>; Fri, 23 Aug 2024 17:21:21 +0000 (UTC)
Message-Id: <20240823172116.7069F43F0B6@smistad.uninett.no>
Date: Fri, 23 Aug 2024 19:21:16 +0200 (CEST)
From: he@NetBSD.org
Reply-To: he@NetBSD.org
To: gnats-bugs@NetBSD.org
Subject: dtrace is "hit or miss", but mostly "miss"
X-Send-Pr-Version: 3.95
>Number: 58630
>Category: bin
>Synopsis: dtrace is "hit or miss", but mostly "miss"
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: bin-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Fri Aug 23 17:25:01 +0000 2024
>Last-Modified: Sat Aug 24 11:10:01 +0000 2024
>Originator: Havard Eidnes
>Release: NetBSD 10.0
>Organization:
I try...
>Environment:
System: NetBSD xxxxxx.uninett.no 10.0 NetBSD 10.0 (GENERIC) #0: Thu Mar 28 08:33:33 UTC 2024 mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/amd64/compile/GENERIC amd64
Architecture: x86_64
Machine: amd64
>Description:
While trying to build BIND 9.20.0, I stumble across this
problem. Trying to process
--- probes.d
/*
* Copyright (C) Internet Systems Consortium, Inc. ("ISC")
*
* SPDX-License-Identifier: MPL-2.0
*
* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, you can obtain one at https://mozilla.org/MPL/2.0/.
*
* See the COPYRIGHT file distributed with this work for additional
* information regarding copyright ownership.
*/
provider libisc {
probe job_cb_after(void *, void *, void *);
probe job_cb_before(void *, void *, void *);
probe rwlock_destroy(void *);
probe rwlock_downgrade(void *);
probe rwlock_init(void *);
probe rwlock_rdlock_acq(void *);
probe rwlock_rdlock_req(void *);
probe rwlock_rdunlock(void *);
probe rwlock_tryrdlock(void *, int);
probe rwlock_tryupgrade(void *, int);
probe rwlock_trywrlock(void *, int);
probe rwlock_wrlock_acq(void *);
probe rwlock_wrlock_req(void *);
probe rwlock_wrunlock(void *);
};
---
with
/usr/sbin/dtrace -s ./probes.d -h -o probes.h
may on some systems result in
dtrace: failed to compile script ./probes.d: "/usr/lib/dtrace/psinfo.d", line 46: syntax error near "u_int"
Looking at /usr/lib/dtrace/psinfo.d near that line does
not reveal anything disturbing to the untrained eye:
$ cat -n /usr/lib/dtrace/psinfo.d
...
33 typedef struct psinfo {
34 int pr_nlwp; /* number of threads */
35 pid_t pr_pid; /* unique process id */
36 pid_t pr_ppid; /* process id of parent */
37 pid_t pr_pgid; /* pid of process group leader */
38 pid_t pr_sid; /* session id */
39 uid_t pr_uid; /* real user id */
40 uid_t pr_euid; /* effective user id */
41 gid_t pr_gid; /* real group id */
42 gid_t pr_egid; /* effective group id */
43 uintptr_t
44 pr_addr; /* address of process */
45 string pr_psargs; /* process arguments */
46 u_int pr_arglen; /* process argument length */
47 } psinfo_t;
48
...
What's even more strange is that this error is not
consistently observed among my different systems:
System #1, where this is *not* observed:
$ egrep 'DISTRIBVER|Build ID' /etc/release
Build ID 202407270640Z
DISTRIBVER = '10.0_STABLE'
$ uname -a
NetBSD bxxx-rxx.uninett.no 10.0 NetBSD 10.0 (GENERIC) #0: Thu Mar 28 08:33:33 UTC 2024 mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/amd64/compile/GENERIC amd64
$
$ openssl dgst -sha1 /usr/sbin/dtrace
SHA1(/usr/sbin/dtrace)= ae65af77aff5ebc80d9db2712852b1b6d5a77131
$
$ openssl dgst -sha1 /netbsd
SHA1(/netbsd)= 7808ca002079afe78961b1da49ff5d48bda71fc0
$
System #2, where this problem *is* observed:
$ egrep 'DISTRIBVER|Build ID' /etc/release
Build ID 202407270640Z
DISTRIBVER = '10.0_STABLE'
$ uname -a
NetBSD oxxxxx.uninett.no 10.0 NetBSD 10.0 (GENERIC) #0: Thu Mar 28 08:33:33 UTC 2024 mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/amd64/compile/GENERIC amd64
$
$ openssl dgst -sha1 /usr/sbin/dtrace
SHA1(/usr/sbin/dtrace)= ae65af77aff5ebc80d9db2712852b1b6d5a77131
$
$ openssl dgst -sha1 /netbsd
SHA1(/netbsd)= 7808ca002079afe78961b1da49ff5d48bda71fc0
$
System #3, where this problem *is* also observed:
$ egrep 'DISTRIBVER|Build ID' /etc/release
DISTRIBVER = '10.0'
$ uname -a
NetBSD sxxxxx.xxx.uninett.no 10.99.11 NetBSD 10.99.11 (GENERIC) #6: Tue Aug 20 18:30:39 UTC 2024 he@stest.urc.uninett.no:/usr/obj/sys/arch/amd64/compile/GENERIC amd64
$
This system has a locally built -current kernel, and is still
running 10.0 user-land, but I suspect also that is locally
built, so /usr/sbin/dtrace isn't the same as above.
Privately, Taylor asked for output from "ctfdump -S /netbsd" on
each system, in particular "number of types", and they are:
#1:
total number of types = 29896
#2:
total number of types = 29896
#3:
total number of types = 30208
I do however see that each of them has more than 32k functions,
for what it's worth:
#1:
total number of functions = 36294
#2:
total number of functions = 36294
#3:
total number of functions = 36927
Not sure about whether that makes any difference, though.
Full output from "ctfdump -S /netbsd" can be appended later,
if need be.
I also dumped the types with "ctfdump -t /netbsd" on each of
them, and the thing related to u_int is:
#1:
<4> INTEGER unsigned int encoding=0x0 offset=0 bits=32
...
<271> TYPEDEF u_int refers to 4
#2:
<4> INTEGER unsigned int encoding=0x0 offset=0 bits=32
...
<271> TYPEDEF u_int refers to 4
#3:
<19> INTEGER unsigned int encoding=0x0 offset=0 bits=32
<20> TYPEDEF u_int refers to 19
"What gives?"
>How-To-Repeat:
Try to build BIND 9.20.0, watch the build on some systems
bomb out with
Making all in isc
DTRACE probes.h
dtrace: failed to compile script ./probes.d: "/usr/lib/dtrace/psinfo.d", line 46: syntax error near "u_int"
*** Error code 1
>Fix:
Yes, please.
>Audit-Trail:
From: RVP <rvp@SDF.ORG>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: bin/58630: dtrace is "hit or miss", but mostly "miss"
Date: Sat, 24 Aug 2024 05:33:58 +0000 (UTC)
On Fri, 23 Aug 2024, he@NetBSD.org wrote:
> What's even more strange is that this error is not
> consistently observed among my different systems:
>
```
$ env DTRACE_DEBUG=1 dtrace -s ./probes.d -h -o probes.h
libdtrace DEBUG: failed to open /dev/ksyms: Permission denied
[...]
dtrace: failed to compile script ./probes.d: "/usr/lib/dtrace/psinfo.d", line 46: syntax error near "u_int"
$ ls -l /dev/ksyms
cr--r----- 1 root kmem 85, 0 Jan 31 2024 /dev/ksyms
$ sudo chmod o+r /dev/ksyms
$ dtrace -s ./probes.d -h -o probes.h && echo OK
OK
$
```
Is /dev/ksyms world-readable on the other systems, or is dtrace setuid there?
I wonder if something like this would do instead of reading from /dev/ksyms?
```
diff -urN a/src/external/cddl/osnet/dist/lib/libdtrace/common/dt_module.c b/src/external/cddl/osnet/dist/lib/libdtrace/common/dt_module.c
--- a/src/external/cddl/osnet/dist/lib/libdtrace/common/dt_module.c 2024-04-02 02:28:51.360422339 +0000
+++ b/src/external/cddl/osnet/dist/lib/libdtrace/common/dt_module.c 2024-08-24 05:23:22.497654556 +0000
@@ -1186,7 +1186,14 @@
bool ismod;
if (strcmp("netbsd", name) == 0) {
- strlcpy(fname, _PATH_KSYMS, sizeof fname);
+ char tmp[MAXPATHLEN];
+ len = sizeof(tmp);
+#if defined(__i386__) || defined(__x86_64__)
+ if (sysctlbyname("machdep.booted_kernel", tmp, &len, NULL, 0) == 0)
+ snprintf(fname, sizeof fname, "%s%s", tmp[0] == '/' ? "" : "/", tmp);
+ else
+#endif
+ strlcpy(fname, _PATH_KSYMS, sizeof fname);
ismod = false;
} else {
```
-RVP
From: matthew green <mrg@eterna23.net>
To: gnats-bugs@netbsd.org, rvp@SDF.ORG
Cc: gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, he@NetBSD.org
Subject: re: bin/58630: dtrace is "hit or miss", but mostly "miss"
Date: Sat, 24 Aug 2024 16:24:16 +1000
> Is /dev/ksyms world-readable on the other systems, or is dtrace setuid =
there?
some time between netbsd-7 and netbsd-9 the default for /dev/ksyms
changed from 444 to 440. ah, here it is:
date: 2018-07-21 00:46:56 -0700; author: maxv; state: Exp; lines: +2 -2=
; commitid: 4dw22L6uN8Y2AYKA;
Create /dev/ksyms as "440 $g_kmem". This prevents unprivileged users from
reading the kernel symbols. Discussed in January 2018 on tech-kern@,
reported by maya@, tested by tih@.
> I wonder if something like this would do instead of reading from /dev/k=
syms?
[ .. ]
> + if (sysctlbyname("machdep.booted_kernel", tmp, &len, NULL, 0) =3D=3D=
0) =
no, this doesn't work if the boot media isn't mounted
and it doesn't support modules, or KALSR, or in the case
i've installed a new /netbsd but haven't yet rebooted.
this is a caveat of dtrace that i don't see - i normally
run it as root. one could enable access to /dev/ksyms
by accepting the info leak (kernel addresses) to all or
perhaps making relevant users part of kmem group.
we could make dtrace more obvious about this failure mode.
.mrg.
From: RVP <rvp@SDF.ORG>
To: matthew green <mrg@eterna23.net>
Cc: gnats-bugs@netbsd.org
Subject: re: bin/58630: dtrace is "hit or miss", but mostly "miss"
Date: Sat, 24 Aug 2024 06:31:04 +0000 (UTC)
On Sat, 24 Aug 2024, matthew green wrote:
> this is a caveat of dtrace that i don't see - i normally
> run it as root. one could enable access to /dev/ksyms
> by accepting the info leak (kernel addresses) to all or
> perhaps making relevant users part of kmem group.
>
No need. Comparing with FreeBSD gives a clue:
```
diff -urN a/src/external/cddl/osnet/lib/libdtrace/psinfo.d b/src/external/cddl/osnet/lib/libdtrace/psinfo.d
--- a/src/external/cddl/osnet/lib/libdtrace/psinfo.d 2018-05-28 21:05:09.000000000 +0000
+++ b/src/external/cddl/osnet/lib/libdtrace/psinfo.d 2024-08-24 06:23:00.489064017 +0000
@@ -30,6 +30,8 @@
* Use is subject to license terms.
*/
+#pragma D depends_on module netbsd
+
typedef struct psinfo {
int pr_nlwp; /* number of threads */
pid_t pr_pid; /* unique process id */
```
Then, that file is skipped if /dev/ksyms can't be read--which isn't fatal
for _this_ dtrace use-case.
HTH,
-RVP
From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@netbsd.org, gnats-admin@netbsd.org
Cc: netbsd-bugs@netbsd.org
Subject: Re: bin/58630: dtrace is "hit or miss", but mostly "miss"
Date: Sat, 24 Aug 2024 13:07:54 +0200 (CEST)
> > What's even more strange is that this error is not
> > consistently observed among my different systems:
>
> ```
> $ env DTRACE_DEBUG=1 dtrace -s ./probes.d -h -o probes.h
> libdtrace DEBUG: failed to open /dev/ksyms: Permission denied
> [...]
> dtrace: failed to compile script ./probes.d: "/usr/lib/dtrace/psinfo.d", line 46: syntax error near "u_int"
> $ ls -l /dev/ksyms
> cr--r----- 1 root kmem 85, 0 Jan 31 2024 /dev/ksyms
> $ sudo chmod o+r /dev/ksyms
> $ dtrace -s ./probes.d -h -o probes.h && echo OK
> OK
> $
> ```
>
> Is /dev/ksyms world-readable on the other systems [...]
Bingo!
> [...], or is dtrace setuid there?
Surely it should not be? (It isn't)
Hm, OK, so this goes perhaps back to the method I use when
updating my systems, and I thought that I mostly did it
"according to one of the recommended procedures", which is:
1) transfer all the sets (or build them from source) for the
version you want to upgrade to
2) extract the new kernel, install the kernel, reboot (after
renaming the old running kernel for easy reversion in case of
disaster)
3) in a shell loop from /, extract all the sets you want
installed (except etc and possibly xetc) with "tar xfzp". I
like using "progress" in this loop as well.
4) run "etcupdate -s <etc set file>"
5) run the postinstall command etcupdate suggests, and clear up
any additional things which needs to be done manually
If /dev/ksyms was initially in some version of MAKEDEV mode 440,
but should now be 444, I would have expected postinstall to warn
me about that, but it has not.
So this may turn into a bug report about postinstall and not
about dtrace, which I guess is an easier problem to deal with.
(It's a tangential issue that etcupdate by default suggests to
change files that are almost always an essential part of the
configuration of the running system, such as /etc/group,
/etc/master.passwd, /etc/rc.conf and /etc/hosts (and I'm sure
there are more in this category...), so it's "a sharp knife" and
requires full alertness when used...)
Returning back to postinstall and MAKEDEV, specifically
MAKEDEV.tmpl:
----------------------------
revision 1.191
date: 2018-07-21 09:46:56 +0200; author: maxv; state: Exp; lines: +2 -2; commitid: 4dw22L6uN8Y2AYKA;
Create /dev/ksyms as "440 $g_kmem". This prevents unprivileged users from
reading the kernel symbols. Discussed in January 2018 on tech-kern@,
reported by maya@, tested by tih@.
----------------------------
so this appears to have been done conciously, possibly as a
"security measure".
> I wonder if something like this would do instead of reading from /dev/ksyms?
>
> ```
> diff -urN a/src/external/cddl/osnet/dist/lib/libdtrace/common/dt_module.c b/src/external/cddl/osnet/dist/lib/libdtrace/common/dt_module.c
> --- a/src/external/cddl/osnet/dist/lib/libdtrace/common/dt_module.c 2024-04-02 02:28:51.360422339 +0000
> +++ b/src/external/cddl/osnet/dist/lib/libdtrace/common/dt_module.c 2024-08-24 05:23:22.497654556 +0000
> @@ -1186,7 +1186,14 @@
> bool ismod;
>
> if (strcmp("netbsd", name) == 0) {
> - strlcpy(fname, _PATH_KSYMS, sizeof fname);
> + char tmp[MAXPATHLEN];
> + len = sizeof(tmp);
> +#if defined(__i386__) || defined(__x86_64__)
> + if (sysctlbyname("machdep.booted_kernel", tmp, &len, NULL, 0) == 0)
> + snprintf(fname, sizeof fname, "%s%s", tmp[0] == '/' ? "" : "/", tmp);
> + else
> +#endif
> + strlcpy(fname, _PATH_KSYMS, sizeof fname);
> ismod = false;
> } else {
>
> ```
That is probably a good idea, especially if we don't revert the
setting of the file mode for /dev/ksyms.
It was quite confusing that e.g. ctfdump could get at the types
(I assume it's using the booted kernel, along the above pattern)
but dtrace could not, don't these tools basically come from the
same place? Or was it just our customization / porting which was
done inconsistently?
Best regards,
- Havard
(Contact us)
$NetBSD: query-full-pr,v 1.47 2022/09/11 19:34:41 kim Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2024
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home