NetBSD Problem Report #53590
From gson@gson.org Tue Sep 11 08:26:03 2018
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 7553B7A1B3
for <gnats-bugs@gnats.NetBSD.org>; Tue, 11 Sep 2018 08:26:03 +0000 (UTC)
Message-Id: <20180911082557.88D8C989770@guava.gson.org>
Date: Tue, 11 Sep 2018 11:25:57 +0300 (EEST)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@NetBSD.org
Subject: DTrace fbt_getargdesc() fails with "Unknown function"
X-Send-Pr-Version: 3.95
>Number: 53590
>Category: kern
>Synopsis: DTrace fbt_getargdesc() fails with "Unknown function"
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: kern-bug-people
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Tue Sep 11 08:30:00 +0000 2018
>Closed-Date: Wed Apr 22 06:49:57 +0000 2020
>Last-Modified: Wed Apr 22 06:49:57 +0000 2020
>Originator: Andreas Gustafsson
>Release: NetBSD 8.0; also -current as of Sep 10
>Organization:
>Environment:
System: NetBSD
Architecture: x86_64
Machine: amd64
>Description:
Examining the arguments of any kernel function call using the DTrace FBT
provider and the args[] array fails. For example:
guido# modload dtrace_fbt
guido# dtrace -n 'fbt::vmem_alloc:entry { trace(args[0]); }'
dtrace: invalid probe specifier fbt::vmem_alloc:entry { trace(args[0]); }: in action list: index 0 is out of range for fbt::vmem_alloc:entry args[ ]
The following message appears on the console:
guido /netbsd: fbt_getargdesc(2067): Unknown function vmem_alloc!
It works on FreeBSD (tested on EC2 with ami-e83e6c97):
root@freebsd:/usr/home/ec2-user # dtrace -n 'fbt::vmem_alloc:entry { trace(args[0]); }'
dtrace: description 'fbt::vmem_alloc:entry ' matched 1 probe
CPU ID FUNCTION:NAME
0 57284 vmem_alloc:entry -2116181376
0 57284 vmem_alloc:entry -2116181376
^C
root@freebsd:/usr/home/ec2-user # dtrace -n 'fbt::vmem_alloc:entry { trace(args[0]->vm_name); }'
dtrace: description 'fbt::vmem_alloc:entry ' matched 1 probe
CPU ID FUNCTION:NAME
0 57284 vmem_alloc:entry buffer arena
0 57284 vmem_alloc:entry kmem arena
0 57284 vmem_alloc:entry buffer arena
0 57284 vmem_alloc:entry kernel arena
^C
>How-To-Repeat:
>Fix:
>Release-Note:
>Audit-Trail:
From: Patrick Welche <prlw1@cam.ac.uk>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/53590: DTrace fbt_getargdesc() fails with "Unknown function"
Date: Tue, 11 Sep 2018 11:29:57 +0100
On Tue, Sep 11, 2018 at 08:30:01AM +0000, Andreas Gustafsson wrote:
> >Release: NetBSD 8.0; also -current as of Sep 10
> guido# modload dtrace_fbt
> guido# dtrace -n 'fbt::vmem_alloc:entry { trace(args[0]); }'
> dtrace: invalid probe specifier fbt::vmem_alloc:entry { trace(args[0]); }: in action list: index 0 is out of range for fbt::vmem_alloc:entry args[ ]
>
> The following message appears on the console:
>
> guido /netbsd: fbt_getargdesc(2067): Unknown function vmem_alloc!
I see:
quark# modload dtrace_fbt
quark# dtrace -n 'fbt::vmem_alloc:entry { trace(args[0]); }'
dtrace: description 'fbt::vmem_alloc:entry ' matched 1 probe
CPU ID FUNCTION:NAME
1 20290 vmem_alloc:entry -2132062368
1 20290 vmem_alloc:entry -2132060864
1 20290 vmem_alloc:entry -2132062368
^C
quark# uname -a
NetBSD quark 8.99.25 NetBSD 8.99.25 (QUARK) #254: Mon Sep 10 11:38:46 BST 2018 prlw1@quark:/usr/src/sys/arch/amd64/compile/QUARK amd64
and in dmesg:
[ 6454.883380] dtrace: XXX dtrace_dof_property not implemented (name=dof-data-0)
From: Andreas Gustafsson <gson@gson.org>
To: Patrick Welche <prlw1@cam.ac.uk>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/53590: DTrace fbt_getargdesc() fails with "Unknown function"
Date: Tue, 11 Sep 2018 17:50:35 +0300
Patrick Welche wrote:
> I see:
>
> quark# modload dtrace_fbt
> quark# dtrace -n 'fbt::vmem_alloc:entry { trace(args[0]); }'
> dtrace: description 'fbt::vmem_alloc:entry ' matched 1 probe
> CPU ID FUNCTION:NAME
> 1 20290 vmem_alloc:entry -2132062368
> 1 20290 vmem_alloc:entry -2132060864
> 1 20290 vmem_alloc:entry -2132062368
> ^C
>
> quark# uname -a
> NetBSD quark 8.99.25 NetBSD 8.99.25 (QUARK) #254: Mon Sep 10 11:38:46 BST 2018 prlw1@quark:/usr/src/sys/arch/amd64/compile/QUARK amd64
Thank you, this is helpful.
I see you are using a custom kernel. How does it differ from GENERIC
- for example, have you done something to enable debug symbols?
--
Andreas Gustafsson, gson@gson.org
From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, kern-bug-people@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
gson@gson.org (Andreas Gustafsson)
Cc:
Subject: Re: kern/53590: DTrace fbt_getargdesc() fails with "Unknown function"
Date: Tue, 11 Sep 2018 12:49:37 -0400
On Sep 11, 2:55pm, gson@gson.org (Andreas Gustafsson) wrote:
-- Subject: Re: kern/53590: DTrace fbt_getargdesc() fails with "Unknown funct
| The following reply was made to PR kern/53590; it has been noted by GNATS.
|
| From: Andreas Gustafsson <gson@gson.org>
| To: Patrick Welche <prlw1@cam.ac.uk>
| Cc: gnats-bugs@NetBSD.org
| Subject: Re: kern/53590: DTrace fbt_getargdesc() fails with "Unknown function"
| Date: Tue, 11 Sep 2018 17:50:35 +0300
|
| Patrick Welche wrote:
| > I see:
| >
| > quark# modload dtrace_fbt
| > quark# dtrace -n 'fbt::vmem_alloc:entry { trace(args[0]); }'
| > dtrace: description 'fbt::vmem_alloc:entry ' matched 1 probe
| > CPU ID FUNCTION:NAME
| > 1 20290 vmem_alloc:entry -2132062368
| > 1 20290 vmem_alloc:entry -2132060864
| > 1 20290 vmem_alloc:entry -2132062368
| > ^C
| >
| > quark# uname -a
| > NetBSD quark 8.99.25 NetBSD 8.99.25 (QUARK) #254: Mon Sep 10 11:38:46 BST 2018 prlw1@quark:/usr/src/sys/arch/amd64/compile/QUARK amd64
|
| Thank you, this is helpful.
|
| I see you are using a custom kernel. How does it differ from GENERIC
| - for example, have you done something to enable debug symbols?
You need a fully modular kernel for dtrace to work, otherwise the ctf
tables overflow and you only get the bupkis symbol.
christos
From: Andreas Gustafsson <gson@gson.org>
To: christos@zoulas.com (Christos Zoulas)
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/53590: DTrace fbt_getargdesc() fails with "Unknown function"
Date: Tue, 11 Sep 2018 22:55:59 +0300
Christos Zoulas wrote:
> You need a fully modular kernel for dtrace to work, otherwise the ctf
> tables overflow and you only get the bupkis symbol.
Looks like you are talking about the CTF typeid overflow issue discussed in
https://mail-index.netbsd.org/tech-kern/2017/10/13/msg022436.html , but I
believe that is a different problem. My GENERIC kernels don't even have
enough type entries to overflow the table:
netbsd-8.0# ctfdump /netbsd | grep "total number of types"
total number of types = 617
netbsd-current# ctfdump /netbsd | grep "total number of types"
total number of types = 816
More to the point, this PR is about a lack of function entries, not
type entries. The GENERIC kernel in the 8.0 release and in the latest
daily build both have only a single dummy function in the "Functions"
section of the ctfdump output:
- Functions ------------------------------------------------------------------
[19881] FUNC (_debugsym_dummyfunc) returns: 608 args: (683)
- Types ----------------------------------------------------------------------
Same thing when I build a release myself using "build.sh release".
However, if I manually build a kernel using "build.sh kernel.gdb=GENERIC",
that not only builds a (huge) netbsd.gdb, but also overwrites the netbsd
(without the .gdb suffix) from "build.sh release" with an only slightly
larger one (27 megabytes instead of 26), and that one *does* include
CTF function entries, 34956 of them, including one for vmem_alloc:
# ctfdump netbsd|grep vmem_alloc
[33669] FUNC (vmem_alloc) returns: 1 args: (29302, 1791, 29301, 29303)
It seems to me this 27-megabyte kernel having the CTF function entries
is what releases ought to install by default, not the 26-megabyte
kernel lacking them.
And until both that and the typeid overflow issue have been fixed,
the HOWTO at https://wiki.netbsd.org/tutorials/how_to_enable_and_run_dtrace/
needs to be changed to give instructions that actually work.
--
Andreas Gustafsson, gson@gson.org
From: christos@zoulas.com (Christos Zoulas)
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/53590: DTrace fbt_getargdesc() fails with "Unknown function"
Date: Tue, 11 Sep 2018 17:09:09 -0400
On Sep 11, 10:55pm, gson@gson.org (Andreas Gustafsson) wrote:
-- Subject: Re: kern/53590: DTrace fbt_getargdesc() fails with "Unknown funct
| Christos Zoulas wrote:
| > You need a fully modular kernel for dtrace to work, otherwise the ctf
| > tables overflow and you only get the bupkis symbol.
|
| Looks like you are talking about the CTF typeid overflow issue discussed in
| https://mail-index.netbsd.org/tech-kern/2017/10/13/msg022436.html , but I
| believe that is a different problem. My GENERIC kernels don't even have
| enough type entries to overflow the table:
|
| netbsd-8.0# ctfdump /netbsd | grep "total number of types"
| total number of types = 617
|
| netbsd-current# ctfdump /netbsd | grep "total number of types"
| total number of types = 816
|
| More to the point, this PR is about a lack of function entries, not
| type entries. The GENERIC kernel in the 8.0 release and in the latest
| daily build both have only a single dummy function in the "Functions"
| section of the ctfdump output:
|
| - Functions ------------------------------------------------------------------
|
| [19881] FUNC (_debugsym_dummyfunc) returns: 608 args: (683)
|
| - Types ----------------------------------------------------------------------
|
| Same thing when I build a release myself using "build.sh release".
| However, if I manually build a kernel using "build.sh kernel.gdb=GENERIC",
| that not only builds a (huge) netbsd.gdb, but also overwrites the netbsd
| (without the .gdb suffix) from "build.sh release" with an only slightly
| larger one (27 megabytes instead of 26), and that one *does* include
| CTF function entries, 34956 of them, including one for vmem_alloc:
So something else broke recently... I will need to take a look.
Thanks,
christos
From: Patrick Welche <prlw1@cam.ac.uk>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/53590: DTrace fbt_getargdesc() fails with "Unknown function"
Date: Wed, 12 Sep 2018 10:32:16 +0100
On Tue, Sep 11, 2018 at 05:50:35PM +0300, Andreas Gustafsson wrote:
> I see you are using a custom kernel. How does it differ from GENERIC
> - for example, have you done something to enable debug symbols?
This one doesn't even have DIAGNOSTIC defined(!) and the modules weren't
compiled into the kernel:
# modstat | grep dtrace
dtrace driver filesys a 1 104233 solaris
dtrace_fbt misc filesys - 0 6824 dtrace,zlib
The kernel conf doesn't have any sort of "security" compiled in either.
From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@NetBSD.org
Cc: christos@zoulas.com (Christos Zoulas), Patrick Welche <prlw1@cam.ac.uk>
Subject: Re: kern/53590: DTrace fbt_getargdesc() fails with "Unknown function"
Date: Wed, 12 Sep 2018 19:17:35 +0300
Earlier, I wrote:
| However, if I manually build a kernel using "build.sh kernel.gdb=GENERIC",
| that not only builds a (huge) netbsd.gdb, but also overwrites the netbsd
| (without the .gdb suffix) from "build.sh release" with an only slightly
| larger one (27 megabytes instead of 26), and that one *does* include
| CTF function entries, 34956 of them, including one for vmem_alloc:
I did some further sleuthing. The part of "build.sh kernel.gdb=GENERIC"
that makes a difference is not the ".gdb", but the "kernel". In other
words,
build.sh tools kernel=GENERIC
compiles each kernel object with "-g" and produces a kernel with CTF
function information, but
build.sh release
using the exact same sources compiles each kernel object without "-g"
and produces a kernel with no CTF function information.
Looks like something in the release build is overriding the
makeoptions DEBUG="-g" in GENERIC.
--
Andreas Gustafsson, gson@gson.org
From: Andreas Gustafsson <gson@gson.org>
To: martin@NetBSD.org, gnats-bugs@NetBSD.org
Cc: christos@NetBSD.org, prlw1@cam.ac.uk, chuq@chuq.com
Subject: Re: kern/53590: DTrace fbt_getargdesc() fails with "Unknown function"
Date: Wed, 31 Jul 2019 13:32:48 +0300
Last year, I wrote:
> Looks like something in the release build is overriding the
> makeoptions DEBUG="-g" in GENERIC.
I found that "something". /etc/Makefile contains "-U DEBUG", which
was added in the following commit by martin:
revision 1.413
date: 2014-05-05 22:10:00 +0300; author: martin; state: Exp; lines: +2 -2; commitid: TIGgqdYim0L6Nnzx;
Use the new -U option for config to avoid generating kernel debug info
during release builds. This should save quite some space in the evbarm
build, where makeoptions DEBUG is heavily used.
Martin, could you please revert this? I see three reasons to do so:
- To fix this PR. This is a necessary step towards making tracing of
function arguments and return types work in DTrace, but sadly not
sufficient; there's still the CTF type table overflow issue
(https://mail-index.netbsd.org/tech-kern/2017/10/13/msg022436.html).
- It is really confusing that the GENERIC kernels produced by
"build.sh release" and "build.sh kernel=GENERIC" are not equivalent
(to the point that when I raised this question in the PR, no one
could explain why)
- It is a necessary step towards get kernel debug symbols included in
releases as discussed in PR 50128.
--
Andreas Gustafsson, gson@gson.org
From: Martin Husemann <martin@duskware.de>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@NetBSD.org, christos@NetBSD.org, prlw1@cam.ac.uk,
chuq@chuq.com
Subject: Re: kern/53590: DTrace fbt_getargdesc() fails with "Unknown function"
Date: Wed, 31 Jul 2019 12:43:17 +0200
On Wed, Jul 31, 2019 at 01:32:48PM +0300, Andreas Gustafsson wrote:
> Last year, I wrote:
> > Looks like something in the release build is overriding the
> > makeoptions DEBUG="-g" in GENERIC.
>
> I found that "something". /etc/Makefile contains "-U DEBUG", which
> was added in the following commit by martin:
>
> revision 1.413
> date: 2014-05-05 22:10:00 +0300; author: martin; state: Exp; lines: +2 -2; commitid: TIGgqdYim0L6Nnzx;
> Use the new -U option for config to avoid generating kernel debug info
> during release builds. This should save quite some space in the evbarm
> build, where makeoptions DEBUG is heavily used.
>
> Martin, could you please revert this? I see three reasons to do so:
We can test it (or mabye better conditionalize it on MKDTRACE).
It was necessary back then to get enough space on the autobuild slaves
to allow to parallel evb*arm* builds, which caused random failure
otherwise due to running out of space.
In the mean time the build cluster has been upgraded, and we reduced the
number of kernels for evbarm, so it may just work.
Martin
From: Thomas Klausner <tk@giga.or.at>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/53590: DTrace fbt_getargdesc() fails with "Unknown function"
Date: Wed, 31 Jul 2019 12:48:39 +0200
On Wed, Jul 31, 2019 at 10:35:01AM +0000, Andreas Gustafsson wrote:
> - To fix this PR. This is a necessary step towards making tracing of
> function arguments and return types work in DTrace, but sadly not
> sufficient; there's still the CTF type table overflow issue
> (https://mail-index.netbsd.org/tech-kern/2017/10/13/msg022436.html).
Is this last statement still correct after the recent commits by
hannken? (both from July 16)
Modified Files:
src/external/cddl/osnet/dist/tools/ctf/cvt: fixup_tdescs.c
Log Message:
Add fixup operation to remove private mutex types for x86.
For GENERIC the number of types goes down from 47351 to 33981.
To generate a diff of this commit:
cvs rdiff -u -r1.3 -r1.4 \
src/external/cddl/osnet/dist/tools/ctf/cvt/fixup_tdescs.c
Modified Files:
src/external/cddl/osnet/dist/tools/ctf/cvt: fixup_tdescs.c
Log Message:
XXX: A crude hack to bring down the number of types for a
GENRIC kernel below 2**15-1 (from ~34000 to ~29800).
Remove the type attributes "volatile", "const" and "restrict",
for DTRACE these attributes are of little value.
To generate a diff of this commit:
cvs rdiff -u -r1.4 -r1.5 \
src/external/cddl/osnet/dist/tools/ctf/cvt/fixup_tdescs.c
Thomas
From: Andreas Gustafsson <gson@gson.org>
To: Thomas Klausner <tk@giga.or.at>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/53590: DTrace fbt_getargdesc() fails with "Unknown function"
Date: Wed, 31 Jul 2019 14:00:16 +0300
Thomas Klausner wrote:
> Is this last statement still correct after the recent commits by
> hannken? (both from July 16)
Maybe not - the version of -current I tested was older than that.
I will retest.
--
Andreas Gustafsson, gson@gson.org
From: Andreas Gustafsson <gson@gson.org>
To: Thomas Klausner <tk@giga.or.at>, gnats-bugs@netbsd.org
Cc: martin@NetBSD.org, christos@NetBSD.org, prlw1@cam.ac.uk, chuq@chuq.com, hannken@NetBSD.org
Subject: Re: kern/53590: DTrace fbt_getargdesc() fails with "Unknown function"
Date: Wed, 31 Jul 2019 20:16:22 +0300
With a newer -current (2019.07.31.06.25.14 aka 9.99.1) that has hannken's
changes to reduce the number of types, the type tables no longer
overflow, so the "-U DEBUG" in /etc/Makefile is the only remaining
issue causing "dtrace -n 'fbt::vmem_alloc:entry { trace(args[0]); }'"
to fail.
--
Andreas Gustafsson, gson@gson.org
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/53590 CVS commit: src/etc
Date: Wed, 31 Jul 2019 18:35:58 +0000
Module Name: src
Committed By: martin
Date: Wed Jul 31 18:35:58 UTC 2019
Modified Files:
src/etc: Makefile
Log Message:
PR kern/53590: remove -U DEBUG from config(1) invocation (that is:
revert r1.413). DTrace needs the full symbol information.
To generate a diff of this commit:
cvs rdiff -u -r1.438 -r1.439 src/etc/Makefile
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->needs-pullups
State-Changed-By: gson@NetBSD.org
State-Changed-When: Sun, 04 Aug 2019 11:22:23 +0000
State-Changed-Why:
Fixed by martin, thanks!
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/53590 CVS commit: [netbsd-9] src/etc
Date: Wed, 7 Aug 2019 10:18:14 +0000
Module Name: src
Committed By: martin
Date: Wed Aug 7 10:18:13 UTC 2019
Modified Files:
src/etc [netbsd-9]: Makefile
Log Message:
Pull up following revision(s) (requested by hannken in ticket #26):
etc/Makefile: revision 1.439
PR kern/53590: remove -U DEBUG from config(1) invocation (that is:
revert r1.413). DTrace needs the full symbol information.
To generate a diff of this commit:
cvs rdiff -u -r1.438 -r1.438.2.1 src/etc/Makefile
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: needs-pullups->feedback
State-Changed-By: maya@NetBSD.org
State-Changed-When: Tue, 21 Apr 2020 19:52:56 +0000
State-Changed-Why:
Do you think netbsd-9 is far enough, or do you want to go further?
State-Changed-From-To: feedback->closed
State-Changed-By: gson@NetBSD.org
State-Changed-When: Wed, 22 Apr 2020 06:49:57 +0000
State-Changed-Why:
I consider this fixed as of the pullup to -9 in August.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.46 2020/01/03 16:35:01 leot Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2020
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home