NetBSD Problem Report #55557

From gson@gson.org  Sun Aug  9 09:13:17 2020
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id C82761A9239
	for <gnats-bugs@gnats.NetBSD.org>; Sun,  9 Aug 2020 09:13:17 +0000 (UTC)
Message-Id: <20200809091312.A7224253EDE@guava.gson.org>
Date: Sun,  9 Aug 2020 12:13:12 +0300 (EEST)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@NetBSD.org
Subject: Releases should include kernel debug symbols
X-Send-Pr-Version: 3.95

>Number:         55557
>Category:       toolchain
>Synopsis:       Releases should include kernel debug symbols
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    toolchain-manager
>State:          open
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Sun Aug 09 09:15:00 +0000 2020
>Last-Modified:  Thu Oct 27 06:20:01 +0000 2022
>Originator:     Andreas Gustafsson
>Release:        NetBSD-current
>Organization:
>Environment:
System: NetBSD
Architecture: all
Machine: all
>Description:

The official NetBSD binary releases have now included debug symbols
for userland programs for some time, in the optional "debug" sets,
but they do not include debug symbols for the kernel(s).

This seems backwards to me - I think kernel debug symbols are both
more important to have, and less costly in terms of space on the
install media, so if we include only one kind, it should be the kernel
ones.  And in most cases we probably could, and should, include both.

PR 50128 is somewhat related to this, but narrower in scope; that PR
is about having a way to do private release builds with kernel debug
symbols for a specific port, but having them in the official releases
for all ports would bring a far greater benefit by making it easier to
analyze kernel bugs when they impact users who have installed a binary
release.  Also, one reason why no progress has been made on 50128 may
be the difficulty of choosing between many different possible
solutions, and restricting the solution space to those that what will
work for the official releases may make the choice easier.

>How-To-Repeat:

>Fix:

>Audit-Trail:
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: toolchain/55557: Releases should include kernel debug symbols
Date: Sun, 9 Aug 2020 11:45:11 +0200

 On Sun, Aug 09, 2020 at 09:15:00AM +0000, Andreas Gustafsson wrote:
 > PR 50128 is somewhat related to this, but narrower in scope

 Building debug info for the kernel(s) is enabled by adding
 MKKDEBUG=yes (I have it in /etc/mk.conf, but should work as well if
 passed via -V on the build.sh command line). For each netbsd
 kernel generated, it creates an additional netbsd.gdb.
 But this is not installed / packaged anywheree.

 Christos recently added .debug files for modules, so with MKDEBUG=yes you
 should have all symbols in the build results.

 So we need to somehow package the netbsd.gdb files and make them available
 in the release dir.

 Martin

From: matthew green <mrg@eterna.com.au>
To: gnats-bugs@netbsd.org, martin@duskware.de
Cc: toolchain-manager@netbsd.org, gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org, gson@gson.org (Andreas Gustafsson)
Subject: re: toolchain/55557: Releases should include kernel debug symbols
Date: Mon, 10 Aug 2020 06:15:08 +1000

 thanks Andreas.

 we really should have a better solution around this.

 Martin Husemann writes:
 > So we need to somehow package the netbsd.gdb files and make them available
 > in the release dir.

 the big difference between these netbsd.gdb files and the
 foo.debug files in the debug sets is that netbsd.gdb has
 the full kernel *and* debug info, ie, it's a super set of
 the kernel itself.  for standard installs, we could have
 a normal path and debug-link link from /netbsd to the
 netbsd.debug, but managing those for multiple installs
 sounds difficult (eg, you want netbsd.old to point to the
 netbsd.old.debug you some how created.)

 or we could simply eat the space (but it's a lot of you
 have a lot of kernels!)


 .mrg.

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: toolchain/55557: Releases should include kernel debug symbols
Date: Sun, 9 Aug 2020 23:37:28 +0200

 On Sun, Aug 09, 2020 at 08:20:01PM +0000, matthew green wrote:
 >  the big difference between these netbsd.gdb files and the
 >  foo.debug files in the debug sets is that netbsd.gdb has
 >  the full kernel *and* debug info, ie, it's a super set of
 >  the kernel itself.

 Yes. And it is not clear for which media/images/install types the full
 debug info kernel would be usefull, so we should not change that.

 But having an easy way to download the matching netbsd.gdb for releases
 (and "daily" builds) would be great.

 I'm not sure that would be enough to resolve this PR, but it is a very simple
 step forward.

 Martin

From: "Christos Zoulas" <christos@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/55557 CVS commit: src
Date: Mon, 20 Dec 2021 09:41:26 -0500

 Module Name:	src
 Committed By:	christos
 Date:		Mon Dec 20 14:41:26 UTC 2021

 Modified Files:
 	src/etc: Makefile
 	src/share/mk: bsd.README bsd.own.mk

 Log Message:
 PR/55557: Andreas Gustafsson" Introduce a new variable MKDEBUGKERNEL which
 as the name implies, includes a netbsd.gdb inside each kernel set:
 $ tar -tzvf kern-GENERIC.tgz
 -rwxr-xr-x  0 root   wheel 29398264 Dec 19 12:50 ./netbsd
 -rwxr-xr-x  0 root   wheel 208125880 Dec 19 12:50 ./netbsd.gdb


 To generate a diff of this commit:
 cvs rdiff -u -r1.452 -r1.453 src/etc/Makefile
 cvs rdiff -u -r1.425 -r1.426 src/share/mk/bsd.README
 cvs rdiff -u -r1.1270 -r1.1271 src/share/mk/bsd.own.mk

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: Andreas Gustafsson <gson@gson.org>
To: christos@NetBSD.org
Cc: gnats-bugs@netbsd.org
Subject: Re: PR/55557 CVS commit: src
Date: Tue, 21 Dec 2021 10:25:05 +0200

 Christos Zoulas wrote:
 >  PR/55557: Andreas Gustafsson" Introduce a new variable MKDEBUGKERNEL which
 >  as the name implies, includes a netbsd.gdb inside each kernel set:
 >  $ tar -tzvf kern-GENERIC.tgz
 >  -rwxr-xr-x  0 root   wheel 29398264 Dec 19 12:50 ./netbsd
 >  -rwxr-xr-x  0 root   wheel 208125880 Dec 19 12:50 ./netbsd.gdb

 I suppose this is a step in the right direction, and it could even be
 considered a fix for PR 50128, because it may now finally be possible
 to construct a custom build that can be installed automatically using
 anita and symbolically debugged using kgdb on a pair of qemu VMs for
 architectures other than i386.

 However, since MKDEBUGKERNEL is not enabled by default, and probably
 shouldn't be due to the set size and disk space requirement, it is not
 sufficient to fix the present PR, which is specifically about the
 official binary releases.  My preferred solution for those would be a
 separate debugkern-GENERIC set containing a netbsd.gdb, and possibly
 but not necessarily, corresponding debugkern-FOO sets for other
 kernels.

 It also does not solve the issue of kernel debugging on the testbeds
 (as in the recent case of PR 56554), because those should be testing
 configurations that are as close as possible to what we ship in the
 official binary releases.
 -- 
 Andreas Gustafsson, gson@gson.org

From: Andreas Gustafsson <gson@gson.org>
To: christos@NetBSD.org
Cc: gnats-bugs@netbsd.org
Subject: Re: PR/55557 CVS commit: src
Date: Thu, 23 Dec 2021 10:25:26 +0200

 Christos Zoulas wrote:
 >  PR/55557: Andreas Gustafsson" Introduce a new variable MKDEBUGKERNEL which
 >  as the name implies, includes a netbsd.gdb inside each kernel set:
 >  $ tar -tzvf kern-GENERIC.tgz
 >  -rwxr-xr-x  0 root   wheel 29398264 Dec 19 12:50 ./netbsd
 >  -rwxr-xr-x  0 root   wheel 208125880 Dec 19 12:50 ./netbsd.gdb

 This is not working for me - after building an amd64 release using
 "build.sh -V MKDEBUGKERNEL=yes", kern-GENERIC.tar.xz does not include
 a netbsd.gdb.

 Where is the code that adds netbsd.gdb to the set?
 -- 
 Andreas Gustafsson, gson@gson.org

From: Christos Zoulas <christos@zoulas.com>
To: Andreas Gustafsson <gson@gson.org>
Cc: "christos@netbsd.org" <christos@NetBSD.org>,
 gnats-bugs@netbsd.org
Subject: Re: PR/55557 CVS commit: src
Date: Thu, 23 Dec 2021 03:29:10 -0500

 --Apple-Mail=_A8BB0961-082F-4D78-93BA-FC5A03B4194D
 Content-Transfer-Encoding: 7bit
 Content-Type: text/plain;
 	charset=us-ascii



 > On Dec 23, 2021, at 3:25 AM, Andreas Gustafsson <gson@gson.org> wrote:
 > 
 > This is not working for me - after building an amd64 release using
 > "build.sh -V MKDEBUGKERNEL=yes", kern-GENERIC.tar.xz does not include
 > a netbsd.gdb.
 > 
 > Where is the code that adds netbsd.gdb to the set?

 I tried it and it worked for me, so I undid my changes in etc/Makefile.

 I thought that the code that adds it was sys/conf/Makefile.kern.inc:

 ....
 # Default DEBUG to -g if kernel debug info is requested by MKDEBUGKERNEL=yes
 .if defined(MKDEBUGKERNEL) && ${MKDEBUGKERNEL} == "yes"
 DEBUG?=-g
 .endif
 ....
 EXTRA_KERNELS+= ${KERNELS:@.KERNEL.@${.KERNEL.}.gdb@}
 ...

 I will start clean and rebuild again to verify.

 Best,

 christo

 --Apple-Mail=_A8BB0961-082F-4D78-93BA-FC5A03B4194D
 Content-Transfer-Encoding: 7bit
 Content-Disposition: attachment;
 	filename=signature.asc
 Content-Type: application/pgp-signature;
 	name=signature.asc
 Content-Description: Message signed with OpenPGP

 -----BEGIN PGP SIGNATURE-----
 Comment: GPGTools - http://gpgtools.org

 iF0EARECAB0WIQS+BJlbqPkO0MDBdsRxESqxbLM7OgUCYcQzVgAKCRBxESqxbLM7
 Osq5AKCvJetKEhj3BILfuUZrVekETPutcQCgzaCOaqfXtfB8fG9dMFb6ohdaNdU=
 =pC2h
 -----END PGP SIGNATURE-----

 --Apple-Mail=_A8BB0961-082F-4D78-93BA-FC5A03B4194D--

From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: toolchain/55557: Releases should include kernel debug symbols
Date: Thu, 27 Oct 2022 09:18:59 +0300

 I believe this bug has been fixed in the releng daily builds since
 January, and therefore will presumably be fixed in NetBSD-10 once
 it is released.

 I'm guessing it was fixed by christos in src/etc/Makefile revisions
 1.456 and 1.457, and/or by kre in revision 1.458, but have not
 bisected to verify.

 I have updated https://wiki.netbsd.org/kernel_debugging_with_qemu/
 to make use of the daily builds now that they are suitable for
 symbolic kernel debugging purposes.

 I will keep the PR open until the fix has made it into an actual
 release.
 -- 
 Andreas Gustafsson, gson@gson.org

Home	PR Database Search