NetBSD Problem Report #54449
From gson@gson.org Fri Aug 9 09:22:45 2019
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 669917A16D
for <gnats-bugs@gnats.NetBSD.org>; Fri, 9 Aug 2019 09:22:45 +0000 (UTC)
Message-Id: <20190809092241.3D47598975B@guava.gson.org>
Date: Fri, 9 Aug 2019 12:22:41 +0300 (EEST)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@NetBSD.org
Subject: _pic.a libraries contain unwanted debug information
X-Send-Pr-Version: 3.95
>Number: 54449
>Category: lib
>Synopsis: _pic.a libraries contain unwanted debug information
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: lib-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Fri Aug 09 09:25:00 +0000 2019
>Last-Modified: Sun Sep 01 10:45:01 +0000 2019
>Originator: Andreas Gustafsson
>Release: NetBSD 8.1, also -current
>Organization:
>Environment:
System: NetBSD
Architecture: x86_64
Machine: amd64
>Description:
As recently discussed on source-changes-d, the _pic.a versions
of libraries in NetBSD releases are much larger than the .a, _p.a,
or .so, using a significant fraction of the disk space taken by
a full NetBSD installation:
http://mail-index.netbsd.org/source-changes-d/2019/08/08/msg011535.html
Although that discussion was in the context of live images, the
problem is not specific to those. For example, looking at libcurses
on an ordinary 8.1 amd64 system installed using sysinst:
guido /usr/lib $ ls -al libcur*
-r--r--r-- 1 root wheel 381166 May 31 11:43 libcurses.a
lrwxr-xr-x 1 root wheel 16 May 31 11:43 libcurses.so -> libcurses.so.7.1
lrwxr-xr-x 1 root wheel 16 May 31 11:43 libcurses.so.7 -> libcurses.so.7.1
-r--r--r-- 1 root wheel 200784 May 31 11:43 libcurses.so.7.1
-r--r--r-- 1 root wheel 406966 May 31 11:43 libcurses_p.a
-r--r--r-- 1 root wheel 2081382 May 31 11:43 libcurses_pic.a
Apparently the _pic.a library contains full debug information, whereas
the others have had it stripped or replaced by CTF. Presumably the same
ought to be done to the _pic.a, too.
>How-To-Repeat:
ls -l /usr/lib
>Fix:
>Audit-Trail:
From: Christos Zoulas <christos@zoulas.com>
To: gnats-bugs@netbsd.org
Cc: lib-bug-people@netbsd.org,
gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org
Subject: Re: lib/54449: _pic.a libraries contain unwanted debug information
Date: Fri, 9 Aug 2019 12:44:38 +0300
--Apple-Mail=_3A58DF82-BFB2-428D-97F3-66428BDCD27A
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii
Try this: https://www.netbsd.org/~christos/strip-piclib.diff =
<https://www.netbsd.org/~christos/strip-piclib.diff>
christos=
--Apple-Mail=_3A58DF82-BFB2-428D-97F3-66428BDCD27A
Content-Transfer-Encoding: 7bit
Content-Type: text/html;
charset=us-ascii
<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Try this: <a href="https://www.netbsd.org/~christos/strip-piclib.diff" class="">https://www.netbsd.org/~christos/strip-piclib.diff</a><div class=""><br class=""></div><div class="">christos</div></body></html>
--Apple-Mail=_3A58DF82-BFB2-428D-97F3-66428BDCD27A--
From: matthew green <mrg@eterna.com.au>
To: gnats-bugs@netbsd.org
Cc: lib-bug-people@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, gson@gson.org (Andreas Gustafsson)
Subject: re: lib/54449: _pic.a libraries contain unwanted debug information
Date: Sat, 10 Aug 2019 07:38:44 +1000
> Try this: https://www.netbsd.org/~christos/strip-piclib.diff
why this? if you're going to do something like this then actually
save the data into the debug sets. anyone who is using these is
likely happy they have debug info, so removing it is likely to be
sad-making.
but i really think the saner move is to leave this as-is and
change the default for MKPICINSTALL to "no". the vast majority
of people are never going to use them so it's wasteful to install,
debug info or not.
.mrg.
From: Andreas Gustafsson <gson@gson.org>
To: Christos Zoulas <christos@zoulas.com>
Cc: gnats-bugs@netbsd.org
Subject: Re: lib/54449: _pic.a libraries contain unwanted debug information
Date: Sat, 10 Aug 2019 11:16:14 +0300
Christos Zoulas wrote:
> Try this: https://www.netbsd.org/~christos/strip-piclib.diff
Works for me. I did a successful release build with this, and
the _pic.a libraries are a reasonable size, for example:
# cd /usr/lib
# ls -al libcurses*
-r--r--r-- 1 root wheel 372206 Aug 8 14:00 libcurses.a
lrwxr-xr-x 1 root wheel 16 Aug 8 14:00 libcurses.so -> libcurses.so.8.2
lrwxr-xr-x 1 root wheel 16 Aug 8 14:00 libcurses.so.8 -> libcurses.so.8.2
-r--r--r-- 1 root wheel 196560 Aug 8 14:00 libcurses.so.8.2
-r--r--r-- 1 root wheel 400126 Aug 8 14:00 libcurses_p.a
-r--r--r-- 1 root wheel 356246 Aug 8 14:00 libcurses_pic.a
I also ran ran the ATF tests, and there were no more failures than usual.
--
Andreas Gustafsson, gson@gson.org
From: Andreas Gustafsson <gson@gson.org>
To: matthew green <mrg@eterna.com.au>
Cc: gnats-bugs@netbsd.org, christos@NetBSD.org
Subject: re: lib/54449: _pic.a libraries contain unwanted debug information
Date: Sat, 10 Aug 2019 11:42:33 +0300
matthew green wrote:
> > Try this: https://www.netbsd.org/~christos/strip-piclib.diff
>
> why this? if you're going to do something like this then actually
> save the data into the debug sets. anyone who is using these is
> likely happy they have debug info, so removing it is likely to be
> sad-making.
When I look in /usr/libdata/debug/usr/lib on my 8.1 system, I only
see .debug files for the .so libraries, so it looks like we are not
shipping debug information for .a nor _p.a. I think we should be
consistent and either ship debug information for all static libraries,
or for none of them. Shipping it for .so and _pic.a but not .a nor
_p.a would just be weird.
> but i really think the saner move is to leave this as-is and
> change the default for MKPICINSTALL to "no". the vast majority
> of people are never going to use them so it's wasteful to install,
> debug info or not.
I don't have an opinion on whether or not that should be done, but it
does not change the need for Christos' patch - even if we set the
default of MKPICINSTALL to "no", the present bug will still affect
those who override the default to "yes", so it still needs to be fixed.
So, to keep this PR focused on the specific issue in its subject, if
you want those other changes, please open your own PRs for "static
libraries should have debug info in /usr/libdata/debug" and/or
"MKPICINSTALL should default to no".
--
Andreas Gustafsson, gson@gson.org
From: matthew green <mrg@eterna.com.au>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@netbsd.org, christos@NetBSD.org
Subject: re: lib/54449: _pic.a libraries contain unwanted debug information
Date: Sat, 10 Aug 2019 21:15:40 +1000
i'm now reminded of another problem. now, my biggest
problem with the patch is that it's forced upon anyone
who sets DBG=-g, AFAICT.
that .a files are missing debuginfo is a problem i've had
a couple of times since switching from pure -g to MKDEBUG
builds, so, i don't really find that argument compelling.
ie, now i can't debug static programs.
if it wasn't for a different benefit i'd go back to the
pure -g solution, but there's an issue where files with
exec mappings end up having gigabytes of unused debug
info in actual memory...
.mrg.
From: matthew green <mrg@eterna.com.au>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@netbsd.org, christos@NetBSD.org
Subject: re: lib/54449: _pic.a libraries contain unwanted debug information
Date: Sat, 10 Aug 2019 21:17:19 +1000
> I don't have an opinion on whether or not that should be done, but it
> does not change the need for Christos' patch - even if we set the
> default of MKPICINSTALL to "no", the present bug will still affect
> those who override the default to "yes", so it still needs to be fixed.
also - if someone wants to use the _pic.a library they're probably
doing something Special, and not having debug info around is not
so fun. i don't think we should change what happens here at all.
we should just turn them off..
.mrg.
From: Christos Zoulas <christos@zoulas.com>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: lib/54449: _pic.a libraries contain unwanted debug information
Date: Sat, 10 Aug 2019 14:19:04 +0300
There has been some discussion to not delete the debugging symbols from =
the libraries,
but instead to not install them by default.
christos
> On Aug 10, 2019, at 11:16 AM, Andreas Gustafsson <gson@gson.org> =
wrote:
>=20
> Christos Zoulas wrote:
>> Try this: https://www.netbsd.org/~christos/strip-piclib.diff
>=20
> Works for me. I did a successful release build with this, and
> the _pic.a libraries are a reasonable size, for example:
>=20
> # cd /usr/lib
> # ls -al libcurses*
> -r--r--r-- 1 root wheel 372206 Aug 8 14:00 libcurses.a
> lrwxr-xr-x 1 root wheel 16 Aug 8 14:00 libcurses.so -> =
libcurses.so.8.2
> lrwxr-xr-x 1 root wheel 16 Aug 8 14:00 libcurses.so.8 -> =
libcurses.so.8.2
> -r--r--r-- 1 root wheel 196560 Aug 8 14:00 libcurses.so.8.2
> -r--r--r-- 1 root wheel 400126 Aug 8 14:00 libcurses_p.a
> -r--r--r-- 1 root wheel 356246 Aug 8 14:00 libcurses_pic.a
>=20
> I also ran ran the ATF tests, and there were no more failures than =
usual.
> --=20
> Andreas Gustafsson, gson@gson.org
From: "Christos Zoulas" <christos@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/54449 CVS commit: src/share/mk
Date: Sat, 10 Aug 2019 08:20:17 -0400
Module Name: src
Committed By: christos
Date: Sat Aug 10 12:20:17 UTC 2019
Modified Files:
src/share/mk: bsd.own.mk
Log Message:
Don't install PIC libraries by default because they are too big since they
contain debug symbols. I supplied a patch in PR/54449 to remove the debugging
symbols but folks preferred to not install them at all.
To generate a diff of this commit:
cvs rdiff -u -r1.1149 -r1.1150 src/share/mk/bsd.own.mk
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Christos Zoulas <christos@zoulas.com>
To: gnats-bugs@netbsd.org
Cc: lib-bug-people@netbsd.org,
gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org,
Andreas Gustafsson <gson@gson.org>
Subject: Re: lib/54449: _pic.a libraries contain unwanted debug information
Date: Sat, 10 Aug 2019 15:56:55 +0300
Historically the libfoo.a libraries never had debugging symbols, this is =
why we have the libfoo_g.a variant which is controlled by ${MKDEBUGLIB}.
Building a system with DBG=3D-g to get the same effect (putting =
debugging info in libfoo.a) was never guaranteed to work.
The fact that libfoo_pic.a ended up with debugging symbols looks like an =
accident to me.
In any case, I am not dogmatic about it and let's decide how we want it =
to work and fix it.=
From: matthew green <mrg@eterna.com.au>
To: Christos Zoulas <christos@zoulas.com>
Cc: lib-bug-people@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, Andreas Gustafsson <gson@gson.org>,
gnats-bugs@netbsd.org
Subject: re: lib/54449: _pic.a libraries contain unwanted debug information
Date: Sun, 11 Aug 2019 06:31:41 +1000
Christos Zoulas writes:
> Historically the libfoo.a libraries never had debugging symbols, this is
> why we have the libfoo_g.a variant which is controlled by ${MKDEBUGLIB}.
> Building a system with DBG=-g to get the same effect (putting
> debugging info in libfoo.a) was never guaranteed to work.
not sure what planet you are talking about, but DBG=-g has been
the right way to me for a long time, though slightly less than
i recalled (more a 90s thing :-)
revision 1.57
date: 2000-05-01 20:44:36 -0700; author: sjg; state: Exp; lines: +4 -3;
Allow optimizer/debug flags to be changed on command line without
breaking everything. Eg. use DBG="-O0 -g" when debugging something.
> The fact that libfoo_pic.a ended up with debugging symbols looks like an
> accident to me.
> In any case, I am not dogmatic about it and let's decide how we want it
> to work and fix it.
nothing is an accident when "DBG=-g" builds have debug info.
i used to build my whole systems this way without using MKDEBUG
(since that's much much newer), initially because i had some
lossage using separate debug info files, though as i mentioned
before, i'm considering switching back to get debug symbols back
in to my static libraries as well. i may have to set execmax
to some very low value to avoid the other bug..
i claim that their lack of debug info is a bug in MKDEBUG.
.mrg.
From: Christos Zoulas <christos@zoulas.com>
To: gnats-bugs@netbsd.org
Cc: lib-bug-people@netbsd.org,
gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org,
Andreas Gustafsson <gson@gson.org>
Subject: Re: lib/54449: _pic.a libraries contain unwanted debug information
Date: Sun, 11 Aug 2019 10:13:02 +0300
> not sure what planet you are talking about, but DBG=-g has been
> the right way to me for a long time, though slightly less than
> i recalled (more a 90s thing :-)
>
> revision 1.57
> date: 2000-05-01 20:44:36 -0700; author: sjg; state: Exp; lines: +4 -3;
> Allow optimizer/debug flags to be changed on command line without
> breaking everything. Eg. use DBG="-O0 -g" when debugging something.
Yes, you might have been using DBG="-O0 -g" for almost 20 years, but this
variable is not documented anywhere outside the .mk files, and most people
(I am guessing) don't know about it.
> nothing is an accident when "DBG=-g" builds have debug info.
But here we are not building with DBG=-g, but with with MKDEBUG=yes
and libfoo_pic.a ends up with debugging info. Is that on purpose or an
accident?
> i used to build my whole systems this way without using MKDEBUG
> (since that's much much newer), initially because i had some
> lossage using separate debug info files, though as i mentioned
> before, i'm considering switching back to get debug symbols back
> in to my static libraries as well. i may have to set execmax
> to some very low value to avoid the other bug..
>
> i claim that their lack of debug info is a bug in MKDEBUG.
Why would we have both libfoo_g.a and libfoo.a with debugging
symbols? What's the difference between them? As for the other
bugs, are they still there?
christos
From: matthew green <mrg@eterna.com.au>
To: Christos Zoulas <christos@zoulas.com>
Cc: lib-bug-people@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, Andreas Gustafsson <gson@gson.org>,
gnats-bugs@netbsd.org
Subject: re: lib/54449: _pic.a libraries contain unwanted debug information
Date: Sun, 11 Aug 2019 20:12:20 +1000
> Yes, you might have been using DBG="-O0 -g" for almost 20 years, but this
> variable is not documented anywhere outside the .mk files, and most people
> (I am guessing) don't know about it.
except in the original commit that created it, yes, before
that one had to have some CFLAGS setting in mk.conf.
i've heard of many folks using DBG here, particularly since
the commit that calls out that usage.
true, not explicitly documented, but often used,
particularly before MKDEBUG was added, and i claim
defacto-supported.
> > nothing is an accident when "DBG=-g" builds have debug info.
>
> But here we are not building with DBG=-g, but with with MKDEBUG=yes
> and libfoo_pic.a ends up with debugging info. Is that on purpose or an
> accident?
upon consideration, i'm happy to be less strict about
MKDEBUG=yes builds, though i was reminded about my failed
static debugging session a few months ago. does external
debug info even work for .a?
i think that means i'm ok with your other patch.
> > i used to build my whole systems this way without using MKDEBUG
> > (since that's much much newer), initially because i had some
> > lossage using separate debug info files, though as i mentioned
> > before, i'm considering switching back to get debug symbols back
> > in to my static libraries as well. i may have to set execmax
> > to some very low value to avoid the other bug..
> >
> > i claim that their lack of debug info is a bug in MKDEBUG.
>
> Why would we have both libfoo_g.a and libfoo.a with debugging
> symbols? What's the difference between them? As for the other
> bugs, are they still there?
i've never used the _g.a libraries. they alway seemed far
annoying to build and use.
i've simply had *all* my install have debug info. i only
really switched to MKDEBUG when i noticed that huge-from-
debug-info shlibs have a bad interaction with exec pages,
and having it separate works around. (i'm now wishing, but
hating myself for it, a build that gives me fat .a files
with debug info, but skinny .so files with links to the
debug info files. :)
.mrg.
From: Christos Zoulas <christos@zoulas.com>
To: gnats-bugs@netbsd.org
Cc: lib-bug-people@netbsd.org,
gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org,
Andreas Gustafsson <gson@gson.org>
Subject: Re: lib/54449: _pic.a libraries contain unwanted debug information
Date: Mon, 12 Aug 2019 07:03:10 +0300
> upon consideration, i'm happy to be less strict about
> MKDEBUG=3Dyes builds, though i was reminded about my failed
> static debugging session a few months ago. does external
> debug info even work for .a?
Nope, I don't think it does (unless we change binutils).
> i think that means i'm ok with your other patch.
But I am not, because this means that DBG=3D-g will also not put symbols
in libfoo_pic.a (my patch will strip it). So I am inclined to leave it =
as it is
for now.
> i've never used the _g.a libraries. they alway seemed far
> annoying to build and use.
>=20
> i've simply had *all* my install have debug info. i only
> really switched to MKDEBUG when i noticed that huge-from-
> debug-info shlibs have a bad interaction with exec pages,
> and having it separate works around. (i'm now wishing, but
> hating myself for it, a build that gives me fat .a files
> with debug info, but skinny .so files with links to the
> debug info files. :)
Well, we should make MKDEBUG work properly. I don't
think we should be documenting DBG=3D (but also not breaking it).
christos=
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/54449 CVS commit: [netbsd-9] src/share/mk
Date: Sun, 1 Sep 2019 10:40:00 +0000
Module Name: src
Committed By: martin
Date: Sun Sep 1 10:40:00 UTC 2019
Modified Files:
src/share/mk [netbsd-9]: bsd.own.mk
Log Message:
Pull up following revision(s) (requested by gson in ticket #124):
share/mk/bsd.own.mk: revision 1.1150
Don't install PIC libraries by default because they are too big since they
contain debug symbols. I supplied a patch in PR/54449 to remove the debugging
symbols but folks preferred to not install them at all.
To generate a diff of this commit:
cvs rdiff -u -r1.1149 -r1.1149.2.1 src/share/mk/bsd.own.mk
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
(Contact us)
$NetBSD: query-full-pr,v 1.43 2018/01/16 07:36:43 maya Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2017
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home