NetBSD Problem Report #35012
From kre@munnari.OZ.AU Wed Nov 8 02:25:51 2006
Return-Path: <kre@munnari.OZ.AU>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by narn.NetBSD.org (Postfix) with ESMTP id 4804663B8CA
for <gnats-bugs@gnats.NetBSD.org>; Wed, 8 Nov 2006 02:25:51 +0000 (UTC)
Message-Id: <200611080225.kA82PV46024664@jade.coe.psu.ac.th>
Date: Wed, 8 Nov 2006 09:25:31 +0700 (ICT)
From: kre@munnari.OZ.AU
To: gnats-bugs@NetBSD.org
Subject: pkgsrc binary package dependancy system is broken
X-Send-Pr-Version: 3.95
>Number: 35012
>Category: pkg
>Synopsis: pkgsrc binary package dependancy system is broken
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: joerg
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Wed Nov 08 02:30:00 +0000 2006
>Closed-Date:
>Last-Modified: Tue Jun 28 14:25:00 +0000 2016
>Originator: Robert Elz
>Release: NetBSD 3.99.15 (pkgsrc current as of last few hours)
>Organization:
Prince of Songkla University
>Environment:
System: NetBSD jade.coe.psu.ac.th 3.99.15 NetBSD 3.99.15 (GENERIC-1.696-20060125) #8: Wed Jan 25 04:59:39 ICT 2006 kre@jade.coe.psu.ac.th:/usr/obj/current/kernels/JADE_ASUS i386
Architecture: i386
Machine: i386
>Description:
Binary packages are built with a copy of the source package
dependencies as their dependencies. That's totally broken.
A binary package *must* depend upon at least the version of
another package that was installed at the time it was built, not
the minimum version that might have been installed in order to
build the source package.
To see this, just consider the NetBSD C library (I know this is
not a pkgsrc binary, but it illustrates the problem). The vast
majority of NetBSD programs will compile, link and run, against
any version of the NetBSD C library (that is, the source dependency
is simply that a C library exists). Those programs that need at
least a particular version - eg: maybe they need statvfs or something,
show the same issue, but we don't need to consider those.
That is, for example, /usr/src/bin/cat/* will happily compile
and link (I guess, if I happened to pick a bad example, just
pick another) against any version of the C library.
However, a version that has been linked with the NetBSD 3 version
of the C library isn't really expected to work on NetBSD 1.6, or
NetBSD 2 (it might, sometimes, if it is lucky, but it isn't really
expected). On the other hand, a version linked against the
NetBSD 3 C library is expected to work on a NetBSD 4 system.
pkgsrc dependency system simply ignores this, and claims that
any version will do, when it builds a binary package. That's bogus.
>How-To-Repeat:
The way I (am fairly sure) I saw this just recently was by
installing firefox2, to see what it was like. First, I was most
pleased to see I could install it without having to delete and
upgrade half the universe first, I hadn't really expected that,
even better, that it installed wthout even needing to remove
firefox1 (or its gtk1 version) first. Credit to the firefox
package developers for that.
However, it really didn't work for me at all once installed,
some functionality was there, but core dumps, ... I haven't
bothered to debug this, but I think the problem is that I always
install from binary packages (and did this time). I build
them myself, but I always build the binary package first, and
then install it.
The binary package I built for firefox2 claims that it needs...
jpeg>=6bnb2
libIDL>=0.8.6nb1
gtk2+>=2.8.17nb1
cairo>=1.0.4nb1
png>=1.2.9nb2
Xft2>=2.1.7nb2
And on the system I installed it, what I have installed is ...
jpeg-6bnb3 IJG's jpeg compression utilities
libIDL-0.8.7 CORBA Interface Definition Language parser
gtk2+-2.8.20nb1 GIMP Toolkit v2 - libraries for building X11 user interfaces
cairo-1.2.0nb2 Vector graphics library with cross-device output support
png-1.2.12 Library for manipulating PNG images
Xft2-2.1.7nb2 Library for configuring and customizing font access
which is ever dependency satisfied, nothing needed upgrading
to install firefox2.
However, now let's look at the packages that were installed
when the firefox2 binary package was built - first, this is
it (and its build date)
-rw-r--r-- 1 root wheel 14789895 Nov 1 01:38 firefox2-2.0.tgz
And the dependency binary packages (which are all what were
installed in my pkg_comp arena when firefox2 was compiled)
-rw-r--r-- 1 root wheel 305432 Jul 19 16:39 jpeg-6bnb3.tgz
-rw-r--r-- 1 root wheel 154330 Aug 6 00:15 libIDL-0.8.7.tgz
-rw-r--r-- 1 root wheel 11155929 Oct 15 22:56 gtk2+-2.10.6.tgz
-rw-r--r-- 1 root wheel 529259 Sep 30 19:42 cairo-1.2.4nb3.tgz
-rw-r--r-- 1 root wheel 307696 Jul 19 16:32 png-1.2.12.tgz
-rw-r--r-- 1 root wheel 78732 Sep 18 22:36 Xft2-2.1.7nb2.tgz
Some of those are fine (ones that have not been upgraded in
a while, like jpeg libIDL png and Xft2)
But, firefox2 was compiled against gtk2+ version 2.10.6
and is not attempting to use version 2.8.20 to run
(which pkg_add doesn't object to, as the package only
claims that it needs 2.8.17 or better). Similarly,
cario version 1.2.4 (the nb3 most likely doesn't matter)
was used to build firefix2, but version 1.2.0 is being
used to run it (again, pkg_add has no idea, as all it is
being told is that anything 1.0.4nb1 or later is OK).
I'm not going to go and attempt to figure out why firefox2
was behaving strangely with a setup like this - I wouldn't
expect it to work properly, and I am not surprised in the
least that it doesn't. Nor is the problem with pkg_add,
as fas as it could tell, everytyhing was fine. The problem
is in pkgsrc, with what dependency information it is putting
in the binary packages when it makes them.
Note that this is what *any* user of binary packages will
experience - note that all of the available binary packages are
consistent, if I removed and reinstalled all of my /usr/pkg
(or a substantial subset of it) this problem wouldn't arise.
That is, it isn't a problem of building one binary package
that's out of step with everything else. There's no way that
you can rationally expect anyone to do that however - rmeove and
reinstall everything, every time they add a new binary package.
And *everything* is th eonly choice most people are likely to
have, as there's no current way to tell them which of their
installed packages do need upgrading (I guess they could
upgrade everything that is out of date - for me, that would
have meant upgrading bind9 and bash and ... even though none of
those has anything to do with firefox at all - I *think*).
And even that assumes that there's a rational way to figure out
what packages I have installed are out of date, if I have no
pkgsrc on my system (relying entirely on binary packages) and
the binary packages are fetched over the net, what technique would
someone use to work out which of their installed packages is
out of date (even if this was the right way forward - which it
is not).
>Fix:
When a binary package is built, the dependencies (for at least
libraries, and anything else that influences the compilation
process - anything that provides C .g files, or their equivalents
in other languages) MUST be (at least) the versions that were
installed when the binary package was built - later versions
are generally OK (most library developers attempt to keep backwards
binary compatability - not all of them, but most - even more than
backwards source compatability).
In this case, that would have told me that I needed
gtk2+>=2.10.6
(etc) and then i would have known that I needed to upgrade
gtk2+ before firefox2 could be installed (which most likely
would have meant upgrading a bunch of other stuff as well).
Note: this cannot (should not, or even must not) be fixed with
the dependencies in the source versions of the packages.
If firefox2 says that gtk2+ version 2.8.17nb1 or better is
OK to use to compile against, then that's all that should be
required - that is, if I compiled firefox2 on the system
I installed it on, upgrading nothing, I'd expect it to work.
(If that's not true, then there's a bug in the firefox2
dependency list, but that would, if it happened to be true,
be a comparatively trivial issue - and I have no reason at all
to assume that anything like that is a problem).
>Release-Note:
>Audit-Trail:
From: "Jeremy C. Reed" <reed@reedmedia.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/35012: pkgsrc binary package dependancy system is broken
Date: Tue, 7 Nov 2006 20:58:57 -0600 (CST)
On Wed, 8 Nov 2006, kre@munnari.OZ.AU wrote:
> >Fix:
> When a binary package is built, the dependencies (for at least
> libraries, and anything else that influences the compilation
> process - anything that provides C .g files, or their equivalents
> in other languages) MUST be (at least) the versions that were
> installed when the binary package was built - later versions
> are generally OK (most library developers attempt to keep backwards
> binary compatability - not all of them, but most - even more than
> backwards source compatability).
>
> In this case, that would have told me that I needed
> gtk2+>=2.10.6
> (etc) and then i would have known that I needed to upgrade
> gtk2+ before firefox2 could be installed (which most likely
> would have meant upgrading a bunch of other stuff as well).
>
> Note: this cannot (should not, or even must not) be fixed with
> the dependencies in the source versions of the packages.
> If firefox2 says that gtk2+ version 2.8.17nb1 or better is
> OK to use to compile against, then that's all that should be
> required - that is, if I compiled firefox2 on the system
> I installed it on, upgrading nothing, I'd expect it to work.
> (If that's not true, then there's a bug in the firefox2
> dependency list, but that would, if it happened to be true,
> be a comparatively trivial issue - and I have no reason at all
> to assume that anything like that is a problem).
For a couple years on a couple systems I have used binary only packages
(with no pkgsrc tree and no builds). I very often have had packages built
with newer dependencies on the build system be used on system with older
dependencies.
The BUILDLINK_API_DEPENDS.${foo} and BUILDLINK_ABI_DEPENDS.${foo}
variables usually work good for this. (They used to have different
variable names last year and before).
But, we do have open-ended dependencies that sometimes cause problems.
The packages can also record the specific library so names needed
(REQUIRES in +BUILD_INFO) and what libraries provided (PROVIDES). If we
had a tool to take advantage of that, it would be good.
As for changing the dependencies to be start from the version used to
build against, that sounds like an okay idea.
Some have advocated just always using the exact version.
Either way, that info is recorded also. Here is an example with lyx-qt I
built today:
@blddep aspell-0.60.4nb2
@pkgdep aspell>=0.50.3
So maybe you are suggesting that @blddep should be forced as at least the
minimum.
Also I wonder in your case if there is a problem with the
BUILDLINK_ABI_DEPENDS?
- Jeremy C. Reed
From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/35012: pkgsrc binary package dependancy system is broken
Date: Wed, 08 Nov 2006 10:41:11 +0700
Date: Wed, 8 Nov 2006 03:05:06 +0000 (UTC)
From: "Jeremy C. Reed" <reed@reedmedia.net>
Message-ID: <20061108030506.CF74C63B8CA@narn.NetBSD.org>
| For a couple years on a couple systems I have used binary only packages
| (with no pkgsrc tree and no builds). I very often have had packages built
| with newer dependencies on the build system be used on system with older
| dependencies.
I'm sure I have too, and in some (or perhaps even many) cases it will
work, but it can't be guaranteed (in the case I saw, it is even possible
that firefox2 is just horribly buggy, and this problem isn't what caused
the problems I saw, I just kind of doubt that).
| But, we do have open-ended dependencies that sometimes cause problems.
Yes, I know, but that's a whole different problem (been meaning to
say something about that one too - I think I know how that could be
mostly handled). The whole way dependencies are used is "sub-optimal",
but this one seems to be an out and out bug (in something).
| Some have advocated just always using the exact version.
That's certainly the conservative, and safe, way - but most of the time
unnecessarily rigorous. While people certainly do sometimes make stupid
changes which break backward binary computability, most library developers
have a fair handle on the problem, and work pretty hard to avoid it.
Requiring everything to be rebuilt every time there's any minor change
is too much (sure, occasionally it would save a problem, but that's the
unusual case).
| So maybe you are suggesting that @blddep should be forced as at least the
| minimum.
Yes, for libraries certainly. For applications that are used, perhaps
but usually not. If an application behaves differently, then the code
that uses it is generally going to expect either the new or old,
and random substitutions don't work - if it doesn't behave differently,
then whatever changes have been made to it internally are irrelevant - that's
not true of libraries, or anything else that influences the compilation
of another program (anything that provides .h files or similar).
| Also I wonder in your case if there is a problem with the
| BUILDLINK_ABI_DEPENDS?
I have no idea, I know nothing about buildlink...
kre
Responsible-Changed-From-To: pkg-manager->joerg
Responsible-Changed-By: joerg@netbsd.org
Responsible-Changed-When: Wed, 08 Nov 2006 10:48:22 +0000
Responsible-Changed-Why:
I am working on it.
State-Changed-From-To: open->suspended
State-Changed-By: joerg@netbsd.org
State-Changed-When: Wed, 08 Nov 2006 10:48:22 +0000
State-Changed-Why:
EuroBSDCon presentation will talk about the intended solution and the
plans for implementation, but this is an issue which needs some time
to finish. It won't get forgotten.
State-Changed-From-To: suspended->open
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Thu, 23 Jun 2016 18:34:06 +0000
State-Changed-Why:
Joerg wrote in 2006: "This is an issue which needs some time to
finish. It won't get forgotten."
ahem.
From: Benny Siegert <bsiegert@gmail.com>
To: gnats-bugs@NetBSD.org
Cc: joerg@NetBSD.org,
pkgsrc-bugs@netbsd.org,
gnats-admin@netbsd.org,
dholland@NetBSD.org,
kre@munnari.OZ.AU
Subject: Re: pkg/35012 (pkgsrc binary package dependancy system is broken)
Date: Tue, 28 Jun 2016 16:23:41 +0200
Are those EuroBSDcon slides available online somewhere? :)
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.