NetBSD Problem Report #35012

From kre@munnari.OZ.AU  Wed Nov  8 02:25:51 2006
Return-Path: <kre@munnari.OZ.AU>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by narn.NetBSD.org (Postfix) with ESMTP id 4804663B8CA
	for <gnats-bugs@gnats.NetBSD.org>; Wed,  8 Nov 2006 02:25:51 +0000 (UTC)
Message-Id: <200611080225.kA82PV46024664@jade.coe.psu.ac.th>
Date: Wed, 8 Nov 2006 09:25:31 +0700 (ICT)
From: kre@munnari.OZ.AU
To: gnats-bugs@NetBSD.org
Subject: pkgsrc binary package dependancy system is broken
X-Send-Pr-Version: 3.95

>Number:         35012
>Category:       pkg
>Synopsis:       pkgsrc binary package dependancy system is broken
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    joerg
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Nov 08 02:30:00 +0000 2006
>Closed-Date:    
>Last-Modified:  Tue Jun 28 14:25:00 +0000 2016
>Originator:     Robert Elz
>Release:        NetBSD 3.99.15   (pkgsrc current as of last few hours)
>Organization:
	Prince of Songkla University
>Environment:


System: NetBSD jade.coe.psu.ac.th 3.99.15 NetBSD 3.99.15 (GENERIC-1.696-20060125) #8: Wed Jan 25 04:59:39 ICT 2006 kre@jade.coe.psu.ac.th:/usr/obj/current/kernels/JADE_ASUS i386
Architecture: i386
Machine: i386
>Description:
	Binary packages are built with a copy of the source package
	dependencies as their dependencies.   That's totally broken.

	A binary package *must* depend upon at least the version of
	another package that was installed at the time it was built, not
	the minimum version that might have been installed in order to
	build the source package.

	To see this, just consider the NetBSD C library (I know this is
	not a pkgsrc binary, but it illustrates the problem).   The vast
	majority of NetBSD programs will compile, link and run, against
	any version of the NetBSD C library (that is, the source dependency
	is simply that a C library exists).  Those programs that need at
	least a particular version - eg: maybe they need statvfs or something,
	show the same issue, but we don't need to consider those.

	That is, for example, /usr/src/bin/cat/* will happily compile
	and link (I guess, if I happened to pick a bad example, just
	pick another) against any version of the C library.

	However, a version that has been linked with the NetBSD 3 version
	of the C library isn't really expected to work on NetBSD 1.6, or
	NetBSD 2 (it might, sometimes, if it is lucky, but it isn't really
	expected).   On the other hand, a version linked against the
	NetBSD 3 C library is expected to work on a NetBSD 4 system.

	pkgsrc dependency system simply ignores this, and claims that
	any version will do, when it builds a binary package.   That's bogus.

>How-To-Repeat:
	The way I (am fairly sure) I saw this just recently was by
	installing firefox2, to see what it was like.   First, I was most
	pleased to see I could install it without having to delete  and
	upgrade half the universe first, I hadn't really expected that,
	even better, that it installed wthout even needing to remove
	firefox1 (or its gtk1 version) first.   Credit to the firefox
	package developers for that.

	However, it really didn't work for me at all once installed,
	some functionality was there, but core dumps, ...   I haven't
	bothered to debug this, but I think the problem is that I always
	install from binary packages (and did this time).   I build
	them myself, but I always build the binary package first, and
	then install it.

	The binary package I built for firefox2 claims that it needs...

jpeg>=6bnb2
libIDL>=0.8.6nb1
gtk2+>=2.8.17nb1
cairo>=1.0.4nb1
png>=1.2.9nb2
Xft2>=2.1.7nb2

	And on the system I installed it, what I have installed is ...

jpeg-6bnb3          IJG's jpeg compression utilities
libIDL-0.8.7        CORBA Interface Definition Language parser
gtk2+-2.8.20nb1     GIMP Toolkit v2 - libraries for building X11 user interfaces
cairo-1.2.0nb2      Vector graphics library with cross-device output support
png-1.2.12          Library for manipulating PNG images
Xft2-2.1.7nb2       Library for configuring and customizing font access

	which is ever dependency satisfied, nothing needed upgrading
	to install firefox2.

	However, now let's look at the packages that were installed
	when the firefox2 binary package was built - first, this is
	it (and its build date)

-rw-r--r--  1 root  wheel  14789895 Nov  1 01:38 firefox2-2.0.tgz

	And the dependency binary packages (which are all what were
	installed in my pkg_comp arena when firefox2 was compiled)

-rw-r--r--  1 root  wheel    305432 Jul 19 16:39 jpeg-6bnb3.tgz
-rw-r--r--  1 root  wheel    154330 Aug  6 00:15 libIDL-0.8.7.tgz
-rw-r--r--  1 root  wheel  11155929 Oct 15 22:56 gtk2+-2.10.6.tgz
-rw-r--r--  1 root  wheel    529259 Sep 30 19:42 cairo-1.2.4nb3.tgz
-rw-r--r--  1 root  wheel    307696 Jul 19 16:32 png-1.2.12.tgz
-rw-r--r--  1 root  wheel     78732 Sep 18 22:36 Xft2-2.1.7nb2.tgz

	Some of those are fine (ones that have not been upgraded in
	a while, like jpeg libIDL png and Xft2)

	But, firefox2 was compiled against gtk2+ version 2.10.6
	and is not attempting to use version 2.8.20 to run
	(which pkg_add doesn't object to, as the package only
	claims that it needs 2.8.17 or better).   Similarly,
	cario version 1.2.4 (the nb3 most likely doesn't matter)
	was used to build firefix2, but version 1.2.0 is being
	used to run it (again, pkg_add has no idea, as all it is
	being told is that anything 1.0.4nb1 or later is OK).

	I'm not going to go and attempt to figure out why firefox2
	was behaving strangely with a setup like this - I wouldn't
	expect it to work properly, and I am not surprised in the
	least that it doesn't.   Nor is the problem with pkg_add,
	as fas as it could tell, everytyhing was fine.   The problem
	is in pkgsrc, with what dependency information it is putting
	in the binary packages when it makes them.

	Note that this is what *any* user of binary packages will
	experience - note that all of the available binary packages are
	consistent, if I removed and reinstalled all of my /usr/pkg
	(or a substantial subset of it) this problem wouldn't arise.
	That is, it isn't a problem of building one binary package
	that's out of step with everything else.   There's no way that
	you can rationally expect anyone to do that however - rmeove and
	reinstall everything, every time they add a new binary package.
	And *everything* is th eonly choice most people are likely to
	have, as there's no current way to tell them which of their
	installed packages do need upgrading (I guess they could
	upgrade everything that is out of date - for me, that would
	have meant upgrading bind9 and bash and ... even though none of
	those has anything to do with firefox at all - I *think*).
	And even that assumes that there's a rational way to figure out
	what packages I have installed are out of date, if I have no
	pkgsrc on my system (relying entirely on binary packages) and
	the binary packages are fetched over the net, what technique would
	someone use to work out which of their installed packages is
	out of date (even if this was the right way forward - which it
	is not).

>Fix:
	When a binary package is built, the dependencies (for at least
	libraries, and anything else that influences the compilation
	process - anything that provides C .g files, or their equivalents
	in other languages) MUST be (at least) the versions that were
	installed when the binary package was built - later versions
	are generally OK (most library developers attempt to keep backwards
	binary compatability - not all of them, but most - even more than
	backwards source compatability).

	In this case, that would have told me that I needed
		gtk2+>=2.10.6
	(etc) and then i would have known that I needed to upgrade
	gtk2+ before firefox2 could be installed (which most likely
	would have meant upgrading a bunch of other stuff as well).

	Note: this cannot (should not, or even must not) be fixed with
	the dependencies in the source versions of the packages.
	If firefox2 says that gtk2+ version 2.8.17nb1 or better is
	OK to use to compile against, then that's all that should be
	required - that is, if I compiled firefox2 on the system
	I installed it on, upgrading nothing, I'd expect it to work.
	(If that's not true, then there's a bug in the firefox2
	dependency list, but that would, if it happened to be true,
	be a comparatively trivial issue - and I have no reason at all
	to assume that anything like that is a problem).

>Release-Note:

>Audit-Trail:
From: "Jeremy C. Reed" <reed@reedmedia.net>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/35012: pkgsrc binary package dependancy system is broken
Date: Tue, 7 Nov 2006 20:58:57 -0600 (CST)

 On Wed, 8 Nov 2006, kre@munnari.OZ.AU wrote:

 > >Fix:
 > 	When a binary package is built, the dependencies (for at least
 > 	libraries, and anything else that influences the compilation
 > 	process - anything that provides C .g files, or their equivalents
 > 	in other languages) MUST be (at least) the versions that were
 > 	installed when the binary package was built - later versions
 > 	are generally OK (most library developers attempt to keep backwards
 > 	binary compatability - not all of them, but most - even more than
 > 	backwards source compatability).
 >
 > 	In this case, that would have told me that I needed
 > 		gtk2+>=2.10.6
 > 	(etc) and then i would have known that I needed to upgrade
 > 	gtk2+ before firefox2 could be installed (which most likely
 > 	would have meant upgrading a bunch of other stuff as well).
 > 
 > 	Note: this cannot (should not, or even must not) be fixed with
 > 	the dependencies in the source versions of the packages.
 > 	If firefox2 says that gtk2+ version 2.8.17nb1 or better is
 > 	OK to use to compile against, then that's all that should be
 > 	required - that is, if I compiled firefox2 on the system
 > 	I installed it on, upgrading nothing, I'd expect it to work.
 > 	(If that's not true, then there's a bug in the firefox2
 > 	dependency list, but that would, if it happened to be true,
 > 	be a comparatively trivial issue - and I have no reason at all
 > 	to assume that anything like that is a problem).

 For a couple years on a couple systems I have used binary only packages 
 (with no pkgsrc tree and no builds). I very often have had packages built 
 with newer dependencies on the build system be used on system with older 
 dependencies.

 The BUILDLINK_API_DEPENDS.${foo} and BUILDLINK_ABI_DEPENDS.${foo} 
 variables usually work good for this. (They used to have different 
 variable names last year and before).

 But, we do have open-ended dependencies that sometimes cause problems.

 The packages can also record the specific library so names needed 
 (REQUIRES in +BUILD_INFO) and what libraries provided (PROVIDES). If we 
 had a tool to take advantage of that, it would be good.

 As for changing the dependencies to be start from the version used to 
 build against, that sounds like an okay idea.

 Some have advocated just always using the exact version.

 Either way, that info is recorded also. Here is an example with lyx-qt I 
 built today:
 @blddep aspell-0.60.4nb2
 @pkgdep aspell>=0.50.3

 So maybe you are suggesting that @blddep should be forced as at least the 
 minimum.

 Also I wonder in your case if there is a problem with the 
 BUILDLINK_ABI_DEPENDS?

 - Jeremy C. Reed

From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/35012: pkgsrc binary package dependancy system is broken 
Date: Wed, 08 Nov 2006 10:41:11 +0700

     Date:        Wed,  8 Nov 2006 03:05:06 +0000 (UTC)
     From:        "Jeremy C. Reed" <reed@reedmedia.net>
     Message-ID:  <20061108030506.CF74C63B8CA@narn.NetBSD.org>

   |  For a couple years on a couple systems I have used binary only packages 
   |  (with no pkgsrc tree and no builds). I very often have had packages built 
   |  with newer dependencies on the build system be used on system with older 
   |  dependencies.

 I'm sure I have too, and in some (or perhaps even many) cases it will
 work, but it can't be guaranteed (in the case I saw, it is even possible
 that firefox2 is just horribly buggy, and this problem isn't what caused
 the problems I saw, I just kind of doubt that).

   |  But, we do have open-ended dependencies that sometimes cause problems.

 Yes, I know, but that's a whole different problem (been meaning to
 say something about that one too - I think I know how that could be
 mostly handled).   The whole way dependencies are used is "sub-optimal",
 but this one seems to be an out and out bug (in something).

   |  Some have advocated just always using the exact version.

 That's certainly the conservative, and safe, way - but most of the time
 unnecessarily rigorous.   While people certainly do sometimes make stupid
 changes which break backward binary computability, most library developers
 have a fair handle on the problem, and work pretty hard to avoid it.
 Requiring everything to be rebuilt every time there's any minor change
 is too much (sure, occasionally it would save a problem, but that's the
 unusual case).

   |  So maybe you are suggesting that @blddep should be forced as at least the 
   |  minimum.

 Yes, for libraries certainly.   For applications that are used, perhaps
 but usually not.  If an application behaves differently, then the code
 that uses it is generally going to expect either the new or old,
 and random substitutions don't work - if it doesn't behave differently,
 then whatever changes have been made to it internally are irrelevant - that's
 not true of libraries, or anything else that influences the compilation
 of another program (anything that provides .h files or similar).

   |  Also I wonder in your case if there is a problem with the 
   |  BUILDLINK_ABI_DEPENDS?

 I have no idea, I know nothing about buildlink...

 kre


Responsible-Changed-From-To: pkg-manager->joerg
Responsible-Changed-By: joerg@netbsd.org
Responsible-Changed-When: Wed, 08 Nov 2006 10:48:22 +0000
Responsible-Changed-Why:
I am working on it.


State-Changed-From-To: open->suspended
State-Changed-By: joerg@netbsd.org
State-Changed-When: Wed, 08 Nov 2006 10:48:22 +0000
State-Changed-Why:
EuroBSDCon presentation will talk about the intended solution and the
plans for implementation, but this is an issue which needs some time
to finish. It won't get forgotten.


State-Changed-From-To: suspended->open
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Thu, 23 Jun 2016 18:34:06 +0000
State-Changed-Why:
Joerg wrote in 2006: "This is an issue which needs some time to
finish. It won't get forgotten."

ahem.


From: Benny Siegert <bsiegert@gmail.com>
To: gnats-bugs@NetBSD.org
Cc: joerg@NetBSD.org,
 pkgsrc-bugs@netbsd.org,
 gnats-admin@netbsd.org,
 dholland@NetBSD.org,
 kre@munnari.OZ.AU
Subject: Re: pkg/35012 (pkgsrc binary package dependancy system is broken)
Date: Tue, 28 Jun 2016 16:23:41 +0200

 Are those EuroBSDcon slides available online somewhere? :)

>Unformatted:

Home	PR Database Search