NetBSD Problem Report #34974

From kre@munnari.OZ.AU  Thu Nov  2 12:55:08 2006
Return-Path: <kre@munnari.OZ.AU>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by narn.NetBSD.org (Postfix) with ESMTP id A8ADD63BC83
	for <gnats-bugs@gnats.NetBSD.org>; Thu,  2 Nov 2006 12:55:08 +0000 (UTC)
Message-Id: <200611021020.kA2AKEc5026809@jade.coe.psu.ac.th>
Date: Thu, 2 Nov 2006 17:20:14 +0700 (ICT)
From: kre@munnari.OZ.AU
To: gnats-bugs@NetBSD.org
Subject: pkgsrc sometimes compiles packages it doesn't need to compile
X-Send-Pr-Version: 3.95

>Number:         34974
>Category:       pkg
>Synopsis:       pkgsrc sometimes compiles packages it doesn't need to compile
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    pkg-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Nov 02 13:00:00 +0000 2006
>Closed-Date:    
>Last-Modified:  Wed May 27 15:10:07 +0000 2015
>Originator:     Robert Elz
>Release:        NetBSD 3.99.15 (pkgsrc current today)
>Organization:
	Prince of Songkla University
>Environment:
System: NetBSD jade.coe.psu.ac.th 3.99.15 NetBSD 3.99.15 (GENERIC-1.696-20060125) #8: Wed Jan 25 04:59:39 ICT 2006 kre@jade.coe.psu.ac.th:/usr/obj/current/kernels/JADE_ASUS i386
Architecture: i386
Machine: i386
>Description:
	When using the "bin-install" pkgsrc target, and a reasonably complete
	set of binary packages (so only packages that have been updated should
	ever need to be rebuilt), pkgsrc sometimes recompiles and rebuilds a
	package that does not need to be recompiled 9that is, for which a
	binary package already exists, and which is unchanged).

>How-To-Repeat:
	I have seen this happen many times over the pasy year(s) - ie: this
	is nothing new, I'm just getting around to remorting it as I noticed
	it again today in a reasonably simple case.

	To make this happen, remove (move) binary packages (if any) for
	security/opencdk and security/wireshark - but have an existing
	package for security/gnutls.

	if you don't have the binary packages already, this is a hard state
	to arrive in, as gnutls depends upon opencdk.

	It happened for me today, as opencdk was updated a day or two ago,
	so my old binary package was removed, and wireshard was updates
	today (so its binary package was rmeoved), but gnutls hasn't been
	updated for a month and a half now.

	Then build (using the bin-install target) security/wireshark.

	pkgsrc notices the dependency upon gnutls, and attempts to install the
	bnary package (which is as it should).  The pkg_install fails, as
	gnutls requires opencdk, and there is no binary package available for
	that one (yet).   At this stage pkgsrc falls back to a source build,
	and attempts the source build of gnutls - that again needs opencdk
	as a dependency, but a source build of that succeeds without problems.

	At this point, a binary install of gnutls would work just fine, but
	pkgsrc has already abandoned that attempt (tried it, it failed...)
	So goes ahead and rebuilds gnutls from source, making a new binary
	package (which overwrites the one that was there, which is really not
	a very nice thing to happen - and in some cases could be a real problem)

>Fix:
	I suspect that the only way to fix this involves a rework of the way
	that pkgsrc handles dependencies - instead of doing recursive builds,
	it really needs to be iterative - that is, pkgsrc needs to find what
	dependencies need to exist for the package being built (the one on the
	original user command line), and then for all dependencies of that
	package, hen combine that list into a sensible form (if A requires
	C>=2 and B reqiores C>=3 and we need both A and B, then we need
	C>=3 (the C>=2 just gets absorbed).  Once we have that list, it needs
	to be sorrted into depenndcncy order, after which the packages can be
	built (whether that's a bin-install build, or just install, or
	whaveer other has been requested) in the order that means no
	(further) recursion happens - everything already exists when it is
	needed.  With a scheme like this, the build of wireshark would have
	discovered that opencdk 9as well as gnutls) was needed, and that as
	opencdk is needed by gnutls, opencdk would be built first.   There
	the pkg_install fails, as there is (was) no binary packate, and a
	source build is done.  Then gnutls comes next, now the binary install
	succeeds, as all dependencies are already installed, and n new
	compilation is required.  After that, the compilation of wireshark
	can be successfully accomplished.

	NOTE: the answer here is **NOT** "gnutls really needsd to be
	recompiled anyway so it could get correct dependency info in the
	binary package" - or if that is the answer, then that just exposes
	a similar (but much harder to fix, and much more stupid) but in
	pkgsrc - as if instead of simply building wireshark, I had first
	built opencdk (which I might have, and would have, had I gotten around
	to updating my binary package collection a day earlier), then the
	bvnary-install of the (old) gnutls package would have succeeded (it
	might have issued a strange error-looking type of message about the
	correct vesion of the binary package not being found - but it would
	have successfully installed from the binary package - and would not
	have been rebuilt.   That's inconsistent with the (assumed here) view
	that the binary package needed to be rebuilt.

	Also note tha the 3 packages opencdk, gnutls, and wireshark (and
	whetaver other dependencies they have which aren't relevant here)
	are not the point f this PR - they just happened to form today's
	example, so nothing specific to any of thse packages can possibly be
	relevant to this PR (ie: bumping PKGREVISION of gnutls would be
	a truly stupid response to this PR, whether or not it might happen
	to be called for because of some other reasons).

	pkgsrc probably needs a major reweite, and that should probably be
	done in sh, rather than in make - it uses almost none of the
	true facilities of make, and continuing to use it just because once,
	many years ago, when the project started (as FreeBSD ports) make
	seemed like the appropriate tool, does not cause that to continue
	to be true...   As a set of shell scripts I suspect pkgsrc could be
	many times more flexible, faster, and MUCH simpler to maintain.

>Release-Note:

>Audit-Trail:

State-Changed-From-To: open->feedback
State-Changed-By: bsiegert@NetBSD.org
State-Changed-When: Sun, 17 May 2015 20:16:49 +0000
State-Changed-Why:
Given that no one will reimplement pkgsrc from scratch any time soon,
and that extra builds like this are not such a big issue, are you okay
with closing this?


From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/34974 (pkgsrc sometimes compiles packages it doesn't need to compile)
Date: Mon, 18 May 2015 06:33:16 +0700

 Since no-one had even commented on the PR in the past 8 and a half years,
 I don't see any great harm in closing it I guess ... Personally I'd still
 like to see pkgsrc reimplemented in sh (or whatever - a programming language
 anyway, rather than make, which is not that) but I also don't see it
 happening any time soon (thinking abut doing it is one of my dreams, but
 it never gets out of fantasyland).

 kre

State-Changed-From-To: feedback->open
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Wed, 27 May 2015 15:04:44 +0000
State-Changed-Why:
I would say this is a genuine infrastructure bug and we should fix it, not
wish it away.


From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/34974 (pkgsrc sometimes compiles packages it doesn't need to
 compile)
Date: Wed, 27 May 2015 15:07:56 +0000

 On Sun, May 17, 2015 at 11:40:00PM +0000, Robert Elz wrote:
  >  Since no-one had even commented on the PR in the past 8 and a half years,
  >  I don't see any great harm in closing it I guess ... Personally I'd still
  >  like to see pkgsrc reimplemented in sh (or whatever - a programming language
  >  anyway, rather than make, which is not that) but I also don't see it
  >  happening any time soon (thinking abut doing it is one of my dreams, but
  >  it never gets out of fantasyland).

 Note fwiw that Gentoo's emerge is basically pkgsrc written in Python.

 Of course, Gentoo has other issues, starting with being Linux... when
 I tried it out for a while some years ago I also found it to be
 annoyingly 133t, but maybe the people responsible have grown up since.

 -- 
 David A. Holland
 dholland@netbsd.org

From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: pkg/34974 (pkgsrc sometimes compiles packages it doesn't need to
 compile)
Date: Wed, 27 May 2015 15:08:22 +0000

 On Wed, May 27, 2015 at 03:04:44PM +0000, dholland@NetBSD.org wrote:
  > Synopsis: pkgsrc sometimes compiles packages it doesn't need to compile
  > 
  > State-Changed-From-To: feedback->open
  > State-Changed-By: dholland@NetBSD.org
  > State-Changed-When: Wed, 27 May 2015 15:04:44 +0000
  > State-Changed-Why:
  > I would say this is a genuine infrastructure bug and we should fix it, not
  > wish it away.

 But I don't see any reason fixing it calls for a total rewrite :-)

 -- 
 David A. Holland
 dholland@netbsd.org

>Unformatted:

Home	PR Database Search