NetBSD Problem Report #55952

From john@ziaspace.com  Sat Jan 23 23:26:05 2021
Return-Path: <john@ziaspace.com>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 078741A923B
	for <gnats-bugs@gnats.NetBSD.org>; Sat, 23 Jan 2021 23:26:04 +0000 (UTC)
Message-Id: <202101232326.10NNQ0K9016959@athena.zia.io>
Date: Sat, 23 Jan 2021 23:26:00 GMT
From: john@ziaspace.com
Reply-To: john@ziaspace.com
To: gnats-bugs@NetBSD.org
Subject: Problems with /usr/pkg hardcoding in pkgsrc
X-Send-Pr-Version: 3.95

>Number:         55952
>Category:       pkg
>Synopsis:       Bootstrapping on Darwin with different prefix is broken
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    cjep
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Jan 23 23:30:00 +0000 2021
>Closed-Date:    Mon Feb 15 12:57:49 +0000 2021
>Last-Modified:  Mon Feb 15 13:00:02 +0000 2021
>Originator:     John Klos
>Release:        Darwin 15.6.0
>Organization:

>Environment:


System: Darwin Mac-Pro.local 15.6.0 Darwin Kernel Version 15.6.0: Thu Jun 23 18:25:34 PDT 2016; root:xnu-3248.60.10~1/RELEASE_X86_64 x86_64
Architecture: x86_64
Machine: x86_64
>Description:

Download latest pkgsrc.tar.gz. Run bootstrap like so:

./bootstrap --abi 64 --full --make-jobs 8 --prefer-pkgsrc --prefix /usr/local

Fails with:

=> Creating binary package /Users/john/pkgsrc/bootstrap/work/wrk/pkgtools/cwrappers/work/.packages/cwrappers-20180325.tgz
===> Installing binary package of cwrappers-20180325
pkg_add: Can't create pkgdb entry: /usr/pkg/pkgdb/cwrappers-20180325: Operation not permitted
pkg_add: no files matching ``/usr/pkg/pkgdb/cwrappers-20180325/+*'' found: No such file or directory
pkg_add: 1 package addition failed
*** Error code 1

>How-To-Repeat:

>Fix:


>Release-Note:

>Audit-Trail:
From: Jonathan Perkin <jperkin@joyent.com>
To: gnats-bugs@netbsd.org
Cc: pkg-manager@netbsd.org, gnats-admin@netbsd.org, pkgsrc-bugs@netbsd.org
Subject: Re: pkg/55952: Problems with /usr/pkg hardcoding in pkgsrc
Date: Sun, 24 Jan 2021 08:58:45 +0000

 * On 2021-01-23 at 23:30 GMT, john@ziaspace.com wrote:

 > => Creating binary package /Users/john/pkgsrc/bootstrap/work/wrk/pkgtools/cwrappers/work/.packages/cwrappers-20180325.tgz
 > ===> Installing binary package of cwrappers-20180325
 > pkg_add: Can't create pkgdb entry: /usr/pkg/pkgdb/cwrappers-20180325: Operation not permitted
 > pkg_add: no files matching ``/usr/pkg/pkgdb/cwrappers-20180325/+*'' found: No such file or directory
 > pkg_add: 1 package addition failed
 > *** Error code 1

 Are you sure this system doesn't have any files or environment
 variables lying around that might affect this?

 None of my macOS/Linux/SmartOS systems use /usr/pkg and they run
 bootstrap for every daily build without issues.

 -- 
 Jonathan Perkin  -  Joyent, Inc.  -  www.joyent.com

From: John Klos <john@ziaspace.com>
To: gnats-bugs@netbsd.org
Cc: pkg-manager@netbsd.org, gnats-admin@netbsd.org, pkgsrc-bugs@netbsd.org
Subject: Re: pkg/55952: Problems with /usr/pkg hardcoding in pkgsrc
Date: Sun, 24 Jan 2021 19:05:42 +0000 (UTC)

 > Are you sure this system doesn't have any files or environment
 > variables lying around that might affect this?
 >
 > None of my macOS/Linux/SmartOS systems use /usr/pkg and they run
 > bootstrap for every daily build without issues.

 This is a clean system, but I'll run it again. It's an El Capitan system 
 (original Mac Pro). Because of SIP, /usr/pkg cannot be used, of course.

 rm -Rf pkgsrc ; tar xzf pkgsrc.tar.gz       (from 22-Jan-2021)
 cd pkgsrc/bootstrap
 ./bootstrap --abi 64 --full --make-jobs 8 --prefer-pkgsrc --prefix /usr/local

 I see things like this:

 ===> Bootstrapping mk-files
 ===> running: (cd /Users/john/pkgsrc/pkgtools/bootstrap-mk-files/files && 
 env CP=/bin/cp  OPSYS=Darwin 
 MK_DST=/Users/john/pkgsrc/bootstrap/work/share/mk ROOT_GROUP=wheel 
 ROOT_USER=root SED=/usr/bin/sed SYSCONFDIR=/usr/pkg/etc /bin/sh 
 ./bootstrap.sh)

 where SYSCONFDIR should be derived from prefix, and like this:

 ===> Building libnbcompat
 ===> running: /bin/sh /Users/john/pkgsrc/bootstrap/work/bin/install-sh -d 
 -o root -g wheel /Users/john/pkgsrc/bootstrap/work/libnbcompat
 ===> running: (cd /Users/john/pkgsrc/bootstrap/work/libnbcompat; /bin/sh 
 ./configure  -C --prefix=/usr/pkg --infodir=/usr/pkg/info 
 --mandir=/usr/pkg/man --sysconfdir=/usr/pkg/etc --enable-bsd-getopt 
 --enable-db && /Users/john/pkgsrc/bootstrap/work/bin/bmake  -j8)

 where prefix should be derived from prefix!


 Also, we have:

 --- automatic.o ---
 cc -isystem /usr/include -DHAVE_CONFIG_H -DDEF_LOG_DIR=\"/usr/pkg/pkgdb\" 
 -I/Users/john/pkgsrc/bootstrap/work/libnbcompat 
 -I/Users/john/pkgsrc/bootstrap/work/libarchive/libarchive -I. -I. 
 -DSYSCONFDIR=\"/usr/pkg/etc\" -DPREFIX=\"\" -DBOOTSTRAP -g -O2 -c 
 automatic.c


 While I think (thought) that pkgdbdir should follow prefix, I didn't 
 check, and I see it isn't. That should be fixed. If pkgdbdir is meant to 
 exist outside of prefix, then that's when it should need to be set. 
 But it still fails with --pkgdbdir /usr/local/pkgdb.

 Then I realized that the tools that are failing are everything built based 
 on --prefer-pkgsrc.

 ./cleanup
 ./bootstrap --abi 64 --full --make-jobs 8 --prefix /usr/local

 This works fine, and nothing mentions /usr/pkg.

 Thanks,
 John

From: Jonathan Perkin <jperkin@joyent.com>
To: John Klos <john@ziaspace.com>
Cc: gnats-bugs@netbsd.org, pkg-manager@netbsd.org, gnats-admin@netbsd.org,
	pkgsrc-bugs@netbsd.org
Subject: Re: pkg/55952: Problems with /usr/pkg hardcoding in pkgsrc
Date: Sun, 24 Jan 2021 20:20:09 +0000

 * On 2021-01-24 at 19:05 GMT, John Klos wrote:

 > > Are you sure this system doesn't have any files or environment
 > > variables lying around that might affect this?
 > > 
 > > None of my macOS/Linux/SmartOS systems use /usr/pkg and they run
 > > bootstrap for every daily build without issues.
 > 
 > This is a clean system, but I'll run it again. It's an El Capitan system
 > (original Mac Pro). Because of SIP, /usr/pkg cannot be used, of course.
 > 
 > rm -Rf pkgsrc ; tar xzf pkgsrc.tar.gz       (from 22-Jan-2021)
 > cd pkgsrc/bootstrap
 > ./bootstrap --abi 64 --full --make-jobs 8 --prefer-pkgsrc --prefix /usr/local

 I think the problem here is that "--prefer-pkgsrc" is not a valid
 argument by itself, it should be "--prefer-pkgsrc yes" or similar.

 The getopt parsing is a little fragile and is simply shifting over
 your "--prefix" argument, and then ignoring an unknown "/usr/local"
 argument, thus you're getting the default, and no helpful warning.

 We should definitely make it more robust.

 -- 
 Jonathan Perkin  -  Joyent, Inc.  -  www.joyent.com

From: John Klos <john@ziaspace.com>
To: Jonathan Perkin <jperkin@joyent.com>
Cc: gnats-bugs@netbsd.org, pkg-manager@netbsd.org, gnats-admin@netbsd.org,
        pkgsrc-bugs@netbsd.org
Subject: Re: pkg/55952: Problems with /usr/pkg hardcoding in pkgsrc
Date: Mon, 25 Jan 2021 20:12:20 +0000 (UTC)

 >> ./bootstrap --abi 64 --full --make-jobs 8 --prefer-pkgsrc --prefix /usr/local
 >
 > I think the problem here is that "--prefer-pkgsrc" is not a valid
 > argument by itself, it should be "--prefer-pkgsrc yes" or similar.
 >
 > The getopt parsing is a little fragile and is simply shifting over
 > your "--prefix" argument, and then ignoring an unknown "/usr/local"
 > argument, thus you're getting the default, and no helpful warning.
 >
 > We should definitely make it more robust.

 That's what it was? Damn. Well, yes, it should be more robust.

 Thank you,
 John Klos

Responsible-Changed-From-To: pkg-manager->cjep
Responsible-Changed-By: cjep@NetBSD.org
Responsible-Changed-When: Mon, 15 Feb 2021 10:38:26 +0000
Responsible-Changed-Why:
Will field this one.


State-Changed-From-To: open->analyzed
State-Changed-By: cjep@NetBSD.org
State-Changed-When: Mon, 15 Feb 2021 10:38:26 +0000
State-Changed-Why:
Will discuss with jperkin how to make two of the options more robust.
--prefer-pkgsrc should only accept yes, no or list. We should abort if 
it isn't set correctly.


State-Changed-From-To: analyzed->closed
State-Changed-By: cjep@NetBSD.org
State-Changed-When: Mon, 15 Feb 2021 12:57:49 +0000
State-Changed-Why:
We've committed a change to make bootstrap check the options better
for --prefer-pkgsrc.


From: "Chris Pinnock" <cjep@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/55952 CVS commit: pkgsrc/bootstrap
Date: Mon, 15 Feb 2021 12:56:53 +0000

 Module Name:	pkgsrc
 Committed By:	cjep
 Date:		Mon Feb 15 12:56:53 UTC 2021

 Modified Files:
 	pkgsrc/bootstrap: bootstrap

 Log Message:
 In the process of looking at PR#55952, we discovered something that
 can catch users out with the two --prefer options. Make the parsing
 of these better by checking their argument exists and is not another
 command line option.

 Addresses PR#55952. Reviewed with jperkin@


 To generate a diff of this commit:
 cvs rdiff -u -r1.296 -r1.297 pkgsrc/bootstrap/bootstrap

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

>Unformatted:

Home	PR Database Search