NetBSD Problem Report #50343

From www@NetBSD.org  Sun Oct 18 20:42:20 2015
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 5887CA5864
	for <gnats-bugs@gnats.NetBSD.org>; Sun, 18 Oct 2015 20:42:20 +0000 (UTC)
Message-Id: <20151018204218.D7E85A6552@mollari.NetBSD.org>
Date: Sun, 18 Oct 2015 20:42:18 +0000 (UTC)
From: mag@icus.se
Reply-To: mag@icus.se
To: gnats-bugs@NetBSD.org
Subject: pkgsrc bootstrap broken on fresh installation of NetBSD 7.0 amd64
X-Send-Pr-Version: www-1.0

>Number:         50343
>Category:       pkg
>Synopsis:       pkgsrc bootstrap broken on fresh installation of NetBSD 7.0 amd64
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    cjep
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Oct 18 20:45:00 +0000 2015
>Closed-Date:    Sun Feb 14 19:33:03 +0000 2021
>Last-Modified:  Sun Feb 14 19:33:03 +0000 2021
>Originator:     Magnus Ihse Bursie
>Release:        7.0
>Organization:
>Environment:
NetBSD pantheon-vbox.ihse.net 7.0 NetBSD 7.0 (GENERIC.201509250726Z) amd64

>Description:
In short: It seems that chosing to install pkgin during the NetBSD installation process will render pkgsrc impossible to install out of the box.

In details: I'm used to Linux but new to NetBSD. I downloaded the 7.0 amd64 iso and booted it in VirtualBox. At the configuration step in the installation, I selected "Enable installation of binary packages" (which enabled pkgin). I also selected "Additional packages" and entered "bash". (I'm not sure if this is needed to reproduce, probably not.)

I did not choose to install pkgsrc, at that time.

Later on, I found out that without pkgsrc, a NetBSD installation is not very useful. I tried to install pkgsrc by running these steps as root:
# cd /tmp
# ftp ftp://ftp.netbsd.org/pub/pkgsrc/stable/pkgsrc.tar.gz
# tar -xzvf pkgsrc.tar.gz -C /usr
# cd /usr/pkgsrc/bootstrap/
# ./bootstrap

As I understand, that would be the recommended way to install pkgsrc. However, this fails. The problematic part of the log:
===> Installing for pkg_install-20150901
ERROR: pkg_install-20150508 is already installed - perhaps an older version?
ERROR: If so, you may use either of:
ERROR:     - "pkg_delete pkg_install-20150508" and "/usr/pkgsrc/bootstrap/work/bin/bmake reinstall"
ERROR:       to upgrade properly
ERROR:     - "/usr/pkgsrc/bootstrap/work/bin/bmake update" to rebuild the package and all
ERROR:       of its dependencies
ERROR:     - "/usr/pkgsrc/bootstrap/work/bin/bmake replace" to replace only the package without
ERROR:       re-linking dependencies, risking various problems.
*** Error code 1

Stop.
bmake[2]: stopped in /usr/pkgsrc/pkgtools/pkg_install
*** Error code 1

Stop.
bmake[1]: stopped in /usr/pkgsrc/pkgtools/pkg_install
*** Error code 1

Stop.
bmake: stopped in /usr/pkgsrc/pkgtools/pkg_install
===> exited with status 1
aborted.

---

As a NetBSD newcomer, I find it highly problematic that selecting an option during installation that was presented as a nice option ("binary packages", yeah, fine), makes a later installation of pkgsrc to fail.

The issue is probably fixable. I googled a bit and tried the three recommended solutions. Neither of these worked, but at that point I might have messed up the system too much. (I ended up doing a complete reinstall instead).

My main point here is not that it's a highly problematic bug, it's that it presents a unnecessary high and very disappointing barrier to a new user.
>How-To-Repeat:
* Boot from NetBSD 7.0 ISO. 
* Select "Enable installation of binary packages" and enable it. 
* Also set "Additional packages" to "bash" (not sure it's needed)
* Download and unpack the stable pkgsrc.
* Run pkgsrc bootstrap.

>Fix:
Workaround: Do not install pkgin during installation time.

>Release-Note:

>Audit-Trail:
From: Magnus Ihse Bursie <mag@icus.se>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: install/50343: pkgsrc bootstrap broken on fresh installation of
 NetBSD 7.0 amd64
Date: Sun, 18 Oct 2015 22:54:00 +0200

 Here's what happened when I followed the suggestions. Not very helpful, 
 really. :(
 ---
 pantheon-vbox# pkg_delete pkg_install-20150508
 Package `pkg_install-20150508' is still required by other packages:
          pkgin-0.8.0nb1

 pantheon-vbox# /usr/pkgsrc/bootstrap/work/bin/bmake update
 bmake: don't know how to make update. Stop

 bmake: stopped in /usr/pkgsrc/bootstrap

 pantheon-vbox# /usr/pkgsrc/bootstrap/work/bin/bmake replace
 bmake: don't know how to make replace. Stop

 bmake: stopped in /usr/pkgsrc/bootstrap
 ---
 When I figured out I should "cd /usr/pkgsrc/pkgtools/pkg_install" before 
 trying option #2 and #3 (not entirely obvious, but sure, not too 
 difficult to figure out) I could run bmake update. However, when 
 restarting the bootstrap I instead get stuck at:
 ===> Installing for bootstrap-mk-files-20150908
 ERROR: bootstrap-mk-files-20150908 is already installed - perhaps an 
 older version?
 ERROR: If so, you may use either of:
 ERROR:     - "pkg_delete bootstrap-mk-files-20150908" and 
 "/usr/pkgsrc/bootstrap/work/bin/bmake reinstall"
 ERROR:       to upgrade properly
 ERROR:     - "/usr/pkgsrc/bootstrap/work/bin/bmake update" to rebuild 
 the package and all
 ERROR:       of its dependencies
 ERROR:     - "/usr/pkgsrc/bootstrap/work/bin/bmake replace" to replace 
 only the package without
 ERROR:       re-linking dependencies, risking various problems.
 *** Error code 1

 Stop.
 bmake[2]: stopped in /usr/pkgsrc/pkgtools/bootstrap-mk-files
 *** Error code 1

 Stop.
 bmake[1]: stopped in /usr/pkgsrc/pkgtools/bootstrap-mk-files
 *** Error code 1

 Stop.
 bmake: stopped in /usr/pkgsrc/pkgtools/bootstrap-mk-files
 ===> exited with status 1
 aborted.
 pantheon-vbox# pkg_delete bootstrap-mk-files-20150908
 Package `bootstrap-mk-files-20150908' is still required by other packages:
          bmake-20150505

 ... and at this point I didn't pursue this further, so I don't know how 
 many more package conflicts that await.

 /Magnus

From: Joerg Sonnenberger <joerg@britannica.bec.de>
To: gnats-bugs@NetBSD.org
Cc: install-manager@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org
Subject: Re: install/50343: pkgsrc bootstrap broken on fresh installation of
 NetBSD 7.0 amd64
Date: Mon, 19 Oct 2015 09:34:53 +0200

 On Sun, Oct 18, 2015 at 08:45:01PM +0000, mag@icus.se wrote:
 > In short: It seems that chosing to install pkgin during the NetBSD
 > installation process will render pkgsrc impossible to install out
 > of the box.

 In short: you are wrong.

 > In details: I'm used to Linux but new to NetBSD. I downloaded the 7.0
 > amd64 iso and booted it in VirtualBox. At the configuration step in the
 > installation, I selected "Enable installation of binary packages"
 > (which enabled pkgin). I also selected "Additional packages" and
 > entered "bash". (I'm not sure if this is needed to reproduce, probably not.)

 This means you will have some pre-installed packages and the associated
 package database entries in /usr/pkg and /var/db/pkg.

 > I did not choose to install pkgsrc, at that time.

 Yes, you did. pkgin is just a tool on top of the normal binary package
 management infrastructure.

 > Later on, I found out that without pkgsrc, a NetBSD installation is not
 > very useful. I tried to install pkgsrc by running these steps as root:
 > # cd /tmp
 > # ftp ftp://ftp.netbsd.org/pub/pkgsrc/stable/pkgsrc.tar.gz
 > # tar -xzvf pkgsrc.tar.gz -C /usr
 > # cd /usr/pkgsrc/bootstrap/
 > # ./bootstrap

 You don't have to do the bootstrap, your system already has all the
 parts.

 > As I understand, that would be the recommended way to install pkgsrc. However, this fails. The problematic part of the log:
 > ===> Installing for pkg_install-20150901
 > ERROR: pkg_install-20150508 is already installed - perhaps an older version?
 > ERROR: If so, you may use either of:
 > ERROR:     - "pkg_delete pkg_install-20150508" and "/usr/pkgsrc/bootstrap/work/bin/bmake reinstall"
 > ERROR:       to upgrade properly
 > ERROR:     - "/usr/pkgsrc/bootstrap/work/bin/bmake update" to rebuild the package and all
 > ERROR:       of its dependencies
 > ERROR:     - "/usr/pkgsrc/bootstrap/work/bin/bmake replace" to replace only the package without
 > ERROR:       re-linking dependencies, risking various problems.
 > *** Error code 1

 ...wich is not very surprising given that it is already installed as
 part of the pkgin choice. Just don't bootstrap.

 Joerg

From: Tobias Nygren <tnn@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: install/50343: pkgsrc bootstrap broken on fresh installation of
 NetBSD 7.0 amd64
Date: Mon, 19 Oct 2015 11:11:34 +0200

 On Mon, 19 Oct 2015 07:40:01 +0000 (UTC)
 Joerg Sonnenberger <joerg@britannica.bec.de> wrote:

 >  ...wich is not very surprising given that it is already installed as
 >  part of the pkgin choice. Just don't bootstrap.

 I wonder if we should make bootstrap fail early rather than late
 with a more useful error message when the prefix or pkgdb already
 contains bootstrap components. Either that or teach bootstrap how
 to gracefully replace the existing packages (but that seems harder).

Responsible-Changed-From-To: install-manager->pkg-manager
Responsible-Changed-By: tnn@NetBSD.org
Responsible-Changed-When: Mon, 19 Oct 2015 10:15:38 +0000
Responsible-Changed-Why:
moving to pkg as it is not a sysinst problem


From: Magnus Ihse Bursie <mag@icus.se>
To: gnats-bugs@NetBSD.org, install-manager@netbsd.org,
 gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Cc: 
Subject: Re: install/50343: pkgsrc bootstrap broken on fresh installation of
 NetBSD 7.0 amd64
Date: Mon, 19 Oct 2015 23:47:35 +0200

 Hi guys,

 Thanks for the quick response!

 On 2015-10-19 09:40, Joerg Sonnenberger wrote:
 >   On Sun, Oct 18, 2015 at 08:45:01PM +0000, mag@icus.se wrote:
 >   > In short: It seems that chosing to install pkgin during the NetBSD
 >   > installation process will render pkgsrc impossible to install out
 >   > of the box.
 >   
 >   In short: you are wrong.
 On the technical side: Yes, most likely; my experience with BSD-land is 
 only about 48 hours long. :-)

 My intention was to report an unfortunate initial user experience, and I 
 tried to keep from speculation on what caused this. My user experience, 
 however, is very much real.
 >   > In details: I'm used to Linux but new to NetBSD. I downloaded the 7.0
 >   > amd64 iso and booted it in VirtualBox. At the configuration step in the
 >   > installation, I selected "Enable installation of binary packages"
 >   > (which enabled pkgin). I also selected "Additional packages" and
 >   > entered "bash". (I'm not sure if this is needed to reproduce, probably not.)
 >   
 >   This means you will have some pre-installed packages and the associated
 >   package database entries in /usr/pkg and /var/db/pkg.
 That seems reasonable. I re-did the installation one more time. This is 
 what I end up with:

 pantheon-vbox$ ls /var/db/pkg
 libarchive-3.1.2nb1     pkg_install-20150508 pkgdb.byfile.db         
 pkgin-0.8.0nb1

 pantheon-vbox$ find /usr/pkg/
 /usr/pkg/
 /usr/pkg/man
 /usr/pkg/man/man1
 /usr/pkg/man/man1/pkg_add.1
 /usr/pkg/man/man1/pkg_admin.1
 /usr/pkg/man/man1/pkg_create.1
 /usr/pkg/man/man1/pkg_delete.1
 /usr/pkg/man/man1/pkg_info.1
 /usr/pkg/man/man1/pkgin.1
 /usr/pkg/man/man5
 /usr/pkg/man/man5/pkg_install.conf.5
 /usr/pkg/man/man5/pkg_summary.5
 /usr/pkg/man/man5/cpio.5
 /usr/pkg/man/man5/libarchive-formats.5
 /usr/pkg/man/man5/mtree.5
 /usr/pkg/man/man5/tar.5
 /usr/pkg/man/man7
 /usr/pkg/man/man7/pkgsrc.7
 /usr/pkg/man/man8
 /usr/pkg/man/man8/audit-packages.8
 /usr/pkg/man/man8/download-vulnerability-list.8
 /usr/pkg/man/man3
 /usr/pkg/man/man3/archive_entry.3
 /usr/pkg/man/man3/archive_entry_acl.3
 /usr/pkg/man/man3/archive_entry_linkify.3
 /usr/pkg/man/man3/archive_entry_paths.3
 /usr/pkg/man/man3/archive_entry_perms.3
 /usr/pkg/man/man3/archive_entry_stat.3
 /usr/pkg/man/man3/archive_entry_time.3
 /usr/pkg/man/man3/archive_read.3
 /usr/pkg/man/man3/archive_read_data.3
 /usr/pkg/man/man3/archive_read_disk.3
 /usr/pkg/man/man3/archive_read_extract.3
 /usr/pkg/man/man3/archive_read_filter.3
 /usr/pkg/man/man3/archive_read_format.3
 /usr/pkg/man/man3/archive_read_free.3
 /usr/pkg/man/man3/archive_read_header.3
 /usr/pkg/man/man3/archive_read_new.3
 /usr/pkg/man/man3/archive_read_open.3
 /usr/pkg/man/man3/archive_read_set_options.3
 /usr/pkg/man/man3/archive_util.3
 /usr/pkg/man/man3/archive_write.3
 /usr/pkg/man/man3/archive_write_blocksize.3
 /usr/pkg/man/man3/archive_write_data.3
 /usr/pkg/man/man3/archive_write_disk.3
 /usr/pkg/man/man3/archive_write_filter.3
 /usr/pkg/man/man3/archive_write_finish_entry.3
 /usr/pkg/man/man3/archive_write_format.3
 /usr/pkg/man/man3/archive_write_free.3
 /usr/pkg/man/man3/archive_write_header.3
 /usr/pkg/man/man3/archive_write_new.3
 /usr/pkg/man/man3/archive_write_open.3
 /usr/pkg/man/man3/archive_write_set_options.3
 /usr/pkg/man/man3/libarchive.3
 /usr/pkg/man/man3/libarchive_changes.3
 /usr/pkg/man/man3/libarchive_internals.3
 /usr/pkg/sbin
 /usr/pkg/sbin/audit-packages
 /usr/pkg/sbin/download-vulnerability-list
 /usr/pkg/sbin/pkg_add
 /usr/pkg/sbin/pkg_admin
 /usr/pkg/sbin/pkg_create
 /usr/pkg/sbin/pkg_delete
 /usr/pkg/sbin/pkg_info
 /usr/pkg/include
 /usr/pkg/include/archive.h
 /usr/pkg/include/archive_entry.h
 /usr/pkg/lib
 /usr/pkg/lib/libarchive.la
 /usr/pkg/lib/libarchive.a
 /usr/pkg/lib/libarchive.so
 /usr/pkg/lib/libarchive.so.13
 /usr/pkg/lib/libarchive.so.13.1.2
 /usr/pkg/lib/pkgconfig
 /usr/pkg/lib/pkgconfig/libarchive.pc
 /usr/pkg/etc
 /usr/pkg/etc/pkgin
 /usr/pkg/etc/pkgin/repositories.conf
 /usr/pkg/bin
 /usr/pkg/bin/pkgin
 /usr/pkg/share
 /usr/pkg/share/examples
 /usr/pkg/share/examples/pkgin
 /usr/pkg/share/examples/pkgin/preferred.conf.example
 /usr/pkg/share/examples/pkgin/repositories.conf.example


 >   
 >   > I did not choose to install pkgsrc, at that time.
 >   
 >   Yes, you did. pkgin is just a tool on top of the normal binary package
 >   management infrastructure.

 What I meant here was that in the installer I was given two option, one 
 to install pkgin (e: Enable installation of binary packages"), and one 
 to install pkgsrc ("f: Fetch and unpack pkgsrc for building from 
 source"). I did not chose to enable the latter in the installer. As a 
 result, I do not have a /usr/pkgsrc directory:

 pantheon-vbox$ ls /usr/
 X11R7   bin     games   include lib     libdata libexec mdec pkg     
 sbin    share   tests

 Maybe it's just a confusion of terms, but to me, this means I do not 
 have pkgsrc installed.

 I've run the installation process a number of times now to figure out 
 what happend. When I did chose the pkgsrc option instead, I did end up 
 with a /usr/pkgsrc directory.

 >   > Later on, I found out that without pkgsrc, a NetBSD installation is not
 >   > very useful. I tried to install pkgsrc by running these steps as root:
 >   > # cd /tmp
 >   > # ftp ftp://ftp.netbsd.org/pub/pkgsrc/stable/pkgsrc.tar.gz
 >   > # tar -xzvf pkgsrc.tar.gz -C /usr
 >   > # cd /usr/pkgsrc/bootstrap/
 >   > # ./bootstrap
 >   
 >   You don't have to do the bootstrap, your system already has all the
 >   parts.

 If that is indeed correct, then this bug report should be turned into 
 something like what Tobias Nygren suggests, namely that running 
 bootstrap when not needed should fail early.

 However, I'm not fully convinced this is correct. Running the bootstrap 
 on a fresh system without the pkgin installation results in a different 
 setup. It might be that all these extra packages are strictly not 
 *necessary* for utilizing pkgsrc. However, the difference is at the very 
 least *problematic*. Case in point, on the pkgin installation system I 
 do not have bmake installed. When running the pkgsrc bootstrap 
 successfully on my comparison machine, I have bmake. Several guides on 
 the 'net for working with pkgsrc suggests using bmake when interacting 
 with pkgsrc. I assume it is technically possible to rectify the 
 sitation, but my point here is that such adhoc fixes should not be 
 needed on a fresh installation.

 This is what I get when I chose not to install pkgin in the installer, 
 but instead download pkgsrc.tar.gz and run the bootstrap script. As you 
 can see, this situation differs from the pkgin setup.

 pantheon-vbox# ls /var/db/pkg/
 bmake-20150505                bootstrap-mk-files-20150908 
 pkg_install-20150901          pkgdb.byfile.db

 pantheon-vbox# find /usr/pkg
 /usr/pkg
 /usr/pkg/share
 /usr/pkg/share/mk
 /usr/pkg/share/mk/bsd.README
 /usr/pkg/share/mk/bsd.dep.mk
 /usr/pkg/share/mk/bsd.depall.mk
 /usr/pkg/share/mk/bsd.doc.mk
 /usr/pkg/share/mk/bsd.files.mk
 /usr/pkg/share/mk/bsd.hostprog.mk
 /usr/pkg/share/mk/bsd.inc.mk
 /usr/pkg/share/mk/bsd.info.mk
 /usr/pkg/share/mk/bsd.kernobj.mk
 /usr/pkg/share/mk/bsd.kinc.mk
 /usr/pkg/share/mk/bsd.kmod.mk
 /usr/pkg/share/mk/bsd.lib.mk
 /usr/pkg/share/mk/bsd.links.mk
 /usr/pkg/share/mk/bsd.man.mk
 /usr/pkg/share/mk/bsd.nls.mk
 /usr/pkg/share/mk/bsd.obj.mk
 /usr/pkg/share/mk/bsd.own.mk
 /usr/pkg/share/mk/bsd.prog.mk
 /usr/pkg/share/mk/bsd.subdir.mk
 /usr/pkg/share/mk/bsd.sys.mk
 /usr/pkg/share/mk/sys.mk
 /usr/pkg/bin
 /usr/pkg/bin/bmake
 /usr/pkg/man
 /usr/pkg/man/cat1
 /usr/pkg/man/man1
 /usr/pkg/man/man1/bmake.1
 /usr/pkg/man/man1/pkg_add.1
 /usr/pkg/man/man1/pkg_admin.1
 /usr/pkg/man/man1/pkg_create.1
 /usr/pkg/man/man1/pkg_delete.1
 /usr/pkg/man/man1/pkg_info.1
 /usr/pkg/man/man5
 /usr/pkg/man/man5/pkg_summary.5
 /usr/pkg/man/man5/pkg_install.conf.5
 /usr/pkg/man/man7
 /usr/pkg/man/man7/pkgsrc.7
 /usr/pkg/man/man8
 /usr/pkg/man/man8/audit-packages.8
 /usr/pkg/man/man8/download-vulnerability-list.8
 /usr/pkg/sbin
 /usr/pkg/sbin/pkg_add
 /usr/pkg/sbin/pkg_admin
 /usr/pkg/sbin/audit-packages
 /usr/pkg/sbin/download-vulnerability-list
 /usr/pkg/sbin/pkg_create
 /usr/pkg/sbin/pkg_delete
 /usr/pkg/sbin/pkg_info
 /usr/pkg/etc
 /usr/pkg/etc/mk.conf


 To phrase it differently: If I were to help a friend install NetBSD, I 
 would at this point tell him: "stay away from installing pkgin using the 
 installer! If you do, you will end up with a system that does not have a 
 standard pkgsrc installation (at least not without a lot of work)." 
 Unless the bootstrap script can be fixed to play more nicely with a 
 system that has pkgin installed from start by the installer, I would 
 recommend that you remove that option from the installer. It is more 
 contraproductive than helpful as it stands now.

 /Magnus

Responsible-Changed-From-To: pkg-manager->cjep
Responsible-Changed-By: cjep@NetBSD.org
Responsible-Changed-When: Sun, 14 Feb 2021 19:33:03 +0000
Responsible-Changed-Why:
Taking old PRs


State-Changed-From-To: open->closed
State-Changed-By: cjep@NetBSD.org
State-Changed-When: Sun, 14 Feb 2021 19:33:03 +0000
State-Changed-Why:
Hi - apologies it has taken so long to close this PR. I can see you've had
some discussion with other developers on it.

In short, there should be no need to bootstrap pkgsrc on NetBSD. It comes
with the tools. If you don't install pksgrc at install time, you can
always get it easily later either by downloading a tar.gz or by checking
it out of anonymous CVS. 

I'm going to close this PR for now. If you are still have trouble please
let me know or ask on one of the NetBSD mailing lists.

All the best
Chris


>Unformatted:

Home	PR Database Search