NetBSD Problem Report #54569

From greywolf@starwolf.com  Wed Sep 25 02:00:27 2019
Return-Path: <greywolf@starwolf.com>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id BAFB37A16B
	for <gnats-bugs@gnats.NetBSD.org>; Wed, 25 Sep 2019 02:00:27 +0000 (UTC)
Message-Id: <20190925020025.CC581BA@eddie.starwolf.com>
Date: Tue, 24 Sep 2019 19:00:25 -0700 (PDT)
From: greywolf@starwolf.com
Reply-To: greywolf@starwolf.com
To: gnats-bugs@NetBSD.org
Subject: build.sh system seems broken
X-Send-Pr-Version: 3.95

>Number:         54569
>Category:       port-amd64
>Synopsis:       build.sh system seems broken
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lukem
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Sep 25 02:05:00 +0000 2019
>Closed-Date:    Sat Jun 10 07:25:36 +0000 2023
>Last-Modified:  Sat Jun 10 07:25:36 +0000 2023
>Originator:     The Grey Wolf
>Release:        NetBSD 9.99.11
>Organization:
Star Wolf Innovations
>Environment:
System: NetBSD eddie.starwolf.com 9.99.11 NetBSD 9.99.11 (EDDIE) #4: Thu Sep 19 01:20:40 PDT 2019 greywolf@eddie.starwolf.com:/sys/arch/amd64/compile/EDDIE amd64
Architecture: x86_64
Machine: amd64
>Description:

I seem to be having some build issues happening.  My pertinent environment/variables are as follows:

MACHINE=<amd64>
MACHINE_ARCH=<(not set)>
MAKEOBJDIR=</usr/src/build/arch/amd64/obj>
DESTDIR=</usr/src/build/arch/amd64/build/>
RELEASEDIR=</usr/src/build/arch/amd64/release/>
PATH=</usr/src/tools/tools-amd64/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/X11/bin:/usr/X11/sbin:/root/bin:/root/bashmodules>
MANPATH=</usr/src/tools/tools-amd64/man:>
BSDSRCDIR=</usr/src>
BSDOBJDIR=</usr/src/build/arch/amd64/obj>
BUILDROOT=</usr/src/build>
TOOLDIR=</usr/src/tools/tools-amd64>

MACHINE_ARCH is not set because things seem to go horribly wrong when it is set to x86_64, even though that appears to be the default MACHINE_ARCH for MACHINE=amd64.

My command line (general) is as such:

./build.sh -m amd64 -D /usr/src/build/arch/amd64/build/ \
	-O /usr/src/build/arch/amd64/obj \
	-R /usr/src/build/arch/amd64/release/ \
	-T /usr/src/tools/tools-amd64 -j12 {target}

Doesn't matter what my target is -- that command line works just fine; UNLESS:

If I add '-u' t o that command line, I get failures.  Isn't -u supposed to
skip cleandir and the tools rebuild?  The tools rebuild fails if the tools
already exist.  (This bug has been present for quite some time.)

If I add '-x' (to build X11), I get failures before the toolchain finishes
building.  Why there?  Does the toolchain contain any weird X11 dependencies?

So right now, I can't do a build-by-stages (tools, build, distribution,
release).  I can only pick one target and build to that, as each build
seems to require a total wipeout and restart.

My /etc/mk.conf:  https://www.starwolf.com/machines/eddie/mk.conf

Build Logs:
"build release", success: https://www.starwolf.com/machines/eddie/buildlogs/2019-09-23/1433-build-release.out.gz
"build release with -x", failure: https://www.starwolf.com/machines/eddie/buildlogs/2019-09-24/0031-build-release-fail.gz

"build tools", success: https://www.starwolf.com/machines/eddie/buildlogs/2019-09-24/0105-build-tools.out.gz
"build tools with -x", failure:  https://www.starwolf.com/machines/eddie/buildlogs/2019-09-24/0228-build-tools.fail.gz

I do in fact have the X sources at /usr/xsrc, updated 2019-09-23 (and
nothing new seems to have come in).

>How-To-Repeat:
	./build.sh -m amd64 -D /usr/src/build/arch/amd64/build/ \
	-O /usr/src/build/arch/amd64/obj -R \
	/usr/src/build/arch/amd64/release/ -T /usr/src/tools/tools-amd64 \
	-j12 -x {tools|build|release}

>Fix:
	Not known at this point.

>Release-Note:

>Audit-Trail:
From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-amd64/54569: build.sh system seems broken
Date: Wed, 25 Sep 2019 06:17:32 +0000

 On Wed, Sep 25, 2019 at 02:05:01AM +0000, greywolf@starwolf.com wrote:
  > If I add '-x' (to build X11), I get failures before the toolchain
  > finishes building.  Why there?  Does the toolchain contain any
  > weird X11 dependencies?

 It might nowadays what with gallium and shaders, but I wouldn't think
 so, should be the other way around if anything.

 My guess would be that your tree is corrupted; there are various
 things that can happen with CVS checkouts that CVS doesn't notice are
 broken, like extra subtrees or subdirectories connected to the wrong
 directory on the server.

 First, though, check for an inconsistent tree caused by tags you
 didn't intend; since you're using current there should be no Tag files
 in any CVS control directory. If you find any, get rid of them by
 doing cvs update -A in the parent.

 Then you might try something like
    find src -path '*/CVS/Repository' | xargs grep . | sed 's,/CVS/Repository:, ,' | awk '$1!=$2'

 to see if any of the CVS subdirectory structure is messed up.

 This is all a guess though... the behavior you're seeing is pretty
 bizarre.

 -- 
 David A. Holland
 dholland@netbsd.org

From: Patrick Welche <prlw1@cam.ac.uk>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-amd64/54569: build.sh system seems broken
Date: Wed, 25 Sep 2019 09:37:57 +0100

 On Wed, Sep 25, 2019 at 02:05:01AM +0000, greywolf@starwolf.com wrote:
 > "build release with -x", failure: https://www.starwolf.com/machines/eddie/buildlogs/2019-09-24/0031-build-release-fail.gz

 That one seems to have disappeared...

From: Patrick Welche <prlw1@cam.ac.uk>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-amd64/54569: build.sh system seems broken
Date: Wed, 25 Sep 2019 10:03:04 +0100

 Concurring with David, looking at 0228-build-tools.fail, I could imagine
 seeing something like that if you had v 1.18 of
 external/bsd/mdocml/Makefile.inc rather than 1.19...

From: Greywolf <greywolf@starwolf.com>
To: gnats-bugs@netbsd.org, port-amd64-maintainer@netbsd.org,
 gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Cc: 
Subject: Re: port-amd64/54569: build.sh system seems broken
Date: Thu, 26 Sep 2019 18:33:21 -0700

 Hi, David, Patrick,

 I always run my updates with 'cvs up -AdP .' in /usr/src; I did the same in 
 /usr/xsrc in case the previous one missed it.

 I'm also mystified that my builds fail if I give the -u option after building 
 tools.

 				--*greywolf;

 On 2019-09-25 02:05, Patrick Welche wrote:
 > The following reply was made to PR port-amd64/54569; it has been noted by GNATS.
 > 
 > From: Patrick Welche <prlw1@cam.ac.uk>
 > To: gnats-bugs@netbsd.org
 > Cc:
 > Subject: Re: port-amd64/54569: build.sh system seems broken
 > Date: Wed, 25 Sep 2019 10:03:04 +0100
 > 
 >   Concurring with David, looking at 0228-build-tools.fail, I could imagine
 >   seeing something like that if you had v 1.18 of
 >   external/bsd/mdocml/Makefile.inc rather than 1.19...
 >   
 > 

Responsible-Changed-From-To: port-amd64-maintainer->lukem
Responsible-Changed-By: lukem@NetBSD.org
Responsible-Changed-When: Wed, 24 May 2023 11:04:05 +0000
Responsible-Changed-Why:


State-Changed-From-To: open->feedback
State-Changed-By: lukem@NetBSD.org
State-Changed-When: Wed, 24 May 2023 11:04:05 +0000
State-Changed-Why:

Is this still a problem?
Your mk.conf in the description doesn't appear to be a valid text file.

build.sh as a tool definitely works; I've built multiple builds of -current
for target amd64 on hosts NetBSD/amd64 and macOS/arm64, unprivileged, in
source and obj in my home dir.

For your build, a couple of features to suggestion:
- use build.sh -U or  mk.conf MKUNPRIVED=yes
- build.sh -u (MKUPDATE=yes) definitely works as advertised;
  I use that in my mk.conf all the time

My build environment is involves setting
   env MAKEOBJDIRPREFIX=some/obj
   env TOOLDIR=some/tool
   env MAKECONF=some/src/mk.conf
and running build.sh from some/src
(those are setenv in tcsh)
and my mk.conf is
MAKEVERBOSE=1
MKUNPRIVED=yes
MKUPDATE=yes
(all of which can be provided to build.sh CLI)

Arguably -U (MKUNPRIVED=yes) should be the default (see PR 42088)
and the N variations of OBJDIR handling could be cleaned up (out of scope
of this PR).



State-Changed-From-To: feedback->closed
State-Changed-By: lukem@NetBSD.org
State-Changed-When: Sat, 10 Jun 2023 07:25:36 +0000
State-Changed-Why:
Unable to reproduce, examples from build environment not available any more,
no response to feedback


>Unformatted:

Home	PR Database Search