NetBSD Problem Report #49276

From gson@gson.org  Sun Oct 12 15:47:45 2014
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 8E6E4A6657
	for <gnats-bugs@gnats.NetBSD.org>; Sun, 12 Oct 2014 15:47:45 +0000 (UTC)
Message-Id: <20141012154733.B57BE745341@guava.gson.org>
Date: Sun, 12 Oct 2014 18:47:33 +0300 (EEST)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@gnats.NetBSD.org
Subject: amd64 MKDEBUG builds randomly fail installing compat libraries
X-Send-Pr-Version: 3.95

>Number:         49276
>Category:       toolchain
>Synopsis:       amd64 MKDEBUG builds randomly fail installing compat libraries
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    toolchain-manager
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Oct 12 15:50:00 +0000 2014
>Closed-Date:    Tue Jul 21 19:05:09 +0000 2015
>Last-Modified:  Tue Jul 21 19:05:09 +0000 2015
>Originator:     Andreas Gustafsson
>Release:        NetBSD-current, source dates around 2014.10.10.17.09.18
>Organization:
>Environment:
System: NetBSD
Architecture: x86_64
Machine: amd64
>Description:

I'm trying to do debug builds (MKDEBUG=YES) of NetBSD-current/amd64
on a NetBSD/amd64 6.1.4 machine.  Since the machine has four cores,
I'm also passing build.sh the option "-j 4".

About half of these builds fail, with errors like this:

  --- install-compat ---
  --- /tmp/bracket/build/2014.10.10.17.09.18-amd64-debug/destdir/usr/lib/i386/libpcap.so.5.0 ---
  x86_64--netbsd-install: libpcap.so.5.0: stat: No such file or directory
  *** [/tmp/bracket/build/2014.10.10.17.09.18-amd64-debug/destdir/usr/lib/i386/libpcap.so.5.0] Error code 1
  nbmake[8]: stopped in /tmp/bracket/build/2014.10.10.17.09.18-amd64-debug/src/external/bsd/libpcap/lib

Or this:

  --- install-libbind9 ---
  --- /tmp/bracket/build/2014.10.11.08.22.34-amd64-debug/destdir/usr/lib/i386/libbind9.so.8.0 ---
  x86_64--netbsd-install: libbind9.so.8.0: stat: No such file or directory
  --- install-libirs ---
  --- /tmp/bracket/build/2014.10.11.08.22.34-amd64-debug/destdir/usr/libdata/debug/usr/lib/i386/libirs.so.8.0.debug ---
  --- install-libbind9 ---
  *** [/tmp/bracket/build/2014.10.11.08.22.34-amd64-debug/destdir/usr/lib/i386/libbind9.so.8.0] Error code 1
  nbmake[9]: stopped in /tmp/bracket/build/2014.10.11.08.22.34-amd64-debug/src/external/bsd/bind/lib/libbind9

The exact library involved changes from run to run, but they're all in
/usr/lib/i386.  This looks like a parallel make race condition of some
sort, like the closed PR 44046 but specific to the i386 compat libraries.

>How-To-Repeat:

Do a few amd64 release builds using build.sh with the options
"-j 4 -V MKDEBUG=YES".  Watch about half of them fail.

>Fix:

>Release-Note:

>Audit-Trail:
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: toolchain-manager@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org
Subject: Re: toolchain/49276: amd64 MKDEBUG builds randomly fail installing compat libraries
Date: Sun, 12 Oct 2014 19:04:19 +0200

 See also PR 49027.

 Martin

From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, toolchain-manager@netbsd.org, 
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
	gson@gson.org (Andreas Gustafsson)
Cc: 
Subject: Re: toolchain/49276: amd64 MKDEBUG builds randomly fail installing compat libraries
Date: Sun, 12 Oct 2014 16:27:41 -0400

 On Oct 12,  5:05pm, martin@duskware.de (Martin Husemann) wrote:
 -- Subject: Re: toolchain/49276: amd64 MKDEBUG builds randomly fail installin

 | The following reply was made to PR toolchain/49276; it has been noted by GNATS.
 | 
 | From: Martin Husemann <martin@duskware.de>
 | To: gnats-bugs@NetBSD.org
 | Cc: toolchain-manager@netbsd.org, gnats-admin@netbsd.org,
 | 	netbsd-bugs@netbsd.org
 | Subject: Re: toolchain/49276: amd64 MKDEBUG builds randomly fail installing compat libraries
 | Date: Sun, 12 Oct 2014 19:04:19 +0200
 | 
 |  See also PR 49027.

 can you replicate it if you set BUILD=yes?

 christos

From: Andreas Gustafsson <gson@gson.org>
To: christos@zoulas.com (Christos Zoulas)
Cc: gnats-bugs@NetBSD.org,
    toolchain-manager@netbsd.org,
    gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org
Subject: Re: toolchain/49276: amd64 MKDEBUG builds randomly fail installing compat libraries
Date: Tue, 14 Oct 2014 09:23:18 +0300

 Christos Zoulas wrote:
 > can you replicate it if you set BUILD=yes?

 I have now done 10 builds with with "-j 4 -V MKDEBUG=YES -V BUILD=yes",
 and there have been no failures so far.
 -- 
 Andreas Gustafsson, gson@gson.org

From: christos@zoulas.com (Christos Zoulas)
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@NetBSD.org, toolchain-manager@netbsd.org, 
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: toolchain/49276: amd64 MKDEBUG builds randomly fail installing compat libraries
Date: Tue, 14 Oct 2014 08:38:04 -0400

 On Oct 14,  9:23am, gson@gson.org (Andreas Gustafsson) wrote:
 -- Subject: Re: toolchain/49276: amd64 MKDEBUG builds randomly fail installin

 | Christos Zoulas wrote:
 | > can you replicate it if you set BUILD=yes?
 | 
 | I have now done 10 builds with with "-j 4 -V MKDEBUG=YES -V BUILD=yes",
 | and there have been no failures so far.

 Yes, as I mentioned in the past the problem is that the install rules don't
 depend on the build rules if BUILD is not set, so you end up installing while
 things are being built which is racy. This is why I added BUILD more than
 a decade ago.

 christos

From: Martin Husemann <martin@duskware.de>
To: Christos Zoulas <christos@zoulas.com>
Cc: Andreas Gustafsson <gson@gson.org>, gnats-bugs@NetBSD.org,
	toolchain-manager@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org
Subject: Re: toolchain/49276: amd64 MKDEBUG builds randomly fail installing compat libraries
Date: Tue, 14 Oct 2014 14:46:27 +0200

 On Tue, Oct 14, 2014 at 08:38:04AM -0400, Christos Zoulas wrote:
 > Yes, as I mentioned in the past the problem is that the install rules don't
 > depend on the build rules if BUILD is not set, so you end up installing while
 > things are being built which is racy. This is why I added BUILD more than
 > a decade ago.

 Can't we make it the default then? (At lest for MAKE_JOBS > 1)

 Martin

From: christos@zoulas.com (Christos Zoulas)
To: Martin Husemann <martin@duskware.de>
Cc: Andreas Gustafsson <gson@gson.org>, gnats-bugs@NetBSD.org, 
	toolchain-manager@netbsd.org, gnats-admin@netbsd.org, 
	netbsd-bugs@netbsd.org
Subject: Re: toolchain/49276: amd64 MKDEBUG builds randomly fail installing compat libraries
Date: Tue, 14 Oct 2014 12:18:26 -0400

 On Oct 14,  2:46pm, martin@duskware.de (Martin Husemann) wrote:
 -- Subject: Re: toolchain/49276: amd64 MKDEBUG builds randomly fail installin

 | On Tue, Oct 14, 2014 at 08:38:04AM -0400, Christos Zoulas wrote:
 | > Yes, as I mentioned in the past the problem is that the install rules don't
 | > depend on the build rules if BUILD is not set, so you end up installing while
 | > things are being built which is racy. This is why I added BUILD more than
 | > a decade ago.
 | 
 | Can't we make it the default then? (At lest for MAKE_JOBS > 1)

 I think it has to be...

 At this point typing make install and having it fail if the source was
 not built does not sound useful to me. But let's discuss it in tech-toolchain.
 If we are going to make it default for jobs > 1 then we might as well make
 it default for everything for consistency, and get rid of hundreds of lines
 of conditionals in the bsd.mk files.

 christos

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: toolchain/49276: amd64 MKDEBUG builds randomly fail installing
 compat libraries
Date: Tue, 14 Oct 2014 17:22:02 +0000

 On Tue, Oct 14, 2014 at 04:20:02PM +0000, Christos Zoulas wrote:
  >  
  >  I think it has to be...
  >  
  >  At this point typing make install and having it fail if the source was
  >  not built does not sound useful to me. But let's discuss it in tech-toolchain.
  >  If we are going to make it default for jobs > 1 then we might as well make
  >  it default for everything for consistency, and get rid of hundreds of lines
  >  of conditionals in the bsd.mk files.

 concur

 -- 
 David A. Holland
 dholland@netbsd.org

From: matthew green <mrg@eterna.com.au>
To: christos@zoulas.com (Christos Zoulas)
Cc: Andreas Gustafsson <gson@gson.org>, gnats-bugs@NetBSD.org,
    toolchain-manager@netbsd.org, gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org, Martin Husemann <martin@duskware.de>
Subject: re: toolchain/49276: amd64 MKDEBUG builds randomly fail installing compat libraries
Date: Wed, 15 Oct 2014 05:45:29 +1100

 why don't the dependancies work enough for this?  submake not
 seeing things?  i have mostly liked the fact that "make install"
 does not build stuff by default...

 what sort of effect does this have on build times?  it must 
 increase the dependancy list for "install" phases quite a lot.

 all that said, i'd rather the build worked properly than not,
 so until dependancies or whatever are fixed, BUILD=yes as a
 default seems quite reasonable.


 .mrg.

From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, toolchain-manager@netbsd.org, 
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
	gson@gson.org (Andreas Gustafsson)
Cc: 
Subject: re: toolchain/49276: amd64 MKDEBUG builds randomly fail installing compat libraries
Date: Tue, 14 Oct 2014 15:54:16 -0400

 On Oct 14,  6:50pm, mrg@eterna.com.au (matthew green) wrote:
 -- Subject: re: toolchain/49276: amd64 MKDEBUG builds randomly fail installin

 | The following reply was made to PR toolchain/49276; it has been noted by GNATS.
 | 
 | From: matthew green <mrg@eterna.com.au>
 | To: christos@zoulas.com (Christos Zoulas)
 | Cc: Andreas Gustafsson <gson@gson.org>, gnats-bugs@NetBSD.org,
 |     toolchain-manager@netbsd.org, gnats-admin@netbsd.org,
 |     netbsd-bugs@netbsd.org, Martin Husemann <martin@duskware.de>
 | Subject: re: toolchain/49276: amd64 MKDEBUG builds randomly fail installing compat libraries
 | Date: Wed, 15 Oct 2014 05:45:29 +1100
 | 
 |  why don't the dependancies work enough for this?  submake not
 |  seeing things?  i have mostly liked the fact that "make install"
 |  does not build stuff by default...
 |  
 |  what sort of effect does this have on build times?  it must 
 |  increase the dependancy list for "install" phases quite a lot.
 |  
 |  all that said, i'd rather the build worked properly than not,
 |  so until dependancies or whatever are fixed, BUILD=yes as a
 |  default seems quite reasonable.

 Or we can try to split the build process into two separate make commands?

 christos

From: Joerg Sonnenberger <joerg@britannica.bec.de>
To: gnats-bugs@NetBSD.org
Cc: toolchain-manager@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org, Andreas Gustafsson <gson@gson.org>
Subject: Re: toolchain/49276: amd64 MKDEBUG builds randomly fail installing
 compat libraries
Date: Tue, 14 Oct 2014 23:02:24 +0200

 On Sun, Oct 12, 2014 at 05:05:01PM +0000, Martin Husemann wrote:
 >  See also PR 49027.

 The correct approach for that would be to adopt bsd.buildinstall.mk for
 the libs.

 Joerg

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: toolchain/49276: amd64 MKDEBUG builds randomly fail installing
 compat libraries
Date: Wed, 15 Oct 2014 06:11:23 +0000

 On Tue, Oct 14, 2014 at 07:55:00PM +0000, Christos Zoulas wrote:
  >  Or we can try to split the build process into two separate make commands?

 What I did the last time I was laying out OS makefiles:

  - the recursive rule for a full build is "build" (not "all", which
 retains its traditional meaning);

  - there's a separate set of "stage-install" rules that hardlink or copy
 into $DESTDIR; the "install" target goes straight to the ultimate
 destination tree;

  - build is defined like this by default:

     # dependall means depend then compile, but it's important to run a
     # new make after depending so the new depends take effect.
     dependall-local: depend-local
 	    $(MAKE) all-local
     dependall: dependall-local

     # build means depend, compile, and stage-install; it also needs a
     # new make after depending. It could use the same one for compile
     # and stage-install, but that turns out to be awkward.
     build-local: dependall-local
 	    $(MAKE) stage-install-local
     build: build-local

 (-local means 'not in subdirs')


 This is probably not as parallelizable as one might wish, but it does
 AFAICT have the redeeming feature that it works, which the netbsd
 scheme doesn't quite always.

 -- 
 David A. Holland
 dholland@netbsd.org

From: David Laight <david@l8s.co.uk>
To: Joerg Sonnenberger <joerg@britannica.bec.de>
Cc: gnats-bugs@NetBSD.org
Subject: Re: toolchain/49276: amd64 MKDEBUG builds randomly fail installing compat libraries
Date: Sun, 7 Dec 2014 23:01:37 +0000

 On Tue, Oct 14, 2014 at 11:02:24PM +0200, Joerg Sonnenberger wrote:
 > On Sun, Oct 12, 2014 at 05:05:01PM +0000, Martin Husemann wrote:
 > >  See also PR 49027.
 > 
 > The correct approach for that would be to adopt bsd.buildinstall.mk for
 > the libs.
 > 
 > Joerg

 Or to link programs against the libraries in the object tree.

 Then you don't need the 'install' phase unless you want to
 create a file tree that matches the installed target system.
 Useful to chroot into, current tools might need for the sets.

 	David

 -- 
 David Laight: david@l8s.co.uk

State-Changed-From-To: open->closed
State-Changed-By: gson@NetBSD.org
State-Changed-When: Tue, 21 Jul 2015 19:05:09 +0000
State-Changed-Why:
Looks like this was fixed by bsd.lib.mk 1.358.  Many thanks to mrg.


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.