NetBSD Problem Report #54657

From gson@gson.org  Tue Oct 29 16:00:09 2019
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 5504E7A25D
	for <gnats-bugs@gnats.NetBSD.org>; Tue, 29 Oct 2019 16:00:09 +0000 (UTC)
Message-Id: <20191029160001.E4C33253F40@guava.gson.org>
Date: Tue, 29 Oct 2019 18:00:01 +0200 (EET)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@NetBSD.org
Subject: releasedir from evbarm "build.sh release" differs from ftp.netbsd.org
X-Send-Pr-Version: 3.95

>Number:         54657
>Category:       misc
>Synopsis:       releasedir from evarm "build.sh release" differs from ftp.netbsd.org
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    misc-bug-people
>State:          needs-pullups
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Oct 29 16:05:00 +0000 2019
>Closed-Date:    
>Last-Modified:  Fri Nov 01 14:33:20 +0000 2019
>Originator:     Andreas Gustafsson
>Release:        NetBSD-current
>Organization:

>Environment:
System: NetBSD
Architecture: arm
Machine: evbarm
>Description:

One might think that "build.sh release" or "make release" would
produce a releasedir with the same directory layout as the official
releases on ftp.netbsd.org, but for the various evbarm ports, this is
not the case.

Specifically, the release directory on ftp.netbsd.org has
subdirectories such as "evbarm-earmv7hf" and "evbarm-aarch64", but if
you run "build.sh release" yourself, the corresponding directory will
be called just "evbarm".

This causes complications for automated testing because you can't use
the same subdirectory when testing a local build as when testing an
official release, and in the case of a local build, you can't deduce
the port from the directory name.

Also, like PR 54544, it means that not all the code needed to fully
reproduce a NetBSD release is open source.

>How-To-Repeat:

>Fix:

Move the code to generate the unique subdirectory names used by the
official releases from localsrc to src/build.sh.

Who even has the authority to "declassify" non-public code like that?
Releng?  Core?  The board?

>Release-Note:

>Audit-Trail:
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: misc/54657: releasedir from evbarm "build.sh release" differs
 from ftp.netbsd.org
Date: Tue, 29 Oct 2019 17:20:28 +0100

 I don't think there is any such non public code, but this is a plain
 bug somewhere in the top level makefile. The difference is (from
 share/mk/bsd.README):

 RELEASEMACHINEDIR
                 Subdirectory used below RELEASEDIR when building
                 a release.  [${MACHINE}]

 and some makefiles using ${RELEASEDIR}/${MACHINE} instead of 
 ${RELEASEMACHINEDIR}. It probably depends on the exact build.sh invocation.
 The build cluster uses explict -a and -m arguments, while you probably 
 had your local invocation use the default for -a and only provided -m.
 (Just guessing, have not tested.)

 But you are right, it is a bug and we should fix it.

 Martin

From: Andreas Gustafsson <gson@gson.org>
To: Martin Husemann <martin@duskware.de>
Cc: gnats-bugs@netbsd.org
Subject: Re: misc/54657: releasedir from evbarm "build.sh release" differs from ftp.netbsd.org
Date: Tue, 29 Oct 2019 18:37:46 +0200

 Martin Husemann wrote:
 >  I don't think there is any such non public code

 localsrc/releng/autobuild/bin/buildmaster.sh lines 546 through 555.
 -- 
 Andreas Gustafsson, gson@gson.org

From: Andreas Gustafsson <gson@gson.org>
To: Martin Husemann <martin@duskware.de>
Cc: gnats-bugs@netbsd.org
Subject: Re: misc/54657: releasedir from evbarm "build.sh release" differs from ftp.netbsd.org
Date: Tue, 29 Oct 2019 18:47:10 +0200

 Martin Husemann wrote:
 >  The build cluster uses explict -a and -m arguments, while you probably 
 >  had your local invocation use the default for -a and only provided -m.
 >  (Just guessing, have not tested.)

 The build cluster suffers from the same issue, hence the non-public
 code to work around it.  When I discussed this with releng two years
 ago, Soren Jacobsen wrote:

   Unfortunately, this is done by a dumb hack in our build scripts :(  The
   actual builds create uninformative and non-unique directories just like
   build.sh does for everyone else.
   [...]
   It would be nice if normal builds just defaulted to generating the
   appropriate directories...

 -- 
 Andreas Gustafsson, gson@gson.org

From: Martin Husemann <martin@duskware.de>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: misc/54657: releasedir from evbarm "build.sh release" differs
 from ftp.netbsd.org
Date: Tue, 29 Oct 2019 17:50:13 +0100

 On Tue, Oct 29, 2019 at 06:47:10PM +0200, Andreas Gustafsson wrote:
 > The build cluster suffers from the same issue, hence the non-public
 > code to work around it.  When I discussed this with releng two years
 > ago, Soren Jacobsen wrote:
 > 
 >   Unfortunately, this is done by a dumb hack in our build scripts :(  The
 >   actual builds create uninformative and non-unique directories just like
 >   build.sh does for everyone else.
 >   [...]
 >   It would be nice if normal builds just defaulted to generating the
 >   appropriate directories...

 Indeed - that is very stupid and should be cleaned up. Maybe with the repo
 move?

 Martin

From: Andreas Gustafsson <gson@gson.org>
To: Martin Husemann <martin@duskware.de>
Cc: gnats-bugs@netbsd.org
Subject: Re: misc/54657: releasedir from evbarm "build.sh release" differs from ftp.netbsd.org
Date: Tue, 29 Oct 2019 18:55:35 +0200

 Martin Husemann wrote:
 > Indeed - that is very stupid and should be cleaned up. Maybe with the repo
 > move?

 I don't see why fixing this would depend on the repo move, nor vice
 versa.
 -- 
 Andreas Gustafsson, gson@gson.org

From: Martin Husemann <martin@duskware.de>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: misc/54657: releasedir from evbarm "build.sh release" differs
 from ftp.netbsd.org
Date: Tue, 29 Oct 2019 17:57:49 +0100

 On Tue, Oct 29, 2019 at 06:55:35PM +0200, Andreas Gustafsson wrote:
 > Martin Husemann wrote:
 > > Indeed - that is very stupid and should be cleaned up. Maybe with the repo
 > > move?
 > 
 > I don't see why fixing this would depend on the repo move, nor vice
 > versa.

 The repo move will cause a build cluster overhaul (or replacing of the
 homegrown auto build system by some off-the-shelf CI environment). So it
 is more a "needs to happen anyway when doing the repo move".

 Martin

From: Andreas Gustafsson <gson@gson.org>
To: Martin Husemann <martin@duskware.de>
Cc: gnats-bugs@netbsd.org
Subject: Re: misc/54657: releasedir from evbarm "build.sh release" differs from ftp.netbsd.org
Date: Tue, 29 Oct 2019 19:13:00 +0200

 Martin Husemann wrote:
 >  The repo move will cause a build cluster overhaul (or replacing of the
 >  homegrown auto build system by some off-the-shelf CI environment). So it
 >  is more a "needs to happen anyway when doing the repo move".

 If we fix build.sh now, there will one less thing to deal with in
 the build cluster overhaul :)
 -- 
 Andreas Gustafsson, gson@gson.org

From: Martin Husemann <martin@duskware.de>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: misc/54657: releasedir from evbarm "build.sh release" differs
 from ftp.netbsd.org
Date: Tue, 29 Oct 2019 19:26:47 +0100

 Here is a patch that should do the equivalent of the magic hidden code;
 not tested yet.

 Martin

 Index: bsd.README
 ===================================================================
 RCS file: /cvsroot/src/share/mk/bsd.README,v
 retrieving revision 1.394
 diff -u -r1.394 bsd.README
 --- bsd.README	10 Aug 2019 12:46:38 -0000	1.394
 +++ bsd.README	29 Oct 2019 18:24:37 -0000
 @@ -796,7 +796,8 @@

  RELEASEMACHINEDIR
  		Subdirectory used below RELEASEDIR when building
 -		a release.  [${MACHINE}]
 +		a release.  [${MACHINE},
 +		or ${MACHINE}-${MACHINE_ARCH} for evb{arm,mips,sh3}*]

  RELEASEMACHINE	Subdirectory or path component used for the following
  		paths:
 Index: bsd.own.mk
 ===================================================================
 RCS file: /cvsroot/src/share/mk/bsd.own.mk,v
 retrieving revision 1.1158
 diff -u -r1.1158 bsd.own.mk
 --- bsd.own.mk	24 Oct 2019 18:46:20 -0000	1.1158
 +++ bsd.own.mk	29 Oct 2019 18:24:38 -0000
 @@ -19,7 +19,12 @@
  #
  # Subdirectory used below ${RELEASEDIR} when building a release
  #
 +.if !empty(MACHINE:Mevbarm) || !empty(MACHINE:Mevbmips) \
 +	|| !empty(MACHINE:Mevbsh3)
 +RELEASEMACHINEDIR?=	${MACHINE}-${MACHINE_ARCH}
 +.else
  RELEASEMACHINEDIR?=	${MACHINE}
 +.endif

  #
  # Subdirectory or path component used for the following paths:

From: Andreas Gustafsson <gson@gson.org>
To: Martin Husemann <martin@duskware.de>
Cc: gnats-bugs@netbsd.org
Subject: Re: misc/54657: releasedir from evbarm "build.sh release" differs from ftp.netbsd.org
Date: Wed, 30 Oct 2019 21:25:52 +0200

 Martin,

 You wrote:
 > Here is a patch that should do the equivalent of the magic hidden code;
 > not tested yet.

 I tested the patch with evbarm-earmv7hf and it works for me.  Also,
 the bracket-based testbeds will work correctly with the patch without
 needing code changes (they will log a warning about not being able to
 symlink evbarm to evbarm-earmv7hf, but otherwise ignore the error).
 But it looks like buildmaster.sh will have to be updated to not
 attempt to rename the directory when it already has the right name,
 or to not treat a failed rename as an error.
 -- 
 Andreas Gustafsson, gson@gson.org

From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/54657 CVS commit: src/share/mk
Date: Thu, 31 Oct 2019 06:43:19 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Thu Oct 31 06:43:19 UTC 2019

 Modified Files:
 	src/share/mk: bsd.README bsd.own.mk

 Log Message:
 PR misc/54657: let RELEASEMACHINEDIR default to ${MACHINE}-${MACHINE_ARCH}
 for evb{arm,mips,sh3}*.


 To generate a diff of this commit:
 cvs rdiff -u -r1.394 -r1.395 src/share/mk/bsd.README
 cvs rdiff -u -r1.1158 -r1.1159 src/share/mk/bsd.own.mk

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->needs-pullups
State-Changed-By: gson@NetBSD.org
State-Changed-When: Fri, 01 Nov 2019 14:33:20 +0000
State-Changed-Why:
Martin committed a fix.


>Unformatted:

Home	PR Database Search