NetBSD Problem Report #54544

From gson@gson.org  Fri Sep 13 15:04:42 2019
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 5D0307A1E9
	for <gnats-bugs@gnats.NetBSD.org>; Fri, 13 Sep 2019 15:04:42 +0000 (UTC)
Message-Id: <20190913150436.753FE98964F@guava.gson.org>
Date: Fri, 13 Sep 2019 18:04:36 +0300 (EEST)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@NetBSD.org
Subject: Release build options are not public
X-Send-Pr-Version: 3.95

>Number:         54544
>Category:       misc
>Synopsis:       Release build options are not public
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    misc-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Sep 13 15:05:00 +0000 2019
>Last-Modified:  Sat Sep 14 12:20:01 +0000 2019
>Originator:     Andreas Gustafsson
>Release:        NetBSD-current and others
>Organization:

>Environment:
System: NetBSD
Architecture: all
Machine: all
>Description:

One might think that "build.sh release" or "make release" would
produce output identical to an official NetBSD release built from the
same sources, but in fact it does not: official releases are built
using additional, non-default make options such as MKREPRO and
MKDEBUG, which vary by architecture.  There appears to be no publicly
documented procedure for building an actual release.  The scripts used
by releng to build the actual releases are in localsrc, which is not
public.

This seems at odds with the idea of NetBSD being a fully open source
system.  It makes it hard or impossible for non-developers to
reproduce a release, it renders NetBSD's support for reproducible
builds somewhat pointless, and it means that we are spending
considerable resources on automated testing of something that is
subtly different from the actual releases.

It was recently suggested on tech-kern that the options used when
building the official releases should be documented in a wiki page.
IMO, this is not an acceptable solution.  For purposes of automated
testing, it must be possible to script release builds such that they
use the actual release options without having to manually track
changes to a wiki page.

>How-To-Repeat:

>Fix:

As a first step, the scripts used to build the official releases, or
at the very least the machine-readable list of architecture dependent
make options used by those scripts, should be moved to a public
respository.

>Audit-Trail:
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: misc/54544: Release build options are not public
Date: Fri, 13 Sep 2019 17:29:44 +0200

 For the record: the suggested wiki page was more meant to be a guide
 how to reproduce a build.

 Furthermore the options used for each build are not secret, but part of
 each "base" set as ./etc/release. Only tricky part is to know which of those
 options are default values and which are not.

 The differences vary a bit for different branches, but for HEAD and netbsd-9
 the only relevant differences are MKDEBUG=yes (used for all architectures)
 and MKX11=no (only for sun2 and rs6000).

 All other options are time stamp specific (that is: the cvs timestamp
 used, which of course varies and needs to be adapted to reproduce a
 build), setting the verbosity of the build log, or build system
 environment specific. If any of the latter two would change anything in
 the build results, something would be wrong (and our builds not
 reproducable).

 Martin

From: Andreas Gustafsson <gson@gson.org>
To: Martin Husemann <martin@duskware.de>, gnats-bugs@netbsd.org
Cc: 
Subject: Re: misc/54544: Release build options are not public
Date: Sat, 14 Sep 2019 14:58:55 +0300

 Martin Husemann wrote:
 >  For the record: the suggested wiki page was more meant to be a guide
 >  how to reproduce a build.
 >  
 >  Furthermore the options used for each build are not secret, but part of
 >  each "base" set as ./etc/release. Only tricky part is to know which of those
 >  options are default values and which are not.

 Surely you don't mean that the documented procedure for building a
 release should involve extracting the build options from the "base"
 set of the release?  That would be a circular defintion, and while it
 could in theory work for reproducing a release that has already been
 made, it wouldn't work for doing advance testing of upcoming releases.

 >  The differences vary a bit for different branches, but for HEAD and netbsd-9
 >  the only relevant differences are MKDEBUG=yes (used for all architectures)
 >  and MKX11=no (only for sun2 and rs6000).

 I'm glad to hear that the differences are minor, but the options may
 change in the future, so if I hardcode the current settings into
 tests, they will sooner or later get out of date.

 If it's really just those two definitions, how hard could it be to
 make them the default in "build.sh release", or to move them into src
 in some other way?
 -- 
 Andreas Gustafsson, gson@gson.org

From: Martin Husemann <martin@duskware.de>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: misc/54544: Release build options are not public
Date: Sat, 14 Sep 2019 14:18:12 +0200

 On Sat, Sep 14, 2019 at 02:58:55PM +0300, Andreas Gustafsson wrote:
 > Surely you don't mean that the documented procedure for building a
 > release should involve extracting the build options from the "base"
 > set of the release?  That would be a circular defintion, and while it
 > could in theory work for reproducing a release that has already been
 > made, it wouldn't work for doing advance testing of upcoming releases.

 No, I mean: if you have a concrete build result, and want to reproduce that,
 it is already possible (though not as trivial as it should).

 Also I meant there are not magic or hidden things involved.

 > If it's really just those two definitions, how hard could it be to
 > make them the default in "build.sh release", or to move them into src
 > in some other way?

 I am actually in favor of making MKDEBUG=yes the default (I have used
 it in all local builds since ages). However, it has a pretty bad  build
 performance impact if your build machine does not have plenty of RAM
 *and* you are building some architecture that requires LLVM parts.
 See PR 54136.

 The other variation is passing -x to build.sh or not - and the two
 architectures where we do not pass -x are the ones where X is not buildable.
 Pretty sure this is not worth any changes.

 The missing documentation should be added, somewhere - I totally agree with
 your intention here.

 Martin

Home	PR Database Search