NetBSD Problem Report #53238

From kre@munnari.OZ.AU  Mon Apr 30 18:15:14 2018
Return-Path: <kre@munnari.OZ.AU>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id E0D7D7A210
	for <gnats-bugs@gnats.NetBSD.org>; Mon, 30 Apr 2018 18:15:13 +0000 (UTC)
Message-Id: <201804301814.w3UIE928003659@jinx.noi.kre.to>
Date: Tue, 1 May 2018 01:14:09 +0700 (+07)
From: kre@munnari.OZ.AU
To: gnats-bugs@NetBSD.org
Subject: MKREPRO builds use reproducible build flags building tools
X-Send-Pr-Version: 3.95

>Number:         53238
>Category:       toolchain
>Synopsis:       MKREPRO builds use reproducible build flags building tools
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    toolchain-manager
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Apr 30 18:20:00 +0000 2018
>Closed-Date:    Sat May 05 21:38:08 +0000 2018
>Last-Modified:  Mon May 07 07:00:01 +0000 2018
>Originator:     Robert Elz
>Release:        NetBSD 8.99.14
>Organization:
>Environment:


System: NetBSD jinx.noi.kre.to 8.99.14 NetBSD 8.99.14 (GENERIC) #3: Fri Apr 13 15:19:55 ICT 2018 kre@onyx.coe.psu.ac.th:/usr/obj/testing/amd64/sys/arch/amd64/compile/GENERIC amd64
Architecture: x86_64
Machine: amd64
>Description:
	If doing a build (build.sh) with -P (MKREPRO) on a host that
	does not have the CC command line flags that are needed for
	reproducible builds, the build fails.

>How-To-Repeat:

	Use a NetBSD-6 vintage build host (or anything else with
	a dated gcc), and attempt to run build.sh -P

	Expect to see something like:

dependall ===> tools/makekeys
/usr/obj/8/tools/bin/nbsed -e 's/#undef NARROWPROTO/#define NARROWPROTO/'   -e '
s/#undef FUNCPROTO/#define FUNCPROTO 15/'  < /release/8/src/xsrc/external/mit/xp
roto/dist/Xfuncproto.h.in > Xfuncproto.h
#    create  makekeys/makekeys.d
/usr/obj/8/tools/bin/nbhost-mkdep -f makekeys.d.tmp  --      -I/usr/obj/8/tools/
include/compat -I/release/8/src/tools/compat -DHAVE_NBTOOL_CONFIG_H=1 -D_FILE_OF
FSET_BITS=64 -I. -Wp,-fno-canonical-system-headers -DCSRG_BASED -DFUNCPROTO=15 -
DNARROWPROTO -I/release/8/amd64/usr/X11R7/include -D__AMD64__     /release/8/src
/xsrc/external/mit/libX11/dist/src/util/makekeys.c &&  mv makekeys.d.tmp makekey
s.d
#    create  makekeys/.depend
rm -f .depend
/usr/obj/8/tools/bin/nbhost-mkdep -s .lo\ .ln\ .d -d -f .depend makekeys.d
#   compile  makekeys/makekeys.lo
cc -O -I/usr/obj/8/tools/include/compat -I/release/8/src/tools/compat -DHAVE_NBT
OOL_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -I. -Wp,-fno-canonical-system-headers -DCS
RG_BASED -DFUNCPROTO=15 -DNARROWPROTO -I/release/8/amd64/usr/X11R7/include -D__A
MD64__ -c -o makekeys.lo.o    /release/8/src/xsrc/external/mit/libX11/dist/src/u
til/makekeys.c
cc1: error: unrecognized command line option "-fno-canonical-system-headers"

*** Failed target:  makekeys.lo

	There's no useful point building tools in a MKREPRO way

	The cause of this seems to be (in bsd.sys.mk)

.if ${MKREPRO:Uno} == "yes" && ${MKLLVM:Uno} != "yes"
.export NETBSDSRCDIR DESTDIR X11SRCDIR

.if !empty(DESTDIR)
CPPFLAGS+=      -Wp,-iremap,${DESTDIR}:
REPROFLAGS+=    -fdebug-prefix-map=\$$DESTDIR=
.endif  

CPPFLAGS+=      -Wp,-fno-canonical-system-headers
CPPFLAGS+=      -Wp,-iremap,${NETBSDSRCDIR}:/usr/src
CPPFLAGS+=      -Wp,-iremap,${X11SRCDIR}:/usr/xsrc
REPROFLAGS+=    -fdebug-prefix-map=\$$NETBSDSRCDIR=/usr/src
REPROFLAGS+=    -fdebug-prefix-map=\$$X11SRCDIR=/usr/xsrc
LINTFLAGS+=     -R${NETBSDSRCDIR}=/usr/src -R${X11SRCDIR}=/usr/xsrc

REPROFLAGS+=    -fdebug-regex-map='/usr/src/(.*)/obj.*=/usr/obj/\1'
REPROFLAGS+=    -fdebug-regex-map='/usr/src/(.*)/obj.*/(.*)=/usr/obj/\1/\2'

CFLAGS+=        ${REPROFLAGS}
CXXFLAGS+=      ${REPROFLAGS}
.endif

>Fix:
	Probably turn all of that off when building tools.

	NB: This affects NetBSD-8 builds, so needs a quick fix & pullup

>Release-Note:

>Audit-Trail:
From: "Christos Zoulas" <christos@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/53238 CVS commit: src/tools
Date: Mon, 30 Apr 2018 20:04:37 -0400

 Module Name:	src
 Committed By:	christos
 Date:		Tue May  1 00:04:36 UTC 2018

 Modified Files:
 	src/tools: Makefile.host
 	src/tools/binstall: Makefile
 	src/tools/binutils: Makefile
 	src/tools/compat: Makefile
 	src/tools/ctfconvert: Makefile
 	src/tools/ctfmerge: Makefile
 	src/tools/cvslatest: Makefile
 	src/tools/dbsym: Makefile
 	src/tools/dtc: Makefile
 	src/tools/gcc: Makefile
 	src/tools/gdb: Makefile
 	src/tools/genassym: Makefile
 	src/tools/gettext: Makefile
 	src/tools/libctf: Makefile
 	src/tools/libdwarf: Makefile
 	src/tools/libelf: Makefile
 	src/tools/libfdt: Makefile
 	src/tools/lorder: Makefile
 	src/tools/makekeys: Makefile
 	src/tools/mandoc: Makefile
 	src/tools/mdsetimage: Makefile
 Added Files:
 	src/tools: Makefile.inc

 Log Message:
 PR/53238: Robert Elz: Disable MKREPRO in tools; the host compiler might
 not support the necessary options. This is done thusly:

 1. Set MKREPRO=no in Makefile.host. This handles all the Makefiles that
    use it and don't include bsd.own.mk.
 2. Create Makefile.inc and set MKREPRO=no in it. Change the Makefiles that
    include bsd.own.mk, to include bsd.init.mk which includes Makefile.inc
    first. This will also allow us to control other tools options from a
    single location if we need to.

 XXX: pullup-8


 To generate a diff of this commit:
 cvs rdiff -u -r1.31 -r1.32 src/tools/Makefile.host
 cvs rdiff -u -r0 -r1.14 src/tools/Makefile.inc
 cvs rdiff -u -r1.11 -r1.12 src/tools/binstall/Makefile
 cvs rdiff -u -r1.26 -r1.27 src/tools/binutils/Makefile
 cvs rdiff -u -r1.81 -r1.82 src/tools/compat/Makefile
 cvs rdiff -u -r1.6 -r1.7 src/tools/ctfconvert/Makefile
 cvs rdiff -u -r1.7 -r1.8 src/tools/ctfmerge/Makefile
 cvs rdiff -u -r1.1 -r1.2 src/tools/cvslatest/Makefile
 cvs rdiff -u -r1.12 -r1.13 src/tools/dbsym/Makefile
 cvs rdiff -u -r1.2 -r1.3 src/tools/dtc/Makefile
 cvs rdiff -u -r1.84 -r1.85 src/tools/gcc/Makefile
 cvs rdiff -u -r1.34 -r1.35 src/tools/gdb/Makefile
 cvs rdiff -u -r1.6 -r1.7 src/tools/genassym/Makefile
 cvs rdiff -u -r1.6 -r1.7 src/tools/gettext/Makefile
 cvs rdiff -u -r1.6 -r1.7 src/tools/libctf/Makefile
 cvs rdiff -u -r1.7 -r1.8 src/tools/libdwarf/Makefile
 cvs rdiff -u -r1.8 -r1.9 src/tools/libelf/Makefile
 cvs rdiff -u -r1.2 -r1.3 src/tools/libfdt/Makefile
 cvs rdiff -u -r1.12 -r1.13 src/tools/lorder/Makefile
 cvs rdiff -u -r1.1 -r1.2 src/tools/makekeys/Makefile
 cvs rdiff -u -r1.10 -r1.11 src/tools/mandoc/Makefile
 cvs rdiff -u -r1.14 -r1.15 src/tools/mdsetimage/Makefile

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->pending-pullups
State-Changed-By: kre@NetBSD.org
State-Changed-When: Tue, 01 May 2018 15:02:20 +0000
State-Changed-Why:
Fixed by Christos, pending pullup-8 #796


From: "Christos Zoulas" <christos@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/53238 CVS commit: src/external/mit/xorg/tools
Date: Tue, 1 May 2018 13:42:01 -0400

 Module Name:	src
 Committed By:	christos
 Date:		Tue May  1 17:42:01 UTC 2018

 Modified Files:
 	src/external/mit/xorg/tools/bdftopcf: Makefile
 	src/external/mit/xorg/tools/fc-cache: Makefile
 	src/external/mit/xorg/tools/makekeys: Makefile
 	src/external/mit/xorg/tools/makestrs: Makefile
 	src/external/mit/xorg/tools/mkfontscale: Makefile
 	src/external/mit/xorg/tools/ucs2any: Makefile
 Added Files:
 	src/external/mit/xorg/tools: Makefile.inc

 Log Message:
 Apply the same fixes from the regular tools dir:

 PR/53238: Robert Elz: Disable MKREPRO in tools; the host compiler might
 not support the necessary options. This is done thusly:

 1. Create Makefile.inc and set MKREPRO=no in it. Change the Makefiles that
    include bsd.own.mk, to include bsd.init.mk which includes Makefile.inc
    first. This will also allow us to control other tools options from a
    single location if we need to.
 2. Centralize NOMAN handling.

 XXX: pullup-8


 To generate a diff of this commit:
 cvs rdiff -u -r0 -r1.1 src/external/mit/xorg/tools/Makefile.inc
 cvs rdiff -u -r1.7 -r1.8 src/external/mit/xorg/tools/bdftopcf/Makefile
 cvs rdiff -u -r1.11 -r1.12 src/external/mit/xorg/tools/fc-cache/Makefile
 cvs rdiff -u -r1.5 -r1.6 src/external/mit/xorg/tools/makekeys/Makefile
 cvs rdiff -u -r1.4 -r1.5 src/external/mit/xorg/tools/makestrs/Makefile
 cvs rdiff -u -r1.8 -r1.9 src/external/mit/xorg/tools/mkfontscale/Makefile
 cvs rdiff -u -r1.2 -r1.3 src/external/mit/xorg/tools/ucs2any/Makefile

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/53238 CVS commit: [netbsd-8] src/tools
Date: Sat, 5 May 2018 19:25:57 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Sat May  5 19:25:57 UTC 2018

 Modified Files:
 	src/tools [netbsd-8]: Makefile.host
 	src/tools/binstall [netbsd-8]: Makefile
 	src/tools/binutils [netbsd-8]: Makefile
 	src/tools/compat [netbsd-8]: Makefile
 	src/tools/ctfconvert [netbsd-8]: Makefile
 	src/tools/ctfmerge [netbsd-8]: Makefile
 	src/tools/cvslatest [netbsd-8]: Makefile
 	src/tools/dbsym [netbsd-8]: Makefile
 	src/tools/dtc [netbsd-8]: Makefile
 	src/tools/gcc [netbsd-8]: Makefile
 	src/tools/gdb [netbsd-8]: Makefile
 	src/tools/genassym [netbsd-8]: Makefile
 	src/tools/gettext [netbsd-8]: Makefile
 	src/tools/libctf [netbsd-8]: Makefile
 	src/tools/libdwarf [netbsd-8]: Makefile
 	src/tools/libelf [netbsd-8]: Makefile
 	src/tools/libfdt [netbsd-8]: Makefile
 	src/tools/lorder [netbsd-8]: Makefile
 	src/tools/makekeys [netbsd-8]: Makefile
 	src/tools/mandoc [netbsd-8]: Makefile
 	src/tools/mdsetimage [netbsd-8]: Makefile
 Added Files:
 	src/tools [netbsd-8]: Makefile.inc

 Log Message:
 Pull up following revision(s) (requested by christos in ticket #796):

 	tools/mdsetimage/Makefile: revision 1.15
 	tools/compat/Makefile: revision 1.82
 	tools/gdb/Makefile: revision 1.35
 	tools/lorder/Makefile: revision 1.13
 	tools/gcc/Makefile: revision 1.85
 	tools/dtc/Makefile: revision 1.3
 	tools/cvslatest/Makefile: revision 1.2
 	tools/ctfmerge/Makefile: revision 1.8
 	tools/libelf/Makefile: revision 1.9
 	tools/libdwarf/Makefile: revision 1.8
 	tools/ctfconvert/Makefile: revision 1.7
 	tools/makekeys/Makefile: revision 1.2
 	tools/gettext/Makefile: revision 1.7
 	tools/binstall/Makefile: revision 1.12
 	tools/libfdt/Makefile: revision 1.3
 	tools/libctf/Makefile: revision 1.7
 	tools/binutils/Makefile: revision 1.27
 	tools/mandoc/Makefile: revision 1.11
 	tools/Makefile.host: revision 1.32
 	tools/dbsym/Makefile: revision 1.13
 	tools/genassym/Makefile: revision 1.7
 	tools/Makefile.inc: revision 1.14

 PR/53238: Robert Elz: Disable MKREPRO in tools; the host compiler might
 not support the necessary options. This is done thusly:

 1. Set MKREPRO=no in Makefile.host. This handles all the Makefiles that
    use it and don't include bsd.own.mk.
 2. Create Makefile.inc and set MKREPRO=no in it. Change the Makefiles that
    include bsd.own.mk, to include bsd.init.mk which includes Makefile.inc
    first. This will also allow us to control other tools options from a
    single location if we need to.

 XXX: pullup-8


 To generate a diff of this commit:
 cvs rdiff -u -r1.31 -r1.31.20.1 src/tools/Makefile.host
 cvs rdiff -u -r0 -r1.16.2.2 src/tools/Makefile.inc
 cvs rdiff -u -r1.11 -r1.11.22.1 src/tools/binstall/Makefile
 cvs rdiff -u -r1.26 -r1.26.6.1 src/tools/binutils/Makefile
 cvs rdiff -u -r1.81 -r1.81.8.1 src/tools/compat/Makefile
 cvs rdiff -u -r1.6 -r1.6.8.1 src/tools/ctfconvert/Makefile
 cvs rdiff -u -r1.7 -r1.7.8.1 src/tools/ctfmerge/Makefile
 cvs rdiff -u -r1.1 -r1.1.8.1 src/tools/cvslatest/Makefile
 cvs rdiff -u -r1.11 -r1.11.6.1 src/tools/dbsym/Makefile
 cvs rdiff -u -r1.2.4.2 -r1.2.4.3 src/tools/dtc/Makefile
 cvs rdiff -u -r1.81.4.1 -r1.81.4.2 src/tools/gcc/Makefile
 cvs rdiff -u -r1.30 -r1.30.6.1 src/tools/gdb/Makefile
 cvs rdiff -u -r1.6 -r1.6.22.1 src/tools/genassym/Makefile
 cvs rdiff -u -r1.6 -r1.6.22.1 src/tools/gettext/Makefile
 cvs rdiff -u -r1.6 -r1.6.8.1 src/tools/libctf/Makefile
 cvs rdiff -u -r1.7 -r1.7.18.1 src/tools/libdwarf/Makefile
 cvs rdiff -u -r1.8 -r1.8.18.1 src/tools/libelf/Makefile
 cvs rdiff -u -r1.2.4.2 -r1.2.4.3 src/tools/libfdt/Makefile
 cvs rdiff -u -r1.12 -r1.12.22.1 src/tools/lorder/Makefile
 cvs rdiff -u -r1.1 -r1.1.8.1 src/tools/makekeys/Makefile
 cvs rdiff -u -r1.9.8.1 -r1.9.8.2 src/tools/mandoc/Makefile
 cvs rdiff -u -r1.13 -r1.13.6.1 src/tools/mdsetimage/Makefile

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: pending-pullups->closed
State-Changed-By: kre@NetBSD.org
State-Changed-When: Sat, 05 May 2018 21:38:08 +0000
State-Changed-Why:
Pullups completed


From: Soren Jacobsen <snj@blef.org>
To: gnats-bugs@NetBSD.org
Cc: kre@munnari.OZ.AU
Subject: Re: toolchain/53238 (MKREPRO builds use reproducible build flags
 building tools)
Date: Sun, 6 May 2018 22:13:58 -0700

 On 05/05 21:38, kre@NetBSD.org wrote:
 > State-Changed-From-To: pending-pullups->closed
 > State-Changed-By: kre@NetBSD.org
 > State-Changed-When: Sat, 05 May 2018 21:38:08 +0000
 > State-Changed-Why:
 > Pullups completed

 The X fixes are still missing, aren't they?

From: Robert Elz <kre@munnari.OZ.AU>
To: Soren Jacobsen <snj@blef.org>
Cc: gnats-bugs@NetBSD.org
Subject: Re: toolchain/53238 (MKREPRO builds use reproducible build flags building tools)
Date: Mon, 07 May 2018 13:57:25 +0700

     Date:        Sun, 6 May 2018 22:13:58 -0700
     From:        Soren Jacobsen <snj@blef.org>
     Message-ID:  <20180507051358.GF2204@blef.org>

   | The X fixes are still missing, aren't they?

 They weren't part of the original issue - not sure if Christos wants those
 pulled up or not.

 kre

>Unformatted:

Home	PR Database Search