NetBSD Problem Report #26072

Received: (qmail 26201 invoked by uid 605); 26 Jun 2004 15:45:20 -0000
Message-Id: <200406261545.i5QFj4Qh001211@seduction.immanent.net>
Date: Sat, 26 Jun 2004 10:45:04 -0500 (CDT)
From: Frederick Bruckman <fredb@immanent.net>
Sender: gnats-bugs-owner@NetBSD.org
Reply-To: fredb@immanent.net
To: gnats-bugs@gnats.NetBSD.org
Subject: Request: an option to build.sh, to build kernels without running "config" and "make depend"
X-Send-Pr-Version: 3.95

>Number:         26072
>Category:       toolchain
>Synopsis:       Request: an option to build.sh, to build kernels without running "config" and "make depend"
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    misc-bug-people
>State:          closed
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Sat Jun 26 15:46:00 +0000 2004
>Closed-Date:    Sat Jun 03 09:00:17 +0000 2023
>Last-Modified:  Sat Jun 03 09:00:17 +0000 2023
>Originator:     Frederick Bruckman
>Release:        NetBSD 2.0_BETA
>Organization:
>Environment:


System: NetBSD seduction.immanent.net 2.0_BETA NetBSD 2.0_BETA (SEDUCTION) #4: Fri Jun 25 22:14:46 CDT 2004 fredb@seduction.immanent.net:/p/i386--netbsd2/obj/usr/src/sys/arch/i386/compile/SEDUCTION i386
Architecture: i386
Machine: i386
>Description:

	Impatient kernel hackers would like to be able to build kernels using
	build.sh without running "config" and "make depend". The "make depend"
	part takes an especially long time, and it often happens that you've
	changed one source file in a way that you know that "make depend" isn't
	needed. It is always possible, of course, to build kernels directly in
	the build directory using the correct "nbmake-<MACHINE>", but it's
	inconvenient to type the tediously long path.
>How-To-Repeat:

>Fix:

	In the patch below, I'm lumping the "config" and "depend" together,
	even though "config" by itself isn't the burden, partly because "-M"
	is already taken. This is most useful when used with "-u". Perhaps it
	should imply "-u"? (It won't work at all with a clean build.)

Index: build.sh
===================================================================
RCS file: /cvsroot/src/build.sh,v
retrieving revision 1.129
diff -u -r1.129 build.sh
--- build.sh	25 Jun 2004 15:03:39 -0000	1.129
+++ build.sh	25 Jun 2004 23:54:25 -0000
@@ -111,6 +111,7 @@
 	runcmd=
 	operations=
 	removedirs=
+	do_config=true
 	do_expertmode=false
 	do_rebuildmake=false
 	do_removedirs=false
@@ -354,7 +355,7 @@
 	fi
 	cat <<_usage_

-Usage: ${progname} [-EnorUux] [-a arch] [-B buildid] [-D dest] [-j njob]
+Usage: ${progname} [-CEnorUux] [-a arch] [-B buildid] [-D dest] [-j njob]
 		[-M obj] [-m mach] [-N noisy] [-O obj] [-R release] [-T tools]
 		[-V var=[value]] [-w wrapper] [-X x11src] [-Z var]
 		operation [...]
@@ -382,6 +383,7 @@
  Options:
     -a arch     Set MACHINE_ARCH to arch.  [Default: deduced from MACHINE]
     -B buildId  Set BUILDID to buildId.
+    -C          Build kernels without running "configure" and "make depend".
     -D dest     Set DESTDIR to dest.  [Default: destdir.MACHINE]
     -E          Set "expert" mode; disables various safety checks.
                 Should not be used without expert knowledge of the build system.
@@ -420,7 +422,7 @@

 parseoptions()
 {
-	opts='a:B:bD:dEhi:j:k:M:m:N:nO:oR:rT:tUuV:w:xX:Z:'
+	opts='a:B:bCD:dEhi:j:k:M:m:N:nO:oR:rT:tUuV:w:xX:Z:'
 	opt_a=no

 	if type getopts >/dev/null 2>&1; then
@@ -464,6 +466,10 @@
 			usage "'-b' has been replaced by 'makewrapper'"
 			;;

+		-C)
+			do_config=false
+			;;
+
 		-D)
 			eval ${optargcmd}; resolvepath
 			setmakeenv DESTDIR "${OPTARG}"
@@ -950,12 +956,17 @@
 		    bomb "Failed to make cleandir in ${kernelbuildpath}"
 		${runcmd} cd "${TOP}"
 	fi
-	${runcmd} "${TOOLDIR}/bin/${toolprefix}config" -b "${kernelbuildpath}" \
-		-s "${TOP}/sys" "${kernelconfpath}" ||
-	    bomb "${toolprefix}config failed for ${kernelconf}"
+	if $do_config; then
+		${runcmd} "${TOOLDIR}/bin/${toolprefix}config" \
+			-b "${kernelbuildpath}" \
+			-s "${TOP}/sys" "${kernelconfpath}" ||
+		    bomb "${toolprefix}config failed for ${kernelconf}"
+	fi
 	${runcmd} cd "${kernelbuildpath}"
-	${runcmd} "${makewrapper}" depend ||
-	    bomb "Failed to make depend in ${kernelbuildpath}"
+	if $do_config; then
+		${runcmd} "${makewrapper}" depend ||
+		    bomb "Failed to make depend in ${kernelbuildpath}"
+	fi
 	${runcmd} "${makewrapper}" ${parallel} all ||
 	    bomb "Failed to make all in ${kernelbuildpath}"
 	${runcmd} cd "${TOP}"

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: misc-bug-people->lukem 
Responsible-Changed-By: fredb 
Responsible-Changed-When: Sat Jun 26 10:50:37 CDT 2004 
Responsible-Changed-Why:  

Luke -- please consider this, or something like it, for inclusion in the tree. 
Thanks. 


From: "Greg A. Woods" <woods@weird.com>
To: fredb@immanent.net
Cc: NetBSD GNATS submissions and followups <gnats-bugs@gnats.netbsd.org>,
  NetBSD Bugs and PR posting List <netbsd-bugs@NetBSD.ORG>
Subject: Re: misc/26072: Request: an option to build.sh, to build kernels without running "config" and "make depend"
Date: Sat, 26 Jun 2004 15:30:29 -0400 (EDT)

 [ On Saturday, June 26, 2004 at 10:45:04 (-0500), Frederick Bruckman wrote: ]
 > Subject: misc/26072: Request: an option to build.sh, to build kernels without running "config" and "make depend"
 >
 > 	Impatient kernel hackers would like to be able to build kernels using
 > 	build.sh without running "config" and "make depend". The "make depend"
 > 	part takes an especially long time, and it often happens that you've
 > 	changed one source file in a way that you know that "make depend" isn't
 > 	needed.

 Hmmm...  maybe "build.sh" isn't the best venue for this.  Below are some
 changes adding a new "custom-kern" target I added to src/Makefile and
 src/etc/Makefile.  These have served me very well.

 These are against the netbsd-1-6 branch, and line numbers are way off,
 but they should give you an idea of what I mean:

 Index: Makefile
 ===================================================================
 RCS file: /cvs/master/m-NetBSD/main/src/Makefile,v
 retrieving revision 1.175.2.3
 diff -u -r1.175.2.3 Makefile
 --- Makefile	30 Nov 2002 15:27:33 -0000	1.175.2.3
 +++ Makefile	1 Nov 2003 22:13:22 -0000
 @@ -219,6 +223,21 @@
  # XXX this needs to change when distrib Makefiles are recursion compliant
  dependall-distrib depend-distrib all-distrib:
  	@true
 +# make CUSTOMKERN=CONFIG [UPDATE=yes] [NODEPEND=yes]
 +#
 +.PHONY: custom-kern
 +custom-kern: .MAKE
 +	(cd ${.CURDIR}/etc && ${MAKE} $@)


 Index: etc/Makefile
 ===================================================================
 RCS file: /cvs/master/m-NetBSD/main/src/etc/Makefile,v
 retrieving revision 1.219.2.4
 diff -u -r1.219.2.4 Makefile
 --- etc/Makefile	24 Nov 2002 15:47:37 -0000	1.219.2.4
 +++ etc/Makefile	21 Jun 2004 21:19:07 -0000
 @@ -378,5 +392,25 @@

  snap_md_post: check_DESTDIR check_RELEASEDIR
  # nothing here -- look in the machine-dependent Makefile.inc
 +
 +.PHONY: custom-kern
 +custom-kern:
 +.for configfile in ${CUSTOMKERN}
 +	@[ ! -d ${KERNOBJDIR} ] && mkdir -p ${KERNOBJDIR} || true
 +	cd ${KERNCONFDIR} && ${CONFIG} -b ${KERNOBJDIR}/${configfile:C/.*\///} -s ${KERNSRCDIR} ${configfile}
 +. if !defined(UPDATE)
 +	cd ${KERNOBJDIR}/${configfile:C/.*\///} && ${MAKE} distclean
 +. endif
 +. if defined(NODEPEND)
 +	cd ${KERNOBJDIR}/${configfile:C/.*\///} && ${MAKE}
 +. else
 +	cd ${KERNOBJDIR}/${configfile:C/.*\///} && ${MAKE} depend && ${MAKE}
 +. endif
 +	@echo ""
 +	@what ${KERNOBJDIR}/${configfile:C/.*\///}/netbsd
 +	@echo ""
 +	@echo "get your new '${configfile}' kernel from:"
 +	@echo "	${KERNOBJDIR}/${configfile:C/.*\///}/netbsd"
 +	@echo ""
 +.endfor

  .include <bsd.prog.mk>


 >        It is always possible, of course, to build kernels directly in
 > 	the build directory using the correct "nbmake-<MACHINE>", but it's
 > 	inconvenient to type the tediously long path.

 Well I'd argue that's also a slightly different problem too.  I solve
 this prolem with a small set of helper functions and aliases and
 environment variable settings that I source into my command shell.

 E.g. I would do the following in a new shell window to update a custom
 test kernel after making a quick change to one or a few source files
 where only those files needed re-compiling:

 	$ cd /usr/src
 	$ . ./myhelpers.ksh
 	$ seti386tools
 	$ nbmake-i386 custom-kern CUSTOMKERN=MYKERN UPDATE=yes NODEPEND=yes

 That last one could be shortened a lot more with either more specific
 makefile targets and/or additional command aliases, but normally I don't
 do it often enough in a fresh window (i.e. one with no shell history)
 that I find it too tedious to type.

 Note that one can be a bit more objective when deciding whether to use a
 makefile target or a script/alias/shell-function by considering the
 namespace issues.  Makefile targets are a convenient way of
 encapsulating shell functions in such a way that they are very easily
 accessed but are in a separate namespace from other commands.  Makefile
 targets also get very easy access to makefile macros (e.g. mk.conf
 settings) too, but on the other hand you can't set $PATH or create
 command aliases for the invoking shell from within a Makefile.

 The "seti386tools" alias sets up the right $PATH with the right
 $TOOLDIR:

 	$ alias setalphatools
 	setalphatools='setNOtools ; export TOOLDIR=${MY_alpha_TOOLDIR} ; dirprepend PATH ${TOOLDIR}/bin'
 	$ setalphatools                                                                                         
 	$ type nbmake-alpha
 	nbmake-alpha is a tracked alias for /build/woods/building/NetBSD-1.6.x-alpha-alpha-tools/bin/nbmake-alpha
 	$ alias nbmakealpha
 	nbmakealpha='${MY_alpha_TOOLDIR}/bin/nbmake-alpha'

 ("building" is $(hostname) in this example)

 I can post (and/or attach to this PR) the full "myHelpers.ksh" file if
 anyone would like it.  It's got other goodies such as aliases like
 'cdi386obj' (uses the ksh "cd old new" trick) which allows me to jump to
 the relative object directory without having to pollute my source tree
 with objdir symlinks.  It's not perfect, but in combination with my own
 build.sh wrapper I can build with source from pretty much anywhere and
 with products going to pretty much anywhere else, within certain naming
 conventions.

 alias cdalphaobj="cd / ${BUILD_HOME}/NetBSD-1.6.x-${HOST_MACHINE}-alpha-obj/"

 (which assumes various symlinks have been created in the objdir tree
 too, and they are by my build.sh wrapper...)

 -- 
 						Greg A. Woods

 +1 416 218-0098                  VE3TCP            RoboHack <woods@robohack.ca>
 Planix, Inc. <woods@planix.com>          Secrets of the Weird <woods@weird.com>

From: fredb@immanent.net (Frederick Bruckman)
To: gnats-bugs@gnats.netbsd.org
Cc:  
Subject: Re: misc/26072: Request: an option to build.sh, to build kernels without running "config" and "make depend"
Date: Wed, 30 Jun 2004 22:30:22 -0500 (CDT)

 In article <m1BeIsf-0002SpC@proven.weird.com>,
 	woods@weird.com (Greg A. Woods) writes:
 > [ On Saturday, June 26, 2004 at 10:45:04 (-0500), Frederick Bruckman wrote: ]
 >> Subject: misc/26072: Request: an option to build.sh, to build kernels without running "config" and "make depend"
 >>
 >> 	Impatient kernel hackers would like to be able to build kernels using
 >> 	build.sh without running "config" and "make depend". The "make depend"
 >> 	part takes an especially long time, and it often happens that you've
 >> 	changed one source file in a way that you know that "make depend" isn't
 >> 	needed.
 > 
 > Hmmm...  maybe "build.sh" isn't the best venue for this.  Below are some
 > changes adding a new "custom-kern" target I added to src/Makefile and
 > src/etc/Makefile.  These have served me very well.

 What I like about having it in "./build.sh", is that sometimes you want
 to update the tools and a custom kernel, sometimes build the tools and
 kernel from scratch, sometimes the kernel from scratch, sometimes you
 need to build a GENERIC kernel, sometimes only update a custom kernel,
 and so on. Sometimes you need to do all that for more than one port. I
 find it convenient to be able to do everything from the same directory,
 and with only minor changes to the recalled command line.

 -- 
 Frederick
Responsible-Changed-From-To: lukem->misc-bug-people
Responsible-Changed-By: lukem@NetBSD.org
Responsible-Changed-When: Sat, 16 Jun 2012 06:08:23 +0000
Responsible-Changed-Why:
I've not got the time to do this


State-Changed-From-To: open->feedback
State-Changed-By: lukem@NetBSD.org
State-Changed-When: Fri, 19 May 2023 20:31:35 +0000
State-Changed-Why:

I'm not convinced that the added complexity of another option is worth the
effort of adding and maintaining yet another option to build.sh.


If you want to recompile a kernel after a change to a source file
without re-running config & make depend, just run nbmake in the
kernel's build directory. E.g
        $TOOLDIR/nbmake-amd64 -C $SOMEDIR/sys/arch/amd64/compile/GENERIC
(with appropriate shell variables already set)
Note that the latter directory path is printed when you first run
build.sh kernel=GENERIC


I did some testing on a modern system (AMD Ryzen 9 5950X 16-Core),
comparing using build.sh (with the config and make depend passes)
versus just building the kernel, both without any file changes
as well as touching a single source file:

        time ./build.sh -j32 kernel=GENERIC
        4.31s

        time nbmake-amd64 -C $SOMEDIR/sys/arch/amd64/compile/GENERIC
        1.18s    


        touch ./sys/dev/usb/xhci.c
        time ./build.sh -j32 kernel=GENERIC
        15.51s

        touch ./sys/dev/usb/xhci.c
        time nbmake-amd64 -C $SOMEDIR/sys/arch/amd64/compile/GENERIC
        12.33s

Not that much quicker even with 1 file change. And if I care about those 3s
I can just run the latter variation without the rerun of config
or make depend.


Thoughts?

regards,
Luke.


From: Luke Mewburn <luke@mewburn.net>
To: gnats-bugs@netbsd.org
Cc: misc-bug-people@netbsd.org, netbsd-bugs@netbsd.org,
	gnats-admin@netbsd.org, lukem@netbsd.org, fredb@immanent.net
Subject: Re: toolchain/26072 (Request: an option to build.sh, to build
 kernels without running "config" and "make depend")
Date: Sat, 20 May 2023 06:36:08 +1000

 On 23-05-19 20:31, Luke Mewburn wrote:
   | I did some testing on a modern system (AMD Ryzen 9 5950X 16-Core),
   | comparing using build.sh (with the config and make depend passes)
   | versus just building the kernel, both without any file changes
   | as well as touching a single source file:
   | 
   |         time ./build.sh -j32 kernel=GENERIC
   |         4.31s
   | 
   |         time nbmake-amd64 -C $SOMEDIR/sys/arch/amd64/compile/GENERIC
   |         1.18s    

 Oops, I forgot to use -j 32 in the nbmake-amd64 cases.
 I didn't mean that, because I wanted a fair comparison.

 Didn't matter; takes the same time.


   |         touch ./sys/dev/usb/xhci.c
   |         time ./build.sh -j32 kernel=GENERIC
   |         15.51s
   | 
   |         touch ./sys/dev/usb/xhci.c
   |         time nbmake-amd64 -C $SOMEDIR/sys/arch/amd64/compile/GENERIC
   |         12.33s

 With -j 32, nbmake isn't any quicker here either, recompiling a single file.


 regards,
 Luke.

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: lukem@NetBSD.org
Subject: Re: toolchain/26072 (Request: an option to build.sh, to build
 kernels without running "config" and "make depend")
Date: Sat, 20 May 2023 04:44:15 +0000

 On Fri, May 19, 2023 at 08:31:35PM +0000, lukem@NetBSD.org wrote:
  > Synopsis: Request: an option to build.sh, to build kernels without running "config" and "make depend"
  > 
  > State-Changed-From-To: open->feedback
  > State-Changed-By: lukem@NetBSD.org
  > State-Changed-When: Fri, 19 May 2023 20:31:35 +0000
  > State-Changed-Why:
  > 
  > I'm not convinced that the added complexity of another option is worth the
  > effort of adding and maintaining yet another option to build.sh.

 I'm reasonably convinced it's not. Not only are build machines far
 faster than in 2004, but in the meantime we've also fixed up config
 (and to some extent also the depend rules) so that running config no
 longer automatically triggers a rebuild of ~half the kernel. Which it
 used to.

 -- 
 David A. Holland
 dholland@netbsd.org

State-Changed-From-To: feedback->closed
State-Changed-By: lukem@NetBSD.org
State-Changed-When: Sat, 03 Jun 2023 09:00:17 +0000
State-Changed-Why:
Feedback request bounced, and we aren't going to add the request anyway.


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.47 2022/09/11 19:34:41 kim Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2023 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.