NetBSD Problem Report #15122

Received: (qmail 23657 invoked from network); 3 Jan 2002 10:26:44 -0000
Message-Id: <200201031026.g03AQOO03538@sam.few.eur.nl>
Date: Thu, 3 Jan 2002 11:26:24 +0100 (MET)
From: Paul Kranenburg <pk@cs.few.eur.nl>
Reply-To: pk@netbsd.org
To: gnats-bugs@gnats.netbsd.org
Subject: system make(1) performance Sucks
X-Send-Pr-Version: 3.95

>Number:         15122
>Category:       toolchain
>Synopsis:       system make(1) performance Sucks
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    toolchain-manager
>State:          closed
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Thu Jan 03 10:27:00 +0000 2002
>Closed-Date:    Mon May 09 23:37:49 +0000 2022
>Last-Modified:  Mon May 09 23:37:49 +0000 2022
>Originator:     Paul Kranenburg
>Release:        NetBSD-current 1/3/2002
>Organization:

>Environment:
System: NetBSD hamfast 1.5ZA NetBSD 1.5ZA (HAMFAST)
Architecture: sparc
Machine: sparc
>Description:
	Recent changes to various system make header files in /usr/share/mk
	cause a remarkable slowdown of the make process in the system
	source hierarchy (actually in any source hierarchy using the
	features provided by /usr/share/mk/*.mk).
>How-To-Repeat:
	Here's an example of a straightforward make process in a
	straightforward subdirectory of the NetBSD source tree:
	`make obj' in .../src/usr.bin. Note: all object directories
	were in place already so no time is spent on I/O to create
	directories. All time is spent by make(1) figuring out what
	to do; I'll let the numbers speak for themselves:

	1. The source tree as of May 2001:
		% cd /usr/src/usr.bin
		% time make obj
		46.3u 19.4s 1:15.61 86.9% 0+0k 731+588io 0pf+0w

	2. Current source tree as of Jan 2, 2002:
		% cd /usr/src/usr.bin
		% time make obj
		278.5u 129.3s 6:41.47 101.6% 0+0k 668+1049io 0pf+0w


	Similar results can be seen with `make cleandir'.

	The following extremely simple test environment also clearly
	shows the point.
	####start####
	#!/bin/sh
	D="b c d e f g h i j k l m n o p q r s t u v w x y z"

	mkdir /tmp/T
	cd /tmp/T

	mkdir a
	for d in $D; do
		ln -s a $d
	done

	cat << EOF > Makefile
	.include <bsd.own.mk>

	SUBDIR=	a $D

	.include <bsd.subdir.mk>
	EOF

	cat << EOF > a/Makefile
	.include <bsd.prog.mk>
	EOF

	time make -m /old/version/of/share/mk obj
	time make -m /usr/src/share/mk obj
	####end####
>Fix:
>Release-Note:
>Audit-Trail:

From: Paul Kranenburg <pk@cs.few.eur.nl>
To: gnats-bugs@gnats.netbsd.org
Cc: netbsd-bugs@netbsd.org
Subject: Re: toolchain/15122: system make(1) performance Sucks
Date: Thu, 3 Jan 2002 11:31:25 +0100 (MET)

[the original PR text sent contained the following typo; this has
 been corrected in text above]
 > 
 > 	1. The source tree as of May 2002:
                                  ^^^^^^^^
 That should be `May 2001'..

From: Robert Elz <kre@munnari.OZ.AU>
To: pk@cs.few.eur.nl
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: toolchain/15122: system make(1) performance Sucks 
Date: Thu, 03 Jan 2002 18:08:10 +0700

     Date:        Thu, 3 Jan 2002 11:26:24 +0100 (MET)
     From:        Paul Kranenburg <pk@cs.few.eur.nl>
     Message-ID:  <200201031026.g03AQOO03538@sam.few.eur.nl>

   | 	Recent changes to various system make header files in /usr/share/mk
   | 	cause a remarkable slowdown of the make process in the system
   | 	source hierarchy (actually in any source hierarchy using the
   | 	features provided by /usr/share/mk/*.mk).

 I'd guess this is the "walk up the tree" loop, looking for the new
 toolchain.

 Try defining _SRC_TOP_ - to a nullstring if you don't have the new tools,
 or to the directory containing build.sh if you do.

 This extra walk up the tree is likely to cause quite a bit of extra
 overhead for the almost null make targets (clean, obj), a noticeable
 overhead for the ones that do some work (depend, install), and almost
 nothing for the actual compiles.

 kre


From: Lennart Augustsson <lennart@augustsson.net>
To: pk@cs.few.eur.nl
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: toolchain/15122: system make(1) performance Sucks
Date: Thu, 03 Jan 2002 12:56:45 +0100

 I just want to second what Paul said.  For those of us with a little
 slower machines this is a real pain.  Obviously, changes that have
 not been performance tested very much have been made to the
 build system.

     -- Lennart



From: fair@netbsd.org
To: Paul Kranenburg <pk@cs.few.eur.nl>
Cc: gnats-bugs@netbsd.org
Subject: Re: toolchain/15122
Date: 4 Jan 2002 04:55:23 -0000

 Is there a reason why this PR is marked confidential?

 	Erik <fair@clock.org>

From: Paul Kranenburg <pk@cs.few.eur.nl>
To: Robert Elz <kre@munnari.OZ.AU>
Cc: gnats-bugs@gnats.netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: toolchain/15122: system make(1) performance Sucks
Date: Mon, 7 Jan 2002 15:02:20 +0100 (MET)

 > Try defining _SRC_TOP_ - to a nullstring if you don't have the new tools,
 > or to the directory containing build.sh if you do.

 It makes a difference. Though system and user times are still more than
 twice as much as they were before.

 Now, _SRC_TOP_ is a static value for a location of a given source
 tree, so there's absolutely no point in re-computing it in every
 invocation of make during a subtree walk.

 I'd rather see it move to build.sh from where it can be added as a make
 variable on the command line. This would have the added advantage that
 just typing `make' in any subdirectory of the source tree will use
 the installed toolchain instead of defaulting to the one in src/tools.

From: Todd Vierling <tv@wasabisystems.com>
To: Paul Kranenburg <pk@cs.few.eur.nl>
Cc: Robert Elz <kre@munnari.OZ.AU>, <gnats-bugs@gnats.netbsd.org>
Subject: Re: toolchain/15122: system make(1) performance Sucks
Date: Mon, 7 Jan 2002 09:52:15 -0500 (Eastern Standard Time)

 On Mon, 7 Jan 2002, Paul Kranenburg wrote:

 : > Try defining _SRC_TOP_ - to a nullstring if you don't have the new tools,
 : > or to the directory containing build.sh if you do.
 :
 : It makes a difference. Though system and user times are still more than
 : twice as much as they were before.

 This message is *not* being sent to netbsd-users (to avoid flamebait from a
 couple people), but is being sent to GNATS to be noted in the PR.

 Much of the time increase can be attributed to the @..@..@->.for transition.
 A while back, several .for loops were mutated into obnoxiously (and
 unreadably) long @..@..@ ODE-make loops.  Though these sped up operations on
 nearly the scale you reported, they made maintenance of the
 src/share/mk/*.mk files impossible.

 I got an OK from Christos to implement the transition back to .for loops,
 with the knowledge that this would slow down Makefile processing (since on a
 full build, the percentage of increased time is not nearly as significant as
 on just a "make obj").

 The solution to fixing this PR correctly is not to move back to unreadable
 @..@..@ variable constructs; rather, the processing of .for loops in make(1)
 needs to be much more optimized -- probably through the use of a loop parser
 that does expansion similarly to how @..@..@ works.

 : Now, _SRC_TOP_ is a static value for a location of a given source
 : tree, so there's absolutely no point in re-computing it in every
 : invocation of make during a subtree walk.

 It's not supposed to recompute it, as it's listed in .MAKEOVERRIDES+= and is
 within a ".if !defined(_SRC_TOP_)" block.

 Note, however, that the very first make(1) invocation does a few more things
 that it never did before, all in <bsd.own.mk>:

 * calculate _SRC_TOP_ (embedded shellcode)
 * calculate _SRC_TOP_OBJ_ (uses _SRC_TOP_ with PRINTOBJDIR)
 * caluclate HOST_OSTYPE (calls "uname" thrice)

 All three of these can be set manually, but they also should only be
 calculated in the first make(1) -- they are under conditionals and included
 in .MAKEOVERRIDES+=.

 : I'd rather see it move to build.sh from where it can be added as a make
 : variable on the command line. This would have the added advantage that
 : just typing `make' in any subdirectory of the source tree will use
 : the installed toolchain instead of defaulting to the one in src/tools.

 There's a very important reason why a full subtree defaults to src/tools
 based toolchain:  consistency with full builds.  Since the default of "make
 build" is the src/tools toolchain, it is assumed that any full tree will use
 this as a default.  (This was discussed at excruciating length on
 tech-toolchain.)

 You can change this behavior on a per-build basis by setting USETOOLS=no.

 -- 
 -- Todd Vierling <tv@wasabisystems.com>  *  Wasabi & NetBSD:  Run with it.
 -- CDs, Integration, Embedding, Support -- http://www.wasabisystems.com/


From: Lennart Augustsson <lennart@augustsson.net>
To: Todd Vierling <tv@wasabisystems.com>
Cc: Paul Kranenburg <pk@cs.few.eur.nl>, Robert Elz <kre@munnari.OZ.AU>,
   gnats-bugs@gnats.netbsd.org
Subject: Re: toolchain/15122: system make(1) performance Sucks
Date: Tue, 08 Jan 2002 00:28:00 +0100

 Todd Vierling wrote:

 > I got an OK from Christos to implement the transition back to .for loops,
 > with the knowledge that this would slow down Makefile processing (since on a
 > full build, the percentage of increased time is not nearly as significant as
 > on just a "make obj").

 I often run `build.sh -u' in which case the make processing takes up most of
 the time spent.


 > The solution to fixing this PR correctly is not to move back to unreadable
 > @..@..@ variable constructs; rather, the processing of .for loops in make(1)
 > needs to be much more optimized -- probably through the use of a loop parser
 > that does expansion similarly to how @..@..@ works.

 That sound lke the right solution to me.  When I find rebuilding too painful I might
 even try to fix make.  But don't hold your breath.

     -- Lennart


Responsible-Changed-From-To: bin-bug-people->toolchain-manager 
Responsible-Changed-By: gnats 
Responsible-Changed-When: Tue Feb 12 00:57:45 PST 2002 
Responsible-Changed-Why:  
There is now a "toolchain-manager" which is the proper default role 
account for handling problem reports in the toolchain category. 

State-Changed-From-To: open->feedback
State-Changed-By: rillig@NetBSD.org
State-Changed-When: Sat, 07 May 2022 18:50:52 +0000
State-Changed-Why:
Does this PR apply to NetBSD 10 as well?
If not, I'd like to close it.
If you find that .for loops are still too slow,
could you provide a minimal self-contained example?


State-Changed-From-To: feedback->closed
State-Changed-By: rillig@NetBSD.org
State-Changed-When: Mon, 09 May 2022 23:37:49 +0000
State-Changed-Why:
Mail to original reporter bounced.


>Unformatted:

Home	PR Database Search