NetBSD Problem Report #60324

From kre@munnari.OZ.AU  Fri Jun 12 08:22:15 2026
Return-Path: <kre@munnari.OZ.AU>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "mail.netbsd.org", Issuer "R13" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id B88001A923A
	for <gnats-bugs@gnats.NetBSD.org>; Fri, 12 Jun 2026 08:22:15 +0000 (UTC)
Message-Id: <202606120821.65C8LJXF003057@jacaranda.noi.kre.to>
Date: Fri, 12 Jun 2026 15:21:19 +0700 (+07)
From: kre@munnari.OZ.AU
To: gnats-bugs@NetBSD.org
Subject: unwind.h build race condition
X-Send-Pr-Version: 3.95
X-From4GNATS: "kre@munnari.OZ.AU via gnats" <gnats-admin@NetBSD.org>

>Number:         60324
>Category:       lib
>Synopsis:       unwind.h build race condition
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          feedback
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Jun 12 08:25:01 +0000 2026
>Closed-Date:    
>Last-Modified:  Thu Jul 09 03:20:01 +0000 2026
>Originator:     Robert Elz
>Release:        NetBSD 11.99.6
>Organization:
	Spooky Bizarre Co-incidences Inc.
>Environment:
	Irrelevant, the actual problem came from the NetBSD build cluster.
>Machine:	any
>Description:

If this PR would be better in toolchain (or something else) rather than
lib, someone please move it.

I believe this issue has been observed before, but in the 2026-06-11 19:51:16
build of the "daily" snapshot builds of HEAD, see:

	https://releng.netbsd.org/cgi-bin/builds.cgi

but you'll need to look at that soon, if needed, as within a few days the
relevant entries will have vanished.

The i386 build failed (click on the build date, look at the faied builds,
and select the i386 failure) with:

ln: unwind.h: File exists
--- unwind.h ---

*** Failed target: unwind.h
*** In directory: /home/source/ab/HEAD/src/external/gpl3/gcc/lib/libgcc/libgcov
*** Failed commands:
	${_MKMSG} "symlink " ${.CURDIR:T}/${.TARGET}
	=> @# "symlink " libgcov/unwind.h
	rm -f ${.TARGET}
	=> rm -f unwind.h
	ln -s ${.ALLSRC} ${.TARGET}
	=> ln -s /home/source/ab/HEAD/src/external/gpl3/gcc/dist/libgcc/unwind-generic.h unwind.h
*** [unwind.h] Error code 1

[The sgimips build failed as well, as it did in the previous and subsequent builds,
but that is (was) just a sets list issue, and should be fixed 2 builds later, unless
I botched the change.]

The following build 2026-06-12 00:35:01 build had no similar problem, and the i386
port (and all others except sgimips) built just fine.  The previous build of 2026-06-11
14:11:52 also had no problems building i386 (just sgimips, all for the same reason).

Needless to say, there have been no commits around this time which could even with
the wildest imagination possibly have caused this error to appear & vanish again.

So (as I believe to be generally understood) this is a build system race condition
that no-one has fixed so far.

I believe I might understand what the problem is, and have a potential solution to
offer, but I am not going to (even attempt to) implement it, as it reqires Makefile
changes which I don't feel confident making (the subtleties of make's programming
language are beyond my limited abilities).   Or rather, there is one very crude
"big stick" change which I could make which will probably fix the issue, but that
one I don't believe is the best solution, and another ugly one which might also work.

I believe the issue is "make includes" in .../gcc/lib/libgcc/libgcov racing against
itself, believe it or not.

In src/Makefile we have:

	_SUBDIR=	tools .WAIT lib
	.if ${MKLLVM} != "no"
	_SUBDIR+=	external/bsd/compiler_rt
	.endif
	_SUBDIR+=	include external crypto/external bin
	_SUBDIR+=	games libexec sbin usr.bin
	_SUBDIR+=	usr.sbin share sys etc tests compat
	_SUBDIR+=	.WAIT rescue .WAIT distrib regress

and in that observe that after tools, up to before rescue, all that can run in parallel.

(That's the crude big stick fix that I could make, insert an extra .WAIT in there
at the right place, and I believe, issue solved).

The failing code above is from the descent into external (and below that).  The racing
version (the one that won the race in this case, and caused the above to fail) is,
I believe, in lib.   src/lib/Makefile contains:

	SUBDIR=		csu .WAIT

	.if (${MKGCC} != "no")
	SUBDIR+=	../external/gpl3/${EXTERNAL_GCC_SUBDIR}/lib/libgcc .WAIT
	.endif

	SUBDIR+=	libc
	SUBDIR+=	.WAIT

	#
	# The SUBDIRs above are included here for completeness but should be built
	# and installed prior to make(dependall) in this file, as libraries listed
	# below will depend on versions from DESTDIR only.
	#

	SUBDIR+=	i18n_module

(followed by lots more SUBDIR additions, but none of them are relevant).

Note that the .WAIT's in that file only affect its internal operation, and have
no effect at all upon the parallel descent into external.

The issue is:

	SUBDIR+=	../external/gpl3/${EXTERNAL_GCC_SUBDIR}/lib/libgcc .WAIT

which is running the same make operations as are being done, potentially at tge
exact same instant as, the descent into external from src/Makefile

If the two makes (when making includes) happen to be running these 2 lines
at the exact same time:

	rm -f ${.TARGET}
	ln -s ${.ALLSRC} ${.TARGET}

so both do the rm in parallel (harmless, one actually removes the file, if
it happens to exist, the other just sees it not existing), then both, at about
the same time attempt the "ln".  One of the two will succeed, the other will
fail, and when it does, the whole build shuts down.

Now another crude (and very poor) fix would be to just make it:

	-ln -s ${.ALLSRC} ${.TARGET}

or perhaps slightly better:

	ln -s ${.ALLSRC} ${.TARGET} || test -e ${.TARGET}

so it only fails if the reason for the ln failing is because ${.TARGET} exists
(which given the preceding rm command, can only be because some other part
of the build is creating it in parallel).

But while kind of appealing (I like doing stuff in sh code), I don't think that
is the ideal solution either (though maybe the second of those might work if my
actual proposed solution has a flaw I am unaware of).

What I'd prefer to do, if someone can get the Makefile syntax and actual names correct
is to change the .if which guards the build of libgcc in src/lib from:

	.if (${MKGCC} != "no")

to something which achieves what I mean by:

	.if (${MKGCC} != "no") && (${.TARGET} != "includes")

so when making the include files, this branch of the parallel make
doesn't reach out into libgcc at all, but simply allows the build in
external/... to do that.   Of course this will fail if someone were
to attempt to build lib alobe (cd src/lib; make)

And naturally, this depends upon there being suitable make syntax to achieve what
I'm suggesting here, and that the descent into external/... when making includes
will always get to libgcov (if MKGCC != no).

Note that the order in which the various include files are added by "make includes"
should be irrelevant, installing them cannot rationally depend upon which other
include files have been installed yet, just as long as all of them get installed,
sometime - and also "make includes" has to be complete before the make system can
start actually making dependencies, or compiling anything.

So, can someone with knowledge of the lib builds, gcc builds, and the make system
take a look at this, and perhaps implement something to make this problem go away,
once and for all ?

In this, I am assuming that the actual builds that happen inside external/.../lib/libgcc
(other than making the include files) are safe from races.  If that's not the case,
and if that run off into libgcc is really needed for the src/lib build, then perhaps
the big stick ".WAIT" added to src/Makefile, somewhere between lib and external
is the correct solution.

>How-To-Repeat:
	Watch the builds and wait, every so often it happens.  Pure (bad) luck.
>Fix:
	See above for potential ways forward.

>Release-Note:

>Audit-Trail:
From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: lib/60324: unwind.h build race condition
Date: Sat, 04 Jul 2026 03:18:18 +0700

 This has happened (at least) twice again since the PR was filed (the most
 recent being the 2026-07-02 04:37:48 UTC i386 build of HEAD from the build
 cluster)

 Unless someone who has a better idea how make & the build system works
 at least claims this PR with the intent to fix it within the next few
 days, I am going to implement one of my proposals - probably just
 shove an extra .WAIT into the Makefile.   Please, you really don't
 want me messing around in the build system makefiles, so someone find
 a good fix for this problem, and implement it!

 kre

From: "Robert Elz" <kre@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/60324 CVS commit: src
Date: Wed, 8 Jul 2026 04:10:14 +0000

 Module Name:	src
 Committed By:	kre
 Date:		Wed Jul  8 04:10:14 UTC 2026

 Modified Files:
 	src: Makefile

 Log Message:
 PR lib/60324 - isolate descent into lib

 Since no-one seems interested in implementing a better fix
 than this for the issue in the PR which (sometimes) breaks
 the builds, simply do the easiest possible thing which will
 avoid the observed symptoms.

 Don't allow builds in lib and external to happen in parallel
 by simply isolating the descent into lib (there is a LOT to
 do in there).   Don't start anything else until that is done.

 If anyone would like to find a different workable solution than
 this one, please go ahead.


 To generate a diff of this commit:
 cvs rdiff -u -r1.342 -r1.343 src/Makefile

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->feedback
State-Changed-By: kre@NetBSD.org
State-Changed-When: Wed, 08 Jul 2026 04:33:45 +0000
State-Changed-Why:
Potential fix installed in HEAD.   Since determining if it works or not
requires simply waiting (for a potentially unbounded period) to see if
the issue is observed again, allow this PR to languish in feedback state
until either the fix is determined to not work (problem observed again),
or until someone gets bored with this PR still being open in a decade or
two and decides to close it.


From: matthew green <mrg@eterna23.net>
To: gnats-bugs@netbsd.org, kre@munnari.OZ.AU
Cc: gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, martin@NetBSD.org
Subject: re: lib/60324: unwind.h build race condition
Date: Thu, 09 Jul 2026 11:11:26 +1000

 sorry, i'd missed this PR being submitted.

 i think the fix you commited is fine, and nicely found on what
 is likely going wrong (i'm convinced by your analysis.)


 maybe the real fix is to not enter this subdirectory twice in
 this same high-level .WAIT group at all.

 there are a few subdirs we traverse twice but perhaps we should
 stop doing that entirely.  eg, src/Makefile had (the first line
 would have a .WAIT now):

 _SUBDIR=3D        tools .WAIT lib
 _SUBDIR+=3D       include external crypto/external bin

 and src/external/Makefile has:

 SUBDIR+=3D amdgpu-firmware apache2 atheros broadcom bsd cddl gpl2 gpl3 his=
 torical

 and src/external/gpl3/Makefile has:

 SUBDIR+=3D        ${EXTERNAL_GCC_SUBDIR}

 etc., where it ends up entering libgcc from multiple parts of
 the same .WAIT segment.

 this isn't the only one, but maybe it's the only one that has
 a build-a-file conflict.  not doing this would mean we could
 take the new .WAIT away again.

 it might be nice to still allow this with some variable set,
 since eg, i will cleandir in src/external/gpl3/gcc and rebuild
 just that sometimes when i'm testing gcc stuff, but i'm fine
 for this to be a special thing you have to ask for.)


 thanks.


 .mrg.

From: Robert Elz <kre@munnari.OZ.AU>
To: matthew green <mrg@eterna23.net>
Cc: gnats-bugs@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: lib/60324: unwind.h build race condition
Date: Thu, 09 Jul 2026 10:17:04 +0700

     Date:        Thu, 09 Jul 2026 11:11:26 +1000
     From:        matthew green <mrg@eterna23.net>
     Message-ID:  <88.1783559486@splode.eterna23.net>

   | maybe the real fix is to not enter this subdirectory twice in
   | this same high-level .WAIT group at all.

 The issue with that, for me anyway, was still allowing users
 to cd to anywhere in the tree, and run "make" and have things
 work.   If building libc doesn't (when needed) get the correct
 bits from libgcc (or whatever it was doing), then just
 	cd src/lib/libc; make
 won't necessarily work.  It should.  And so should
 	cd src/external/gpl3/gcc; make
 so both of those need to (at least be able to) descend into the
 part of the tree where the issue occurred (libgcov).

   | this isn't the only one, but maybe it's the only one that has
   | a build-a-file conflict.

 It might just be because of using ln -- most other commands would just
 replace whatever is there, whereas ln fails if the dest exists, it
 could be switched to use ln -f, but I doubt that's sufficient,
 it still ends up doing a symlink() call, just after (yet another)
 unlink() perhaps, but the two ln(1)'s both doing an unlink() if needed,
 followed by a symlink() in parallel are going to fail in just the same
 way - perhaps slightly less often as the sync between the two would need
 to be tighter.

 One possibility I thought of, but didn't try, would be to make the
 ln line of the relevant make recipe be (something like)

 	ln -s ${.ALLSRC} ${.TARGET} || \
 		 { X=$(readlink ${.TARGET}) && [ "$X" = "${.ALLSRC}" ] ; }

 converted into the correct make syntax ($$X for $X and the cmdsub etc) and
 with someone who knows how checking that the uses of TARGET and ALLSRC are
 correct.

 I didn't attempt that, as I never found where the relevant recipe appears
 (somewhere in share/mk I'd assume - a cesspit I like to avoid), and I'm
 also not sure that a change there like this would be OK for every possible
 use (if ever used to link multiple files into a directory, this wouldn't
 work at all).

 It also would mean needing readlink as a tool, which I doubt we have now
 (might need to be a specialised version, making stat(1) (which is what our
 readlink(1) really is) into a tool could be a challenge, but a very simple,
 portable, readlink(1), with limited abilities, just enough for this special
 case, should be easy enough).

 Or we could toolize ln, and give it a new option, that if set, and an
 EEXIST error is reported, would readlink(2) the existing file, and if that
 works, and the result is the same as the link that was to be created, to
 just treat that as OK, and not fail (which is more or less doing the above,
 internal to ln).   (Might also want to do permission & ownership checks).

   | not doing this would mean we could take the new .WAIT away again.

 Perhaps.   I'm not sure that would make any real difference, the build
 time for a full set of builds doesn't seem to have been altered, and
 for all the builds I checked (where the info is available without needing
 to do it myself!) the number of lines in the build logfile didn't change
 out of the range that had existed from recent previous builds, so I don't
 think adding that .WAIT really harmed anything - I had half expected the
 builds to take a bit longer, but they don't seem to have.

 Of course, it is also possible that the .WAIT is largely a no-op, but from
 what I could tell from my own test builds when I inserted it, it did seem
 to make (for each phase: obj, includes, dependall) the lib builds to be
 complete before the rest of the tree started.    That's where I became
 confused, without the .WAIT there, what was stopping a dependall of (say)
 bin/cat from starting before the new lib/libc is available?   If there is
 (was) nothing preventing that, how does (did) anything actually work?  But
 clearly it all does, and did before, at least 99% of the time.

 kre

 ps: I am also still not convinced that the problem is fixed, it is likely
 that the cases I detected will no longer occur, but the same thing might
 be possible from a two parts of the tree not including "lib" - just perhaps
 even more rarely occurring.


>Unformatted:

Home	PR Database Search