NetBSD Problem Report #34934
From dholland@eecs.harvard.edu Sat Oct 28 22:26:18 2006
Return-Path: <dholland@eecs.harvard.edu>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by narn.NetBSD.org (Postfix) with ESMTP id A5D6963B8CA
for <gnats-bugs@gnats.NetBSD.org>; Sat, 28 Oct 2006 22:26:18 +0000 (UTC)
Message-Id: <20061028222519.038085426@weatherwax.eecs.harvard.edu>
Date: Sat, 28 Oct 2006 18:25:18 -0400 (EDT)
From: dholland@eecs.harvard.edu
Reply-To: dholland@eecs.harvard.edu
To: gnats-bugs@NetBSD.org
Subject: make sometimes fails to handle multiple targets on the left correctly
X-Send-Pr-Version: 3.95
>Number: 34934
>Category: bin
>Synopsis: make sometimes fails to handle multiple targets on the left correctly
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: rillig
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sat Oct 28 22:30:00 +0000 2006
>Closed-Date: Sun May 15 09:03:56 +0000 2022
>Last-Modified: Sun May 15 09:03:56 +0000 2022
>Originator: David A. Holland
>Release: NetBSD 4.99.3 (20061018)
>Organization:
Moderately organized.
>Environment:
System: NetBSD weatherwax 4.99.3 NetBSD 4.99.3 (WEATHERWAX) #2: Wed Oct 18 18:55:11 EDT 2006 dholland@weatherwax:/usr/src/sys/arch/i386/compile/WEATHERWAX i386
Architecture: i386
Machine: i386
/usr/bin/make:
$NetBSD: crt0.c,v 1.16 2006/05/17 17:08:54 christos Exp $
$NetBSD: arch.c,v 1.52 2006/10/15 08:38:21 dsl Exp $
$NetBSD: buf.c,v 1.19 2005/08/08 16:42:54 christos Exp $
$NetBSD: compat.c,v 1.63 2006/10/15 08:38:21 dsl Exp $
$NetBSD: cond.c,v 1.34 2006/10/15 08:38:21 dsl Exp $
$NetBSD: dir.c,v 1.50 2006/10/15 08:38:21 dsl Exp $
$NetBSD: for.c,v 1.23 2006/10/15 08:38:21 dsl Exp $
$NetBSD: hash.c,v 1.16 2005/08/04 00:20:12 christos Exp $
$NetBSD: job.c,v 1.123 2006/10/15 08:38:21 dsl Exp $
$NetBSD: main.c,v 1.133 2006/10/15 08:38:21 dsl Exp $
$NetBSD: make.c,v 1.66 2006/10/15 08:38:22 dsl Exp $
$NetBSD: parse.c,v 1.118 2006/10/15 21:17:27 dsl Exp $
$NetBSD: str.c,v 1.25 2006/08/11 19:11:00 christos Exp $
$NetBSD: suff.c,v 1.55 2006/10/15 08:38:22 dsl Exp $
$NetBSD: targ.c,v 1.43 2006/10/15 08:38:22 dsl Exp $
$NetBSD: trace.c,v 1.7 2006/01/04 21:35:44 dsl Exp $
$NetBSD: var.c,v 1.114 2006/10/15 08:38:22 dsl Exp $
$NetBSD: util.c,v 1.39 2005/08/08 16:42:54 christos Exp $
$NetBSD: lstAppend.c,v 1.10 2004/05/07 00:04:41 ross Exp $
$NetBSD: lstAtEnd.c,v 1.11 2005/02/16 15:11:53 christos Exp $
$NetBSD: lstAtFront.c,v 1.11 2005/02/16 15:11:53 christos Exp $
$NetBSD: lstClose.c,v 1.10 2004/05/07 00:04:41 ross Exp $
$NetBSD: lstConcat.c,v 1.14 2005/08/08 16:42:54 christos Exp $
$NetBSD: lstDatum.c,v 1.10 2004/05/07 00:04:41 ross Exp $
$NetBSD: lstDeQueue.c,v 1.11 2005/02/16 15:11:53 christos Exp $
$NetBSD: lstDestroy.c,v 1.14 2005/08/09 21:36:42 christos Exp $
$NetBSD: lstDupl.c,v 1.13 2005/08/09 21:36:42 christos Exp $
$NetBSD: lstEnQueue.c,v 1.11 2005/02/16 15:11:53 christos Exp $
$NetBSD: lstFind.c,v 1.12 2005/02/16 15:11:53 christos Exp $
$NetBSD: lstFindFrom.c,v 1.11 2004/05/07 00:04:41 ross Exp $
$NetBSD: lstFirst.c,v 1.10 2004/05/07 00:04:41 ross Exp $
$NetBSD: lstForEach.c,v 1.11 2004/05/07 00:04:41 ross Exp $
$NetBSD: lstForEachFrom.c,v 1.11 2004/05/07 00:04:41 ross Exp $
$NetBSD: lstInit.c,v 1.10 2004/05/07 00:04:41 ross Exp $
$NetBSD: lstInsert.c,v 1.10 2004/05/07 00:04:41 ross Exp $
$NetBSD: lstIsAtEnd.c,v 1.10 2004/05/07 00:04:41 ross Exp $
$NetBSD: lstIsEmpty.c,v 1.10 2004/05/07 00:04:41 ross Exp $
$NetBSD: lstLast.c,v 1.10 2004/05/07 00:04:41 ross Exp $
$NetBSD: lstMember.c,v 1.10 2004/05/07 00:04:41 ross Exp $
$NetBSD: lstNext.c,v 1.10 2004/05/07 00:04:41 ross Exp $
$NetBSD: lstOpen.c,v 1.10 2004/05/07 00:04:41 ross Exp $
$NetBSD: lstRemove.c,v 1.12 2005/08/08 16:42:54 christos Exp $
$NetBSD: lstReplace.c,v 1.10 2004/05/07 00:04:41 ross Exp $
$NetBSD: lstSucc.c,v 1.11 2004/05/07 00:04:41 ross Exp $
>Description:
If you have two (or presumably more) targets on the left hand side of
a rule, and they're files that aren't in the current directory, make
will sometimes fail to detect that they got built together and run the
recipe twice, once "for" each target.
If the recipe is in some way generative, this can result in a broken
build; e.g. if you generate matching .c and .h files and this happens
twice, some parts of a build can end up including one version of the
header file and some the other, which in obscure cases could
conceivably be lethal.
>How-To-Repeat:
Use the included simple example, which has a simple code generator
that generates a source and header file into a subdirectory.
Unpack the example:
tar -xvzf example.tgz
cd example
Compile it:
make
This does
gdir/gen.sh gdir/gen
cc -Igdir -c src.c
cc -Igdir -c gdir/gen.c
cc src.o gen.o -o prog
Note that it runs gen.sh once.
Now, having built it, do this:
touch gdir/gen.sh
make
This causes
gdir/gen.sh gdir/gen
cc -Igdir -c src.c
gdir/gen.sh gdir/gen
cc -Igdir -c gdir/gen.c
cc src.o gen.o -o prog
Note that gen.sh gets run twice and the two source files in the
program are compiled with different copies of gen.h.
I haven't contrived this example so the resulting program breaks
(maybe I should have) but it's clearly not supposed to happen this
way.
Example follows:
begin 644 example.tgz
M'XL(`!K50T4``^U8ST\3012>`@J,)B(G$P\^`9,6PS*[;1=3@0@5B`8TL?'F
M9=D=V(W;W69W40Q!#YR\^0=YTXO_@U<3#QST;H(S^Z,=VB*:T*)QOJ:=>6_>
MSGLSK]^;:>FN46^X%/42A)3('"&LG=/ULL;:&%D;]U52ULA<62O.E9E>U9B,
MRCV-*L5.&!D!`+)LWW4-SSK)[J5-J=N/@/H+FN1_=MMR@E[Y^+/\EWC^BZ2H
MROSW`6+^9[>II_AG[X.HA.@GYU\MM?*OZEK,_Q+//SG[4#KQG^?_S<KZ:BZ7
M:\HY-(ARPOC[M"W%GWF$T3!Z^O;+P>'(P>%EF^N^_C@Z.CKX./;I@TU=U\<(
MK56K%<BO/7I:@*+"7I!_1*/EVGWP-HN@\6K`<EU`2`E?U2-CD[51D+1VU@NH
MJT1T-T**940&4C;#D"G]1##]>IUZT6^O\4:\+H0N"FNY)HP/I>UU]AX5]-\&
M$!IC[4CZ_%#:OY7*@ZG=>IN_;+[;J?UI=B2=;T"P&Q;LLGS<$?H<-<0STFEW
M5?#)L<?>2UWL1)^Q7R:\0\D>#*?QC:6R.-]G9C?>9;YVB/HA]/U('!ML>VKP
MF`<N#[7)%]KDB\T]XKD<9_-=$L9YW(C7,Y-_HD;@>-$64V$>%YOJ"FL'1DX(
M_#]"1_T/[3/WP>O_K\[_HJJUSG]=C^N_QG2R_O<>DS=G-QUO-K3Q)"3IAQD(
M'?Z=@(9KF)1MBT4#V/(#,,#T+<K-:&!$?H"Q:42P"%-[ZKYBP_S\RN-5_,)W
M+&Z2YYW"7<QUHIV9VDTZGNGNL.GFP\AR?,5>;*DF>"3V1-M<L(<!(&%R?B(^
M;)YY$\S%?NSDO'?RWT3&_S`P6:GL#4ZY__'BT.0_*1;C^[]*)/_[@0[.848O
MJ!N.ET\)Q_G'2,9Z`8UV`@\(9]QYQRUQ-LCXOV$\IUM.;_X(.HW_Y9+>.O\U
M+>:_KA')_SZ@]J1:6XB+/S2O@"9^O/PPT?KQI<#'N%I=,$U<75U?6JLMS#S@
MMA@W`G^[`E-Y;E[`HU/Y:K60B3#CP]0]C%NSMAS8%1#NFWA4$)H#[$GNN2*&
MU?20Q,%\'(L:QQ%7(%Y.5^-D!&<U3[%H@WH6QDE;Z?+,Q@:3^"85V`6F:6^Z
MU/"8>5"'F2W@VP!=UPG3;`>G7__%U3+C?[JTGO@X]?S7RRW^EY/__]@/`,G_
H/D"DC/#%[22?.'C>04M(2$A(2$A(2$A(2$A(2/P1?@+?4`\4`"@``-@/
`
end
>Fix:
I don't have time to look into it right now, but I'll put it on my list...
(As a workaround to the broken build problem, one can always run make
repeatedly.)
>Release-Note:
>Audit-Trail:
From: David Laight <david@l8s.co.uk>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: bin/34934: make sometimes fails to handle multiple targets on the left correctly
Date: Sun, 29 Oct 2006 07:18:35 +0000
On Sat, Oct 28, 2006 at 10:30:00PM +0000, dholland@eecs.harvard.edu wrote:
> >Number: 34934
> >Category: bin
> >Synopsis: make sometimes fails to handle multiple targets on the left correctly
...
> >Description:
>
> If you have two (or presumably more) targets on the left hand side of
> a rule, and they're files that aren't in the current directory, make
> will sometimes fail to detect that they got built together and run the
> recipe twice, once "for" each target.
....
If you have:
a1 a1: b
echo $@
The make runs the rules twice, once for a1 and once for a2.
This is the correct and required behaviour.
(Otherwise the above could't output 'a1' and 'a2'.)
If you have something that generates two files then you have to
do something else (eg have one of the files depend on the other one).
David
--
David Laight: david@l8s.co.uk
From: dholland@eecs.harvard.edu (David Holland)
To: gnats-bugs@NetBSD.org
Cc: gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
dholland@eecs.harvard.edu
Subject: Re: bin/34934: make sometimes fails to handle multiple targets on the left correctly
Date: Sun, 29 Oct 2006 13:29:39 -0500 (EST)
> If you have:
>
> a1 a1: b
^2
> echo $@
>
> The make runs the rules twice, once for a1 and once for a2.
> This is the correct and required behaviour.
> (Otherwise the above could't output 'a1' and 'a2'.)
Yes.
It's not a matter of what the rules mean; it's a matter of when
timestamps can and cannot be cached. If you have
a1 a2: b
echo foo > a1; echo bar > a2
then after running the recipe make is required to check and ascertain
that both a1 and a2 have been updated. Then it won't run the recipe a
second time. Whereas if you don't list both a1 and a2 on the left make
is allowed to assume that the one not listed isn't modified, and this
can cause things to go wrong later. (Lying to make rarely goes
unpunished.)
You'll note that make does actually do this; the problem is that
something breaks some of the time if a1 and a2 are in another
directory.
--
- David A. Holland / dholland@eecs.harvard.edu
From: Roland Illig <roland.illig@gmx.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: toolchain/34934
Date: Sat, 12 Feb 2022 01:18:57 +0100
David A. Holland wrote on 29 Oct 2006:
> It's not a matter of what the rules mean
I disagree. It's exactly a matter of what the rules mean. From your
description, you seem to think that "target1 target2: source" means that
this is a single rule that builds 2 targets.
Unfortunately,
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/make.html#tag_2=
0_76_13_04
talks a lot about the syntax of the dependency declarations but misses
to mention explicitly that each target is handled separately, there are
no groups of targets. The manual page for make(1) is as silent as POSIX
in this regard.
NetBSD make and GNU make agree on how to interpret the dependency
"target1 target2: source" though. They both treat the targets as
independent. Therefore I don't see any need to change the code. It
would be good to mention this independence in the documentation and ask
POSIX to add it to the specification for their next issue.
David A. Holland wrote on 29 Oct 2006:
> after running the recipe make is required to check and
> ascertain that both a1 and a2 have been updated
If we assume that a1 and a2 are independent (as GNU make and NetBSD make
treat them), why would make be required to check that both files have
been updated?
Responsible-Changed-From-To: bin-bug-people->rillig
Responsible-Changed-By: rillig@NetBSD.org
Responsible-Changed-When: Sat, 12 Feb 2022 00:21:09 +0000
Responsible-Changed-Why:
I'll handle this one.
From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: toolchain/34934
Date: Mon, 14 Feb 2022 21:32:50 +0000
On Sat, Feb 12, 2022 at 12:20:02AM +0000, Roland Illig wrote:
> David A. Holland wrote on 29 Oct 2006:
> > It's not a matter of what the rules mean
>
> I disagree. It's exactly a matter of what the rules mean. From your
> description, you seem to think that "target1 target2: source" means that
> this is a single rule that builds 2 targets.
Unfortunately, you are wrong.
I "seem to think" that because it's true: rules of the form "target1
target2: source" have always been used for this purpose, as well as
*also* for rules that build several targets independently.
Traditionally, this works because a non-parallel make sees the rule,
says "oh, target1 is out of date", runs the recipe, and then
- if target2 is now up to date, it doesn't need to run it again (and
doesn't)
- if target2 does not exist, it runs it again for target2
This does not work correctly for parallel make and that's been a
long-standing problem. Several solutions have been proposed but none
have been adopted. I think there's another PR for it, but whether or
not that's the case it's a relatively well-known issue.
*This* PR is about the sequential form breaking sometimes for targets
in other directories. As I pointed out in 2006, by default it works
correctly. (If by chance you have broken this recently, please fix
it.)
viz.:
valkyrie% cat Makefile
all: foo bar
foo bar: baz
touch foo bar
valkyrie% touch baz
valkyrie% make
touch foo bar
valkyrie%
Note that gmake behaves the same way:
valkyrie% touch baz
valkyrie% gmake
touch foo bar
valkyrie%
> Unfortunately,
> [irrelevant]
> David A. Holland wrote on 29 Oct 2006:
> > after running the recipe make is required to check and
> > ascertain that both a1 and a2 have been updated
>
> If we assume that a1 and a2 are independent (as GNU make and NetBSD make
> treat them), why would make be required to check that both files have
> been updated?
Because that's how it's supposed to work.
--
David A. Holland
dholland@netbsd.org
From: Roland Illig <roland.illig@gmx.de>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: toolchain/34934
Date: Sat, 7 May 2022 12:40:24 +0200
Am 14.02.2022 um 22:35 schrieb David Holland:
> *This* PR is about the sequential form breaking sometimes for targets
> in other directories. As I pointed out in 2006, by default it works
> correctly. (If by chance you have broken this recently, please fix
> it.)
Thanks for the detailed explanation.
Do you see any chance of making this "sometimes" or "in some cases" more
specific?
When you say "sometimes", does this mean that when running the same
scenario 20 times in a row, 3 of them might fail and the other 17 might
succeed?
When you say "in some cases", does this mean that the behavior is then
either "always fail" or "always succeed", but no mixture?
Could it have something to do with .PATH, or is it enough to have
subdir/file?
I ran the following loop for several minutes in the example directory
that you provided; it always generated consistent output.
make clean
make > good.out
while :; do
make -s clean
make > new.out
cmp -s new.out good.out || break
printf .
done
Any other ideas to reproduce the wrong behavior?
Roland
From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: toolchain/34934
Date: Sun, 15 May 2022 08:55:23 +0000
On Sat, May 07, 2022 at 10:45:02AM +0000, Roland Illig wrote:
> Am 14.02.2022 um 22:35 schrieb David Holland:
> > *This* PR is about the sequential form breaking sometimes for targets
> > in other directories. As I pointed out in 2006, by default it works
> > correctly. (If by chance you have broken this recently, please fix
> > it.)
>
> Thanks for the detailed explanation.
>
> Do you see any chance of making this "sometimes" or "in some cases" more
> specific?
Not without doing a bunch of experiments. I have long forgotten what I
was doing in 2006 that triggered the initial report :-)
Reading between the lines of what I originally wrote, which does (or
did then) exhibit both behaviors, and looking at the example, the
difference is triggered by touching the generator script; that is, in
the case where it worked, the dep from the output files to the
generator was satisfied, and where it didn't, it was out of date
before the recipe was run the first time (but not before it was run
the second time).
So,
> When you say "sometimes", does this mean that when running the same
> scenario 20 times in a row, 3 of them might fail and the other 17 might
> succeed?
I think it meant "depending on the mtime state of the files involved"
as opposed to "nondeterministically". Very little in non-parallel make
is nondeterministic :-)
I have figured that the problem was somewhere in the (then, at least)
rudimentary directory caching layer, something maybe like not flushing
cached mtimes of files in other directories.
However...
...it no longer behaves according to the description, that is, it
works in both cases.
I also vaguely recall something in this context involving ".."
directories, but I just tried it with ../gdir/ insted of gdir/ and it
still works.
Therefore, I'm inclined to think it's fixed.
> Could it have something to do with .PATH, or is it enough to have
> subdir/file?
Very unlikely; I don't ordinarily write makefiles with .PATH (since
long before 2006) so I wouldn't have hit any related problems.
--
David A. Holland
dholland@netbsd.org
State-Changed-From-To: open->closed
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Sun, 15 May 2022 09:03:56 +0000
State-Changed-Why:
No longer reproducible.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.46 2020/01/03 16:35:01 leot Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2020
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home