NetBSD Problem Report #46750

From ef@math.uni-bonn.de  Sat Jul 28 17:54:08 2012
Return-Path: <ef@math.uni-bonn.de>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
	by www.NetBSD.org (Postfix) with ESMTP id 4194963B85F
	for <gnats-bugs@gnats.NetBSD.org>; Sat, 28 Jul 2012 17:54:08 +0000 (UTC)
Message-Id: <20120728175405.694D91BD42@trave.math.uni-bonn.de>
Date: Sat, 28 Jul 2012 19:54:05 +0200 (CEST)
From: ef@math.uni-bonn.de
Reply-To: ef@math.uni-bonn.de
To: gnats-bugs@gnats.NetBSD.org
Subject: make oddity expanding variables assigned to with +=
X-Send-Pr-Version: 3.95

>Number:         46750
>Category:       toolchain
>Synopsis:       make oddity expanding variables assigned to with +=
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    toolchain-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Jul 28 17:55:00 +0000 2012
>Closed-Date:    
>Last-Modified:  Tue May 10 09:10:01 +0000 2022
>Originator:     Edgar Fuß
>Release:        NetBSD 6.0_BETA2
>Organization:
	Mathematisches Institut der Uni Bonn
>Environment:


System: NetBSD trave.math.uni-bonn.de 6.0_BETA2 NetBSD 6.0_BETA2 (MI12serv) #3: Wed Jul 25 14:41:45 CEST 2012 support@trave.math.uni-bonn.de:/usr/obj/sys/arch/amd64/compile/mi12serv amd64
Architecture: x86_64
Machine: amd64
>Description:
	Variables that have been assigned to with += expand oddly.
	The variable's full expansion will start with a space, while
	the first element's expansion doesn't.
	Explicitly transforming to space separation (e.g. ":ts ")
	also makes the leading space disappear.
>How-To-Repeat:
	The following Makefile:

		x=
		x+= foo bar

		default:
			echo _${x}
			echo _${x:[1]}
			echo _${x:ts }

	outputs:

		echo _ foo bar
		_ foo bar
		echo _foo
		_foo
		echo _foo bar
		_foo bar

	So while asking for x's first element gives "foo" (not " foo"),
	x nevertheless expands to " foo bar" (not "foo bar").
	And explicitely asking the words to be separated by a space
	gives the expected result (no leading space).

	If you just assign to x (e.g. "x= foo bar"), it works as expected.
>Fix:


>Release-Note:

>Audit-Trail:
From: David Laight <david@l8s.co.uk>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: toolchain/46750: make oddity expanding variables assigned to with +=
Date: Sun, 29 Jul 2012 14:58:13 +0100

 This is the way make works.
 make variables hold character strings and have to maintain any
 embedded white space.
 In certain places the variable contents are treated as white-space
 separated tokens (usually filenames).

 Whitespace after = (or :=) can be removed, but whitespace
 after += must be kept - otherwise you can't build strings.

 	David

 -- 
 David Laight: david@l8s.co.uk

From: =?iso-8859-1?Q?Edgar_Fu=DF?= <ef@math.uni-bonn.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: toolchain/46750: make oddity expanding variables assigned to with +=
Date: Sun, 29 Jul 2012 16:20:31 +0200

 > This is the way make works.
 Then either the first element (e.g. :[1]) should be empty or start with =
 a space.
 You currently get different results from
 - asking for the whole value, which should be a space-separated list =
 (you get " foo bar")
 - asking explicitly for a space-separated list (you get "foo bar")
 - asking for the individual elements and space-joining them (you get =
 "foo bar")

 > Whitespace after =3D (or :=3D) can be removed, but whitespace
 > after +=3D must be kept - otherwise you can't build strings.
 The manpage says (for all variants of assignments):
 	Any white-space before the assigned value is removed=

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: toolchain/46750: make oddity expanding variables assigned to
 with +=
Date: Mon, 30 Jul 2012 07:26:24 +0000

 On Sun, Jul 29, 2012 at 02:05:03PM +0000, David Laight wrote:
  >  This is the way make works.
  >  make variables hold character strings and have to maintain any
  >  embedded white space.
  >  In certain places the variable contents are treated as white-space
  >  separated tokens (usually filenames).
  >  
  >  Whitespace after = (or :=) can be removed, but whitespace
  >  after += must be kept - otherwise you can't build strings.

 That's not true - try it.

 -- 
 David A. Holland
 dholland@netbsd.org

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: toolchain/46750: make oddity expanding variables assigned to
 with +=
Date: Mon, 30 Jul 2012 07:30:21 +0000

 On Sun, Jul 29, 2012 at 02:25:02PM +0000, Edgar Fu? wrote:
  >  > This is the way make works.
  >  Then either the first element (e.g. :[1]) should be empty or start with =
  >  a space.

 :[1] gives the first word; since words are delimited by whitespace,
 any leading whitespace is not part of the first word.

  >  You currently get different results from
  >  - asking for the whole value, which should be a space-separated list =
  >  (you get " foo bar")
  >  - asking explicitly for a space-separated list (you get "foo bar")
  >  - asking for the individual elements and space-joining them (you get =
  >  "foo bar")
  >  
  >  > Whitespace after =3D (or :=3D) can be removed, but whitespace
  >  > after +=3D must be kept - otherwise you can't build strings.
  >
  >  The manpage says (for all variants of assignments):
  >  	Any white-space before the assigned value is removed=

 It is.

 What it's doing is inserting a space between the old value of the
 variable and the appended material. This is what it's supposed to do;
 otherwise, if you did

 FOO=bar
 FOO+=baz

 you'd get "barbaz" instead of "bar baz".

 The problem is that it's doing this even if the old value is the empty
 string, so you get a leading space. Relatedly, if you do

 FOO=foo
 FOO+=
 FOO+=
 FOO+=

 foo:
 	echo /${FOO}/

 you get

 /foo   /

 which is also wrong.

 -- 
 David A. Holland
 dholland@netbsd.org

Responsible-Changed-From-To: toolchain-manager->dholland
Responsible-Changed-By: dholland@NetBSD.org
Responsible-Changed-When: Mon, 30 Jul 2012 07:35:59 +0000
Responsible-Changed-Why:
I've dealt with this code before


From: Edgar =?iso-8859-1?B?RnXf?= <ef@math.uni-bonn.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: toolchain/46750: make oddity expanding variables assigned to
 with +=
Date: Mon, 30 Jul 2012 11:47:27 +0200

 > The problem is that it's doing this even if the old value is the empty 
 > string, so you get a leading space.
 If we, for a moment, set aside how make internally handles lists (and 
 talk only about list elements not containing whitespace), the question is 
 what we expect "FOO=" to do.
 Either we expect it to set FOO to the empty list. Then, after "FOO+=foo", 
 the value should be "foo".
 Or we expect it to set FOO to a list containing an empty value. Then, 
 after "FOO+=foo", the value should indeed be " foo", but FOO:[1] should 
 return that empty value.

 > Relatedly, if you do
 Here, the question is whether we expect "FOO+=" to append an empty list 
 to FOO (e.g. don't change it) or whether we expext it to append an empty 
 value to the list.

 Probably that should be discussed on tech-toolchain and then made consistent.

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: toolchain/46750: make oddity expanding variables assigned to
 with +=
Date: Wed, 22 Aug 2012 07:10:37 +0000

 On Mon, Jul 30, 2012 at 09:50:05AM +0000, Edgar Fu? wrote:
  >  > The problem is that it's doing this even if the old value is the empty 
  >  > string, so you get a leading space.
  >
  >  If we, for a moment, set aside how make internally handles lists (and 
  >  talk only about list elements not containing whitespace), the question is 
  >  what we expect "FOO=" to do.
  >  Either we expect it to set FOO to the empty list. Then, after "FOO+=foo", 
  >  the value should be "foo".
  >  Or we expect it to set FOO to a list containing an empty value. Then, 
  >  after "FOO+=foo", the value should indeed be " foo", but FOO:[1] should 
  >  return that empty value.

 Right.

  >  > Relatedly, if you do
  >
  >  Here, the question is whether we expect "FOO+=" to append an empty list 
  >  to FOO (e.g. don't change it) or whether we expext it to append an empty 
  >  value to the list.

 Right.

  >  Probably that should be discussed on tech-toolchain and then made
  >  consistent.

 The historic behavior of make has always been that variables hold
 strings which are whitespace-separated lists of words. There are no
 empty values. I don't think this needs discussion (nor can it be
 changed without risking a lot of fallout and incompatibility...)

 -- 
 David A. Holland
 dholland@netbsd.org

State-Changed-From-To: open->feedback
State-Changed-By: rillig@NetBSD.org
State-Changed-When: Mon, 07 Feb 2022 23:11:18 +0000
State-Changed-Why:
Edgar, do you still think that '+=' semantics are broken?  I'm on
David's side on this topic, in that variable values are strings, and
in these, whitespace is relevant and is preserved.  While parsing an
assignment, whitespace around the value is removed though.

As soon as one of the split-into-words modifiers is applied to an
expression, the value of the variable is treated as a list of words, so
whitespace between the words becomes irrelevant and thus gets removed.
This is why an expression like ${:M*} normalizes whitespace in a list
expression.

I remember that for pkgsrc packages using GNU configure, pkglint warns:

// In GNU configure scripts, a few variables need to be passed through
// the :M* modifier before they reach the configure scripts. Otherwise
// the leading or trailing spaces will lead to strange caching errors
// since the GNU configure scripts cannot handle these space characters.

Do you have other practical examples where the space between words 
matters?


From: Edgar =?iso-8859-1?B?RnXf?= <ef@math.uni-bonn.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: toolchain/46750 (make oddity expanding variables assigned to
 with +=)
Date: Tue, 8 Feb 2022 11:22:01 +0100

 > I'm on David's side on this topic
 Which one of the two Davids involved?

 I now agree that make's behaviour is mostly OK as there are no lists, only strings that can be interpreted as space-seperated lists.

 There are, however, two cases (FOO+= ${BAR} where either FOO or BAR are empty) where one could argue that not inserting a space would be more rational. I've brought that up on tech-toolchain@.

 > Do you have other practical examples where the space between words matters?
 Given that that PR is nearly ten years old (thanks for picking it up anyway!): No.

State-Changed-From-To: feedback->open
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Wed, 06 Apr 2022 23:21:50 +0000
State-Changed-Why:
Feedback was received. Nobody responded to the tech-toolchain post, so it's
not clear what the path forward is. There seem to be two choices:

1. Do nothing; the current behavior is only wrong in the sense that empty
appends accumulate extra whitspace, which is unsightly but has no semantic
consequences. Also it's not clear how you generate strings with multiple
consecutive spaces in them any other way.

2. Fix append so that if either side of the append is empty it just copies
the other string instead of also inserting a space.

Inertia favors #1. I kind of prefer #2, but I'm no longer in a position to
handle it myself; besides not having time after all rillig's changes I'm
no longer familiar with the code.


Responsible-Changed-From-To: dholland->toolchain-manager
Responsible-Changed-By: dholland@NetBSD.org
Responsible-Changed-When: Wed, 06 Apr 2022 23:22:14 +0000
Responsible-Changed-Why:
probably cannot deal with this one myself at this point


From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: toolchain/46750 (make oddity expanding variables assigned to with +=)
Date: Thu, 07 Apr 2022 19:30:56 +0700

     Date:        Wed,  6 Apr 2022 23:21:51 +0000 (UTC)
     From:        dholland=40NetBSD.org
     Message-ID:  <20220406232151.63B331A923B=40mollari.NetBSD.org>

   =7C 2. Fix append so that if either side of the append is empty it just=
  copies
   =7C the other string instead of also inserting a space.
   =7C
   =7C Inertia favors =231. I kind of prefer =232

 FWIW, what POSIX says (or perhaps will say, there are lots of changes
 in this area, and I'm not really keeping track of what is current, and
 what is new).

 	The following form (the append form) appends additional text
 	to the value of a macro:

 		string1 +=3D =5Bstring2=5D

 	When using the append form:

 	    =B7 If the macro named by string1 does not exist, this form shall
 	      be equivalent to the delayed-expansion form

        		string1 =3D =5Bstring2=5D

 That would be your =232 (or part of it).

 	    =B7 If the macro named by string1 exists and is an
   	      immediate-expansion macro, then a <space> or <tab> character
 	      followed by the evaluation of string2 shall be appended to
 	      the value currently assigned to the macro named by string1.

 	    =B7 If the macro named by string1 exists and is a delayed-expansion
 	      macro, then a <space> or <tab> character followed by the
 	      unevaluated string2 shall be appended to the value currently
 	      assigned to the macro named by string1.

 Nothing in those about =22unless nothing would be appended=22 where the <=
 space>
 (or tab) is inserted, so this case would be your =231.

 Not sure whether there's a distinction (wrt string1) between =22doesn't e=
 xist=22
 and =22exists with an empty value=22.

 Kind of unrelated to this point, but it also says:

 	The value of string2 is defined as all characters from the first
 	non-<blank> character, if any, after the <equals-sign>, up to but
 	not including a comment character ('=23') or an unescaped <newline>.

 which implies that trailing white space (any amount) is included.

 kre

From: Edgar =?iso-8859-1?B?RnXf?= <ef@math.uni-bonn.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: toolchain/46750 (make oddity expanding variables assigned to
 with +=)
Date: Thu, 7 Apr 2022 16:21:53 +0200

 >  Kind of unrelated to this point, but it also says:
 >  
 >  	The value of string2 is defined as all characters from the first
 >  	non-<blank> character, if any, after the <equals-sign>, up to but
 >  	not including a comment character ('=23') or an unescaped <newline>.
 >  
 >  which implies that trailing white space (any amount) is included.
 Which bmake fourtunately doesn't adhere to. I doubt you want
 	foo= bar	# see section 47.11
 to assign trailing whitespace to foo.

From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: toolchain/46750 (make oddity expanding variables assigned to with +=)
Date: Thu, 07 Apr 2022 22:56:38 +0700

     Date:        Thu,  7 Apr 2022 14:25:02 +0000 (UTC)
     From:        Edgar =?iso-8859-1?B?RnXf?= <ef@math.uni-bonn.de>
     Message-ID:  <20220407142502.4D98E1A923B@mollari.NetBSD.org>

   |  Which bmake fourtunately doesn't adhere to. I doubt you want
   |  	foo= bar	# see section 47.11
   |  to assign trailing whitespace to foo.

 Yet gmake and smake both do include it.

 On the other hand while with

 foo=
 foo += data       #comment

 all:
 	printf '<%s>\n' "$(foo)"

 gmake (like bmake) inserts a leading space from the += smake
 doesn't.

 kre

From: Roland Illig <roland.illig@gmx.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: toolchain/46750
Date: Tue, 10 May 2022 02:20:34 +0200

 Robert Elz wrote:
  > Edgar Fu=C3=9F wrote:
  > |  Which bmake fourtunately doesn't adhere to. I doubt you want
  > |  	foo=3D bar	# see section 47.11
  > |  to assign trailing whitespace to foo.
  >
  > Yet gmake and smake both do include it.

 They don't. All of bmake, gmake and smake discard the whitespace between
 'bar' and '#'.

 https://pubs.opengroup.org/onlinepubs/9699919799/utilities/make.html#tag_2=
 0_76_13_05

 Even though POSIX requires that the macro value includes trailing
 whitespace, I doubt that bmake or gmake will ever adhere to that
 specification. It's rather the POSIX specification that should be
 adjusted to reflect existing behavior.

 Roland

From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: toolchain/46750
Date: Tue, 10 May 2022 15:41:06 +0700

     Date:        Tue, 10 May 2022 00:25:01 +0000 (UTC)
     From:        Roland Illig <roland.illig@gmx.de>
     Message-ID:  <20220510002501.C1E6E1A923C@mollari.NetBSD.org>

   |   > |  Which bmake fourtunately doesn't adhere to. I doubt you want
   |   > |  	foo=3Dbar	# see section 47.11
   |   > |  to assign trailing whitespace to foo.
   |   >
   |   > Yet gmake and smake both do include it.
   |  =

   |  They don't. All of bmake, gmake and smake discard the whitespace
   |  between 'bar' and '#'.

 Really??   Given /tmp/mf containing:


 FOO=3DText      # and a comment

 all:
 	printf '<%s>\n' '$(FOO)'


 (4 lines, first line is FOO=3D  Last line is printf)

 What I see is (this is NetBSD so "make" is bmake):

 jinx$ make -f /tmp/mf
 printf '<%s>\n' 'Text'
 <Text>
 jinx$ gmake -f /tmp/mf
 printf '<%s>\n' 'Text      '
 <Text      >
 jinx$ smake -f /tmp/mf
 ...printf '<%s>\n' 'Text      '
 <Text      >
 jinx$ gmake --version
 GNU Make 4.1
 Built for x86_64--netbsd
 [noise deleted]
 jinx$ smake --version
 Smake release 1.3 2020/03/30 (amd64-unknown-netbsd9.99.74) Copyright (C) 1=
 985, 87, 88, 91, 1995-2020 J=EF=BF=BDg Schilling
   =

   |  Even though POSIX requires that the macro value includes trailing
   |  whitespace,

 POSIX requires it because that is what other make versions do.
 Not necessarily because anyone finds it particularly desireable.

 It is possible that later gmake (and less likely, smake) might
 have changed thus, but as tgat would break backwards compat
 and make them drop out of conformance in this area, I somehow
 doubt it.

 kre

>Unformatted:

Home	PR Database Search