NetBSD Problem Report #50834

From kre@munnari.OZ.AU  Sun Feb 21 23:07:16 2016
Return-Path: <kre@munnari.OZ.AU>
Received: from mail.netbsd.org (mail.NetBSD.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id CEFB97AC07
	for <gnats-bugs@www.NetBSD.org>; Sun, 21 Feb 2016 23:07:16 +0000 (UTC)
Message-Id: <201602212306.u1LN6kSg022268@andromeda.noi.kre.to>
Date: Mon, 22 Feb 2016 06:06:46 +0700 (ICT)
From: kre@munnari.OZ.AU
To: gnats-bugs@www.NetBSD.org
Subject: /bin/sh incorrectly treats ${unset_or_null_var:-} as ''
X-Send-Pr-Version: 3.95

>Number:         50834
>Category:       bin
>Synopsis:       /bin/sh incorrectly treats ${unset_or_null_var:-} as ''
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    bin-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Feb 21 23:10:00 +0000 2016
>Closed-Date:    Mon Apr 04 12:38:18 +0000 2016
>Last-Modified:  Mon Apr 04 12:38:18 +0000 2016
>Originator:     Robert Elz
>Release:        NetBSD 7.99.26
>Organization:
	Prince of Songkla University
>Environment:
System: NetBSD andromeda.noi.kre.to 7.99.26 NetBSD 7.99.26 (VBOX64-1.1-20160128) #43: Thu Jan 28 16:09:08 ICT 2016 kre@onyx.coe.psu.ac.th:/usr/obj/current/kernels/amd64/VBOX64 amd64
Architecture: x86_64
Machine: amd64
>Description:
	When expanding (unquoted) ${N} an unset or null value for N
	will result in nothing being produced, so

		N=
		set -- a ${N} b; echo $# ; 		# echoes 2

	Of course, if quoted:
		set -- a "${N}" b; echo $# ;		# echoes 3

	That's all good.

	But
		set -- a ${N:-} b; echo $# ;		# echoes 3

	This is broken.

	[Aside: note, if N is set, but null,
		N=; set -- a ${N-} b; echo $# ;		# echoes 2
	that is interpolating the value of N rather than the empty word,
	and that's nothing, so that's what is (correctly) produced.

	This really doesn't matter all that much, as ${N:-} is
	identical to ${N} in meaning (if N is set and non-null, it is
	the value of N, otherwise it is nothing, which is what $N iis)
	so there's no reason for anyone to ever use this form.

	Fortunately, the case that does matter:

		N=; E=; set -- a ${N:-${E}}; echo $#	# echoes 2

	If that one failed, this would need fixing, as that's a
	useful construct, unlike the bare ${N:-}

>How-To-Repeat:
	As above.

	I noticed this as there is (was - or will be was soonish)
	an ATF sh test that explicitly tested for unquoted ${N:-}
	producing a null string, rather than nothing, and failed
	if that was not true....

	That test is about to become an expected failure...

>Fix:
	I'm not sure I will bother, this is just so meaningless...

	This PR is just so there is a number to put in the
	ATF failure message so it is possible for people who see
	the expected failure to discover exacly what is going on,
	and why it isn't fixed.

	Of course, if there were to be a great demand to make ${N:-}
	work properly, I might reconsider this, but for now there
	are more important sh bugs that need fixing (IMO).

>Release-Note:

>Audit-Trail:
From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: bin/50834 (/bin/sh incorrectly treats ${unset_or_null_var:-} as '')
Date: Sat, 05 Mar 2016 03:04:34 +0700

 For the record (which should be no surprise given the implementation)
 the same is true of ${x+} if x is set.

 And probably related, so I am going to include it in this PR,
 the shell expands the arg to ':' in...

 	set --; :	$@""

 as nothing, rather than as a null string as it should.  It does however
 do the right thing with

 	set --; : ""$@

 (to be observable, you'd want a different test case than this one of course).

 kre

From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: bin/50834 (/bin/sh incorrectly treats ${unset_or_null_var:-} as '')
Date: Sat, 05 Mar 2016 12:59:15 +0700

 I tried to send this message (a message like this) soon after the previous
 one - it arrived at NetBSD's mailer, but after that, no idea (maybe I
 made a typo in the addresses somewhere).

 The point was that in the previous message, I reversed the test
 cases, $@"" is the one that works, and ""$@ is the one that doesn't.

 The cause turns out to be the output format of the parser in the
 shell - the parser appears to process everything correctly, but the
 output is "quoted" var @ or var @ "quoted" (with a null string, ie: nothing,
 after "quoted" in both cases.)   The problem is that "$@" also produces
 "quoted" var @.   Here the "var @" is the operand of "quoted".  That one
 is a very special case, if $@ is empty, the "$@" simply vanishes when it
 is evaluated.  Since the evaluation routine cannot tell the difference
 between ""$@ and "$@" because they look just the same in the intermediate
 form, ""$@ also vanishes when $@ is empty.

 The intermediate form has an "endquote" operator, but it is used rarely.
 I suspect that the fix for this is going to have it used more frequently.

 And after all this, this problem turns out to have nothing to do with the
 subject of the PR, which doesn't involve quotes at all (if quotes were
 present, anywhere that is legal according to the syntax, a null string would
 be the correct result.)   I am getting a handle on that bug as well, but it
 turns out to be a devious one to locate...   I know I said in the PR that
 I didn't think I would bother looking, as ${N} and ${N-} both produce the
 same output, so the form with the '-' seems useless - but it was pointed
 out to me that the ${N-} form is the correct way to refer to a variable that
 might be unset, if it is possible that the -u (nounset) might be set.
 ${N} would generate an error, ${N-} produces the same output, regardless
 of the value of the nounset option.  So, this form is not as useless as I
 expected.   It helps that FreeBSD's shell (which shares much of its history
 with ours) has these bugs fixed - but they have changed so much more than
 we have, that finding just what fixed it is not trivial.   They also have
 bugs that we don't (any more) so simply using their code is also not the
 best solution.

 kre

State-Changed-From-To: open->closed
State-Changed-By: wiz@NetBSD.org
State-Changed-When: Mon, 04 Apr 2016 12:38:18 +0000
State-Changed-Why:
Fixed, thanks!


>Unformatted:

Home	PR Database Search