NetBSD Problem Report #37493

From martin@duskware.de  Thu Dec  6 21:50:04 2007
Return-Path: <martin@duskware.de>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by narn.NetBSD.org (Postfix) with ESMTP id 8636563BA9A
	for <gnats-bugs@gnats.netbsd.org>; Thu,  6 Dec 2007 21:50:04 +0000 (UTC)
Message-Id: <20071206214739.E84EE63B946@narn.NetBSD.org>
Date: Thu,  6 Dec 2007 21:47:39 +0000 (UTC)
From: rillig@NetBSD.org
Reply-To: rillig@NetBSD.org
To: netbsd-bugs-owner@NetBSD.org
Subject: /bin/sh exits if shift fails, but shouldn't
X-Send-Pr-Version: www-1.0

>Number:         37493
>Category:       bin
>Synopsis:       /bin/sh exits if shift fails, but shouldn't
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    bin-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Dec 06 21:55:01 +0000 2007
>Closed-Date:    Sun Feb 21 18:39:21 +0000 2016
>Last-Modified:  Sun Feb 21 18:39:21 +0000 2016
>Originator:     Roland Illig
>Release:        NetBSD 4.99.30
>Organization:
>Environment:
>Description:
When "shift" is used in "set -e" mode, the shell exits if the "shift"
occurs in a conditional statement. This is unexpected and not covered by
the documentation.


>How-To-Repeat:
$ sh -c 'set -ex; shift || echo failed.'
+ shift
shift: can't shift that many

$ bash -c 'set -ex; shift || echo failed.'
+ shift
+ echo failed.
failed.

>Fix:

>Release-Note:

>Audit-Trail:
From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: bin/37493: /bin/sh exits if shift fails, but shouldn't
Date: Sun, 25 May 2008 07:01:31 +0000

 On Thu, Dec 06, 2007 at 09:55:01PM +0000, rillig@NetBSD.org wrote:
  > When "shift" is used in "set -e" mode, the shell exits if the "shift"
  > occurs in a conditional statement. This is unexpected and not covered by
  > the documentation.
  > 
  > 
  > >How-To-Repeat:
  > $ sh -c 'set -ex; shift || echo failed.'
  > + shift
  > shift: can't shift that many

 So, the issue is that on line 1039 of eval.c there's a test that
 re-raises the exception (EXERROR, aka "error") if the command was a
 special builtin. This gets caught in main() and results in _exit(2),
 whether or not -e is used; a bad shift is immediately and
 unconditionally fatal.

 Commenting that test out yields the expected behavior.

 However, commenting it out clearly is the wrong thing. It would make
 things like sh -c 'set -Z || echo failed' not bail out, which would be
 wrong, or at least inconsistent with ksh/bash/zsh. (-Z is an illegal
 option.)

 I'm thinking that shift should be changed to an ordinary builtin
 instead of a special builtin. But I'm not sure what other implications
 this might have. Would someone who understands the finer points of sh
 like to comment?

 Also... ksh exhibits the same behavior. So, apparently, does the
 original Bourne shell. bash and zsh do not. Which makes it unclear if
 it's really a bug or not, too...

 -- 
 David A. Holland
 dholland@netbsd.org

From: Michael van Elst <mlelstv@serpens.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: bin/37493 /bin/sh exits if shift fails, but shouldn't
Date: Sun, 25 May 2008 09:04:27 +0200

 Hi,

 to add some data points: almost no sh or ksh does this as expected.
 I have tested the issue with sh and ksh on AIX5.3, HPUX11.23 and SunOS5.9.

 The exceptions seem to be bash and zsh.

 Greetings,
 -- 
                                 Michael van Elst
 Internet: mlelstv@serpens.de
                                 "A potential Snark may lurk in every tree."

From: Alan Barrett <apb@cequrux.com>
To: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Cc: 
Subject: Re: bin/37493: /bin/sh exits if shift fails, but shouldn't
Date: Sun, 25 May 2008 09:49:53 +0000

 On Sun, 25 May 2008, David Holland wrote:
 >  > >How-To-Repeat:
 >  > $ sh -c 'set -ex; shift || echo failed.'
 >  > + shift
 >  > shift: can't shift that many

 > I'm thinking that shift should be changed to an ordinary builtin
 > instead of a special builtin. But I'm not sure what other implications
 > this might have. Would someone who understands the finer points of sh
 > like to comment?

 shift is required to be a special built-in utility
 <http://www.opengroup.org/onlinepubs/009695399/utilities/shift.html>.

 According to one part of SUSv3, syntax errors in
 special built-in utilities are permitted, but
 not required, to make the shell exit immediately
 <http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_14>.

 According to another part of SUSv3, an "option or operand error"
 in a special built-in utility means that the shell must exit
 <http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_08_01>.
 I think that trying to shift too many parameters would fall under this
 category.

 > Also... ksh exhibits the same behavior. So, apparently, does the
 > original Bourne shell. bash and zsh do not. Which makes it unclear if
 > it's really a bug or not, too...

 I say "not a bug".

 --apb (Alan Barrett)

From: Roland Illig <roland.illig@gmx.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: bin/37493: /bin/sh exits if shift fails, but shouldn't
Date: Sun, 25 May 2008 15:16:50 +0200

 Alan Barrett schrieb:
 >  According to another part of SUSv3, an "option or operand error"
 >  in a special built-in utility means that the shell must exit
 >  <http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_08_01>.
 >  I think that trying to shift too many parameters would fall under this
 >  category.

 Thank you for the explanation. I think that makes the PR a false one. 
 I'll document this quirk in the "pkgsrc portability guide" instead.

 Roland

From: "Greg A. Woods; Planix, Inc." <woods@planix.ca>
To: Alan Barrett <apb@cequrux.com>
Cc: gnats-bugs@NetBSD.org,
 Roland Illig <roland.illig@gmx.de>
Subject: Re: bin/37493: /bin/sh exits if shift fails, but shouldn't
Date: Sun, 25 May 2008 11:44:22 -0400

 On 25-May-08, at 5:49 AM, Alan Barrett wrote:

 > On Sun, 25 May 2008, David Holland wrote:
 >>>> How-To-Repeat:
 >>> $ sh -c 'set -ex; shift || echo failed.'
 >>> + shift
 >>> shift: can't shift that many
 >
 >> I'm thinking that shift should be changed to an ordinary builtin
 >> instead of a special builtin. But I'm not sure what other  
 >> implications
 >> this might have. Would someone who understands the finer points of sh
 >> like to comment?
 >
 > shift is required to be a special built-in utility
 > <http://www.opengroup.org/onlinepubs/009695399/utilities/shift.html>.
 >
 > According to one part of SUSv3, syntax errors in
 > special built-in utilities are permitted, but
 > not required, to make the shell exit immediately
 > <http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_14 
 > >.

 A plain "shift" command, as in the original example above, is never a  
 "syntax" error in that form regardless of whether the command itself  
 can perform the required operation or not.  The syntax of the command  
 as given is itself correct.

 >
 > According to another part of SUSv3, an "option or operand error"
 > in a special built-in utility means that the shell must exit
 > <http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_08_01 
 > >.
 > I think that trying to shift too many parameters would fall under this
 > category.

 As I will show below this is effectively a data error, not anything to  
 do with command options or operands (or syntax).

 This next quote is slightly out of context, but in XBD Chapter 12 it  
 is stated:

 	"The requirement on the standard utilities is that numbers in the  
 specified range do not cause a syntax error, although the  
 specification of a number need not be semantically correct for a  
 particular operand or option-argument of a utility."

 So, the lack of any operand for "shift" is the same as an operand of  
 "1" ("If n is not given, it shall be assumed to be 1.") and as such  
 that number is well within the specified range ("The value n shall be  
 an unsigned decimal integer...") of valid operand values for the  
 "shift" command.  Yes that second quote goes on to say something else  
 that could be construed to be an excuse for a syntax error, however I  
 would strongly suggest that it is then trumped by the following:

 The standard does fully describe the required behaviour of the "shift"  
 utility when its operand value exceeds the number of remaining  
 positional parameters:

 "
 EXIT STATUS

      The exit status is >0 if n>$#; otherwise, it is zero.
 "
 (from <URL:http://www.opengroup.org/onlinepubs/009695399/utilities/shift.html 
  >)

 I.e. "shift" must not trigger a syntax error when its operand is  
 greater than the number of positional parameters remaining -- it must  
 exit with a non-zero status instead.

 -- 
 					Greg A. Woods; Planix, Inc.
 					<woods@planix.ca>

From: David Holland <dholland-bugs@netbsd.org>
To: Alan Barrett <apb@cequrux.com>
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: bin/37493: /bin/sh exits if shift fails, but shouldn't
Date: Sun, 25 May 2008 17:16:02 +0000

 On Sun, May 25, 2008 at 09:49:53AM +0000, Alan Barrett wrote:
  > shift is required to be a special built-in utility
  > <http://www.opengroup.org/onlinepubs/009695399/utilities/shift.html>.

 Well, then...

  > According to one part of SUSv3, syntax errors in
  > special built-in utilities are permitted, but
  > not required, to make the shell exit immediately
  > <http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_14>.
  > 
  > According to another part of SUSv3, an "option or operand error"
  > in a special built-in utility means that the shell must exit
  > <http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_08_01>.
  > I think that trying to shift too many parameters would fall under this
  > category.

 Well, no. It says "Utility syntax error (option or operand error)",
 and it doesn't define this any more specifically anywhere in the whole
 chapter. At least that I can find. I don't see that an out-of-range
 value qualifies as a syntax error.

 Meanwhile, the definition of shift explicitly says that its exit
 status is nonzero if the count is out of range, which would be
 meaningless if the shell were required to exit in this context.
 By the magical properties of reasoning about standards, it therefore
 must not be required to exit in this context.

 (I see Greg A. Woods has deployed approximately the same reasoning...)

  > > Also... ksh exhibits the same behavior. So, apparently, does the
  > > original Bourne shell. bash and zsh do not. Which makes it unclear if
  > > it's really a bug or not, too...
  > 
  > I say "not a bug".

 Nah, it's clearly allowed by POSIX, but I don't think we can claim
 it's required, and it's certainly not *desirable* behavior.

 Fixing it, on the other hand, will require a considerable rework of
 error handling in sh, which I don't think would be a good idea this
 weekend.

 -- 
 David A. Holland
 dholland@netbsd.org

From: Alan Barrett <apb@cequrux.com>
To: gnats-bugs@netbsd.org, netbsd-bugs@netbsd.org
Cc: 
Subject: Re: bin/37493: /bin/sh exits if shift fails, but shouldn't
Date: Sun, 25 May 2008 22:18:08 +0000

 On Sun, 25 May 2008, David Holland wrote:
 >  > <http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_08_01>.
 >  > I think that trying to shift too many parameters would fall under this
 >  > category.
 > 
 > Well, no. It says "Utility syntax error (option or operand error)",
 > and it doesn't define this any more specifically anywhere in the whole
 > chapter. At least that I can find. I don't see that an out-of-range
 > value qualifies as a syntax error.

 I had thought that the value passed to shift was an "operand", and
 that it was an "error" for this value to be larger than the number of
 positional parameters available to the shell; hence an "operand error".
 However, Greg's and your argument ...

 > Meanwhile, the definition of shift explicitly says that its exit
 > status is nonzero if the count is out of range, which would be
 > meaningless if the shell were required to exit in this context.  By
 > the magical properties of reasoning about standards, it therefore must
 > not be required to exit in this context.
 >
 > (I see Greg A. Woods has deployed approximately the same reasoning...)

 ... has convinced me that I was wrong.

 >  > I say "not a bug".
 > 
 > Nah, it's clearly allowed by POSIX, but I don't think we can claim
 > it's required, and it's certainly not *desirable* behavior.

 I agree with all that.  (I previously thought that it was required but
 undesirable; I now think that it's allowed, not required, and still
 undesirable.)

 --apb (Alan Barrett)

From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: bin/37493 (/bin/sh exits if shift fails, but shouldn't)
Date: Mon, 22 Feb 2016 01:17:40 +0700

 I think this PR can be closed.   Nothing has changed in the
 past 7 years (nothing in sh anyway), and POSIX (now) says explicitly...

 (relating to shift)

 EXIT STATUS

     If the n operand is invalid or is greater than "$#", this may be
     considered a syntax error and a non-interactive shell may exit;
     if the shell does not exit in this case, a non-zero exit status
     shall be returned. Otherwise, zero shall be returned.

 which explicitly makes what NetBSD's /bin/sh (and other shells) does
 acceptable (it also makes what bash, etc, do acceptable).

 That is, no portable sh script can assume that a shift with a count
 greater than the number of available args will leave the shell running
 afterwards.

 So, as there is no real requirement to change this, and it hasn't been
 changed already, I think the PR can just go away.

 kre

State-Changed-From-To: open->closed
State-Changed-By: rillig@NetBSD.org
State-Changed-When: Sun, 21 Feb 2016 18:39:21 +0000
State-Changed-Why:
The behavior is explicitly allowed by POSIX.


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.