NetBSD Problem Report #51142

From www@NetBSD.org  Sun May 15 17:03:02 2016
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 0A6CD7AAB4
	for <gnats-bugs@gnats.NetBSD.org>; Sun, 15 May 2016 17:03:02 +0000 (UTC)
Message-Id: <20160515170300.C814A7AAB5@mollari.NetBSD.org>
Date: Sun, 15 May 2016 17:03:00 +0000 (UTC)
From: richard@NetBSD.org
Reply-To: richard@NetBSD.org
To: gnats-bugs@NetBSD.org
Subject: pdksh has occasional issues with empty 'for' loops
X-Send-Pr-Version: www-1.0

>Number:         51142
>Category:       pkg
>Synopsis:       pdksh has occasional issues with empty 'for' loops
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    pkg-manager
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun May 15 17:05:00 +0000 2016
>Closed-Date:    Tue Jul 05 08:48:20 +0000 2016
>Last-Modified:  Tue Jul 05 08:48:20 +0000 2016
>Originator:     Richard PALO
>Release:        current
>Organization:
>Environment:
SunOS omnis 5.11 omnios-master-5409e8f i86pc i386 i86pc

>Description:
the following configure snippet always fails on my platform when using pdksh as my configure shell (noticed with evince3):

# ================ yelp-tools stuff ============= #

YELP_LC_MEDIA_LINKS=true
YELP_LC_DIST=true

for yelpopt in ; do
  case $yelpopt in
    lc-media-links)    YELP_LC_MEDIA_LINKS=true ;;
    no-lc-media-links) YELP_LC_MEDIA_LINKS= ;;
    lc-dist)           YELP_LC_DIST=true ;;
    no-lc-dist)        YELP_LC_DIST= ;;
    *) as_fn_error $? "Unrecognized YELP_HELP_INIT option $yelpopt\"" "$LINENO" 5 ;;
  esac
done;

at configure time this gives (for example):
>configure: error: Unrecognized YELP_HELP_INIT option dummy"
>*** Error code 1

'dummy' seems to only be used via 'set dummy blahbla; ac_word=$2'
type of things.

Seems to be cases where something in pdksh isn't reinitialised
correctly, as manually running the snippet in isolation doesn't
show the problem.

bash and ksh93 work fine too



>How-To-Repeat:
see above, but a quick work-around patch that seems to work is:
diff --git a/print/evince3/patches/patch-configure b/print/evince3/patches/patch-configure
new file mode 100644
index 0000000..f81f857
--- /dev/null
+++ b/print/evince3/patches/patch-configure
@@ -0,0 +1,21 @@
+$NetBSD$
+
+deal with YELP_HELP_INIT invoked with no options
+
+--- configure.orig     2016-03-21 16:28:17.000000000 +0000
++++ configure
+@@ -22382,12 +22382,13 @@ fi
+ YELP_LC_MEDIA_LINKS=true
+ YELP_LC_DIST=true
+ 
+-for yelpopt in ; do
++for yelpopt in z-end; do
+   case $yelpopt in
+     lc-media-links)    YELP_LC_MEDIA_LINKS=true ;;
+     no-lc-media-links) YELP_LC_MEDIA_LINKS= ;;
+     lc-dist)           YELP_LC_DIST=true ;;
+     no-lc-dist)        YELP_LC_DIST= ;;
++    z-end)             ;;
+     *) as_fn_error $? "Unrecognized YELP_HELP_INIT option $yelpopt\"" "$LINENO" 5 ;;
+   esac
+ done;

(I also have a patch for yelp.m4 in wip/yelp-tools)
>Fix:

>Release-Note:

>Audit-Trail:
From: Joerg Sonnenberger <joerg@bec.de>
To: gnats-bugs@NetBSD.org
Cc: pkg-manager@netbsd.org, gnats-admin@netbsd.org, pkgsrc-bugs@netbsd.org
Subject: Re: pkg/51142: pdksh has occasional issues with empty 'for' loops
Date: Sun, 15 May 2016 19:28:04 +0200

 On Sun, May 15, 2016 at 05:05:00PM +0000, richard@NetBSD.org wrote:
 > >Synopsis:       pdksh has occasional issues with empty 'for' loops

 That's not a shell bug, but a "feature" of the standard. A common
 pattern found e.g. in mk/ is to explicit append '' (empty string) and
 short cut with [ -n "$i" ] || break

 Joerg

From: Robert Elz <kre@munnari.OZ.AU>
To: Joerg Sonnenberger <joerg@bec.de>
Cc: gnats-bugs@NetBSD.org, pkgsrc-bugs@netbsd.org
Subject: Re: pkg/51142: pdksh has occasional issues with empty 'for' loops
Date: Mon, 16 May 2016 08:01:11 +0700

     Date:        Sun, 15 May 2016 19:28:04 +0200
     From:        Joerg Sonnenberger <joerg@bec.de>
     Message-ID:  <20160515172804.GA11874@britannica.bec.de>

   | On Sun, May 15, 2016 at 05:05:00PM +0000, richard@NetBSD.org wrote:
   | > >Synopsis:       pdksh has occasional issues with empty 'for' loops
   | 
   | That's not a shell bug, but a "feature" of the standard.

 I believe you're thinking of "for i; do" rather than "for i in; do"

 It certainly looks like a bug in pdksh to me (it isn't as if it does not
 have many) - certainly if that were reported against /bin/sh I would be
 fixing it.

 I notice the PR says that bash and ksh93 work (as they should) but says
 nothing about /bin/sh.  Is there a reason for that?  (In general, unless you
 want it for interactive use reasons, I'd always suggest /bin/sh on NetBSD
 over /bin/ksh)

 kre


From: Richard PALO <richard@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/51142: pdksh has occasional issues with empty 'for' loops
Date: Mon, 16 May 2016 06:52:36 +0200

 Le 15/05/16 19:30, Joerg Sonnenberger a écrit :
 >  That's not a shell bug, but a "feature" of the standard. A common
 >  pattern found e.g. in mk/ is to explicit append '' (empty string) and
 >  short cut with [ -n "$i" ] || break
 >  
 >  Joerg

 Indeed! 
 http://pubs.opengroup.org/onlinepubs/009604499/utilities/xcu_chap02.html#tag_02_14
 does say:
 > The format for the for loop is as follows:
 > 
 >     for name [ in [word ... ]]do
 >         compound-list
 > 
 > 
 >     done
 > 
 > First, the list of words following in shall be expanded to generate a list of items.
 > Then, the variable name shall be set to each item, in turn, and the compound-list
 > executed each time. If no items result from the expansion, the compound-list shall
 > not be executed. Omitting:
 > 
 >     in word ...
 > 
 > shall be equivalent to:
 > 
 >     in "$@"
 > 

 and using "set dummy; shift" prior to the 'for' loop gets over the yelp problem too.

 So it looks as if pdksh inadvertently performs the "$@" substitution even if the "in" remains.

 Testing again bash and ksh93, ommiting the "in" gives 'dummy' as with pdksh so it does
 look like pdksh is the culprit here.

 cheers,
 -- 
 Richard PALO

From: Richard PALO <richard@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/51142: pdksh has occasional issues with empty 'for' loops
Date: Mon, 16 May 2016 07:00:47 +0200

 Le 16/05/16 03:05, Robert Elz a écrit :
 >  I believe you're thinking of "for i; do" rather than "for i in; do"
 >  
 Guess I should have refreshed first, after coming to the same conclusion...

 >  It certainly looks like a bug in pdksh to me (it isn't as if it does not
 >  have many) - certainly if that were reported against /bin/sh I would be
 >  fixing it.
 >  
 >  I notice the PR says that bash and ksh93 work (as they should) but says
 >  nothing about /bin/sh.  Is there a reason for that?  (In general, unless you
 >  want it for interactive use reasons, I'd always suggest /bin/sh on NetBSD
 >  over /bin/ksh)
 >  
 >  kre

 I'm running on SunOS 5.11 where native /bin/sh is the same ksh93, but my ksh93
 test was with pkgsrc ksh93... 
 (PM: bootstrap --full gives pdksh as the default pkgsrc shell; which I use)

 cheers
 -- 
 Richard PALO

From: Richard PALO <richard.palo@free.fr>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/51142: pdksh has occasional issues with empty 'for' loops
Date: Mon, 16 May 2016 09:10:17 +0200

 Le 16/05/16 06:55, Richard PALO a écrit :
 >  Testing again bash and ksh93, ommiting the "in" gives 'dummy' as with pdksh so it does
 >  look like pdksh is the culprit here.
 >  

 Although by no means specialist in shell processing, this did seem to come down to around
 the following in the wordlist() function found in syn.c

 Prior to this patch, the result was the same whether "in" was specified with a null set of options or not.

 > diff --git a/shells/pdksh/files/syn.c b/shells/pdksh/files/syn.c
 > index d3bce2e..3b19d0a 100644
 > --- a/shells/pdksh/files/syn.c
 > +++ b/shells/pdksh/files/syn.c
 > @@ -609,13 +609,8 @@ wordlist()
 >                 XPput(args, yylval.cp);
 >         if (c != '\n' && c != ';')
 >                 syntaxerr((char *) 0);
 > -       if (XPsize(args) == 0) {
 > -               XPfree(args);
 > -               return NULL;
 > -       } else {
 > -               XPput(args, NULL);
 > -               return (char **) XPclose(args);
 > -       }
 > +       XPput(args, NULL);
 > +       return (char **) XPclose(args);
 >  }
 >  
 >  /*

 before:
 > richard@omnis:/home/richard/src/pkgsrc/shells/pdksh$ /opt/local/bin/pdksh 
 > $ set dummy 
 > $ for i in ; do echo $i; done
 > dummy
 > $ for i ; do echo $i; done
 > dummy

 after:
 > richard@omnis:/tmp/pkgsrc/shells/pdksh/work/.destdir/opt/local/bin$ ./pdksh 
 > $ set dummy
 > $ for i in ; do echo $i; done
 > $ for i ; do echo $i; done
 > dummy


 does this look reasonable?

 -- 
 Richard PALO

From: Joerg Sonnenberger <joerg@bec.de>
To: gnats-bugs@NetBSD.org
Cc: pkg-manager@netbsd.org, gnats-admin@netbsd.org, pkgsrc-bugs@netbsd.org,
	richard@NetBSD.org
Subject: Re: pkg/51142: pdksh has occasional issues with empty 'for' loops
Date: Mon, 16 May 2016 10:58:26 +0200

 On Mon, May 16, 2016 at 04:55:01AM +0000, Richard PALO wrote:
 > The following reply was made to PR pkg/51142; it has been noted by GNATS.
 > 
 > From: Richard PALO <richard@netbsd.org>
 > To: gnats-bugs@NetBSD.org
 > Cc: 
 > Subject: Re: pkg/51142: pdksh has occasional issues with empty 'for' loops
 > Date: Mon, 16 May 2016 06:52:36 +0200
 > 
 >  Le 15/05/16 19:30, Joerg Sonnenberger a écrit :
 >  >  That's not a shell bug, but a "feature" of the standard. A common
 >  >  pattern found e.g. in mk/ is to explicit append '' (empty string) and
 >  >  short cut with [ -n "$i" ] || break
 >  >  
 >  >  Joerg
 >  
 >  Indeed! 
 >  http://pubs.opengroup.org/onlinepubs/009604499/utilities/xcu_chap02.html#tag_02_14
 >  does say:
 >  > The format for the for loop is as follows:
 >  > 
 >  >     for name [ in [word ... ]]do
 >  >         compound-list
 >  > 
 >  > 
 >  >     done

 Actually, you are looking at a too-new version. Try SUSv2 for example:
 http://pubs.opengroup.org/onlinepubs/7908799/xcu/chap2.html#tag_001_009_004_002

     for name [ in word ... ]
     do
         compound-list
     done

 ...and that has been interpreted historically by some shells (this one
 included) to mean that after the "in" something must be found. The BNF
 grammar is similar problematic. I'm not really objecting to fixing this
 is pdksh, but be careful to call it a bug.

 Joerg

From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/51142: pdksh has occasional issues with empty 'for' loops
Date: Mon, 16 May 2016 17:13:46 +0700

     Date:        Mon, 16 May 2016 07:10:01 +0000 (UTC)
     From:        Richard PALO <richard.palo=40free.fr>
     Message-ID:  <20160516071001.CEF3F7AAB4=40mollari.NetBSD.org>

   =7C  does this look reasonable?

 I can't comment on the patch, as I don't know the code, but the
 result looks to be what is wanted.

 joerg=40bec.de said:
   =7C  Actually, you are looking at a too-new version.=20

 The problem with a lot of the older shell specs is that they were
 wildly wrong (did not say what they really intended).   Whether
 an implementation which adapted to what those seemed to require can
 be said to have bugs or not is just semantics - no-one is assigning
 blame here (or shouldn't be) just trying to get everything operating
 as it should.   And whether that =22word...=22 was actually intended to m=
 ean
 one more more words, or zero or more words doesn't really matter any
 more - anything that believed the 1 or more interpretation must now
 know that was incorrect, and should be fixed.

 kre

From: Richard PALO <richard@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/51142: pdksh has occasional issues with empty 'for' loops
Date: Thu, 19 May 2016 13:43:30 +0200

 I'm concerned about the extent of the fix wherein the code is working on *both* 'for' and 'select'.

 Judging by http://linux.die.net/man/1/pdksh it appears that 'select' shouldn't necessarily
 support the 'in ;' sort of usage.

 Could I get any suggestions, confirmation (or infirmation) of this to determine if there needs to be a
 a bit of adapting for 'select' case.

 Other than this mention, outside of a bulk build (which I can't do since a while until
 I resync my tree's 'current' pbulk with 'joyent' bits) the patch seems be working well.

 -- 
 Richard PALO

From: Richard PALO <richard@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/51142: pdksh has occasional issues with empty 'for' loops
Date: Tue, 31 May 2016 17:49:06 +0200

 Le 19/05/16 13:45, Richard PALO a écrit :
 > 
 >  I'm concerned about the extent of the fix wherein the code is working on *both* 'for' and 'select'.
 >  

 Adding as well the following patch seems to get 'select' to a similarly working situation:
 > diff --git a/shells/pdksh/files/exec.c b/shells/pdksh/files/exec.c
 > index 734d484..14e216b 100644
 > --- a/shells/pdksh/files/exec.c
 > +++ b/shells/pdksh/files/exec.c
 > @@ -334,7 +334,7 @@ execute(t, flags)
 >                 }
 >  #ifdef KSH
 >                 else { /* TSELECT */
 > -                       for (;;) {
 > +                       while (*ap != NULL) {
 >                                 if (!(cp = do_selectargs(ap, is_first))) {
 >                                         rv = 1;
 >                                         break;


 before:
 > richard@omnis:/home/richard/src/pkgsrc/shells/pdksh$ /opt/local/bin/pdksh.orig 
 > $ set dummy
 > $ select i in ; do echo $i; break; done
 > 1) dummy
 > #? 1
 > dummy
 > $ select i ; do echo $i; break; done
 > 1) dummy
 > #? 1
 > dummy


 after:
 > richard@omnis:/home/richard/src/pkgsrc/shells/pdksh$  /tmp/pkgsrc/shells/pdksh/work/.destdir/opt/local/bin/pdksh 
 > $ set dummy
 > $  select i in ; do echo $i; break; done
 > $ echo $?
 > 0
 > $ select i ; do echo $i; break; done
 > 1) dummy
 > #? 1
 > dummy
 > $ echo $?
 > 0


 Request some 'okays' to commit
 -- 
 Richard PALO

From: Richard PALO <richard@netbsd.org>
To: gnats-bugs@NetBSD.org, =?UTF-8?B?SsO2cmcgU29ubmVuYmVyZ2Vy?=
 <joerg@netbsd.org>, Robert Elz <kre@munnari.OZ.AU>
Cc: 
Subject: Re: pkg/51142: pdksh has occasional issues with empty 'for' loops
Date: Wed, 01 Jun 2016 08:32:50 +0200

 I put a copy of the complete patchset proposed here:
 > http://www.netbsd.org/~richard/pdksh.patch

 as jörg indicated, this seems not to be a bug per se but, but as already indicated in the source,
 this is for ast-ksh compatibility as encountered.

 btw, seems in bash, as opposed to ast-ksh, the select clause without the 'in ...' gives a syntax error.
 I don't believe this has any impact, strictly speaking, on pkgsrc as a build shell.

 cheers
 -- 
 Richard PALO

From: "Richard PALO" <richard@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/51142 CVS commit: pkgsrc/shells/pdksh
Date: Sun, 5 Jun 2016 04:44:56 +0000

 Module Name:	pkgsrc
 Committed By:	richard
 Date:		Sun Jun  5 04:44:56 UTC 2016

 Modified Files:
 	pkgsrc/shells/pdksh: Makefile
 	pkgsrc/shells/pdksh/files: exec.c syn.c

 Log Message:
 PR/51142: address for/select loop compatibility with ksh93 in pdksh


 To generate a diff of this commit:
 cvs rdiff -u -r1.23 -r1.24 pkgsrc/shells/pdksh/Makefile
 cvs rdiff -u -r1.3 -r1.4 pkgsrc/shells/pdksh/files/exec.c \
     pkgsrc/shells/pdksh/files/syn.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->feedback
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Tue, 05 Jul 2016 01:40:06 +0000
State-Changed-Why:
Is this fixed?


From: Richard PALO <richard@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/51142 (pdksh has occasional issues with empty 'for' loops)
Date: Tue, 05 Jul 2016 08:24:45 +0200

 Le 05/07/16 03:40, dholland@NetBSD.org a écrit :
 > Synopsis: pdksh has occasional issues with empty 'for' loops
 > 
 > State-Changed-From-To: open->feedback
 > State-Changed-By: dholland@NetBSD.org
 > State-Changed-When: Tue, 05 Jul 2016 01:40:06 +0000
 > State-Changed-Why:
 > Is this fixed?
 > 
 > 
 > 
 > 
 I have not received any [negative] feedback since the patch
 committed.  It can probably be closed, at least for now.

 -- 
 Richard PALO

State-Changed-From-To: feedback->closed
State-Changed-By: bsiegert@NetBSD.org
State-Changed-When: Tue, 05 Jul 2016 08:48:20 +0000
State-Changed-Why:
Submitter says fixed.


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.