NetBSD Problem Report #53919
From woods@weird.com Mon Jan 28 20:41:31 2019
Return-Path: <woods@weird.com>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 447757A1B1
for <gnats-bugs@gnats.NetBSD.org>; Mon, 28 Jan 2019 20:41:31 +0000 (UTC)
Message-Id: <m1goDiO-0031oCC@building.local>
Date: Mon, 28 Jan 2019 12:41:08 -0800 (PST)
From: "Greg A. Woods" <woods@planix.ca>
Sender: "Greg A. Woods" <woods@weird.com>
Reply-To: "Greg A. Woods" <woods@planix.ca>
To: gnats-bugs@NetBSD.org
Subject: please revert changes that make /bin/sh evaluate $ENV non-interactively
X-Send-Pr-Version: 3.95
>Number: 53919
>Category: bin
>Synopsis: please revert changes that make /bin/sh evaluate $ENV non-interactively
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: kre
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Mon Jan 28 20:45:00 +0000 2019
>Closed-Date: Fri Jan 31 17:38:55 +0000 2020
>Last-Modified: Fri Jan 31 17:38:55 +0000 2020
>Originator: Greg A. Woods
>Release: NetBSD 8.99.27
>Organization:
Planix, Inc.; Kelowna, BC; Canada
>Environment:
System: NetBSD future 8.99.27 NetBSD 8.99.27 (XEN3_DOMU) #1: Mon Jan 14 20:32:17 PST 2019 woods@future:/build/woods/future/current-amd64-amd64-obj/building/work/woods/m-NetBSD-current/sys/arch/amd64/compile/XEN3_DOMU amd64
Architecture: x86_64
Machine: amd64
>Description:
Please revert the changes to /bin/sh which cause it to evaluate
$ENV when not running in "interactive" mode.
I.e. revert, or disable, the changes made for PR 42829
The expansion of $ENV by non-interactive /bin/sh process makes
the system effectively and entirely unusable by some of us users
of /bin/ksh who have been happily using a certain feature of
/bin/ksh and similar shells for over 30 years now.
Way back when I was discussing this issue for PR 42828 I failed
to fully appreciate the implications of this change. I am now
vehemently opposed to these changes, at least in their current
form.
>How-To-Repeat:
I finally got around to upgrading a server built with Jan 14
sources and was surprised to find a plethora of error messages
that appeared to be coming from /bin/sh, sometimes appearing in
the most unwelcome and unexpected places.
Observe:
$ /usr/bin/true
/usr/bin/true: 0: Syntax error: Bad substitution
Not having paid enough attention to recent changes in NetBSD
made such an error message extremely confusing and concerning.
Debugging this was also a bit of an adventure with the new GDB
as it seems to have sometimes become more difficult to use
without a fully populated /usr/libdata/debug, even for
static-linked binaries with a full .debug file present. But
that's an issue for another PR. Suffice it to say that without
use of the debugger I would likely have taken far far longer to
figure out what was going on.
>Fix:
I don't know what the best fix is. As I said in PR 42828 I
think the best solution would be _full_ compatability with the
original source of this feature, namely the original Korn Shell.
However as I discussed this would require implementation of
arrays for /bin/sh, and that's probably not a short term fix.
My sort-term hack, just to make my system usable again, is to
change the name of the variable to SHENV. However I doubt I
will ever set $SHENV for my own use, because:
The current implementation, i.e. no matter what the variable is
named, is extremely unsatisfactory as is.
The reason the current implementation is unacceptable is because
it forces the shell to open the file referenced by $ENV and to
read the entire file's contents (in most cases, as per the
recommended hack for interactive-only use) and, if I'm not
mistaken, parse all of what it reads as well. What a horrible
useless waste, and for _EVERY_ shell invocation! That is not
acceptable to me, and it should not be acceptable to anyone who
has even the slightest concern about userland performance.
However I still can't think of a better fix than to implement
arrays in /bin/sh.
I also add a small error message to try to help explain the
mystery error message. That needs to be done to /bin/ksh too.
(The real ksh93 seems to suppress all errors expanding $ENV,
which might also be acceptable, though it certainly adds yet
more mystery to $ENV expansion failures.)
Index: main.c
===================================================================
RCS file: /cvs/master/m-NetBSD/main/src/bin/sh/main.c,v
retrieving revision 1.80
diff -u -r1.80 main.c
--- main.c 19 Jan 2019 14:20:22 -0000 1.80
+++ main.c 28 Jan 2019 19:46:13 -0000
@@ -213,9 +213,15 @@
struct stackmark env_smark;
setstackmark(&env_smark);
- if ((shinit = lookupvar("ENV")) != NULL && *shinit != '\0') {
- state = 3;
- read_profile(expandenv(shinit));
+ if ((shinit = lookupvar("SHENV")) != NULL && *shinit != '\0') {
+ const char *shenvf = expandenv(shinit);
+
+ if (strcmp(shinit, shenvf) == 0) {
+ sh_warnx("failed to expand SHENV");
+ } else {
+ state = 3;
+ read_profile(shenvf);
+ }
}
popstackmark(&env_smark);
}
Index: sh.1
===================================================================
RCS file: /cvs/master/m-NetBSD/main/src/bin/sh/sh.1,v
retrieving revision 1.217
diff -u -r1.217 sh.1
--- sh.1 21 Jan 2019 14:09:24 -0000 1.217
+++ sh.1 28 Jan 2019 19:54:38 -0000
@@ -137,7 +137,7 @@
.Pq \&$HOME ,
if they exist.
If the environment variable
-.Ev ENV
+.Ev SHENV
is set on entry to a shell,
or is set in the
.Pa .profile
@@ -147,40 +147,40 @@
option is not set,
the shell then performs parameter and arithmetic
expansion on the value of
-.Ev ENV ,
+.Ev SHENV ,
(these are described later)
and then reads commands from the file name that results.
If
-.Ev ENV
+.Ev SHENV
contains a command substitution, or one of the
other expansions fails, or if there are no expansions
to expand, the value of
-.Ev ENV
+.Ev SHENV
is used as the file name.
.Pp
Therefore, a user should place commands that are to be executed only at
login time in the
.Pa .profile
file, and commands that are executed for every shell inside the
-.Ev ENV
+.Ev SHENV
file.
To set the
-.Ev ENV
+.Ev SHENV
variable to some file, place the following line in your
.Pa .profile
of your home directory
.Pp
-.Dl ENV=$HOME/.shinit; export ENV
+.Dl SHENV=$HOME/.shinit; export SHENV
.Pp
substituting for
.Dq .shinit
any filename you wish.
Since the
-.Ev ENV
+.Ev SHENV
file can be read for every invocation of the shell, including shell scripts
-and non-interactive shells, the following paradigm is useful for
+and non-interactive shells, the following paradigm is useful, though ugly and not ideal, for
restricting commands in the
-.Ev ENV
+.Ev SHENV
file to interactive invocations.
Place commands within the
.Dq Ic case
@@ -410,7 +410,7 @@
.Pa /etc/profile ,
.Pa .profile ,
and the file specified by the
-.Ev ENV
+.Ev SHENV
environment variable.
.It Fl s Em stdin
Read commands from standard input (set automatically if
@@ -518,7 +518,7 @@
option on the command line.
Currently this option controls whether (!posix) or not (posix)
the file given by the
-.Ev ENV
+.Ev SHENV
variable is read at startup by a non-interactive shell.
It also controls whether file descriptors greater than 2
opened using the
@@ -4055,7 +4057,7 @@
If unset
.Dq $HOME/.editrc
is used.
-.It Ev ENV
+.It Ev SHENV
Names the file sourced at startup by the shell.
Unused by this shell after initialization,
but is usually passed through the environment to
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: bin-bug-people->kre
Responsible-Changed-By: kre@NetBSD.org
Responsible-Changed-When: Tue, 29 Jan 2019 04:19:13 +0000
Responsible-Changed-Why:
Mine.
From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: bin/53919: please revert changes that make /bin/sh evaluate $ENV non-interactively
Date: Tue, 29 Jan 2019 13:44:27 +0700
Date: Mon, 28 Jan 2019 20:45:00 +0000 (UTC)
From: "Greg A. Woods" <woods@planix.ca>
Message-ID: <20190128204500.DD68C7A1FE@mollari.NetBSD.org>
| Please revert the changes to /bin/sh which cause it to evaluate
| $ENV when not running in "interactive" mode.
I can interpret this in several ways. If you are requesting that
we re-open PR 42828, then sorry, no.
If ...
| I.e. revert, or disable, the changes made for PR 42829
That (alone) I am certainly willing to condsider, that one was only
processed in the name of being more POSIX compatible, and with
the assumption that it would do no harm. I personally see no
rational benefit in having $ENV parameter expanded, it ought simply
be set to a file name (with any expansions needed being done
when it is set) - though from your usage I doubt that you agree.
I do not think it would be rational or sane to make the decision on
whether or not to parameter expand $ENV be done based upon
whether the shell is interactive or not - whatever happens, it should
be the same both ways (again, no plans to avoid using $ENV in
non-ineractive shells) - not that I believe that you are suggesting
that change. Here I am just mentioning all the possibilities.
| The expansion of $ENV by non-interactive /bin/sh process makes
| the system effectively and entirely unusable by some of us users
| of /bin/ksh who have been happily using a certain feature of
| /bin/ksh and similar shells for over 30 years now.
I don't really belive this is a rational thing to do, or to expect to work.
While it may be been functional 30 years ago, when the sh choices
available were limited, these days there are lots more shells around,
and when they extend POSIX, which almost all do, one way or another,
they all tend to be different, though there are some common extensions.
ENV is a POSIX specified variable, and parameter expanding it, is
a POSIX mandated activity (whether we decide to continue that
practice in /bin/sh or not), so any setting of $ENV ought to be limited
to use only POSIX defnied mechanisms if you have any real hope
that it is going to work.
Just doing something as simple as "I see a new shell appeared in pkgsrc,
let's see if it is any good" ... so you install it using whatever method you
use for installing packages, and then try it out
newshell
to see what happens - only to find that being a POSIX compatible (or
mostly) shell, and interactive, it parameter expands $ENV and because
your version of that uses non-POSIX mechanisms, it fails.
What you're effectively doing in this PR is requesting that all shells
which do not happen to have the extensions that you want to use to
ignore the POSIX spec and not parameter expand $ENV - only shells
that happen to implement the extensions that you want to use are to
be permitted to parameter expand $ENV.
[Aside: for casual readers, note it is $ENV that this relates to, the
name ENV has always been expanded, what is of concern here is
whether the string produced from generating ${ENV} is expanded
again before being used.]
Your usage has always been broken, it just has not affected you
until now.
You should also be aware that your particular usage not only
requires that arrays be implemented in the shell in question (something
that will never happen in /bin/sh while I am maintaing it, they are
entirely unnecessary, and cause all kinds of bizarre parsing issues
as commonly defined) - but you're also expecting that they be
implemented and accessed the way that ksh first implemened them,
and that the array indexes be interpreted and expanded using the
parsing rules that ksh defined for them, neither of which is guaranteed
in anything except ksh. That is, you should ensure that your magic
expression is only ever expanded by a ksh type shell, and is never
seen by anything else. Including it in $ENV 30 years ago only ever
worked because only ksh (and bash, which copied ksh for this) actually
did parameter expansion -- and I note you are, or were in 2010 when
you commented on PR 42828, also expecting tilde expansion of $ENV,
which POSIX does not specify (and we do not implement) - of course,
that would easily be accommodated, and perhaps has been since,
by just using '${HOME}' instead of '~'. (You were (maybe still are)
expecting ~ expansion to happen (for this anyway) after parameter
expansion, which never happens anywhere in a POSIX shell.)
A better solution for your problem, which should be much more
portable, is simply to make $ENV be a very short file, that determines
whether it should be used at all (if you don't want it in non-interactive
shells, it should simply return) and then determines which shell is
being run (without using unsafe tricks like uses of $RANDOM unless
there is absolutely no other way) and then sources a shell specific
setup script, or a generic one (which uses only standard POSIX) if
it cannot determine with some degree of certainty, which shell is
currently being used. This has to be done inside the file set as
$ENV, and not as part of login time init, as the shell expanding
$ENV and then reading and processing the result might not even
exist at the time you logged in.
| >Fix:
|
| I don't know what the best fix is. As I said in PR 42828 I
| think the best solution would be _full_ compatability with the
| original source of this feature, namely the original Korn Shell.
| However as I discussed this would require implementation of
| arrays for /bin/sh, and that's probably not a short term fix.
Not short term, not long term, not any term as far as I am concerned.
If you want ksh to be /bin/sh you know how to make that happen, I
presume... If you use NetBSD's (ancient pdksh version in /bin/ksh)
for that purpose, don't expect the rest f the system to work properly.
| My sort-term hack, just to make my system usable again, is to
| change the name of the variable to SHENV. However I doubt I
| will ever set $SHENV for my own use, because:
That would be to acceed to PR 42828, which we have already decided
we are not going to do (more than what was done years ago.)
| The reason the current implementation is unacceptable is because
| it forces the shell to open the file referenced by $ENV and to
| read the entire file's contents
That depends upon the shell. /bin/sh reads only as much as it needs
to before the script in there is done. But it should be a very small file,
if it exceeds one read buffer worth, then it is probably doing more than
it should. Some other shells wll read the entire thing (though those also
tend to be ones that only process ENV in interactive shells.)
| recommended hack for interactive-only use) and, if I'm not
| mistaken, parse all of what it reads as well.
You're mistaken for /bin/sh - it reads and parses a command at a
time, then executes it. If that results in a "return" we're done, and
it neither reads (though more was probably already read into the
file buffer), nor parses, any more. Other shells (perhaps ksh - or
some variants - and maybe bosh) parse the whole thing before
executing any of it (they are more likely to do that for files read
using '.' but sometimes the same mechanism is used to process
$ENV).
| However I still can't think of a better fix than to implement
| arrays in /bin/sh.
You could just not sent ENV, and instead of modifying sh to use
a different variable name, instead modify ksh to use KSH_ENV or
something, in which you could put as much KSH specific syntax
as you like. ksh could continue processing ENV as well, for
posix conformance, but you wouldn't be using that.
That would be a much more portable solution. I believe bash
has that functionality - it has its own startup script var name
which it will read.
That is something I could do for sh as well, if people would like
that capability, a NetBSD sh specific var which names a startup
file only processed by the NetBSD sh (or two, one for interactive
shells, and one for others) if people would like that (if so, please
file a "change-request" PR to suggest it.) That would not help for
this issue however, as $ENV is still going to be there, and processed
the same way (whichever way that ends up being - the -8 way, or
the HEAD way.)
| I also add a small error message to try to help explain the
| mystery error message.
Since there are many ways that expanding $ENV can lead to problems
if it is not set correctly, and not all of them are related to expansion
errors, I am not sure that would help a lot, though it might be useful
in some cases.
Your implementation won't work in general (it works for you because
you're expecting the expanded $ENV to be different to the unexpanded
one, but that is not the usual case) but that would be easily fixed if we
decide getting (more) errors from faulty settings of $ENV is a good idea.
Now I believe we need input from others, before any changes are made,
but pending that, my position would be that reverting the PR42829 change
is not out of the question, but that (aside from more possible error messages,
or perhaps just enhancing what is there now to be more specific as to
context, if we don't revert 42829 changes) that is the only change that is
likely to result from this PR.
kre
From: "Greg A. Woods" <woods@planix.ca>
To: NetBSD GNATS <gnats-bugs@NetBSD.org>
Cc:
Subject: Re: bin/53919: please revert changes that make /bin/sh evaluate $ENV non-interactively
Date: Tue, 29 Jan 2019 10:51:27 -0800
--pgp-sign-Multipart_Tue_Jan_29_10:51:10_2019-1
Content-Type: text/plain; charset=US-ASCII
At Tue, 29 Jan 2019 06:45:01 +0000 (UTC), Robert Elz <kre@munnari.OZ.AU> wrote:
Subject: Re: bin/53919: please revert changes that make /bin/sh evaluate $ENV non-interactively
>
> Your usage has always been broken, it just has not affected you
> until now.
Broken? I think not. It just hasn't ever interacted so horribly with
NetBSD /bin/sh (or any other POSIX-ish system shell) ever before. I've
installed and used Ksh or something compatible enough on almost all of
the extremely many and varied systems I've used since 1985, and I've
used the same $ENV setting since before 1993, and I've never seen any
other system shell misbehave when faced with any aspect of my Ksh
environment. It is though somehow pleasant to see a situation where the
REPL of a system's /bin/sh isn't entirely useless to Ksh users (which of
course has been more or less true of NetBSD's /bin/sh since 1.0).
However I'd like to prevent any /bin/sh in any current or future NetBSD
release from being the first to so misbehave.
So, with a bit more debugging, I see that the fundamental reason NetBSD
/bin/sh hasn't been a problem for Korn and Korn-like Shell users since
growing its $ENV hack is that it has been silently failing to do
anything with David Korn's long-recommended setting for $ENV. Here's
why, from ktrace+kdump:
13571 1 sh CALL open(0x7f7ffffffb6d,0,0x7f7ffde06440)
13571 1 sh NAMI "${ENVFILE[(_$-=1)+(_=0)-(_$-!=_${-%%*i*})]}"
13571 1 sh RET open -1 errno 2 No such file or directory
However this failure is silent and un-obtrusive, and efficient.
Now that I see this clearly in this context it makes perfect sense.
So, the thing that made my new -current system entirely unusable was a
new error message, and the most frustrating thing was the futility and
uselessness of that feckless error message.
One thing is certain: The $ENV form using arrays is not going away. It
has worked for 30+ years, and is widely recommended, including in the
"The Kornshell" book (since 1989). This method has worked A-OK with
_all_ Ksh derivatives I've ever encountered, including even Zsh (older
Zsh, that is, IIRC -- newer Zsh is, if I'm not mistaken, even more Ksh
compatible). Even if I'm willing to change my own interactive shell
settings, there's no changing the behaviour of shells that now prevail
in the vast enormous majority of all other systems in the world.
I'm rather sad that the POSIX committee was somehow able to justify, at
least to to themselves, a half-baked appropriation of this ancient Ksh
feature. But that's water under the bridge. (Perhaps I'm most sad that
I personally wasn't able to be more involved in POSIX, but that's also
flowed long ago into sea of time.)
So I think I would be happy enough if /bin/sh guaranteed never to make a
peep about $ENV expansion or open() failures in all normal circumstances.
(i.e. maintain the status quo of behaviour in existing releases).
Of course I would also be happy if there were some command-line flag
that made the shell very noisy about $ENV issues. Perhaps "-x" would
suffice, though I see it is already quelled by read_profile(), so
something stronger would probably be needed for profile and ENV file
debugging.
> A better solution for your problem, which should be much more
> portable, is simply to make $ENV be a very short file, that determines
> whether it should be used at all (if you don't want it in non-interactive
> shells, it should simply return) and then determines which shell is
> being run [[...]]
Sorry, I don't think that is a better solution at all. It's a bad
(inefficient) hack. Change for the better is fine, but not change for
the worse.
The only good solution is to do something smart during the runtime
expansion of $ENV to prevent the open from occurring or at least from
succeeding, just as Korn suggested in the beginning. I think I could
come up with a new hack to be compatible with a no-arrays shell, but as
I said, the arrays method simply isn't ever going away in wider circles.
--
Greg A. Woods <gwoods@acm.org>
+1 250 762-7675 RoboHack <woods@robohack.ca>
Planix, Inc. <woods@planix.com> Avoncote Farms <woods@avoncote.ca>
--pgp-sign-Multipart_Tue_Jan_29_10:51:10_2019-1
Content-Type: application/pgp-signature
Content-Transfer-Encoding: 7bit
Content-Description: OpenPGP Digital Signature
-----BEGIN PGP SIGNATURE-----
iF0EABECAB0WIQRuK6dmwVAucmRxuh9mfXG3eL/0fwUCXFCgpAAKCRBmfXG3eL/0
f6ArAJ9R+NHJj6J9uIXJ9CCfOrxVYiu8JACeNF9WupOyGbVXT/raih9kMb88moo=
=S8DS
-----END PGP SIGNATURE-----
--pgp-sign-Multipart_Tue_Jan_29_10:51:10_2019-1--
From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: bin/53919: please revert changes that make /bin/sh evaluate $ENV non-interactively
Date: Wed, 30 Jan 2019 19:06:40 +0700
Date: Tue, 29 Jan 2019 18:55:01 +0000 (UTC)
From: "Greg A. Woods" <woods@planix.ca>
Message-ID: <20190129185501.744187A1E5@mollari.NetBSD.org>
| Broken? I think not.
It is, and always was. Not in the sense of not working, more in the
sense of being something that is and always was unsafe and
unpredictable, but just happened to work OK in the environments
that you have encountered (perhaps even most that you could have
encountered for a long time.)
This is very much like the ancient C programs that used to assume that
all chars were signed (so they were a safe place to put small negative
integers, as well as small positive ones) or that computers are all
little endian, so *(char *)&var = 1; works to assign 1 to a var which
was 0 before, regardless of the size of the int type that var happened to
be, or that *(char *)NULL == '\0';
All of those, for a long time, worked on many systems, and were used
by many programs, and worked just fine - or seemed to. Until they
didn't.
Those were always broken, never guaranteed to work, and as much as
some people would have preferred that the underlying assumptions could
have been converted into requirements, that was never going to happen.
Your usage of ENV is just the same kind of thing. It is making
assumptions that neither are, nor ever have been, promised by
anything, but which for a long time just happened to be satisfactory.
| However I'd like to prevent any /bin/sh in any current or future NetBSD
| release from being the first to so misbehave.
It isn't misbehaving, it just is not doing what you want.
| So, with a bit more debugging, I see that the fundamental reason NetBSD
| /bin/sh hasn't been a problem for Korn and Korn-like Shell users since
| growing its $ENV hack is that it has been silently failing to do
| anything with David Korn's long-recommended setting for $ENV.
Yes, if the file named fails to exist, sh (and all shells, I think this may
even be part of the rules, and if not, probably should be) silently ignore
attempts to open profile type files that don't exist.
Note however, that if you don't cause $ENV to have a fully qualified
file name then ...
| 13571 1 sh NAMI "${ENVFILE[(_$-=1)+(_=0)-(_$-!=_${-%%*i*})]}"
[I deleted some spaces for line length reasons]
if you're in a directory (say /tmp) where others have write permission,
and someone creates a file with just that name (which is easy to do,
especially as you are so public about what you use) then that file is
going to be read as a sh script and executed (as you). I'd call that a
security problem that you ought to address. Doing so is easy, you
just need to make sure that (no matter what) ENV is set to a path that
starts with / and always goes to somewhere safe. Regardless of how
it is expanded (or is not expanded.)
One might think that sh could help with this, by checking ownership or
something, but it can't - there's no rule that says that a $ENV file
needs to be owned by the user whose shell uses it, which is something
that has been used for years, with less experienced users using the
setup files of more experienced ones who do a better job of making a
nice environment. Sure, they are taking a risk - but that is just of
trusting another (known, and specific) user not to be malicious, not
simply running a strangely named named file that happens to be
in $PWD when the shell starts.
| So, the thing that made my new -current system entirely unusable was a
| new error message, and the most frustrating thing was the futility and
| uselessness of that feckless error message.
If all you need is for the error message to go away, that's probably
something that could happen.
| One thing is certain: The $ENV form using arrays is not going away. It
| has worked for 30+ years, and is widely recommended, including in the
| "The Kornshell" book (since 1989).
As I recall it (it has been a long time since I looked there), the
Kornshell book is about using ksh right? Obviously if you're using
ksh (especially where /bin/sh is a ksh variant) that is going to work
just fine. And in 1989 or thereabouts, *(char *)NULL worked fine too...
| This method has worked A-OK with
| _all_ Ksh derivatives I've ever encountered,
Of course it would, it is ksh specific, any ksh variant, or anything that
emulates ksh, is (or should) handle that (well, perhaps, see below.)
How about if you were to stick zsh'isms in your $ENV (perhaps as
recommended by some zsh book ... or these days, more likely, website)
and then attempt to use those when the shell is ksh. Would you
complain just as loudly that ksh is not doing the right thing by not
implementing the zsh extensions (there are *many*) or not interpreting
them the same way ... or by giving an error message when it fails
to comprehend what the zsh extension means?
| I'm rather sad that the POSIX committee was somehow able to justify, at
| least to to themselves, a half-baked appropriation of this ancient Ksh
| feature. But that's water under the bridge. (Perhaps I'm most sad that
| I personally wasn't able to be more involved in POSIX, but that's also
| flowed long ago into sea of time.)
You could always try now, and good luck with it - I agree that the POSIX
spec for ENV isn't really adequate, but nothing has, or is likely to, alter
now. This is just like the people who really believe that char should
have been specified to always be a signed type. Whoever is right about
issues like this, it is just too late, the spec is set, and the chances of
it changing now are close to 0 ... and if it did, all it could change to
is "unspecified", which really helps no-one.
| So I think I would be happy enough if /bin/sh guaranteed never to make a
| peep about $ENV expansion or open() failures in all normal circumstances.
| (i.e. maintain the status quo of behaviour in existing releases).
The open failures, sure, it has always been that way, I suspect might be
required to be that way, and isn't going to change (for many reasons.)
The other, yes, I could suppress the error msgs when an expansion fails,
though in that case, I think I'd tend to make the result be something
safe, like "" rather than the original value. I will wait on opinions
from others before making any changes here though, as I believe that
some people are of the opinion that errors should always be reported.
| Of course I would also be happy if there were some command-line flag
| that made the shell very noisy about $ENV issues.
There is no real need for that, if you suspect $ENV is not doing what
you want, then:
eval printf '%s\\n' "${ENV}"
would tell you soon enough (in the shell in question, both interactively
and not, if it is intended that there be a difference).
And
| Perhaps "-x" would suffice,
that will let you see if anything that is in $ENV is being executed
(if you're not expecting it to be, there is no issue with why it is not,
and you would not be concerned) as...
| though I see it is already quelled by read_profile(),
No, it isn't, unless you use -q, which exists for exactly that purpose -
but which has unfortunate side effects which are hard to overcome,
so I do not recommend it.
| so something stronger would probably be needed for profile and
| ENV file debugging.
I doubt anything "stronger" is needed there.
| Sorry, I don't think that is a better solution at all. It's a bad
| (inefficient) hack. Change for the better is fine, but not change for
| the worse.
It would be reliable, and use only defined mechanisms. Your current
method is even more broken than you suspect, it really is only working
correctly because of a whole set of ksh design choices, some of which
are unlikely to ever change in ksh, but others which easily could.
For example, to use your explanation of why your hack (which is
what it is) works, as you included it in PR42828 in 2010 ...
First:
# It does work with all the versions of Ksh I've run across,
# including ksh-85 and the one AIX-3.2, as well as pdksh.
that I am not disputing, for for most of the rest of this, let's assume
the shell evaluating the expression is not ksh or something which
has deliberately copied it.
# Note that any string
# variable can be accessed as an array with one element in the
Not if it was a shell with arrays that I had implemented (as unlikely as
that is to ever happen). Accessing a regular var as an array is more
likely to be a mistake, and would quite likely just generate an error
message. The variable would need to be declared (somehow) as
an array for it to be able to be accessed that way.
Further, I am not sure I would use the conventional '[' ']' to indicate
array subscripts are enclosed - I might decide to use '(' and ')'
instead (as in fortran and tcl) as name(xxx) has no current defined
meaning in sh (whether in a var de-ref or elsewhere) whereas
when the $ is missing, name[xxx] is just a glob expression. The
only place where a name (that is, not a keyword) followed by a
'(' now has any meaning at all in sh, is in a function definition, and
in one of those the '(' is always followed by a ')' (with only white
space intervening) which is not useful as an array index - that
must be something, even if it evaluates to a null string).
That is, when he added arrays, I think Korn got the syntax wrong.
But then, he was trying to make it look good, and familiar, to
C and csh programmers, for whom '[' ']' is the natural choice,
even if it isn't really for sh.
# Also remember that arithmetic expressions are evaluated
# inside array index references (i.e. inside the square brackets)
Again, not if I were implementing arrays - there would only be one
type, associative arrays, which could be indexed by any string,
which just might happen to be an integer for some uses. Of course
overcoming this one is as simple as enclosing the expression in
$(( )) to force arithmetic evaluation, so this just requires a minor
variation. This "automatic arithmetic evaluation" is disgusting.
# The flags variable ($-) is the key to forming our magic
# expression. If the shell is interactive, the flags will
# contain an 'i'.
Sorry, not required at all. There is no 'i' flag or option in a purely
posix shell. Just about all shells have it, so this is a fairly safe
assumption, but it is certainly not required to happen. (Not all shells
allow the 'i' flag, to be altered, but that's not relevant here.)
In the following, I have, I hope, undone the QP encoding that
persists in the PR (don't we all just love gnats!) but if I manage
to miss some, or do it incorrectly, my apologies...
# (_$-=1) This creates a variable named "_BLAH", where
# "BLAH" is the value of the variable "$-", and
# assigns it the value "1". The expression has
# the result "1" as well.
That's correct.
# (_=0) This creates a variable named "_" and assigns
# it the value "0". The expression has the result
# "0" as well.
You'd think that would be OK, and safe, but $_ is kind of magic and
almost anything is possible. Especially in ksh which seems to have
about 50 different uses for it, and add more every new revision...
The "assigns 0 to it" part is likely to work OK, even ksh is unlikely to
break that - but arithmetic is supposed to be evaluated according to
how it would be done in C, and in C the value of an assignment operator
is the value of the variable after the assignment. Even assigning 0 to _
does not guarantee that $_ will expand to 0. Especially in ksh.
Probably this will work, but no guarantees.
Of course, this one is also easy to fix, you just use __ instead of _
(or _X for any other letter, or digit, that you choose) throughout the
expression, or anything else unlikely to matter instead of the _.
# (_$- != _${-%%*i*}) This compares the value of _BLAH with
# the value of another variable that
# will either have the same name (and
# thus the same value), or the name "_".
Same problem with $_ but otherwise that's correct. Of course,
BLAH might be the empty string, so it would be comparing $_ to $_
in all cases, but this version of that expression anticipates that
# For an interactive shell these variables will be guaranteed
# to have different names
Outside ksh, that's not necessarily true. If there's no 'i' in $-
(ever) then the two vars will always be the same.
But assuming the 'i' flag does exist, as is very common, the complete
expression (after variable substitution) turns out to be either ...
# (_BLAH=1) + (_=0) - (_BLAH != _BLAH)
or
# (_BLAH=1) + (_=0) - (_BLAH != _)
(the latter if $- contains 'i'). That looks like it should be
# (1) + (0) - (1 != 1)
or
# (1) + (0) - (1 != 0)
which is 1 in the first case (no -i flag) and 0 in the second (-i)
which is what you want.
Except that part is not guaranteed (even ignoring the issues
accessing the _ variable, and the perhaps non-existent 'i' flag)
First, neither C, nor posix, guarantees anything about the order of
evaluation of subexpressions in an expression like this, a shell could
just as easily evaluate the sub-expressions right to left as left to right.
In the first of them this makes no difference at all, as whatever value
_BLAH has, it will not compare unequal to itself, so the third term is
always 0 there, and that case works.
But in the second one, we do not know what the relationship between
the values of _BLAH and _ will be when that expression is evaluated.
_ may not even be numeric, and you might get an arithmetic error (and
POSIX requires that an error message be issued in such a case, and
evaluation abort). Bash on the other hand (maybe ksh as well) will,
I think, just try to treat the value as another var name, and keep hunting
till it finds either a number, or an unset var (which is interpreted as 0).
The most likely outcome for this, if evaluated this way, is 0 != 0 which
is not what you want.
But even if you think that order of evaluation (which I suspect that even
ksh does not guarantee) is unlikely to be anything except left to right in
an expression like this, it has another problem. Neither POSIX nor C
require that the assignments to the variables be made during evaluation
of the expression, merely that the values of the variables, after the
expression has been evaluated, be as set by the expression. That's
a part of the fallout from unordered sub-expr evaluation, but is worthy
of specific mention, as it means that you cannot expect an assignment
to any var to be available for use anywhere else in the same expression.
So:
# Since
# the first variable has been assigned the value "1" and the
# second one has the value "0", result of this last expression
# will always be 1 for any interactive shell.
is simply not guaranteed, and even a simple change (perhaps
optimisation) to how ksh evaluates arithmetic could make that fail.
# Remember class, the test is tomorrow!
And it always helps if the instructor gets the correct answer, and
is not mistaken about the way that things are defined to work, as
distinct from the way they just happen to work.
| The only good solution is to do something smart during the runtime
| expansion of $ENV to prevent the open from occurring or at least from
| succeeding,
Personally, I think the number 1 rule in deciding what is good is to
be correct first, and then once you're correct, you can worry about
finding a way to be more efficient. Being fast, but wrong, is not
anyone I know's definition of good.
kre
ps: bosh (which prides itself on being a true SysVR4 /bin/sh clone,
with some more recent posix additions), dash, the FreeBSD sh, and ours,
all give a "bad substitution" type error when asked to evaluate your
$ENV expression, though that might not appear in all (or perhaps even
any) when it is de-referenced at sh startup, either because they do not
param expand $ENV (in which case they will be attempting to open a file
of that name in $PWD) or by just suppressing the errors at that point.
From: "Greg A. Woods" <woods@planix.ca>
To: NetBSD GNATS <gnats-bugs@NetBSD.org>
Cc:
Subject: Re: bin/53919: Please suppress all possible error messages that might arise from the $ENV expansion and use by /bin/sh at startup
Date: Wed, 30 Jan 2019 14:05:43 -0800
--pgp-sign-Multipart_Wed_Jan_30_14:05:16_2019-1
Content-Type: text/plain; charset=US-ASCII
Perhaps we can change the subject of this PR, and the intent, to:
Please suppress all possible error messages that might arise
from the $ENV expansion and use by /bin/sh at startup
At Wed, 30 Jan 2019 12:10:01 +0000 (UTC), Robert Elz <kre@munnari.OZ.AU> wrote:
>
> [[....]]
> Not if it was a shell with arrays that I had implemented
> [[....]]
I guess you don't like Lua either? :-) (i.e. in addition to Ksh?)
> ps: bosh (which prides itself on being a true SysVR4 /bin/sh clone,
> with some more recent posix additions), dash, the FreeBSD sh, and ours,
> all give a "bad substitution" type error when asked to evaluate your
> $ENV expression, though that might not appear in all (or perhaps even
> any) when it is de-referenced at sh startup, either because they do not
> param expand $ENV (in which case they will be attempting to open a file
> of that name in $PWD) or by just suppressing the errors at that point.
Well, the part from "though" onward is kind of exactly my point,
especially if you include the /bin/sh in all NetBSD releases to date,
and you exclude all those that don't expand $ENV on non-interactive
startup (bosh, for example, only uses $ENV on interactive startup, as,
FYI, does Bash when invoked as "sh"). As an aside I'll note that
expanding $ENV on non-interactive startup seems rather extremely rare in
the shell-as-/bin/sh business -- in fact I can't find any other example
where that happens other than systems where something very compatible
with AT&T Ksh itself has been installed as /bin/sh (even OpenBSD long
ago took away the expansion of $ENV on non-interactive shells for their
ksh, which is also installed as /bin/sh). This is of course no doubt
due to the "when and only when an interactive shell is invoked" language
in POSIX. It is pretty strong language, for a standard; though I agree
in principle with your original argument from PR42828.
Anyways, the appearance of unwanted output (on either stdout or stderr)
during the expansion and use of $ENV (i.e. the automated use, during
startup) is what actually makes NetBSD-current /bin/sh entirely unusable
and broken for anyone who uses Ksh or its many compatible derivatives
(and expects who and uses traditional Ksh semantics and syntax for
$ENV). All the rest is bikeshedding (well, of course the security
implications are important, but a somewhat different kind of issue
entirely as they don't affect the fundamental usability of an isolated
system per se).
--
Greg A. Woods
Planix, Inc.
<woods@planix.com> +1 250 762-7675 http://www.planix.com/
--pgp-sign-Multipart_Wed_Jan_30_14:05:16_2019-1
Content-Type: application/pgp-signature
Content-Transfer-Encoding: 7bit
Content-Description: OpenPGP Digital Signature
-----BEGIN PGP SIGNATURE-----
iF0EABECAB0WIQRuK6dmwVAucmRxuh9mfXG3eL/0fwUCXFIfoAAKCRBmfXG3eL/0
f43gAJ9I0oIPEGi84pJFbHMuVgq//wS71ACcCP+k0lI9eAjI3k8yk6BLIHK0BxI=
=0qZg
-----END PGP SIGNATURE-----
--pgp-sign-Multipart_Wed_Jan_30_14:05:16_2019-1--
From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: bin/53919: Please suppress all possible error messages that might arise from the $ENV expansion and use by /bin/sh at startup
Date: Thu, 31 Jan 2019 18:02:53 +0700
Date: Wed, 30 Jan 2019 22:10:00 +0000 (UTC)
From: "Greg A. Woods" <woods@planix.ca>
Message-ID: <20190130221000.DC6347A1F7@mollari.NetBSD.org>
| Perhaps we can change the subject of this PR, and the intent, to:
I doubt a Subject: change is really needed, and the intent has
obviously already changed.
| I guess you don't like Lua either? :-)
Actually, no idea, I have never taken the time to look at it
(for no other reason than that.)
| (i.e. in addition to Ksh?)
Bourne's sh was a masterpiece of design, and given the constraints
of making it work on a PDP-11, not a bad implementation either (though
we are still suffering the after effects of some of what those
limitations caused.)
Korn's additions are mostly (not all) crap (design).
| Well, the part from "though" onward is kind of exactly my point,
| especially if you include the /bin/sh in all NetBSD releases to date,
Which did not expand ENV - even though it was supposed to according to
POSIX.
| and you exclude all those that don't expand $ENV on non-interactive
| startup (bosh, for example, only uses $ENV on interactive startup,
For your usage that should really make no difference, the error just
occurs more frequently. If you were to somehow guarantee that you'd
never start an interactive /bin/sh then I guess that might be more
recevant, but I am not sure how you could do that.
Further, the whole point of your ENV hack is because non-interactive
shells expand $ENV and run the script, the idea is to suppress that.
Doing it using non-standard mechanisms is bound to lead to more
problems later, on some system, with some other shell.
| Anyways, the appearance of unwanted output (on either stdout or stderr)
| during the expansion and use of $ENV (i.e. the automated use, during
| startup) is what actually makes NetBSD-current /bin/sh entirely unusable
I have (locally) made changes that avoid the errors. They seem to
work... If no-one objects, I will commit those sometime soon. (It
is not a complex change).
kre
From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: bin/53919: please revert changes that make /bin/sh evaluate $ENV
non-interactively
Date: Fri, 1 Feb 2019 14:46:18 +0000
On Wed, Jan 30, 2019 at 12:10:01PM +0000, Robert Elz wrote:
> The other, yes, I could suppress the error msgs when an expansion fails,
> though in that case, I think I'd tend to make the result be something
> safe, like "" rather than the original value. I will wait on opinions
> from others before making any changes here though, as I believe that
> some people are of the opinion that errors should always be reported.
I'll stick my oar in here to note that while I'm more or less one of
those, I also think the $ENV behavior is fundamentally flawed and
should never be used; I don't really care if additional hacks
agglomerate around it as long as they don't impede normal operation.
--
David A. Holland
dholland@netbsd.org
From: "Robert Elz" <kre@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/53919 CVS commit: src/bin/sh
Date: Mon, 4 Feb 2019 11:16:41 +0000
Module Name: src
Committed By: kre
Date: Mon Feb 4 11:16:41 UTC 2019
Modified Files:
src/bin/sh: error.c error.h eval.c eval.h main.c parser.c sh.1
Log Message:
PR bin/53919
Suppress shell error messages while expanding $ENV (which also causes
errors while expanding $PS1 $PS2 and $PS4 to be suppressed as well).
This allows any random garbage that happens to be in ENV to not
cause noise when the shell starts (which is effectively all it did).
On a parse error (for any of those vars) we also use "" as the result,
which will be a null prompt, and avoid attempting to open any file for ENV.
This does not in any way change what happens for a correctly parsed command
substitution (either when it is executed when permitted for one of the
prompts, or when it is not (which is always for ENV)) and commands run
from those can still produce error output (but shell errors remain suppressed).
To generate a diff of this commit:
cvs rdiff -u -r1.42 -r1.43 src/bin/sh/error.c
cvs rdiff -u -r1.21 -r1.22 src/bin/sh/error.h
cvs rdiff -u -r1.170 -r1.171 src/bin/sh/eval.c
cvs rdiff -u -r1.22 -r1.23 src/bin/sh/eval.h
cvs rdiff -u -r1.80 -r1.81 src/bin/sh/main.c
cvs rdiff -u -r1.164 -r1.165 src/bin/sh/parser.c
cvs rdiff -u -r1.217 -r1.218 src/bin/sh/sh.1
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->feedback
State-Changed-By: kre@NetBSD.org
State-Changed-When: Mon, 04 Feb 2019 12:26:29 +0000
State-Changed-Why:
Does this satisfy your requirements now?
From: "Greg A. Woods" <woods@avoncote.ca>
To: NetBSD GNATS <gnats-bugs@NetBSD.org>
Cc:
Subject: Re: bin/53919 (please revert changes that make /bin/sh evaluate $ENV non-interactively)
Date: Fri, 22 Feb 2019 12:30:27 -0800
--pgp-sign-Multipart_Fri_Feb_22_12:30:10_2019-1
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
At Mon, 4 Feb 2019 12:26:29 +0000 (UTC), kre@NetBSD.org wrote:
Subject: Re: bin/53919 (please revert changes that make /bin/sh evaluate $E=
NV non-interactively)
>=20
> Synopsis: please revert changes that make /bin/sh evaluate $ENV non-inter=
actively
>=20
> State-Changed-From-To: open->feedback
> State-Changed-By: kre@NetBSD.org
> State-Changed-When: Mon, 04 Feb 2019 12:26:29 +0000
> State-Changed-Why:
> Does this satisfy your requirements now?
Hi, I'm very sorry for the delay. I've been busy with work and other
things, including a buggy upgrade on my build server and its Xen dom0
host, though I'm getting close to sorting those issues out.
=46rom your description of the fix, and from your diff, it looks like this
will work very well for me. However I've as yet still been unable to do
a build from the update and test it.
--=20
Greg A. Woods
Planix, Inc.
<woods@planix.com> +1 250 762-7675 http://www.planix.com/
--pgp-sign-Multipart_Fri_Feb_22_12:30:10_2019-1
Content-Type: application/pgp-signature
Content-Transfer-Encoding: 7bit
Content-Description: OpenPGP Digital Signature
-----BEGIN PGP SIGNATURE-----
iF0EABECAB0WIQRuK6dmwVAucmRxuh9mfXG3eL/0fwUCXHBb2gAKCRBmfXG3eL/0
f4W2AKD86Ik5/FI385WGnSD9bSG+aRoCSQCgkC9hgr/8RO+xbdsO60s2Ht+i1V8=
=Sahy
-----END PGP SIGNATURE-----
--pgp-sign-Multipart_Fri_Feb_22_12:30:10_2019-1--
From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: bin/53919 (please revert changes that make /bin/sh evaluate $ENV non-interactively)
Date: Sat, 23 Feb 2019 05:55:05 +0700
Date: Fri, 22 Feb 2019 20:35:01 +0000 (UTC)
From: "Greg A. Woods" <woods@avoncote.ca>
Message-ID: <20190222203501.20CC27A1D0@mollari.NetBSD.org>
| However I've as yet still been unable to do
| a build from the update and test it.
There is no hurry, whenever you get an opportunity.
kre
State-Changed-From-To: feedback->closed
State-Changed-By: kre@NetBSD.org
State-Changed-When: Fri, 31 Jan 2020 17:38:55 +0000
State-Changed-Why:
Feedback timeout, and issue believed to have been adequately addressed.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.46 2020/01/03 16:35:01 leot Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2020
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home