NetBSD Problem Report #42828
From www@NetBSD.org Tue Feb 16 17:50:08 2010
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by www.NetBSD.org (Postfix) with ESMTP id 0EAC763C356
for <gnats-bugs@gnats.NetBSD.org>; Tue, 16 Feb 2010 17:50:08 +0000 (UTC)
Message-Id: <20100216175005.DCB1063C142@www.NetBSD.org>
Date: Tue, 16 Feb 2010 17:50:05 +0000 (UTC)
From: rhansen@bbn.com
Reply-To: rhansen@bbn.com
To: gnats-bugs@NetBSD.org
Subject: Almquist shell always evaluates the contents of ${ENV} even if non-interactive
X-Send-Pr-Version: www-1.0
>Number: 42828
>Category: standards
>Synopsis: Almquist shell always evaluates the contents of ${ENV} even if non-interactive
>Confidential: no
>Severity: non-critical
>Priority: medium
>Responsible: standards-manager
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Tue Feb 16 17:55:00 +0000 2010
>Closed-Date: Wed Dec 05 08:21:06 +0000 2018
>Last-Modified: Wed Dec 05 08:21:06 +0000 2018
>Originator: Richard Hansen
>Release: 5.0
>Organization:
>Environment:
NetBSD deer-netbsd.bbn.com 5.0_STABLE NetBSD 5.0_STABLE (GENERIC) #4: Tue Feb 2 10:43:58 EST 2010 gdt@fnord.ir.bbn.com:/n0/obj/gdt-5/amd64/sys/arch/amd64/compile/GENERIC amd64
>Description:
NetBSD 5.0's /bin/sh always evaluates the contents of the file identified by ${ENV} even if the shell is non-interactive. This has a few consequences:
* NetBSD's /bin/sh is not compliant with POSIX:2004 [1] or POSIX:2008 [2] (at least)
* shell script execution is unnecessarily slow if the ${ENV} file does a lot of stuff
* if the ${ENV} file executes a shell script, or executes something that executes a shell script, you'll get a fork bomb
[1] <http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_05_03>
[2] <http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_05_03>
>How-To-Repeat:
$ printf '#!/bin/sh\necho bar\n' > bar
$ chmod a+x bar
$ echo './bar' > foo
$ ENV=$(pwd)/foo /bin/sh -i
./bar: Cannot vfork
bar
bar
bar
bar
^C
$
>Fix:
The following patch is completely untested (I haven't even compiled it yet). I'm not sure when iflag is initialized, so this may not be the correct fix:
Index: bin/sh/main.c
===================================================================
RCS file: /cvsroot/src/bin/sh/main.c,v
retrieving revision 1.53
diff -u -p -r1.53 main.c
--- bin/sh/main.c 18 Jan 2009 00:30:54 -0000 1.53
+++ bin/sh/main.c 8 Feb 2010 06:08:26 -0000
@@ -184,7 +184,7 @@ state1:
}
state2:
state = 3;
- if (getuid() == geteuid() && getgid() == getegid()) {
+ if (iflag && getuid() == geteuid() && getgid() == getegid()) {
if ((shinit = lookupvar("ENV")) != NULL && *shinit != '\0') {
state = 3;
read_profile(shinit);
>Release-Note:
>Audit-Trail:
From: Christos Zoulas <christos@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/42828 CVS commit: src/bin/sh
Date: Wed, 17 Feb 2010 10:57:24 -0500
Module Name: src
Committed By: christos
Date: Wed Feb 17 15:57:24 UTC 2010
Modified Files:
src/bin/sh: main.c
Log Message:
PR/42828: Richard Hansen: Don't evaluate ${ENV} if not interactive.
http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_05_03
To generate a diff of this commit:
cvs rdiff -u -r1.53 -r1.54 src/bin/sh/main.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: PR/42828 CVS commit: src/bin/sh
Date: Wed, 17 Feb 2010 13:51:26 -0500
On Wed, 17 Feb 2010 16:00:07 +0000 (UTC)
Christos Zoulas <christos@netbsd.org> wrote:
> Modified Files:
> src/bin/sh: main.c
>
> Log Message:
> PR/42828: Richard Hansen: Don't evaluate ${ENV} if not interactive.
> http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_05_03
>
>
> To generate a diff of this commit:
> cvs rdiff -u -r1.53 -r1.54 src/bin/sh/main.c
Thanks for the fix;
would this be worthy of a pullup to netbsd-5*? I've seen occasional
user complaints about this, although if being careful and adding
interactive-intended code inside "case "$-" in *i*)" this wasn't a
major problem. But I myself got bitten by this long ago when I
switched from another shell to /bin/sh, yet I didn't know this was a
standards compatibility issue.
Thanks,
--
Matt
From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, standards-manager@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, rhansen@bbn.com
Cc:
Subject: Re: PR/42828 CVS commit: src/bin/sh
Date: Wed, 17 Feb 2010 14:25:35 -0500
On Feb 17, 6:55pm, mm_lists@pulsar-zone.net (Matthew Mondor) wrote:
-- Subject: Re: PR/42828 CVS commit: src/bin/sh
| The following reply was made to PR standards/42828; it has been noted by GNATS.
|
| From: Matthew Mondor <mm_lists@pulsar-zone.net>
| To: gnats-bugs@NetBSD.org
| Cc:
| Subject: Re: PR/42828 CVS commit: src/bin/sh
| Date: Wed, 17 Feb 2010 13:51:26 -0500
|
| On Wed, 17 Feb 2010 16:00:07 +0000 (UTC)
| Christos Zoulas <christos@netbsd.org> wrote:
|
| > Modified Files:
| > src/bin/sh: main.c
| >
| > Log Message:
| > PR/42828: Richard Hansen: Don't evaluate ${ENV} if not interactive.
| > http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_05_03
| >
| >
| > To generate a diff of this commit:
| > cvs rdiff -u -r1.53 -r1.54 src/bin/sh/main.c
|
| Thanks for the fix;
|
| would this be worthy of a pullup to netbsd-5*? I've seen occasional
| user complaints about this, although if being careful and adding
| interactive-intended code inside "case "$-" in *i*)" this wasn't a
| major problem. But I myself got bitten by this long ago when I
| switched from another shell to /bin/sh, yet I didn't know this was a
| standards compatibility issue.
Sure thing.
christos
From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: standards/42828: Almquist shell always evaluates the contents of ${ENV} even if non-interactive
Date: Thu, 18 Feb 2010 05:36:43 +0700
Note that this change makes sh fail to live up to its documentation ...
If the environment variable ENV is set on entry to a shell, or is set in
the .profile of a login shell, the shell next reads commands from the
file named in ENV. Therefore, a user should place commands that are to
be executed only at login time in the .profile file, and commands that
are executed for every shell inside the ENV file. To set the ENV vari-
able to some file, place the following line in your .profile of your home
directory
ENV=$HOME/.shinit; export ENV
substituting for ``.shinit'' any filename you wish. Since the ENV file
is read for every invocation of the shell, including shell scripts and
non-interactive shells, the following paradigm is useful for restricting
commands in the ENV file to interactive invocations. Place commands
within the ``case'' and ``esac'' below (these commands are described
later):
case $- in *i*)
# commands for interactive use only
...
esac
In particular, with this being a significant change, it is definitely
not suitable for a pullup to NetBSD 5 (or any existing released system).
Further, this is a case where sanity ought to win over standards, there's
lots of stuff where our shell isn't the same as wha POSIX says it should
be, some of considerably more importance than this one.
It is trivial (as the doc says) to make the script be a no-op in
non-interactive shells, if that's what's needed. On the other hand,
if ${ENV} isn't run when *every* shell starts, what is? If there's
nothing, ten all that's left is what's in the environment to tailor
the shell, and not everything can be done there, and even when it can,
passing around all kinds of shell specific noise in the environment, to
every process that runs, isn't a sane way of operating.
What's more, "ENV" is the name of the env var that sets the startup
script for everthing that's a basically Bourne compatible shell (though
that shell had no similar concept, of course) - but not all shells are
the same, and stuff that fits one shell won't fit. The effect of this
is that it isn't sane to put just any random script as $ENV, the script
needs to be aware of why it will be invoked, and in what possible
environments, and do sane things (for which, running other shell scripts,
without resetting ENV first, is not included).
I'd put this pack the way it was. Screw POSIX in this case.
kre
From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, standards-manager@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, rhansen@bbn.com
Cc:
Subject: Re: standards/42828: Almquist shell always evaluates the contents of ${ENV} even if non-interactive
Date: Wed, 17 Feb 2010 18:43:02 -0500
On Feb 17, 10:50pm, kre@munnari.OZ.AU (Robert Elz) wrote:
-- Subject: Re: standards/42828: Almquist shell always evaluates the contents
| I'd put this pack the way it was. Screw POSIX in this case.
|
| kre
I agree with Robert here and I vote to put it back.
christos
From: Richard Hansen <rhansen@bbn.com>
To: gnats-bugs@NetBSD.org
Cc: standards-manager@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org
Subject: Re: standards/42828: Almquist shell always evaluates the contents
of ${ENV} even if non-interactive
Date: Thu, 18 Feb 2010 02:14:07 -0500
Robert Elz wrote:
> In particular, with this being a significant change, it is definitely
> not suitable for a pullup to NetBSD 5 (or any existing released system).
Agreed.
> On the other hand,
> if ${ENV} isn't run when *every* shell starts, what is? If there's
> nothing, ten all that's left is what's in the environment to tailor
> the shell, and not everything can be done there, and even when it can,
> passing around all kinds of shell specific noise in the environment, to
> every process that runs, isn't a sane way of operating.
I agree that the POSIX design is suboptimal, but at this point there is
greater value in being compatible than in being right.
Cross-platform scripts can't rely on NetBSD's nonstandard behavior, so
they won't take advantage of it. NetBSD-targeted scripts that require
the nonstandard behavior would be better served by an extension
specifically for that purpose (e.g., dot ~/.nbshrc upon every shell
invocation).
It would be unfortunate if an ordinary script ran everywhere except NetBSD.
> What's more, "ENV" is the name of the env var that sets the startup
> script for everthing that's a basically Bourne compatible shell (though
> that shell had no similar concept, of course) - but not all shells are
> the same, and stuff that fits one shell won't fit. The effect of this
> is that it isn't sane to put just any random script as $ENV,
These are arguments for standards compliance, not against.
> I'd put this pack the way it was. Screw POSIX in this case.
I think it should be kept, just not pulled into netbsd-5* in order to
minimize surprises. Fixing this bug might raise some issues, but I
believe the long-term benefit from standards compliance will outweigh
the short-term pain.
-Richard
From: Robert Elz <kre@munnari.OZ.AU>
To: Richard Hansen <rhansen@bbn.com>
Cc: gnats-bugs@NetBSD.org, standards-manager@NetBSD.org,
netbsd-bugs@NetBSD.org
Subject: Re: standards/42828: Almquist shell always evaluates the contents of ${ENV} even if non-interactive
Date: Thu, 18 Feb 2010 19:53:25 +0700
Date: Thu, 18 Feb 2010 02:14:07 -0500
From: Richard Hansen <rhansen@bbn.com>
Message-ID: <4B7CE8BF.3020401@bbn.com>
| I agree that the POSIX design is suboptimal, but at this point there is
| greater value in being compatible than in being right.
In some cases I might agree, in this one, I don't.
| Cross-platform scripts can't rely on NetBSD's nonstandard behavior,
One of us is confused about what $ENV is for? What scripts are ever going to
set that (clear(unset) it perhaps, but set it?) Particularly the way you
expect it to work - where it applies only to interactive shells, which isn't
something many scripts ever care about.
Do you have a real example of something that breaks (other than a conformance
test) because of this? That is, excluding user environments, we know they're
different.
| It would be unfortunate if an ordinary script ran everywhere except NetBSD.
Yes, but there are thousands of scripts around, the only problems I have ever
seen reported relate to scripts that use non-standard shell features and
occasionally, an undisputed bug. I've never seen a report of a problem
that related to $ENV and a script.
| but I believe the long-term benefit from standards compliance will outweigh
| the short-term pain.
Better long term benefit would be to convince the standard to be rational.
We don't achieve that (not any hope of achieving that) if everyone just does
what they say because they say it, and not because it is the right thing to
do. Following the standards is great when it is a matter of a choice between
two ways of which we could pick either - and sometimes when it is a matter of
adding something that we don't really see the need for, but does no real harm
to have - but never when their design is flawed.
kre
From: Richard Hansen <rhansen@bbn.com>
To: Robert Elz <kre@munnari.OZ.AU>
Cc: gnats-bugs@NetBSD.org, standards-manager@NetBSD.org,
netbsd-bugs@NetBSD.org
Subject: Re: standards/42828: Almquist shell always evaluates the contents
of ${ENV} even if non-interactive
Date: Fri, 19 Feb 2010 14:36:53 -0500
>> Cross-platform scripts can't rely on NetBSD's nonstandard behavior,
>
> One of us is confused about what $ENV is for? What scripts are ever going to
> set that (clear(unset) it perhaps, but set it?)
My .profile sets ENV. (By "script" I meant a file containing
environment setup commands. That was not the best word to use, but I
couldn't think of a better word and I thought that context would make
the meaning clear. I apologize for the imprecision. In the future I'll
use "script" to only refer to executable files beginning with the
'#!/bin/sh' shebang.)
> Particularly the way you expect it to work
It's not just me; everyone under the impression that NetBSD conforms to
POSIX also expects it.
> - where it applies only to interactive shells, which isn't
> something many scripts ever care about.
>
> Do you have a real example of something that breaks (other than a conformance
> test) because of this? That is, excluding user environments, we know they're
> different.
Why exclude user environments? The purpose of ${ENV} is to customize
the user's interactive shell.
This bug broke my dot files, which is how I discovered it. I just
installed NetBSD for the first time the other day, and my first
experience was watching it lock up after copying my dot files from a
Linux machine. The freeze was thanks to a fork bomb that wouldn't have
happened had NetBSD's shell conformed to the POSIX spec. Ctrl-C
eventually worked, but only after several minutes of unwinding the
process stack.
>> but I believe the long-term benefit from standards compliance will outweigh
>> the short-term pain.
>
> Better long term benefit would be to convince the standard to be rational.
Good point.
> We don't achieve that (not any hope of achieving that) if everyone just does
> what they say because they say it, and not because it is the right thing to
> do.
Likewise, the standard won't change because the NetBSD community
silently resists it; someone has to fight for change. I don't have the
time to champion the cause. Would you be willing to take this up with
the POSIX working group?
> Following the standards is great when it is a matter of a choice between
> two ways of which we could pick either - and sometimes when it is a matter of
> adding something that we don't really see the need for, but does no real harm
> to have - but never when their design is flawed.
I agree that the POSIX design is limited relative to the NetBSD design,
but I would not call it flawed. Suboptimal perhaps, but not flawed.
Flawed implies that there are legitimate use cases that are not
supported, but I have not yet come across any such use cases. Can you
provide an example where a user needs ${ENV} evaluated upon invocation
of every non-interactive shell?
Perhaps the flaw is in NetBSD's design: The ${ENV} file can arbitrarily
modify the shell execution environment (change the working directory,
modify positional parameters, close file descriptors, trap signals,
define functions, set shell options, etc.). This means that shell
script writers can't rely on a consistent initial environment. It is
doubtful this would be a significant problem in practice, but in theory
this could lead to bugs that appear to be in a script file but are
actually caused by the user's ${ENV} file.
I want to pause and wax introspective for a moment. A number of BSD
fans give Linux users a hard time, but due to this bug my first
impression was that NetBSD is half-baked; that it fails to conform to
even the pedestrian parts of POSIX. I know that bugs happen so I'm
willing to forgive flaws and even contribute to fixes, but only if the
community respectfully deals with legitimate user complaints. The
antagonistic response I've received to this bug is disheartening and
makes me disinclined to contribute to NetBSD in the future.
-Richard
From: Robert Elz <kre@munnari.OZ.AU>
To: Richard Hansen <rhansen@bbn.com>
Cc: gnats-bugs@NetBSD.org, standards-manager@NetBSD.org,
netbsd-bugs@NetBSD.org
Subject: Re: standards/42828: Almquist shell always evaluates the contents of ${ENV} even if non-interactive
Date: Sun, 21 Feb 2010 20:52:23 +0700
Date: Fri, 19 Feb 2010 14:36:53 -0500
From: Richard Hansen <rhansen@bbn.com>
Message-ID: <4B7EE855.1050108@bbn.com>
| My .profile sets ENV. (By "script" I meant a file containing
| environment setup commands.
Ah, OK, fine (my .profile sets ENV as well...)
| Why exclude user environments? The purpose of ${ENV} is to customize
| the user's interactive shell.
Because it is a different degree of problem than I thought you were
suggesting. With the use of $ENV that we're agreeing on, you get to
decide what's in it, and you know the environments where it will be used,
and you can read the doc for those environments and make sure that your
setup applies there.
Now that you know how NetBSD applies $ENV, I would assume that you have
no further problems - ie: for this usage, the difference from other systems
is a minor annoyance, not a serious impediment.
On the other hand, for "scripts" with the other (more common usage),
the author doesn't know all the environments where the script will be
used, and for that, it is far more important that anything that is
reasonably relying upon some standard behaviour gets what they're expecting.
If you had examples of the latter where NetBSD's application of $ENV
was a problem I'd be much more tempted to agree to a change, for the sake of
compatibility, for just user setup files, I'm not.
| Would you be willing to take this up with the POSIX working group?
I don't have the time either ... and in any case, I've pretty much given
up on the current set of standards bodies - they've all become full of
people for whom the standard is what is most important, rather than the
systems that are built using the standard (ie: making the standard itself
seem better looks to me to be of greater importance than making the system
that uses the standard actually work better.)
| Flawed implies that there are legitimate use cases that are not
| supported,
flawed just implies there is a defect - something that could be improved,
or which is less than perfect.
| but I have not yet come across any such use cases. Can you
| provide an example where a user needs ${ENV} evaluated upon invocation
| of every non-interactive shell?
"of every non-interactive shell" ? No, of course not, but that's not
what is needed - the converse of "only used by non-interactive shells"
is "used by at least one non-interactive shell" (ie: the opposite of
"none" is "one" not "all").
And that's easy - I make scripts (personal use scripts) all the time
by simply cut and paste from my history - that is, whenever I detect that
I'm going to re-execute a set of commands that I have done in the not
too distant past, I assume that if I'm doing it twice, I'm going to be
doing it again, so I simply take what I did and stick it in a script, then
run the script. That's a non-interactive shell running the script, and
in order to run correctly, it needs to run with the same environment that
my interactive shell where I first ran the commands had - that is, it
needs to have access to whatever is in ${ENV} because my interactive shell
had that access as well.
And yes, of course, I can ". ${ENV}" in the script, but then I need to
remember I have to add that, every time.
Another use, that is less common for me, but I have done on occasion,
is to define shell functions that replace standard commands, and either do
something compatible, but in a different way (like turning "rm" into
a "mv" to a trash directory) or add trace/debugging to what is happening,
so allow debugging of complex sets of scripts to find out just where the
script is doing something it shouldn't be doing, and why. For that, I
can set ENV so a suitable environment is established for the scripts that
I want to investigate - but only if they respect it.
| Perhaps the flaw is in NetBSD's design: The ${ENV} file can arbitrarily
| modify the shell execution environment (change the working directory,
| modify positional parameters, close file descriptors, trap signals,
| define functions, set shell options, etc.). This means that shell
| script writers can't rely on a consistent initial environment.
Huh? They can't rely upon most of that anyway - when the shell that starts
the script begins, the environment can be anything. Of your list, about all
that can't be done is trapped signals, but signals can be ignored, which is
close enough to the same, and (I think) defined functions (or can they come
from the environment?), which is one of the very things I want to be able to do.
Any of that that is important to the script needs to be explicitly either
tested, or set, in the script, to ensure the environment is as it expects.
Whether the change came from something done in $ENV, or was established
before the shell running the script began is largely irrelevant.
| It is doubtful this would be a significant problem in practice, but in
| theory this could lead to bugs that appear to be in a script file but are
| actually caused by the user's ${ENV} file.
Yes, of course it can. If you put stuff in there that breaks things, you
get what you deserve (just the same as if you put commands in directories early
in your PATH with standard names but which don't do what the standard commands
with the same names do). You're entitled to break things if you want to.
So is any other user.
| but only if the
| community respectfully deals with legitimate user complaints.
I agree, but you need to understand that there's more than one way to
deal with a problem - simply making every change everyone asks for,
regardless of whether that change makes things better or not would
hardly be a responsible attitude, would it? Nor does it mean that
whatever POSIX says is necessarily correct for us.
I don't think there's been anyone here who has been antagonistic, just
not agreeing with what you're asking - there is a difference.
Lastly, you're mostly just hearing from me, and I do not represent NetBSD,
I'm just offering an opinion - what will actually happens in NetBSD will
end up being decided by someone else.
kre
From: Richard Hansen <rhansen@bbn.com>
To: Robert Elz <kre@munnari.OZ.AU>
Cc: gnats-bugs@NetBSD.org, standards-manager@NetBSD.org,
netbsd-bugs@NetBSD.org
Subject: Re: standards/42828: Almquist shell always evaluates the contents
of ${ENV} even if non-interactive
Date: Mon, 22 Feb 2010 18:07:46 -0500
> Now that you know how NetBSD applies $ENV, I would assume that you have
> no further problems - ie: for this usage, the difference from other systems
> is a minor annoyance, not a serious impediment.
Correct; it's an annoyance and not a serious impediment. I'm arguing
for a fix to remove that annoyance.
> If you had examples of the latter where NetBSD's application of $ENV
> was a problem I'd be much more tempted to agree to a change, for the sake of
> compatibility, for just user setup files, I'm not.
The annoyance is the problem. It's not a severe or urgent problem, but
it is a problem.
I would like to reach consensus on whether the benefits of fixing the
annoyance outweigh the costs. The benefits aren't huge because the
annoyance isn't huge, but I argue that the costs of fixing this are small.
Benefits:
* reduced risk of login shell freezes caused by fork bombs
* faster shell invocation for scripts
* standards compliance/meets user expectations
* reliable initial execution environment for scripts
Costs:
* implementing the fix (already done)
* backward incompatibility (only affects users that have code in
their ${ENV} file that is evaluated by non-interactive shells)
* makes some tasks less convenient
Is there anything I'm forgetting?
>
>> Would you be willing to take this up with the POSIX working group?
>
> I don't have the time either ... and in any case, I've pretty much given
> up on the current set of standards bodies - they've all become full of
> people for whom the standard is what is most important, rather than the
> systems that are built using the standard (ie: making the standard itself
> seem better looks to me to be of greater importance than making the system
> that uses the standard actually work better.)
This criticism brings up a broader question: How much should NetBSD
care about POSIX compliance?
I agree that standards committees can sometimes fail to appreciate
real-world issues in their zeal to create a theoretically better world
(see the controversy around C++98's 'export' feature [1]), but I do
appreciate the uniformity and long-term perspective they provide.
I think a POSIX violation should be classified as a bug. It might not
be fixed if there isn't enough manpower or user interest, but it should
still be considered a bug.
[1] <http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2003/n1426.pdf>
>
>> Flawed implies that there are legitimate use cases that are not
>> supported,
>
> flawed just implies there is a defect - something that could be improved,
> or which is less than perfect.
I like to distinguish "design flaw" from "implementation flaw". I
define a design flaw as something that precludes any implementation from
meeting a requirement (such as a use case that must be supported).
I don't think there is a requirement to evaluate ${ENV} for
non-interactive shells. It might be handy in some cases, but I can't
think of a case where it would be needed. That's why I prefer the word
"suboptimal" over "flawed".
>
>> but I have not yet come across any such use cases. Can you
>> provide an example where a user needs ${ENV} evaluated upon invocation
>> of every non-interactive shell?
>
> "of every non-interactive shell" ? No, of course not, but that's not
> what is needed - the converse of "only used by non-interactive shells"
> is "used by at least one non-interactive shell" (ie: the opposite of
> "none" is "one" not "all").
Let me rephrase by using an example: Suppose a user needed to change
the behavior of one particular shell script. In NetBSD, there are three
ways this might be done:
1. modify the script
2. via ${ENV}
3. change the invoker's environment (e.g., export environment
variables, open/close file descriptors, etc.) before running the script
Option #1 may not be available. Maybe the file is not writable, or
maybe modifications are too impractical (e.g., upgrading the package
providing the script will revert any changes). Option #2 is not
available on other platforms and may have unintended consequences in
other scripts. Option #3 may not be powerful enough.
The big question: Do options #1 and #3 provide enough flexibility in
practice? If so, then option #2 is not needed and we should fix this
bug. Certainly options #1 and #3 are insufficient in come cases, but I
would argue that those cases are rare and the result of an exceptional
flaw that should be dealt with in another way.
>
> And that's easy - I make scripts (personal use scripts) all the time
> by simply cut and paste from my history - that is, whenever I detect that
> I'm going to re-execute a set of commands that I have done in the not
> too distant past, I assume that if I'm doing it twice, I'm going to be
> doing it again, so I simply take what I did and stick it in a script, then
> run the script. That's a non-interactive shell running the script, and
> in order to run correctly, it needs to run with the same environment that
> my interactive shell where I first ran the commands had - that is, it
> needs to have access to whatever is in ${ENV} because my interactive shell
> had that access as well.
>
> And yes, of course, I can ". ${ENV}" in the script, but then I need to
> remember I have to add that, every time.
This is an example of where NetBSD's behavior is handy but not needed.
If this bug was fixed, you could still make your scripts work.
For this particular example, current behavior vs. fixed is a tradeoff
between having to remember to add the following at the top of the ${ENV}
file:
case "$-" in *i*);; *) return;; esac
versus having the remember to add the following to the scripts:
[ -r "${ENV}" ] && . "${ENV}"
Either one is annoying, but fixing it is standards compliant and won't
cause your login shell to lock up if you forget the magic line.
>
> Another use, that is less common for me, but I have done on occasion,
> is to define shell functions that replace standard commands, and either do
> something compatible, but in a different way (like turning "rm" into
> a "mv" to a trash directory) or add trace/debugging to what is happening,
> so allow debugging of complex sets of scripts to find out just where the
> script is doing something it shouldn't be doing, and why. For that, I
> can set ENV so a suitable environment is established for the scripts that
> I want to investigate - but only if they respect it.
This is another example of where NetBSD's behavior is handy but not
needed. The same results can be achieved by prepending a special
directory to PATH and populating that directory with custom versions of
standard commands.
>
>> Perhaps the flaw is in NetBSD's design: The ${ENV} file can arbitrarily
>> modify the shell execution environment (change the working directory,
>> modify positional parameters, close file descriptors, trap signals,
>> define functions, set shell options, etc.). This means that shell
>> script writers can't rely on a consistent initial environment.
>
> Huh? They can't rely upon most of that anyway - when the shell that starts
> the script begins, the environment can be anything.
You're right about file descriptors, but the others have well-defined
initial values. Changing them in ${ENV} could cause problems. Here are
some (somewhat silly) examples to illustrate the potential danger:
* working directory: Initially the current directory matches the
working directory of the invoker. If the user adds 'cd "${HOME}"' to
the ${ENV} file, then every shell script will assume it was invoked from
${HOME}.
* positional parameters: Initially the positional parameters match
the command line arguments passed by the invoker. If the user adds
'shift' somewhere in the ${ENV} file, all shell scripts will ignore the
first command line argument.
* shell options: Initially they're all turned off. If the user adds
'set -C' to ${ENV}, '>' will no longer work as expected for every shell
script.
> Any of that that is important to the script needs to be explicitly either
> tested, or set, in the script, to ensure the environment is as it expects.
Shell scripts could use $- to test for sane shell options, but none of
them do. (Why would they? The POSIX spec says they're all turned off
by default.) Shell scripts can't tell if the current working directory
or positional parameters were modified so there's nothing to test or set.
>> It is doubtful this would be a significant problem in practice, but in
>> theory this could lead to bugs that appear to be in a script file but are
>> actually caused by the user's ${ENV} file.
>
> Yes, of course it can. If you put stuff in there that breaks things, you
> get what you deserve (just the same as if you put commands in directories early
> in your PATH with standard names but which don't do what the standard commands
> with the same names do). You're entitled to break things if you want to.
> So is any other user.
If a user's ${ENV} file follows the POSIX spec, it is reasonable for the
user to expect it to work; the user doesn't "deserve" to have the login
shell lock up.
>
>> but only if the
>> community respectfully deals with legitimate user complaints.
>
> I agree, but you need to understand that there's more than one way to
> deal with a problem - simply making every change everyone asks for,
> regardless of whether that change makes things better or not would
> hardly be a responsible attitude, would it? Nor does it mean that
> whatever POSIX says is necessarily correct for us.
I do not mean for respect to entail capitulation, only that the tone be
professional and not condescending or dismissive.
-Richard
From: Robert Elz <kre@munnari.OZ.AU>
To: Richard Hansen <rhansen@bbn.com>
Cc: gnats-bugs@NetBSD.org, standards-manager@NetBSD.org,
netbsd-bugs@NetBSD.org
Subject: Re: standards/42828: Almquist shell always evaluates the contents of ${ENV} even if non-interactive
Date: Tue, 23 Feb 2010 21:30:38 +0700
Date: Mon, 22 Feb 2010 18:07:46 -0500
From: Richard Hansen <rhansen@bbn.com>
Message-ID: <4B830E42.9060504@bbn.com>
| I would like to reach consensus on whether the benefits of fixing the
| annoyance outweigh the costs.
Yes, that would be good - and that's never going to happen as long as it is
just the two of us having this discussion, so after this message I'm going to
leave it for others to give an opinion, and will (probably) say no more.
| This criticism brings up a broader question: How much should NetBSD
| care about POSIX compliance?
I suspect my opinion on that one is clear, so I'll dodge that bullet...
| I like to distinguish "design flaw" from "implementation flaw".
That's reasonable, but I'm not sure it is material.
| I don't think there is a requirement to evaluate ${ENV} for
| non-interactive shells.
My take is that if ${ENV} isn't wanted for non-interactive shells, it can
make itself a no-op. On the other hand, if it is, other than adding some
other way to do the same thing to the shell, there is no remedy.
| Let me rephrase by using an example: Suppose a user needed to change
| the behavior of one particular shell script. In NetBSD, there are three
| ways this might be done:
|
| 1. modify the script
| 2. via ${ENV}
| 3. change the invoker's environment (e.g., export environment
| variables, open/close file descriptors, etc.) before running the script
|
| Option #1 may not be available. Maybe the file is not writable, or
| maybe modifications are too impractical (e.g., upgrading the package
| providing the script will revert any changes).
Yes, and in cases that sometimes matter, it isn't clear which script needs
to be modified in any case - the whole point of using ${ENV} is so the
modifications apply to every script, which might then allow the actual
change that's needed to be detected.
| The big question: Do options #1 and #3 provide enough flexibility in
| practice?
No, or not without considerable work (one can always replace /bin/sh
with something else, that does whatever is required, and then exec's the
real /bin/sh from its new location - but that's rarely convenient...)
| Certainly options #1 and #3 are insufficient in come cases, but I
| would argue that those cases are rare and the result of an exceptional
| flaw that should be dealt with in another way.
What other way?
| This is an example of where NetBSD's behavior is handy but not needed.
| If this bug was fixed, you could still make your scripts work.
Yes, but ...
| For this particular example, current behavior vs. fixed is a tradeoff
| between having to remember to add the following at the top of the ${ENV}
| file:
|
| case "$-" in *i*);; *) return;; esac
|
| versus having the remember to add the following to the scripts:
|
| [ -r "${ENV}" ] && . "${ENV}"
|
| Either one is annoying, but fixing it is standards compliant and won't
| cause your login shell to lock up if you forget the magic line.
My login shell (nor any other shell) has never locked up, so that isn't
a problem I'm particularly worried about - perhaps just because I'm a little
more conservative about what I have the shell do in ${ENV}
But for those two remedies, the former (modifying ${ENV}) you do exactly
once (once in all time) - the other (adding a line to the script) needs to
be repeated for every script that is created, which can be several times
a day.
That is, this is hardly a "add a line here vs add a line there, which is
better?" debate (if it were, that's exactly the kind of place where following
the standards - ie: doing what others have already decided is best - is the
better way). The alternatives you're suggesting are not comparable that
way however, there's a big difference between a one off, and a repeat forever.
| This is another example of where NetBSD's behavior is handy but not
| needed. The same results can be achieved by prepending a special
| directory to PATH and populating that directory with custom versions of
| standard commands.
No, that doesn't work, built in commands are executed before commands
found in $PATH, so that doesn't work to override stuff like test, let
alone the built in commands that can't go in $PATH (like cd) which happens
to be one of the commands I most often want to override. That can be
done by defining a cd() function, and that works in ${ENV}.
| * working directory: Initially the current directory matches the
| working directory of the invoker. If the user adds 'cd "${HOME}"' to
| the ${ENV} file, then every shell script will assume it was invoked from
| ${HOME}.
Of course, and it would be, and if that's what the user wanted to happen
(which we can assume they must have when they put that in ${ENV}) then
what's the problem? If that causes things to break, the user who added
the cd can remove it (or make it conditional).
| * positional parameters: Initially the positional parameters match
| the command line arguments passed by the invoker. If the user adds
| 'shift' somewhere in the ${ENV} file, all shell scripts will ignore the
| first command line argument.
But those parameters all depend upon how the script is invoked anyway?
I can just invoke any script without its first parameter, can't I?
The script needs to handle that possibility in some rational way, and if
it does that, then I don't see that it matters that ${ENV} could do the
same thing (though in normal circumstances, I don't really see that
as a problem I'd typically worry about - that is, I don't think it's
very likely that a normal ${ENV} will just randomly decide that a shift
or two might be nice...)
On the other hand, the ability to rearrange parameters allows for version
shift variances to be handled: if you have two packages X and Y. where X
needs stuff from Y, upgrades are available for both, and the new X wants
the new Y, but while you need the new X (bug fixes, or new features) you
don't want the new Y (platform not supported, or has new bugs that break
your work, or just cost) then you might (OK, it is a stretch) be able to
massage the exec from X into a script from Y to rearrange the args to
be what the older Y expects rather than what the newer X is handing it.
Perhaps not very plausible, but it is the kind of thing that only works when
it is possible to alter the environment of every script that runs.
| * shell options: Initially they're all turned off. If the user adds
| 'set -C' to ${ENV}, '>' will no longer work as expected for every shell
| script.
And if I run the script as "sh -C script" ???
Or for that matter, if I run the script as "sh -i script" ?
Options are only all off initially if the script is invoked as "./script"
(or whatever) and even that's only true if the #! line doesn't set any
#! /bin/sh -C
should work (but that's not relevant to our discussion).
| Shell scripts could use $- to test for sane shell options, but none of
| them do. (Why would they? The POSIX spec says they're all turned off
| by default.)
By default, yes, but the script is not required to be run "by default".
And while I totally understand that there's no particularly good reason
that a script author needs to necessarily anticipate some idiot running
a script with "sh -C script" when the script doesn't expect noclobber mode,
there's also no real reason to anticipate that someone will do set -C
in ${ENV} (outside a test for an interactive shell).
| Shell scripts can't tell if the current working directory
| or positional parameters were modified so there's nothing to test or set.
They cannot tell that they were modified, but nor can they control the
environment from which they were invoked. If a script needs to be in
a particular directory, or to have some options set, or reset, it needs
to make those things happen itself. It always needs to test its parameters
for sanity (correct number, types, ...) If a script just wants to operate
upon whatever happens to be ${PWD} when it starts, it should just do that,
and whether that was altered as a result of
(cd /some/where && ./script)
or as a result of
case "$0" in script) cd /some/where;; esac
in $ENV shouldn't matter very much, should it?
| If a user's ${ENV} file follows the POSIX spec,
Local system documentation trumps the POSIX spec every time. Remember
here you're asking for NetBSD to alter its documented behaviour, not to
alter the way some undocumented event is handled, nor to actually comply
with what it documents about itself.
| I do not mean for respect to entail capitulation, only that the tone be
| professional and not condescending or dismissive.
I apologise if I ever sounded condescending, that wasn't the intent.
However, I don't think your proposal should be implemented, so dismissive,
yes - but that's just me, and I have nothing more than an opinion here,
nothing I can do can do more than perhaps influence others (I don't get to
make the change, or prevent it.)
kre
ps: If there's anyone else bothering to read all of this stuff, sometime
around now would be a good time to offer opinions.
From: Bernd Ernesti <netbsd@lists.veego.de>
To: gnats-bugs@NetBSD.org
Cc: Robert Elz <kre@munnari.OZ.AU>
Subject: Re: standards/42828: Almquist shell always evaluates the contents of ${ENV} even if non-interactive
Date: Tue, 23 Feb 2010 18:35:19 +0100
On Tue, Feb 23, 2010 at 09:30:38PM +0700, Robert Elz wrote:
[..]
> ps: If there's anyone else bothering to read all of this stuff, sometime
> around now would be a good time to offer opinions.
It would be nice to have a summary on tech-userlevel@NetBSD.org and not in
a pr.
Bernd
From: matthew green <mrg@eterna.com.au>
To: Robert Elz <kre@munnari.OZ.AU>
Cc: gnats-bugs@NetBSD.org, standards-manager@NetBSD.org,
netbsd-bugs@NetBSD.org, Richard Hansen <rhansen@bbn.com>
Subject: re: standards/42828: Almquist shell always evaluates the contents of ${ENV} even if non-interactive
Date: Wed, 24 Feb 2010 08:12:52 +1100
ps: If there's anyone else bothering to read all of this stuff, sometime
around now would be a good time to offer opinions.
i dunno about all this stuff, but i'm glad that christos has changed
our /bin/sh back to how it was, but made it handle POSIXLY_CORRECT.
.mrg.
From: Richard Hansen <rhansen@bbn.com>
To: Robert Elz <kre@munnari.OZ.AU>
Cc: gnats-bugs@NetBSD.org, standards-manager@NetBSD.org,
netbsd-bugs@NetBSD.org
Subject: Re: standards/42828: Almquist shell always evaluates the contents
of ${ENV} even if non-interactive
Date: Wed, 24 Feb 2010 00:37:55 -0500
>> The same results can be achieved by prepending a special
>> directory to PATH and populating that directory with custom versions of
>> standard commands.
>
> No, that doesn't work, built in commands are executed before commands
> found in $PATH, so that doesn't work to override stuff like test, let
> alone the built in commands that can't go in $PATH (like cd) which happens
> to be one of the commands I most often want to override. That can be
> done by defining a cd() function, and that works in ${ENV}.
Ah ha! That's what I was looking for. I can see how overriding
built-ins could be useful, and the POSIX design makes it impossible.
Now I see the benefit of evaluating a file at the start of
non-interactive shells (I just couldn't think of a practical use for the
non-standard behavior).
So, my opinion is that NetBSD's functionality should be preserved, just
moved elsewhere (such as a different environment variable) so that it
doesn't conflict with POSIX.
>> * working directory: Initially the current directory matches the
>> working directory of the invoker. If the user adds 'cd "${HOME}"' to
>> the ${ENV} file, then every shell script will assume it was invoked from
>> ${HOME}.
>
> Of course, and it would be, and if that's what the user wanted to happen
> (which we can assume they must have when they put that in ${ENV}) then
> what's the problem?
I'm coming from the perspective of users that expect POSIX compliance.
They might add 'cd "${HOME}"' not realizing that it also affects shell
scripts.
> there's also no real reason to anticipate that someone will do set -C
> in ${ENV} (outside a test for an interactive shell).
I think there is a reason: users like me that assume POSIX compliance
and fail to test for an interactive shell. They might not realize that
errors in 3rd party scripts are their fault. They might not even see
the errors if the script is executed by an X application.
> Local system documentation trumps the POSIX spec every time.
Good point. But aren't users expecting POSIX more and more? Perhaps
it's time to take another small step toward compliance.
-Richard
From: David Laight <david@l8s.co.uk>
To: Richard Hansen <rhansen@bbn.com>
Cc: Robert Elz <kre@munnari.OZ.AU>, gnats-bugs@NetBSD.org,
standards-manager@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: standards/42828: Almquist shell always evaluates the contents of ${ENV} even if non-interactive
Date: Wed, 24 Feb 2010 20:44:36 +0000
On Wed, Feb 24, 2010 at 12:37:55AM -0500, Richard Hansen wrote:
> >
> >No, that doesn't work, built in commands are executed before commands
> >found in $PATH, so that doesn't work to override stuff like test, let
> >alone the built in commands that can't go in $PATH (like cd) which happens
> >to be one of the commands I most often want to override. That can be
> >done by defining a cd() function, and that works in ${ENV}.
>
> Ah ha! That's what I was looking for. I can see how overriding
> built-ins could be useful, and the POSIX design makes it impossible.
> Now I see the benefit of evaluating a file at the start of
> non-interactive shells (I just couldn't think of a practical use for the
> non-standard behavior).
Actually, for non-special builtins, posix requires that the builtin version
only be used if the one that would be executed when $PATH is searched
would be the systems 'standard' version.
Which would mean that the builtin 'echo' could only be used if the
shells command hash table contained /usr/bin/echo.
Netbsd's /bin/sh doesn't behave that way!
(and I'm not going to change it!)
David
--
David Laight: david@l8s.co.uk
From: Richard Hansen <rhansen@bbn.com>
To: Richard Hansen <rhansen@bbn.com>, Robert Elz <kre@munnari.OZ.AU>,
gnats-bugs@NetBSD.org, standards-manager@NetBSD.org,
netbsd-bugs@NetBSD.org, david@l8s.co.uk
Cc:
Subject: Re: standards/42828: Almquist shell always evaluates the contents
of ${ENV} even if non-interactive
Date: Wed, 24 Feb 2010 17:20:56 -0500
David Laight wrote:
> Actually, for non-special builtins, posix requires that the builtin version
> only be used if the one that would be executed when $PATH is searched
> would be the systems 'standard' version.
I'm not 100% sure that's true... I stared at the "Command Search and
Execution" section [1] for a while (it's not easy to grok!) and came
away with the impression that if the system had echo as a regular
built-in, the built-in would be invoked instead of ~/bin/echo even if
~/bin was first in PATH.
Here's how I interpret the logic:
1. 'echo' doesn't contain any slashes, so rule 1. applies
2. echo is not a special built-in, so rule 1.a. doesn't apply
3. there is no function named echo, so rule 1.b. doesn't apply
4. echo is not in the exception list, so 1.c. doesn't apply
5. rule 1.d. applies and the PATH is searched for echo
6. echo is in the PATH (in ~/bin), so rule 1.d.i. applies
7. echo is implemented as a regular built-in, so rule 1.d.i.a.
applies and the built-in version is invoked
I hope I'm misunderstanding POSIX because I'd prefer the algorithm you
described.
On a related note: It's unclear what POSIX says should happen if a user
defines a function whose name matches the name of a regular built-in
(e.g., 'cd' or 'echo'). Specifically, rule 1.b. is awkward: Either the
shell must be able to distinguish implementation-provided functions from
user-defined functions (my interpretation) or it's undefined which one
wins (rule 1.d.i.a. doesn't break ties between regular built-ins and
shell functions).
[1]
<http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01>
-Richard
From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: standards/42828: Almquist shell always evaluates the contents
of ${ENV} even if non-interactive
Date: Sun, 28 Feb 2010 23:27:51 +0000
On Wed, Feb 24, 2010 at 10:25:03PM +0000, Richard Hansen wrote:
> I hope I'm misunderstanding POSIX because I'd prefer the algorithm you
> described.
There are lots of things wrong with POSIX; it would be nice in theory
(although I doubt anyone reading this has the time or energy) to
collect them up somewhere for reference.
The shell-related thing I've run across is that the behavior
specification for -e mode is so poorly written that it's nearly
impossible to tell what it's supposed to be, let alone test any given
implementation for conformance. And some of the parts that are clear
are at variance with common sense and historical practice.
NetBSD is in general more committed to maintaining historical practice
(and to correctness) than to strict compliance. And since the POSIX
folks seem to be running somewhat off the track lately (e.g. on
getline) I suspect that we will start to diverge somewhat more from
conformance too...
--
David A. Holland
dholland@netbsd.org
From: "Greg A. Woods" <woods@planix.ca>
To: Robert Elz <kre@munnari.OZ.AU>
Cc: Richard Hansen <rhansen@bbn.com>,
NetBSD GNATS <gnats-bugs@NetBSD.org>,
standards-manager@NetBSD.org
Subject: Re: standards/42828: Almquist shell always evaluates the contents of ${ENV} even if non-interactive
Date: Wed, 03 Mar 2010 14:18:10 -0500
--pgp-sign-Multipart_Wed_Mar__3_14:18:10_2010-1
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
At Tue, 23 Feb 2010 21:30:38 +0700, Robert Elz <kre@munnari.OZ.AU> wrote:
Subject: Re: standards/42828: Almquist shell always evaluates the contents =
of ${ENV} even if non-interactive
>=20
> ps: If there's anyone else bothering to read all of this stuff, sometime
> around now would be a good time to offer opinions.
I haven't caught up to the full discussion yet, but I've skimmed through
and I'd like to offer a few of my observations of related issues gleaned
over the past 20 years or so. :-)
> My take is that if ${ENV} isn't wanted for non-interactive shells, it can
> make itself a no-op. On the other hand, if it is, other than adding some
> other way to do the same thing to the shell, there is no remedy.
I think this is sort of the way it was originally intended, if I
understand correctly, and if I understand all the history and lore
correctly too.
I've had the following in my ~/.kshlogin file for some time now, which
sets ENV in such a way that it will only be the appropriate filename IFF
the shell is running in interactive mode. Some comments and
explanations are included.
Unfortunately this does not, or at least did not, work with /bin/sh (due
to lack of support for arrays), but perhaps it should (support arrays,
that is, despite them not being required by POSIX). In fact I think the
failure of POSIX to also include arrays in the shell is one of the
reasons why the definition of ENV got bunged up, since without arrays
the shell cannot support the de facto standard way of managing
interactive vs. non-interactive ENV variants.
See also my PR#33693.
Note that my ~/.profile sources an appropriately named ~/.*login file
based on what it thinks the type of shell being used is. Thus=20
if [ ${RANDOM:-0} -ne ${RANDOM:-0} -a -z "${BASH}" ] ; then
# TODO: try to remember why we don't trust this...
SHELL=3D""
[ -x $PKG/bin/ksh ] && export SHELL=3D"$PKG/bin/ksh"
[ -x $CONTRIB/bin/ksh ] && export SHELL=3D"$CONTRIB/bin/ksh"
[ -x $LOCAL/bin/ksh ] && export SHELL=3D"$LOCAL/bin/ksh"
[ -z "$SHELL" -a -x /usr/bin/ksh ] && export SHELL=3D"/usr/bin/ksh"
[ -z "$SHELL" -a -x /bin/ksh ] && export SHELL=3D"/bin/ksh"
if [ -r $HOME/.kshlogin ] ; then
. $HOME/.kshlogin
fi
elif ....
#
# .kshlogin - login shell startup for ksh
#
#ident "@(#)HOME:.kshlogin 29.1 09/04/09 20:45:32 (woods)"
set -o ignoreeof # we only do this for our login shell....
export FPATH=3D$HOME/lib/ksh
# this may be a good idea (from Frederick Bruckman <fredb@immanent.net>)
#
#for func in $(ls "$FPATH"); do
# autoload $func
#done
export FCEDIT=3D"$(whence ed)"
export ENVFILE=3D$HOME/.kshrc
if [ -n "$ENVFILE" -a -r "$ENVFILE" ] ; then
#
# This magic expression prevents a non-interactive Ksh
# instance from even trying to open and read any file.
#
# The original Korn & Bolsky book [1989] suggests....
#
#export ENV=3D'${ENVFILE[(_$-=3D0)+(_=3D1)-_${-%%*i*}]}'
#
# However the expression below is better as it handles the
# empty $- case, which the one above fails on (0+1-1 =3D 0).
#
# According to a post by Jon H. LaBadie to comp.unix.questions
# on 1989/12/06 (with the Message-ID <826@jonlab.UUCP>) the
# one below is what David Korn actually suggested to him. I'm
# reasonably certain I read a post by Korn suggesting this
# expression as well, but I may simply have read Jon's post.
#
# It does work with all the versions of Ksh I've run across,
# including ksh-85 and the one AIX-3.2, as well as pdksh.
#
export ENV=3D'${ENVFILE[(_$-=3D1)+(_=3D0)-(_$-!=3D_${-%%*i*})]}'
# The following explanation is derived from a Usenet post by
# David Young to comp.sys.apollo (of all places) 1992/10/21:
#
# The idea behind this scheme is to set up an array (ENVFILE)
# whose first element is the file we want executed at startup,
# and whose second element is empty. Note that any string
# variable can be accessed as an array with one element in the
# first (zero) position and any attempt to access other
# elements will result in an empty value.
#
# $ export ENVFILE=3D'~/.envfile'
# $ echo $ENVFILE[0]
# ~/.envfile
# $ echo $ENVFILE[1]
#=09
# $=20
#
# So, we want to expand $ENV, when it is referenced by the
# shell during startup, to ENVFILE[0] for interactive shells
# and to ENVFILE[1] for non-interactive shells. To do this we
# need an expression that will evaluate to "0" for an
# interactive shell but which will evaluate to "1" for a
# non-interactive shell.
#
# Keep in mind the two environment variables ENV and PS1 are
# special in Ksh insofar as they get evaluated when they are
# used! So if we assign the value of ENV using single quotes
# as a fixed string it will only be evaluated when it is used
# by the shell.
#
# Also remember that arithmetic expressions are evaluated
# inside array index references (i.e. inside the square
# brackets), but only after '$' variable refs are expanded.
#
# The flags variable ($-) is the key to forming our magic
# expression. If the shell is interactive, the flags will
# contain an 'i'. The expression used as the index consists
# of three parts that are combined using normal arithmetic
# operators to form the final index value:
#
# (_$-=3D1) + (_=3D0) - (_$- !=3D _${-%%*i*})
#
# (_$-=3D1) This creates a variable named "_BLAH", where
# "BLAH" is the value of the variable "$-", and
# assigns it the value "1". The expression has
# the result "1" as well.
#
# (_=3D0) This creates a variable named "_" and assigns
# it the value "0". The expression has the result
# "0" as well.
#
# (_$- !=3D _${-%%*i*}) This compares the value of _BLAH with
# the value of another variable that
# will either have the same name (and
# thus the same value), or the name "_".
#
# For an interactive shell these variables will be guaranteed
# to have different names, i.e. the first variable will have
# the name of the variable created in the first expression
# ("_BLAH") and the second will have name of the variable
# created in the second expression ("_"), respectively. Since
# the first variable has been assigned the value "1" and the
# second one has the value "0", result of this last expression
# will always be 1 for any interactive shell.
#
# For a non-interactive shell this last expression will have
# the value 0 because both parameters will have the same name
# (there was no "i" to match when forming the name of the
# second var, regardless of whether $- is empty or not).
#
# So for an interactive shell, e.g. when $- is "i", the index is:
#
# (_i=3D1) + (_=3D0) - (_i !=3D _)
# 1 + 0 - (1 !=3D 0)
# i.e. 0, thus $ENV[0], for the first and only element
#
# and for a non-interactive shell (e.g. $i is "BLAH") the index is:
#
# (_BLAH=3D1) + (_=3D0) - (_BLAH !=3D _BLAH)
# 1 + 0 - (0 !=3D 0)
# i.e. 1, thus $ENV[1], for the second, empty, element
#
# or for a non-interactive shell where $i is "", the index is:
#
# (_=3D1) + (_=3D0) - (_ !=3D _)
# 1 + 0 - (0 !=3D 0)
# i.e. 1, thus $ENV[1], for the second, empty, element
#
# Remember class, the test is tomorrow!
fi
export HISTSIZE=3D2000
# we could set HISTFILE if we wanted persistent history, but we may
# only want it for login shells, not all interactive shells....
#
# NOTE: many implementations have HISTFILE inter-locking bugs!
#
#HISTFILE=3D$HOME/.ksh_history
if [ -z "$LOGNAME" ] ; then
# some systems won't allow this...
export LOGNAME=3D${HOME##*/}
fi
# for emacs timeclock.el
#
if [ ! -f ~/.timelog ] ; then
touch ~/.timelog
fi
# note this depends on and requires the "cd" at the end of ~/.profile
#
export PWD=3D$HOME
export OLDPWD=3D$HOME
--=20
Greg A. Woods
Planix, Inc.
<woods@planix.com> +1 416 218 0099 http://www.planix.com/
--pgp-sign-Multipart_Wed_Mar__3_14:18:10_2010-1
Content-Type: application/pgp-signature
Content-Transfer-Encoding: 7bit
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (NetBSD)
iD8DBQBLjrXyZn1xt3i/9H8RArxDAKDVv7k93f+57bYehaFebDVdsE1UWwCgutOS
Rcia2aEzpON+iilR+VVKWfs=
=eOfT
-----END PGP SIGNATURE-----
--pgp-sign-Multipart_Wed_Mar__3_14:18:10_2010-1--
From: "Greg A. Woods" <woods@planix.ca>
To: gnats-bugs@NetBSD.org
Cc: standards-manager@netbsd.org,
rhansen@bbn.com
Subject: Re: standards/42828: Almquist shell always evaluates the contents of ${ENV} even if non-interactive
Date: Wed, 03 Mar 2010 14:41:25 -0500
--pgp-sign-Multipart_Wed_Mar__3_14:41:25_2010-1
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
At Wed, 3 Mar 2010 19:20:03 +0000 (UTC), "Greg A. Woods" <woods@planix.ca>=
wrote:
Subject: Re: standards/42828: Almquist shell always evaluates the contents =
of ${ENV} even if non-interactive
>=20
> Unfortunately this does not, or at least did not, work with /bin/sh (due
> to lack of support for arrays), but perhaps it should (support arrays,
> that is, despite them not being required by POSIX). In fact I think the
> failure of POSIX to also include arrays in the shell is one of the
> reasons why the definition of ENV got bunged up, since without arrays
> the shell cannot support the de facto standard way of managing
> interactive vs. non-interactive ENV variants.
Perhaps I should be more clear that the advantage to the trick using
arrays in KSH is the following:
> # This magic expression prevents a non-interactive Ksh
> # instance from even trying to open and read any file.
This same advantage is preserved with the strict POSIX requirements for
$ENV, but it is lost entirely in NetBSD (at least without the recent fix
and without POSIXLY_CORRECT being set).
Worse, with default NetBSD behaviour not only must the shell try to open
the file referenced by $ENV, it must also read possibly the entire
contents and, if I'm not mistaken, parse much of what it reads as well.
At least the POSIX requirement maintained the efficiency of not using
$ENV for non-interactive shells, though it also destroys the possibility
of using $ENV for non-interactive shells.
With arrays both possibilities are easily and compatibly (to KSH at
least, and perhaps Bash too) maintained without any necessary sacrifice
to efficiency.
--=20
Greg A. Woods
Planix, Inc.
<woods@planix.com> +1 416 218 0099 http://www.planix.com/
--pgp-sign-Multipart_Wed_Mar__3_14:41:25_2010-1
Content-Type: application/pgp-signature
Content-Transfer-Encoding: 7bit
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (NetBSD)
iD8DBQBLjrtlZn1xt3i/9H8RAlWBAKDTF24+npxK2O74K2JPfLNui8/cgACfblIb
7TikiT8LTIakGBLzW4gwYRs=
=6l6y
-----END PGP SIGNATURE-----
--pgp-sign-Multipart_Wed_Mar__3_14:41:25_2010-1--
From: Richard Hansen <rhansen@bbn.com>
To: "Greg A. Woods" <woods@planix.ca>
Cc: gnats-bugs@NetBSD.org, standards-manager@netbsd.org
Subject: Re: standards/42828: Almquist shell always evaluates the contents
of ${ENV} even if non-interactive
Date: Wed, 03 Mar 2010 15:39:54 -0500
Greg A. Woods wrote:
> With arrays both possibilities are easily and compatibly (to KSH at
> least, and perhaps Bash too) maintained without any necessary sacrifice
> to efficiency.
I'm not sure I understand... Are you suggesting that ${ENV[0]} (or the
equivalent ${ENV}) be used for interactive shells and ${ENV[1]} be used
for non-interactive shells?
If so, I like it. It's POSIX compliant and it allows us to maintain
NetBSD's functionality (but not backward compatibility) without having
to come up with a new environment variable that could collide with
someone else.
Plus it would mean arrays. :) Lack of arrays is my biggest gripe with
the POSIX shell spec ("set -e" ambiguity is a close second). However,
adding array support would be a *huge* change. It'd be a challenge to
get community buy-in.
-Richard
From: David Laight <david@l8s.co.uk>
To: gnats-bugs@NetBSD.org
Cc: standards-manager@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, rhansen@bbn.com
Subject: Re: standards/42828: Almquist shell always evaluates the contents of ${ENV} even if non-interactive
Date: Wed, 3 Mar 2010 21:21:20 +0000
On Wed, Mar 03, 2010 at 08:40:03PM +0000, Richard Hansen wrote:
>
> Plus it would mean arrays. :) Lack of arrays is my biggest gripe with
> the POSIX shell spec ("set -e" ambiguity is a close second). However,
> adding array support would be a *huge* change. It'd be a challenge to
> get community buy-in.
Especially since, to a large extent, they can be simulated using eval.
David
--
David Laight: david@l8s.co.uk
State-Changed-From-To: open->closed
State-Changed-By: kre@NetBSD.org
State-Changed-When: Wed, 05 Dec 2018 08:21:06 +0000
State-Changed-Why:
It has been over 8 years now, and nothing else has changed, nor
(as best I can tell) have there been more complaints about the
NetBSD behaviour.
Unlike when the original discussion was held, I am now in a position
where I could change things, but I am no more inclined to now than
I was (would have been, if able) then.
The current situation is that $ENV is read by all sh startups,
interactive or not, unless the posix option is set - either by
"-o posix" or by having POSIXLY_CORRECT in the environment.
I also know more about the POSIX standards process now than I did
then, and am a bit less disenchanted with that group than some
others around - however from that I have also discovered that a
system is supposed to document what is needed for it to be POSIX
conformant, and that it is not simply to be assumed that any
random system will offer a posix conformant environment by
default.
That is, if you want posix conformance, you need to read the
system doc, and do whatever it says is required to obtain that.
In this case, for at least this issue, that is to include
POSIXLY_CORRECT in your environment (which can be done in
.profile). That will not make the NetBSD shell conform
to POSIX, but it will in this particular case.
I was idly wondering whether there was anything else I could
do which would improve things, but I have concluded that there
is nothing which would not break backward compatibility that
would be any simpler than this.
There is no point keeping this PR open any longer, there does
not seem to be much chance of any more progress than that which
has already been made.
kre
ps: on the other hand, the companion PR (42829) which requests
parameter expansion on $ENV is likely to see some action soon.
I see no particularly good reason that one should not be handled.
The NetBSD shell finally has the internal mechanisms to allow
that to be done without a lot (more) messing around.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.43 2018/01/16 07:36:43 maya Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2017
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home