NetBSD Problem Report #33693

From woods@building.weird.com  Sat Jun 10 20:15:36 2006
Return-Path: <woods@building.weird.com>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by narn.NetBSD.org (Postfix) with ESMTP id 4012363BA5F
	for <gnats-bugs@gnats.netbsd.org>; Sat, 10 Jun 2006 20:15:36 +0000 (UTC)
Message-Id: <m1Fp8mO-002IhbC@building.weird.com>
Date: Sat, 10 Jun 2006 15:05:52 -0400 (EDT)
From: "Greg A. Woods" <woods@planix.com>
Sender: "Greg A. Woods" <woods@building.weird.com>
Reply-To: "Greg A. Woods" <woods@planix.com>
To: gnats-bugs@NetBSD.org
Subject: pdksh in 2.0 and newer violates de facto standard Ksh startup behaviour
X-Send-Pr-Version: 3.95

>Number:         33693
>Category:       bin
>Synopsis:       pdksh in 2.0 and newer violates de facto standard Ksh startup behaviour
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    bin-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Jun 10 20:20:00 +0000 2006
>Last-Modified:  Tue Jul 17 04:43:23 +0000 2007
>Originator:     Greg A. Woods
>Release:        NetBSD 2.0 and all subsequent releases
>Organization:
Planix, Inc.; Toronto, Ontario; Canada
>Environment:
System: NetBSD
Architecture: all
Machine: all
>Description:

	Pdksh on all recent releases of NetBSD violate de facto standard
	Ksh compatibility with respect to how $ENV works.  Every other
	ksh and ksh-compatible shell, including pdksh on 1.6 and
	earlier, does not, by default, do anything special if $ENV is
	not set.

	The 1989 KSH book, as well as Usenet folklore going back as much
	as 14 years, says that Ksh will only source ~/.profile if it's
	an interactive shell and that it'll only source the file pointed
	to by $ENV IFF that variable is set (and the file exits).

	This was the behaviour of /bin/ksh on NetBSD since /bin/ksh was
	added.  The current behaviour (since 2.0) breaks rshd, sshd, and
	perhaps all other remote execution facilities when the user's
	shell is /bin/ksh.

	Note also that the NetBSD ksh(1) manual does not say that
	/bin/ksh was built with some magic value for DEFAULT_ENV leaving
	this aberrant behaviour completely undocumented too (though it
	does hint that such a feature is possible in non-standard
	compiles).

	Note that the "configure" script used for configuring and
	building pdksh on systems other than NetBSD says:

		NOTE: This is a non-standard feature
		(ie, at&t ksh has no default environment)."

	As an example of the issue I've noticed an apparent problem with
	sftp and scp to _some_ NetBSD systems for a couple of years now
	but never was able to debug the issue since the error message
	seemed to be related only to OpenSSH on NetBSD and it seemed to
	be deep in the internal protocol.  My work-around in the past
	was to install SSH-2 on the target systems and forget about it
	(or to do the transfer in the opposite direction).  Not sure why
	that works yet, but it does (sshd2 is probably smart enough not
	to use the user's $SHELL for sftp).

	However with recent installs of 3.0_STABLE in a customer
	environment where "rsh" is still used extensively (only on a
	physically separate and private administrative network, of
	course), similar problems suddenly appeared with all "rsh" jobs
	being executed on the upgraded and new machines.

	Most importantly note that it is difficult (to the point of
	being practically impossible) with rshd (and perhaps with sshd
	as well) to pre-set $ENV to a null value in order to counter-act
	this aberrant behaviour.

	Note also that those who want $ENV to be set and used for their
	interactive sessions can trivially follow the nearly decades old
	advice (replicated below) to achieve this goal.  It might look
	like magic, but it's been used successfully by millions for many
	years now.

	Finally I'll note that I can't even find any discussion of this
	rather critical change anywhere, at least not with Google.


>How-To-Repeat:

	On a target system that's 2.0 or newer set up a normal ~/.kshrc
	using the de facto standard Ksh trick for setting the ENV
	variable only in interactive shells (put the following in
	~/.profile, and then create a ~/.kshrc file which includes
	commands to be run for _interactive_ sessions, such as showing
	$HOME with "cd"):

	export ENVFILE=$HOME/.kshrc

	if [ -n "$ENVFILE" -a -r "$ENVFILE" ] ; then
		#
		# This magic expression prevents a non-interactive Ksh
		# instance from even trying to open and read any file.
		#
		# The original Korn & Bolsky KSH book [1989] suggests....
		#
		#export ENV='${ENVFILE[(_$-=0)+(_=1)-_${-%%*i*}]}'
		#
		# However the expression below is better as it handles the
		# empty $- case, which the one above fails on (0+1-1 = 0).
		#
		# According to a post by Jon H. LaBadie to comp.unix.questions
		# on 1989/12/06 (with the Message-ID <826@jonlab.UUCP>) the
		# one below is what David Korn actually suggested to him.  I'm
		# reasonably certain I read a post by Korn suggesting this
		# expression as well, but I may simply have read Jon's post.
		#
		# It does work with all the versions of Ksh I've run across,
		# including ksh-85 and the one AIX-3.2, as well as pdksh.
		#
		export ENV='${ENVFILE[(_$-=1)+(_=0)-(_$-!=_${-%%*i*})]}'

		# The following explanation is derived from a Usenet post by
		# David Young to comp.sys.apollo (of all places) 1992/10/21:
		#
		# The idea behind this scheme is to set up an array (ENVFILE)
		# whose first element is the file we want executed at startup,
		# and whose second element is empty.  Note that any string
		# variable can be accessed as an array with one element in the
		# first (zero) position and any attempt to access other
		# elements will result in an empty value.
		#
		#	$ export ENVFILE='~/.envfile'
		#	$ echo $ENVFILE[0]
		#	~/.envfile
		#	$ echo $ENVFILE[1]
		#	
		#	$ 
		#
		# So, we want to expand $ENV, when it is referenced by the
		# shell during startup, to ENVFILE[0] for interactive shells
		# and to ENVFILE[1] for non-interactive shells.  To do this we
		# need an expression that will evaluate to "0" for an
		# interactive shell but which will evaluate to "1" for a
		# non-interactive shell.
		#
		# Keep in mind the two environment variables ENV and PS1 are
		# special in Ksh insofar as they get evaluated when they are
		# used!  So if we assign the value of ENV using single quotes
		# as a fixed string it will only be evaluated when it is used
		# by the shell.
		#
		# Also remember that arithmetic expressions are evaluated
		# inside array index references (i.e. inside the square
		# brackets), but only after '$' variable refs are expanded.
		#
		# The flags variable ($-) is the key to forming our magic
		# expression.  If the shell is interactive, the flags will
		# contain an 'i'.  The expression used as the index consists
		# of three parts that are combined using normal arithmetic
		# operators to form the final index value:
		#
		#	(_$-=1) + (_=0) - (_$- != _${-%%*i*})
		#
		# (_$-=1)	This creates a variable named "_BLAH", where
		#		"BLAH" is the value of the variable "$-", and
		#		assigns it the value "1".  The expression has
		#		the result "1" as well.
		#
		# (_=0)		This creates a variable named "_" and assigns
		#		it the value "0".  The expression has the result
		#		"0" as well.
		#
		# (_$- != _${-%%*i*})	This compares the value of _BLAH with
		#			the value of another variable that
		#			will either have the same name (and
		#			thus the same value), or the name "_".
		#
		# For an interactive shell these variables will be guaranteed
		# to have different names, i.e. the first variable will have
		# the name of the variable created in the first expression
		# ("_BLAH") and the second will have name of the variable
		# created in the second expression ("_"), respectively.  Since
		# the first variable has been assigned the value "1" and the
		# second one has the value "0", result of this last expression
		# will always be 1 for any interactive shell.
		#
		# For a non-interactive shell this last expression will have
		# the value 0 because both parameters will have the same name
		# (there was no "i" to match when forming the name of the
		# second var, regardless of whether $- is empty or not).
		#
		# So for an interactive shell, e.g. when $- is "i", the index is:
		#
		#	(_i=1) + (_=0) - (_i != _)
		#	 1     +  0    - (1 != 0)
		#	i.e. 0, thus $ENV[0], for the first and only element
		#
		# and for a non-interactive shell (e.g. $i is "BLAH") the index is:
		#
		#	(_BLAH=1) + (_=0) - (_BLAH != _BLAH)
		#	 1        +  0    - (0 != 0)
		#	i.e. 1, thus $ENV[1], for the second, empty, element
		#
		# or for a non-interactive shell where $i is "", the index is:
		#
		#	(_=1) + (_=0) - (_ != _)
		#	 1     +  0   - (0 != 0)
		#	i.e. 1, thus $ENV[1], for the second, empty, element
		#
		# Remember class, the test is tomorrow!
	fi

	Then try using rsh or similar to a 2.0 or newer system and see
	that it spits out strange error messages:

	$ rsh 192.168.0.220 uname -srm
	/home/woods
	ksh: No controlling tty (open /dev/tty: Device not configured)
	ksh: Can't find tty file descriptor
	NetBSD 3.0_STABLE i386
	ksh: warning: won't have full job control

	See that on on 1.6 and prior target systems it still works
	correctly (i.e. only the output from the command is shown):

	$ rsh 192.168.0.251 uname -srm
	NetBSD 1.6.2_STABLE alpha

	See that similarly sftp is also now _completely_ broken and
	unusable:

	$ sftp whome.planix.com
	Keyboard-interactive:
	woods@PLANIX.COM's Password:
	Keyboard-interactive:
	Warning: ssh_packet_wrapper_input: invalid packet received: len 796210483 closing the offending input channel.

	Thanks to Andreas Wrede who thought to explore other
	representations of that weird value we see that it is 2F75352F
	in hex and thus it corresponds to the ASCII string "/u5/", which
	just so happens to be the first few characters in my $HOME on
	that machine, a string which will be printed initially by my
	_interactive_ ~/.kshrc.

>Fix:

	revert the following change.  ksh should _not_ ever have a
	default ENV in a base system install, _especially_ when it is
	installed as /bin/ksh!

----------------------------
revision 1.6
date: 2002/11/11 19:03:57;  author: jdolecek;  state: Exp;  lines: +2 -2
default to "$HOME/.kshrc" if ENV is not set
----------------------------

>Release-Note:

>Audit-Trail:

>Unformatted:

Home	PR Database Search