NetBSD Problem Report #44063

From cheusov@tut.by  Sun Nov  7 20:23:52 2010
Return-Path: <cheusov@tut.by>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by www.NetBSD.org (Postfix) with ESMTP id A5EA563BABA
	for <gnats-bugs@gnats.netbsd.org>; Sun,  7 Nov 2010 20:23:52 +0000 (UTC)
Message-Id: <87fwvcykfm.fsf@asrock.chizhovka.net>
Date: Sun, 07 Nov 2010 21:59:09 +0200
From: cheusov@tut.by
To: gnats-bugs@gnats.NetBSD.org
Subject: awk: setting NF doesn't affect $<fieldnum>
X-Send-Pr-Version: 3.95

>Number:         44063
>Category:       standards
>Synopsis:       awk: setting NF doesn't change $i
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    cheusov
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Nov 07 20:25:00 +0000 2010
>Closed-Date:    Tue Nov 22 20:59:50 +0000 2011
>Last-Modified:  Tue Nov 22 20:59:50 +0000 2011
>Originator:     Aleksey Cheusov
>Release:        NetBSD 5.1_RC4
>Organization:
>Environment:
System: NetBSD asrock.chizhovka.net 5.1_RC4 NetBSD 5.1_RC4 (GENERIC) #0: Sat Oct 30 14:10:25 EEST 2010 cheusov@asrock.chizhovka.net:/srv/obj/sys/arch/amd64/compile/GENERIC amd64
Architecture: x86_64
Machine: amd64
>Description:

http://www.opengroup.org/onlinepubs/009695399/utilities/awk.html

    ...
    References to nonexistent fields (that is, fields after $NF),
    shall evaluate to the uninitialized value.

Now awk works like the following

    $ echo ' 1  2  3 ' | awk.orig '{NF=2; print "`" $3 "`"}'
    `3`
    $ echo ' 1  2  3 ' | awk.orig '{NF=2; print "`" $0 "`"}' 
    ` 1  2  3 `
    $

As far as I can see POSIX allows variantions about changing $0,
but gawk and mawk change it. The following patch solves the problem
with $3 and makes awk compatible with gawk and mawk.

Result:

    $ echo ' 1  2  3 ' | awk '{NF=2; print "`" $3 "`"}' 
    ``
    $ echo ' 1  2  3 ' | awk '{NF=2; print "`" $0 "`"}' 
    `1 2`
    $ 

>How-To-Repeat:
See above
>Fix:

Index: lib.c
===================================================================
RCS file: /cvsroot/src/dist/nawk/lib.c,v
retrieving revision 1.14
diff -u -r1.14 lib.c
--- lib.c	19 Oct 2008 19:33:47 -0000	1.14
+++ lib.c	7 Nov 2010 19:38:29 -0000
@@ -362,6 +362,7 @@
 		}
 	}
 	setfval(nfloc, (Awkfloat) lastfld);
+	donerec = 1; /* restore */
 	if (dbg) {
 		for (j = 0; j <= lastfld; j++) {
 			p = fldtab[j];
@@ -393,6 +394,19 @@
 	setfval(nfloc, (Awkfloat) n);
 }

+void setlastfld(int n)	/* set lastfld cleaning fldtab cells if necessary */
+{
+	if (n > nfields)
+		growfldtab(n);
+
+	if (lastfld < n)
+	    cleanfld(lastfld+1, n);
+	else
+	    cleanfld(n+1, lastfld);
+
+	lastfld = n;
+}
+
 Cell *fieldadr(int n)	/* get nth field */
 {
 	if (n < 0)
Index: proto.h
===================================================================
RCS file: /cvsroot/src/dist/nawk/proto.h,v
retrieving revision 1.6.12.2
diff -u -r1.6.12.2 proto.h
--- proto.h	14 Aug 2009 20:32:21 -0000	1.6.12.2
+++ proto.h	7 Nov 2010 19:38:29 -0000
@@ -127,6 +127,7 @@
 extern	void	fldbld(void);
 extern	void	cleanfld(int, int);
 extern	void	newfld(int);
+extern	void	setlastfld(int);
 extern	int	refldbld(const char *, const char *);
 extern	void	recbld(void);
 extern	Cell	*fieldadr(int);
Index: tran.c
===================================================================
RCS file: /cvsroot/src/dist/nawk/tran.c,v
retrieving revision 1.14
diff -u -r1.14 tran.c
--- tran.c	19 Oct 2008 19:33:48 -0000	1.14
+++ tran.c	7 Nov 2010 19:38:29 -0000
@@ -298,6 +298,10 @@
 		if (fldno > *NF)
 			newfld(fldno);
 		   dprintf( ("setting field %d to %g\n", fldno, f) );
+	} else if (&vp->fval == NF) {
+		donerec = 0;	/* mark $0 invalid */
+		setlastfld(f);
+		dprintf( ("setting NF to %g\n", f) );
 	} else if (isrec(vp)) {
 		donefld = 0;	/* mark $1... invalid */
 		donerec = 1;
@@ -324,6 +328,7 @@
 {
 	char *t;
 	int fldno;
+	Awkfloat f;

 	   dprintf( ("starting setsval %p: %s = \"%s\", t=%o, r,f=%d,%d\n", 
 		vp, NN(vp->nval), s, vp->tval, donerec, donefld) );
@@ -347,6 +352,14 @@
 	vp->tval &= ~DONTFREE;
 	   dprintf( ("setsval %p: %s = \"%s (%p) \", t=%o r,f=%d,%d\n", 
 		vp, NN(vp->nval), t,t, vp->tval, donerec, donefld) );
+
+	if (&vp->fval == NF) {
+		donerec = 0;	/* mark $0 invalid */
+		f = getfval(vp);
+		setlastfld(f);
+		dprintf( ("setting NF to %g\n", f) );
+	}
+
 	return(vp->sval = t);
 }


>Release-Note:

>Audit-Trail:
From: "Christos Zoulas" <christos@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/44063 CVS commit: src/external/historical/nawk/dist
Date: Sun, 7 Nov 2010 17:55:27 -0500

 Module Name:	src
 Committed By:	christos
 Date:		Sun Nov  7 22:55:27 UTC 2010

 Modified Files:
 	src/external/historical/nawk/dist: lib.c proto.h tran.c

 Log Message:
 PR/44063: Aleksey Cheusov: awk: setting NF doesn't change $i
 http://www.opengroup.org/onlinepubs/009695399/utilities/awk.html

     ...
     References to nonexistent fields (that is, fields after $NF),
     shall evaluate to the uninitialized value.


 To generate a diff of this commit:
 cvs rdiff -u -r1.2 -r1.3 src/external/historical/nawk/dist/lib.c \
     src/external/historical/nawk/dist/proto.h \
     src/external/historical/nawk/dist/tran.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->feedback
State-Changed-By: wiz@NetBSD.org
State-Changed-When: Mon, 08 Nov 2010 00:26:07 +0000
State-Changed-Why:
christos committed a fix, ok to close?


From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: standards/44063: awk: setting NF doesn't affect $<fieldnum>
Date: Mon, 08 Nov 2010 17:06:34 +0700

     Date:        Sun,  7 Nov 2010 20:25:00 +0000 (UTC)
     From:        cheusov@tut.by
     Message-ID:  <20101107202501.0146A63BAC6@www.NetBSD.org>

   | http://www.opengroup.org/onlinepubs/009695399/utilities/awk.html
   | 
   |     ...
   |     References to nonexistent fields (that is, fields after $NF),
   |     shall evaluate to the uninitialized value.
   | 

 I think you might be reading more in to that than what it is actually
 intended to say - I'd suspect that if changing NF explicitly was meant
 to be supported at all, that doc would say something about its effects,

 It does say how NF can be set (including how it can be incremented if
 a previously non-existent field is assigned to), but it says nothing about
 the effects of increasing or decreasing the value of NF using an
 explicit assignment.

 That is, I see nothing to support the assertion that a field should become
 unassigned (nonexistant) merely because NF is altered.  I certainly don't
 think that that parenthetical remark can be used as justification for
 that interpretation - I'd interpret that as just saying that when a getline()
 happens, fields after the value NF gets set to are non-existent, and
 evaluate as uninitialised (not whatever value they may have had from
 an earlier line for example).

 kre

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: standards/44063: awk: setting NF doesn't affect $<fieldnum>
Date: Mon, 8 Nov 2010 19:06:53 +0000

 On Mon, Nov 08, 2010 at 10:10:07AM +0000, Robert Elz wrote:
  >  I think you might be reading more in to that than what it is actually
  >  intended to say - I'd suspect that if changing NF explicitly was meant
  >  to be supported at all, that doc would say something about its effects,
  >  [...]

 I don't disagree; on the other hand, awk has traditionally supported
 assigning to builtins, such as $0, and it seems pretty clear from
 analogy and the principle of least surprise what the results of
 assigning to NF should be. Having it do something else arbitrary
 doesn't seem useful, and having it blow up at runtime seems
 undesirable.

 So I think this is a good change regardless of whether the standard
 really demands it...

 I'd say there there is one open question, which is whether

    % echo 1 2 3 | awk '{ NF=2; NF=3; print; }'

 should print "1 2" or "1 2 3", IOW, whether assiging NF destroys or
 merely hides subsequent values. That one i could argue either way...

 -- 
 David A. Holland
 dholland@netbsd.org

From: Aleksey Cheusov <cheusov@tut.by>
To: gnats-bugs@NetBSD.org
Cc: standards-manager@netbsd.org, gnats-admin@netbsd.org,
	 netbsd-bugs@netbsd.org
Subject: Re: standards/44063: awk: setting NF doesn't affect $<fieldnum>
Date: Mon, 08 Nov 2010 22:37:51 +0200

 >  So I think this is a good change regardless of whether the standard
 >  really demands it...
 >  
 >  I'd say there there is one open question, which is whether
 >  
 >     % echo 1 2 3 | awk '{ NF=2; NF=3; print; }'
 >  
 >  should print "1 2" or "1 2 3", IOW, whether assiging NF destroys or
 >  merely hides subsequent values. That one i could argue either way...

 GNU awk and mawk destroy them.  This is why
 I implemented assigning NF this way.

 -- 
 Best regards, Aleksey Cheusov.

From: Alexander Nasonov <alnsn@yandex.ru>
To: gnats-bugs@NetBSD.org
Cc: standards-manager@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org
Subject: Re: standards/44063: awk: setting NF doesn't affect $<fieldnum>
Date: Mon, 8 Nov 2010 21:31:31 +0000

 cheusov@tut.by wrote:
 > http://www.opengroup.org/onlinepubs/009695399/utilities/awk.html
 > 
 >     ...
 >     References to nonexistent fields (that is, fields after $NF),
 >     shall evaluate to the uninitialized value.
 > 
 > Now awk works like the following
 > 
 >     $ echo ' 1  2  3 ' | awk.orig '{NF=2; print "`" $3 "`"}'
 >     `3`
 >     $ echo ' 1  2  3 ' | awk.orig '{NF=2; print "`" $0 "`"}' 
 >     ` 1  2  3 `
 >     $
 > 
 Hi guys,
 Can we add an atf test for this change (and the test from bin/42320
 as well)?
 Thanks,
 Alex

Responsible-Changed-From-To: standards-manager->cheusov
Responsible-Changed-By: cheusov@NetBSD.org
Responsible-Changed-When: Sun, 22 May 2011 08:28:03 +0000
Responsible-Changed-Why:
I'll add regression test for this PR.


State-Changed-From-To: feedback->open
State-Changed-By: cheusov@NetBSD.org
State-Changed-When: Sun, 22 May 2011 08:28:03 +0000
State-Changed-Why:


From: Aleksey Cheusov <cheusov@tut.by>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: standards/44063 (awk: setting NF doesn't change $i)
Date: Sat, 19 Nov 2011 22:22:51 +0300

 I'd like to commit the following regression test and close this PR.
 I think it answers all answered questions.

 Objections?

 Index: distrib/sets/lists/tests/mi
 ===================================================================
 RCS file: /cvsroot/src/distrib/sets/lists/tests/mi,v
 retrieving revision 1.424
 diff -u -r1.424 mi
 --- distrib/sets/lists/tests/mi	17 Nov 2011 01:22:30 -0000	1.424
 +++ distrib/sets/lists/tests/mi	19 Nov 2011 16:38:20 -0000
 @@ -2701,6 +2701,9 @@
  ./usr/tests/util/Atffile			tests-util-tests
  ./usr/tests/util/awk				tests-util-tests
  ./usr/tests/util/awk/Atffile			tests-util-tests
 +./usr/tests/util/awk/d_assign_NF.awk		tests-util-tests
 +./usr/tests/util/awk/d_assign_NF.in		tests-util-tests
 +./usr/tests/util/awk/d_assign_NF.out		tests-util-tests
  ./usr/tests/util/awk/d_big_regexp.awk		tests-util-tests
  ./usr/tests/util/awk/d_big_regexp.in		tests-util-tests
  ./usr/tests/util/awk/d_big_regexp.out		tests-util-tests
 Index: tests/util/awk/Makefile
 ===================================================================
 RCS file: /cvsroot/src/tests/util/awk/Makefile,v
 retrieving revision 1.2
 diff -u -r1.2 Makefile
 --- tests/util/awk/Makefile	30 Apr 2011 11:24:14 -0000	1.2
 +++ tests/util/awk/Makefile	19 Nov 2011 16:38:20 -0000
 @@ -29,5 +29,8 @@
  FILES+=		d_toupper.awk
  FILES+=		d_toupper.in
  FILES+=		d_toupper.out
 +FILES+=		d_assign_NF.awk
 +FILES+=		d_assign_NF.in
 +FILES+=		d_assign_NF.out

  .include <bsd.test.mk>
 Index: tests/util/awk/d_assign_NF.awk
 ===================================================================
 RCS file: tests/util/awk/d_assign_NF.awk
 diff -N tests/util/awk/d_assign_NF.awk
 --- /dev/null	1 Jan 1970 00:00:00 -0000
 +++ tests/util/awk/d_assign_NF.awk	19 Nov 2011 16:38:20 -0000
 @@ -0,0 +1,16 @@
 +# $NetBSD$
 +
 +{
 +	NF = 2
 +	print "$0=`" $0 "`"
 +	print "$3=`" $3 "`"
 +	print "$4=`" $4 "`"
 +	NF = 3
 +	print "$0=`" $0 "`"
 +	print "$3=`" $3 "`"
 +	print "$4=`" $4 "`"
 +	NF = 4
 +	print "$0=`" $0 "`"
 +	print "$3=`" $3 "`"
 +	print "$4=`" $4 "`"
 +}
 Index: tests/util/awk/d_assign_NF.in
 ===================================================================
 RCS file: tests/util/awk/d_assign_NF.in
 diff -N tests/util/awk/d_assign_NF.in
 --- /dev/null	1 Jan 1970 00:00:00 -0000
 +++ tests/util/awk/d_assign_NF.in	19 Nov 2011 16:38:20 -0000
 @@ -0,0 +1 @@
 + 1    2  3
 Index: tests/util/awk/d_assign_NF.out
 ===================================================================
 RCS file: tests/util/awk/d_assign_NF.out
 diff -N tests/util/awk/d_assign_NF.out
 --- /dev/null	1 Jan 1970 00:00:00 -0000
 +++ tests/util/awk/d_assign_NF.out	19 Nov 2011 16:38:20 -0000
 @@ -0,0 +1,9 @@
 +$0=`1 2`
 +$3=``
 +$4=``
 +$0=`1 2 `
 +$3=``
 +$4=``
 +$0=`1 2  `
 +$3=``
 +$4=``
 Index: tests/util/awk/t_awk.sh
 ===================================================================
 RCS file: /cvsroot/src/tests/util/awk/t_awk.sh,v
 retrieving revision 1.6
 diff -u -r1.6 t_awk.sh
 --- tests/util/awk/t_awk.sh	2 May 2011 08:30:21 -0000	1.6
 +++ tests/util/awk/t_awk.sh	19 Nov 2011 16:38:23 -0000
 @@ -98,6 +98,16 @@
  	h_check period -v x=0.5
  }

 +atf_test_case assign_NF
 +assign_NF_head()
 +{
 +	atf_set "descr" 'Checks that assign to NF changes $0 and $n (PR/44063)'
 +}
 +assign_NF_body()
 +{
 +	h_check assign_NF
 +}
 +
  atf_init_test_cases()
  {
  	atf_add_test_case big_regexp
 @@ -105,4 +115,5 @@
  	atf_add_test_case string1
  	atf_add_test_case multibyte
  	atf_add_test_case period
 +	atf_add_test_case assign_NF
  }

 -- 
 Best regards, Aleksey Cheusov.

From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, cheusov@NetBSD.org, gnats-admin@netbsd.org, 
	netbsd-bugs@netbsd.org, cheusov@tut.by
Cc: 
Subject: Re: standards/44063 (awk: setting NF doesn't change $i)
Date: Sat, 19 Nov 2011 15:37:05 -0500

 On Nov 19,  7:25pm, cheusov@tut.by (Aleksey Cheusov) wrote:
 -- Subject: Re: standards/44063 (awk: setting NF doesn't change $i)

 |  I'd like to commit the following regression test and close this PR.
 |  I think it answers all answered questions.

 Go for it.

 christos

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: standards/44063 (awk: setting NF doesn't change $i)
Date: Mon, 21 Nov 2011 12:36:14 +0000

 On Sat, Nov 19, 2011 at 07:25:02PM +0000, Aleksey Cheusov wrote:
  > The following reply was made to PR standards/44063; it has been noted by GNATS.
  > 
  > From: Aleksey Cheusov <cheusov@tut.by>
  > To: gnats-bugs@NetBSD.org
  > Cc: 
  > Subject: Re: standards/44063 (awk: setting NF doesn't change $i)
  > Date: Sat, 19 Nov 2011 22:22:51 +0300
  > 
  >  I'd like to commit the following regression test and close this PR.
  >  I think it answers all answered questions.
  >  
  >  Objections?

 Looks like a good plan to me.

 -- 
 David A. Holland
 dholland@netbsd.org

From: "Aleksey Cheusov" <cheusov@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/44063 CVS commit: src
Date: Tue, 22 Nov 2011 20:22:10 +0000

 Module Name:	src
 Committed By:	cheusov
 Date:		Tue Nov 22 20:22:10 UTC 2011

 Modified Files:
 	src/distrib/sets/lists/tests: mi
 	src/tests/util/awk: Makefile t_awk.sh
 Added Files:
 	src/tests/util/awk: d_assign_NF.awk d_assign_NF.in d_assign_NF.out

 Log Message:
 Regression tests for awk(1) (PR 44063)


 To generate a diff of this commit:
 cvs rdiff -u -r1.425 -r1.426 src/distrib/sets/lists/tests/mi
 cvs rdiff -u -r1.2 -r1.3 src/tests/util/awk/Makefile
 cvs rdiff -u -r0 -r1.1 src/tests/util/awk/d_assign_NF.awk \
     src/tests/util/awk/d_assign_NF.in src/tests/util/awk/d_assign_NF.out
 cvs rdiff -u -r1.6 -r1.7 src/tests/util/awk/t_awk.sh

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->closed
State-Changed-By: cheusov@NetBSD.org
State-Changed-When: Tue, 22 Nov 2011 20:59:50 +0000
State-Changed-Why:
Fixed


>Unformatted:

Home	PR Database Search