NetBSD Problem Report #51726

From www@NetBSD.org  Sat Dec 17 17:19:09 2016
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 18A4F7A2FD
	for <gnats-bugs@gnats.NetBSD.org>; Sat, 17 Dec 2016 17:19:09 +0000 (UTC)
Message-Id: <20161217171907.ECC477A302@mollari.NetBSD.org>
Date: Sat, 17 Dec 2016 17:19:07 +0000 (UTC)
From: abs@absd.org
Reply-To: abs@absd.org
To: gnats-bugs@NetBSD.org
Subject: sort -n ignored if given after -k
X-Send-Pr-Version: www-1.0

>Number:         51726
>Category:       bin
>Synopsis:       sort -n ignored if given after -k
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    abhinav
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Dec 17 17:20:00 +0000 2016
>Last-Modified:  Tue Dec 27 08:05:01 +0000 2016
>Originator:     David Brownlee
>Release:        7.0_STABLE
>Organization:
>Environment:
NetBSD iris.absd.org 7.0_STABLE NetBSD 7.0_STABLE (GENERIC.201610051330Z) amd64

>Description:
If -n is given after -k on the command line it appears to be ignored.

I found a script which was running "sort -k 3 -n -r"

On NetBSD-7 it looks like the -n does not take effect as it happens after the -k 3, while "sort -n -k 3 -r" works as expected
>How-To-Repeat:
Create a file which sorts differently for alpha vs numeric sort, eg, 
  9
  10

Then

sort -k 1 -n ${file}

vs

sort -n -k 1 ${file}

>Fix:

>Release-Note:

>Audit-Trail:

Responsible-Changed-From-To: bin-bug-people->abhinav
Responsible-Changed-By: abhinav@NetBSD.org
Responsible-Changed-When: Sat, 17 Dec 2016 18:05:35 +0000
Responsible-Changed-Why:
Looking into it


From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: bin/51726: sort -n ignored if given after -k
Date: Sun, 18 Dec 2016 02:02:29 +0000

 On Sat, Dec 17, 2016 at 05:20:00PM +0000, abs@absd.org wrote:
  > If -n is given after -k on the command line it appears to be ignored.
  > 
  > I found a script which was running "sort -k 3 -n -r"

 If you're using -k you're supposed to put the key flags in the -k
 argument, that is, sort -k 3nr.

 Technically I think if you write -k 3 -n -r and it doesn't honor the
 -n it's doing what you asked. And I think if you write -k 3 -n -r and
 it *does* sort in reverse order, then *that*'s a bug. sigh.

 sort's argument handling is a trainwreck. And thus the code in sort
 that deals with it is horrifying. I know why POSIX wanted to get rid
 of "sort +3", but it *worked*...

 -- 
 David A. Holland
 dholland@netbsd.org

From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: bin/51726: sort -n ignored if given after -k
Date: Sun, 18 Dec 2016 15:23:38 +0700

     Date:        Sun, 18 Dec 2016 02:05:01 +0000 (UTC)
     From:        David Holland <dholland-bugs@netbsd.org>
     Message-ID:  <20161218020501.468347A302@mollari.NetBSD.org>

   |  If you're using -k you're supposed to put the key flags in the -k
   |  argument, that is, sort -k 3nr.

 Not quite, using global options is still fine, they're just not
 supposed to apply to a key specified if that key has any sort selections
 of its own.

 Our sort doesn't do that, it merges global options and key specific
 options, in a kind of weird way --- but the behaviour described (while
 perhaps not strictly correct) is I think what is intended.

 That is, when sorting a key field, you get whatever sort options are
 specified for that key, merged with whatever global options had already
 been given - later global options are supposed to affect the next key
 (or the backup default sort if the keys are equal).   That is, that's how
 the NetBSD sort is written.

   |  Technically I think if you write -k 3 -n -r and it doesn't honor the
   |  -n it's doing what you asked.

 According to our sort's design, yes, though I don't think that is posix.

   | And I think if you write -k 3 -n -r and
   | it *does* sort in reverse order, then *that*'s a bug. sigh.

 Yes, probably.

   |  sort's argument handling is a trainwreck.

 That mild?

   |  And thus the code in sort that deals with it is horrifying.

 Yes.   We could probably simplify it a lot if we made it posix
 conformat (where any key specific ordering options disable all
 the globals for that key) but it has been as it is for a long time
 now (mayve even, modulo the k stuff) has been like it since 6th or
 7th edition unix.   So changing it might break a lot - who knows?

   | I know why POSIX wanted to get rid of "sort +3", but it *worked*...

 That change is actually a separate isue, and other than that people tend
 not to really understand what the key selection stuff does, that works
 as well.   And the +3 variant is still there as well...

 kre


From: Abhinav Upadhyay <er.abhinav.upadhyay@gmail.com>
To: NetBSD GNATS <gnats-bugs@netbsd.org>
Cc: abhinav@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
	abs@absd.org
Subject: Re: bin/51726: sort -n ignored if given after -k
Date: Sun, 18 Dec 2016 21:54:42 +0530

 On Sun, Dec 18, 2016 at 1:55 PM, Robert Elz <kre@munnari.oz.au> wrote:
 > The following reply was made to PR bin/51726; it has been noted by GNATS.
 >
 > From: Robert Elz <kre@munnari.OZ.AU>
 > To: gnats-bugs@NetBSD.org
 > Cc:
 > Subject: Re: bin/51726: sort -n ignored if given after -k
 > Date: Sun, 18 Dec 2016 15:23:38 +0700
 >
 >      Date:        Sun, 18 Dec 2016 02:05:01 +0000 (UTC)
 >      From:        David Holland <dholland-bugs@netbsd.org>
 >      Message-ID:  <20161218020501.468347A302@mollari.NetBSD.org>
 >
 >    |  If you're using -k you're supposed to put the key flags in the -k
 >    |  argument, that is, sort -k 3nr.
 >
 >  Not quite, using global options is still fine, they're just not
 >  supposed to apply to a key specified if that key has any sort selections
 >  of its own.
 >
 >  Our sort doesn't do that, it merges global options and key specific
 >  options, in a kind of weird way --- but the behaviour described (while
 >  perhaps not strictly correct) is I think what is intended.
 >
 >  That is, when sorting a key field, you get whatever sort options are
 >  specified for that key, merged with whatever global options had already
 >  been given - later global options are supposed to affect the next key
 >  (or the backup default sort if the keys are equal).   That is, that's how
 >  the NetBSD sort is written.
 >
 >    |  Technically I think if you write -k 3 -n -r and it doesn't honor the
 >    |  -n it's doing what you asked.
 >
 >  According to our sort's design, yes, though I don't think that is posix.
 >
 >    | And I think if you write -k 3 -n -r and
 >    | it *does* sort in reverse order, then *that*'s a bug. sigh.
 >
 >  Yes, probably.
 >
 >    |  sort's argument handling is a trainwreck.
 >
 >  That mild?
 >
 >    |  And thus the code in sort that deals with it is horrifying.
 >
 >  Yes.   We could probably simplify it a lot if we made it posix
 >  conformat (where any key specific ordering options disable all
 >  the globals for that key) but it has been as it is for a long time
 >  now (mayve even, modulo the k stuff) has been like it since 6th or
 >  7th edition unix.   So changing it might break a lot - who knows?

 I have been going over the sort(1) man page from posix, and at one
 place (in the APPLICATION USAGE section) it says the following about
 the use of ordering options after -k:

 "The wording in the OPTIONS section clarifies that the -b, -d, -f, -i,
 -n, and -r options have to come before the first sort key specified if
 they are intended to apply to all specified keys. The way it is
 described in this volume of POSIX.1-2008 matches historical practice,
 not historical documentation. The results are unspecified if these
 options are specified after a -k option."

 So, I guess using `sort -k3 -n -r` is undefined behavior?. That said,
 the example mentioned in the PR (sort -k 2 -n -r) does work with GNU
 sort(1). I had a patch ready yesterday, so just posting it here :)

 This patch lets sort(1) associate -b, -d, -f, -i, -n, -r options to
 specific fields if specified after -k.  All ATF tests for sort(1) are
 passing except for a couple of cases in kflag_alpha, but they look
 ambiguous to me and one of the similar tests is commented out as
 broken. If we want to accept this behavior in our sort(1) and the
 patch looks in the right direction, I will try to dig in and get those
 test cases passing  :)

 http://www.netbsd.org/~abhinav/sort.c.diff

 -
 Abhinav

From: Abhinav Upadhyay <er.abhinav.upadhyay@gmail.com>
To: NetBSD GNATS <gnats-bugs@netbsd.org>
Cc: abhinav@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
	David Brownlee <abs@absd.org>
Subject: Re: bin/51726: sort -n ignored if given after -k
Date: Tue, 20 Dec 2016 21:30:14 +0530

 On Sun, Dec 18, 2016 at 9:54 PM, Abhinav Upadhyay
 <er.abhinav.upadhyay@gmail.com> wrote:
 > On Sun, Dec 18, 2016 at 1:55 PM, Robert Elz <kre@munnari.oz.au> wrote:
 >> The following reply was made to PR bin/51726; it has been noted by GNATS.
 >>
 >> From: Robert Elz <kre@munnari.OZ.AU>
 >> To: gnats-bugs@NetBSD.org
 >> Cc:
 >> Subject: Re: bin/51726: sort -n ignored if given after -k
 >> Date: Sun, 18 Dec 2016 15:23:38 +0700
 >>
 >>      Date:        Sun, 18 Dec 2016 02:05:01 +0000 (UTC)
 >>      From:        David Holland <dholland-bugs@netbsd.org>
 >>      Message-ID:  <20161218020501.468347A302@mollari.NetBSD.org>
 >>
 >>    |  If you're using -k you're supposed to put the key flags in the -k
 >>    |  argument, that is, sort -k 3nr.
 >>
 >>  Not quite, using global options is still fine, they're just not
 >>  supposed to apply to a key specified if that key has any sort selections
 >>  of its own.
 >>
 >>  Our sort doesn't do that, it merges global options and key specific
 >>  options, in a kind of weird way --- but the behaviour described (while
 >>  perhaps not strictly correct) is I think what is intended.
 >>
 >>  That is, when sorting a key field, you get whatever sort options are
 >>  specified for that key, merged with whatever global options had already
 >>  been given - later global options are supposed to affect the next key
 >>  (or the backup default sort if the keys are equal).   That is, that's how
 >>  the NetBSD sort is written.
 >>
 >>    |  Technically I think if you write -k 3 -n -r and it doesn't honor the
 >>    |  -n it's doing what you asked.
 >>
 >>  According to our sort's design, yes, though I don't think that is posix.
 >>
 >>    | And I think if you write -k 3 -n -r and
 >>    | it *does* sort in reverse order, then *that*'s a bug. sigh.
 >>
 >>  Yes, probably.
 >>
 >>    |  sort's argument handling is a trainwreck.
 >>
 >>  That mild?
 >>
 >>    |  And thus the code in sort that deals with it is horrifying.
 >>
 >>  Yes.   We could probably simplify it a lot if we made it posix
 >>  conformat (where any key specific ordering options disable all
 >>  the globals for that key) but it has been as it is for a long time
 >>  now (mayve even, modulo the k stuff) has been like it since 6th or
 >>  7th edition unix.   So changing it might break a lot - who knows?
 >
 > I have been going over the sort(1) man page from posix, and at one
 > place (in the APPLICATION USAGE section) it says the following about
 > the use of ordering options after -k:
 >
 > "The wording in the OPTIONS section clarifies that the -b, -d, -f, -i,
 > -n, and -r options have to come before the first sort key specified if
 > they are intended to apply to all specified keys. The way it is
 > described in this volume of POSIX.1-2008 matches historical practice,
 > not historical documentation. The results are unspecified if these
 > options are specified after a -k option."
 >
 > So, I guess using `sort -k3 -n -r` is undefined behavior?. That said,
 > the example mentioned in the PR (sort -k 2 -n -r) does work with GNU
 > sort(1). I had a patch ready yesterday, so just posting it here :)
 >
 > This patch lets sort(1) associate -b, -d, -f, -i, -n, -r options to
 > specific fields if specified after -k.  All ATF tests for sort(1) are
 > passing except for a couple of cases in kflag_alpha, but they look
 > ambiguous to me and one of the similar tests is commented out as
 > broken. If we want to accept this behavior in our sort(1) and the
 > patch looks in the right direction, I will try to dig in and get those
 > test cases passing  :)
 >
 > http://www.netbsd.org/~abhinav/sort.c.diff

 Just tested on OpenBSD and FreeBSD, their sort(1) also supports -n after -k.

 -
 Abhinav

From: Abhinav Upadhyay <er.abhinav.upadhyay@gmail.com>
To: NetBSD GNATS <gnats-bugs@netbsd.org>
Cc: abhinav@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
	David Brownlee <abs@absd.org>
Subject: Re: bin/51726: sort -n ignored if given after -k
Date: Tue, 27 Dec 2016 10:30:49 +0530

 On Tue, Dec 20, 2016 at 9:30 PM, Abhinav Upadhyay
 <er.abhinav.upadhyay@gmail.com> wrote:
 > On Sun, Dec 18, 2016 at 9:54 PM, Abhinav Upadhyay
 > <er.abhinav.upadhyay@gmail.com> wrote:
 >> On Sun, Dec 18, 2016 at 1:55 PM, Robert Elz <kre@munnari.oz.au> wrote:
 >>> The following reply was made to PR bin/51726; it has been noted by GNATS.
 >>>
 >>> From: Robert Elz <kre@munnari.OZ.AU>
 >>> To: gnats-bugs@NetBSD.org
 >>> Cc:
 >>> Subject: Re: bin/51726: sort -n ignored if given after -k
 >>> Date: Sun, 18 Dec 2016 15:23:38 +0700
 >>>
 >>>      Date:        Sun, 18 Dec 2016 02:05:01 +0000 (UTC)
 >>>      From:        David Holland <dholland-bugs@netbsd.org>
 >>>      Message-ID:  <20161218020501.468347A302@mollari.NetBSD.org>
 >>>
 >>>    |  If you're using -k you're supposed to put the key flags in the -k
 >>>    |  argument, that is, sort -k 3nr.
 >>>
 >>>  Not quite, using global options is still fine, they're just not
 >>>  supposed to apply to a key specified if that key has any sort selections
 >>>  of its own.
 >>>
 >>>  Our sort doesn't do that, it merges global options and key specific
 >>>  options, in a kind of weird way --- but the behaviour described (while
 >>>  perhaps not strictly correct) is I think what is intended.
 >>>
 >>>  That is, when sorting a key field, you get whatever sort options are
 >>>  specified for that key, merged with whatever global options had already
 >>>  been given - later global options are supposed to affect the next key
 >>>  (or the backup default sort if the keys are equal).   That is, that's how
 >>>  the NetBSD sort is written.
 >>>
 >>>    |  Technically I think if you write -k 3 -n -r and it doesn't honor the
 >>>    |  -n it's doing what you asked.
 >>>
 >>>  According to our sort's design, yes, though I don't think that is posix.
 >>>
 >>>    | And I think if you write -k 3 -n -r and
 >>>    | it *does* sort in reverse order, then *that*'s a bug. sigh.
 >>>
 >>>  Yes, probably.
 >>>
 >>>    |  sort's argument handling is a trainwreck.
 >>>
 >>>  That mild?
 >>>
 >>>    |  And thus the code in sort that deals with it is horrifying.
 >>>
 >>>  Yes.   We could probably simplify it a lot if we made it posix
 >>>  conformat (where any key specific ordering options disable all
 >>>  the globals for that key) but it has been as it is for a long time
 >>>  now (mayve even, modulo the k stuff) has been like it since 6th or
 >>>  7th edition unix.   So changing it might break a lot - who knows?
 >>
 >> I have been going over the sort(1) man page from posix, and at one
 >> place (in the APPLICATION USAGE section) it says the following about
 >> the use of ordering options after -k:
 >>
 >> "The wording in the OPTIONS section clarifies that the -b, -d, -f, -i,
 >> -n, and -r options have to come before the first sort key specified if
 >> they are intended to apply to all specified keys. The way it is
 >> described in this volume of POSIX.1-2008 matches historical practice,
 >> not historical documentation. The results are unspecified if these
 >> options are specified after a -k option."
 >>
 >> So, I guess using `sort -k3 -n -r` is undefined behavior?. That said,
 >> the example mentioned in the PR (sort -k 2 -n -r) does work with GNU
 >> sort(1). I had a patch ready yesterday, so just posting it here :)
 >>
 >> This patch lets sort(1) associate -b, -d, -f, -i, -n, -r options to
 >> specific fields if specified after -k.  All ATF tests for sort(1) are
 >> passing except for a couple of cases in kflag_alpha, but they look
 >> ambiguous to me and one of the similar tests is commented out as
 >> broken. If we want to accept this behavior in our sort(1) and the
 >> patch looks in the right direction, I will try to dig in and get those
 >> test cases passing  :)
 >>
 >> http://www.netbsd.org/~abhinav/sort.c.diff
 >

 I have a new patch which fixes this issue without breaking any tests,
 please have a look:

 http://netbsd.org/~abhinav/sort.c.diff_v2

 -
 Abhinav

From: Robert Elz <kre@munnari.OZ.AU>
To: Abhinav Upadhyay <er.abhinav.upadhyay@gmail.com>
Cc: NetBSD GNATS <gnats-bugs@netbsd.org>, abhinav@netbsd.org,
        gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
        David Brownlee <abs@absd.org>
Subject: Re: bin/51726: sort -n ignored if given after -k
Date: Tue, 27 Dec 2016 14:59:37 +0700

     Date:        Tue, 27 Dec 2016 10:30:49 +0530
     From:        Abhinav Upadhyay <er.abhinav.upadhyay@gmail.com>
     Message-ID:  <CAHwRYJnPTdDbQi1FOROG_MbLDrUHGpD7BOfPRuLnhydUAGOqFw@mail.gmail.com>

   | I have a new patch which fixes this issue without breaking any tests,
   | please have a look:
   | 
   | http://netbsd.org/~abhinav/sort.c.diff_v2

 The first 2 chunks of that appear to change the way that global opts
 after a -k are linked in ... before the change

 	sort -d -k1,2f -i k2,3n

 would (incorrectly according to posix) sort on the first key with the d & f
 flags (posix would expect only f) and then the 2nd key with d i & n flags
 (and then if needed, the whole record with d and i flags).

 After your proposed change, as I read it, the effect would be to sort
 on the first key with d f and i flags, and then the second key with d and n
 flags (and then the whole record with just d).

 Since the meaning of global opts after a -k is explicitly undefined, making
 that change is something that could be done, but I am not sure that there
 is really any overall merit in it.   What we have now isn'tgreat, but at
 least can be explained, global options apply to all following -k's, and
 so has some potential use.

 After the change, glbal options after a -k are essentially just attached
 to the preceding -k, right, so

 	sork -k1,2 -n

 is exactly the same as

 	sort -1,2n

 I am not sure there is any real point to that.

 The third change in the diff looks to be just churn, whcih changes nothing,
 and shouldn't really be there at all.

 kre

 ps: neither you, nor I above, made any reference to the r flag, which is
 currently just different, and really should not be (but its current impl
 is scattered all over.)

>Unformatted:

Home	PR Database Search