NetBSD Problem Report #40977

From riastradh@joule.t.continuation.org  Sat Mar  7 04:24:00 2009
Return-Path: <riastradh@joule.t.continuation.org>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by www.NetBSD.org (Postfix) with ESMTP id A0DCC63B8EC
	for <gnats-bugs@gnats.NetBSD.org>; Sat,  7 Mar 2009 04:24:00 +0000 (UTC)
Message-Id: <20090307042358.2C2C61236@joule.t.continuation.org>
Date: Sat,  7 Mar 2009 04:23:58 +0000 (UTC)
From: Taylor R Campbell <campbell@mumble.net>
Reply-To: Taylor R Campbell <campbell@mumble.net>
To: gnats-bugs@gnats.NetBSD.org
Subject: /var/backups/work/device.current is overly sensitive to ls(1) columnar alignment
X-Send-Pr-Version: 3.95

>Number:         40977
>Category:       misc
>Synopsis:       /var/backups/work/device.current is overly sensitive to ls(1) columnar alignment
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    misc-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Mar 07 04:25:00 +0000 2009
>Last-Modified:  Tue Sep 08 09:15:02 +0000 2009
>Originator:     Taylor R Campbell <campbell@mumble.net>
>Release:        NetBSD 4.0.1
>Organization:
>Environment:
System: NetBSD joule.t.continuation.org 4.0.1 NetBSD 4.0.1 (GENERIC) #0: Tue Oct 7 22:58:48 PDT 2008 builds@wb30:/home/builds/ab/netbsd-4-0-1-RELEASE/i386/200810080053Z-obj/home/builds/ab/netbsd-4-0-1-RELEASE/src/sys/arch/i386/compile/GENERIC i386
Architecture: i386
Machine: i386
>Description:

	Any change in the columnar alignment in the listing of a
	single device in /var/backups/work/device.current as generated
	by /etc/security causes every other line to change.  This
	makes the RCS history accumulate excessively, about a thousand
	times faster than the useful information is accumulating, and
	renders the diffs useless for the naked eye.

>How-To-Repeat:

	Create a user whose name is sufficiently long (say nine
	characters).  On some days, own a tty when /etc/security
	runs.  On other days, let root own every device listed.
	Observe the entire file /var/backups/work/device.current
	change.

>Fix:

	Yes, please!  Can ls(1) be persuaded to adjust columns on a
	line-by-line basis, rather than having the adjustment of a
	single line affect the entire output?  Nothing leaps out of
	the man page about this.

>Audit-Trail:
From: "David H. Gutteridge" <dhgutteridge@sympatico.ca>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: misc/40977: /var/backups/work/device.current is overly sensitive to ls(1) co
Date: Sat, 07 Mar 2009 02:32:23 -0500

 I could be mistaken, but I believe you can use the variable
 $check_devices_ignore_paths that's defined in
 /etc/defaults/security.conf (and can be overwritten in
 /etc/security.conf) to specify you want to ignore tty devices.  That
 string gets turned into find(1) "-path" arguments that are then pruned.
 That could be a workaround for you.

 Regards,

 Dave


From: Taylor R Campbell <campbell@mumble.net>
To: gnats-bugs@NetBSD.org
Cc: misc-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: misc/40977: /var/backups/work/device.current is overly sensitive to ls(1) co
Date: Sun, 8 Mar 2009 11:40:26 -0400

    Date: Sat,  7 Mar 2009 02:32:23 -0500
    From: "David H. Gutteridge" <dhgutteridge@sympatico.ca>

    I could be mistaken, but I believe you can use the variable
    $check_devices_ignore_paths that's defined in
    /etc/defaults/security.conf (and can be overwritten in
    /etc/security.conf) to specify you want to ignore tty devices.  That
    string gets turned into find(1) "-path" arguments that are then pruned.
    That could be a workaround for you.

 Yes, I think that will approximately work, but as you say, it's only a
 workaround, not a solution, and it works around only one of the
 problems, namely that of the log file's thousandfold increase in
 expansion rate.  It won't help me to see important changes in the
 diffs.  For instance, if a user named `meddling-interloper' takes some
 tty devices, I'd like to see that.

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/40977: /var/backups/work/device.current is overly
	sensitive to ls(1) columnar alignment
Date: Sun, 29 Mar 2009 18:38:26 +0000

 On Sat, Mar 07, 2009 at 04:25:00AM +0000, Taylor R Campbell wrote:
  > 	Any change in the columnar alignment in the listing of a
  > 	single device in /var/backups/work/device.current as generated
  > 	by /etc/security causes every other line to change.  This
  > 	makes the RCS history accumulate excessively, about a thousand
  > 	times faster than the useful information is accumulating, and
  > 	renders the diffs useless for the naked eye.

 Does setting "diff_options=-uw" in security.conf produce suitable
 diffs?

 (And, does it mess up any other diffs? Does that mechanism need to be
 extended to allow per-object diff flags?)

 -- 
 David A. Holland
 dholland@netbsd.org

From: Alan Barrett <apb@cequrux.com>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/40977: /var/backups/work/device.current is overly
	sensitive to ls(1) columnar alignment
Date: Mon, 30 Mar 2009 10:31:19 +0200

 On Sun, 29 Mar 2009, David Holland wrote:
 >  On Sat, Mar 07, 2009 at 04:25:00AM +0000, Taylor R Campbell wrote:
 >   > 	Any change in the columnar alignment in the listing of a
 >   > 	single device in /var/backups/work/device.current as generated
 >   > 	by /etc/security causes every other line to change.

 I was thinking of using "find ... -ls" instead of "find ... -print0
 | xargs -0 ls ...", because the output from "find -ls" always has
 consistent horizontal space.  However, the the output from "find -ls"
 has some extra columns (easy to remove via sed), and a different date
 format (similar to "ls -l", different from "ls -lT").

 --apb (Alan Barrett)

From: Taylor R Campbell <campbell@mumble.net>
To: gnats-bugs@NetBSD.org
Cc: misc-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, David Holland <dholland-bugs@netbsd.org>
Subject: Re: misc/40977: /var/backups/work/device.current is overly
	sensitive to ls(1) columnar alignment
Date: Mon, 3 Aug 2009 11:02:49 -0400

 This is a multi-part message in MIME format.
 --=_1/aDh3DSoId+Vui0mPhmC5hHEL47dOLj

    Date: Sun, 29 Mar 2009 18:40:04 +0000 (UTC)
    From: David Holland <dholland-bugs@netbsd.org>

    On Sat, Mar 07, 2009 at 04:25:00AM +0000, Taylor R Campbell wrote:
    > 	Any change in the columnar alignment in the listing of a
    > 	single device in /var/backups/work/device.current as generated
    > 	by /etc/security causes every other line to change.  This
    > 	makes the RCS history accumulate excessively, about a thousand
    > 	times faster than the useful information is accumulating, and
    > 	renders the diffs useless for the naked eye.

    Does setting "diff_options=-uw" in security.conf produce suitable
    diffs?

 Sorry for the late reply.  Setting diff_options doesn't change what
 gets recorded in the RCS file; it affects only the messages that the
 security script sends by mail.  Here's a patch to the security script
 to make it do what I want, by using the -exec option to find(1) rather
 than xargs(1).  Unfortunately, this slows the security script down a
 great deal by causing it to fork many more times, but it does reduce
 informationless clutter, which causes backups to use much less disk
 space and which makes them more useful to humans.

 Perhaps find(1) should be extended with an option to set the `ls(1)
 flags' that it uses (emulates) for `find -ls', which is faster than
 either `find -exec ls' or xargs(1).

 --=_1/aDh3DSoId+Vui0mPhmC5hHEL47dOLj
 Content-Type: text/plain; charset="ISO-8859-1"; name="security"
 Content-Transfer-Encoding: quoted-printable
 Content-Disposition: attachment; filename="security.patch"

 --- security.~1~	2009-08-03 15:00:12.000000000 +0000
 +++ security	2009-08-03 15:01:09.000000000 +0000
 @@ -673,8 +673,8 @@
  	find / $ignexpr \
  	    \( \( -perm -u+s -a ! -type d \) -o \
  	       \( -perm -g+s -a ! -type d \) -o \
 -	       -type b -o -type c \) -print0 | \
 -	xargs -0 ls -ldgTq | sort +9 > $LIST
 +	       -type b -o -type c \) \
 +	    -exec ls -ldgTq \{\} \; | sort +9 > $LIST
 =20
  	) 2> $OUTPUT
 =20

 --=_1/aDh3DSoId+Vui0mPhmC5hHEL47dOLj--

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/40977: /var/backups/work/device.current is overly
	sensitive to ls(1) columnar alignment
Date: Mon, 7 Sep 2009 19:38:14 +0000

 On Mon, Aug 03, 2009 at 03:05:10PM +0000, Taylor R Campbell wrote:
  >  Perhaps find(1) should be extended with an option to set the `ls(1)
  >  flags' that it uses (emulates) for `find -ls', which is faster than
  >  either `find -exec ls' or xargs(1).

 Or perhaps ls should have a flag that disables the automatic column
 sizing.

 Another approach is to forcibly set the column widths...  or just
 prune all column padding.

 The former isn't entirely trivial because there are a lot of cases.

 The latter would give something like

     	       -type b -o -type c \) -print0 | \
    -	xargs -0 ls -ldgTq | sort +9 > $LIST
    +	xargs -0 ls -ldgTq | sed 's/  */ /g' | sort +9 > $LIST

 although looking at this I am starting to have real doubts about that
 sort +9. It sorts the regular files and directories by name, but the
 devices by year...

 -- 
 David A. Holland
 dholland@netbsd.org

From: David Laight <david@l8s.co.uk>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/40977: /var/backups/work/device.current is overly sensitive to ls(1) columnar alignment
Date: Mon, 7 Sep 2009 21:16:14 +0100

 On Mon, Sep 07, 2009 at 07:40:04PM +0000, David Holland wrote:
 >  Another approach is to forcibly set the column widths...  or just
 >  prune all column padding.
 ...
 >  The latter would give something like
 >  
 >      	       -type b -o -type c \) -print0 | \
 >     -	xargs -0 ls -ldgTq | sort +9 > $LIST
 >     +	xargs -0 ls -ldgTq | sed 's/  */ /g' | sort +9 > $LIST
 >  
 >  although looking at this I am starting to have real doubts about that
 >  sort +9. It sorts the regular files and directories by name, but the
 >  devices by year...

 Perhaps: sed -e 's/  */ /g' -e 's/, /,/'

 However I'm 100% sure that spaces are guaranteed after user and group
 names (I've seen 'ls' that don't generate them for maximal length names).

 Not to mention the problems of spaces in user/group names and odd
 characters in filenames.

 So maybe 'ls -nb' should be used.
 (Netbsd's ls only seems to have gained -b recently!)

 	David

 -- 
 David Laight: david@l8s.co.uk

From: Alan Barrett <apb@cequrux.com>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/40977: /var/backups/work/device.current is overly
 sensitive to ls(1) columnar alignment
Date: Mon, 7 Sep 2009 22:38:47 +0200

 On Mon, 07 Sep 2009, David Laight wrote:
 >  Not to mention the problems of spaces in user/group names and odd
 >  characters in filenames.
 >  
 >  So maybe 'ls -nb' should be used.
 >  (Netbsd's ls only seems to have gained -b recently!)

 What about extending "find -ls" to take options?  I mean, adding a new
 primary to find(1) that works a lot like "-ls" but takes options.  Then
 ew could use

 	find -s ... -ls-with-options -ldgTqnb

 and eliminate the xargs and sort.

 --apb (Alan Barrett)

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/40977: /var/backups/work/device.current is overly
	sensitive to ls(1) columnar alignment
Date: Mon, 7 Sep 2009 23:27:00 +0000

 On Mon, Sep 07, 2009 at 08:40:06PM +0000, Alan Barrett wrote:
  >>  Not to mention the problems of spaces in user/group names and odd
  >>  characters in filenames.

 Because the file name is last, it doesn't really matter unless one
 turns up with a \n in it. But since this is /dev, that shouldn't
 happen and if it does the resulting lossage will encourage the
 sysadmin to clean it up. So I'm not worried about that. Then again,
 adding -b to the ls won't hurt either.

 I'm inclined to say that anyone who uses spaces in user and group
 names deserves the consequences.

 Ultimately it doesn't really matter if the sort is meaningful as long
 as it's consistent. Note how long we've been sorting devices by
 last-modified-year without anyone noticing or caring.

  >>  So maybe 'ls -nb' should be used.
  >>  (Netbsd's ls only seems to have gained -b recently!)
  >  
  >  What about extending "find -ls" to take options?  I mean, adding a new
  >  primary to find(1) that works a lot like "-ls" but takes options.  Then
  >  ew could use
     ^^
 freudian slip there? :-)

  >  	find -s ... -ls-with-options -ldgTqnb
  >  
  >  and eliminate the xargs and sort.

 That seems like a lot of arbitrary grossness to wedge into find just
 for a small optimization to the nightly maintenance script. Each
 program should do one thing well...

 I think adding s/,  */,/ should be enough to take care of the sort
 problem.

 Anyone have thoughts on the column spacing part?

 -- 
 David A. Holland
 dholland@netbsd.org

From: Alan Barrett <apb@cequrux.com>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/40977: /var/backups/work/device.current is overly
 sensitive to ls(1) columnar alignment
Date: Tue, 8 Sep 2009 11:11:45 +0200

 On Mon, 07 Sep 2009, David Holland wrote:
 >   >  	find -s ... -ls-with-options -ldgTqnb
 >   >  
 >   >  and eliminate the xargs and sort.
 >  
 >  That seems like a lot of arbitrary grossness to wedge into find just
 >  for a small optimization to the nightly maintenance script. Each
 >  program should do one thing well...

 The general form of -ls-with-options is gross, but I do think that
 there's a place for something like the existing "-ls" except with full
 time display (like "ls -lT") and with escaping of problematic characters
 (like "ls -b").

 >  I think adding s/,  */,/ should be enough to take care of the sort
 >  problem.

 "find -s" is even easier, so I think we should do that anyway.

 >  Anyone have thoughts on the column spacing part?

 The main reason behind my suggestion to use "find -ls" was that it
 provides consistent column spacing.  I see that I neglected to mention
 that in my message yesterday, but I did mention it in my message soon
 after the PR was created.  The main reason why plain "find -ls" is
 not good enough is that it switches between displaying the year or
 displaying the time of day, according to how old the file is.

 We could use an awk script to reformat the output with fixed column
 widths.

 We could add yet another ls option to give fixed column widths; I see
 that some letters of the alphabet remain unused for ls(1) options.

 --apb (Alan Barrett)

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.