NetBSD Problem Report #10625

Received: (qmail 14773 invoked from network); 18 Jul 2000 08:21:36 -0000
Message-Id: <200007180821.BAA01217@nbwww.isc.org>
Date: Tue, 18 Jul 2000 01:21:35 -0700 (PDT)
From: uklaus@hgb-leipzig.de
Reply-To: uklaus@hgb-leipzig.de
To: gnats-bugs@gnats.netbsd.org
Subject: /usr/bin/cmp is unable to compare rather large files
X-Send-Pr-Version: www-1.0

>Number:         10625
>Category:       bin
>Synopsis:       /usr/bin/cmp is unable to compare rather large files
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    bin-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Jul 18 08:22:01 +0000 2000
>Closed-Date:    Fri Apr 04 22:14:17 +0000 2003
>Last-Modified:  Fri Apr 04 22:14:17 +0000 2003
>Originator:     Uwe Klaus
>Release:        1.4.2
>Organization:
HGB Leipzig
>Environment:
NetBSD hgb1.hgb-leipzig.de 1.4.2 NetBSD 1.4.2 (UWE) #0: Tue Jun 27 17:56:21 GMT 2000     root@hgb1.hgb-leipzig.de:/usr/src/sys/arch/i386/compile/UWE i386

>Description:
<root@hgb1> ~# ls -lai /wd0/Backup/media48.tar.gz /wd1/Backup/media48.tar.gz 
   7942 -rw-r--r--  1 root  daemon  1409578717 Jul 12 08:16 /wd0/Backup/media48.tar.gz
4551315 -rw-r--r--  1 root  daemon  1409578717 Jul 12 08:16 /wd1/Backup/media48.tar.gz
<root@hgb1> ~# cmp /wd0/Backup/media48.tar.gz /wd1/Backup/media48.tar.gz 
cmp: /wd1/Backup/media48.tar.gz: Cannot allocate memory


>How-To-Repeat:

>Fix:


>Release-Note:
>Audit-Trail:

From: woods@weird.com (Greg A. Woods)
To: uklaus@hgb-leipzig.de
Cc: gnats-bugs@gnats.netbsd.org,
	netbsd-bugs@NetBSD.ORG (NetBSD Bugs and PR posting List)
Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files
Date: Thu, 20 Jul 2000 12:10:02 -0400 (EDT)

 [ On Tuesday, July 18, 2000 at 01:21:35 (-0700), uklaus@hgb-leipzig.de wrote: ]
 > Subject: bin/10625: /usr/bin/cmp is unable to compare rather large files
 >
 > <root@hgb1> ~# ls -lai /wd0/Backup/media48.tar.gz /wd1/Backup/media48.tar.gz 
 >    7942 -rw-r--r--  1 root  daemon  1409578717 Jul 12 08:16 /wd0/Backup/media48.tar.gz
 > 4551315 -rw-r--r--  1 root  daemon  1409578717 Jul 12 08:16 /wd1/Backup/media48.tar.gz
 > <root@hgb1> ~# cmp /wd0/Backup/media48.tar.gz /wd1/Backup/media48.tar.gz 
 > cmp: /wd1/Backup/media48.tar.gz: Cannot allocate memory

 because of BAD non-unix-like programming practices....

 The damn program uses mmap()!

 If read() isn't fast enough then something's wrong in the land of *BSD....

 (Of course if mmap() is broken and can't handle paging transparently
 through any reasonable number of *VERY* large files then something's
 very wrong with it too....)

 -- 
 							Greg A. Woods

 +1 416 218-0098      VE3TCP      <gwoods@acm.org>      <robohack!woods>
 Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>

From: Jaromír Doleček <dolecek@ibis.cz>
To: gnats-bugs@gnats.netbsd.org
Cc: uklaus@hgb-leipzig.de
Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files
Date: Thu, 20 Jul 2000 19:30:56 +0200 (CEST)

 It turned out the thing called mmap() with MAP_PRIVATE|MAP_FILE, which is
 of course wrong - the MAP_PRIVATE should not be there.

 BTW - watching madvise(MADV_SEQUENTIAL,...) in work is quite cool :)

 Jaromir

 Greg A. Woods wrote:
 > [ On Tuesday, July 18, 2000 at 01:21:35 (-0700), uklaus@hgb-leipzig.de wrote: ]
 > > Subject: bin/10625: /usr/bin/cmp is unable to compare rather large files
 > >
 > > <root@hgb1> ~# ls -lai /wd0/Backup/media48.tar.gz /wd1/Backup/media48.tar.gz 
 > >    7942 -rw-r--r--  1 root  daemon  1409578717 Jul 12 08:16 /wd0/Backup/media48.tar.gz
 > > 4551315 -rw-r--r--  1 root  daemon  1409578717 Jul 12 08:16 /wd1/Backup/media48.tar.gz
 > > <root@hgb1> ~# cmp /wd0/Backup/media48.tar.gz /wd1/Backup/media48.tar.gz 
 > > cmp: /wd1/Backup/media48.tar.gz: Cannot allocate memory
 > 
 > because of BAD non-unix-like programming practices....
 > 
 > The damn program uses mmap()!
 > 
 > If read() isn't fast enough then something's wrong in the land of *BSD....
 > 
 > (Of course if mmap() is broken and can't handle paging transparently
 > through any reasonable number of *VERY* large files then something's
 > very wrong with it too....)
 > 
 > -- 
 > 							Greg A. Woods
 > 
 > +1 416 218-0098      VE3TCP      <gwoods@acm.org>      <robohack!woods>
 > Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>
 > 
 > 


 -- 
 Jaromir Dolecek <jdolecek@NetBSD.org>      http://www.ics.muni.cz/~dolecek/
 @@@@  Wanna a real operating system ? Go and get NetBSD, damn!  @@@@

From: woods@weird.com (Greg A. Woods)
To: gnats-bugs@gnats.netbsd.org (NetBSD GNATS submissions and followups),
	netbsd-bugs@NetBSD.ORG (NetBSD Bugs and PR posting List)
Cc:  
Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files 
Date: Wed, 26 Jul 2000 23:33:22 -0400 (EDT)

 [ On Wednesday, July 26, 2000 at 18:45:44 (-0700), R. C. Dowdeswell wrote: ]
 > Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files 
 >
 > Well, mmap() by its very operation can't handle files that are too
 > large, especially on 32bit arches.  Mmap(2) will map the entire
 > file into the processes address space, which is limited.  Looks
 > like the above files together total 2.6GB and cmp only uses two
 > mmaps so it is conceivable that problems might occur.

 Ah yes -- I guess didn't really pay attention to that little detail....

 > The issue is not whether read(2) or mmap(2) is used, but rather
 > the fact that cmp tries to get both files in their entirety into
 > its process address space.

 I think those two issues are orthogonal, no?  Avoiding mmap() would have
 prevented this bug from ever even peeping over the horizon.

 I can see how mmap() can be more efficient in terms of CPU resources and
 memory throughput (buffers never have to be copied to user-space if the
 kernel does the mapping and page-faulting carefully) but in the case of
 a program like cmp there's almost always going to be lots of time spent
 waiting for disks to spin (and I suspect the ratio of secondary storage
 access speeds and CPU/RAM speeds is fairly constant across all major
 classifications of machines too).

 So, given the extra complications and complexities that mmap()
 introduces into cmp I've still got to wonder at the sanity of it all.
 Is mmap() the hammer in every programmer's tool kit these days?

 What's even more silly is that if indeed mmap() is a much more efficient
 way to access files (at least those smaller than SIZE_T_MAX :-) then
 wouldn't it be a heck of a lot smarter and simpler to write this code
 once in a stdio-like library (pkgsrc/devel/sfio already does this, for
 example) and then keep all the applications dead simple and straight
 forward?

 In fact now that SFIO is really freeware (well no fee for redistribution
 is permitted) perhaps it should be considered directly.

 >  One would have the same problem if
 > one tried to compare these two files with only two read(2)s.  :-)

 nobody in their right mind would try to do that!  ;-)

 > In fact perusing the source code (prior to being fixed), it will
 > only fail for large files that are not too large on 32bit
 > architechtures.  :-)

 Yup!

 However I'm not sure the bug is *really* fixed.  Shouldn't the test used
 to avoid mmap() be to see if the length is bigger than (SIZE_T_MAX / 2)?

 Does mmap() really fail if the file is bigger than the *remaining* size
 of the process's address space (the manual page makes no obvious claims
 along these lines)?

 -- 
 							Greg A. Woods

 +1 416 218-0098      VE3TCP      <gwoods@acm.org>      <robohack!woods>
 Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>

From: Andrew Brown <atatat@atatdot.net>
To: "Greg A. Woods" <woods@weird.com>
Cc: NetBSD GNATS submissions and followups <gnats-bugs@gnats.netbsd.org>,
   NetBSD Bugs and PR posting List <netbsd-bugs@netbsd.org>
Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files
Date: Thu, 27 Jul 2000 10:50:36 -0400

 >However I'm not sure the bug is *really* fixed.  Shouldn't the test used
 >to avoid mmap() be to see if the length is bigger than (SIZE_T_MAX / 2)?

 if mmap is so great, wouldn't it make sense to use a moving mmap (ie,
 mmap, scan, munmap, mmap next chunk...) instead of falling back to
 plain old read?

 then again...if mmap is so great, why doesn't cat use it?  then cat
 would be just six system calls for each (regular) file.

 >Does mmap() really fail if the file is bigger than the *remaining* size
 >of the process's address space (the manual page makes no obvious claims
 >along these lines)?

 common sense says it should.

 -- 
 |-----< "CODE WARRIOR" >-----|
 codewarrior@daemon.org             * "ah!  i see you have the internet
 twofsonet@graffiti.com (Andrew Brown)                that goes *ping*!"
 andrew@crossbar.com       * "information is power -- share the wealth."

From: woods@weird.com (Greg A. Woods)
To: NetBSD GNATS submissions and followups <gnats-bugs@gnats.netbsd.org>,
	NetBSD Bugs and PR posting List <netbsd-bugs@netbsd.org>
Cc:  
Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files
Date: Thu, 27 Jul 2000 15:29:23 -0400 (EDT)

 [ On Thursday, July 27, 2000 at 10:50:36 (-0400), Andrew Brown wrote: ]
 > Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files
 >
 > >However I'm not sure the bug is *really* fixed.  Shouldn't the test used
 > >to avoid mmap() be to see if the length is bigger than (SIZE_T_MAX / 2)?
 > 
 > if mmap is so great, wouldn't it make sense to use a moving mmap (ie,
 > mmap, scan, munmap, mmap next chunk...) instead of falling back to
 > plain old read?

 Well, yeah!

 > then again...if mmap is so great, why doesn't cat use it?  then cat
 > would be just six system calls for each (regular) file.

 "If I had a hammer...."  :-)

 > >Does mmap() really fail if the file is bigger than the *remaining* size
 > >of the process's address space (the manual page makes no obvious claims
 > >along these lines)?
 > 
 > common sense says it should.

 There've been some more or less similarly stupid bugs in various mmap()
 implementations in the past....

 -- 
 							Greg A. Woods

 +1 416 218-0098      VE3TCP      <gwoods@acm.org>      <robohack!woods>
 Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>

From: woods@weird.com (Greg A. Woods)
To: netbsd-bugs@NetBSD.ORG (NetBSD Bugs and PR posting List),
	gnats-bugs@gnats.netbsd.org (NetBSD GNATS submissions and followups)
Cc:  
Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files 
Date: Thu, 27 Jul 2000 15:35:48 -0400 (EDT)

 [ On Thursday, July 27, 2000 at 09:15:04 (-0700), R. C. Dowdeswell wrote: ]
 > Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files 
 >
 > Well the current code falls back to stdio in any case where the
 > mmap(2)s fail, so it should work.
 > 
 > SIZE_T_MAX / 2 is still too large, because it does not account for
 > the kernel address space and the memory that is used by cmp for
 > other purposes.

 Yeah, I know -- but it's at least closer to correct than the currently implemented test!

 >  Probably the best approach would be to use mmap(2)
 > in a manner more consistent with how one would use read(2), i.e.:
 > mmap(2) relatively small chunks of the file in a loop.

 Hmmm....   Sfio springs to mind again here....

 > >of the process's address space (the manual page makes no obvious claims
 > >along these lines)?
 > 
 > It really has to fail, since there is no way to give you back a
 > char * that can access data beyond the end of your address space.
 > And in fact it is more hairy than that because it must find a free
 > contiguous region in your address space to hand back to you.  This
 > may be the reason that 2.6GB of files couldn't be mapped.

 But does it actually *really* always fail when there's no more room in
 your maximum allowed process size?  I've not tested it yet, and I am
 very leary of mmap() given previous bugs it has suffered.

 If it does fail then this failure case should be documented (more?)
 clearly in mmap(2).

 Is there a regression test to make sure it continues to fail when it
 should?  It doesn't appear src/regress/sys/uvm/mmap/mmap.c has a test
 for this case....

 -- 
 							Greg A. Woods

 +1 416 218-0098      VE3TCP      <gwoods@acm.org>      <robohack!woods>
 Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>

From: woods@weird.com (Greg A. Woods)
To: netbsd-bugs@netbsd.org (NetBSD Bugs and PR posting List), gnats-bugs@gnats.netbsd.org (NetBSD GNATS submissions and followups)
Cc:  
Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files 
Date: Thu, 27 Jul 2000 15:35:48 -0400 (EDT)

 [ On Thursday, July 27, 2000 at 09:15:04 (-0700), R. C. Dowdeswell wrote: ]
 > Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files 
 >
 > Well the current code falls back to stdio in any case where the
 > mmap(2)s fail, so it should work.
 > 
 > SIZE_T_MAX / 2 is still too large, because it does not account for
 > the kernel address space and the memory that is used by cmp for
 > other purposes.

 Yeah, I know -- but it's at least closer to correct than the currently implemented test!

 >  Probably the best approach would be to use mmap(2)
 > in a manner more consistent with how one would use read(2), i.e.:
 > mmap(2) relatively small chunks of the file in a loop.

 Hmmm....   Sfio springs to mind again here....

 > >of the process's address space (the manual page makes no obvious claims
 > >along these lines)?
 > 
 > It really has to fail, since there is no way to give you back a
 > char * that can access data beyond the end of your address space.
 > And in fact it is more hairy than that because it must find a free
 > contiguous region in your address space to hand back to you.  This
 > may be the reason that 2.6GB of files couldn't be mapped.

 But does it actually *really* always fail when there's no more room in
 your maximum allowed process size?  I've not tested it yet, and I am
 very leary of mmap() given previous bugs it has suffered.

 If it does fail then this failure case should be documented (more?)
 clearly in mmap(2).

 Is there a regression test to make sure it continues to fail when it
 should?  It doesn't appear src/regress/sys/uvm/mmap/mmap.c has a test
 for this case....

 -- 
                                                         Greg A. Woods

 +1 416 218-0098      VE3TCP      <gwoods@acm.org>      <robohack!woods>
 Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>



From: woods@weird.com (Greg A. Woods)
To: NetBSD GNATS submissions and followups <gnats-bugs@gnats.netbsd.org>,
	NetBSD Bugs and PR posting List <netbsd-bugs@netbsd.org>
Cc:  
Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files
Date: Thu, 27 Jul 2000 15:29:23 -0400 (EDT)

 [ On Thursday, July 27, 2000 at 10:50:36 (-0400), Andrew Brown wrote: ]
 > Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files
 >
 > >However I'm not sure the bug is *really* fixed.  Shouldn't the test used
 > >to avoid mmap() be to see if the length is bigger than (SIZE_T_MAX / 2)?
 > 
 > if mmap is so great, wouldn't it make sense to use a moving mmap (ie,
 > mmap, scan, munmap, mmap next chunk...) instead of falling back to
 > plain old read?

 Well, yeah!

 > then again...if mmap is so great, why doesn't cat use it?  then cat
 > would be just six system calls for each (regular) file.

 "If I had a hammer...."  :-)

 > >Does mmap() really fail if the file is bigger than the *remaining* size
 > >of the process's address space (the manual page makes no obvious claims
 > >along these lines)?
 > 
 > common sense says it should.

 There've been some more or less similarly stupid bugs in various mmap()
 implementations in the past....

 -- 
                                                         Greg A. Woods

 +1 416 218-0098      VE3TCP      <gwoods@acm.org>      <robohack!woods>
 Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>



From: Andrew Brown <atatat@atatdot.net>
To: "Greg A. Woods" <woods@weird.com>
Cc: NetBSD GNATS submissions and followups <gnats-bugs@gnats.netbsd.org>,
	NetBSD Bugs and PR posting List <netbsd-bugs@netbsd.org>
Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files
Date: Thu, 27 Jul 2000 10:50:36 -0400

 >However I'm not sure the bug is *really* fixed.  Shouldn't the test used
 >to avoid mmap() be to see if the length is bigger than (SIZE_T_MAX / 2)?

 if mmap is so great, wouldn't it make sense to use a moving mmap (ie,
 mmap, scan, munmap, mmap next chunk...) instead of falling back to
 plain old read?

 then again...if mmap is so great, why doesn't cat use it?  then cat
 would be just six system calls for each (regular) file.

 >Does mmap() really fail if the file is bigger than the *remaining* size
 >of the process's address space (the manual page makes no obvious claims
 >along these lines)?

 common sense says it should.

 -- 
 |-----< "CODE WARRIOR" >-----|
 codewarrior@daemon.org             * "ah!  i see you have the internet
 twofsonet@graffiti.com (Andrew Brown)                that goes *ping*!"
 andrew@crossbar.com       * "information is power -- share the wealth."



From: woods@weird.com (Greg A. Woods)
To: gnats-bugs@gnats.netbsd.org (NetBSD GNATS submissions and followups),
	netbsd-bugs@netbsd.org (NetBSD Bugs and PR posting List)
Cc:  
Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files 
Date: Wed, 26 Jul 2000 23:33:22 -0400 (EDT)

 [ On Wednesday, July 26, 2000 at 18:45:44 (-0700), R. C. Dowdeswell wrote: ]
 > Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files 
 >
 > Well, mmap() by its very operation can't handle files that are too
 > large, especially on 32bit arches.  Mmap(2) will map the entire
 > file into the processes address space, which is limited.  Looks
 > like the above files together total 2.6GB and cmp only uses two
 > mmaps so it is conceivable that problems might occur.

 Ah yes -- I guess didn't really pay attention to that little detail....

 > The issue is not whether read(2) or mmap(2) is used, but rather
 > the fact that cmp tries to get both files in their entirety into
 > its process address space.

 I think those two issues are orthogonal, no?  Avoiding mmap() would have
 prevented this bug from ever even peeping over the horizon.

 I can see how mmap() can be more efficient in terms of CPU resources and
 memory throughput (buffers never have to be copied to user-space if the
 kernel does the mapping and page-faulting carefully) but in the case of
 a program like cmp there's almost always going to be lots of time spent
 waiting for disks to spin (and I suspect the ratio of secondary storage
 access speeds and CPU/RAM speeds is fairly constant across all major
 classifications of machines too).

 So, given the extra complications and complexities that mmap()
 introduces into cmp I've still got to wonder at the sanity of it all.
 Is mmap() the hammer in every programmer's tool kit these days?

 What's even more silly is that if indeed mmap() is a much more efficient
 way to access files (at least those smaller than SIZE_T_MAX :-) then
 wouldn't it be a heck of a lot smarter and simpler to write this code
 once in a stdio-like library (pkgsrc/devel/sfio already does this, for
 example) and then keep all the applications dead simple and straight
 forward?

 In fact now that SFIO is really freeware (well no fee for redistribution
 is permitted) perhaps it should be considered directly.

 >  One would have the same problem if
 > one tried to compare these two files with only two read(2)s.  :-)

 nobody in their right mind would try to do that!  ;-)

 > In fact perusing the source code (prior to being fixed), it will
 > only fail for large files that are not too large on 32bit
 > architechtures.  :-)

 Yup!

 However I'm not sure the bug is *really* fixed.  Shouldn't the test used
 to avoid mmap() be to see if the length is bigger than (SIZE_T_MAX / 2)?

 Does mmap() really fail if the file is bigger than the *remaining* size
 of the process's address space (the manual page makes no obvious claims
 along these lines)?

 -- 
                                                         Greg A. Woods

 +1 416 218-0098      VE3TCP      <gwoods@acm.org>      <robohack!woods>
 Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>



From: woods@weird.com (Greg A. Woods)
To: uklaus@hgb-leipzig.de
Cc: gnats-bugs@gnats.netbsd.org, netbsd-bugs@netbsd.org (NetBSD Bugs and PR posting List)
Subject: Re: bin/10625: /usr/bin/cmp is unable to compare rather large files
Date: Thu, 20 Jul 2000 12:10:02 -0400 (EDT)

 [ On Tuesday, July 18, 2000 at 01:21:35 (-0700), uklaus@hgb-leipzig.de wrote: ]
 > Subject: bin/10625: /usr/bin/cmp is unable to compare rather large files
 >
 > <root@hgb1> ~# ls -lai /wd0/Backup/media48.tar.gz /wd1/Backup/media48.tar.gz 
 >    7942 -rw-r--r--  1 root  daemon  1409578717 Jul 12 08:16 /wd0/Backup/media48.tar.gz
 > 4551315 -rw-r--r--  1 root  daemon  1409578717 Jul 12 08:16 /wd1/Backup/media48.tar.gz
 > <root@hgb1> ~# cmp /wd0/Backup/media48.tar.gz /wd1/Backup/media48.tar.gz 
 > cmp: /wd1/Backup/media48.tar.gz: Cannot allocate memory

 because of BAD non-unix-like programming practices....

 The damn program uses mmap()!

 If read() isn't fast enough then something's wrong in the land of *BSD....

 (Of course if mmap() is broken and can't handle paging transparently
 through any reasonable number of *VERY* large files then something's
 very wrong with it too....)

 -- 
                                                         Greg A. Woods

 +1 416 218-0098      VE3TCP      <gwoods@acm.org>      <robohack!woods>
 Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>


State-Changed-From-To: open->feedback 
State-Changed-By: ross 
State-Changed-When: Wed Feb 12 17:35:39 PST 2003 
State-Changed-Why:  
Looks like Jaromir fixed this bug 2.5 years ago. 
State-Changed-From-To: feedback->closed 
State-Changed-By: perry 
State-Changed-When: Fri Apr 4 14:14:01 PST 2003 
State-Changed-Why:  
Looks like Jaromir fixed this bug 2.5 years ago.  
>Unformatted:

Home	PR Database Search