NetBSD Problem Report #43697

From yamt@NetBSD.org  Mon Aug  2 22:51:35 2010
Return-Path: <yamt@NetBSD.org>
Received: by www.NetBSD.org (Postfix, from userid 1270)
	id 4B64363BB73; Mon,  2 Aug 2010 22:51:35 +0000 (UTC)
Message-Id: <20100802225135.4B64363BB73@www.NetBSD.org>
Date: Mon,  2 Aug 2010 22:51:35 +0000 (UTC)
From: yamt@NetBSD.org
Reply-To: yamt@NetBSD.org
To: gnats-bugs@NetBSD.org
Subject: "No such file or directory" on build.sh -j 128
X-Send-Pr-Version: 3.95

>Number:         43697
>Category:       bin
>Synopsis:       "No such file or directory" on build.sh -j 128
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lukem
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Aug 02 22:55:00 +0000 2010
>Closed-Date:    Sat Jun 10 07:30:13 +0000 2023
>Last-Modified:  Sat Jun 10 07:30:13 +0000 2023
>Originator:     YAMAMOTO Takashi
>Release:        NetBSD-current
>Organization:

>Environment:


Architecture: i386
Machine: i386
>Description:
	build.sh -j 128 often fail.  it seems where to fail is vary.
	the following is an example.

--- /nfs/eos-fs.nfskuro/obj/o429.i386.NetBSD-5.99.38-i386/siro/nbsd/src/destdir.i386/usr/lib/libss.so.6.0 ---
i486--netbsdelf-install: libss.so.6.0: stat: No such file or directory

	/tmp is tmpfs and everything else are nfs.

>How-To-Repeat:
	cd /siro/nbsd/src
	./build.sh -m i386 -M /nfs/eos-fs.nfskuro/obj/o429.i386.NetBSD-5.99.38-i386 -N 3 -j 128 -U -u -x -X /siro/nbsd/src/../xsrc/ release

	(the objdir was not exist before running build.sh.)
>Fix:


>Release-Note:

>Audit-Trail:
From: David Laight <david@l8s.co.uk>
To: gnats-bugs@NetBSD.org
Cc: gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: bin/43697: "No such file or directory" on build.sh -j 128
Date: Thu, 5 Aug 2010 21:30:38 +0100

 On Mon, Aug 02, 2010 at 10:55:01PM +0000, yamt@NetBSD.org wrote:
 > >Number:         43697
 > >Synopsis:       "No such file or directory" on build.sh -j 128

 At a guess this is a side effect of hitting some limit on
 the number of processes the user is allowed.

 You need a LOT more than 128!

 	David

 -- 
 David Laight: david@l8s.co.uk

From: christos@zoulas.com (Christos Zoulas)
To: David Laight <david@l8s.co.uk>, gnats-bugs@NetBSD.org
Cc: gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: bin/43697: "No such file or directory" on build.sh -j 128
Date: Thu, 5 Aug 2010 18:43:11 -0400

 On Aug 5,  9:30pm, david@l8s.co.uk (David Laight) wrote:
 -- Subject: Re: bin/43697: "No such file or directory" on build.sh -j 128

 | On Mon, Aug 02, 2010 at 10:55:01PM +0000, yamt@NetBSD.org wrote:
 | > >Number:         43697
 | > >Synopsis:       "No such file or directory" on build.sh -j 128
 | 
 | At a guess this is a side effect of hitting some limit on
 | the number of processes the user is allowed.
 | 
 | You need a LOT more than 128!

 Well, it could be number of fd's and select...

 christos

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: bin/43697: "No such file or directory" on build.sh -j 128
Date: Mon, 9 Aug 2010 03:23:08 +0000

 On Thu, Aug 05, 2010 at 08:30:05PM +0000, David Laight wrote:
  >  At a guess this is a side effect of hitting some limit on
  >  the number of processes the user is allowed.
  >  
  >  You need a LOT more than 128!

 I was running some excessively parallelized builds to burn in some new
 machines a couple months ago, and at least once forgot to adjust
 maxproc, and it doesn't cause this behavior.

 I fixed the problem I ran into at the time, too, so unless this is a
 newly invented parallelism bug in bsd.lib.mk I vote for it being tmpfs
 lossage.

 -- 
 David A. Holland
 dholland@netbsd.org

From: yamt@mwd.biglobe.ne.jp (YAMAMOTO Takashi)
To: david@l8s.co.uk
Cc: gnats-bugs@NetBSD.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: bin/43697: "No such file or directory" on build.sh -j 128
Date: Tue, 10 Aug 2010 03:10:27 +0000 (UTC)

 hi,

 > On Mon, Aug 02, 2010 at 10:55:01PM +0000, yamt@NetBSD.org wrote:
 >> >Number:         43697
 >> >Synopsis:       "No such file or directory" on build.sh -j 128
 > 
 > At a guess this is a side effect of hitting some limit on
 > the number of processes the user is allowed.
 > 
 > You need a LOT more than 128!

 i already have a lot more than 128. :-)

 YAMAMOTO Takashi

 nfskuro% ulimit -a
 -t: cpu time (seconds)         unlimited
 -f: file size (blocks)         unlimited
 -d: data seg size (kbytes)     3145728
 -s: stack size (kbytes)        2048
 -c: core file size (blocks)    unlimited
 -m: resident set size (kbytes) 3141632
 -l: locked-in-memory size (kb) 1047210
 -u: processes                  29999
 -n: file descriptors           99999
 -N  9: socket buffer size (kb) unlimited
 -v: virtual memory size (kb)   unlimited
 nfskuro% sysctl kern.maxfiles
 kern.maxfiles = 99999
 nfskuro% 

 > 
 > 	David
 > 
 > -- 
 > David Laight: david@l8s.co.uk

From: Nicolas Joly <njoly@pasteur.fr>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: bin/43697: "No such file or directory" on build.sh -j 128
Date: Tue, 10 Aug 2010 08:31:54 +0200

 On Mon, Aug 02, 2010 at 10:55:01PM +0000, yamt@NetBSD.org wrote:
 > >Number:         43697
 > >Category:       bin
 > >Synopsis:       "No such file or directory" on build.sh -j 128
 [...]
 > >Description:
 > 	build.sh -j 128 often fail.  it seems where to fail is vary.
 > 	the following is an example.
 > 
 > --- /nfs/eos-fs.nfskuro/obj/o429.i386.NetBSD-5.99.38-i386/siro/nbsd/src/destdir.i386/usr/lib/libss.so.6.0 ---
 > i486--netbsdelf-install: libss.so.6.0: stat: No such file or directory
 > 
 > 	/tmp is tmpfs and everything else are nfs.
 > 
 > >How-To-Repeat:
 > 	cd /siro/nbsd/src
 > 	./build.sh -m i386 -M /nfs/eos-fs.nfskuro/obj/o429.i386.NetBSD-5.99.38-i386 -N 3 -j 128 -U -u -x -X /siro/nbsd/src/../xsrc/ release
 > 
 > 	(the objdir was not exist before running build.sh.)

 Looks familiar ... parallel mkdebug builds are currently broken.

   http://mail-index.netbsd.org/tech-toolchain/2010/05/26/msg001283.html

 -- 
 Nicolas Joly

 Biological Software and Databanks.
 Institut Pasteur, Paris.

From: christos@zoulas.com (Christos Zoulas)
To: yamt@mwd.biglobe.ne.jp (YAMAMOTO Takashi), david@l8s.co.uk
Cc: gnats-bugs@NetBSD.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: bin/43697: "No such file or directory" on build.sh -j 128
Date: Tue, 10 Aug 2010 06:41:46 -0400

 On Aug 10,  3:10am, yamt@mwd.biglobe.ne.jp (YAMAMOTO Takashi) wrote:
 -- Subject: Re: bin/43697: "No such file or directory" on build.sh -j 128

 | hi,
 | 
 | > On Mon, Aug 02, 2010 at 10:55:01PM +0000, yamt@NetBSD.org wrote:
 | >> >Number:         43697
 | >> >Synopsis:       "No such file or directory" on build.sh -j 128
 | > 
 | > At a guess this is a side effect of hitting some limit on
 | > the number of processes the user is allowed.
 | > 
 | > You need a LOT more than 128!
 | 
 | i already have a lot more than 128. :-)

 I guess it only uses select when you don't have poll. And for select
 it is not the limit that matters; there is FD_SETSIZE too.

 christos

State-Changed-From-To: open->feedback
State-Changed-By: lukem@NetBSD.org
State-Changed-When: Thu, 18 May 2023 13:45:31 +0000
State-Changed-Why:
Is this still an issue?
Per Nicolas Joly's comment on 2010-08-10 and the link to the mail thread
  http://mail-index.netbsd.org/tech-toolchain/2010/05/26/msg001283.html
I looked into the current history of share/mk/bsd.lib.mk,
and there appear to be a few -j MKDEBUG fixes, including by mrg on 2015-06-04
per the thread
   http://mail-index.netbsd.org/tech-toolchain/2015/06/03/msg002616.html


Responsible-Changed-From-To: bin-bug-people->lukem
Responsible-Changed-By: lukem@NetBSD.org
Responsible-Changed-When: Sat, 20 May 2023 08:35:28 +0000
Responsible-Changed-Why:


State-Changed-From-To: feedback->closed
State-Changed-By: lukem@NetBSD.org
State-Changed-When: Sat, 10 Jun 2023 07:30:13 +0000
State-Changed-Why:
Unable to reproduce in -current.
There have been various fixes to -j handling in the build
since 2010.
It's /also/ possible it was an NFS client or server bug given
the directories listed in the description all seem to be on /nfs
and not in the tmpdir.


>Unformatted:

Home	PR Database Search