NetBSD Problem Report #45718

From campbell@mumble.net  Sat Dec 17 05:21:31 2011
Return-Path: <campbell@mumble.net>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by www.NetBSD.org (Postfix) with ESMTP id E7F2763BF13
	for <gnats-bugs@gnats.NetBSD.org>; Sat, 17 Dec 2011 05:21:30 +0000 (UTC)
Message-Id: <20111217052131.F1BA3982AB@pluto.mumble.net>
Date: Sat, 17 Dec 2011 05:21:31 +0000 (UTC)
From: Taylor R Campbell <campbell+netbsd@mumble.net>
Reply-To: Taylor R Campbell <campbell+netbsd@mumble.net>
To: gnats-bugs@gnats.NetBSD.org
Subject: processes sometimes get stuck and spin in vm_map
X-Send-Pr-Version: 3.95

>Number:         45718
>Category:       kern
>Synopsis:       processes sometimes get stuck and spin in vm_map
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          feedback
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Dec 17 05:25:00 +0000 2011
>Closed-Date:    
>Last-Modified:  Tue Apr 10 08:44:40 +0000 2018
>Originator:     Taylor R Campbell <campbell+netbsd@mumble.net>
>Release:        NetBSD 5.99.58
>Organization:
>Environment:
System: NetBSD oberon.local 5.99.58 NetBSD 5.99.58 (RIAMONOHACK) #4: Sat Dec 10 23:18:29 UTC 2011 root@oberon.local:/home/riastradh/netbsd/current/obj/sys/arch/i386/compile/RIAMONOHACK i386
Architecture: i386
Machine: i386
>Description:

	Over the past six to eight months (maybe longer), I've
	occasionally noticed that after a lot of heavy file system
	activity, some processes seem to get stuck.  They occupy some
	CPU time, but spend most of their time in vm_map.

	Tonight I tried to create a null mount, and mount_null got
	stuck like this; ^C did nothing, and umount wedged too.  A
	little while later, mount_null seemed to get unstuck, but when
	I tried again, it got stuck again.  Now several processes,
	including mount_null, zsh, and xargs, are stuck like this.

21204 root     117    0  2904K  728K vm_map/0   2:05 16.41% 16.41% mount_null

	In the past, the only way I have recovered from this situation
	is by rebooting.

	Memory is not scarce; I have a good half a gigabyte of RAM
	free, out of two gigabytes total, and plenty of swap handy.

	I haven't observed a pattern to what processes get stuck like
	this, although if there are file system mounts involved (which
	there often are because I use null mounts all the time), and
	any process is stuck trying to do something like `df -h /mnt',
	then any other process will get stuck if it tries to handle
	/mnt too.

>How-To-Repeat:

	Not sure.  Try a few concurrent rsyncs, cvs updates, finds,
	tars, &c.

>Fix:

	Yes, please!

>Release-Note:

>Audit-Trail:

State-Changed-From-To: open->feedback
State-Changed-By: rmind@NetBSD.org
State-Changed-When: Tue, 20 Dec 2011 11:28:03 +0000
State-Changed-Why:
Can you get into DDB (can use crash(8) as well) and grab (ps/l, t/a) the
backtrace of those LWPs?  Also, the output of 'show uvm'.


From: rudolf <netbsd@eq.cz>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/45718 (processes sometimes get stuck and spin in vm_map)
Date: Fri, 24 Feb 2012 15:48:22 +0100

 I've recently observerd the same phenomenon.

 "ps /l" in DDB revealed 4 lwps stuck in vm_map: xulrunner-bin, ksh, 
 login, cron.

 "bt /a <lwp>" for xulrunner-bin:
 sleepq_block
 cv_timedwait
 uvm_map_prepare
 uvm_map
 uvm_km_alloc
 uarea_poolpage_alloc
 pool_allocator_alloc
 pool_grow
 poop_get
 poop_cache_get_slow
 pool_cache_get_paddr
 uvm_uarea_alloc
 sys__lwp_create
 sy_call
 syscall

 "bt /a <lwp>" for the other three lwps looked the same, with "fork1, 
 sys_fork" instead of the "sys_lwp_create". Unfortunately, I've not 
 collected the "show uvm", maybe next time.

 My environment: i386 with netbsd-5 userland and netbsd-6 MONOLITHIC 
 kernel updated 2012/02/22, with PAE, 8 GB RAM.

 r.

From: rudolf <netbsd@eq.cz>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/45718 (processes sometimes get stuck and spin in vm_map)
Date: Mon, 27 Feb 2012 18:10:10 +0100

 This time 4 lwps in vm_map, 2x sh, 2x xulrunner-bin, all with the same 
 backtrace:
 sleepq_block
 cv_timedwait
 uvm_map_prepare
 uvm_map
 uvm_km_alloc
 pipe_ctor
 pool_cache_get_slow
 pool_cache_get_paddr
 pipe_create
 pipe1
 sys_pipe
 sy_call
 syscall

 Current UVM status:
 pagesize=4096, pagemask=0xfff, pageshift=12, ncolors=64
 2015729 VM pages: 445408 active, 0 inactive, 1479 wired, 1481348 free
 pages 135016 anon, 296929 file, 14942 exec
 freemin=512 free-target=682, wired-max=671909
 cpu0:
 faults=7976773, traps=7613655, intrs=6473188, ctxswitch=14547360
 softint=5749848, syscalls=136291738
 cpu1:
 faults=7132353, traps=6663764, intrs=0, ctxswitch=13848441
 softint=2613137, syscalls=103586607
 fault counts:
 noram=0, noanon=0, pgwait=0, pgrele=0
 ok relocks(total)=6605(6638)
 anget(retrys)=1330518(0)
 anapcopy=712393
 neighbor anon/obj pg=77203/1108057, gets(lock/unlock)=507179/6638
 cases: anon=1309558, anoncow=20966, obj=454788, prcopy=52356,
 przero=12489369
 daemon and swap counts:
 nswapdev=1, swpgavail=524457, swpages=524457, all other attributes = 0.

 r.

From: Taylor R Campbell <campbell+netbsd@mumble.net>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org, netbsd-bugs@netbsd.org,
	gnats-admin@netbsd.org, rmind@NetBSD.org
Subject: Re: kern/45718 (processes sometimes get stuck and spin in vm_map)
Date: Sat, 14 Apr 2012 16:57:01 +0000

 I caught some processes wedged in vm_map, although they don't seem to
 be spinning -- just wedged.  ps(1) reports these processes as zombies;
 top(1) does not report their taking any CPU time.  One is a subprocess
 of newsyslog; the other is a subprocess of git-pull.

 crash> t/a ca73daa0
 trace: pid 25970 lid 1 at 0xdc10781c
 sleepq_block(64,0,c0c5fd9d,c0ce63d0,0,0,0,c08e40d3,0,0) at sleepq_block+0xda
 cv_timedwait(c0d291d0,c0d291cc,64,dc1078e8,c0d28fe0,ffffffff,ffffffff,0,801727,c66ef880) at cv_timedwait+0x126
 uvm_map_prepare(c0d291c0,c0000000,40000,c0d28fe0,ffffffff,ffffffff,0,801727,dc107928,dc107b54) at uvm_map_prepare+0x167
 uvm_map(c0d291c0,dc1079b0,40000,c0d28fe0,ffffffff,ffffffff,0,801727,800002,0) at uvm_map+0x78
 uvm_km_alloc(c0d291c0,40000,0,800002,c0d15ea0,1,dc107a4c,c07e117a,c0d15ea0,1) at uvm_km_alloc+0xe6
 exec_pool_alloc(c0d15ea0,1,dc107a0c,c09548fd,0,0,0,0,0,0) at exec_pool_alloc+0x2b
 pool_grow(c0d15f14,1,c385ac12,0,0,c054,ca5ff000,c385ac09,9,c0d15f18) at pool_grow+0x2a
 pool_get(c0d15ea0,1,ce4e4a80,c098bb91,0,c31eec40,dc107adc,c056764b,c3137d00,c3137bc0) at pool_get+0x79
 execve_loadvm(8063f44,c055c480,dc107b3c,c3137d00,ca73daa0,0,8063f1c,c3ec4400,c4519400,c380e000) at execve_loadvm+0x1da
 execve1(ca73daa0,8063f1c,8063f3c,8063f44,c055c480,6300,dc107d1c,0,c06ba833,cd722744) at execve1+0x32
 sys_execve(ca73daa0,dc107cf4,dc107d1c,c08096f0,0,cdb17e3c,c0c8b800,dc107d30,c06babb9,cd722730) at sys_execve+0x30
 syscall(dc107d48,b3,ab,bfbf001f,806001f,8063f1c,8063f3c,bfbfec48,8063f1c,7d7b7cff) at syscall+0x95

 crash> t/a ccc79540
 trace: pid 21497 lid 1 at 0xdc32581c
 sleepq_block(64,0,c0c5fd9d,c0ce63d0,0,0,0,c08e40d3,0,0) at sleepq_block+0xda
 cv_timedwait(c0d291d0,c0d291cc,64,dc3258e8,c0d28fe0,ffffffff,ffffffff,0,801727,c66ef880) at cv_timedwait+0x126
 uvm_map_prepare(c0d291c0,c0000000,40000,c0d28fe0,ffffffff,ffffffff,0,801727,dc325928,dc325b54) at uvm_map_prepare+0x167
 uvm_map(c0d291c0,dc3259b0,40000,c0d28fe0,ffffffff,ffffffff,0,801727,800002,0) at uvm_map+0x78
 uvm_km_alloc(c0d291c0,40000,0,800002,c0d15ea0,1,dc325a4c,c07e117a,c0d15ea0,1) at uvm_km_alloc+0xe6
 exec_pool_alloc(c0d15ea0,1,dc325a0c,c09548fd,0,0,0,0,0,0) at exec_pool_alloc+0x2b
 pool_grow(c0d15f14,1,c387f80b,0,0,c054,cd15fa80,c387f809,2,c0d15f18) at pool_grow+0x2a
 pool_get(c0d15ea0,1,c88faae0,c057c7e1,cda0986c,10,0,c3889700,ccc79540,0) at pool_get+0x79
 execve_loadvm(bb92d404,c055c480,dc325b3c,c38b3e80,c30822f0,0,bb92d48c,c3ec5c00,c3ec4800,ca9b9800) at execve_loadvm+0x1da
 execve1(ccc79540,bb92d48c,8063f9c,bb92d404,c055c480,106,dc325d3c,c0840430,0,0) at execve1+0x32
 sys_execve(ccc79540,dc325cf4,dc325d1c,c08096f0,0,cd722a78,c3663000,cda09868,0,bbb81010) at sys_execve+0x30
 syscall(dc325d48,bb9200b3,ab,bfbf001f,bbbb001f,bb92d48c,8063f9c,bfbfe1f8,bb92d48c,bb92d49c) at syscall+0x95

From: Taylor R Campbell <campbell+netbsd@mumble.net>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org,
	netbsd-bugs@netbsd.org, gnats-admin@netbsd.org, rmind@NetBSD.org
Subject: Re: kern/45718 (processes sometimes get stuck and spin in vm_map)
Date: Sat, 14 Apr 2012 17:01:52 +0000

 Forgot to add: `show uvm' in crash(8) produces no output.  Also, this
 is under a 6.99.4 kernel from about 20120325 with a ~5.1ish userland
 (and crash(8) from a 6.99.4ish userland in a chroot).

From: matthew green <mrg@eterna.com.au>
To: Taylor R Campbell <campbell+netbsd@mumble.net>
Cc: kern-bug-people@netbsd.org, netbsd-bugs@netbsd.org,
    gnats-admin@netbsd.org, rmind@NetBSD.org, gnats-bugs@NetBSD.org,
    oster@netbsd.org, martin@netbsd.org
Subject: re: kern/45718 (processes sometimes get stuck and spin in vm_map)
Date: Sun, 15 Apr 2012 05:14:06 +1000

 > I caught some processes wedged in vm_map, although they don't seem to
 > be spinning -- just wedged.  ps(1) reports these processes as zombies;
 > top(1) does not report their taking any CPU time.  One is a subprocess
 > of newsyslog; the other is a subprocess of git-pull.
 > 
 > crash> t/a ca73daa0
 > trace: pid 25970 lid 1 at 0xdc10781c
 > sleepq_block(64,0,c0c5fd9d,c0ce63d0,0,0,0,c08e40d3,0,0) at sleepq_block+0xda
 > cv_timedwait(c0d291d0,c0d291cc,64,dc1078e8,c0d28fe0,ffffffff,ffffffff,0,801727,c66ef880) at cv_timedwait+0x126
 > uvm_map_prepare(c0d291c0,c0000000,40000,c0d28fe0,ffffffff,ffffffff,0,801727,dc107928,dc107b54) at uvm_map_prepare+0x167
 > uvm_map(c0d291c0,dc1079b0,40000,c0d28fe0,ffffffff,ffffffff,0,801727,800002,0) at uvm_map+0x78
 > uvm_km_alloc(c0d291c0,40000,0,800002,c0d15ea0,1,dc107a4c,c07e117a,c0d15ea0,1) at uvm_km_alloc+0xe6
 > exec_pool_alloc(c0d15ea0,1,dc107a0c,c09548fd,0,0,0,0,0,0) at exec_pool_alloc+0x2b
 > pool_grow(c0d15f14,1,c385ac12,0,0,c054,ca5ff000,c385ac09,9,c0d15f18) at pool_grow+0x2a
 > pool_get(c0d15ea0,1,ce4e4a80,c098bb91,0,c31eec40,dc107adc,c056764b,c3137d00,c3137bc0) at pool_get+0x79
 > execve_loadvm(8063f44,c055c480,dc107b3c,c3137d00,ca73daa0,0,8063f1c,c3ec4400,c4519400,c380e000) at execve_loadvm+0x1da
 > execve1(ca73daa0,8063f1c,8063f3c,8063f44,c055c480,6300,dc107d1c,0,c06ba833,cd722744) at execve1+0x32
 > sys_execve(ca73daa0,dc107cf4,dc107d1c,c08096f0,0,cdb17e3c,c0c8b800,dc107d30,c06babb9,cd722730) at sys_execve+0x30
 > syscall(dc107d48,b3,ab,bfbf001f,806001f,8063f1c,8063f3c,bfbfec48,8063f1c,7d7b7cff) at syscall+0x95

 this looks like the execargs pool leak that greg oster has a patch for.
 (reproduced below.)


 .mrg.

 Index: kern_exec.c
 ===================================================================
 RCS file: /cvsroot/src/sys/kern/kern_exec.c,v
 retrieving revision 1.349
 diff -u -p -r1.349 kern_exec.c
 --- kern_exec.c	9 Apr 2012 19:42:06 -0000	1.349
 +++ kern_exec.c	13 Apr 2012 20:28:14 -0000
 @@ -1991,6 +1991,8 @@ spawn_return(void *arg)
  		rw_exit(&exec_lock);
  	}

 +	execve_free_data(&spawn_data->sed_exec);
 +
  	/* release our refcount on the data */
  	spawn_exec_data_release(spawn_data);


From: Taylor R Campbell <campbell+netbsd@mumble.net>
To: matthew green <mrg@eterna.com.au>
Cc: kern-bug-people@netbsd.org, netbsd-bugs@netbsd.org,
	gnats-admin@netbsd.org, rmind@NetBSD.org,
	gnats-bugs@NetBSD.org, oster@netbsd.org, martin@netbsd.org
Subject: Re: kern/45718 (processes sometimes get stuck and spin in vm_map)
Date: Sat, 14 Apr 2012 19:23:58 +0000

    Date: Sun, 15 Apr 2012 05:14:06 +1000
    From: matthew green <mrg@eterna.com.au>

    this looks like the execargs pool leak that greg oster has a patch for.
    (reproduced below.)

    @@ -1991,6 +1991,8 @@ spawn_return(void *arg)
                    rw_exit(&exec_lock);
            }

    +	execve_free_data(&spawn_data->sed_exec);

 Isn't that code path reachable only if someone uses posix_spawn?
 Certainly nothing in a 5.1 userland will use posix_spawn, and although
 I suppose my having run the 6.99.4 tests in a chroot might have
 triggered it, it wouldn't explain the earlier problem I reported,
 because I believe I reported that before posix_spawn was imported.

From: Greg Oster <oster@cs.usask.ca>
To: matthew green <mrg@eterna.com.au>
Cc: Taylor R Campbell <campbell+netbsd@mumble.net>,
 kern-bug-people@netbsd.org, netbsd-bugs@netbsd.org, gnats-admin@netbsd.org,
 rmind@NetBSD.org, gnats-bugs@NetBSD.org, oster@netbsd.org,
 martin@netbsd.org
Subject: Re: kern/45718 (processes sometimes get stuck and spin in vm_map)
Date: Sat, 14 Apr 2012 22:55:25 -0600

 On Sun, 15 Apr 2012 05:14:06 +1000
 matthew green <mrg@eterna.com.au> wrote:

 > 
 > > I caught some processes wedged in vm_map, although they don't seem
 > > to be spinning -- just wedged.  ps(1) reports these processes as
 > > zombies; top(1) does not report their taking any CPU time.  One is
 > > a subprocess of newsyslog; the other is a subprocess of git-pull.
 > > 
 > > crash> t/a ca73daa0
 > > trace: pid 25970 lid 1 at 0xdc10781c
 > > sleepq_block(64,0,c0c5fd9d,c0ce63d0,0,0,0,c08e40d3,0,0) at
 > > sleepq_block+0xda
 > > cv_timedwait(c0d291d0,c0d291cc,64,dc1078e8,c0d28fe0,ffffffff,ffffffff,0,801727,c66ef880)
 > > at cv_timedwait+0x126
 > > uvm_map_prepare(c0d291c0,c0000000,40000,c0d28fe0,ffffffff,ffffffff,0,801727,dc107928,dc107b54)
 > > at uvm_map_prepare+0x167
 > > uvm_map(c0d291c0,dc1079b0,40000,c0d28fe0,ffffffff,ffffffff,0,801727,800002,0)
 > > at uvm_map+0x78
 > > uvm_km_alloc(c0d291c0,40000,0,800002,c0d15ea0,1,dc107a4c,c07e117a,c0d15ea0,1)
 > > at uvm_km_alloc+0xe6
 > > exec_pool_alloc(c0d15ea0,1,dc107a0c,c09548fd,0,0,0,0,0,0) at
 > > exec_pool_alloc+0x2b
 > > pool_grow(c0d15f14,1,c385ac12,0,0,c054,ca5ff000,c385ac09,9,c0d15f18)
 > > at pool_grow+0x2a
 > > pool_get(c0d15ea0,1,ce4e4a80,c098bb91,0,c31eec40,dc107adc,c056764b,c3137d00,c3137bc0)
 > > at pool_get+0x79
 > > execve_loadvm(8063f44,c055c480,dc107b3c,c3137d00,ca73daa0,0,8063f1c,c3ec4400,c4519400,c380e000)
 > > at execve_loadvm+0x1da
 > > execve1(ca73daa0,8063f1c,8063f3c,8063f44,c055c480,6300,dc107d1c,0,c06ba833,cd722744)
 > > at execve1+0x32
 > > sys_execve(ca73daa0,dc107cf4,dc107d1c,c08096f0,0,cdb17e3c,c0c8b800,dc107d30,c06babb9,cd722730)
 > > at sys_execve+0x30
 > > syscall(dc107d48,b3,ab,bfbf001f,806001f,8063f1c,8063f3c,bfbfec48,8063f1c,7d7b7cff)
 > > at syscall+0x95
 > 
 > this looks like the execargs pool leak that greg oster has a patch
 > for. (reproduced below.)
 > 
 > 
 > .mrg.
 > 
 > Index: kern_exec.c
 > ===================================================================
 > RCS file: /cvsroot/src/sys/kern/kern_exec.c,v
 > retrieving revision 1.349
 > diff -u -p -r1.349 kern_exec.c
 > --- kern_exec.c	9 Apr 2012 19:42:06 -0000	1.349
 > +++ kern_exec.c	13 Apr 2012 20:28:14 -0000
 > @@ -1991,6 +1991,8 @@ spawn_return(void *arg)
 >  		rw_exit(&exec_lock);
 >  	}
 >  
 > +	execve_free_data(&spawn_data->sed_exec);
 > +
 >  	/* release our refcount on the data */
 >  	spawn_exec_data_release(spawn_data);
 >  

 Actually, info from Martin leads me to believe that the
 execve_free_data() needs to go into the: 

  if (have_reflock) {

  }

 bit just above here... I've run tests with that, and they all pass
 too...

 Later...

 Greg Oster

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/45718 (processes sometimes get stuck and spin in vm_map)
Date: Sun, 15 Apr 2012 16:49:03 +0200

 On Sat, Apr 14, 2012 at 07:15:06PM +0000, matthew green wrote:
 >  this looks like the execargs pool leak that greg oster has a patch for.

 You should see a log message "should not happen" if this would be the case.

 However, if we go through exec_pool_alloc, we can't have hit the hard limit,
 can we?

 Martin

From: Taylor R Campbell <campbell+netbsd@mumble.net>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org, netbsd-bugs@netbsd.org,
	gnats-admin@netbsd.org, rmind@NetBSD.org
Subject: Re: kern/45718 (processes sometimes get stuck and spin in vm_map)
Date: Wed, 9 May 2012 00:28:25 +0000

 Caught another one, different stack trace this time.  Forgot to run
 `show uvm', sorry.

 crash> bt/t 0t24257
 trace: pid 24257 lid 1 at 0xfbd0d85c
 sleepq_block(64,0,c0c5fd9d,c0ce63d0,0,0,0,c08e40d3,0,0) at sleepq_block+0xda
 cv_timedwait(c0d291d0,c0d291cc,64,fbd0d928,0,ffffffff,ffffffff,0,801727,c9a83200) at cv_timedwait+0x126
 uvm_map_prepare(c0d291c0,c0000000,3000,0,ffffffff,ffffffff,0,801727,fbd0d968,10) at uvm_map_prepare+0x167
 uvm_map(c0d291c0,fbd0d9f0,3000,0,ffffffff,ffffffff,0,801727,800001,0) at uvm_map+0x78
 uvm_km_alloc(c0d291c0,3000,0,800001,c30ba6c0,1,fbd0da8c,c07e117a,c30ba6c0,1) at uvm_km_alloc+0xe6
 uarea_poolpage_alloc(c30ba6c0,1,fbd0da4c,c09548fd,0,0,0,0,0,0) at uarea_poolpage_alloc+0x41
 pool_grow(c30ba734,c07e1c4e,6,c0d7436c,6,0,c07e1b13,c30ba7bc,cce4fd20,c30ba738) at pool_grow+0x2a
 pool_get(c30ba6c0,1,fbd0db2c,c056697e,0,1,28b8d000,5,21,0) at pool_get+0x79
 pool_cache_get_slow(0,1,0,c08de98e,0,0,1,3,fbd0dc7c,0) at pool_cache_get_slow+0x19a
 pool_cache_get_paddr(c30ba6c0,1,0,0,fbd0dc40,cce4fd20,fbd0dc6c,c05619e5,0,1) at pool_cache_get_paddr+0x21c
 uvm_uarea_alloc(0,1,c6fde044,e9,0,0,ffffffff,3f8,cce4fd20,c4436a40) at uvm_uarea_alloc+0x23
 fork1(cce4fd20,3,14,0,0,0,0,fbd0dd1c,0,106) at fork1+0x115
 sys___vfork14(cce4fd20,fbd0dcf4,fbd0dd1c,c08096f0,7d49f000,c0c8b800,cce4fd20,c838a350,0,8054095) at sys___vfork14+0x50
 syscall(fbd0dd48,b3,bbbb00ab,bfbf001f,806001f,0,0,bfbfe678,bb9102fc,bb9102fc) at syscall+0x95

From: Taylor R Campbell <campbell+netbsd@mumble.net>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org,
	netbsd-bugs@netbsd.org, gnats-admin@netbsd.org, rmind@NetBSD.org
Subject: Re: kern/45718 (processes sometimes get stuck and spin in vm_map)
Date: Sat, 23 Jun 2012 15:27:06 +0000

 I caught some more processes wedged in vm_map today.  I wasn't able to
 get a stack trace because my attempt to chroot into a recent enough
 userland to use crash(8) also wedged in vm_map.

 But then after five minutes, all the processes that were wedged in
 vm_map spontaneously unwedged and continued to make progress!

 When I next looked, there was nothing wedged in vm_map.

From: "Jonathan A. Kollasch" <jakllsch@kollasch.net>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/45718: processes sometimes get stuck and spin in vm_map
Date: Sat, 1 Sep 2012 14:15:44 -0500

 I've been seeing the hangs of the exec_pool_alloc() variety in 6.0RC1 on
 i386.  Using  dd if=/dev/zero of=/dev/null count=1 bs=1G  (machine has
 about 2G of RAM, sometimes needs to be more than 1G though) tends to
 unwedge everything that gets stuck in vm_map wait channel here.

From: Chuck Silvers <chuq@chuq.com>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/45718: processes sometimes get stuck and spin in vm_map
Date: Sat, 1 Sep 2012 17:48:23 -0700

 On Sat, Sep 01, 2012 at 07:20:05PM +0000, Jonathan A. Kollasch wrote:
 >  I've been seeing the hangs of the exec_pool_alloc() variety in 6.0RC1 on
 >  i386.  Using  dd if=/dev/zero of=/dev/null count=1 bs=1G  (machine has
 >  about 2G of RAM, sometimes needs to be more than 1G though) tends to
 >  unwedge everything that gets stuck in vm_map wait channel here.

 this sounds like KVA exhaustion.  this was a problem on i386 some years ago
 and I don't remember whether it was ever really fixed.  I vaguely recall
 there was a particular PR that was the main means of tracking that issue
 but I can't find it now.

 -Chuck

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/45718: processes sometimes get stuck and spin in vm_map
Date: Sun, 2 Sep 2012 03:45:14 +0000

 On Sun, Sep 02, 2012 at 12:50:04AM +0000, Chuck Silvers wrote:
  >  On Sat, Sep 01, 2012 at 07:20:05PM +0000, Jonathan A. Kollasch wrote:
  >  >  I've been seeing the hangs of the exec_pool_alloc() variety in 6.0RC1 on
  >  >  i386.  Using  dd if=/dev/zero of=/dev/null count=1 bs=1G  (machine has
  >  >  about 2G of RAM, sometimes needs to be more than 1G though) tends to
  >  >  unwedge everything that gets stuck in vm_map wait channel here.
  >  
  >  this sounds like KVA exhaustion.  this was a problem on i386 some years ago
  >  and I don't remember whether it was ever really fixed.  I vaguely recall
  >  there was a particular PR that was the main means of tracking that issue
  >  but I can't find it now.

 Do you remember anything else about that PR? I can't find it from
 obvious keywords. Is it likely to have been closed?

 -- 
 David A. Holland
 dholland@netbsd.org

From: Chuck Silvers <chuq@chuq.com>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/45718: processes sometimes get stuck and spin in vm_map
Date: Tue, 11 Sep 2012 07:07:40 -0700

 On Sun, Sep 02, 2012 at 03:50:05AM +0000, David Holland wrote:
 >  Do you remember anything else about that PR? I can't find it from
 >  obvious keywords. Is it likely to have been closed?

 I think I found the one I was thinking of: 33185

 -Chuck

From: Taylor R Campbell <campbell+netbsd@mumble.net>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/45718: processes sometimes get stuck and spin in vm_map
Date: Sun, 9 Dec 2012 06:13:43 +0000

 Every once in a while I still find processes stuck like this.  Tonight
 I investigated closer with the help of crash(8).  Here are some clues
 I've gathered:

 - The processes that are stuck are all stuck in the cv_timedwait in
 uvm_map_prepare deep inside exec.

 - I can still exec new processes, so the problem is *not* simply kva
 exhaustion causing the uvm_km_alloc in exec_pool_alloc to hang
 indefinitely waiting for kva -- all the requests for kva are for the
 same size, so if I can make new requests, the old ones should be
 serviceable too.

 - Examination of the lwp structures and their l_timeout_ch members
 reveals that the callouts for cv_timedwait are firing (the callouts'
 c_time values keep changing), so it's not that the sleepq mechanism is
 stuck or anything.

 - Examintion of kernel_map itself reveals that UVM_MAP_WANTVA is
 persistently flagged.

 So it looks like something freed up kva, but failed to signal to the
 waiters that the kva was freed up.

 I looked around for a race condition surrounding map->flags, but
 although there is a wacky locking dance surrounding vm_maps, I didn't
 see anything obvious there: uvm_unmap_remove looks like it does the
 right thing to signal to the waiters.  However, I suspect that
 uvm_map_replace and uvm_map_extract can free up space in the map, and
 neither of them signals UVM_MAP_WANTVA waiters.

 There may be other places in uvm_map.c that free up space -- it's huge
 and I haven't gone through it all.  Is it plausible that at least
 uvm_map_replace and uvm_map_extract, and perhaps other parts of
 uvm_map.c as well, need to signal UVM_MAP_WANTVA waiters?


 Caveat: I am currently using a kernel from back in March, because
 something broke related to drm in more recent kernels, but uvm hasn't
 changed much lately, so I suspect the problem is still here.

From: Taylor R Campbell <campbell+netbsd@mumble.net>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/45718: processes sometimes get stuck and spin in vm_map
Date: Sun, 9 Dec 2012 07:25:25 +0000

    Date: Sun, 9 Dec 2012 06:13:43 +0000
    From: Taylor R Campbell <campbell+netbsd@mumble.net>

    - Examintion of kernel_map itself reveals that UVM_MAP_WANTVA is
    persistently flagged.

    So it looks like something freed up kva, but failed to signal to the
    waiters that the kva was freed up.

 Manually clearing UVM_MAP_WANTVA in kernel_map->flags by writing to
 /dev/kmem failed to unwedge the processes that were wedged in vm_map,
 so there goes that hypothesis.

    - I can still exec new processes, so the problem is *not* simply kva
    exhaustion causing the uvm_km_alloc in exec_pool_alloc to hang
    indefinitely waiting for kva -- all the requests for kva are for the
    same size, so if I can make new requests, the old ones should be
    serviceable too.

 This analysis was a little too simple-minded.  It may be that most new
 execs are serviced by exec_pool without going through exec_pool_alloc,
 but every one that does go through exec_pool_alloc gets wedged in.

From: Taylor R Campbell <campbell+netbsd@mumble.net>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/45718: processes sometimes get stuck and spin in vm_map
Date: Sun, 9 Dec 2012 08:12:19 +0000

    Date: Sun, 9 Dec 2012 07:25:25 +0000
    From: Taylor R Campbell <campbell+netbsd@mumble.net>

       Date: Sun, 9 Dec 2012 06:13:43 +0000
       From: Taylor R Campbell <campbell+netbsd@mumble.net>

       - I can still exec new processes, so the problem is *not* simply kva
       exhaustion causing the uvm_km_alloc in exec_pool_alloc to hang
       indefinitely waiting for kva -- all the requests for kva are for the
       same size, so if I can make new requests, the old ones should be
       serviceable too.

    This analysis was a little too simple-minded.  It may be that most new
    execs are serviced by exec_pool without going through exec_pool_alloc,
    but every one that does go through exec_pool_alloc gets wedged in [vm_ma=
 p].

 It also seems that although there is a hard limit on the number of
 objects currently pooled in exec_pool, namely maxexec (16 on my
 system), there is no limit on the number of pending calls to the
 pool's allocator.  This strikes me as suboptimal...

From: Taylor R Campbell <campbell+netbsd@mumble.net>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/45718: processes sometimes get stuck and spin in vm_map
Date: Sun, 9 Dec 2012 08:43:58 +0000

 pmap(8) on the kernel_map (`pmap -V 0x87654321' with the address of
 kernel_map as shown by crash(8)) shows that kva is fragmented enough
 that there is indeed no empty space of at least #x40000 bytes, and
 that every request to exec_pool requiring a new kva allocation is
 guaranteed to fail.

From: David Laight <david@l8s.co.uk>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/45718: processes sometimes get stuck and spin in vm_map
Date: Sun, 9 Dec 2012 18:22:22 +0000

 On Sun, Dec 09, 2012 at 08:45:03AM +0000, Taylor R Campbell wrote:
 > The following reply was made to PR kern/45718; it has been noted by GNATS.
 > 
 > From: Taylor R Campbell <campbell+netbsd@mumble.net>
 > To: gnats-bugs@NetBSD.org
 > Cc: 
 > Subject: Re: kern/45718: processes sometimes get stuck and spin in vm_map
 > Date: Sun, 9 Dec 2012 08:43:58 +0000
 > 
 >  pmap(8) on the kernel_map (`pmap -V 0x87654321' with the address of
 >  kernel_map as shown by crash(8)) shows that kva is fragmented enough
 >  that there is indeed no empty space of at least #x40000 bytes, and
 >  that every request to exec_pool requiring a new kva allocation is
 >  guaranteed to fail.

 Allocating 256k blocks has to be sub-optimal!
 Especially for something that is going to be chopped into pieces
 (which is what I suspect is happening here).

 Personally I've no idea why we have this large proliferation of
 memory 'pools' - rather than just allocating items as-needed from
 general free lists.

 There might be some mileage in keeping stats for some uses - and
 predetermining the correct list for fixed size items - but not
 big private memory free lists.

 	David

 -- 
 David Laight: david@l8s.co.uk

From: Jeff Rizzo <riz@NetBSD.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/45718: processes sometimes get stuck and spin in vm_map
Date: Sun, 27 Jan 2013 13:58:23 -0800

 I just hit what appears to be this problem in 6.0_STABLE with 6.0.1 (or 
 maybe 6.0_STABLE, I forget exactly) userland on NetBSD/evbarm (a 
 Sheevaplug):


 db> show uvm
 Current UVM status:
    pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
 , ncolors=1  127411 VM pages: 87390 active, 0 inactive, 1345 wired, 
 23658 free
    pages  8564 anon, 75753 file, 4418 exec
    freemin=256, free-target=341, wired-max=42470
    cpu0:
      faults=1285187, traps=11298455, intrs=2416539, ctxswitch=2281911
      softint=989950, syscalls=4468244
    fault counts:
      noram=1, noanon=0, pgwait=0, pgrele=0
      ok relocks(total)=54213(54213), anget(retrys)=70879(0), amapcopy=125623
      neighbor anon/obj pg=48318/1379218, gets(lock/unlock)=449220/54213
      cases: anon=47679, anoncow=23189, obj=334661, prcopy=114559, 
 przero=241394
    daemon and swap counts:
      woke=450, revs=450, scans=520574, obscans=499526, anscans=0
      busy=0, freed=499526, reactivate=1194, deactivate=561810
      pageouts=0, pending=0, nswget=0
      nswapdev=1, swpgavail=1874999
      swpages=1874999, swpginuse=0, swpgonly=0, paging=0
 db> ps /l
 PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 1758     1 3   0         0           c2f4ad20               cron vm_map
 1181     1 3   0         0           c2f36020               cron wait
 7580     1 3   0         0           c2f362c0               cron vm_map
 1627     1 3   0         0           c2f36560               cron wait
 1434     1 3   0         0           c2f36800               cron vm_map
 13657    1 3   0         0           c2f36aa0               cron wait
 13848    1 3   0         0           c2f36d40               cron vm_map
 1175     1 3   0         0           c2f16000               cron wait
 1238     1 3   0         0           c1be4540               cron vm_map
 13333    1 3   0         0           c1ce2aa0               cron vm_map
 12564    1 3   0         0           c1bb0560               cron wait
 1747     1 3   0         0           c1bb0aa0               cron wait
 1426     1 3   0         0           c2f162a0               cron vm_map
 1233     1 3   0         0           c2f16540               cron wait
 976      1 3   0         0           c2f167e0             dhcpcd vm_map
 1103     1 3   0         0           c2f16a80               cron vm_map
 13646    1 3   0         0           c2f16d20               cron wait
 13965    1 3   0         0           c2ef7020               cron vm_map
 5836     1 3   0         0           c2ef72c0               cron wait
 6667     1 3   0         0           c2ef7560               cron vm_map
 14218    1 3   0         0           c2ef7800               cron wait
 137      1 3   0         0           c2ef7aa0               cron vm_map
 328      1 3   0         0           c2ef7d40               cron wait
 135      1 3   0         0           c2eda000               cron vm_map
 326      1 3   0         0           c2eda2a0               cron vm_map
 12549    1 3   0         0           c2eda540               cron wait
 12996    1 3   0         0           c2eda7e0               cron wait
 1859     1 3   0         0           c2edaa80               cron vm_map
 11458    1 3   0         0           c2edad20               cron wait
 193      1 3   0         0           c2ebf020               cron vm_map
 192      1 3   0         0           c2ebf2c0               cron wait
 13363    1 3   0         0           c2ebf560               cron vm_map
 13233    1 3   0         0           c2ebf800               cron wait
 13371    1 3   0         0           c1ce2020             master vm_map
 13269    1 3   0         0           c2b4aaa0             master vm_map
 13784    1 3   0         0           c2ebfaa0               cron vm_map
 14142    1 3   0         0           c2ebfd40               cron wait
 13114    1 3   0         0           c2e8e000               cron vm_map
 14243    1 3   0         0           c2e8e2a0               cron wait
 13454    1 3   0         0           c2e8e540               cron vm_map
 12647    1 3   0         0           c2e8e7e0               cron vm_map
 13810    1 3   0         0           c2e8ea80               cron wait
 13423    1 3   0         0           c2e8ed20               cron wait
 11504    1 3   0         0           c1d65020               cron vm_map
 11266    1 3   0         0           c1d652c0               cron wait
 13559    1 3   0         0           c1d65560               cron vm_map
 13901    1 3   0         0           c1d65800               cron wait
 13309    1 3   0         0           c1d65aa0               cron vm_map
 12485    1 3   0         0           c1d65d40               cron wait
 12774    1 3   0         0           c2df7000               cron vm_map
 14154    1 3   0         0           c2df72a0               cron wait
 13183    1 3   0         0           c2df7540               cron vm_map
 11177    1 3   0         0           c2df77e0               cron wait
 10922    1 3   0         0           c2dc4020               cron vm_map
 12500    1 3   0         0           c2df7a80               cron vm_map
 11816    1 3   0         0           c2dc4aa0               cron wait
 12065    1 3   0         0           c2dc4800               cron wait
 12804    1 3   0         0           c2df7d20               make vm_map
 7516     1 3   0         0           c2dc42c0               make wait
 8630     1 3   0        80           c2dc4560                 sh wait
 6603     1 3   0        80           c2dc4d40               make wait
 5644     1 3   0        80           c2d89000                 sh wait
 96       1 3   0        80           c2d892a0                 sh wait
 1667     1 3   0        80           c2d89540        pbulk-build wait
 1566     1 3   0        80           c2d897e0                 sh wait
 1272     1 3   0        80           c2d89d20               sshd select
 136      1 3   0        80           c2d89a80                ssh select
 134      1 3   0        80           c2b4a020        pbulk-build select
 1555     1 3   0        80           c2b4a560                 sh wait
 98       1 3   0        80           c197e540                 sh wait
 1205     1 3   0        80           c197e000                 sh wait
 71       1 3   0        80           c1bb0800                zsh pause
 73       1 3   0        80           c1bb02c0               tmux kqueue
 1435     1 3   0   1000000           c1bb0d40              getty vm_map
 1262     1 3   0        80           c2b4a800               cron nanoslp
 784      1 3   0        80           c197e2a0               qmgr kqueue
 1117     1 3   0        80           c2b4a2c0              inetd kqueue
 1453     1 3   0        80           c2b4ad40              mdnsd select
 1316     1 3   0        80           c197e7e0             master kqueue
 1111     1 3   0        80           c1ce22c0               sshd select
 1041     1 3   0        80           c1be42a0               ntpd pause
 933      1 3   0        80           c197ea80               qmgr kqueue
 1046     1 3   0        80           c197ed20             master kqueue
 884      1 3   0        80           c1be4d20               sshd select
 364      1 3   0        80           c1bb0020            syslogd kqueue
 188      1 3   0         0           c1be4000             dhcpcd wait
 1        1 3   0        80           c1af82c0               init wait
 0       45 3   0       200           c1ce2560              nfsio nfsiod
 0       44 3   0       200           c1ce2800              nfsio nfsiod
 0       43 3   0       200           c1be4a80              nfsio nfsiod
 0       42 3   0       200           c1ce2d40              nfsio nfsiod
 0       41 3   0       200           c1be47e0            physiod physiod
 0       40 3   0       200           c1af9000           aiodoned aiodoned
 0       39 3   0       200           c1af9540            ioflush syncer
 0       38 3   0       200           c1af92a0           pgdaemon pgdaemon
 0       35 3   0       200           c1af97e0           swdmover swdmvr
 0       34 3   0       200           c1af8d40          cryptoret crypto_w
 0       33 3   0       200           c1a04000           scsibus0 sccomp
 0       32 3   0       200           c1af8aa0         usbtask-dr usbtsk
 0       31 3   0       200           c1af8800         usbtask-hc usbtsk
 0       30 3   0       200           c1af8560               usb0 usbevt
 0       29 3   0       200           c1af8020              unpgc unpgc
 0       28 3   0       200           c1af9d20        vmem_rehash vmem_rehash
 0       27 3   0       200           c1af9a80             sdmmc0 mmctaskq
 0       18 3   0       200           c1a042a0            atabus1 atath
 0       17 3   0       200           c1a04540            atabus0 atath
 0       16 3   0       200           c1a047e0               iic0 iicintr
 0       15 3   0       200           c1a04a80         pmfsuspend pmfsuspend
 0       14 3   0       200           c1a04d20           pmfevent pmfevent
 0       13 3   0       200           c19fd020         sopendfree sopendfr
 0       12 3   0       200           c19fd2c0           nfssilly nfssilly
 0       11 3   0       200           c19fd560            cachegc cachegc
 0       10 3   0       200           c19fd800              vrele vrele
 0        9 3   0       200           c19fdaa0             vdrain vdrain
 0        8 3   0       200           c19fdd40          modunload mod_unld
 0        7 3   0       200           c19f3000            xcall/0 xcall
 0        6 1   0       200           c19f32a0          softser/0
 0        5 1   0       200           c19f3540          softclk/0
 0        4 1   0       200           c19f37e0          softbio/0
 0        3 1   0       200           c19f3a80          softnet/0
 0    >   2 7   0       201           c19f3d20             idle/0
 0        1 3   0       200           c04cc960            swapper uvm
 db> t/a c2f4ad20
 trace: pid 1758 lid 1 at 0xcb691b58
 netbsd:mi_switch+0x10
          scp=0xc014787c rlv=0xc0143eb0 (netbsd:sleepq_block+0x88)
          rsp=0xcb691b5c rfp=0xcb691b80
          r10=0xc198e09c r9=0xc198e558
          r8=0x00000064 r7=0xc2f4ad20 r6=0xc04e9bd4 r5=0x00000000
          r4=0xc04ee2b4
 netbsd:sleepq_block+0x10
          scp=0xc0143e38 rlv=0xc011ae20 (netbsd:cv_timedwait+0xf0)
          rsp=0xcb691b84 rfp=0xcb691bb0
          r8=0x00000064 r7=0xc0367498
          r6=0xc04e9bd4 r5=0xc2f4ad20 r4=0xc04e9bd8
 netbsd:cv_timedwait+0x10
          scp=0xc011ad40 rlv=0xc02ddb40 (netbsd:uvm_map_prepare+0x178)
          rsp=0xcb691bb4 rfp=0xcb691c08
          r10=0x00801727 r9=0x00040000
          r8=0x00069138 r7=0xc04e9bd8 r6=0xc03e318c r5=0xc04e9bd4
          r4=0xc04e9bc4
 netbsd:uvm_map_prepare+0x10
          scp=0xc02dd9d8 rlv=0xc02ddda0 (netbsd:uvm_map+0x88)
          rsp=0xcb691c0c rfp=0xcb691c68
          r10=0x00000000 r9=0xcb691c24
          r8=0x00801727 r7=0xffffffff r6=0xcb691c98 r5=0x00040000
          r4=0xc04e9bc4
 netbsd:uvm_map+0x10
          scp=0xc02ddd28 rlv=0xc02d9394 (netbsd:uvm_km_alloc+0xbc)
          rsp=0xcb691c6c rfp=0xcb691cc4
          r10=0xc036b58c r9=0xc04dbc7c
          r8=0xc04e11c0 r7=0xffffffff r6=0xffffffff r5=0x00000000
          r4=0x00040000
 netbsd:uvm_km_alloc+0x10
          scp=0xc02d92e8 rlv=0xc0230040 (netbsd:pool_grow+0x38)
          rsp=0xcb691cc8 rfp=0xcb691cfc
          r10=0xc04dbc78 r9=0xc04dbc7c
          r8=0xc040934c r7=0xc04dbc78 r6=0x00000001 r5=0xc04dbc04
          r4=0xc04dbc04
 netbsd:pool_grow+0x10
          scp=0xc0230018 rlv=0xc022f994 (netbsd:pool_get+0x80)
          rsp=0xcb691d00 rfp=0xcb691d2c
          r10=0xc04dbc78 r9=0xc04dbc7c
          r8=0x00000001 r7=0x00000001 r6=0x00000000 r5=0x00000000
          r4=0xc04dbc04
 netbsd:pool_get+0x10
          scp=0xc022f924 rlv=0xc0123cd4 (netbsd:execve_loadvm+0x2d4)
          rsp=0xcb691d30 rfp=0xcb691d88
          r10=0xc04ee2b4 r9=0xcb691d5c
          r8=0xc2f38550 r7=0xc2f4ad20 r6=0x00000000 r5=0xc2f4ad20
          r4=0xcb691d94
 netbsd:execve_loadvm+0x10
          scp=0xc0123a10 rlv=0xc0126594 (netbsd:execve1+0x28)
          rsp=0xcb691d8c rfp=0xcb691ee0
          r10=0xc0407b14 r9=0x00000003
          r8=0xc2f38550 r7=0x0000003b r6=0xc2f4ad20 r5=0xc2f4ad20
          r4=0xcb691d94
 netbsd:execve1+0x10
          scp=0xc012657c rlv=0xc01265e4 (netbsd:sys_execve+0x2c)
          rsp=0xcb691ee4 rfp=0xcb691ef4
          r5=0xc2f4ad20 r4=0xcb691fb4
 netbsd:sys_execve+0x10
          scp=0xc01265c8 rlv=0xc02497a0 (netbsd:syscall+0x84)
          rsp=0xcb691ef8 rfp=0xcb691f84
 netbsd:syscall+0x10
          scp=0xc024972c rlv=0xc02499d0 (netbsd:swi_handler+0xb4)
          rsp=0xcb691f88 rfp=0xcb691fb0
          r10=0x00000000 r9=0x5105bad0
          r8=0x00000000 r7=0x00000001 r6=0xc03e322c r5=0xc2f4ad20
          r4=0xcb691fb4
 netbsd:swi_handler+0x10
          scp=0xc024992c rlv=0xc005f12c (netbsd:swi_entry+0x2c)
          rsp=0xcb691fb4 rfp=0xbfffe860
          r8=0xbfffede0 r7=0x00018ecc
          r6=0x2020c0a6 r5=0x00001230 r4=0xc03e322c
 db>

State-Changed-From-To: feedback->analyzed
State-Changed-By: riastradh@NetBSD.org
State-Changed-When: Sat, 20 Feb 2016 15:34:14 +0000
State-Changed-Why:
feedback provided and problem analyzed


From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/45718 CVS commit: src/sys/kern
Date: Fri, 20 Oct 2017 14:48:43 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Fri Oct 20 14:48:43 UTC 2017

 Modified Files:
 	src/sys/kern: kern_exec.c

 Log Message:
 Carve out KVA for execargs on boot from an exec_map like we used to.

 Candidate fix for PR kern/45718: `processes sometimes get stuck and
 spin in vm_map', a problem that has been plaguing all our 32-bit
 ports for years.

 Since we currently use large (256k) buffers for execargs, and since
 nobody has stepped up to tackle breaking them into bite-sized (or at
 least page-sized) chunks, after KVA gets sufficiently fragmented we
 can't allocate new execargs buffers from kernel_map.

 Until 2008, we always carved out KVA for execargs on boot with a uvm
 submap exec_map of kernel_map.  Then ad@ found that the uvm_km_free
 call, to discard them when done, cost about 100us, which a pool
 avoided:

 https://mail-index.NetBSD.org/tech-kern/2008/06/25/msg001854.html
 https://mail-index.NetBSD.org/tech-kern/2008/06/26/msg001859.html

 ad@ _simultaneously_ introduced a pool _and_ eliminated the reserved
 KVA in the exec_map submap.  This change preserves the pool, but
 restores exec_map (with less code, by putting it in MI code instead
 of copying it in every MD initialization routine).

 Patch proposed on tech-kern:
 https://mail-index.NetBSD.org/tech-kern/2017/10/19/msg022461.html

 Patch tested by bouyer@:
 https://mail-index.NetBSD.org/tech-kern/2017/10/20/msg022465.html

 I previously discussed the issue on tech-kern before I knew of the
 history around exec_map:
 https://mail-index.NetBSD.org/tech-kern/2012/12/09/msg014695.html

 The candidate workaround I proposed of using pool_setlowat to force
 preallocation of KVA would also force preallocation of physical RAM,
 which is a waste not incurred by using exec_map, and which is part of
 why I never committed it.

 There may remain a general problem that if thread A calls pool_get
 and tries to service that request by a uvm_km_alloc call that hangs
 because KVA is scarce, and thread B does pool_put, the pool_put in
 thread B will not notify the pool_get in thread A that it doesn't
 need to wait for KVA, and so thread A may continue to hang in
 uvm_km_alloc.  However,

 (a) That won't apply here, because there is exactly as much KVA
 available in exec_map as exec_pool will ever try to use.

 (b) It is possible that may not even matter in other cases as long as
 the page daemon eventually tries to shrink the pool, which will cause
 a uvm_km_free that can unhang the hung uvm_km_alloc.

 XXX pullup-8
 XXX pullup-7
 XXX pullup-6
 XXX pullup-5, perhaps...


 To generate a diff of this commit:
 cvs rdiff -u -r1.447 -r1.448 src/sys/kern/kern_exec.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/45718 CVS commit: src/sys/kern
Date: Sat, 28 Oct 2017 17:06:43 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Sat Oct 28 17:06:43 UTC 2017

 Modified Files:
 	src/sys/kern: subr_pool.c

 Log Message:
 Allow only one pending call to a pool's backing allocator at a time.

 Candidate fix for problems with hanging after kva fragmentation related
 to PR kern/45718.

 Proposed on tech-kern:

 https://mail-index.NetBSD.org/tech-kern/2017/10/23/msg022472.html

 Tested by bouyer@ on i386.

 This makes one small change to the semantics of pool_prime and
 pool_setlowat: they may fail with EWOULDBLOCK instead of ENOMEM, if
 there is a pending call to the backing allocator in another thread but
 we are not actually out of memory.  That is unlikely because nearly
 always these are used during initialization, when the pool is not in
 use.

 XXX pullup-8
 XXX pullup-7
 XXX pullup-6 (requires tweaking the patch)
 XXX pullup-5...


 To generate a diff of this commit:
 cvs rdiff -u -r1.208 -r1.209 src/sys/kern/subr_pool.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/45718 CVS commit: [netbsd-8] src/sys
Date: Tue, 27 Feb 2018 09:07:33 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Tue Feb 27 09:07:33 UTC 2018

 Modified Files:
 	src/sys/arch/alpha/alpha [netbsd-8]: pmap.c
 	src/sys/arch/m68k/m68k [netbsd-8]: pmap_motorola.c
 	src/sys/arch/powerpc/oea [netbsd-8]: pmap.c
 	src/sys/arch/sparc64/sparc64 [netbsd-8]: pmap.c
 	src/sys/arch/x86/x86 [netbsd-8]: pmap.c
 	src/sys/dev/dtv [netbsd-8]: dtv_scatter.c
 	src/sys/dev/marvell [netbsd-8]: mvxpsec.c
 	src/sys/kern [netbsd-8]: subr_extent.c subr_pool.c uipc_mbuf.c
 	src/sys/opencrypto [netbsd-8]: crypto.c
 	src/sys/sys [netbsd-8]: mbuf.h pool.h
 	src/sys/ufs/chfs [netbsd-8]: chfs_malloc.c
 	src/sys/uvm [netbsd-8]: uvm_fault.c

 Log Message:
 Pull up following revision(s) (requested by mrg in ticket #593):
 	sys/dev/marvell/mvxpsec.c: revision 1.2
 	sys/arch/m68k/m68k/pmap_motorola.c: revision 1.70
 	sys/opencrypto/crypto.c: revision 1.102
 	sys/arch/sparc64/sparc64/pmap.c: revision 1.308
 	sys/ufs/chfs/chfs_malloc.c: revision 1.5
 	sys/arch/powerpc/oea/pmap.c: revision 1.95
 	sys/sys/pool.h: revision 1.80,1.82
 	sys/kern/subr_pool.c: revision 1.209-1.216,1.219-1.220
 	sys/arch/alpha/alpha/pmap.c: revision 1.262
 	sys/kern/uipc_mbuf.c: revision 1.173
 	sys/uvm/uvm_fault.c: revision 1.202
 	sys/sys/mbuf.h: revision 1.172
 	sys/kern/subr_extent.c: revision 1.86
 	sys/arch/x86/x86/pmap.c: revision 1.266 (via patch)
 	sys/dev/dtv/dtv_scatter.c: revision 1.4

 Allow only one pending call to a pool's backing allocator at a time.
 Candidate fix for problems with hanging after kva fragmentation related
 to PR kern/45718.

 Proposed on tech-kern:
 https://mail-index.NetBSD.org/tech-kern/2017/10/23/msg022472.html
 Tested by bouyer@ on i386.

 This makes one small change to the semantics of pool_prime and
 pool_setlowat: they may fail with EWOULDBLOCK instead of ENOMEM, if
 there is a pending call to the backing allocator in another thread but
 we are not actually out of memory.  That is unlikely because nearly
 always these are used during initialization, when the pool is not in
 use.

 Define the new flag too for previous commit.

 pool_grow can now fail even when sleeping is ok. Catch this case in pool_get
 and retry.

 Assert that pool_get failure happens only with PR_NOWAIT.
 This would have caught the mistake I made last week leading to null
 pointer dereferences all over the place, a mistake which I evidently
 poorly scheduled alongside maxv's change to the panic message on x86
 for null pointer dereferences.

 Since pr_lock is now used to wait for two things now (PR_GROWING and
 PR_WANTED) we need to loop for the condition we wanted.
 make the KASSERTMSG/panic strings consistent as '%s: [%s], __func__, wchan'
 Handle the ERESTART case from pool_grow()

 don't pass 0 to the pool flags
 Guess pool_cache_get(pc, 0) means PR_WAITOK here.
 Earlier on in the same context we use kmem_alloc(sz, KM_SLEEP).

 use PR_WAITOK everywhere.
 use PR_NOWAIT.

 Don't use 0 for PR_NOWAIT

 use PR_NOWAIT instead of 0

 panic ex nihilo -- PR_NOWAITing for zerot

 Add assertions that either PR_WAITOK or PR_NOWAIT are set.
 - fix an assert; we can reach there if we are nowait or limitfail.
 - when priming the pool and failing with ERESTART, don't decrement the number
   of pages; this avoids the issue of returning an ERESTART when we get to 0,
   and is more correct.
 - simplify the pool_grow code, and don't wakeup things if we ENOMEM.

 In pmap_enter_ma(), only try to allocate pves if we might need them,
 and even if that fails, only fail the operation if we later discover
 that we really do need them.  This implements the requirement that
 pmap_enter(PMAP_CANFAIL) must not fail when replacing an existing
 mapping with the first mapping of a new page, which is an unintended
 consequence of the changes from the rmind-uvmplock branch in 2011.

 The problem arises when pmap_enter(PMAP_CANFAIL) is used to replace an existing
 pmap mapping with a mapping of a different page (eg. to resolve a copy-on-write).
 If that fails and leaves the old pmap entry in place, then UVM won't hold
 the right locks when it eventually retries.  This entanglement of the UVM and
 pmap locking was done in rmind-uvmplock in order to improve performance,
 but it also means that the UVM state and pmap state need to be kept in sync
 more than they did before.  It would be possible to handle this in the UVM code
 instead of in the pmap code, but these pmap changes improve the handling of
 low memory situations in general, and handling this in UVM would be clunky,
 so this seemed like the better way to go.

 This somewhat indirectly fixes PR 52706, as well as the failing assertion
 about "uvm_page_locked_p(old_pg)".  (but only on x86, various other platforms
 will need their own changes to handle this issue.)
 In uvm_fault_upper_enter(), if pmap_enter(PMAP_CANFAIL) fails, assert that
 the pmap did not leave around a now-stale pmap mapping for an old page.
 If such a pmap mapping still existed after we unlocked the vm_map,
 the UVM code would not know later that it would need to lock the
 lower layer object while calling the pmap to remove or replace that
 stale pmap mapping.  See PR 52706 for further details.
 hopefully workaround the irregularly "fork fails in init" problem.
 if a pool is growing, and the grower is PR_NOWAIT, mark this.
 if another caller wants to grow the pool and is also PR_NOWAIT,
 busy-wait for the original caller, which should either succeed
 or hard-fail fairly quickly.

 implement the busy-wait by unlocking and relocking this pools
 mutex and returning ERESTART.  other methods (such as having
 the caller do this) were significantly more code and this hack
 is fairly localised.
 ok chs@ riastradh@

 Don't release the lock in the PR_NOWAIT allocation. Move flags setting
 after the acquiring the mutex. (from Tobias Nygren)
 apply the change from arch/x86/x86/pmap.c rev. 1.266 commitid vZRjvmxG7YTHLOfA:

 In pmap_enter_ma(), only try to allocate pves if we might need them,
 and even if that fails, only fail the operation if we later discover
 that we really do need them.  If we are replacing an existing mapping,
 reuse the pv structure where possible.

 This implements the requirement that pmap_enter(PMAP_CANFAIL) must not fail
 when replacing an existing mapping with the first mapping of a new page,
 which is an unintended consequence of the changes from the rmind-uvmplock
 branch in 2011.

 The problem arises when pmap_enter(PMAP_CANFAIL) is used to replace an existing
 pmap mapping with a mapping of a different page (eg. to resolve a copy-on-write).
 If that fails and leaves the old pmap entry in place, then UVM won't hold
 the right locks when it eventually retries.  This entanglement of the UVM and
 pmap locking was done in rmind-uvmplock in order to improve performance,
 but it also means that the UVM state and pmap state need to be kept in sync
 more than they did before.  It would be possible to handle this in the UVM code
 instead of in the pmap code, but these pmap changes improve the handling of
 low memory situations in general, and handling this in UVM would be clunky,
 so this seemed like the better way to go.

 This somewhat indirectly fixes PR 52706 on the remaining platforms where
 this problem existed.


 To generate a diff of this commit:
 cvs rdiff -u -r1.261 -r1.261.8.1 src/sys/arch/alpha/alpha/pmap.c
 cvs rdiff -u -r1.69 -r1.69.8.1 src/sys/arch/m68k/m68k/pmap_motorola.c
 cvs rdiff -u -r1.94 -r1.94.8.1 src/sys/arch/powerpc/oea/pmap.c
 cvs rdiff -u -r1.307 -r1.307.6.1 src/sys/arch/sparc64/sparc64/pmap.c
 cvs rdiff -u -r1.245.6.1 -r1.245.6.2 src/sys/arch/x86/x86/pmap.c
 cvs rdiff -u -r1.3 -r1.3.2.1 src/sys/dev/dtv/dtv_scatter.c
 cvs rdiff -u -r1.1 -r1.1.12.1 src/sys/dev/marvell/mvxpsec.c
 cvs rdiff -u -r1.80 -r1.80.8.1 src/sys/kern/subr_extent.c
 cvs rdiff -u -r1.207 -r1.207.6.1 src/sys/kern/subr_pool.c
 cvs rdiff -u -r1.172 -r1.172.6.1 src/sys/kern/uipc_mbuf.c
 cvs rdiff -u -r1.78.2.4 -r1.78.2.5 src/sys/opencrypto/crypto.c
 cvs rdiff -u -r1.170.2.1 -r1.170.2.2 src/sys/sys/mbuf.h
 cvs rdiff -u -r1.79 -r1.79.10.1 src/sys/sys/pool.h
 cvs rdiff -u -r1.4 -r1.4.30.1 src/sys/ufs/chfs/chfs_malloc.c
 cvs rdiff -u -r1.199.6.2 -r1.199.6.3 src/sys/uvm/uvm_fault.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: analyzed->feedback
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Tue, 10 Apr 2018 08:44:40 +0000
State-Changed-Why:
Is this fixed?


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.43 2018/01/16 07:36:43 maya Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2017 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.