NetBSD Problem Report #2755

Received: (qmail-queue invoked from smtpd); 14 Sep 1996 23:06:30 -0000
Message-Id: <199609142305.QAA18091@Cup.DSG.Stanford.EDU>
Date: Sat, 14 Sep 1996 16:05:15 -0700 (PDT)
From: Jonathan Stone <jonathan@DSG.Stanford.EDU>
Reply-To: jonathan@DSG.Stanford.EDU
To: gnats-bugs@gnats.netbsd.org
Subject: Touching virtual memory==physical memory causes pathological
              VM behaviour
X-Send-Pr-Version: 3.95

>Number:         2755
>Category:       kern
>Synopsis:       Exhausting memory causes a machine to freeze for up to a minute
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    kern-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Sep 15 21:50:03 +0000 1996
>Closed-Date:    Sun Jun 08 01:55:59 +0000 1997
>Last-Modified:  Wed Aug 07 22:23:26 +0000 2002
>Originator:     Jonathan Stone
>Release:        1.2_BETA
>Organization:

>Environment:

System: NetBSD Cup.DSG.Stanford.EDU 1.2_BETA NetBSD 1.2_BETA (DSG) #47: Mon Jul 29 12:05:57 PDT 1996 jonathan@Cup.DSG.Stanford.EDU:/aga/n1/src/NetBSD/IP-PLUS/src/sys/arch/i386/compile/DSG i386


>Description:

NetBSD exhibits pathological behaviour somewhere in the VM system,
under at least some circumstances when the active memory (desired
working-set size) is greater than physical memory.  I have replicated
the behaviour several times on a 64Mbyte machine.  I describe the
bebaviour as "pathological" because (as observed from either local X
applications or remote logins from another machine) *all* processes on
the afflicted machine are blocked for 60 to 65 seconds.

During this time, X applications on the thrashing machine do not
repaint when iconified and de-iconified; the kernel answers TCP
connects to listening ports, but inetd can't seem to fork new daemons
(e.g., telnet gets a "connect" message but no login prompt, and a
telnet interrupt or "are you there" are not answered.  system monitoring
tools also seem to get stopped.

Listening to the swap disk suggests that not much disk I/O activity
is taking place during the one-minute hiatus.


This is

>How-To-Repeat:

Compile and run the following program on a NetBSD/i386 machine, with
arguments "-v N", where N is the amount of physical memory, in megabytes,
installed in the machine.   I find that on a 64Mbyte machine,  the
pathological behaviour starts approximately when touching the 50th Mbyte.
Re-running the program on a previously-idle system forces idle processes
to get swapped out, and increases the amount of memory that can be touched
before the pathological regime is entered.



>Fix:

I dont have a clue.


>Release-Note:
>Audit-Trail:

From: Jonathan Stone <jonathan@DSG.Stanford.EDU>
To: gnats-bugs@gnats.netbsd.org
Cc: developers@netbsd.org
Subject: Re: kern/2755: Touching virtual memory==physical memory causes pathological
Date: Tue, 17 Sep 1996 18:34:04 -0700

 Executive SummarY:
    A  fix for the VM "freeze" bug in  NetBSD.
    Please consider for  inclusion in  1.2.


 The following patch originates from John Dyson. 
 It fixes a serious bug in the 1.2 (and earlier) VM system,
 which is by now well-known.

 The bug is exhibited when free pages get exhausted.  At that point,
 a NetBSD (or OpenBSD, or 4.4-lite) system will "freeze".  Any
 process that needs to allocate new pages, or that pages, is blocked.
 The freeze persists for several seconds to over a minute.

 Summary:

 the pageout daemon is sleeping on lbolt, That means when pages are
 exhausted, the pageout daemon does nothing execpt wake up once a
 second.  That would be bad enough, except due to the design of the
 pageout daemon, it never manages to free up pages and let user
 processes make progress until it's cleaned *all* the dirty pages oin
 the inactive list --- and all that at the slow rate of progress
 implied by running once per second.

 The fix is therefore to make the pageout daemon sleep on
 vm_pages_wanted, rather than lbolt.  Deeper analysis follows.


 When the pager daemon wakes up, it' coded to do nothing except the
 minimal amount of "page cleaning" required to meet the VM goals
 encoded in cnt.   This means the pageout daemon wakes up and
 transfers at least "cnt.v_free_min pages from the VM system's
 "active" list to the "inactive" list.   Pages on the inactive list
 are then candidates for cleaning.   Here, cleaning means initiating
 asynchronous writeback,  or what the code refers to as "laundry".


 I have diagnosed what is going on here by running "systat vmstat"
 in such a way that it never pagefaults, and using that to obseve
 a system during the freeze.   If  "systat -w 1 vmstat"
 is started *before* the system enters the freeze, and started on
 a process that's in memory (e.g., an rlogind, instead of an xterm)
 then the *current* systat display continues during the "frozen" state.

 Experimenting with the memory-touching hog in pr 2755 shows clearly
 that the pageout daemon's cleaning activity _does_ not_ appear to
 result in any pages being freed until *all* the pags on the inactive
 list are freed.   Listening to the disk (I have a good ear) shows
 clearly that it's only writing a small burst once per second.
 Watching  the scan rate shows that the pageout daemon is waking
 up and scanning the  inactive list.  Watchign the free-page list
 shows that the system stays in a state with *no* free pages
 (blocking fork() and any pages) until the short, 1Hz scans of
 the inactive list have gone on for long enough to write back
 the entire inactive list.  At that point, hundreds of pages become
 available.  (I've run the test on both 32M and 64M machines.)




 The patch below changes a sleep on lbolt to a sleep on
 vm_pages_needed. This ensures that the pageout daemon runs each
 time vm_pages_needed is signalled.  Grepping on src/sys/*/*.c
 shows that this variable is signalled only by the swap pager,
 when vm/swap_pager.c:swap_pages_iodone() completes a page-writeback
 IO, or when a vm_object_allocote() allocates a new VM object
 but notices a shortfall of either free pages, or pages on the
 inactive list.

 Analysis of patch:

 I've spent several hours since Sunday running the patch in VM overload
 conditions.  It  does definitely avoid the scenario where
 there are zero free pages (none at all!) which is what causes
 the freeze behaviour.   It does so at some cost in system time under
 page shortfalls.   The patch  doesn't directly address the root cause
 of the problem, namely that the VM system isn't freeing pages fast
 enough to meet either the offered workload, *or* the write bandwidth
 of the swap device. 

 Without the patch, in non-frozen state, I'ce observed sustained page
 writes of 1.5Mbytes/sec on a P/120 with BusLogic 596 controller and a
 7200rpm fast/wide drive.  With the patch applied, the system does
 *not* freeze, but only sustains page writeback (and thus pagefaults)
 at 25% to 50% of the unpatched state.

 I've also spent some hours watching "systat -w {1,5} vmstat"
 with the patch applied, under overloda conditions.  I'm seeing low
 pageout rates, as described  above.  I'm also seeing that the
 sustained free-page list size (as reported by systat vmstat)
 is low. I've seen the freelist size stay at 10-12 pages for tens
 of seconds.  I know paging activity is taking place during this time,
 because both systat vmstat shows it, and the VM pig in pr 2755 is
 making progress.   My experience with other VM systems (V, other
 research kernels at Stanford, other BSD variants) says that' s
 not a large enough free pool of pages to maintain system activity.
 I think the VM system still needs more work even with the patch applied.


 I beleive this may be due, in part, to lockstep behavour between two
 activities:

  a) writing back a page in vm_swap.c (where swap_pager_iodone signals
     vm_pages_needed)
  b) page cleaning in the pageout daemon's scan loop.

 John Dyson suggests that with the patch applied, with a VM writeback
 system that does no page clustering (like ours) and with an IO
 subsystem that does *not* do asynch writeback, one is likely to see a
 maximum rate of about 60 transfers/sec to a 3600rpm drive.
 The hardware  I have manages from 80 to 200 transfers, or 320
 to 800 Kbytes/sec, substantially less than  in the unpatched
 case; but overall throughput is significantly higher.
 Though John has experience with tweaking a similar VM system to ours,
 before replacing it with a new one in FreeBSD, I'd prefer to see
 hard measurements.


 To emphasise a point: This patch *only*  makes a difference under
 VM overload conditions.  I don't think  it should have a signficant
 effect at all under the normal operating regime for mot NetBSD
 systems, as they should have enough free pages, and inactive pages,
 that the pageout daemon is doing essentially nothing.



 Recommended action:

 1. Apply this patch as soon as feasible, both to the trunk and to
    1.2, either pre-release or as an Official patch as soon as practical
    (at the Release Engineer's discretion).
    (If I had my say, I'd just ship kernels built with the patch.)

    Having the pageout daemon sleep for 1 second when pages are exhausted
    is clearly wrong. (I don't think that needs further defense).
    The fix below is semantically correct, in that it's the most
    appropriate thing to do if pages are short.  It's also the best
    that can be  done without some nontrivial changes to the VM
    system design.

 2. In *addition to* applying this patch,  change the VM system.
    (note: not completely thought out!)

     I think the root cause of this problem is that the VM system
     is not being aggresive enough, under page shortage conditions,
     at actually returning  pages to the freelist. This is demonstrated
     by running for tens of seconds of paging activity with a sustained
     mean free list of 12 pages, as described  above.

     The VM system already enforces a per-scan minimum on the number
     of pages moved from the "active" to the "inactive" list. (only the
     latter are candidates for pageout.  A more complete fix is to
     ensure that, under heavy paging conditions, some minimal amout of
     pages are actully *freed*, per scan.  This may imply forcing
     non-asynchronous writeback when pages are low; I haven't checked
     the code.



 Index: vm_pageout.c
 ===================================================================
 RCS file: /cvsroot/src/sys/vm/vm_pageout.c,v
 retrieving revision 1.23
 diff -c -r1.23 vm_pageout.c
 *** vm_pageout.c	1996/02/05 01:54:07	1.23
 --- vm_pageout.c	1996/09/18 00:14:45
 ***************
 *** 70,75 ****
 --- 70,76 ----

   #include <sys/param.h>
   #include <sys/proc.h>
 + #include <sys/kernel.h>

   #include <vm/vm.h>
   #include <vm/vm_page.h>
 ***************
 *** 326,332 ****
   		 * shortage, so we put pause for awhile and try again.
   		 * XXX could get stuck here.
   		 */
 ! 		(void) tsleep((caddr_t)&lbolt, PZERO|PCATCH, "pageout", 0);
   		break;
   	}
   	case VM_PAGER_FAIL:
 --- 327,333 ----
   		 * shortage, so we put pause for awhile and try again.
   		 * XXX could get stuck here.
   		 */
 ! 		(void) tsleep((caddr_t)&vm_pages_needed, PZERO|PCATCH, "pageout", hz);
   		break;
   	}
   	case VM_PAGER_FAIL:
 ***************
 *** 453,459 ****
   	if (postatus == VM_PAGER_AGAIN) {
   		extern int lbolt;

 ! 		(void) tsleep((caddr_t)&lbolt, PZERO|PCATCH, "pageout", 0);
   		goto again;
   	} else if (postatus == VM_PAGER_BAD)
   		panic("vm_pageout_cluster: VM_PAGER_BAD");
 --- 454,461 ----
   	if (postatus == VM_PAGER_AGAIN) {
   		extern int lbolt;

 ! 		(void) tsleep((caddr_t)&vm_pages_needed, PZERO|PCATCH, "pageout", hz);
 ! 
   		goto again;
   	} else if (postatus == VM_PAGER_BAD)
   		panic("vm_pageout_cluster: VM_PAGER_BAD");

From: Jonathan Stone <jonathan@netbsd1.cygnus.com>
To: Kernel Bug People <gnats-admin@NetBSD.ORG>
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: kern/2755: Exhausting memory causes a machine to freeze for up to a minute
Date: 8 Jun 1997 01:58:39 -0000

 	`Jonathan Stone' made changes to this PR.

 --- /tmp/gnats029031	Sat Jun  7 18:55:53 1997
 +++ /tmp/gnatsa29031	Sat Jun  7 18:56:00 1997
 @@ -18,7 +18,7 @@
  >Severity:       critical
  >Priority:       high
  >Responsible:    kern-bug-people
 ->State:          open
 +>State:          closed
  >Class:          sw-bug
  >Submitter-Id:   net
  >Arrival-Date:   Sun Sep 15 21:50:03 1996

 Fixed by the patch John Dyson suggested after discussions with me;
 applied by mrg as rev. 1.24 of sys/vm/vm_pageout.c.
>Unformatted:
 >Sharfile of Example Program:


 # This is a shell archive.  Save it in a file, remove anything before
 # this line, and then unpack it by entering "sh file".  Note, it may
 # create directories; files and directories will be owned by you and
 # have default permissions.
 #
 # This archive contains:
 #
 #	memtest.c
 #
 echo x - memtest.c
 sed 's/^X//' >memtest.c << 'END-of-memtest.c'
 X/*
 X * Write test pattern into N megabytes of virtual memory.
 X * May also induce thrashing or demonstrate VM performance problems
 X * when the size tested is at or near physical memory size.
 X */
 X
 X
 X#include <stdio.h>
 X#include <stdlib.h>
 X#include <unistd.h>
 X
 X#define ONE_MBYTE (1024* 1024)
 X
 X/*
 X *  Flags
 X */
 Xint verboseflag = 0;
 Xint foreverflag = 0;
 X
 X
 X/*
 X * Fill a hunk of memory with a test pattern, then do a read-after-write
 X * comparison.
 X */
 Xint
 Xmemtest (int *mem, int size, int pattern)
 X{
 X	register int i;
 X	
 X	for (i = 0; i < size; i++) {
 X		mem[i] = pattern ;
 X	}
 X
 X	for (i = 0; i < size; i++) {
 X		if (mem[i] != pattern) {
 X			printf("failed at %p, should be 0x%x, is 0x%x\n",
 X			       mem+i, pattern, mem[i]);
 X			((volatile int*)mem)[i] = pattern ;
 X			if (mem[i] != pattern) 
 X			printf("failed again at %p, should be 0x%x, is 0x%x\n",
 X			       mem+i, pattern, mem[i]);
 X		}
 X	}
 X}
 X
 Xvoid
 Xusage()
 X{
 X	extern char *__progname;
 X	printf("usage: %s [-v] [-f] <megabytes>\n", __progname);
 X	exit(2);
 X}
 X
 X
 Xint
 Xparseopts(int argc, char *argv[])
 X{
 X	extern int optind;
 X	int ch;
 X
 X	while ((ch = getopt(argc, argv, "vf")) != -1) {
 X		switch (ch) {
 X		case 'v':
 X			verboseflag = 1;
 X			break;
 X		case 'f':
 X			foreverflag = 1;
 X			break;
 X		default:
 X			usage();
 X		}
 X	}
 X	
 X	argc -= optind;
 X	argv += optind;
 X
 X	if (argc != 1) usage();
 X
 X	return (atoi (argv[0]));
 X}
 X
 X
 X
 Xint
 Xmain(int argc, char *argv[])
 X{
 X	void  *p;
 X	int i, megs;
 X
 X	megs = parseopts(argc, argv);
 X	p = (void*)malloc(ONE_MBYTE * megs);
 X
 Xtest:
 X	for (i = 0; i < megs; i++) {
 X		int *thishunk = p+(ONE_MBYTE*i);
 X		if (verboseflag)
 X			printf("testing: %d Mbytes\n", i);
 X		memtest(thishunk, (ONE_MBYTE / sizeof(int)), 0);
 X		memtest(thishunk, (ONE_MBYTE / sizeof(int)), 0xffffffff);
 X		memtest(thishunk, (ONE_MBYTE / sizeof(int)), 0xa5a5a5a5);
 X		memtest(thishunk, (ONE_MBYTE / sizeof(int)), 0x5a5a5a5a);
 X		memtest(thishunk, (ONE_MBYTE / sizeof(int)), 0);
 X	}
 X	if (foreverflag) goto test;
 X}
 END-of-memtest.c
 exit

Home	PR Database Search