NetBSD Problem Report #41766

From Wolfgang.Stukenbrock@nagler-company.com  Wed Jul 22 12:44:47 2009
Return-Path: <Wolfgang.Stukenbrock@nagler-company.com>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by www.NetBSD.org (Postfix) with ESMTP id AAD9A63B879
	for <gnats-bugs@gnats.NetBSD.org>; Wed, 22 Jul 2009 12:44:47 +0000 (UTC)
Message-Id: <20090722124445.DDB774EA9FE@s012.nagler-company.com>
Date: Wed, 22 Jul 2009 14:44:45 +0200 (CEST)
From: Wolfgang.Stukenbrock@nagler-company.com
Reply-To: Wolfgang.Stukenbrock@nagler-company.com
To: gnats-bugs@gnats.NetBSD.org
Subject: system will hang in reboot or shutdown if tmpfs contains more data than phys mem available
X-Send-Pr-Version: 3.95

>Number:         41766
>Category:       bin
>Synopsis:       system will hang in reboot or shutdown if tmpfs contains more data than phys mem available
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    bin-bug-people
>State:          closed
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Wed Jul 22 12:45:00 +0000 2009
>Closed-Date:    Fri Mar 31 06:48:42 +0000 2023
>Last-Modified:  Fri Mar 31 06:48:42 +0000 2023
>Originator:     Wolfgang Stukenbrock
>Release:        NetBSD 4.0
>Organization:
Dr. Nagler & Company GmbH
>Environment:


System: NetBSD s012 4.0 NetBSD 4.0 (NSW-S012) #9: Fri Mar 13 12:31:52 CET 2009 wgstuken@s012:/usr/src/sys/arch/amd64/compile/NSW-S012 amd64
Architecture: x86_64
Machine: amd64
>Description:
	During reboot or shutdown the last step is to execute /etc/rc.d/swap1 in case
	"swapoff=YES" is set in rc.conf. Setting this is required if the swap space resides
	on a raid device to avoid rebuilding the parity on every boot.
	If there are some tmpfs filesystems, the data of theese resides in the swap space and
	if there is a large amount of data present, unmounting of the swap space will fail.
	The system will hang.
>How-To-Repeat:
	Setup a large swap space, add a tmpfs, fill e.g. twice the amount of data of the phys memory
	available in the system into the tmpfs.
	Set swapoff=YES in rc.conf and try to reboot the system. You will get into problems.
>Fix:
	The fix below simply removes all tmpfs filesystems prior unmounting the swap space.
	The fix is added to swap1 that is executed as last script according to the rcorder output
	created in rc.shutdown.
	remark: If someone is using network swap space it may be already too late to do it here,
		but umounting in the swap2 script would happen too early and some problems with
		still running processes may pop up.
	It is clear that this fix will not solve all possible problems with "out of swap space" during
	shutdown. E.g. if there is more memroy allocated to processes than phys memory is available,
	this fix will help nothing. but in this case the kernel itself tries to work around the problem
	by killing some processes.

	The following patch will add the removeal of all tmpfs filesystems in etc/rc.d/swap1
	in front of the swapspace removal.

--- swap1       2009/07/22 12:32:13     1.1
+++ swap1       2009/07/22 12:32:24
@@ -29,6 +29,16 @@
 {
 	if checkyesno swapoff || [ -n "$rc_force" ]; then
 		echo "Removing block-type swap devices"
+#
+#	Check if there is tmpfs present.
+#	The problem is that we my hang in swapctl -U if there
+#	is not enougth physical memory to hold the contents.
+#
+#	Remark: We assume that there are no other mount points inside of tmpfs
+#		and this will not solve the problem for more process memory
+#		than physical memory.
+#
+		umount -aft tmpfs
 		swapctl -U -t blk
 	fi
 }

>Release-Note:

>Audit-Trail:
From: Patrick Welche <prlw1@cam.ac.uk>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: bin/41766
Date: Thu, 5 Mar 2015 12:45:10 +0000

 This bug still exists in NetBSD-7.99.5/amd64 of 2015-Mar-01:

 # sysctl hw.physmem
 hw.physmem = 2136272896
 # swapctl -lh
 Device      Size     Used    Avail Capacity  Priority
 /dev/wd0b   2.0G       0B     2.0G     0%    0
 # df -h /tmp
 Filesystem         Size       Used      Avail %Cap Mounted on
 tmpfs              3.5G       8.0K       3.5G   0% /tmp
 # dd if=/dev/zero of=big.fs count=2600272896
 dd: big.fs: No space left on device
 7508857+0 records in
 7508856+0 records out
 3844534272 bytes transferred in 92.338 secs (41635450 bytes/sec)
 # df -h .
 Filesystem         Size       Used      Avail %Cap Mounted on
 tmpfs              3.6G       3.6G         0B 100% /tmp
 # shutdown -r now
 ...
 UVM:pid 1.1 (init), uid 0 killed: out of swap
 UVM:pid 1.1 (init), uid 0 killed: out of swap
 ...


 The patch included in the original report to add "umount -aft tmpfs"
 to rc.d/swap1 before swapctl -U seems to make sense.

From: "Patrick Welche" <prlw1@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/41766 CVS commit: src/etc/rc.d
Date: Mon, 20 Apr 2015 18:01:46 +0000

 Module Name:	src
 Committed By:	prlw1
 Date:		Mon Apr 20 18:01:46 UTC 2015

 Modified Files:
 	src/etc/rc.d: swap1

 Log Message:
 Apply modified version of patch from Wolfgang Stukenbrock in PR bin/41766

 This forcibly unmounts tmpfs file systems, so that swapctl doesn't run
 out of swap itself, eventually trying to kill pid 1.

 No objections since:
   http://mail-index.netbsd.org/current-users/2015/03/16/msg026908.html


 To generate a diff of this commit:
 cvs rdiff -u -r1.11 -r1.12 src/etc/rc.d/swap1

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->feedback
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Tue, 08 Sep 2015 04:03:48 +0000
State-Changed-Why:
Committed this past April. Do you need the change pulled up to any of
the stable branches? (It is likely too late to get it into the 7.0
release)


From: Wolfgang Stukenbrock <Wolfgang.Stukenbrock@nagler-company.com>
To: <gnats-bugs@NetBSD.org>
Cc: <dholland@NetBSD.org>, <gnats-admin@netbsd.org>, <netbsd-bugs@netbsd.org>,
        <Wolfgang.Stukenbrock@nagler-company.com>
Subject: Re: bin/41766 (system will hang in reboot or shutdown if tmpfs
 contains more data than phys mem available)
Date: Fri, 11 Sep 2015 11:13:01 +0200

 Hi,

 I don't need it in a specific branch. (we have aded a user-level-workaround to all of our systems)
 I suggest it will be in 7.1 then.

 best regards.

 W. Stukenbrock

 be
 OK - it is not 100% saveOn Tue, 8 Sep 2015 04:03:48 +0000
 <dholland@NetBSD.org> wrote:

 > Synopsis: system will hang in reboot or shutdown if tmpfs contains more data than phys mem available
 > 
 > State-Changed-From-To: open->feedback
 > State-Changed-By: dholland@NetBSD.org
 > State-Changed-When: Tue, 08 Sep 2015 04:03:48 +0000
 > State-Changed-Why:
 > Committed this past April. Do you need the change pulled up to any of
 > the stable branches? (It is likely too late to get it into the 7.0
 > release)
 > 
 > 
 > 

State-Changed-From-To: feedback->open
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Sun, 11 Oct 2015 01:00:34 +0000
State-Changed-Why:
feedback received; let's do pullup-7.


Responsible-Changed-From-To: bin-bug-people->dholland
Responsible-Changed-By: pgoyette@NetBSD.org
Responsible-Changed-When: Wed, 01 Jun 2016 07:02:50 +0000
Responsible-Changed-Why:
dholland suggested that pullups to NetBSD 7 were appropriate.


State-Changed-From-To: open->pending-pullups
State-Changed-By: pgoyette@NetBSD.org
State-Changed-When: Wed, 01 Jun 2016 07:02:50 +0000
State-Changed-Why:

A fix is committed, need pullups to NetBSD-7


State-Changed-From-To: pending-pullups->open
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Thu, 02 Jun 2016 02:44:20 +0000
State-Changed-Why:
pending-pullups is for when pullups have been filed and not processed.


Responsible-Changed-From-To: dholland->bin-bug-people
Responsible-Changed-By: dholland@NetBSD.org
Responsible-Changed-When: Mon, 20 Jun 2016 17:22:19 +0000
Responsible-Changed-Why:
don't hide it by assigning it to me, I'll see it anyway


State-Changed-From-To: open->needs-pullups
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Mon, 20 Jun 2016 17:22:19 +0000
State-Changed-Why:
we have needs-pullups now


State-Changed-From-To: needs-pullups->closed
State-Changed-By: riastradh@NetBSD.org
State-Changed-When: Fri, 31 Mar 2023 06:48:42 +0000
State-Changed-Why:
netbsd-7 is EOL, pullups no longer needed


>Unformatted:

Home	PR Database Search