NetBSD Problem Report #56329

From paul@whooppee.com  Sun Jul 25 03:59:47 2021
Return-Path: <paul@whooppee.com>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id B0B3E1A921F
	for <gnats-bugs@gnats.NetBSD.org>; Sun, 25 Jul 2021 03:59:47 +0000 (UTC)
Message-Id: <20210725035941.F086D30F2C4@speedy.whooppee.com>
Date: Sat, 24 Jul 2021 20:59:41 -0700 (PDT)
From: paul@whooppee.com
Reply-To: paul@whooppee.com
To: gnats-bugs@NetBSD.org
Subject: nvme(4) takes long time to umount
X-Send-Pr-Version: 3.95

>Number:         56329
>Category:       kern
>Synopsis:       nvme(4) takes long time to umount
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Jul 25 04:00:00 +0000 2021
>Last-Modified:  Tue Nov 28 14:55:01 +0000 2023
>Originator:     Paul Goyette
>Release:        NetBSD 9.99.87
>Organization:
+--------------------+--------------------------+----------------------+
| Paul Goyette       | PGP Key fingerprint:     | E-mail addresses:    |
| (Retired)          | FA29 0E3B 35AF E8AE 6651 | paul@whooppee.com    |
| Software Developer | 0786 F758 55DE 53BA 7731 | pgoyette@netbsd.org  |
|                    |                          | pgoyette99@gmail.com |
+--------------------+--------------------------+----------------------+
>Environment:


System: NetBSD speedy.whooppee.com 9.99.87 NetBSD 9.99.87 (SPEEDY 2021-07-23 13:58:03 UTC) #0: Sat Jul 24 00:01:27 UTC 2021 paul@speedy.whooppee.com:/build/netbsd-local/obj/amd64/sys/arch/amd64/compile/SPEEDY amd64
Architecture: x86_64
Machine: amd64
>Description:
	Using nvme(4) device as input AND output device for running
	standard system builds, so there is lots of file creation and
	re-creation and deletion.  Device details are

	<lspci>
	03:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961

	<dmesg>
	[     8.118210] nvme0 at pci3 dev 0 function 0: Samsung Electronics (3rd vendor ID) product a804 (rev. 0x00)

	Depending on just how much activity has occurred, it can take
	up to 15 seconds (or even longer) for the device/file-system
	to umount.  During this time, neither xosview nor systat seem
	to observe any disk activity (presumably, no commands are being
	issued), yet the device does not respond.  It "feels like" the
	device is dequeuing a large number of deferred operations which
	are all being processed by the nvme controller without any host
	intervention.

	Here is a time-stamped console log of a system shutdown showing
	the lengthy delay for umount (in this case, ~15 seconds):

	...
	[ 900738.108348] unmounting 0xffff9afce05b4000 /kern (kernfs)...
	[ 900738.108348] unmounted kernfs on /kern type kernfs
	[ 900738.108348] unmounting 0xffff9afce0d30000 /build (/dev/ld0e)...
	[ 900753.244179] unmounted /dev/ld0e on /build type ffs
	[ 900753.244179] unmounting 0xffff9afce052f000 /home (/dev/wd0f)... 
	[ 900753.624326] unmounted /dev/wd0f on /home type ffs
	...

	Given that there seems to be some queue of deferred operations,
	it is not surprising that, after a system crash, a forced
	``fsck -fp'' identifies [tens of] thousands of unreferenced
	files.  So far, fsck has been successful at restoring the file
	system without any data loss.

	This misbehavior occurs whether or not the file system used
	``-o log'' (wapbl).  And it occurs without using ``-o discard''.

	Not sure how to debug this further.  However, dh@ suggests
	that this behavior is definitely suboptimal, and that it
	should be a blocker for the NetBSD-10 release.  (At least, we
	ought to know enough to describe the actual bug, and maybe
	provide a work-around.

>How-To-Repeat:
	See above

>Fix:
	Yes, please


>Audit-Trail:
From: Paul Goyette <paul@whooppee.com>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/56329: nvme(4) takes long time to umount
Date: Fri, 6 Aug 2021 06:25:49 -0700 (PDT)

 Additional information...

 The particular nvme SSD device is mounted on /build and there are
 several null-fs mounts on top of it.  (These are so I can build.sh
 using read-only sources.)  There doesn't seem to be any issue with
 the null-fs mounts.


 # mount
 /dev/wd0a on / type ffs (log, local)
 /dev/wd0e on /var type ffs (log, local)
 /dev/wd0f on /home type ffs (log, local)
 /dev/ld0e on /build type ffs (log, noatime, local)
 kernfs on /kern type kernfs (local)
 ptyfs on /dev/pts type ptyfs (local)
 procfs on /proc type procfs (local)
 tmpfs on /tmp type tmpfs (local)
 tmpfs on /var/shm type tmpfs (local)
 /build/netbsd-local/src on /build/netbsd-local/src_ro type null (read-only, local)
 /build/netbsd-local/xsrc on /build/netbsd-local/xsrc_ro type null (read-only, local)
 /build/netbsd-current/src on /build/netbsd-current/src_ro type null (read-only, local)


From: "J. Hannken-Illjes" <hannken@eis.cs.tu-bs.de>
To: NetBSD GNATS <gnats-bugs@netbsd.org>
Cc: 
Subject: Re: kern/56329: nvme(4) takes long time to umount
Date: Fri, 6 Aug 2021 15:39:19 +0200

 --Apple-Mail=_856441C4-36E4-4AA9-845E-2E7F68D06998
 Content-Transfer-Encoding: 7bit
 Content-Type: text/plain;
 	charset=us-ascii

 Given the null mounts it is possible you removed nodes
 below the null mount and these nodes become open-but-removed
 because the null layer keeps a (last) reference.

 Does the unmount speed up if you unmount the null mounts
 first and sync?

 --
 J. Hannken-Illjes - hannken@eis.cs.tu-bs.de - TU Braunschweig

 --Apple-Mail=_856441C4-36E4-4AA9-845E-2E7F68D06998
 Content-Transfer-Encoding: 7bit
 Content-Disposition: attachment;
 	filename=signature.asc
 Content-Type: application/pgp-signature;
 	name=signature.asc
 Content-Description: Message signed with OpenPGP

 -----BEGIN PGP SIGNATURE-----

 iQEzBAEBCAAdFiEE2BL3ha7Xao4WUZVYKoaVJdNr+uEFAmENO4cACgkQKoaVJdNr
 +uHzIAf/X4w9RGQy7/JWaxhzf0IjEtp/QfdyUvqi3hMql5hp1L60koezGyV4+qW0
 6ike23cenrRjAKvEMVFBfiqRwEH+xy7FXC21MPaMkxa+Xy15npl4SKguCteDhsKE
 txKS6FpIrtnq3i/rqDMgrQtqL791TkbSLxRWzcAsgiNLyD875Dl5Q/tenWZh9AVP
 YouhSZ62bVCSg6PkiEFuv+v5vNvicVmolOu9zi2liL7JUDw71gMxLH6QrEF90JZT
 yX1cBaeGBSpBdi9UYB9pTYWm/Vm1L15539bJqbUIE9lncv9qOUHyuJuhS4y+OXKs
 dQLYWZ2Yb6Pso0LEj6a3YvWR2C4v0A==
 =e1RJ
 -----END PGP SIGNATURE-----

 --Apple-Mail=_856441C4-36E4-4AA9-845E-2E7F68D06998--

From: Paul Goyette <paul@whooppee.com>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/56329: nvme(4) takes long time to umount
Date: Fri, 6 Aug 2021 06:54:56 -0700 (PDT)

 On Fri, 6 Aug 2021, J. Hannken-Illjes wrote:

 > Given the null mounts it is possible you removed nodes
 > below the null mount and these nodes become open-but-removed
 > because the null layer keeps a (last) reference.
 >
 > Does the unmount speed up if you unmount the null mounts
 > first and sync?

 If I try to umount(8) the SSD while the null mounts are still in
 use, I would expect to see EBUSY when trying to umount(8) the SSD
 since the mount-point itself is still referenced.

 I hvae tried to umount(8) the null-fs first, and I have tried to
 sync(8) first, and I have tried doing both before attempting to
 umount(8) the SSD.  It does not matter, it still takes "a long
 time" for umount(8) to complete.


 +--------------------+--------------------------+----------------------+
 | Paul Goyette       | PGP Key fingerprint:     | E-mail addresses:    |
 | (Retired)          | FA29 0E3B 35AF E8AE 6651 | paul@whooppee.com    |
 | Software Developer | 0786 F758 55DE 53BA 7731 | pgoyette@netbsd.org  |
 |                    |                          | pgoyette99@gmail.com |
 +--------------------+--------------------------+----------------------+

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/56329: nvme(4) takes long time to umount
Date: Tue, 29 Mar 2022 17:54:31 +0200

 I am not sure this is the same problem (but rumours said for a long time
 that this all is not nvme(4) specific) - but I found a very nice reproducer
 for something that smells the same or at least related issue:

  - find a usb stick with a write LED
  - format it from scratch
  - create a msdos FS on it and mount it (I used newfs_msdos -F 32 /dev/rsd0)
  - copy (say) 400 MB of data to the fresh new filesystem
  - umount the filesystem

 Now you can watch the usb stick blink in ~1 HZ interval for like 10 minutes.

 Martin

From: "J. Hannken-Illjes" <hannken@mailbox.org>
To: NetBSD GNATS <gnats-bugs@netbsd.org>
Cc: 
Subject: Re: kern/56329: nvme(4) takes long time to umount
Date: Wed, 20 Apr 2022 16:41:50 +0200

 --Apple-Mail=_17CD9E4F-B7AE-4F43-A659-DBD0CC42533F
 Content-Transfer-Encoding: quoted-printable
 Content-Type: text/plain;
 	charset=us-ascii


 Finally got a laptop with
     nvme0: SAMSUNG MZVLB256HAHQ-000L7, firmware 0L2QEXD7, serial =
 S41GNB0K593889
 and see unmount times of 10 .. 12 seconds with a DIAGNOSTIC + DEBUG + =
 LOCKDEBUG
 kernel.  All this time is spent in the first vflush().

 On a GENERIC kernel without DIAGNOSTIC etc the unmount time goes down to =
 less
 than a second and the disk becomes much faster.

 What kind of kernel are/were you using?

 --
 J. Hannken-Illjes - hannken@mailbox.org

 --Apple-Mail=_17CD9E4F-B7AE-4F43-A659-DBD0CC42533F
 Content-Transfer-Encoding: 7bit
 Content-Disposition: attachment;
 	filename=signature.asc
 Content-Type: application/pgp-signature;
 	name=signature.asc
 Content-Description: Message signed with OpenPGP

 -----BEGIN PGP SIGNATURE-----

 iQIzBAEBCAAdFiEEyLVMkhxs8fxixv+2IOocBq6p/bMFAmJgG64ACgkQIOocBq6p
 /bO+DA//eNF5fJSbBOrLnw8F1MeeSGQGuU/CTdpn9a++Ti1ZEiDI7vhmPGtmjqPK
 jSOM7FEg9G2zmF2pBgoH2kOd1F3RrImjHSROq75TR+JUrVcqpnOZ/gEGzVGKxVHC
 T3phgwkywHai+eAGlhVLUY85ah2KNp4K5/xuRlLcuDW1yCAdlmV8MVXwnWtRieBb
 XQGcL0UEt1INWvcUq+Pjqf4TcTFOgih0yMS/UUEy/xAEGsfee/z88Hw0ZYwO3kcg
 dOhm/CviQryc7l7+t5SlTrwnDen9vWGBs1kWjreEGf0feBYHLJ/5GMHreaMin14S
 +zprDRMQAy5g7nZaytNEnOM6DS/SfpviUpkzbYkOQ7n5fxEnSGFklN4VXEoGhKQ8
 0QnM5SB6sCUBnjE4gGnCGovO2W5OBmJ0Xs7ZJbYE6A590hToi3PPVWxGaVcnGDD3
 7kol8mfvR0bmgPpGdm8UHHZadVg0EJ9eUXy2xglEPj6FGdklUjtiHbOGMu62/Tpk
 Zar7zUeelEiheF7m+1ojIPIWReAgN/90ZyW5/DKJaJeatbvrodjhExlnjwqHUlaN
 ZEA1KKzncntwHDUfbXSD19HahZq5WLteV8gHY8gtpscz7RQIwdLkRlp1cfXQMlbW
 j2444G0wJcNl2maErcSfmW1EiQQwdgKTJE6miRk7U/WDzp7b1Jw=
 =PkKw
 -----END PGP SIGNATURE-----

 --Apple-Mail=_17CD9E4F-B7AE-4F43-A659-DBD0CC42533F--

From: Paul Goyette <paul@whooppee.com>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/56329: nvme(4) takes long time to umount
Date: Wed, 20 Apr 2022 07:50:15 -0700 (PDT)

 On Wed, 20 Apr 2022, J. Hannken-Illjes wrote:

 > Finally got a laptop with
 >     nvme0: SAMSUNG MZVLB256HAHQ-000L7, firmware 0L2QEXD7, serial =
 > S41GNB0K593889
 > and see unmount times of 10 .. 12 seconds with a DIAGNOSTIC + DEBUG + =
 > LOCKDEBUG
 > kernel.  All this time is spent in the first vflush().
 >
 > On a GENERIC kernel without DIAGNOSTIC etc the unmount time goes down to =
 > less
 > than a second and the disk becomes much faster.
 >
 > What kind of kernel are/were you using?

 All of my kernels have DEBUG and DIAGNOSTIC enabled.  I do not have
 LOCKDEBUG.


 +--------------------+--------------------------+----------------------+
 | Paul Goyette       | PGP Key fingerprint:     | E-mail addresses:    |
 | (Retired)          | FA29 0E3B 35AF E8AE 6651 | paul@whooppee.com    |
 | Software Developer | 0786 F758 55DE 53BA 7731 | pgoyette@netbsd.org  |
 | & Network Engineer |                          | pgoyette99@gmail.com |
 +--------------------+--------------------------+----------------------+

From: "J. Hannken-Illjes" <hannken@mailbox.org>
To: NetBSD GNATS <gnats-bugs@netbsd.org>
Cc: 
Subject: Re: kern/56329: nvme(4) takes long time to umount
Date: Wed, 20 Apr 2022 17:04:34 +0200

 --Apple-Mail=_1D601804-510B-4949-BEC5-AB6E43F5EE3C
 Content-Transfer-Encoding: 7bit
 Content-Type: text/plain;
 	charset=us-ascii

 > On 20. Apr 2022, at 16:55, Paul Goyette <paul@whooppee.com> wrote:
 <snip>
 > All of my kernels have DEBUG and DIAGNOSTIC enabled.  I do not have
 > LOCKDEBUG.

 Could you try a kernel without DEBUG/DIAGNOSTIC and see if it makes
 a visible difference?

 --
 J. Hannken-Illjes - hannken@mailbox.org

 --Apple-Mail=_1D601804-510B-4949-BEC5-AB6E43F5EE3C
 Content-Transfer-Encoding: 7bit
 Content-Disposition: attachment;
 	filename=signature.asc
 Content-Type: application/pgp-signature;
 	name=signature.asc
 Content-Description: Message signed with OpenPGP

 -----BEGIN PGP SIGNATURE-----

 iQIzBAEBCAAdFiEEyLVMkhxs8fxixv+2IOocBq6p/bMFAmJgIQIACgkQIOocBq6p
 /bMghQ//bW2v8hC0QLXO5ekEfNiN16VTlS04uWcJ4pg2wMVg9J+H+eBTuqFcvgvB
 idqIn/AuyBMHiX6FLvUTFtxBM4r2tclDgpHGf4j9YE+8BG4hl+TjcSGWm7GUGChh
 T6WJ7SilAWBwaqNXA2aOj1JB5zBl98imJyWP7Z+b1CSsZpndXnOxeaT3lkndGIm8
 IWwu1JIpGN+c64+ijawe8QEDW322yWgaYla4YMeCgbirEpgkyAkMrNJQ6Oy2aFGW
 4pH8axmbjuuf2fkANB6w67xgQD06lZ74pabLVsndVDv/4pL4ZvlJ2rlTN0Ps60Qn
 uoXxoteE+VlOmErWoFlqeu2F4+Md9b/JFET28vJ5L7a4b+jkLMnfm6VknmmsHA4E
 YkiZQ+uUMAsqGWyxNrfcX0syH2WfDAEcmx1IJQtaOhviqIuTUnSFfvlVlIH+GHjX
 UJTGSNmFzzpc8+1VuDBW31MOsRxkrF02jNIY1NdhBTKjfolnrHG9bwvV5s/mLv3d
 P4l2f5J/yrozryd4ohHsdf5a4+XceDz3EAYtl4mWSKeUDz8C8w5oRvzt5wZHyaj1
 v74Vxh3sh5BzdnM98oR7l4mETv3PNbt3l4q1AWMByk3diBpiawEHySb2sUAVq0hl
 lBC/JU8wdRHOeulv6pSw/psPcTpmtqUBNhhGuyVE0pFVGRjyZT8=
 =NtI+
 -----END PGP SIGNATURE-----

 --Apple-Mail=_1D601804-510B-4949-BEC5-AB6E43F5EE3C--

From: matthew green <mrg@eterna.com.au>
To: gnats-bugs@netbsd.org, "J. Hannken-Illjes" <hannken@mailbox.org>
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org, paul@whooppee.com
Subject: re: kern/56329: nvme(4) takes long time to umount
Date: Thu, 21 Apr 2022 06:26:55 +1000

 >  Finally got a laptop with
 >      nvme0: SAMSUNG MZVLB256HAHQ-000L7, firmware 0L2QEXD7, serial S41GNB=
 0K593889
 >  and see unmount times of 10 .. 12 seconds with a DIAGNOSTIC + DEBUG + L=
 OCKDEBUG
 >  kernel.  All this time is spent in the first vflush().
 >  =

 >  On a GENERIC kernel without DIAGNOSTIC etc the unmount time goes down t=
 o less
 >  than a second and the disk becomes much faster.
 >  =

 >  What kind of kernel are/were you using?

 FWIW, i see this problem on non-nvme sometimes too, and it
 seems related to how much data was written between booting
 and shutdown.

 DIAGNOSTIC shouldn't affect speed nearly this much!  code
 like that should be in DEBUG probably.


 .mrg.

From: Paul Goyette <paul@whooppee.com>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/56329: nvme(4) takes long time to umount
Date: Fri, 22 Apr 2022 07:15:49 -0700 (PDT)

 On Wed, 20 Apr 2022, J. Hannken-Illjes wrote:

 > > All of my kernels have DEBUG and DIAGNOSTIC enabled.  I do not have
 > > LOCKDEBUG.
 > 
 > Could you try a kernel without DEBUG/DIAGNOSTIC and see if it makes
 > a visible difference?

 OK, I tried a 9.99.96 kernel with both DEBUG and DIAGNOSTIC turned
 off.

 I ran my usual home-brew version of bulk-package-building script,
 with about 680 packages, in a chroot sandbox rooted on the nvme
 device.

 When the build finished, I removed the chroot environment with
 no problem (so, the ``remnant on /tmp'' bug previously reported
 is gone!).

 I then tried to unmount the nvme itself.  It took more than 10
 seconds before it reported EBUSY.  I guess I must have had some
 process somewhere with a reference to the device.

 So, the unmount did not complete, but it still took a long time
 to fail.


 +--------------------+--------------------------+----------------------+
 | Paul Goyette       | PGP Key fingerprint:     | E-mail addresses:    |
 | (Retired)          | FA29 0E3B 35AF E8AE 6651 | paul@whooppee.com    |
 | Software Developer | 0786 F758 55DE 53BA 7731 | pgoyette@netbsd.org  |
 | & Network Engineer |                          | pgoyette99@gmail.com |
 +--------------------+--------------------------+----------------------+

From: "J. Hannken-Illjes" <hannken@mailbox.org>
To: NetBSD GNATS <gnats-bugs@netbsd.org>
Cc: 
Subject: Re: kern/56329: nvme(4) takes long time to umount
Date: Sat, 23 Apr 2022 11:11:38 +0200

 --Apple-Mail=_40ED12DE-DD07-4479-BEF2-87F103F39CED
 Content-Type: multipart/mixed;
 	boundary="Apple-Mail=_64336509-853A-4381-9767-48DDBFA081A2"


 --Apple-Mail=_64336509-853A-4381-9767-48DDBFA081A2
 Content-Transfer-Encoding: 7bit
 Content-Type: text/plain;
 	charset=us-ascii

 > On 22. Apr 2022, at 16:20, Paul Goyette <paul@whooppee.com> wrote:
 <snip>
 > OK, I tried a 9.99.96 kernel with both DEBUG and DIAGNOSTIC turned
 > off.
 <snip>
 > So, the unmount did not complete, but it still took a long time
 > to fail.

 So it is independent of DIAGNOSTIC and DEBUG -- good.

 Next time you run a bulk build please do the unmount with the
 script attached as "sh do_unmount.sh | tee do_unmount.log" and
 send the files do_unmount.log, pre-null.log and pre-build.log
 off list to hannken@mailbox.org

 --
 J. Hannken-Illjes - hannken@mailbox.org


 --Apple-Mail=_64336509-853A-4381-9767-48DDBFA081A2
 Content-Disposition: attachment;
 	filename=do_unmount.sh
 Content-Type: application/octet-stream;
 	name=do_unmount.sh;
 	x-unix-mode=0600
 Content-Transfer-Encoding: 7bit

 #! /bin/sh

 do_unmount(){

 	iostat -I -x ld0
 	date "+=== %T unmount ${1}"
 	umount ${1}
 	date "+=== %T"
 	iostat -I -x ld0
 }

 pstat -v > pre-null.log

 do_unmount /build/netbsd-local/src
 do_unmount /build/netbsd-local/xsrc
 do_unmount /build/netbsd-current/src

 pstat -v > pre-build.log

 do_unmount /build

 --Apple-Mail=_64336509-853A-4381-9767-48DDBFA081A2--

 --Apple-Mail=_40ED12DE-DD07-4479-BEF2-87F103F39CED
 Content-Transfer-Encoding: 7bit
 Content-Disposition: attachment;
 	filename=signature.asc
 Content-Type: application/pgp-signature;
 	name=signature.asc
 Content-Description: Message signed with OpenPGP

 -----BEGIN PGP SIGNATURE-----

 iQIzBAEBCAAdFiEEyLVMkhxs8fxixv+2IOocBq6p/bMFAmJjwsoACgkQIOocBq6p
 /bO3BhAAphW1ArzU0aHgNTzcs9zGyQmL6SOllXEZkzll94MWDoUyW5+ZLkBJ0kUB
 VRIhdiN9gtZXoupr/SP9lTv7HT8PnezBNZfi3ARhVmUeySKC+vQUrqhS6PFHWMNJ
 X4BeFCNxjganZ8w3XKI4vAcseb+ZRyYdGNVkZYfYa4X6LWT0qgMmxumGhEznmrss
 FrVlLLg9mZbu1UcTe6frI5o82X1xjo2DjV3wEln0n8PBr/OlRruGBexfUYngkNZ7
 neqQtvC5eGTsl3Op8L9ui+7GDjojStu6SQJ/P9nQx70Hb241SZlNDbf/lafHGLgD
 6WIZD74PiZGZz/48gzrnQmnvwyAPHHcaYXO2DYIuT38u0LXP/B+NRjHHv9ap9lR/
 eKPwAt9uCtjHJpSY7YxcmwEC4DI9B1QHnbxXe0vrbDBtTWD5bY47Qu9RkyVRhlip
 1YM1iQCJhWAiE2+j242OFjN05lJR17b+WsS89Ji26+dzk23bnUsEn4lhVciFdd6A
 GrSp9GmJAgUhc4ThA7uvNBvRdvEph2GQlObowZs0so+/Or6aPK/UPTpy5wqRd+sI
 DfyjhIUYRh+sDEfIWYYGvjLnaHToCDXuOl4SDvVu80YNTR9M4/fhRO5nkn7ve5um
 HtJcR8BSKfXRa3rWmCDrLrUimjG4zpNjQFPlJt+4X3kE/tndLuk=
 =sbBE
 -----END PGP SIGNATURE-----

 --Apple-Mail=_40ED12DE-DD07-4479-BEF2-87F103F39CED--

From: matthew green <mrg@eterna.com.au>
To: gnats-bugs@netbsd.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org, paul@whooppee.com
Subject: re: kern/56329: nvme(4) takes long time to umount
Date: Sun, 24 Apr 2022 02:58:51 +1000

 >  Next time you run a bulk build please do the unmount with the
 >  script attached as "sh do_unmount.sh | tee do_unmount.log" and
 [ ...]
 >  do_unmount /build/netbsd-local/src
 >  do_unmount /build/netbsd-local/xsrc
 >  do_unmount /build/netbsd-current/src
 [ ...]
 >  do_unmount /build

 one of the systems i see the problem on is currently running
 a pkg build of over 1000 packages, and usually will see this
 problem.  this is an arm64 16-core machine with 64GB ram and
 a single 1T nvme.  ie, everything is just "/" for me.  is there
 a way to modify this script usefully for me?  i'm not sure how
 much will be written to any fs :)

 thanks.


 .mrg.

From: "J. Hannken-Illjes" <hannken@mailbox.org>
To: NetBSD GNATS <gnats-bugs@netbsd.org>
Cc: 
Subject: Re: kern/56329: nvme(4) takes long time to umount
Date: Mon, 25 Apr 2022 16:02:40 +0200

 With help from Paul Goyette I'm able to reproduce the behaviour here.

 - It is not related to nvme, it depends on the number of cached
   vnodes, 3456591 in the last run taking ~7 seconds.

 - With 64GByte memory my test machine takes ~6 seconds to unmount
   a file system with 3332133 cached vnodes with "ld at virtio".

 - This time to unmount didn't change in the last 7 years, testing
   kernels from October 2014, 2015 ... until now it always took 5 +- 1
   seconds to unmount.

 - Latest run took 5.864 seconds, 1.311 were spent in cache_purgevfs(),
   0.290 in VFS_SYNC() and 4.263 in vflush().

 To me this doesn't look like a regression and recycling ~570000 vnodes
 per second looks ok.

 --
 J. Hannken-Illjes - hannken@mailbox.org

From: Christos Zoulas <christos@zoulas.com>
To: gnats-bugs@netbsd.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
 netbsd-bugs@netbsd.org, paul@whooppee.com
Subject: Re: kern/56329: nvme(4) takes long time to umount
Date: Mon, 25 Apr 2022 11:07:38 -0400

 But should't be something applying backpressure so that the number of vnodes=
  to be recycled is kept reasonably sized?

 christos

 > On Apr 25, 2022, at 10:05 AM, J. Hannken-Illjes <hannken@mailbox.org> wrot=
 e:
 >=20
 > =EF=BB=BFThe following reply was made to PR kern/56329; it has been noted b=
 y GNATS.
 >=20
 > From: "J. Hannken-Illjes" <hannken@mailbox.org>
 > To: NetBSD GNATS <gnats-bugs@netbsd.org>
 > Cc:=20
 > Subject: Re: kern/56329: nvme(4) takes long time to umount
 > Date: Mon, 25 Apr 2022 16:02:40 +0200
 >=20
 > With help from Paul Goyette I'm able to reproduce the behaviour here.
 >=20
 > - It is not related to nvme, it depends on the number of cached
 >   vnodes, 3456591 in the last run taking ~7 seconds.
 >=20
 > - With 64GByte memory my test machine takes ~6 seconds to unmount
 >   a file system with 3332133 cached vnodes with "ld at virtio".
 >=20
 > - This time to unmount didn't change in the last 7 years, testing
 >   kernels from October 2014, 2015 ... until now it always took 5 +- 1
 >   seconds to unmount.
 >=20
 > - Latest run took 5.864 seconds, 1.311 were spent in cache_purgevfs(),
 >   0.290 in VFS_SYNC() and 4.263 in vflush().
 >=20
 > To me this doesn't look like a regression and recycling ~570000 vnodes
 > per second looks ok.
 >=20
 > --
 > J. Hannken-Illjes - hannken@mailbox.org
 >=20

From: mlelstv@serpens.de (Michael van Elst)
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/56329: nvme(4) takes long time to umount
Date: Mon, 25 Apr 2022 15:31:49 -0000 (UTC)

 christos@zoulas.com (Christos Zoulas) writes:

 >But should't be something applying backpressure so that the number of vnodes=
 > to be recycled is kept reasonably sized?

 The system tries to keep 'desired_vnodes' and that value was bumped.

 Saying that, recycling unused, unreferenced vnodes should still be faster
 and if that isn't possible, should be done more aggressively.

From: matthew green <mrg@eterna.com.au>
To: gnats-bugs@netbsd.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org, paul@whooppee.com
Subject: re: kern/56329: nvme(4) takes long time to umount
Date: Sun, 01 May 2022 14:57:18 +1000

 >  To me this doesn't look like a regression and recycling ~570000 vnodes
 >  per second looks ok.

 it definiately feels like it got slower sometime in the last
 6mo-2 years.  it sure didn't take 10s of seconds to reboot
 because unmounting root took so long before:

 [ 2943701.7107086] unmounting 0xffff00000065b000 / (/dev/ld4a)...
 [ 2943725.4316875] unmounting done

 this was just now, on a system up ~34 days.


 i'll see if i can reproduce this on a system with both netbsd-9
 and -current installed.


 .mrg.

From: "J. Hannken-Illjes" <hannken@mailbox.org>
To: NetBSD GNATS <gnats-bugs@netbsd.org>
Cc: 
Subject: Re: kern/56329: nvme(4) takes long time to umount
Date: Sun, 1 May 2022 15:56:19 +0200

 --Apple-Mail=_29F66330-9AC2-47E3-8D96-E4332FBE06DE
 Content-Transfer-Encoding: 7bit
 Content-Type: text/plain;
 	charset=us-ascii

 > On 1. May 2022, at 07:00, matthew green <mrg@eterna.com.au> wrote:
 >> To me this doesn't look like a regression and recycling ~570000 vnodes
 >> per second looks ok.
 > 
 > it definiately feels like it got slower sometime in the last
 > 6mo-2 years.  it sure didn't take 10s of seconds to reboot
 > because unmounting root took so long before:
 > 
 > [ 2943701.7107086] unmounting 0xffff00000065b000 / (/dev/ld4a)...
 > [ 2943725.4316875] unmounting done
 > 
 > this was just now, on a system up ~34 days.

 - Memory size and therefore kern.maxvnodes didn't change over
   the last two years?

 - How large is your vnode cache (sysctl -n kern.maxvnodes),
   did you change it manually?

 --
 J. Hannken-Illjes - hannken@mailbox.org

 --Apple-Mail=_29F66330-9AC2-47E3-8D96-E4332FBE06DE
 Content-Transfer-Encoding: 7bit
 Content-Disposition: attachment;
 	filename=signature.asc
 Content-Type: application/pgp-signature;
 	name=signature.asc
 Content-Description: Message signed with OpenPGP

 -----BEGIN PGP SIGNATURE-----

 iQIzBAEBCAAdFiEEyLVMkhxs8fxixv+2IOocBq6p/bMFAmJukYMACgkQIOocBq6p
 /bO/Aw/7Bk7yMYelV7ub3swS6MQ9W7KnXV2sI4nIFNTBbyamIaQPstaF8tQYZCZK
 +EgUvkQnrUm+uxu5DAZt4hZjMbeop3crTsy3gWQTOl/iQ1JtFp12Yt9dpqgFV/YN
 0DlA/n1R3z3KDu0a5+zCsNJL0tVBZM0/nO5gGFtO8ADeH3Hj9B8bVFr1N4OKo4H8
 QaYiQJQq+M75RtMau2y/xXgz/S/jjy0gflG5YCOwPOpJFF4ptQiGAiqRgkxaq+Sa
 FCeeBUT8QqJZS2QqQN5RPpfKCDV4qknznZh19vaATDEx/71ztep15aJRGYsJfQCS
 G7C0UV0eX2BpRDWvDaBINm5UgLRO1DGOO0pxT8IS0TrpztdxsXTWNGNFZ+yBQO/y
 RVWH/mE8bKeBifvbdUFBigAkSyOkJZuGZYsHhlGzC3zIdhtU0ZhirQpyVRJHsx4/
 AZnbR65ygqtWGuHzsqy8DHWpQ/BcpC4ihMvWmrZJ9C0MeIr9YNrc26cxboLbnNBT
 Hwi4WlrgTxLDdFmRzSu1/PqlUjl1EdFiZMaTyH8QxU8JJyy/ww+GlrcuK3zJNOd/
 3u/VyxS9VMs0PjRDW34mPPeH466AgLZr8ZMqR5RJtEwnC/ibETC7I+Vtbqe5fg73
 3Gs8Ma3zjiVzuYPcaTtmIELi6aUQMeQxoFS2OIBNATb7phxxqZk=
 =YIcn
 -----END PGP SIGNATURE-----

 --Apple-Mail=_29F66330-9AC2-47E3-8D96-E4332FBE06DE--

From: Paul Goyette <paul@whooppee.com>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/56329: nvme(4) takes long time to umount
Date: Tue, 28 Nov 2023 06:51:10 -0800 (PST)

 J. Hannken-Illjes <hannken@mailbox.org> asked here some 18 months
 ago (sorry for the delayed response)

 >  - Memory size and therefore kern.maxvnodes didn't change over
 >    the last two years?
 >
 >  - How large is your vnode cache (sysctl -n kern.maxvnodes),
 >    did you change it manually?

 # sysctl -n kern.maxvnodes
 kern.maxvnodes = 6706308

 This is the boot-time default, based on the machine has 128GB RAM.


 +---------------------+--------------------------+----------------------+
 | Paul Goyette (.sig) | PGP Key fingerprint:     | E-mail addresses:    |
 | (Retired)           | 1B11 1849 721C 56C8 F63A | paul@whooppee.com    |
 | Software Developer  | 6E2E 05FD 15CE 9F2D 5102 | pgoyette@netbsd.org  |
 | & Network Engineer  |                          | pgoyette99@gmail.com |
 +---------------------+--------------------------+----------------------+

>Unformatted:

Home	PR Database Search