NetBSD Problem Report #57503

From www@netbsd.org  Wed Jul  5 10:12:30 2023
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 46DCF1A923D
	for <gnats-bugs@gnats.NetBSD.org>; Wed,  5 Jul 2023 10:12:30 +0000 (UTC)
Message-Id: <20230705101228.A29BD1A923E@mollari.NetBSD.org>
Date: Wed,  5 Jul 2023 10:12:28 +0000 (UTC)
From: jperkin@pkgsrc.org
Reply-To: jperkin@pkgsrc.org
To: gnats-bugs@NetBSD.org
Subject: nullfs locking contention
X-Send-Pr-Version: www-1.0

>Number:         57503
>Category:       kern
>Synopsis:       nullfs locking contention
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Jul 05 10:15:00 +0000 2023
>Last-Modified:  Wed Jul 05 11:30:01 +0000 2023
>Originator:     Jonathan Perkin
>Release:        NetBSD 10.0_BETA (GENERIC) #0: Tue Jun 27 18:56:25 UTC 2023
>Organization:
>Environment:
NetBSD pkgsrc-nb0 10.0_BETA NetBSD 10.0_BETA (GENERIC) #0: Tue Jun 27 18:56:25 UTC 2023  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/amd64/compile/GENERIC amd64
>Description:
I figured there would already be a ticket about this, but abs suggests that there isn't and requested I raise this one.

mount_null(8) does not appear to scale very well. Attempting to use pkgsrc/pkgtools/mksandbox to create a single chroot and run pbulk scan processes inside it runs incredibly slowly.

Switching to unpacked sets instead as suggested by Joerg and others instantly made everything around 3.5x faster, with more scope for running additional processes and improving performance even further.

Obviously this approach has many drawbacks though, and I'd much rather use null mounts if they had the same performance that loopback/bind mounts have on Linux/illumos/etc.
>How-To-Repeat:
Use mount_null(8) and observe lack of scaling with additional CPUs (my host has 16).
>Fix:
From what I recall when mentioning this on IRC a while back, one of the FreeBSD developers mentioned that they had fixed a number of similar issues in their nullfs, so you may just want to see what they did there.

>Audit-Trail:
From: mlelstv@serpens.de (Michael van Elst)
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/57503: nullfs locking contention
Date: Wed, 5 Jul 2023 11:27:32 -0000 (UTC)

 jperkin@pkgsrc.org writes:

 >mount_null(8) does not appear to scale very well. Attempting to use pkgsrc/pkgtools/mksandbox to create a single chroot and run pbulk scan processes inside it runs incredibly slowly.

 Works fine for me, although I recently had to adjust some mounts for performance reasons.

 A single builder looks like:

 procfs on /s1/proc type procfs (local)
 ptyfs on /s1/dev/pts type ptyfs (local)
 tmpfs on /s1/tmp type tmpfs (local)
 tmpfs on /s1/usr/pkg type tmpfs (local)
 /home/pbulk/pkgsrc-2023Q2 on /s1/usr/pkgsrc type null (read-only, local)
 /home/pbulk/amd64-10.0-2023Q2/pbulk/bulklog on /s1/pbulk/bulklog type null (local)
 /home/pbulk/amd64-10.0-2023Q2/pbulk/cache on /s1/pbulk/cache type null (local)
 tmpfs on /s1/pbulk/work type tmpfs (local)
 /scratch/pbulk/distfiles on /s1/pbulk/distfiles type null (local)
 /home/pbulk/packages-amd64-10.0-2023Q2 on /s1/pbulk/packages type null (local)
 /tank/data/pbulk/final-amd64-10.0-2023Q2 on /s1/pbulk/final type null (local)



 Previously I had only mounted /home/pbulk/amd64-10.0-2023Q2/pbulk -> /s1/pbulk
 which started to cause contention for /s1/pbulk/work. Avoiding the stacked
 mounts by mounting subdirectories of /s1/pbulk individually avoids this.

Home	PR Database Search