NetBSD Problem Report #60607

From www@netbsd.org  Mon Aug 17 05:32:06 2026
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "mail.netbsd.org", Issuer "YR2" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id DAB5E1A923B
	for <gnats-bugs@gnats.NetBSD.org>; Mon, 17 Aug 2026 05:32:06 +0000 (UTC)
Message-Id: <20260817053205.897461A923C@mollari.NetBSD.org>
Date: Mon, 17 Aug 2026 05:32:05 +0000 (UTC)
From: thorpej@me.com
Reply-To: thorpej@me.com
To: gnats-bugs@NetBSD.org
Subject: /etc/rc does too much throw-away work and is needlessly slow on constrained systems
X-Send-Pr-Version: www-1.0
X-From4GNATS: "thorpej@me.com via gnats" <gnats-admin@NetBSD.org>

>Number:         60607
>Category:       bin
>Synopsis:       /etc/rc does too much throw-away work and is needlessly slow on constrained systems
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    thorpej
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Aug 17 05:35:00 +0000 2026
>Closed-Date:    Thu Aug 20 14:47:30 +0000 2026
>Last-Modified:  Thu Aug 20 14:47:49 +0000 2026
>Originator:     Jason Thorpe
>Release:        NetBSD 11.99.7
>Organization:
Retrogrouches'R'Us
>Environment:
Any sufficiently constrained machine will do, but wrap030 is the exemplar.
>Description:
NetBSD's /etc/rc and /etc/rc.d are a well-designed facility that provide a good mix of flexibility, extensibility, and ease of configuration.  However, the implementation performs quite a lot of work that, on constrained systems, leads to frustratingly long boot times.  Part of this issue stems from per-service isolation implemented using sub-shells. the "rc_fast_and_loose" config setting was introduced to paper over this.

To illustrate the point, boot times of ~13 minutes have been observed on a 25MHz 68030 with a PIO-only ATA disk interface.  Instrumentation showed that huge amounts of time, with no progress reporting, were being spent between start-up messages running start-up scripts that did nothing because they were not enabled.

The issue stems from the fact that a lot of processing work goes into an rc.d script even if the rcvar that controls it is set to NO.  This is throw-away work that consumes time for no benefit.
>How-To-Repeat:
Boot even a minimally configured system on a constrained machine.  Real 32-bit SPARC, VAX, and m68k hardware are more likely to exhibit the performance problem of the system.
>Fix:
I've devised a simple solution to the problem that's minimally invasive and, on the wrap030 homebrew, cuts the boot time (with ldconfig=NO) from ~13 minutes to ~7 minutes.

The solution is comprised of two parts:

1. Some small modifications to /etc/rc.subr that add a new standard "doeswork" directive to rc.d scripts.  This directive interrogates the script to ask if it's going to do any work.  The answer is YES if the script's rcvar is set to YES or if the script is not controlled by an rcvar.  Otherwise, it means the script's rcvar is set to NO and the script will thus do no useful work at boot time.

2. Another small change to /etc/rc where the real work of invoking the rc.d scripts is performed.  This second change introduces the notion of the "rcorder.cache".  At boot time, if it is determined that the cache needs to be rebuilt, then all of the rc.d scripts are interrogated in rcorder-order to see if they will do work.  If so, they're added to the list, otherwise they are ignored.  Once the list is built, then the list is invoked to start each service in the normal way.  The list is saved in the "rcorder.cache" and consulted at the next boot.  The cache is invalidated by modifying /etc/rc.conf, /etc/rc.d, or by manually removing /etc/rcorder.cache.

On some selected sample systems, this reduced the number of rc.d scripts that had to be run by ~66% while fully preserving the dependency ordering.  On very I/O-constrained systems, this cut system boot time by ~50%.

Patch that implements this change is here:

    https://www.netbsd.org/~thorpej/rcorder-cache-diff.txt

It almost certainly need some additional polish work, and probably some integration with etcupdate(8).

>Release-Note:

>Audit-Trail:
From: Taylor R Campbell <riastradh@NetBSD.org>
To: Jason Thorpe <thorpej@me.com>
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: bin/60607: /etc/rc does too much throw-away work and is needlessly slow on constrained systems
Date: Mon, 17 Aug 2026 13:19:16 +0000

 > Date: Mon, 17 Aug 2026 05:35:00 +0000 (UTC)
 > From: Jason Thorpe <thorpej@me.com>
 > 
 > I've devised a simple solution to the problem that's minimally
 > invasive and, on the wrap030 homebrew, cuts the boot time (with
 > ldconfig=NO) from ~13 minutes to ~7 minutes.

 Nice!

 > 2. Another small change to /etc/rc where the real work of invoking
 > the rc.d scripts is performed.  This second change introduces the
 > notion of the "rcorder.cache".  At boot time, if it is determined
 > that the cache needs to be rebuilt, then all of the rc.d scripts are
 > interrogated in rcorder-order to see if they will do work.  If so,
 > they're added to the list, otherwise they are ignored.  Once the
 > list is built, then the list is invoked to start each service in the
 > normal way.  The list is saved in the "rcorder.cache" and consulted
 > at the next boot.  The cache is invalidated by modifying
 > /etc/rc.conf, /etc/rc.d, or by manually removing /etc/rcorder.cache.

 Can you make this part optional, and/or populated up front at
 build-time or repopulated on request?  It would be nice if we could
 boot to multi-user from a read-only /, with all state stored on other
 file systems like /var, and without spurious EROFS warnings spewed to
 the console.  (I know we'll hit issues with, e.g., /etc/motd today,
 but I'd like to avoid adding more issues.)

From: Jason Thorpe <thorpej@me.com>
To: Taylor R Campbell <riastradh@NetBSD.org>
Cc: "gnats-bugs@netbsd.org" <gnats-bugs@NetBSD.org>,
 "netbsd-bugs@netbsd.org" <netbsd-bugs@NetBSD.org>
Subject: Re: bin/60607: /etc/rc does too much throw-away work and is
 needlessly slow on constrained systems
Date: Mon, 17 Aug 2026 06:42:14 -0700

 > On Aug 17, 2026, at 6:19=E2=80=AFAM, Taylor R Campbell =
 <riastradh@NetBSD.org> wrote:
 >=20
 > Can you make this part optional, and/or populated up front at
 > build-time or repopulated on request?  It would be nice if we could
 > boot to multi-user from a read-only /, with all state stored on other
 > file systems like /var, and without spurious EROFS warnings spewed to
 > the console.  (I know we'll hit issues with, e.g., /etc/motd today,
 > but I'd like to avoid adding more issues.)

 The cache can=E2=80=99t live on /var because /var might not be available =
 when the cache is required (this would especially be the case for =
 read-only roots).  The only path that=E2=80=99s currently guaranteed to =
 be available when /etc/rc starts processing the rc.d scripts is /etc.  =
 It=E2=80=99s pretty straightforward to suppress any error messages that =
 might occur when writing out the cache (which happens as the last step =
 of the boot process).  And, yes, I can refactor the code that builds and =
 writes out the cache into rc.subr so that there can be a =E2=80=9Cnostart=E2=
 =80=9D rc.d script to rebuild it for those who wish to do that before =
 the next reboot.

 -- thorpej

From: Jason Thorpe <thorpej@me.com>
To: Taylor R Campbell <riastradh@NetBSD.org>
Cc: "gnats-bugs@netbsd.org" <gnats-bugs@NetBSD.org>,
 "netbsd-bugs@netbsd.org" <netbsd-bugs@NetBSD.org>
Subject: Re: bin/60607: /etc/rc does too much throw-away work and is
 needlessly slow on constrained systems
Date: Mon, 17 Aug 2026 18:28:57 -0700

 > On Aug 17, 2026, at 6:42=E2=80=AFAM, Jason Thorpe <thorpej@me.com> =
 wrote:
 >=20
 > The cache can=E2=80=99t live on /var because /var might not be =
 available when the cache is required (this would especially be the case =
 for read-only roots).  The only path that=E2=80=99s currently guaranteed =
 to be available when /etc/rc starts processing the rc.d scripts is /etc. =
  It=E2=80=99s pretty straightforward to suppress any error messages that =
 might occur when writing out the cache (which happens as the last step =
 of the boot process).  And, yes, I can refactor the code that builds and =
 writes out the cache into rc.subr so that there can be a =E2=80=9Cnostart=E2=
 =80=9D rc.d script to rebuild it for those who wish to do that before =
 the next reboot.

 Updated patch: https://www.netbsd.org/~thorpej/rcorder-cache-diff-v2.txt

 Also ran it on my AlphaStation 200 4/233.  Even with a few heavy hitters =
 in the boot of that machine (ntpd wiring itself, sshd, fontconfig cache, =
 IPv6 DAD, etc.) that consume most of the time, it still managed to save =
 20 seconds, reducing =E2=80=9Cdate-to-date=E2=80=9D boot time from 2min =
 20sec to 2min flat.

 -- thorpej

From: "Jason R Thorpe" <thorpej@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/60607 CVS commit: src
Date: Thu, 20 Aug 2026 14:43:32 +0000

 Module Name:	src
 Committed By:	thorpej
 Date:		Thu Aug 20 14:43:32 UTC 2026

 Modified Files:
 	src/distrib/sets/lists/etc: mi
 	src/etc: rc rc.subr
 	src/etc/defaults: rc.conf
 	src/etc/rc.d: Makefile
 Added Files:
 	src/etc/rc.d: rcorder_cache

 Log Message:
 Add an optional mechanism, enabled by setting rcorder_cache=YES in
 /etc/rc.conf, to precompute and cache the list of rc.d scripts that
 will actually result in useful boot-time work.  This cached list is
 then consulted on subsequent boots to reduce the number of scripts
 run during boot.  This can have a huge impact on boot time on machines
 with slow CPUs and/or highly constrained I/O.  On the machine that
 inspired this change, it reduced the number of scripts visited during
 boot from 130 to 45 and reduced the boot time from 13 minutes to 7.

 PR bin/60607


 To generate a diff of this commit:
 cvs rdiff -u -r1.278 -r1.279 src/distrib/sets/lists/etc/mi
 cvs rdiff -u -r1.175 -r1.176 src/etc/rc
 cvs rdiff -u -r1.111 -r1.112 src/etc/rc.subr
 cvs rdiff -u -r1.169 -r1.170 src/etc/defaults/rc.conf
 cvs rdiff -u -r1.119 -r1.120 src/etc/rc.d/Makefile
 cvs rdiff -u -r0 -r1.1 src/etc/rc.d/rcorder_cache

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->closed
State-Changed-By: thorpej@NetBSD.org
State-Changed-When: Thu, 20 Aug 2026 14:47:30 +0000
State-Changed-Why:
After discussion on tech-userlevel, feedback incorporated and "v4"
of the patch committed.


Responsible-Changed-From-To: bin-bug-people->thorpej
Responsible-Changed-By: thorpej@NetBSD.org
Responsible-Changed-When: Thu, 20 Aug 2026 14:47:49 +0000
Responsible-Changed-Why:
I made the change.


>Unformatted:

Home	PR Database Search