NetBSD Problem Report #59870

From www@netbsd.org  Wed Dec 31 04:18:41 2025
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits)
	 client-signature RSA-PSS (2048 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id DAE4A1A923C
	for <gnats-bugs@gnats.NetBSD.org>; Wed, 31 Dec 2025 04:18:41 +0000 (UTC)
Message-Id: <20251231041840.7E9031A923D@mollari.NetBSD.org>
Date: Wed, 31 Dec 2025 04:18:40 +0000 (UTC)
From: campbell+netbsd@mumble.net
Reply-To: campbell+netbsd@mumble.net
To: gnats-bugs@NetBSD.org
Subject: kernel lock runtime diagnostics are difficult
X-Send-Pr-Version: www-1.0

>Number:         59870
>Category:       kern
>Synopsis:       kernel lock runtime diagnostics are difficult
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Dec 31 04:20:00 +0000 2025
>Last-Modified:  Sat Jan 03 23:10:02 +0000 2026
>Originator:     Taylor R Campbell
>Release:        current, 11, 10, 9, ...
>Organization:
The NetBSD Locker, Inc.
>Environment:
>Description:

	Sometimes the legacy kernel lock is held for an unreasonably
	long time.

	How do you tell who holds it when this happens?  If you're
	lucky, you can enter ddb and find the threads running on each
	cpuN with `ps' and switch to `mach cpu N' and run `bt' to find
	a code path that obviously holds the kernel lock.

	If you're not lucky, you have a heartbeat panic because some
	softint tried to take the kernel lock and waited too long for
	it and crash dumps failed because suspendsched has
	KASSERT(!cpu_intr_p()) and the heartbeat panic happens within
	an interrupt handler, as in:

	https://mail-index.NetBSD.org/current-users/2025/12/27/msg047183.html

	If you have enabled LOCKDEBUG, and the kernel lock is held for
	more than 10sec, you get a kernel lock spinout and an IPI is
	sent to the hogging CPU to geta stack trace.  But since
	autoconf(9) runs kernel-locked, loading a module for a driver
	can trigger this panic.

	It's also annoying when the kernel lock is held for enough time
	to make the system flaky (partly because, e.g., wscons(4) and
	pckbport(4) run with it, and so do some network drivers like
	iwm(4)), but not enough to trigger other diagnostics.  However,
	attempts to dtrace the kernel_lock function, along the lines of
	https://mail-index.netbsd.org/tech-kern/2022/10/30/msg028499.html,
	only show that it was taken in sleepq_block because something
	that held the kernel lock slept and then woke up again.

>How-To-Repeat:

	- chase bad interactive system latency due to kernel lock hogs
	- try to diagnose panics like
          https://mail-index.netbsd.org/current-users/2025/12/27/msg047183.html

>Fix:

	1. Enable the logic to provoke an IPI to dump a stack trace
	   _without_ LOCKDEBUG.

	2. Pass a cookie across the unlock/sleep/relock logic so that
	   dtrace can tell on whose behalf the relock happened.

	Other ideas welcome!

>Audit-Trail:
From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/59870 CVS commit: src/sys/kern
Date: Fri, 2 Jan 2026 04:40:10 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Fri Jan  2 04:40:10 UTC 2026

 Modified Files:
 	src/sys/kern: kern_lock.c

 Log Message:
 kern_lock.c: Sort includes.

 No functional change intended.

 Prompted by:

 PR kern/59870: kernel lock runtime diagnostics are difficult


 To generate a diff of this commit:
 cvs rdiff -u -r1.188 -r1.189 src/sys/kern/kern_lock.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: David Holland <dholland-sourcechanges@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/59870: kernel lock runtime diagnostics are difficult
Date: Fri, 2 Jan 2026 05:53:30 +0000

 From: Taylor R Campbell <riastradh@netbsd.org>
 To: source-changes@NetBSD.org
 Subject: CVS commit: src/sys/kern
 Date: Fri, 2 Jan 2026 04:40:26 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Fri Jan  2 04:40:26 UTC 2026

 Modified Files:
 	src/sys/kern: kern_lock.c

 Log Message:
 KERNEL_LOCK(9): Improve diagnostics when hogging kernel lock.

 1. Log diagnostics unconditionally, even if !LOCKDEBUG -- only put the
    _KERNEL_LOCK_ABORT("spinout") under LOCKDEBUG.

 2. Spin up to ten seconds (not 0x0fffffff iterations of the loop), as
    measured by getticks() and hz, before logging diagnostics.

 3. Avoid printing anything if it was already printed in the last ten
    seconds.

 This uses a somewhat grody symbol alias hack to keep the declaration
 of kernel_lock intact so it doesn't require changes in sys/lock.h or
 its users.

 kern/59870: kernel lock runtime diagnostics are difficult


 To generate a diff of this commit:
 cvs rdiff -u -r1.189 -r1.190 src/sys/kern/kern_lock.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/59870 CVS commit: src/sys/kern
Date: Sat, 3 Jan 2026 23:08:16 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Sat Jan  3 23:08:16 UTC 2026

 Modified Files:
 	src/sys/kern: kern_lock.c

 Log Message:
 KERNEL_LOCK(9): New dtrace probes for taking and dropping the lock.

 At some point, fbt::_kernel_lock:return worked to probe the `taking'
 part (judging by the message I wrote some years ago at
 https://mail-index.NetBSD.org/tech-kern/2022/10/30/msg028499.html)
 but no longer, so let's just make sure there is always a probe that
 explicitly works, on both sides:

 - sdt:kernel:lock:entry(nlocks) fires when we have determined we can
   take the kernel lock and we're about to increment its depth by
   nlocks.

 - sdt:kernel:lock:exit(nlocks) fires after we have released the kernel
   lock and already decremented its depth by nlocks.

 On first acquisition, or last release, of the kernel lock by a thread
 or CPU, in both probes the value of curlwp->l_blcnt or
 curcpu()->ci_biglock_count will be zero.

 PR kern/59870: kernel lock runtime diagnostics are difficult


 To generate a diff of this commit:
 cvs rdiff -u -r1.190 -r1.191 src/sys/kern/kern_lock.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

Home	PR Database Search