NetBSD Problem Report #60631

From www@netbsd.org  Sat Aug 22 08:46:00 2026
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "mail.netbsd.org", Issuer "YR2" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 73B761A923C
	for <gnats-bugs@gnats.NetBSD.org>; Sat, 22 Aug 2026 08:46:00 +0000 (UTC)
Message-Id: <20260822084559.07B781A923E@mollari.NetBSD.org>
Date: Sat, 22 Aug 2026 08:45:59 +0000 (UTC)
From: phil.netbsd@ashbysoft.com
Reply-To: phil.netbsd@ashbysoft.com
To: gnats-bugs@NetBSD.org
Subject: Intel N150 cpu hang leading to kernel panic in heartbeat
X-Send-Pr-Version: www-1.0
X-From4GNATS: "phil.netbsd@ashbysoft.com via gnats" <gnats-admin@NetBSD.org>

>Number:         60631
>Category:       port-amd64
>Synopsis:       Intel N150 cpu hang leading to kernel panic in heartbeat
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    port-amd64-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Aug 22 08:50:00 +0000 2026
>Last-Modified:  Sat Aug 22 15:25:01 +0000 2026
>Originator:     Phil Ashby
>Release:        NetBSD 11.0 (GENERIC)
>Organization:
>Environment:
NetBSD timmy-iv 11.0 NetBSD 11.0 (GENERIC) #0: Thu Jul 30 15:23:12 UTC 2026  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/amd64/compile/GENERIC amd64
>Description:
On a Beelink ME Pro NAS, which uses the Intel N150 SoC and associated Intel PCH chipset, I find the following with current NetBSD 11.0 (GENERIC):

1/ USB image installer works OK
2/ Single user boot (boot -s) works OK
3/ Multi user boot (boot) hangs during /etc/rc execution
  3.1/ Further execution appears to stop (no visible progress)
  3.2/ heartbeat(9) triggers a kernel panic after 15s (1501 beats)

>How-To-Repeat:
The problem is completely repeatable on every default multi user boot
>Fix:
After experimenting with boot options (thanks to Stephen Borill) I have discovered that avoiding ACPI (boot -2) provides a reliable solution. Sometimes a uniprocessor boot (boot -1) works, but not always.

Further information:

DDB shows that the stuck cpu is asleep in x86_stihlt() at the point when the heartbeat panic fires.

I am happy to collect crash dumps for analysis.

>Audit-Trail:
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-amd64/60631: Intel N150 cpu hang leading to kernel panic in
 heartbeat
Date: Sat, 22 Aug 2026 11:59:57 +0200

 On Sat, Aug 22, 2026 at 08:50:00AM +0000, phil.netbsd@ashbysoft.com via gnats wrote:
 > 3/ Multi user boot (boot) hangs during /etc/rc execution

 What is the last few lines printed on console before it stops?

 Can you show the contents of /etc/rc.conf and any /etc/ifconfig* files
 (in case you created some)?

 Can you provide full dmesg output from both a non-ACPI boot and an
 ACPI boot to single-user?

 The latter can be created by "boot -s", and when in the single user shell
 run something like: "mount -u /; dmesg > /dmesg.acpi"
 and then reboot into non-ACPI mode and get the /dmesg.acpi out.

 Thanks,

 Martin

From: Phil Ashby <phil.netbsd@ashbysoft.com>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-amd64/60631: Intel N150 cpu hang leading to kernel panic
 in heartbeat
Date: Sat, 22 Aug 2026 13:21:04 +0100

 Hi Martin,

 As requested, /etc/rc.conf pasted below, I shall now create dmesg dumps
 and reproduce the failure a couple of times to collect last few console
 lines and crash dumps. I should note that the earlier failures did not
 have tap0, tinc, dnsmasq, openntpd, sshd configured, but things never
 get this far.

 Dmesg and crash dumps all here:
 https://phil.ashbysoft.com/dumps/

 Last few console lines on a failed multi user boot with DDB enabled:

 Sat Aug 22 11:53:02 UTC 2026
 Starting root file system check:
 /dev/rdk1: file system is clean; not checking
 Setting sysctl variables:
 ddb.onpanic: 1 -> 1
 swapctl: setting dump device to /dev/dk2
 swapctl: adding /dev/dk2 as swap device at priority 0
 Starting file system checks:
 Loaded entropy from /var/db/entropy-file.
 Waiting for entropy...done
 Setting tty flags.

 ...15 seconds later...
 [DDB crash handler]

 --8<-- /etc/rc.conf:
 # Load the defaults in from /etc/defaults/rc.conf (if it's readable).
 # These can be overridden below.
 #
 if [ -r /etc/defaults/rc.conf ]; then
 	. /etc/defaults/rc.conf
 fi

 # If this is not set to YES, the system will drop into single-user mode.
 #
 rc_configured=YES

 # Add local overrides below.
 #
 wscons=YES
 # Networking:
 # - explicit list of interfaces to configure
 auto_ifconfig=NO
 net_interfaces="igc0 tap0"
 # - igc0 (2.5G) port is internal..
 ifconfig_igc0="inet 192.168.0.10/24"
 # - tap0 (tinc), also runs tincd (see /usr/pkg/etc/tinc)
 ifconfig_tap0="inet 10.1.1.19/24; !/usr/pkg/sbin/tincd -n tinc0
 --pidfile=/tmp/tincd.pid" # Default route..
 defaultroute="192.168.0.1"
 # DNS (see /usr/pkg/etc/dnsmasq.conf)
 dnsmasq=YES
 # Clock sync (see /usr/pkg/etc/ntpd.conf)
 openntpd=YES
 # Remote access (see /etc/ssh/sshd_config)
 sshd=YES
 --8<--

From: Taylor R Campbell <riastradh@NetBSD.org>
To: gnats-bugs@netbsd.org
Cc: port-amd64-maintainer@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org, phil.netbsd@ashbysoft.com
Subject: Re: port-amd64/60631: Intel N150 cpu hang leading to kernel panic
	in heartbeat
Date: Sat, 22 Aug 2026 15:20:05 +0000

 Crash dump 2 makes it look like maybe there's an interrupt storm, or a
 buggy interrupt handler going in a loop, preventing hardclock from
 making progress:

 crash> mach cpu 0
 using CPU 0
 crash> bt
 __kernel_end() at 0
 intr_kdtrace_wrapper() at intr_kdtrace_wrapper+0x32
 Xhandle_ioapic_edge2() at Xhandle_ioapic_edge2+0x75
 --- interrupt ---
 Xspllower() at Xspllower+0xe
 hardclock() at hardclock+0xb3
 Xresume_lapic_ltimer() at Xresume_lapic_ltimer+0x1e
 --- interrupt ---
 Xspllower() at Xspllower+0xe
 comopen() at comopen+0x2a1
 cdev_open() at cdev_open+0x12a
 spec_open() at spec_open+0x1e8
 VOP_OPEN() at VOP_OPEN+0x3e
 vn_open() at vn_open+0x32c
 do_open() at do_open+0xb3
 do_sys_openat() at do_sys_openat+0x72
 sys_open() at sys_open+0x24
 syscall() at syscall+0x9a
 --- syscall (number 5) ---
 syscall+0x9a:

 Let's see how many interrupts have arrived so far:

 crash> show event/i
 evcnt type 1: TLB shootdown = 342
 evcnt type 1: cpu0 timer = 301
 evcnt type 1: msi1 vec 0 = 57
 evcnt type 1: ioapic0 pin 16 = 5
 evcnt type 1: msix2 vec 1 = 1084
 evcnt type 1: msix2 vec 2 = 8587
 evcnt type 1: msix2 vec 3 = 87
 evcnt type 1: msix2 vec 4 = 3

 Looks like these correspond to nvme queues:

 nvme0: for admin queue interrupting at msix2 vec 0
 nvme0: WD PC SN540 SDDPNPF-512G, firmware 33006000, serial 251051800551
 nvme0: for io queue 1 interrupting at msix2 vec 1 affinity to cpu0
 nvme0: for io queue 2 interrupting at msix2 vec 2 affinity to cpu1
 nvme0: for io queue 3 interrupting at msix2 vec 3 affinity to cpu2
 nvme0: for io queue 4 interrupting at msix2 vec 4 affinity to cpu3

 Is 9-10k interrupts from nvme by this time a lot?  Seems like it might
 be a lot.  But only 1k of them were on cpu0, and it doesn't seem like
 that should be enough to hold up the timecounter for 15sec.


 The x86_stihlt part is a red herring -- cpu3 is just idling (with the
 HLT instruction as ACPI recommended), and then when the hardclock
 timer fired on cpu3, it detected that the timecounter hasn't advanced
 and panicked:

 crash> bt
 __kernel_end() at 0
 vpanic() at vpanic+0x171
 panic() at vprintf
 heartbeat() at heartbeat+0x175
 hardclock() at hardclock+0x9c
 Xresume_lapic_ltimer() at Xresume_lapic_ltimer+0x1e
 --- interrupt ---
 x86_stihlt() at x86_stihlt+0x6
 acpicpu_cstate_idle() at acpicpu_cstate_idle+0x128
 idle_loop() at idle_loop+0x89

 heartbeat+0x175 is not very helpful, but going back one instruction
 gives:

 (gdb) info line *(heartbeat+0x170)
 Line 681 of "/usr/src/sys/kern/kern_heartbeat.c"
    starts at address 0xffffffff80ce9f39 <heartbeat+359>
    and ends at 0xffffffff80ce9f47 <heartbeat+373>.

     676 		stamp =
     677 		    atomic_load_relaxed(&curcpu()->ci_heartbeat_uptime_stamp);
     678 		d = count - stamp;
     679 		if (__predict_false(d > period_ticks) &&
     680 		    !heartbeat_timecounter_suspended()) {
  => 681 			panic("%s: time has not advanced in %u heartbeats",
     682 			    cpu_name(curcpu()), d);
     683 		}

 https://nxr.NetBSD.org/xref/src/sys/kern/kern_heartbeat.c?r=1.14#681

Home	PR Database Search