NetBSD Problem Report #58715

From www@netbsd.org  Thu Oct  3 13:48:29 2024
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits)
	 client-signature RSA-PSS (2048 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id A13C51A923B
	for <gnats-bugs@gnats.NetBSD.org>; Thu,  3 Oct 2024 13:48:29 +0000 (UTC)
Message-Id: <20241003134828.2B39D1A923D@mollari.NetBSD.org>
Date: Thu,  3 Oct 2024 13:48:28 +0000 (UTC)
From: campbell+netbsd@mumble.net
Reply-To: campbell+netbsd@mumble.net
To: gnats-bugs@NetBSD.org
Subject: cpu0: softints stuck for 16 seconds
X-Send-Pr-Version: www-1.0

>Number:         58715
>Category:       kern
>Synopsis:       cpu0: softints stuck for 16 seconds
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Oct 03 13:50:00 +0000 2024
>Last-Modified:  Sat Oct 05 09:15:01 +0000 2024
>Originator:     Taylor R Campbell
>Release:        current
>Organization:
The RoyBSD Foundation
>Environment:
NetBSD netbsd 10.99.12 NetBSD 10.99.12 (GENERIC) #2: Thu Oct  3 00:35:53 UTC 2024  roy@netbsd:/home/roy/src/out/sys/arch/amd64/compile/GENERIC amd64
>Description:
panic: cpu0: softints stuck for 16 seconds

db{0}> bt
breakpoint() at netbsd:breakpoint+0x5
vpanic() at netbsd:vpanic+0x171
panic() at netbsd:panic+0x3c
heartbeat() at netbsd:heartbeat+0x34c
hardclock() at netbsd:hardclock+0x8b
Xresume_lapic_timer() at netbsd:Xresume_lapic_timer+0x1e
--- interrupt ---
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0x19a
idle_loop() at netbsd:idle_loop+0x128

In another instance (without the patch below), same panic but the stack trace was:

Crash version 10.99.6, image version 10.99.6.
Kernel compiled without options LOCKDEBUG.
System panicked: cpu0: softints stuck for 16 seconds
Backtrace from time of crash is available.
trace: pid 700 lid 700 at 0xffffc6844fccb670
sleepq_block() at sleepq_block+0x13a
cv_wait() at cv_wait+0x49
vioif_ctrl_send_command() at vioif_ctrl_send_command+0x1e6
vioif_set_rx_filter() at vioif_set_rx_filter+0x174
vioif_rx_filter() at vioif_rx_filter+0x157
vioif_ioctl() at vioif_ioctl+0x73
if_mcast_op() at if_mcast_op+0x54
in6m_destroy() at in6m_destroy+0x118
in6_leavegroup() at in6_leavegroup+0x34
in6_purgeaddr() at in6_purgeaddr+0xa5
in6_control1() at in6_control1+0x1205
in6_control() at in6_control+0x87
udp6_ioctl_wrapper() at udp6_ioctl_wrapper+0x33
compat_ifioctl() at compat_ifioctl+0xf5
doifioctl() at doifioctl+0xf06
sys_ioctl() at sys_ioctl+0x56d
syscall() at syscall+0x196
--- syscall (number 54) ---
syscall+0x196:
>How-To-Repeat:
Configure host with bridge0(tap0, tap1).

Configure a single guest with guest vioif0 <-> host tap0, guest vioif1 <-> host tap1.

Run dhcpcd with default configuration in the guest.

With this patch it triggers more reliably, but it d:

diff -r 05014b1ae511 sys/netinet/if_arp.c
--- a/sys/netinet/if_arp.c	Thu Sep 26 21:31:09 2024 +0000
+++ b/sys/netinet/if_arp.c	Thu Oct 03 13:33:45 2024 +0000
@@ -793,14 +793,6 @@
 	if (ah->ar_pln != sizeof(struct in_addr))
 		goto out;

-	ifp = if_get_bylla(ar_sha(ah), ah->ar_hln, &psref);
-	if (ifp) {
-		/* it's from me, ignore it. */
-		if_put(ifp, &psref);
-		ARP_STATINC(ARP_STAT_RCVLOCALSHA);
-		goto out;
-	}
-
 	rcvif = ifp = m_get_rcvif_psref(m, &psref);
 	if (__predict_false(rcvif == NULL))
 		goto out;
@@ -911,6 +903,12 @@
 	myaddr = ia->ia_addr.sin_addr;

 	/* XXX checks for bridge case? */
+	if (!memcmp(ar_sha(ah), CLLADDR(ifp->if_sadl), ifp->if_addrlen)) {
+		ARP_STATINC(ARP_STAT_RCVLOCALSHA);
+		goto out;	/* it's from me, ignore it. */
+	}
+
+	/* XXX checks for bridge case? */
 	if (!memcmp(ar_sha(ah), ifp->if_broadcastaddr, ifp->if_addrlen)) {
 		ARP_STATINC(ARP_STAT_RCVBCASTSHA);
 		log(LOG_ERR,
diff -r 05014b1ae511 sys/netinet6/nd6_nbr.c
--- a/sys/netinet6/nd6_nbr.c	Thu Sep 26 21:31:09 2024 +0000
+++ b/sys/netinet6/nd6_nbr.c	Thu Oct 03 13:33:45 2024 +0000
@@ -677,23 +677,21 @@
 	}

 	if (ndopts.nd_opts_tgt_lladdr != NULL) {
-		struct ifnet *ifp_ll;
-		struct psref psref_ll;
-
 		lladdr = (char *)(ndopts.nd_opts_tgt_lladdr + 1);
 		lladdrlen = ndopts.nd_opts_tgt_lladdr->nd_opt_len << 3;
+	}

-		if (lladdr && ((ifp->if_addrlen + 2 + 7) & ~7) != lladdrlen) {
+	if (lladdr != NULL) {
+		if (((ifp->if_addrlen + 2 + 7) & ~7) != lladdrlen) {
 			nd6log(LOG_INFO, "lladdrlen mismatch for %s "
-			    "(if %d, NA packet %d)\n", IN6_PRINT(ip6buf, &taddr6),
+			    "(if %d, NA packet %d)\n",
+			    IN6_PRINT(ip6buf, &taddr6),
 			    ifp->if_addrlen, lladdrlen - 2);
 			goto bad;
 		}

-		ifp_ll = if_get_bylla(lladdr, ifp->if_addrlen, &psref_ll);
-		if (ifp_ll != NULL) {
+		if (!memcmp(lladdr, CLLADDR(ifp->if_sadl), ifp->if_addrlen)) {
 			/* it's from me, ignore it. */
-			if_put(ifp_ll, &psref_ll);
 			goto freeit;
 		}
 	}

>Fix:
Yes, please!

>Release-Note:

>Audit-Trail:
From: Roy Marples <roy@marples.name>
To: "gnats-bugs" <gnats-bugs@NetBSD.org>
Cc: "campbell+netbsd" <campbell+netbsd@mumble.net>
Subject: Re: kern/58715: cpu0: softints stuck for 16 seconds
Date: Thu, 03 Oct 2024 15:07:06 +0100

 Some more repro data.

 The host has two tap(4) interfaces into qemu guest configured using virtio-net-pci which are in a bridge(4).
 The host runs ISC DHCPd which is the only DHCP server I know which allows you to configure the same IP address for two different hardware addresses.
 Configure ISC DHCPd to ensure that both virtual nics on the guest will get the same IP address in the same subnet.

 The NetBSD guest is configured just to start dhcpcd as a service.
 With the above kernel patch, we no longer allow interfaces we know but don't receive on to bypass DAD.

 Watch dhcpcd in the VM configure one but get stuck in the other in a DAD/DECLINE/DROP/DISCOVER/DAD/DECLINE loop and watch the host panic at this point.
 Takes a few seconds for dhcpcd to suddenly stop and then about 20 more for the kernel to actually panic.

 Roy


From: Roy Marples <roy@marples.name>
To: "gnats-bugs" <gnats-bugs@NetBSD.org>
Cc: 
Subject: Re: kern/58715: cpu0: softints stuck for 16 seconds
Date: Fri, 04 Oct 2024 14:12:30 +0100

 The QEMU command I use to startup is this:

 qemu-system-x86_64 -machine type=q35,accel=nvmm -drive file=netbsd.qcow2,if=none,id=disk2 -device virtio-blk-pci,drive=disk2 -object rng-random,id=rng2,filename=/dev/urandom -device virtio-rng-pci,rng=rng2 -netdev tap,ifname=tap1,id=net1,script=no,downscript=no -device virtio-net-pci,netdev=net1,mac=52:54:00:73:78:01 -netdev tap,ifname=tap2,id=net2,script=no,downscript=no -device virtio-net-pci,netdev=net2,mac=52:54:00:73:78:02 -pidfile /home/roy/qemu/run/qemu-netbsd.pid -daemonize -smp cpus=16 -m 16G -vnc :1


From: Roy Marples <roy@marples.name>
To: "gnats-bugs" <gnats-bugs@NetBSD.org>
Cc: 
Subject: Re: kern/58715: cpu0: softints stuck for 16 seconds
Date: Sat, 05 Oct 2024 10:13:38 +0100

 The kernel patches in the initial PR have now landed and this can now be easily reproduced in a stock kernel.

 Roy

>Unformatted:

Home	PR Database Search