NetBSD Problem Report #57694
From wiz@exadelic.gatalith.at Sun Nov 12 19:43:35 2023
Return-Path: <wiz@exadelic.gatalith.at>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 12BB41A9238
for <gnats-bugs@gnats.NetBSD.org>; Sun, 12 Nov 2023 19:43:35 +0000 (UTC)
Message-Id: <20231112194328.DE6C92EBBAD3@exadelic.gatalith.at>
Date: Sun, 12 Nov 2023 20:43:28 +0100 (CET)
From: Thomas Klausner <wiz@NetBSD.org>
Reply-To: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Subject: rge(4) hang
X-Send-Pr-Version: 3.95
>Number: 57694
>Category: kern
>Synopsis: rge(4) hang
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: kern-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sun Nov 12 19:45:00 +0000 2023
>Closed-Date:
>Last-Modified: Tue Nov 19 05:35:01 +0000 2024
>Originator: Thomas Klausner
>Release: NetBSD 10.99.10
>Organization:
>Environment:
Architecture: x86_64
Machine: amd64
>Description:
After the latest fixes, rge(4) is better, but it's completely hung up
the network interface twice so far - no network traffic possible on it
- both times so hard, that the BIOS had some kind of issue on the next
boot and needed 15 minutes to sort itself out (before even showing
anything on the screen).
I'm running a kernel from Oct 22.
That's in 1Gb mode.
dmesg:
rge0 at pci7 dev 0 function 0: Realtek Semiconductor 8125 10/100/1G/2.5G Ethernet (rev. 0x05)
rge0: interrupting at msix1 vec 0
rge0: Ethernet address xx.xx.xx.xx.xx.xx
In /var/log/messages I see:
Nov 1 18:59:43 exadelic dhcpcd[2191]: rge0: xxxx::1 is unreachable
Nov 1 18:59:43 exadelic dhcpcd[2191]: rge0: soliciting an IPv6 router
Nov 1 18:59:45 exadelic dhcpcd[2191]: rge0: Router Advertisement from xxxx::1 1 18:59:46 exadelic dhcpcd[2191]: rge0: xxxx::1 is unreachable
Nov 1 18:59:46 exadelic dhcpcd[2191]: rge0: soliciting an IPv6 router
Nov 1 18:59:58 exadelic dhcpcd[2191]: rge0: no IPv6 Routers available
Nov 1 19:01:11 exadelic dhcpcd[2191]: rge0: xxxx::1 is reachable again
Nov 1 19:01:19 exadelic dhcpcd[2191]: rge0: xxxx::1 is unreachable
Nov 1 19:01:19 exadelic dhcpcd[2191]: rge0: soliciting an IPv6 router
Nov 1 19:01:31 exadelic dhcpcd[2191]: rge0: no IPv6 Routers available
Nov 1 19:01:57 exadelic dhcpcd[2191]: rge0: xxxx::1 is reachable again
Nov 1 19:02:05 exadelic dhcpcd[2191]: rge0: xxxx::1 is unreachable
Nov 1 19:02:05 exadelic dhcpcd[2191]: rge0: soliciting an IPv6 router
Nov 1 19:02:17 exadelic dhcpcd[2191]: rge0: no IPv6 Routers available
Nov 1 19:04:27 exadelic dhcpcd[2191]: rge0: xxxx::1 is reachable again
Nov 1 19:04:35 exadelic dhcpcd[2191]: rge0: xxxx::1 is unreachable
Nov 1 19:04:35 exadelic dhcpcd[2191]: rge0: soliciting an IPv6 router
Nov 1 19:04:47 exadelic dhcpcd[2191]: rge0: no IPv6 Routers available
Nov 1 19:06:12 exadelic dhcpcd[2191]: rge0: xxxx::1 is reachable again
Nov 1 19:06:20 exadelic dhcpcd[2191]: rge0: xxxx::1 is unreachable
Nov 1 19:06:21 exadelic dhcpcd[2191]: rge0: soliciting an IPv6 router
Nov 1 19:06:33 exadelic dhcpcd[2191]: rge0: no IPv6 Routers available
Nov 1 19:09:27 exadelic /netbsd: [ 91537.5847758] nfs server 192.168.178.19:/path: not responding
Nov 1 19:15:16 exadelic dhcpcd[2191]: rge0: xxxx::1 is reachable again
Nov 1 19:15:24 exadelic dhcpcd[2191]: rge0: xxxx::1 is unreachable
Nov 1 19:15:24 exadelic dhcpcd[2191]: rge0: soliciting an IPv6 router
Nov 1 19:15:36 exadelic dhcpcd[2191]: rge0: no IPv6 Routers available
Nov 1 19:16:51 exadelic dhcpcd[2191]: rge0: xxxx::1 is reachable again
Nov 1 19:16:52 exadelic dhcpcd[2191]: ps_root_recvmsg: No buffer space available
Nov 1 19:16:52 exadelic dhcpcd[2191]: ps_root_recvmsg: No buffer space available
Nov 1 19:16:59 exadelic dhcpcd[2191]: rge0: xxxx::1 is unreachable
Nov 1 19:16:59 exadelic dhcpcd[2191]: rge0: soliciting an IPv6 router
Nov 1 19:16:59 exadelic dhcpcd[2191]: ps_root_recvmsg: No buffer space available
Nov 1 19:17:11 exadelic syslogd[2290]: last message repeated 3 times
Nov 1 19:17:11 exadelic dhcpcd[2191]: rge0: no IPv6 Routers available
Nov 1 19:17:44 exadelic dhcpcd[2191]: ps_root_recvmsg: No buffer space available
Just in case it matters, I'm not running with default sysctl's, I have
kern.sbmax: 262144 -> 16777216
net.inet.tcp.recvbuf_max: 262144 -> 16777216
net.inet.tcp.sendbuf_max: 262144 -> 16777216
net.inet.tcp.recvspace: 32768 -> 262144
net.inet.tcp.sendspace: 32768 -> 262144
because of
https://mail-index.netbsd.org/current-users/2017/09/21/msg032369.html
>How-To-Repeat:
Make heavy use of an rge(4) device with NFS + SFTP.
>Fix:
Yes, please.
>Release-Note:
>Audit-Trail:
From: Michael van Elst <mlelstv@serpens.de>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/57694: rge(4) hang
Date: Sat, 16 Dec 2023 13:38:25 +0100
The following patch helped here:
--- sys/dev/pci/if_rge.c 19 Oct 2023 23:43:40 -0000 1.28
+++ sys/dev/pci/if_rge.c 16 Dec 2023 12:33:04 -0000
@@ -1404,24 +1404,14 @@ rge_txeof(struct rge_softc *sc)
sc->rge_ldata.rge_txq_considx = cons;
-#if 0
- if (ifq_is_oactive(&ifp->if_snd))
- ifq_restart(&ifp->if_snd);
- else if (free == 2)
- ifq_serialize(&ifp->if_snd, &sc->sc_task);
- else
- ifp->if_timer = 0;
-#else
-#if 0
+ if (free == 2)
+ rge_txstart(&sc->sc_task, sc);
+
+ CLR(ifp->if_flags, IFF_OACTIVE);
if (!IF_IS_EMPTY(&ifp->if_snd))
rge_start(ifp);
else
- if (free == 2)
- if (0) { rge_txstart(&sc->sc_task, sc); }
- else
-#endif
ifp->if_timer = 0;
-#endif
return (1);
}
--
Michael van Elst
Internet: mlelstv@serpens.de
"A potential Snark may lurk in every tree."
From: "Michael van Elst" <mlelstv@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/57694 CVS commit: src/sys/dev/pci
Date: Sat, 16 Dec 2023 16:35:49 +0000
Module Name: src
Committed By: mlelstv
Date: Sat Dec 16 16:35:49 UTC 2023
Modified Files:
src/sys/dev/pci: if_rge.c
Log Message:
- handle stuck transmitter (descriptor still owned)
- restart send queue after transmit
- count output packets
- use deferred start
Should fix PR 57694
To generate a diff of this commit:
cvs rdiff -u -r1.28 -r1.29 src/sys/dev/pci/if_rge.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->closed
State-Changed-By: wiz@NetBSD.org
State-Changed-When: Sun, 31 Dec 2023 10:03:34 +0000
State-Changed-Why:
mlelstv's commit improved the situation a lot - I saw no more hangs
running the new kernel for over a week.
From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/57694: rge(4) hang
Date: Sun, 31 Dec 2023 11:02:24 +0100
After running with rge(4) with mlelstv's change for more than a week,
there were no problems. So this has improved stability a lot.
Thank you, Michael!
Thomas
State-Changed-From-To: closed->open
State-Changed-By: wiz@NetBSD.org
State-Changed-When: Tue, 02 Jan 2024 16:03:24 +0000
State-Changed-Why:
Not solved yet.
From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@netbsd.org
Cc: Michael van Elst <mlelstv@serpens.de>
Subject: Re: kern/57694 (rge(4) hang)
Date: Tue, 2 Jan 2024 17:08:45 +0100
Today the machine's network hung again, after about 10 days of uptime.
In syslog I saw:
Jan 2 16:34:27 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:34:27 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:34:31 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:34:39 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:34:39 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:34:42 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:34:47 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:34:50 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:34:51 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:34:51 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:34:53 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:34:55 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:34:55 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:35:01 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:35:02 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:35:02 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:35:05 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:35:06 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:35:06 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:35:13 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:35:14 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:35:14 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:35:17 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:35:18 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:35:18 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:35:25 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:35:26 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:35:26 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:35:29 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:35:30 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:35:30 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:35:37 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:35:37 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:35:37 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:35:41 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:35:41 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:35:41 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:35:49 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:35:49 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:35:49 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:35:49 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:35:53 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:35:57 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:35:57 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:36:01 exadelic dhcpcd[1790]: rge0: no IPv6 Routers available
Jan 2 16:36:01 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:36:02 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:36:02 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:36:04 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:36:06 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:36:06 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:36:12 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:36:13 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:36:13 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:36:16 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:36:17 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:36:17 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:36:24 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:36:24 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:36:24 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:36:24 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:36:30 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:36:32 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:36:32 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:36:36 exadelic dhcpcd[1790]: rge0: no IPv6 Routers available
Jan 2 16:36:38 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:36:39 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:36:39 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:36:51 exadelic syslogd[2503]: last message repeated 3 times
Jan 2 16:36:51 exadelic dhcpcd[1790]: rge0: no IPv6 Routers available
Jan 2 16:36:54 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:37:02 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:37:03 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:37:03 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:37:05 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:37:07 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:37:07 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:37:13 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:37:13 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:37:13 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:37:16 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:37:17 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:37:17 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:37:24 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:37:25 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:37:25 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:37:25 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:37:30 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:37:33 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:37:33 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:37:37 exadelic dhcpcd[1790]: rge0: no IPv6 Routers available
Jan 2 16:37:38 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:37:39 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:37:39 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:37:42 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:37:43 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:37:43 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:37:50 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:37:50 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:37:50 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:38:02 exadelic syslogd[2503]: last message repeated 3 times
Jan 2 16:38:02 exadelic dhcpcd[1790]: rge0: no IPv6 Routers available
Jan 2 16:38:15 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:38:15 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:38:23 exadelic syslogd[2503]: last message repeated 2 times
Jan 2 16:38:23 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:38:23 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:38:23 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:38:35 exadelic syslogd[2503]: last message repeated 3 times
Jan 2 16:38:35 exadelic dhcpcd[1790]: rge0: no IPv6 Routers available
Jan 2 16:38:45 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:38:53 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:38:53 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:38:53 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:39:05 exadelic syslogd[2503]: last message repeated 3 times
Jan 2 16:39:05 exadelic dhcpcd[1790]: rge0: no IPv6 Routers available
Jan 2 16:39:39 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:39:47 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:39:47 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:39:47 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:39:50 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:39:51 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:39:51 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:39:58 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:39:58 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:39:58 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:39:58 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:40:03 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:40:06 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:40:06 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:40:10 exadelic dhcpcd[1790]: rge0: no IPv6 Routers available
Jan 2 16:40:11 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:40:12 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:40:12 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:40:12 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:40:19 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is reachable again
Jan 2 16:40:20 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:40:20 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
Jan 2 16:40:24 exadelic dhcpcd[1790]: rge0: no IPv6 Routers available
Jan 2 16:40:27 exadelic dhcpcd[1790]: rge0: fe80::de15:c8ff:fe36:2e98 is unreachable
Jan 2 16:40:28 exadelic dhcpcd[1790]: rge0: soliciting an IPv6 router
Jan 2 16:40:28 exadelic dhcpcd[1790]: ps_root_recvmsg: No buffer space available
This was during a high load situation (bulk build). The machine
stopped being accessible from the network.
Thomas
From: matthew green <mrg@eterna23.net>
To: gnats-bugs@netbsd.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org
Subject: re: kern/57694: rge(4) hang
Date: Sun, 11 Aug 2024 05:32:22 +1000
i saw this problem last night, when doing fairly light NFS work (
about 2MB/s reads.)
i noticed i started trying to merge recent fixes from openbsd into
if_rge.c late last year but didn't finish. i'll look again.
.mrg.
From: matthew green <mrg@eterna23.net>
To: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, wiz@netbsd.org,
netbsd-bugs@netbsd.org, gnats-bugs@netbsd.org
Cc:
Subject: re: kern/57694: rge(4) hang
Date: Sun, 11 Aug 2024 08:13:46 +1000
matthew green writes:
> i saw this problem last night, when doing fairly light NFS work (
> about 2MB/s reads.)
FWIW, "ifconfig rge0 down up" made it come back, so this seems like
a bug in the descriptor ring handling, as that will all be reset by
the down/up sequence.
i had a look at the latest openbsd changes and i didn't spot anything
that seemed relevant for this problem.
.mrg.
From: matthew green <mrg@eterna23.net>
To: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, gnats-bugs@netbsd.org
Cc:
Subject: re: kern/57694: rge(4) hang
Date: Sun, 11 Aug 2024 09:23:24 +1000
ah - i just noticed that my system is has an older netbsd-10
kernel, that does not have revision 1.24.4.3 included, but
also that netbsd-10 itself is missing rev 1.29, which fixed
some hangs for mlelstv and made this problem less bad for wiz.
so that may be all i am seeing..
.mrg.
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/57694 CVS commit: [netbsd-10] src/sys/dev/pci
Date: Thu, 22 Aug 2024 19:22:35 +0000
Module Name: src
Committed By: martin
Date: Thu Aug 22 19:22:35 UTC 2024
Modified Files:
src/sys/dev/pci [netbsd-10]: if_rge.c
Log Message:
Pull up following revision(s) (requested by mrg in ticket #782):
sys/dev/pci/if_rge.c: revision 1.29
- handle stuck transmitter (descriptor still owned)
- restart send queue after transmit
- count output packets
- use deferred start
Should fix PR 57694
To generate a diff of this commit:
cvs rdiff -u -r1.24.4.3 -r1.24.4.4 src/sys/dev/pci/if_rge.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: matthew green <mrg@eterna23.net>
To: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, gnats-bugs@netbsd.org
Cc:
Subject: re: kern/57694: rge(4) hang
Date: Fri, 23 Aug 2024 11:44:15 +1000
matthew green writes:
> ah - i just noticed that my system is has an older netbsd-10
> kernel, that does not have revision 1.24.4.3 included, but
> also that netbsd-10 itself is missing rev 1.29, which fixed
> some hangs for mlelstv and made this problem less bad for wiz.
>
> so that may be all i am seeing..
unfortunately, i did see one more soft-hang that was fixed
with 'ifconfig rge0 down up', but haven't seen it for over
12 more days, and my attempts to trigger failure with the
same card in a zen2 testbox (the host i have seen hangs on
is zen1+) have not gained any insight.
netbsd-10 has the latest fixes now at least, thanks martin.
.mrg.
From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/57694 (rge(4) hang)
Date: Mon, 7 Oct 2024 23:40:24 +0200
Ok, the plot thickens.
Some months ago I switched to using an igc0 instead:
igc0 at pci6 dev 0 function 0, 64-bit DMA: Intel(R) Ethernet Controller I225-LM (rev. 0x03)
igc0: txrx interrupting at msix1 vec 0
igc0: txrx interrupting at msix1 vec 1
igc0: txrx interrupting at msix1 vec 2
igc0: txrx interrupting at msix1 vec 3
igc0: link interrupting at msix1 vec 4
igc0: PHY: OUI 0x984fee model 0x0000, rev. 0, PHY FW version 0x8754
igc0: NVM image version 1.57, EtrackID 80000180
igc0: Ethernet address ...
today the machine stopped being accessible from the network, and in the log I saw
Oct 7 23:03:39 exadelic dhcpcd[2783]: ps_root_recvmsg: No buffer space available
which looks like the symptoms I saw with rge0.
Could there be a common underlying cause in the network code?
Thomas
From: matthew green <mrg@eterna23.net>
To: gnats-bugs@netbsd.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, Thomas Klausner <wiz@NetBSD.org>
Subject: re: kern/57694 (rge(4) hang)
Date: Tue, 08 Oct 2024 16:15:26 +1100
"Thomas Klausner via gnats" writes:
> Could there be a common underlying cause in the network code?
i wonder. is there something in your setup that is triggering
it because i've spent weeks sending packets between rge(4) and
cdce(4) in 2.5T mode (realtek ethernet), without seeing issues
that aren't already fixed.
most of my kernels don't have INET6 is perhaps the first thing
i can think of...
.mrg.
From: Thomas Klausner <wiz@NetBSD.org>
To: matthew green <mrg@eterna23.net>
Cc: NetBSD bugtracking <gnats-bugs@NetBSD.org>
Subject: Re: kern/57694 (rge(4) hang)
Date: Tue, 8 Oct 2024 11:22:36 +0200
On Tue, Oct 08, 2024 at 04:15:26PM +1100, Matthew Green wrote:
> i wonder. is there something in your setup that is triggering
> it because i've spent weeks sending packets between rge(4) and
> cdce(4) in 2.5T mode (realtek ethernet), without seeing issues
> that aren't already fixed.
>
> most of my kernels don't have INET6 is perhaps the first thing
> i can think of...
Yes, I'm using IPv6 - I don't have a public IPv4 so much of the
Internet traffic goes that way.
Internally, I'm using a lot of NFS but that's via IPv4.
Not sure what else is special in my setup.
Thomas
From: Rin Okuyama <rokuyama.rk@gmail.com>
To: gnats-bugs@netbsd.org, kern-bug-people@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
Thomas Klausner <wiz@NetBSD.org>, matthew green <mrg@NetBSD.org>
Cc:
Subject: Re: kern/57694 (rge(4) hang)
Date: Wed, 9 Oct 2024 14:18:34 +0900
On 2024/10/08 14:20, matthew green via gnats wrote:
> i wonder. is there something in your setup that is triggering
> it because i've spent weeks sending packets between rge(4) and
> cdce(4) in 2.5T mode (realtek ethernet), without seeing issues
> that aren't already fixed.
>
> most of my kernels don't have INET6 is perhaps the first thing
> i can think of...
Similar for me. One of my main machines is running with igc(4) over
a year without problems, but IPv6 is not used (mostly).
For igc(4), some firmware versions are buggy, and update
https://www.intel.com/content/www/us/en/download/15084/intel-ethernet-adapter-complete-driver-pack.html
may help (as per suggestion by msaitoh@).
Thanks,
rin
From: matthew green <mrg@eterna23.net>
To: Rin Okuyama <rokuyama.rk@gmail.com>
Cc: gnats-bugs@netbsd.org, kern-bug-people@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
Thomas Klausner <wiz@NetBSD.org>, tech-net@netbsd.org
Subject: re: kern/57694 (rge(4) hang)
Date: Sat, 09 Nov 2024 06:16:08 +1100
one of my rge(4)'s has soft-hung twice recently. after a power
outage, it lost connectivity soon after coming back online (on
the order of 10s of minutes less than a hour), and ifconfig =
down/up sequence restored connectivity. then, about 2 days
later, the same thing happened.
i wrote a stupid script to notice and the down/up for me.
i also started looking at rge_rxeof(), and i wonder if there's
a bad case we get into and can't get out of without manually
resetting the descriptors (eg, the down/up sequence):
1249 for (i =3D sc->rge_ldata.rge_rxq_considx; ; i =3D RGE_NEXT=
_RX_DESC(i)) {
...
1255 cur_rx =3D &sc->rge_ldata.rge_rx_list[i];
1256
1257 if (RGE_OWN(cur_rx))
1258 break;
...
1361 }
1362
1363 sc->rge_ldata.rge_rxq_considx =3D i;
if for some reason rge_rxq_considx ends up pointing to a desc
that is owned, then i is never changed from rge_rxq_considx and
the assignment on L1363 is a no-op. could it be being not
filled in _ever_ for some other reason , this loop will never
find other filled in descriptors. can that happen? i am not
nearly familiar enough with ethernet drivers/hardware...
i guess it might be useful to get a dump of all the descriptors
at the hang-time, to see their status.
i'll see about doing that, but this is the only machine i've
seen this hang on and i'm loathe to reboot it ever as it does
not reboot without power cycle, and recently, it took about 5
power cycles for it to get past whatever problem it has.
.mrg.
From: mlelstv@serpens.de (Michael van Elst)
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/57694 (rge(4) hang)
Date: Sat, 9 Nov 2024 22:14:28 -0000 (UTC)
gnats-admin@NetBSD.org ("matthew green via gnats") writes:
> 1249 for (i =3D sc->rge_ldata.rge_rxq_considx; ; i =3D RGE_NEXT=
> _RX_DESC(i)) {
> ...
> 1255 cur_rx =3D &sc->rge_ldata.rge_rx_list[i];
> 1256
> 1257 if (RGE_OWN(cur_rx))
> 1258 break;
> ...
> 1361 }
> 1362
> 1363 sc->rge_ldata.rge_rxq_considx =3D i;
>
> if for some reason rge_rxq_considx ends up pointing to a desc
> that is owned, then i is never changed from rge_rxq_considx and
> the assignment on L1363 is a no-op.
This is a ring buffer and all descriptors are processed until
the loop hits a descriptor that is still 'owned' by the device,
because no data had been received.
rge_rxq_considx is updated correctly to the descriptor that
needs to be checked when the next interrupt arrives.
Maybe an interrupt is dropped and when you don't fetch
all data, there might be a condition that no further
interrupt is raised.
Or maybe the chip firmware 'forgets' to clear the OWN bit
under rare conditions. I've seen similar bugs before.
From: matthew green <mrg@eterna23.net>
To: gnats-bugs@netbsd.org, kern-bug-people@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
Thomas Klausner <wiz@NetBSD.org>, tech-net@netbsd.org,
Rin Okuyama <rokuyama.rk@gmail.com>
Cc:
Subject: re: kern/57694 (rge(4) hang)
Date: Tue, 19 Nov 2024 16:31:19 +1100
> i guess it might be useful to get a dump of all the descriptors
> at the hang-time, to see their status.
>
> i'll see about doing that, but this is the only machine i've
> seen this hang on and i'm loathe to reboot it ever as it does
> not reboot without power cycle, and recently, it took about 5
> power cycles for it to get past whatever problem it has.
i recently patched this system's kernel and rebooted it, it only
took 6 or 7 power cycles to work this time (i really really need
to replace this system), and then another power outage lead it
two dumps of the rx descriptors when it has hung.
both times the rge(4) hung about 1h after booting.
unfortunately, AFAICT, all 1024 are currently "owned by the
hardware", and appear to be ready to go, both times. ie, there
is no indication what is wrong here at all.
i don't know enough about how this sort of h/w works to really
judge what is happening here.
.mrg.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.49 2026/05/14 01:52:41 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home