NetBSD Problem Report #59745

From www@netbsd.org  Mon Nov  3 01:22:53 2025
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits)
	 client-signature RSA-PSS (2048 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id A90D11A9239
	for <gnats-bugs@gnats.NetBSD.org>; Mon,  3 Nov 2025 01:22:53 +0000 (UTC)
Message-Id: <20251103012252.565DD1A923C@mollari.NetBSD.org>
Date: Mon,  3 Nov 2025 01:22:52 +0000 (UTC)
From: campbell+netbsd@mumble.net
Reply-To: campbell+netbsd@mumble.net
To: gnats-bugs@NetBSD.org
Subject: guest network hosed, host says: xvif6i0 GNTTABOP_copy[0] Rx -3
X-Send-Pr-Version: www-1.0

>Number:         59745
>Category:       port-xen
>Synopsis:       guest network hosed, host says: xvif6i0 GNTTABOP_copy[0] Rx -3
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    port-xen-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Nov 03 01:25:00 +0000 2025
>Last-Modified:  Tue Nov 04 11:25:01 +0000 2025
>Originator:     Taylor R Campbell
>Release:        10.1_STABLE
>Organization:
The GrantBSD Foundabop
>Environment:
NetBSD cole.netbsd.de 10.1_STABLE NetBSD 10.1_STABLE (TNF_XENSERVER_PVH) #1: Sat Oct  4 22:20:13 UTC 2025  spz@franklin.NetBSD.org:/home/netbsd/10/amd64/obj/sys/arch/amd64/compile/TNF_XENSERVER_PVH amd64
>Description:

	In two incidents recently, we've had a guest on TNF
	infrastructure become unreachable over the network -- no
	response to pings or TCP connections, nothing logged in
	npflog0, system mainly idle.

	The host's dmesg is full of messages like this:

[ 2451339.802864] xvif6i0 GNTTABOP_copy[0] Rx -3
[ 2451339.802864] xvif6i0 GNTTABOP_copy[0] Rx -3
[ 2451339.802864] xvif6i0 GNTTABOP_copy[0] Rx -3
[ 2451339.802864] xvif6i0 GNTTABOP_copy[0] Rx -3
[ 2451339.802864] xvif6i0 GNTTABOP_copy[0] Rx -3
[ 2451339.802864] xvif6i0 GNTTABOP_copy[0] Rx -3

	That's this message:

    638 	/*
    639 	 * Copy the data and ack it. Delaying it until the mbuf is
    640 	 * freed will stall transmit.
    641 	 */
    642 	if (HYPERVISOR_grant_table_op(GNTTABOP_copy, gop, copycnt) != 0) {
    643 		printf("%s: GNTTABOP_copy %s failed", ifp->if_xname, dir);
    644 		return EINVAL;
    645 	}
    646 
    647 	for (int i = 0; i < copycnt; i++) {
    648 		if (gop->status != GNTST_okay) {
    649 			printf("%s GNTTABOP_copy[%d] %s %d\n",
    650 			    ifp->if_xname, i, dir, gop->status);
    651 			return EINVAL;
    652 		}
    653 	}

	https://nxr.NetBSD.org/xref/src/sys/arch/xen/xen/xennetback_xenbus.c?r=1.108.4.3#649

	Restarting the guest makes the problem go away, for a little
	while.

	Internal admins ticket references:
	https://rt.NetBSD.org/Ticket/Display.html?id=377840
	https://rt.NetBSD.org/Ticket/Display.html?id=372971

>How-To-Repeat:

	No idea, maybe run a network-oriented guest for a while.

>Fix:

	Yes, please!

>Audit-Trail:
From: Manuel Bouyer <bouyer@antioche.eu.org>
To: gnats-bugs@netbsd.org
Cc: port-xen-maintainer@netbsd.org, gnats-admin@netbsd.org,
        netbsd-bugs@netbsd.org
Subject: Re: port-xen/59745: guest network hosed, host says: xvif6i0
 GNTTABOP_copy[0] Rx -3
Date: Mon, 3 Nov 2025 11:31:44 +0100

 On Mon, Nov 03, 2025 at 01:25:00AM +0000, campbell+netbsd@mumble.net wrote:
 > >Number:         59745
 > >Category:       port-xen
 > >Synopsis:       guest network hosed, host says: xvif6i0 GNTTABOP_copy[0] Rx -3
 > >Confidential:   no
 > >Severity:       serious
 > >Priority:       medium
 > >Responsible:    port-xen-maintainer
 > >State:          open
 > >Class:          sw-bug
 > >Submitter-Id:   net
 > >Arrival-Date:   Mon Nov 03 01:25:00 +0000 2025
 > >Originator:     Taylor R Campbell
 > >Release:        10.1_STABLE
 > >Organization:
 > The GrantBSD Foundabop
 > >Environment:
 > NetBSD cole.netbsd.de 10.1_STABLE NetBSD 10.1_STABLE (TNF_XENSERVER_PVH) #1: Sat Oct  4 22:20:13 UTC 2025  spz@franklin.NetBSD.org:/home/netbsd/10/amd64/obj/sys/arch/amd64/compile/TNF_XENSERVER_PVH amd64
 > >Description:
 > 
 > 	In two incidents recently, we've had a guest on TNF
 > 	infrastructure become unreachable over the network -- no
 > 	response to pings or TCP connections, nothing logged in
 > 	npflog0, system mainly idle.
 > 
 > 	The host's dmesg is full of messages like this:
 > 
 > [ 2451339.802864] xvif6i0 GNTTABOP_copy[0] Rx -3
 > [ 2451339.802864] xvif6i0 GNTTABOP_copy[0] Rx -3
 > [ 2451339.802864] xvif6i0 GNTTABOP_copy[0] Rx -3
 > [ 2451339.802864] xvif6i0 GNTTABOP_copy[0] Rx -3
 > [ 2451339.802864] xvif6i0 GNTTABOP_copy[0] Rx -3
 > [ 2451339.802864] xvif6i0 GNTTABOP_copy[0] Rx -3
 > 
 > 	That's this message:
 > 
 >     638 	/*
 >     639 	 * Copy the data and ack it. Delaying it until the mbuf is
 >     640 	 * freed will stall transmit.
 >     641 	 */
 >     642 	if (HYPERVISOR_grant_table_op(GNTTABOP_copy, gop, copycnt) != 0) {
 >     643 		printf("%s: GNTTABOP_copy %s failed", ifp->if_xname, dir);
 >     644 		return EINVAL;
 >     645 	}

 Are you running a debug hypervisor (xen-debug.gz from the package) ?
 If so, does anything shows up in xl dmesg or on serial console ?

 Also you don't mention the xen version

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

From: Taylor R Campbell <riastradh@NetBSD.org>
To: Manuel Bouyer <bouyer@antioche.eu.org>
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: port-xen/59745: guest network hosed, host says: xvif6i0
	GNTTABOP_copy[0] Rx -3
Date: Tue, 4 Nov 2025 01:41:25 +0000

 > Date: Mon, 3 Nov 2025 11:31:44 +0100
 > From: Manuel Bouyer <bouyer@antioche.eu.org>
 > 
 > Are you running a debug hypervisor (xen-debug.gz from the package) ?

 Nope, xen.gz.

 > If so, does anything shows up in xl dmesg or on serial console ?

 Unsurprisingly, I don't see anything obviously relevant in xl dmesg.
 Not sure offhand how to get at the host's serial console.

 > Also you don't mention the xen version

 Happened on:

 4.20.2-pre_20250821nb0
 4.18.0_20231116nb0

From: Manuel Bouyer <bouyer@antioche.eu.org>
To: Taylor R Campbell <riastradh@NetBSD.org>
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: port-xen/59745: guest network hosed, host says: xvif6i0
 GNTTABOP_copy[0] Rx -3
Date: Tue, 4 Nov 2025 12:19:57 +0100

 On Tue, Nov 04, 2025 at 01:41:25AM +0000, Taylor R Campbell wrote:
 > > Date: Mon, 3 Nov 2025 11:31:44 +0100
 > > From: Manuel Bouyer <bouyer@antioche.eu.org>
 > > 
 > > Are you running a debug hypervisor (xen-debug.gz from the package) ?
 > 
 > Nope, xen.gz.
 > 
 > > If so, does anything shows up in xl dmesg or on serial console ?
 > 
 > Unsurprisingly, I don't see anything obviously relevant in xl dmesg.

 it would help to run xen-debug.gz then

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

Home	PR Database Search