NetBSD Problem Report #54645

From gson@gson.org  Wed Oct 23 16:23:52 2019
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id B294C7A165
	for <gnats-bugs@gnats.NetBSD.org>; Wed, 23 Oct 2019 16:23:52 +0000 (UTC)
Message-Id: <20191023162346.D10B9253F40@guava.gson.org>
Date: Wed, 23 Oct 2019 19:23:46 +0300 (EEST)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@NetBSD.org
Subject: bnx(4) direct cable connection is unreliable
X-Send-Pr-Version: 3.95

>Number:         54645
>Category:       kern
>Synopsis:       Netbooting over a direct cable connection is unreliable
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Oct 23 16:25:00 +0000 2019
>Last-Modified:  Mon Jul 20 18:42:21 +0000 2020
>Originator:     Andreas Gustafsson
>Release:        NetBSD 9.0_BETA
>Organization:

>Environment:
System: NetBSD
Architecture: x86_64
Machine: amd64
>Description:

I have an automated testing setup that netboots an INSTALL kernel and
performs a scripted installation of NetBSD/amd64 on physical hardware.
It consists of two HP DL360 G7 machines, one acting as the netboot
server and the other as the client.  Both have quad bnx(4) gigabit
Ethernet interfaces.

To avoid any risk of accidentally auto-installing on the wrong
machine, the network connecting the server and client is physically
separate and used to simply consist of a short patch cable directly
connecting a port on the server to a port on the client (not even
using a crossover cable, but rather relying on "Auto MDI-X").

This worked fairly reliably until recently, when a significant
fraction of the test runs started failing.  This may or may not have
been triggered by upgrading the server from NetBSD 8 to 9.0_BETA.

Examining the network traffic using tcpdump, it looks like the server
is receiving packets from the client at about the expected times, but
not with the expected contents; rather, the packets received look like
duplicates of packets the server has received earlier.  For example,
in one case, the first packet received by the server after the client
was powered on was not the expected BOOTP packet, but a TCP packet to
port 80 (without a SYN flag, so from an existing connection).

Since the client can't actually have sent that immediately after
power-on, I suspect this is a server-side issue where the server
somehow gets confused after losing and regaining carrier on the
interface, causing it to inject packets from the wrong part of its
receive ring buffer into the network stack.

That the problem is triggered by a temporary loss of carrier is
supported by the fact that the problem has not recurred after
I added an Ethernet switch between the two machines.

I can make pcap files available on request.

>How-To-Repeat:

>Fix:

>Release-Note:

>Audit-Trail:
From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/54645: Netbooting over a direct cable connection is unreliable
Date: Fri, 20 Mar 2020 11:05:58 +0200

 After replacing the client machine by a Dell PowerEdge R630 and the
 straight-through four-pair cable with a two-pair crossover cable,
 the issue remains, which seems consistent with my hypothesis 
 that the server side is at fault.

 Pinging the client shows sporadic duplicate packets, and most of them
 are duplicates of the 8th or 16th previous ping response, which seems
 consistent with my hypothesis of some kind of receive ring corruption
 in the server's bnx(4) driver.

 $ ping 10.169.1.2
 PING 10.169.1.2 (10.169.1.2): 56 data bytes
 64 bytes from 10.169.1.2: icmp_seq=0 ttl=254 time=0.811067 ms
 64 bytes from 10.169.1.2: icmp_seq=1 ttl=254 time=0.720762 ms
 [...some uneventful packets omitted...]
 64 bytes from 10.169.1.2: icmp_seq=42 ttl=254 time=0.685701 ms
 64 bytes from 10.169.1.2: icmp_seq=43 ttl=254 time=0.685701 ms
 64 bytes from 10.169.1.2: icmp_seq=44 ttl=254 time=0.655042 ms
 64 bytes from 10.169.1.2: icmp_seq=45 ttl=254 time=0.686330 ms
 64 bytes from 10.169.1.2: icmp_seq=46 ttl=254 time=0.686121 ms
 64 bytes from 10.169.1.2: icmp_seq=47 ttl=254 time=0.686610 ms
 64 bytes from 10.169.1.2: icmp_seq=48 ttl=254 time=0.674597 ms
 64 bytes from 10.169.1.2: icmp_seq=49 ttl=254 time=0.672781 ms
 64 bytes from 10.169.1.2: icmp_seq=50 ttl=254 time=0.774330 ms
 64 bytes from 10.169.1.2: icmp_seq=51 ttl=254 time=0.686959 ms
 64 bytes from 10.169.1.2: icmp_seq=44 DUP! ttl=254 time=8000.432877 ms
 64 bytes from 10.169.1.2: icmp_seq=53 ttl=254 time=0.687588 ms
 64 bytes from 10.169.1.2: icmp_seq=54 ttl=254 time=0.687098 ms
 64 bytes from 10.169.1.2: icmp_seq=55 ttl=254 time=0.732495 ms
 64 bytes from 10.169.1.2: icmp_seq=56 ttl=254 time=0.686610 ms
 64 bytes from 10.169.1.2: icmp_seq=57 ttl=254 time=0.759314 ms
 64 bytes from 10.169.1.2: icmp_seq=58 ttl=254 time=0.771327 ms
 64 bytes from 10.169.1.2: icmp_seq=59 ttl=254 time=0.715105 ms
 64 bytes from 10.169.1.2: icmp_seq=44 DUP! ttl=254 time=16000.308070 ms
 64 bytes from 10.169.1.2: icmp_seq=61 ttl=254 time=0.682769 ms
 64 bytes from 10.169.1.2: icmp_seq=62 ttl=254 time=0.719156 ms
 64 bytes from 10.169.1.2: icmp_seq=63 ttl=254 time=0.687238 ms
 64 bytes from 10.169.1.2: icmp_seq=64 ttl=254 time=0.672222 ms
 64 bytes from 10.169.1.2: icmp_seq=65 ttl=254 time=0.682838 ms
 64 bytes from 10.169.1.2: icmp_seq=66 ttl=254 time=0.745486 ms
 64 bytes from 10.169.1.2: icmp_seq=67 ttl=254 time=0.663702 ms
 64 bytes from 10.169.1.2: icmp_seq=68 ttl=254 time=0.683885 ms
 64 bytes from 10.169.1.2: icmp_seq=69 ttl=254 time=0.686120 ms
 64 bytes from 10.169.1.2: icmp_seq=70 ttl=254 time=0.687238 ms
 64 bytes from 10.169.1.2: icmp_seq=71 ttl=254 time=0.686051 ms
 64 bytes from 10.169.1.2: icmp_seq=72 ttl=254 time=0.685702 ms
 64 bytes from 10.169.1.2: icmp_seq=73 ttl=254 time=0.730539 ms
 64 bytes from 10.169.1.2: icmp_seq=74 ttl=254 time=0.686121 ms
 64 bytes from 10.169.1.2: icmp_seq=75 ttl=254 time=0.817423 ms
 64 bytes from 10.169.1.2: icmp_seq=76 ttl=254 time=0.770489 ms
 64 bytes from 10.169.1.2: icmp_seq=77 ttl=254 time=0.686330 ms
 64 bytes from 10.169.1.2: icmp_seq=78 ttl=254 time=0.731796 ms
 64 bytes from 10.169.1.2: icmp_seq=79 ttl=254 time=0.686540 ms
 64 bytes from 10.169.1.2: icmp_seq=80 ttl=254 time=0.686540 ms
 64 bytes from 10.169.1.2: icmp_seq=81 ttl=254 time=0.686680 ms
 64 bytes from 10.169.1.2: icmp_seq=74 DUP! ttl=254 time=8000.666077 ms
 64 bytes from 10.169.1.2: icmp_seq=83 ttl=254 time=0.687028 ms
 64 bytes from 10.169.1.2: icmp_seq=84 ttl=254 time=0.731518 ms
 64 bytes from 10.169.1.2: icmp_seq=85 ttl=254 time=0.539524 ms
 64 bytes from 10.169.1.2: icmp_seq=86 ttl=254 time=0.532470 ms
 ^C

 -- 
 Andreas Gustafsson, gson@gson.org

From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@NetBSD.org
Cc: jdolecek@NetBSD.org
Subject: Re: kern/54645: Netbooting over a direct cable connection is unreliable
Date: Sat, 18 Jul 2020 19:38:22 +0300

 I have now found a way to reproduce the issue using a setup that does
 not involve netbooting, which should make it easier for others to
 reproduce.  This was with a -current system of source date
 2020.07.17.21.04.14.

 Here's how to do it:

 You need two systems, the system under test (or "sut" for short) with
 at least one bnx network interface (let's call it bnx3) and a second
 system, the traffic generator (or "tg" for short) with a network
 interface of any supported type (let's call it "bge1").  The two
 interfaces are connected by a patch cable (no switch).

 On sut, run this script (replace bnx3 by the actual interface
 name if needed):

 sysctl -w net.inet.ip.dad_count=0
 ifconfig bnx3 10.89.0.1 netmask 255.255.255.0

 On tg, run this script (replace bge1 with the actual interface
 name if needed):

 sysctl -w net.inet.ip.dad_count=0
 ifconfig bge1 10.89.0.2 netmask 255.255.255.0
 while true
 do
   ifconfig bge1 down
   sleep 1
   ifconfig bge1 up
   ping -i 0.1 -c 100 10.89.0.1
 done

 This will run ping repeatedly.  Normally, each invocation of ping will
 report a packet loss of some 30-40% as packets are lost while the
 network interface is being configured "up", but if you are suffering
 from the bug, the packet loss will eventually jump to 100%.  This can
 take several hours.

 The scripts disable DAD because it slows things down, and to eliminate
 DAD as a possible cause of the bug.
 -- 
 Andreas Gustafsson, gson@gson.org

>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.46 2020/01/03 16:35:01 leot Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2020 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.