NetBSD Problem Report #51531
From gson@gson.org Wed Oct 5 13:59:52 2016
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
by mollari.NetBSD.org (Postfix) with ESMTPS id 375B87A281
for <gnats-bugs@gnats.NetBSD.org>; Wed, 5 Oct 2016 13:59:52 +0000 (UTC)
Message-Id: <20161005135943.0FDE37449CD@guava.gson.org>
Date: Wed, 5 Oct 2016 16:59:43 +0300 (EEST)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@NetBSD.org
Subject: Recent networking regression affecting installs
X-Send-Pr-Version: 3.95
>Number: 51531
>Category: kern
>Synopsis: Recent networking regression affecting installs
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: kern-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Wed Oct 05 14:00:00 +0000 2016
>Last-Modified: Wed Sep 29 21:35:01 +0000 2021
>Originator: Andreas Gustafsson
>Release: NetBSD-current, source date >= 2016.09.16.23.01.53
>Organization:
>Environment:
System: NetBSD
Architecture: i386
Machine: i386
>Description:
I have an automated testing setup that netboots an INSTALL kernel and
performs a scripted installation of NetBSD/i386 or /amd64 on physical
hardware using sysinst.
This stopped working on both i386 and amd64 some time during the
period of build breakage lasting from 2016.09.16.11.35.07 to
2016.09.16.23.01.53, and is still broken as of 2016.10.05.03.46.38.
The ping test performed by sysinst fails with "100.0% packet loss, and
then the HTTP download of the first installation set fails:
ftp: Can't connect to `10.169.0.1:80': Operation timed out
A log from the most recent installation attempt is here, but a bit
hard to read because it contains terminal escape sequences from
sysinst (with the ESC characters stripped):
http://www.gson.org/netbsd/bugs/build/i386-baremetal/2016/2016.10.05.03.46.38/install.log
The problem started with one of the following commits:
2016.09.16.11.35.07 jdolecek src/sys/dev/pci/nvme_pci.c 1.6
2016.09.16.11.35.07 jdolecek src/sys/modules/Makefile 1.177
2016.09.16.11.35.07 jdolecek src/sys/modules/nvme/Makefile 1.1
2016.09.16.11.35.07 jdolecek src/sys/modules/nvme/nvme.ioconf 1.1
2016.09.16.11.41.40 jdolecek src/sys/dev/ic/nvme.c 1.6
2016.09.16.11.48.10 maxv src/sys/arch/amd64/amd64/trap.c 1.85
2016.09.16.11.48.10 maxv src/sys/arch/i386/i386/trap.c 1.279
2016.09.16.12.28.41 maxv src/sys/arch/i386/i386/copy.S 1.25
2016.09.16.12.31.27 jdolecek src/share/man/man4/nvme.4 1.3
2016.09.16.12.43.37 wiz src/share/man/man4/nvme.4 1.4
2016.09.16.12.57.26 jdolecek src/sys/dev/ic/nvme.c 1.7
2016.09.16.13.47.47 roy src/sys/netinet/if_arp.c 1.226
2016.09.16.13.56.36 jdolecek src/share/man/man9/pci_msi.9 1.13
2016.09.16.14.17.23 roy src/sys/net/if_spppsubr.c 1.152
2016.09.16.14.17.23 roy src/sys/netinet/in.c 1.182
2016.09.16.14.17.23 roy src/sys/netinet/in_var.h 1.83
2016.09.16.14.55.28 jdolecek src/doc/roadmaps/storage 1.16
2016.09.16.15.02.23 jdolecek src/doc/roadmaps/storage 1.17
2016.09.16.15.20.50 jdolecek src/sys/dev/ata/ld_ataraid.c 1.42
2016.09.16.15.20.50 jdolecek src/sys/dev/i2o/ld_iop.c 1.36
2016.09.16.15.20.50 jdolecek src/sys/dev/ic/ld_aac.c 1.29
2016.09.16.15.20.50 jdolecek src/sys/dev/ic/ld_cac.c 1.29
2016.09.16.15.20.50 jdolecek src/sys/dev/ic/ld_icp.c 1.28
2016.09.16.15.20.50 jdolecek src/sys/dev/ic/ld_mlx.c 1.22
2016.09.16.15.20.50 jdolecek src/sys/dev/ic/ld_nvme.c 1.2
2016.09.16.15.20.50 jdolecek src/sys/dev/ld.c 1.95
2016.09.16.15.20.50 jdolecek src/sys/dev/ldvar.h 1.28
2016.09.16.15.20.50 jdolecek src/sys/dev/pci/ld_amr.c 1.24
2016.09.16.15.20.50 jdolecek src/sys/dev/pci/ld_twa.c 1.18
2016.09.16.15.20.50 jdolecek src/sys/dev/pci/ld_twe.c 1.38
2016.09.16.15.20.50 jdolecek src/sys/dev/pci/ld_virtio.c 1.11
2016.09.16.15.20.50 jdolecek src/sys/dev/sdmmc/ld_sdmmc.c 1.22
2016.09.16.15.24.47 jdolecek src/sys/dev/ic/ld_nvme.c 1.3
2016.09.16.17.12.06 christos src/lib/libc/time/Makefile 1.33
2016.09.16.17.12.06 christos src/lib/libc/time/NEWS 1.16
2016.09.16.17.12.06 christos src/lib/libc/time/Theory 1.20
2016.09.16.17.12.06 christos src/lib/libc/time/tz-art.htm 1.13
2016.09.16.17.12.06 christos src/lib/libc/time/tz-how-to.html 1.1
2016.09.16.17.12.06 christos src/lib/libc/time/tz-link.htm 1.26
2016.09.16.17.12.06 christos src/lib/libc/time/zic.c 1.59
2016.09.16.17.13.06 christos src/doc/3RDPARTY 1.1361
2016.09.16.17.13.06 christos src/doc/CHANGES 1.2185
2016.09.16.17.27.09 matt src/sys/uvm/pmap/pmap.c 1.22
2016.09.16.17.32.36 scole src/sys/arch/macppc/dev/platinumfb.c 1.3
2016.09.16.20.30.14 christos src/external/gpl3/gcc/dist/gcc/config/m68k/m68k.md 1.5
2016.09.16.20.31.00 christos src/external/gpl3/gcc/dist/gcc/config/m68k/m68k.c 1.2
2016.09.16.22.39.35 macallan src/sys/arch/sparc/dev/cgfourteen.c 1.82
2016.09.16.23.01.53 pgoyette src/distrib/sets/lists/modules/mi 1.95
Those by roy are the only ones that look networking related to me.
>How-To-Repeat:
Attempt a network install of -current.
>Fix:
>Release-Note:
>Audit-Trail:
From: Roy Marples <roy@NetBSD.org>
To: Andreas Gustafsson <gson@gson.org>, gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Wed, 5 Oct 2016 15:11:53 +0100
Hi Andreas
On 05/10/2016 15:03, Andreas Gustafsson wrote:
> Could you please have a look at PR kern/51531, since it looks like you
> were the only one to touch the networking code during the period when
> the problem appeared?
I think ifconfig -w 15 -W 5 should be added to sysinst before the ping
test to wait for DaD to clear.
Roy
From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org,
gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org,
gson@gson.org (Andreas Gustafsson)
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Wed, 5 Oct 2016 17:42:18 +0300
Roy Marples wrote:
> I think ifconfig -w 15 -W 5 should be added to sysinst before the ping
> test to wait for DaD to clear.
I think sysinst, and other existing code that configures IP addresses,
is not incorrect as written, and should not be required to do this.
--
Andreas Gustafsson, gson@gson.org
From: Joerg Sonnenberger <joerg@bec.de>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, Andreas Gustafsson <gson@gson.org>
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Wed, 5 Oct 2016 17:07:34 +0200
On Wed, Oct 05, 2016 at 02:45:01PM +0000, Andreas Gustafsson wrote:
> The following reply was made to PR kern/51531; it has been noted by GNATS.
>
> From: Andreas Gustafsson <gson@gson.org>
> To: gnats-bugs@NetBSD.org
> Cc: kern-bug-people@netbsd.org,
> gnats-admin@netbsd.org,
> netbsd-bugs@netbsd.org,
> gson@gson.org (Andreas Gustafsson)
> Subject: Re: kern/51531: Recent networking regression affecting installs
> Date: Wed, 5 Oct 2016 17:42:18 +0300
>
> Roy Marples wrote:
> > I think ifconfig -w 15 -W 5 should be added to sysinst before the ping
> > test to wait for DaD to clear.
>
> I think sysinst, and other existing code that configures IP addresses,
> is not incorrect as written, and should not be required to do this.
I disagree, quite a bit. There are three options for such a program:
(1) It is dumb and just expects a newly configured interface to work
after some time. Nothing new here.
(2) It tries to be smart and checks immediately after configuration if
ping works. This has been wrong for a variety of network conditions
already, since link configuration itself can already take time in the
second range. WiFi can be even worse with slow authorisation. This
really is an application bug and not related to this issue at all.
(3) It tries to be even smarter and checks for link state changes before
ping. This still doesn't work properly for WiFi or 802.1x ports. DaD
timeouts are no different here.
In short, *any* assumption that a freshly configured network interface
is going to be ready RSN are wrong. There are some architectural issues
in that we currently have no way to report "this interface should be
usable now", which is shared with all the BSDs. But it is essentially
unrelated to the recent DaD changes.
Joerg
From: mlelstv@serpens.de (Michael van Elst)
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Wed, 5 Oct 2016 20:20:36 +0000 (UTC)
joerg@bec.de (Joerg Sonnenberger) writes:
>In short, *any* assumption that a freshly configured network interface
>is going to be ready RSN are wrong.
Of course nobody really cared because in practice it just worked.
It started to be wrong when enforcing DAD.
It is so well-known to be wrong that people invented optimistic DAD
to reduce the extra delay, but so far only for IPv6. DAD on IPv4
seems to be less common.
--
--
Michael van Elst
Internet: mlelstv@serpens.de
"A potential Snark may lurk in every tree."
From: Andreas Gustafsson <gson@gson.org>
To: Roy Marples <roy@NetBSD.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Mon, 21 Nov 2016 18:50:42 +0200
Roy,
On October 5, you wrote:
> I think ifconfig -w 15 -W 5 should be added to sysinst before the ping
> test to wait for DaD to clear.
And I responded:
< I think sysinst, and other existing code that configures IP addresses,
< is not incorrect as written, and should not be required to do this.
Please don't let our disagreement over the correct way to fix this
issue keep you from fixing it. If you won't fix it my way, then fix
it your way.
--
Andreas Gustafsson, gson@gson.org
From: "Roy Marples" <roy@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/51531 CVS commit: src/usr.sbin/sysinst
Date: Tue, 22 Nov 2016 12:04:35 +0000
Module Name: src
Committed By: roy
Date: Tue Nov 22 12:04:35 UTC 2016
Modified Files:
src/usr.sbin/sysinst: net.c
Log Message:
Fix PR kern/51531 by using ifconfig to wait for addresses to become
valid rather than sleeping a fixed ammount of time.
To generate a diff of this commit:
cvs rdiff -u -r1.21 -r1.22 src/usr.sbin/sysinst/net.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Roy Marples <roy@NetBSD.org>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Tue, 22 Nov 2016 12:05:24 +0000
Hi
On 21/11/2016 16:50, Andreas Gustafsson wrote:
> On October 5, you wrote:
>> I think ifconfig -w 15 -W 5 should be added to sysinst before the ping
>> test to wait for DaD to clear.
>
> And I responded:
> < I think sysinst, and other existing code that configures IP addresses,
> < is not incorrect as written, and should not be required to do this.
>
> Please don't let our disagreement over the correct way to fix this
> issue keep you from fixing it. If you won't fix it my way, then fix
> it your way.
Fixed.
I don't have the resource to actually test it though.
Roy
From: Andreas Gustafsson <gson@gson.org>
To: Roy Marples <roy@NetBSD.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Tue, 22 Nov 2016 20:03:37 +0200
Roy Marples wrote:
> Fixed.
>
> I don't have the resource to actually test it though.
I ran an automated test on -current sources from after the commit of
sysinst/net.c 1.22, and the install still failed. The install log is
here:
http://www.gson.org/netbsd/bugs/build/amd64-baremetal/2016/2016.11.22.12.04.35/install.log
Based on the log:
- The command "/sbin/ifconfig -w 15 -W 5" is now being run
- The ping test fails with 100% packet loss as before
- Retrieving the first installation set fails as before, with
"ftp: Can't connect to `10.169.01:80': Operation timed out"
- Running ifconfig after the failed install shows the
TENTATIVE flag is set:
sk0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ec_capabilities=5<VLAN_MTU,JUMBO_MTU>
ec_enabled=0
address: 00:1b:fc:9e:0f:b4
media: Ethernet autoselect (10baseT half-duplex)
status: active
inet 10.169.0.2/24 broadcast 10.169.0.255 flags 0x1<TENTATIVE>
inet6 fe80::21b:fcff:fe9e:fb4%sk0/64 flags 0x2<TENTATIVE> scopeid 0x3
--
Andreas Gustafsson, gson@gson.org
From: Roy Marples <roy@marples.name>
To: Andreas Gustafsson <gson@gson.org>,Roy Marples <roy@NetBSD.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Tue, 22 Nov 2016 18:40:29 +0000
------R3199Y3SOPHUWPG8W7NG3I01JVO8QX
Content-Transfer-Encoding: 8bit
Content-Type: text/plain;
charset=UTF-8
Do you know if the new ifconfig command did in fact wait for around 15 seconds before running ping?
On 22 November 2016 18:03:37 GMT+00:00, Andreas Gustafsson <gson@gson.org> wrote:
>Roy Marples wrote:
>> Fixed.
>>
>> I don't have the resource to actually test it though.
>
>I ran an automated test on -current sources from after the commit of
>sysinst/net.c 1.22, and the install still failed. The install log is
>here:
>
>http://www.gson.org/netbsd/bugs/build/amd64-baremetal/2016/2016.11.22.12.04.35/install.log
>
>Based on the log:
>
> - The command "/sbin/ifconfig -w 15 -W 5" is now being run
> - The ping test fails with 100% packet loss as before
> - Retrieving the first installation set fails as before, with
> "ftp: Can't connect to `10.169.01:80': Operation timed out"
> - Running ifconfig after the failed install shows the
> TENTATIVE flag is set:
>
> sk0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> ec_capabilities=5<VLAN_MTU,JUMBO_MTU>
> ec_enabled=0
> address: 00:1b:fc:9e:0f:b4
> media: Ethernet autoselect (10baseT half-duplex)
> status: active
> inet 10.169.0.2/24 broadcast 10.169.0.255 flags 0x1<TENTATIVE>
> inet6 fe80::21b:fcff:fe9e:fb4%sk0/64 flags 0x2<TENTATIVE> scopeid 0x3
>
>--
>Andreas Gustafsson, gson@gson.org
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
------R3199Y3SOPHUWPG8W7NG3I01JVO8QX
Content-Type: text/html;
charset=utf-8
Content-Transfer-Encoding: 8bit
<html><head></head><body>Do you know if the new ifconfig command did in fact wait for around 15 seconds before running ping?<br><br><div class="gmail_quote">On 22 November 2016 18:03:37 GMT+00:00, Andreas Gustafsson <gson@gson.org> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<pre class="k9mail">Roy Marples wrote:<br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #729fcf; padding-left: 1ex;"> Fixed.<br /> <br /> I don't have the resource to actually test it though.<br /></blockquote><br />I ran an automated test on -current sources from after the commit of<br />sysinst/net.c 1.22, and the install still failed. The install log is<br />here:<br /><br /> <a href="http://www.gson.org/netbsd/bugs/build/amd64-baremetal/2016/2016.11.22.12.04.35/install.log">http://www.gson.org/netbsd/bugs/build/amd64-baremetal/2016/2016.11.22.12.04.35/install.log</a><br /><br />Based on the log:<br /><br /> - The command "/sbin/ifconfig -w 15 -W 5" is now being run<br /> - The ping test fails with 100% packet loss as before<br /> - Retrieving the first installation set fails as before, with<br /> "ftp: Can't connect to `10.169.01:80': Operation timed out"<br /> - Running ifconfig after the failed install shows the<br />
TENTATIVE flag is set:<br /><br /> sk0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500<br /> ec_capabilities=5<VLAN_MTU,JUMBO_MTU><br /> ec_enabled=0<br /> address: 00:1b:fc:9e:0f:b4<br /> media: Ethernet autoselect (10baseT half-duplex)<br /> status: active<br /> inet <a href="http://10.169.0.2/24">10.169.0.2/24</a> broadcast <a href="http://10.169.0.255">10.169.0.255</a> flags 0x1<TENTATIVE><br /> inet6 fe80::21b:fcff:fe9e:fb4%sk0/64 flags 0x2<TENTATIVE> scopeid 0x3<br /></pre></blockquote></div><br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.</body></html>
------R3199Y3SOPHUWPG8W7NG3I01JVO8QX--
From: Andreas Gustafsson <gson@gson.org>
To: Roy Marples <roy@marples.name>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Tue, 22 Nov 2016 21:05:05 +0200
Roy Marples wrote:
> Do you know if the new ifconfig command did in fact wait for around
> 15 seconds before running ping?
I don't know, as I do not have a timestamped log. I will add a
timestamping facility and rerun the test tomorrow.
I should also mention that when the machine netboots, it gets the
address 10.169.0.2 using DHCP at the PXE and tftpboot stages, and
then in sysinst, it is statically configured with the same address it
previously got using DHCP. Could this be confusing DAD somehow?
This is a dedicated network segment with only the server (10.169.0.1)
and the machine being installed on, so there should be no possibility
of address conflicts with third parties.
--
Andreas Gustafsson, gson@gson.org
From: Roy Marples <roy@marples.name>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Tue, 22 Nov 2016 19:58:40 +0000
On 22/11/2016 19:05, Andreas Gustafsson wrote:
> Roy Marples wrote:
>> Do you know if the new ifconfig command did in fact wait for around
>> 15 seconds before running ping?
>
> I don't know, as I do not have a timestamped log. I will add a
> timestamping facility and rerun the test tomorrow.
>
> I should also mention that when the machine netboots, it gets the
> address 10.169.0.2 using DHCP at the PXE and tftpboot stages, and
> then in sysinst, it is statically configured with the same address it
> previously got using DHCP. Could this be confusing DAD somehow?
>
> This is a dedicated network segment with only the server (10.169.0.1)
> and the machine being installed on, so there should be no possibility
> of address conflicts with third parties.
I'm going to guess that ifconfig -w 15 -W 5 did infact wait the time.
I'll also guess that my recentish changes have unmasked the actual
problem in that for some reason DaD isn't completing on your interface.
Could you wait a few minutes and then look at the ifconfig output again?
Guessing the addresses still say TENTATIVE.
Roy
From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Wed, 23 Nov 2016 07:16:23 +0700
In all of this (and regardless of whatever bugs may be uncovered in
IPv4 DaD processing, and which clearly should get fixed), I still
fail to understand this fetish of not sending from a tentative address.
Note, tentative is not invalid - it is an address whose address is not
invalid but that is not yet known for certain.
I haven't looked at whatever spec now exists for testing v4 addresses
in this way, but as I recall it, for v6, it is absolutely required to
send from a tentative address in some circumstances (particularly when
2 hosts are attempting to configure the same address at the same time,
and both are performing DaD ... if both stay silent, neither will see
the conflict, and both will assume they can transition the addr to valid.)
The real fix for this problem (in addition to any others that come to
attention because of it) would seem to be to remove the restriction on
sending from tentative addresses (that is, to stop treating them as invalid.)
Almost always, DaD "succeeds" and the tentative address becomes valid.
This may not have been quite so true in the past, when people commonly
manually configured IPv4 addresses, and it what let to using ARP to check,
which eventually led to DaD for IPv6, and then its formalisation for IPv4.
These days, almost no-one manually configures addresses, and when they do
it is most commonly of routers/major servers, and they're very rarely
configured incorrectly, and automatic addr assignment (DHCP, or for v6,
autoconfig) almost never produces duplicates.
But - for DaD to succeed, we must wait - if we really believe that there's
more than a negligible chance of duplicates, we really should be sending
multiple queries, and waiting after each. That means noticeable delays,
which almost always are pointless.
On the other hand when DaD does happen to fail (the addr is duplicated)
it almost always fails very very quickly (usually within a millisecond.)
That is, the other host to which the addr has been assigned responds to
the DaD query very very quickly. That is, usually, by the time we have
finished configuring the address, and are ready to actually send a packet
if DaD were ever going to fail, it almost certainly always will have.
And if we want to reduce our chances of accidentally using addr that might
become invalid very soon, we can just delay, 10 ms or so should be enough,
after configuring the address. If DaD has not failed by then, the chances
are very small that it ever will - and if we wanted to improve the chances,
we could send the second query about 5ms after the first rather than waiting
for the entire timeout - just in case something happened to the first packet
(corrupted on the wire, or similar) so it had no chance to be received.
Also note that there is 0 chance of never using an address that might
become invalid in the future (and no way of knowing how far in the future
that might happen) so we either need to simply tolerate occasional comms
failures (which is what we mostly just do - just try again if things don't
work first time - for any reason) or apps need to get a lot smarter to
learn to recover from such events. That is, avoiding using a tentative
address, because it might become invalid is silly, because any address
might become invalid. Even avoiding it because it might become duplicated
is pointless, as the (admittedly rare) case of 2 net segments, reconnecting
after the same address happened to be assigned (and DaD carried out without
complaint) on each of the segments before they re-joined shows.
Please, just put the networking code back to the state it used to be, and
allow tentative addresses to be considered valid for the purposes of
packet sending/reception.
kre
From: Andreas Gustafsson <gson@gson.org>
To: Roy Marples <roy@marples.name>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Wed, 23 Nov 2016 10:33:02 +0200
Roy Marples wrote:
> >> Do you know if the new ifconfig command did in fact wait for around
> >> 15 seconds before running ping?
It waits for about 5 seconds, not 15. Here's the relevant part
of the timestamped log (the big numbers are seconds since the epoch):
recv(1479853346.806, '[37m\x1b[44mStatus: \x1b[7mRun')
recv(1479853346.826, 'ning\x1b[2;5H\x1b[27mC')
recv(1479853346.846, 'ommand: \x1b[7m/sbi')
recv(1479853346.866, 'n/ifconfig -w 15 -W 5\r\n\r')
recv(1479853346.886, '\n\x1b[27m----------')
recv(1479853346.906, '------------------------')
recv(1479853346.926, '----------------')
recv(1479853346.946, '----------------')
recv(1479853346.966, '--------------\x1b[4;1H\x1b[m\r')
recv(1479853346.986, '\n\x1b[m')
recv(1479853351.815, '\x1b[1;14H\x1b')
recv(1479853351.828, '[37m\x1b[44')
recv(1479853351.848, 'm\x1b[7mFinished\x1b[2')
> Could you wait a few minutes and then look at the ifconfig output again?
> Guessing the addresses still say TENTATIVE.
No, they don't. I changed the test to run ifconfig -a twice, with a
delay of 120 seconds inbetween. The first one printed:
sk0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ec_capabilities=5<VLAN_MTU,JUMBO_MTU>
ec_enabled=0
address: 00:1b:fc:9e:0f:b4
media: Ethernet autoselect (10baseT half-duplex)
status: active
inet 10.169.0.2/24 broadcast 10.169.0.255 flags 0x1<TENTATIVE>
inet6 fe80::21b:fcff:fe9e:fb4%sk0/64 flags 0x2<TENTATIVE> scopeid 0x3
and the second one printed:
sk0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ec_capabilities=5<VLAN_MTU,JUMBO_MTU>
ec_enabled=0
address: 00:1b:fc:9e:0f:b4
media: Ethernet autoselect (10baseT half-duplex)
status: active
inet 10.169.0.2/24 broadcast 10.169.0.255 flags 0x0
inet6 fe80::21b:fcff:fe9e:fb4%sk0/64 flags 0x0 scopeid 0x3
--
Andreas Gustafsson, gson@gson.org
From: Roy Marples <roy@marples.name>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Wed, 23 Nov 2016 09:15:09 +0000
On 23/11/2016 08:33, Andreas Gustafsson wrote:
> Roy Marples wrote:
>>>> Do you know if the new ifconfig command did in fact wait for around
>>>> 15 seconds before running ping?
>
> It waits for about 5 seconds, not 15. Here's the relevant part
> of the timestamped log (the big numbers are seconds since the epoch):
OK.
That tells me that the address is DETACHED during this time and not
TENTATIVE.
In other words, the interface thinks there is no carrier.
Could you try changing my commit locally from -w 15 -W 5 to -w 15 -W 15
and see if that helps at all?
If not, do other numbers help? -w 120 -W 120 is insane, but it's worth
trying to see if my theory is right.
If so, we just have to increase the default numbers, and maybe in
rc.conf as well OR look into why the interface is taking more than 5
seconds to gain a carrier as sk0 is cabled yes?
Roy
From: Andreas Gustafsson <gson@gson.org>
To: Roy Marples <roy@marples.name>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Wed, 23 Nov 2016 12:07:19 +0200
Roy Marples wrote:
> > It waits for about 5 seconds, not 15. Here's the relevant part
> > of the timestamped log (the big numbers are seconds since the epoch):
>
> OK.
> That tells me that the address is DETACHED during this time and not
> TENTATIVE.
Shouldn't there be a DETACHED flag in the ifconfig output, then?
> In other words, the interface thinks there is no carrier.
>
> Could you try changing my commit locally from -w 15 -W 5 to -w 15 -W 15
> and see if that helps at all?
Not easily. The testbed in case is generating public reports that are
supposed to reflect the true state of -current as a function of the
CVS source date. To test something that is not actually -current,
I'd have to set up a separate instance of the testbed.
> If not, do other numbers help? -w 120 -W 120 is insane, but it's worth
> trying to see if my theory is right.
>
> If so, we just have to increase the default numbers, and maybe in
> rc.conf as well OR look into why the interface is taking more than 5
> seconds to gain a carrier as sk0 is cabled yes?
It is cabled to a 10baseT (sic) hub. I could try a 100baseTX switch
and see if that makes a difference.
--
Andreas Gustafsson, gson@gson.org
From: Roy Marples <roy@marples.name>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Wed, 23 Nov 2016 10:30:47 +0000
On 23/11/2016 10:07, Andreas Gustafsson wrote:
> Roy Marples wrote:
>>> It waits for about 5 seconds, not 15. Here's the relevant part
>>> of the timestamped log (the big numbers are seconds since the epoch):
>>
>> OK.
>> That tells me that the address is DETACHED during this time and not
>> TENTATIVE.
>
> Shouldn't there be a DETACHED flag in the ifconfig output, then?
Maybe not.
Addresses transition from DETACHED -> TENTATIVE when the carrier comes
up and this triggers DaD.
If the carrier (or interface) goes down then all addresses are marked
DETACHED.
As Joerg indicated earlier this could happen with wireless networking
very easily.
Personally I would be inclinded to remove the ping test entirely - the
fact it passes is not a guarantee it can download sets.
ifconfig -w 15 -W 5 says wait 5 seconds for DETACHED to clear and 15 for
TENTATIVE to clear. So if it's waiting for 5 seconds as you say then the
carrier has been down for at least 5 seconds for some reason.
rc.d/network was modified a long time ago with this exact same change.
>> In other words, the interface thinks there is no carrier.
>>
>> Could you try changing my commit locally from -w 15 -W 5 to -w 15 -W 15
>> and see if that helps at all?
>
> Not easily. The testbed in case is generating public reports that are
> supposed to reflect the true state of -current as a function of the
> CVS source date. To test something that is not actually -current,
> I'd have to set up a separate instance of the testbed.
>
>> If not, do other numbers help? -w 120 -W 120 is insane, but it's worth
>> trying to see if my theory is right.
>>
>> If so, we just have to increase the default numbers, and maybe in
>> rc.conf as well OR look into why the interface is taking more than 5
>> seconds to gain a carrier as sk0 is cabled yes?
>
> It is cabled to a 10baseT (sic) hub. I could try a 100baseTX switch
> and see if that makes a difference.
Please do.
Roy
From: Andreas Gustafsson <gson@gson.org>
To: Roy Marples <roy@marples.name>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Wed, 23 Nov 2016 15:48:16 +0200
Roy Marples wrote:
> Personally I would be inclinded to remove the ping test entirely - the
> fact it passes is not a guarantee it can download sets.
I agree - not only is the test passing no guarantee that sets can be
downloaded, but also, the ping test failing is no guarantee that
downloading the sets will fail. But I don't think that has anything
to do with the present issue: the problem is that downloading the sets
fails, and whether the ping test succeeds, fails, or even exists is
irrelevant.
> > It is cabled to a 10baseT (sic) hub. I could try a 100baseTX switch
> > and see if that makes a difference.
>
> Please do.
Done. It made no difference. Here's the ifconfig output from immediately
after the ftp command failed:
sk0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ec_capabilities=5<VLAN_MTU,JUMBO_MTU>
ec_enabled=0
address: 00:1b:fc:9e:0f:b4
media: Ethernet autoselect (100baseTX full-duplex,flowcontrol,rxpause,txpause)
status: active
inet 10.169.0.2/24 broadcast 10.169.0.255 flags 0x1<TENTATIVE>
inet6 fe80::21b:fcff:fe9e:fb4%sk0/64 flags 0x2<TENTATIVE> scopeid 0x3
And 120 seconds later:
sk0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ec_capabilities=5<VLAN_MTU,JUMBO_MTU>
ec_enabled=0
address: 00:1b:fc:9e:0f:b4
media: Ethernet autoselect (100baseTX full-duplex,flowcontrol,rxpause,txpause)
status: active
inet 10.169.0.2/24 broadcast 10.169.0.255 flags 0x0
inet6 fe80::21b:fcff:fe9e:fb4%sk0/64 flags 0x0 scopeid 0x3
--
Andreas Gustafsson, gson@gson.org
From: Andreas Gustafsson <gson@gson.org>
To: Roy Marples <roy@marples.name>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/51531: Recent networking regression affecting installs
Date: Wed, 23 Nov 2016 17:16:06 +0200
Roy,
I took packet captures on the server (10.169.0.1) of a successful
network install (source date 2016.09.16.10.54.45) and an unsuccessful
one (sorce date 2016.11.22.12.04.35). Maybe you can glean some
useful information from them.
Here are extracts showing the last couple of packets of the TFTP boot
followed by the DAD, ping test, and first couple of packets of the HTTP
set download (if present). The successful one looks like this:
16:42:58.073675 00:50:fc:fb:10:45 (oui Unknown) > 00:1b:fc:9e:0f:b4 (oui Unknown), ethertype IPv4 (0x0800), length 558: 10.169.0.1.62529 > 10.169.0.2.dict-lookup: UDP, length 516
16:42:58.087396 00:1b:fc:9e:0f:b4 (oui Unknown) > 00:50:fc:fb:10:45 (oui Unknown), ethertype IPv4 (0x0800), length 60: 10.169.0.2.dict-lookup > 10.169.0.1.62529: UDP, length 5
16:44:05.360799 00:1b:fc:9e:0f:b4 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 60: Request who-has 10.169.0.2 tell 10.169.0.2, length 46
16:44:13.346801 00:1b:fc:9e:0f:b4 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 60: Request who-has 10.169.0.1 tell 10.169.0.2, length 46
16:44:13.346811 00:50:fc:fb:10:45 (oui Unknown) > 00:1b:fc:9e:0f:b4 (oui Unknown), ethertype ARP (0x0806), length 42: Reply 10.169.0.1 is-at 00:50:fc:fb:10:45 (oui Unknown), length 28
16:44:13.348603 00:1b:fc:9e:0f:b4 (oui Unknown) > 00:50:fc:fb:10:45 (oui Unknown), ethertype IPv4 (0x0800), length 98: 10.169.0.2 > 10.169.0.1: ICMP echo request, id 19520, seq 0, length 64
16:44:13.348614 00:50:fc:fb:10:45 (oui Unknown) > 00:1b:fc:9e:0f:b4 (oui Unknown), ethertype IPv4 (0x0800), length 98: 10.169.0.1 > 10.169.0.2: ICMP echo reply, id 19520, seq 0, length 64
16:44:16.479645 00:1b:fc:9e:0f:b4 (oui Unknown) > 00:50:fc:fb:10:45 (oui Unknown), ethertype IPv4 (0x0800), length 98: 10.169.0.2 > 10.0.1.1: ICMP echo request, id 22346, seq 0, length 64
16:44:16.480349 00:50:fc:fb:10:45 (oui Unknown) > 00:1b:fc:9e:0f:b4 (oui Unknown), ethertype IPv4 (0x0800), length 98: 10.0.1.1 > 10.169.0.2: ICMP echo reply, id 22346, seq 0, length 64
16:44:19.104649 00:1b:fc:9e:0f:b4 (oui Unknown) > 00:50:fc:fb:10:45 (oui Unknown), ethertype IPv4 (0x0800), length 78: 10.169.0.2.65535 > 10.169.0.1.http: Flags [S], seq 2279312074, win 32768, options [mss 1460,nop,wscale 3,sackOK,nop,nop,nop,nop,TS val 1 ecr 0], length 0
16:44:19.104665 00:50:fc:fb:10:45 (oui Unknown) > 00:1b:fc:9e:0f:b4 (oui Unknown), ethertype IPv4 (0x0800), length 78: 10.169.0.1.http > 10.169.0.2.65535: Flags [S.], seq 241726766, ack 2279312075, win 32768, options [mss 1460,nop,wscale 3,nop,nop,TS val 1 ecr 1,sackOK,nop,nop], length 0
(successful HTTP download follows)
The unsuccssful one looks like this:
21:02:02.196177 00:50:fc:fb:10:45 (oui Unknown) > 00:1b:fc:9e:0f:b4 (oui Unknown), ethertype IPv4 (0x0800), length 558: 10.169.0.1.62681 > 10.169.0.2.netadmin: UDP, length 516
21:02:02.210352 00:1b:fc:9e:0f:b4 (oui Unknown) > 00:50:fc:fb:10:45 (oui Unknown), ethertype IPv4 (0x0800), length 60: 10.169.0.2.netadmin > 10.169.0.1.62681: UDP, length 5
21:04:44.366636 00:1b:fc:9e:0f:b4 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 60: Request who-has 10.169.0.2 tell 0.0.0.0, length 46
21:04:44.939141 00:1b:fc:9e:0f:b4 (oui Unknown) > 33:33:ff:9e:0f:b4 (oui Unknown), ethertype IPv6 (0x86dd), length 78: :: > ff02::1:ff9e:fb4: ICMP6, neighbor solicitation, who has fe80::21b:fcff:fe9e:fb4, length 24
21:04:45.772737 00:1b:fc:9e:0f:b4 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 60: Request who-has 10.169.0.2 tell 0.0.0.0, length 46
21:04:47.670958 00:1b:fc:9e:0f:b4 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 60: Request who-has 10.169.0.2 tell 0.0.0.0, length 46
21:04:49.679675 00:1b:fc:9e:0f:b4 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 60: Request who-has 10.169.0.2 tell 10.169.0.2, length 46
21:04:51.688397 00:1b:fc:9e:0f:b4 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 60: Request who-has 10.169.0.2 tell 10.169.0.2, length 46
(no further packets)
--
Andreas Gustafsson, gson@gson.org
Responsible-Changed-From-To: kern-bug-people->roy
Responsible-Changed-By: gson@NetBSD.org
Responsible-Changed-When: Mon, 12 Dec 2016 16:42:56 +0000
Responsible-Changed-Why:
roy broke it
From: "Roy Marples" <roy@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/51531 CVS commit: src/usr.sbin/sysinst
Date: Tue, 13 Dec 2016 19:03:49 +0000
Module Name: src
Committed By: roy
Date: Tue Dec 13 19:03:49 UTC 2016
Modified Files:
src/usr.sbin/sysinst: net.c
Log Message:
ping is not a reliable means of testing if connectivity to download sets
actually works, so remove it.
Hopefully fixes PR kern/51531.
To generate a diff of this commit:
cvs rdiff -u -r1.22 -r1.23 src/usr.sbin/sysinst/net.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Andreas Gustafsson <gson@gson.org>
To: roy@NetBSD.org
Cc: gnats-bugs@NetBSD.org
Subject: Re: PR/51531 CVS commit: src/usr.sbin/sysinst
Date: Wed, 14 Dec 2016 10:28:05 +0200
Roy Marples wrote:
> ping is not a reliable means of testing if connectivity to download sets
> actually works, so remove it.
As I said in an earlier comment to this PR, I support this change,
but I don't think it is related to the present issue.
> Hopefully fixes PR kern/51531.
It did not. Console log:
http://www.gson.org/netbsd/bugs/build/amd64-baremetal/2016/2016.12.13.19.03.49/install.log
Timestamped log:
http://www.gson.org/netbsd/bugs/51531/install-2016.12.13.19.03.49-slog.txt
Some observations based on the timestamped log:
The "ifconfig -w 15 -W 5" command runs in five seconds. The ftp
command then takes some 74 seconds to time out.
Then, a diagnostic "ifconfig -a" is run, and it shows a TENTATIVE flag
even though 83 seconds (!) have now elapsed since the "ifconfig
-w 15 -W 5" started.
Ten seconds later, a second diagnostic "ifconfig -a" is run, and then,
the TENTATIVE flag is gone.
When reading the timestamped log, note that the console output is not
being read during the 10-second sleep between the two "ifconfig -a"
commands, so the output from the first "ifconfig -a" only appears in
the log after the second "ifconfig -a" has been sent, and its timestamp
is not meaningful. The timestamps on the "send(..., 'ifconfig -a\n')"
lines are correct.
--
Andreas Gustafsson, gson@gson.org
From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@NetBSD.org
Cc: roy@NetBSD.org
Subject: Re: PR/51531 CVS commit: src/usr.sbin/sysinst
Date: Wed, 14 Dec 2016 10:45:00 +0200
A short while ago, I wrote:
> Ten seconds later, a second diagnostic "ifconfig -a" is run, and then,
> the TENTATIVE flag is gone.
A correction to the above: the delay between the two diagnostic
"ifconfig -a" commands is actually 120 seconds, not 10 seconds.
--
Andreas Gustafsson, gson@gson.org
From: Roy Marples <roy@marples.name>
To: gnats-bugs@netbsd.org
Cc: gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, gson@gson.org
Subject: Re: PR/51531 CVS commit: src/usr.sbin/sysinst
Date: Wed, 14 Dec 2016 13:40:50 +0000
On 2016-12-14 08:30, Andreas Gustafsson wrote:
> The following reply was made to PR kern/51531; it has been noted by
> GNATS.
>
> From: Andreas Gustafsson <gson@gson.org>
> To: roy@NetBSD.org
> Cc: gnats-bugs@NetBSD.org
> Subject: Re: PR/51531 CVS commit: src/usr.sbin/sysinst
> Date: Wed, 14 Dec 2016 10:28:05 +0200
>
> Roy Marples wrote:
> > ping is not a reliable means of testing if connectivity to download
> sets
> > actually works, so remove it.
>
> As I said in an earlier comment to this PR, I support this change,
> but I don't think it is related to the present issue.
>
> > Hopefully fixes PR kern/51531.
>
> It did not. Console log:
>
>
> http://www.gson.org/netbsd/bugs/build/amd64-baremetal/2016/2016.12.13.19.03.49/install.log
>
> Timestamped log:
>
>
> http://www.gson.org/netbsd/bugs/51531/install-2016.12.13.19.03.49-slog.txt
>
> Some observations based on the timestamped log:
>
> The "ifconfig -w 15 -W 5" command runs in five seconds. The ftp
> command then takes some 74 seconds to time out.
>
> Then, a diagnostic "ifconfig -a" is run, and it shows a TENTATIVE flag
> even though 83 seconds (!) have now elapsed since the "ifconfig
> -w 15 -W 5" started.
>
> Ten seconds later, a second diagnostic "ifconfig -a" is run, and then,
> the TENTATIVE flag is gone.
>
> When reading the timestamped log, note that the console output is not
> being read during the 10-second sleep between the two "ifconfig -a"
> commands, so the output from the first "ifconfig -a" only appears in
> the log after the second "ifconfig -a" has been sent, and its
> timestamp
> is not meaningful. The timestamps on the "send(..., 'ifconfig -a\n')"
> lines are correct.
This sounds like one of two things:
1) there is a timing issue
2) the link status is flapping
Because at the point of the ifconfig -w command there are no futher
interface changes, nothing should be resetting the PHY at all so one of
the above must be true.
I say this because the maximum length from DETACHED -> TENTATIVE ->
ready to go is 10 seconds once the link is active and your logs show
it's at TENTATIVE way beyond that which implies link state is flapping
or 10 seconds is not 10 seconds on this device.
Some idea of how to solve this are
1) Implement RFC4429 and apply it to IPv4 as well
(needs a toggle like ifconfig bge0 inet 1.2.3.4/24 optimistic)
2) Add a toggle to sysinst to disable DaD
I don't suppose you could try swapping the interfacec out with another
one?
Roy
From: Andreas Gustafsson <gson@gson.org>
To: Roy Marples <roy@marples.name>
Cc: gnats-bugs@netbsd.org
Subject: Re: PR/51531 CVS commit: src/usr.sbin/sysinst
Date: Wed, 14 Dec 2016 18:39:57 +0200
Roy Marples wrote:
> This sounds like one of two things:
> 1) there is a timing issue
> 2) the link status is flapping
>
> Because at the point of the ifconfig -w command there are no futher
> interface changes, nothing should be resetting the PHY at all so one of
> the above must be true.
> I say this because the maximum length from DETACHED -> TENTATIVE ->
> ready to go is 10 seconds once the link is active and your logs show
> it's at TENTATIVE way beyond that which implies link state is flapping
> or 10 seconds is not 10 seconds on this device.
I just ran another pair of tests. First, I made the test script run
"ifconfig -a" in a loop, once per second for 60 seconds, starting two
seconds after the failed "ftp". The "ifconfig -a" output had the
TENTATIVE flag set in the first five runs, and clear in the remaining
55 runs.
Then I did a second run, but now with a 30-second delay between the
failed "ftp" and the first "ifconfig -a" instead of a 2-second delay.
Now, the output had the TENTATIVE flag set in the first six runs, and
clear in the remaining 54.
So, the TENTATIVE flag appears to be set for some 5-6 seconds,
counting from the time when "ifconfig -a" is first run, *regardless*
of when that first run takes place. The only explanation I can think
of is that it must be the "ifconfig -a" itself that starts the timer,
despite supposedly being a read-only operation.
Is it possible that the initial DAD attempt could somehow get stuck
indefinitely, for example because of the NIC being in an unexpected
initial state after the netboot, and that the "ifconfig -a" could
then cause it to come unstuck?
> Some idea of how to solve this are
> 1) Implement RFC4429 and apply it to IPv4 as well
This, too, seems like a worthwhile change for reasons unrelated to the
present issue, and it might be a viable work-around, but I don't think
it addresses the root cause.
> (needs a toggle like ifconfig bge0 inet 1.2.3.4/24 optimistic)
> 2) Add a toggle to sysinst to disable DaD
IMO, it needs to work with the default settings. Someone trying to
install NetBSD for the first time and running into this issue is far
more likely to give up and move on to the next OS candidate than to
find out what special settings are needed.
> I don't suppose you could try swapping the interfacec out with another
> one?
I currently have two machines configured for this automated test.
I already tried the other one, and it behaved the same way.
Unfortunately, even though they have different motherboards and CPUs,
they have the same type of on-board Ethernet interface (Marvell Yukon
Lite) so the possibility remains that this issue could be specific
to that chip or the sk driver.
I'll see if I can find a suitable PCI card (this is not as trivial as
it may sound because it needs to have a working PXE ROM). Or maybe
I'll set up a third machine...
--
Andreas Gustafsson, gson@gson.org
From: Andreas Gustafsson <gson@gson.org>
To: Roy Marples <roy@marples.name>
Cc: gnats-bugs@netbsd.org
Subject: Re: PR/51531 CVS commit: src/usr.sbin/sysinst
Date: Wed, 14 Dec 2016 21:34:08 +0200
Roy,
I found an old Intel PCI network card (fxp) with a PXE ROM, and using
that, the installation succeeded. So it looks like your DAD changes
have exposed a problem with the sk driver.
--
Andreas Gustafsson, gson@gson.org
From: Roy Marples <roy@marples.name>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: PR/51531 CVS commit: src/usr.sbin/sysinst
Date: Wed, 14 Dec 2016 20:44:57 +0000
Hi Andreas
On 14/12/16 19:34, Andreas Gustafsson wrote:
> Roy,
>
> I found an old Intel PCI network card (fxp) with a PXE ROM, and using
> that, the installation succeeded. So it looks like your DAD changes
> have exposed a problem with the sk driver.
Well, that's good and bad news I guess.
Good news that you can continue testing, bad news that the underlying
issue isn't resolved.
So what do we want to do with this bug?
I think the sysinst changes to date are still valid.
Close this and make a new ticket?
Roy
Responsible-Changed-From-To: roy->kern-bug-people
Responsible-Changed-By: gson@NetBSD.org
Responsible-Changed-When: Wed, 14 Dec 2016 21:12:43 +0000
Responsible-Changed-Why:
The root cause of the problem exposed by roy's change appears
to be in sk(4).
From: Andreas Gustafsson <gson@gson.org>
To: Roy Marples <roy@marples.name>
Cc: gnats-bugs@netbsd.org
Subject: Re: PR/51531 CVS commit: src/usr.sbin/sysinst
Date: Wed, 14 Dec 2016 23:10:16 +0200
Roy Marples wrote:
> Well, that's good and bad news I guess.
> Good news that you can continue testing, bad news that the underlying
> issue isn't resolved.
Right.
> So what do we want to do with this bug?
> I think the sysinst changes to date are still valid.
They can stay; removing the ping test is the right thing to do
regardless, and I suppose the "ifconfig -w 15 -W 5" is correct given
the current current behavior of ifconfig where it can exit before the
address being configured is usable. I do still think that behavior is
incorrect, but that is a separate issue which should have a separate PR.
> Close this and make a new ticket?
I think we should keep this ticket open, consider it a probable sk(4)
bug, and assign it back to kern-bug-people.
--
Andreas Gustafsson, gson@gson.org
From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, roy@NetBSD.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, gson@gson.org (Andreas Gustafsson)
Cc:
Subject: Re: PR/51531 CVS commit: src/usr.sbin/sysinst
Date: Wed, 14 Dec 2016 17:21:37 -0500
On Dec 14, 8:50pm, roy@marples.name (Roy Marples) wrote:
-- Subject: Re: PR/51531 CVS commit: src/usr.sbin/sysinst
| Hi Andreas
|
| On 14/12/16 19:34, Andreas Gustafsson wrote:
| > Roy,
| >
| > I found an old Intel PCI network card (fxp) with a PXE ROM, and using
| > that, the installation succeeded. So it looks like your DAD changes
| > have exposed a problem with the sk driver.
|
| Well, that's good and bad news I guess.
| Good news that you can continue testing, bad news that the underlying
| issue isn't resolved.
|
| So what do we want to do with this bug?
| I think the sysinst changes to date are still valid.
|
| Close this and make a new ticket?
Can you try 1.84 of if_sk.c?
christos
From: Andreas Gustafsson <gson@gson.org>
To: christos@zoulas.com (Christos Zoulas)
Cc: gnats-bugs@NetBSD.org,
roy@NetBSD.org
Subject: Re: PR/51531 CVS commit: src/usr.sbin/sysinst
Date: Thu, 15 Dec 2016 19:46:53 +0200
Christos Zoulas wrote:
> Can you try 1.84 of if_sk.c?
Done. I'm still seeing the same strange behavior where running
ifconfig -a repeatedly shows the TENTATIVE flag set for the first six
seconds or so and clear thereafter, even though the loop of "ifconfig
-a" commands is started at a completely arbitrary point in time
relative to when the interface was first configured or last used.
Log:
http://www.gson.org/netbsd/bugs/build/amd64-baremetal/2016/2016.12.15.13.06.08/install.log
Timestamped log:
http://www.gson.org/netbsd/bugs/51531/install-2016.12.15.13.06.08.slog.txt
--
Andreas Gustafsson, gson@gson.org
From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/51531 (Recent networking regression affecting installs)
Date: Tue, 28 Sep 2021 20:51:53 +0300
This five-year-old issue affecting sk(4) is still present in -current
as of source date 2021.09.28.11.38.07.
This time I happened to be using a kernel built with "options DEBUG",
and this enabled an additional kernel message that might be helpful in
tracking down the cause of the problem.
As before, an attempt to install NetBSD over the network using an
sk(4) interface failed. The INSTALL kernel successfully booted over
TFTP, but when sysinst (re-)configured the interface and attempted to
download the first install set over HTTP, the connection timed out.
To diagnose the failure, "ifconfig -a" was run from the shell more
than ten minutes after sysinst had terminated, and even though
"ifconfig -a" is supposed to only display things, not change them, and
there had been no recent changes to the link state, it had the side
effect of triggering the kernel debug message "sk0: link state UP (was
DOWN)":
# ifconfig -a
sk0: link state UP (was DOWN)
lo0: flags=0x8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33624
inet 127.0.0.1/8 flags 0x0
inet6 ::1/128 flags 0x20<NODAD>
inet6 fe80::1%lo0/64 flags 0x0 scopeid 0x1
sk0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ec_capabilities=5<VLAN_MTU,JUMBO_MTU>
ec_enabled=0
address: 00:17:31:16:dd:fa
media: Ethernet autoselect (1000baseT full-duplex)
status: active
inet 10.169.3.2/24 broadcast 10.169.3.255 flags 0x1<TENTATIVE>
inet6 fe80::217:31ff:fe16:ddfa%sk0/64 flags 0x2<TENTATIVE> scopeid 0x2
To me this looks like sk(4) is failing to update the link state when
it actually changes, and instead updating it much later at the point
when "ifconfig -a" is run.
--
Andreas Gustafsson, gson@gson.org
From: Jason Thorpe <thorpej@me.com>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/51531 (Recent networking regression affecting installs)
Date: Tue, 28 Sep 2021 10:58:34 -0700
Can you send me the "dmesg" for this system?
> On Sep 28, 2021, at 10:55 AM, Andreas Gustafsson <gson@gson.org> wrote:
>
> To me this looks like sk(4) is failing to update the link state when
> it actually changes, and instead updating it much later at the point
> when "ifconfig -a" is run.
-- thorpej
From: Andreas Gustafsson <gson@gson.org>
To: Jason Thorpe <thorpej@me.com>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/51531 (Recent networking regression affecting installs)
Date: Tue, 28 Sep 2021 21:22:55 +0300
Jason Thorpe wrote:
> Can you send me the "dmesg" for this system?
A console log including the dmesg from the INSTALL kernel is at
https://www.gson.org/netbsd/bugs/build/amd64-athlon64/2021/2021.09.28.11.38.07/install.log
--
Andreas Gustafsson, gson@gson.org
From: Jason Thorpe <thorpej@me.com>
To: Andreas Gustafsson <gson@gson.org>
Cc: kern-bug-people@netbsd.org,
gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org,
gnats-bugs@netbsd.org
Subject: Re: kern/51531 (Recent networking regression affecting installs)
Date: Wed, 29 Sep 2021 14:33:41 -0700
> On Sep 28, 2021, at 10:55 AM, Andreas Gustafsson <gson@gson.org> =
wrote:
>=20
> To me this looks like sk(4) is failing to update the link state when
> it actually changes, and instead updating it much later at the point
> when "ifconfig -a" is run.
Ok, so I took a quick look and this looks like this is a logic bug in =
sk_tick().
Background: The MII layer uses mii_tick() to perform periodic link =
checks and send link status changes up the network stack. These are =
generally assumed to happen once per second (for example, the MII =
auto-negotiation logic counts "ticks", where a tick is assumed to be 1 =
second, to determine how often to re-initiate auto-negotiation if the =
link is down).
The "sk" driver is doing something different, and possibly being a =
little too clever (and certainly not working the way the MII layer =
assumes drivers work). It appears to be using an interrupt on one of =
the MAC's GPIO pins to determine if link status has changed, and then =
conditionally disables the 1-second periodic timer if it detects link. =
It also has some code that reads a GPIO pin and skips work if that pin =
reads back asserted.
It does something entirely different if the device has a Broadcom PHY =
(which your interface doesn't have; you have a Marvell "Alaska" PHY).
Anyway, I think it would probably work a lot better if it wasn't trying =
to be quite so clever around link change interrupts and the GPIO link =
status input. If it just called mii_tick() once per second like other =
drivers do, it would probably work just fine. But I don't have any =
manuals for these parts, nor generally much familiarity with them, so =
I'll let someone else who's done some actual work on the "sk" driver to =
chime in and take a deeper look.
-- thorpej
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.46 2020/01/03 16:35:01 leot Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2020
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home