NetBSD Problem Report #54057

From he@smistad.uninett.no  Tue Mar 12 08:48:10 2019
Return-Path: <he@smistad.uninett.no>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 3C7AD7A1B3
	for <gnats-bugs@gnats.NetBSD.org>; Tue, 12 Mar 2019 08:48:10 +0000 (UTC)
Message-Id: <20190312084802.5B4ED43F3E7@smistad.uninett.no>
Date: Tue, 12 Mar 2019 09:48:02 +0100 (CET)
From: he@uninett.no
Reply-To: he@uninett.no
To: gnats-bugs@NetBSD.org
Subject: ipv6 neighbor discovery fails to resolve address
X-Send-Pr-Version: 3.95

>Number:         54057
>Category:       kern
>Synopsis:       ipv6 neighbor discovery fails to resolve address
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Mar 12 08:50:00 +0000 2019
>Last-Modified:  Tue Mar 12 15:45:00 +0000 2019
>Originator:     he@uninett.no
>Release:        NetBSD 8.0_STABLE
>Organization:
	Uninett AS
>Environment:
System: NetBSD granlund.uninett.no 8.0_STABLE NetBSD 8.0_STABLE (GENERIC) #5: Mon Sep 24 11:24:29 CEST 2018  he@granlund.uninett.no:/usr/obj/sys/arch/amd64/compile/GENERIC amd64
Architecture: x86_64
Machine: amd64
>Description:
	I get assigned an IPv6 address on this wireless network:

granlund# ifconfig iwn0
iwn0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        ssid eduroam nwkey 65536:"",0xXXXXXXXXXXXXXXXXccd564d49b94,"",""
        powersave off
        bssid ac:a3:1e:09:a1:61 chan 6
        address: a4:4e:31:e4:43:a8
        media: IEEE802.11 autoselect (OFDM6 mode 11g)
        status: active
        inet a.b.c.d/24 broadcast a.b.c.255 flags 0x0
        inet6 fe80::4968:f8ab:9024:84a5%iwn0/64 flags 0x0 scopeid 0x2
        inet6 2001:700:1:21::11bb/128 flags 0x0
granlund# 

	and a corresponding default gateway via a link-local address:

granlund# netstat -rn -f inet6 | grep default
default                                 fe80::120e:7e00:15c6:cbf0      UGS         -        -      -  iwn0
granlund# 

	However, trying to use this setup does not work; I get a
	time-out with ssh on IPv6 and eventually resorts to IPv4.

	Tcpdump'ing the ND traffic shows that the router is indeed
	replying, but apparently the kernel is refusing to install the
	neighbor discovery entry, since "ndp -an" continues to be
	empty:

granlund# ndp -an
Neighbor                                Linklayer Address  Netif Expire    S Fl
granlund# 

	The tcpdump shows:

09:28:31.835893 a4:4e:31:e4:43:a8 > 33:33:ff:c6:cb:f0, ethertype IPv6 (0x86dd), length 86: 2001:700:1:21::11bb > ff02::1:ffc6:cbf0: ICMP6, neighbor solicitation, who has fe80::120e:7e00:15c6:cbf0, length 32
09:28:31.838053 10:0e:7e:c6:cb:f0 > a4:4e:31:e4:43:a8, ethertype IPv6 (0x86dd), length 86: 2001:700:1:21::2 > 2001:700:1:21::11bb: ICMP6, neighbor advertisement, tgt is fe80::120e:7e00:15c6:cbf0, length 32
09:28:32.838212 a4:4e:31:e4:43:a8 > 33:33:ff:c6:cb:f0, ethertype IPv6 (0x86dd), length 86: 2001:700:1:21::11bb > ff02::1:ffc6:cbf0: ICMP6, neighbor solicitation, who has fe80::120e:7e00:15c6:cbf0, length 32
09:28:32.850136 10:0e:7e:c6:cb:f0 > a4:4e:31:e4:43:a8, ethertype IPv6 (0x86dd), length 86: 2001:700:1:21::2 > 2001:700:1:21::11bb: ICMP6, neighbor advertisement, tgt is fe80::120e:7e00:15c6:cbf0, length 32
09:28:33.848998 a4:4e:31:e4:43:a8 > 33:33:ff:c6:cb:f0, ethertype IPv6 (0x86dd), length 86: 2001:700:1:21::11bb > ff02::1:ffc6:cbf0: ICMP6, neighbor solicitation, who has fe80::120e:7e00:15c6:cbf0, length 32
09:28:33.850563 10:0e:7e:c6:cb:f0 > a4:4e:31:e4:43:a8, ethertype IPv6 (0x86dd), length 86: 2001:700:1:21::2 > 2001:700:1:21::11bb: ICMP6, neighbor advertisement, tgt is fe80::120e:7e00:15c6:cbf0, length 32

	If I remove the default route and configure it to the global
	IPv6 address of the (VRRP-provided) address:

granlund# route delete -inet6 default
delete net default
granlund# route add -inet6 default 2001:700:1:21::1
add net default: gateway 2001:700:1:21::1
granlund# 

	things start working as they should.

09:45:20.573201 a4:4e:31:e4:43:a8 > 33:33:ff:00:00:01, ethertype IPv6 (0x86dd), length 86: 2001:700:1:21::11bb > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 2001:700:1:21::1, length 32
09:45:20.574790 10:0e:7e:c6:cb:f0 > a4:4e:31:e4:43:a8, ethertype IPv6 (0x86dd), length 86: 2001:700:1:21::2 > 2001:700:1:21::11bb: ICMP6, neighbor advertisement, tgt is 2001:700:1:21::1, length 32
09:45:20.574818 a4:4e:31:e4:43:a8 > 00:00:5e:00:02:00, ethertype IPv6 (0x86dd), length 70: 2001:700:1:21::11bb > 2001:700:1:0:eeb1:d7ff:fe59:fbaa: ICMP6, echo request, seq 0, length 16
09:45:20.576775 10:0e:7e:c6:cb:f0 > a4:4e:31:e4:43:a8, ethertype IPv6 (0x86dd), length 70: 2001:700:1:0:eeb1:d7ff:fe59:fbaa > 2001:700:1:21::11bb: ICMP6, echo reply, seq 0, length 16
09:45:20.819682 10:0e:7e:c6:cf:f0 > a4:4e:31:e4:43:a8, ethertype IPv6 (0x86dd), length 86: fe80::200:5eff:fe00:200 > fe80::4968:f8ab:9024:84a5: ICMP6, neighbor solicitation, who has fe80::4968:f8ab:9024:84a5, length 32
09:45:20.819715 a4:4e:31:e4:43:a8 > 00:00:5e:00:02:00, ethertype IPv6 (0x86dd), length 78: fe80::4968:f8ab:9024:84a5 > fe80::200:5eff:fe00:200: ICMP6, neighbor advertisement, tgt is fe80::4968:f8ab:9024:84a5, length 24
09:45:21.586068 a4:4e:31:e4:43:a8 > 00:00:5e:00:02:00, ethertype IPv6 (0x86dd), length 70: 2001:700:1:21::11bb > 2001:700:1:0:eeb1:d7ff:fe59:fbaa: ICMP6, echo request, seq 1, length 16
09:45:21.591958 10:0e:7e:c6:cb:f0 > a4:4e:31:e4:43:a8, ethertype IPv6 (0x86dd), length 70: 2001:700:1:0:eeb1:d7ff:fe59:fbaa > 2001:700:1:21::11bb: ICMP6, echo reply, seq 1, length 16

granlund# ndp -an
Neighbor                                Linklayer Address  Netif Expire    S Fl
2001:700:1:21::1                        00:00:5e:00:02:00   iwn0 23h59m17s S R
fe80::200:5eff:fe00:200%iwn0            00:00:5e:00:02:00   iwn0 23h58m38s S
granlund# 	

	Should not the first setup with a link-local address as
	default gateway work?  Why is the neighbor discovery traffic
	ignored?

>How-To-Repeat:
	Come visit us, get IPv6 on eduroam, watch it fail for NetBSD. :)

>Fix:
	Sorry, don't know.

>Audit-Trail:
From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/54057: ipv6 neighbor discovery fails to resolve address
Date: Tue, 12 Mar 2019 19:58:33 +0700

 First, yes, the link local default router should work, in
 fact that's the way it is supposed to be used (that way the
 router can send a redirect that will be accepted - redirects
 from global addresses should be ignored, too easy to spoof,
 and redirects are only accepted when they come from the address
 to which the packet was originally sent - so if you send to
 a global addr default router, it can never redirect you to a
 more appropriate router).

 Apart from that, ND for the default router is normally not needed,
 unless you have configured the router to send an anycast address
 as the default (in which case ND is needed so you can find the
 "nearest" router that shares the anycast addr) the MAC addr (link
 layer addr) of the router is normally sent in the RA which announces
 the default router.

 However, ND should work, and it isn't clear from your tcpdump why
 it isn't.   Any chance you could repeat the tcpdump with -v (or
 even -vv) to get the whole packet dumped, rather than just a summary
 of what the packets are about, which is what the default output
 shows.

 kre

From: Havard Eidnes <he@uninett.no>
To: gnats-bugs@NetBSD.org, kre@munnari.OZ.AU
Cc: kern-bug-people@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/54057: ipv6 neighbor discovery fails to resolve address
Date: Tue, 12 Mar 2019 14:24:47 +0100 (CET)

 >  First, yes, the link local default router should work, in
 >  fact that's the way it is supposed to be used (that way the
 >  router can send a redirect that will be accepted - redirects
 >  from global addresses should be ignored, too easy to spoof,
 >  and redirects are only accepted when they come from the address
 >  to which the packet was originally sent - so if you send to
 >  a global addr default router, it can never redirect you to a
 >  more appropriate router).
 >
 >  Apart from that, ND for the default router is normally not needed,
 >  unless you have configured the router to send an anycast address
 >  as the default (in which case ND is needed so you can find the
 >  "nearest" router that shares the anycast addr) the MAC addr (link
 >  layer addr) of the router is normally sent in the RA which announces
 >  the default router.

 The two connected router interfaces are configured with a shared
 global address supported by VRRP.  That's probably not what you
 refer to above as an IPv6 anycast address.

 >  However, ND should work, and it isn't clear from your tcpdump why
 >  it isn't.   Any chance you could repeat the tcpdump with -v (or
 >  even -vv) to get the whole packet dumped, rather than just a summary
 >  of what the packets are about, which is what the default output
 >  shows.

 I saved the pcap file, here it is decoded with -v -v:

 09:28:30.484990 10:0e:7e:c6:cf:f0 > a4:4e:31:e4:43:a8, ethertype IPv6 (0x86dd), length 110: (hlim 255, next-header ICMPv6 (58) payload length: 56) fe80::200:5eff:fe00:200 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 56
         hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 1800s, reachable time 0s, retrans time 0s
           source link-address option (1), length 8 (1): 00:00:5e:00:02:00
             0x0000:  0000 5e00 0200
           prefix info option (3), length 32 (4): 2001:700:1:21::/64, Flags [onlink], valid time 2592000s, pref. time 604800s
             0x0000:  4080 0027 8d00 0009 3a80 0000 0000 2001
             0x0010:  0700 0001 0021 0000 0000 0000 0000
 09:28:30.552798 10:0e:7e:c6:cb:f0 > a4:4e:31:e4:43:a8, ethertype IPv6 (0x86dd), length 110: (hlim 255, next-header ICMPv6 (58) payload length: 56) fe80::120e:7e00:15c6:cbf0 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 56
         hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 1800s, reachable time 0s, retrans time 0s
           source link-address option (1), length 8 (1): 10:0e:7e:c6:cb:f0
             0x0000:  100e 7ec6 cbf0
           prefix info option (3), length 32 (4): 2001:700:1:21::/64, Flags [onlink], valid time 2592000s, pref. time 604800s
             0x0000:  4080 0027 8d00 0009 3a80 0000 0000 2001
             0x0010:  0700 0001 0021 0000 0000 0000 0000
 09:28:30.575962 10:0e:7e:c6:cb:f0 > a4:4e:31:e4:43:a8, ethertype IPv6 (0x86dd), length 110: (hlim 255, next-header ICMPv6 (58) payload length: 56) fe80::200:5eff:fe00:200 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 56
         hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 1800s, reachable time 0s, retrans time 0s
           source link-address option (1), length 8 (1): 00:00:5e:00:02:00
             0x0000:  0000 5e00 0200
           prefix info option (3), length 32 (4): 2001:700:1:21::/64, Flags [onlink], valid time 2592000s, pref. time 604800s
             0x0000:  4080 0027 8d00 0009 3a80 0000 0000 2001
             0x0010:  0700 0001 0021 0000 0000 0000 0000
 09:28:30.665716 10:0e:7e:c6:cf:f0 > a4:4e:31:e4:43:a8, ethertype IPv6 (0x86dd), length 110: (hlim 255, next-header ICMPv6 (58) payload length: 56) fe80::120e:7e00:15c6:cff0 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 56
         hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 1800s, reachable time 0s, retrans time 0s
           source link-address option (1), length 8 (1): 10:0e:7e:c6:cf:f0
             0x0000:  100e 7ec6 cff0
           prefix info option (3), length 32 (4): 2001:700:1:21::/64, Flags [onlink], valid time 2592000s, pref. time 604800s
             0x0000:  4080 0027 8d00 0009 3a80 0000 0000 2001
             0x0010:  0700 0001 0021 0000 0000 0000 0000
 09:28:31.835893 a4:4e:31:e4:43:a8 > 33:33:ff:c6:cb:f0, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:700:1:21::11bb > ff02::1:ffc6:cbf0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::120e:7e00:15c6:cbf0
           source link-address option (1), length 8 (1): a4:4e:31:e4:43:a8
             0x0000:  a44e 31e4 43a8
 09:28:31.838053 10:0e:7e:c6:cb:f0 > a4:4e:31:e4:43:a8, ethertype IPv6 (0x86dd), length 86: (class 0xc0, hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:700:1:21::2 > 2001:700:1:21::11bb: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fe80::120e:7e00:15c6:cbf0, Flags [router, solicited, override]
           destination link-address option (2), length 8 (1): 10:0e:7e:c6:cb:f0
             0x0000:  100e 7ec6 cbf0
 09:28:32.838212 a4:4e:31:e4:43:a8 > 33:33:ff:c6:cb:f0, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:700:1:21::11bb > ff02::1:ffc6:cbf0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::120e:7e00:15c6:cbf0
           source link-address option (1), length 8 (1): a4:4e:31:e4:43:a8
             0x0000:  a44e 31e4 43a8
 09:28:32.850136 10:0e:7e:c6:cb:f0 > a4:4e:31:e4:43:a8, ethertype IPv6 (0x86dd), length 86: (class 0xc0, hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:700:1:21::2 > 2001:700:1:21::11bb: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fe80::120e:7e00:15c6:cbf0, Flags [router, solicited, override]
           destination link-address option (2), length 8 (1): 10:0e:7e:c6:cb:f0
             0x0000:  100e 7ec6 cbf0
 09:28:33.848998 a4:4e:31:e4:43:a8 > 33:33:ff:c6:cb:f0, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:700:1:21::11bb > ff02::1:ffc6:cbf0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::120e:7e00:15c6:cbf0
           source link-address option (1), length 8 (1): a4:4e:31:e4:43:a8
             0x0000:  a44e 31e4 43a8
 09:28:33.850563 10:0e:7e:c6:cb:f0 > a4:4e:31:e4:43:a8, ethertype IPv6 (0x86dd), length 86: (class 0xc0, hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:700:1:21::2 > 2001:700:1:21::11bb: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fe80::120e:7e00:15c6:cbf0, Flags [router, solicited, override]
           destination link-address option (2), length 8 (1): 10:0e:7e:c6:cb:f0
             0x0000:  100e 7ec6 cbf0

 ...and now when I try to repeat it, of course I can't reproduce
 it at my desk; an entry is now entered in my neighbor cache:

 granlund: {6} ndp -an
 Neighbor                                Linklayer Address  Netif Expire    S Fl
 fe80::200:5eff:fe00:200%iwn0            00:00:5e:00:02:00   iwn0 23h50m12s S 
 granlund: {7}
 granlund: {7} netstat -rn -f inet6 | grep default
 default                                 fe80::200:5eff:fe00:200        UGS         -        -      -  iwn0
 granlund: {8}

 Regards,

 - Havard

From: Robert Elz <kre@munnari.OZ.AU>
To: Havard Eidnes <he@uninett.no>
Cc: gnats-bugs@NetBSD.org, kern-bug-people@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/54057: ipv6 neighbor discovery fails to resolve address
Date: Tue, 12 Mar 2019 22:44:29 +0700

     Date:        Tue, 12 Mar 2019 14:24:47 +0100 (CET)
     From:        Havard Eidnes <he@uninett.no>
     Message-ID:  <20190312.142447.676762096099488390.he@uninett.no>

   | The two connected router interfaces are configured with a shared
   | global address supported by VRRP.  That's probably not what you
   | refer to above as an IPv6 anycast address.

 Probably not (and it appears not from the packet trace you appended
 in this message) but (given that I know nothing about how VRRP works)
 that's the kind of operation where an anycast default router addr
 might make sense.   It all depends upon just how seamless the dual
 (or more) router config is designed to be - ones where it is essentially
 impossible to tell which router is processing the packets would not
 normally use anycast (they tend to need to use the same MAC addr for
 example) less tightly coupled setups might (and the hosts deal with the
 loss of one of the routers, and switch to another, knowing what it is
 they are doing ... which usually means a brief (but detectable) period
 of network outage, but generally designed to be much less than what would
 cause a TCP session to time out - it looks more like a few lost packets.

 All this is much quicker than simply having 2 (or more) independent
 routers, and relying upon RA timeouts to detect a vanished router,
 and pick a new one - that generally takes long enough to reconnect
 that TCP (and many UDP applications, like DNS lookups) has given up
 before the host picks a new default router and can reach the world again.

   | I saved the pcap file,

 That's great - I wish more people would do things that way.   It makes
 it much easier to investigate once you have managed to capture the
 "interesting event".

   | here it is decoded with -v -v:

 I will look at that more carefully tomorrow, but from what I can see
 the router(s) is/are configured to require the hosts to use DHCPv6
 for address assignment (etc).   That will mean that (I assume) dhcpcd
 is configuring the v6 default router - which might be why its link addr
 isn't being configured at the same time (which should be happening if
 stateless autoconfig were being used.)

 I also have a suspicion what might have happened, but unless someone
 else analyses this first (in the next dozen hours or so) I need to look
 at our ND code to see just how it would process that NS/NA sequence.

   | ...and now when I try to repeat it, of course I can't reproduce
   | it at my desk; an entry is now entered in my neighbor cache:

 Of course, that's always the way!   At least that confirms that it
 should work (and generally does work)  we just need to work out why
 it did not work, which is where having that packet trace is so useful.

 kre

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.43 2018/01/16 07:36:43 maya Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2017 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.