NetBSD Problem Report #57957
From www@netbsd.org Fri Feb 23 21:57:44 2024
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id A3C891A9239
for <gnats-bugs@gnats.NetBSD.org>; Fri, 23 Feb 2024 21:57:44 +0000 (UTC)
Message-Id: <20240223215742.D583D1A923A@mollari.NetBSD.org>
Date: Fri, 23 Feb 2024 21:57:42 +0000 (UTC)
From: campbell+netbsd@mumble.net
Reply-To: campbell+netbsd@mumble.net
To: gnats-bugs@NetBSD.org
Subject: dhcpcd fails to unconfigure addresses from released lease
X-Send-Pr-Version: www-1.0
>Number: 57957
>Category: bin
>Synopsis: dhcpcd fails to unconfigure addresses from released lease
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: roy
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Fri Feb 23 22:00:00 +0000 2024
>Last-Modified: Sun Feb 25 20:00:02 +0000 2024
>Originator: Taylor R Campbell
>Release: netbsd-10
>Organization:
The NetBSDCDCPCHCPR Foundation
>Environment:
NetBSD nanocons.local 10.0_RC3 NetBSD 10.0_RC3 (GENERIC64) #15: Wed Jan 17 05:31:14 UTC 2024 root@manticore.local:/usr/obj/10/evbarm64/sys/arch/evbarm/compile/GENERIC64 evbarm
>Description:
My network had an IPv6 prefix, say 2001:db8:0:1111::/64, delegated from my upstream ISP by DHCPv6-PD.
Downstream to my LAN, my router serves this prefix and default route through SLAAC and DHCPv6.
On my LAN is a NetBSD device whose interface ure0 is configured with dhcpcd, say to 2001:db8:0:1111::1234.
The prefix and prefix length changed upstream, say to 2001:db8:0:2220::/60, so all the downstream DHCPv6 leases stopped working.
DHCP server didn't do FORCERENEW, so I tried `dhcpcd -n ure0' to force renewal, giving ure0 a new address 2001:db8:0:2220::1234. But ure0 still had the old address 2001:db8:0:1111::1234 and kept using it as a source address, so IPv6 connectivity was effectively lost.
Tried `dhcpcd -k ure0', but that's broken (https://gnats.NetBSD.org/57956). Tried `dhcpcd -U ure0' to dump why the old address 2001:db8:0:1111::1234 is still configured, and that failed too.
Ran `dhcpcd -k' to make it release leases on all interfaces (fortunately I still have console access to this machine), and it removed the _new_ (working) address 2001:db8:0:2220::1234 but left the _old_ (now broken) address 2001:db8:0:1111::1234 in place.
>How-To-Repeat:
1. change DHCP server to new configuration
2. dhcpcd -n
3. dhcpcd -k
4. observe old configuration is still there
>Fix:
Yes, please!
>Release-Note:
>Audit-Trail:
From: Taylor R Campbell <riastradh@NetBSD.org>
To: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Cc:
Subject: Re: bin/57957: dhcpcd fails to unconfigure addresses from released lease
Date: Fri, 23 Feb 2024 22:03:04 +0000
Addendum: The old and new addresses were both provided by DHCPv6, not
by SLAAC.
dhcpcd also configured other addresses with SLAAC, but it successfully
deleted the old SLAAC addresses with the old prefix and added new ones
with the new prefix.
Responsible-Changed-From-To: bin-bug-people->roy
Responsible-Changed-By: riastradh@NetBSD.org
Responsible-Changed-When: Fri, 23 Feb 2024 22:05:29 +0000
Responsible-Changed-Why:
over to dhcpcd wizard
From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: bin/57957: dhcpcd fails to unconfigure addresses from released lease
Date: Sun, 25 Feb 2024 16:12:24 +0700
I used to see this quite frequently - it happened when (for whatever
reason) there was no network connectivity at the time the router was
trying to alter the IPv6 addr (ISPs that don't like any customers to
have a stable address for very long!)
If the RA (or sequence of RAs perhaps) indicating that the old address
was deprecated were missed, the old addr would simply hang around
forever, with the effects as stated in the PR. A new one would appear,
but wouldn't be used.
Whatever is managing RAs (dhcpcd now, I suppose) should be looking for
all prefixes being announced in all RAs, and if long enough passes with
a prefix simply missing from the announcements, then drop it.
Of course, it needs to know which addresses (and corresponding prefix)
were assigned this way, not delete addresses which might have been
statically added (someone typing "ifconfig xx0 ..."), and that needs
to happen even if dhcpcd was restarted in the meantime (this was one of
the advantages of kernel processing - if the kernel restarts, all old
addrs (and everything else) simply vanish - dhcpcd restarting doesn't
make that happen, and nor should it).
Note that simply missing from one RA is not enough - there might be
more than one router on a LAN advertising different prefixes, or a
router might not announce everything in every RA.
kre
ps: What I used to do when this happened was to use ifconfig and simply
remove the bad address, after which it all recovers - but naturally that
means having a method to access the system, distinct from the badly
configured network interface.
From: Roy Marples <roy@marples.name>
To: "gnats-bugs" <gnats-bugs@netbsd.org>
Cc: "Robert Elz" <kre@munnari.OZ.AU>, "gnats-admin" <gnats-admin@netbsd.org>,
"netbsd-bugs" <netbsd-bugs@netbsd.org>,
"campbell+netbsd" <campbell+netbsd@mumble.net>
Subject: Re: bin/57957: dhcpcd fails to unconfigure addresses from released
lease
Date: Sun, 25 Feb 2024 12:18:03 +0000
---- On Sun, 25 Feb 2024 09:15:02 +0000 Robert Elz wrote ---
> I used to see this quite frequently - it happened when (for whatever
> reason) there was no network connectivity at the time the router was
> trying to alter the IPv6 addr (ISPs that don't like any customers to
> have a stable address for very long!)
>
> If the RA (or sequence of RAs perhaps) indicating that the old address
> was deprecated were missed, the old addr would simply hang around
> forever, with the effects as stated in the PR. A new one would appear,
> but wouldn't be used.
>
> Whatever is managing RAs (dhcpcd now, I suppose) should be looking for
> all prefixes being announced in all RAs, and if long enough passes with
> a prefix simply missing from the announcements, then drop it.
>
> Of course, it needs to know which addresses (and corresponding prefix)
> were assigned this way, not delete addresses which might have been
> statically added (someone typing "ifconfig xx0 ..."), and that needs
> to happen even if dhcpcd was restarted in the meantime (this was one of
> the advantages of kernel processing - if the kernel restarts, all old
> addrs (and everything else) simply vanish - dhcpcd restarting doesn't
> make that happen, and nor should it).
>
> Note that simply missing from one RA is not enough - there might be
> more than one router on a LAN advertising different prefixes, or a
> router might not announce everything in every RA.
>
> kre
>
> ps: What I used to do when this happened was to use ifconfig and simply
> remove the bad address, after which it all recovers - but naturally that
> means having a method to access the system, distinct from the badly
> configured network interface.
If dhcpcd received an address via DHCPv6 or RA with a lifetime of zero then it is removed from the interface.
Otherwise dhcpcd leaves it alone and the address will naturally expire.
This behaviour is pretty well documented in then RFCs.
So the only issue left is one of source address selection.
There is a bug where if the source address is the only thing to change on the route then the kernel route isn't updated with the new source address.
This is being tracked upstream here:
https://github.com/NetworkConfiguration/dhcpcd/issues/300
Roy
From: Robert Elz <kre@munnari.OZ.AU>
To: Roy Marples <roy@marples.name>
Cc: "gnats-bugs" <gnats-bugs@netbsd.org>,
"gnats-admin" <gnats-admin@netbsd.org>,
"netbsd-bugs" <netbsd-bugs@netbsd.org>,
"campbell+netbsd" <campbell+netbsd@mumble.net>
Subject: Re: bin/57957: dhcpcd fails to unconfigure addresses from released lease
Date: Sun, 25 Feb 2024 21:57:57 +0700
Date: Sun, 25 Feb 2024 12:18:03 +0000
From: Roy Marples <roy@marples.name>
Message-ID: <18de0336fed.c6fdde1d1909855.1370758253774638123@marples.=
name>
| If dhcpcd received an address via DHCPv6 or RA with a lifetime
| of zero then it is removed from the interface.
Yes, that part is/was working fine - the normal case caused no
issues at all. (At one time I was getting addr changes every
day - now they're reduced to one a week I think, so the chances
to observe this have decreases a lot, and my general net stability
is much better now, so things rarely get lost).
| Otherwise dhcpcd leaves it alone and the address will naturally expire=
.
But that one didn't seem to. There was no apparent expiration happening
at all (and at times this was after many hours).
| This behaviour is pretty well documented in then RFCs.
Yes, what should happen is clear - the question is more what
does happen.
| So the only issue left is one of source address selection.
Well, yes, and no. If the old addr was just left sitting there,
and the replacement used for the source addr, then probably no-one
would really ever notice (though if the old addresses built up over
time it would start getting annoying, eventually).
But that would still be wrong, the old addresses should expire,
but weren't (and apparently didn't for Taylor either).
kre
From: Taylor R Campbell <riastradh@NetBSD.org>
To: Robert Elz <kre@munnari.OZ.AU>
Cc: gnats-bugs@NetBSD.org, roy@NetBSD.org, gnats-admin@NetBSD.org,
netbsd-bugs@NetBSD.org
Subject: Re: bin/57957: dhcpcd fails to unconfigure addresses from released lease
Date: Sun, 25 Feb 2024 15:13:03 +0000
> Date: Sun, 25 Feb 2024 15:05:01 +0000 (UTC)
> From: Robert Elz <kre@munnari.OZ.AU>
>
> But that would still be wrong, the old addresses should expire,
> but weren't (and apparently didn't for Taylor either).
Just to clarify: the issue I observed is that the old DHCPv6-assigned
addresses weren't removed (and kept getting used as default source
addresses). The SLAAC addresses with the old prefix, in contrast, did
get removed and replaced by SLAAC addresses with the new prefix.
From: Roy Marples <roy@marples.name>
To: "Taylor R Campbell" <riastradh@NetBSD.org>
Cc: "Robert Elz" <kre@munnari.OZ.AU>, "gnats-bugs" <gnats-bugs@NetBSD.org>,
"roy" <roy@NetBSD.org>, "gnats-admin" <gnats-admin@NetBSD.org>,
"netbsd-bugs" <netbsd-bugs@NetBSD.org>
Subject: Re: bin/57957: dhcpcd fails to unconfigure addresses from released
lease
Date: Sun, 25 Feb 2024 16:28:33 +0000
---- On Sun, 25 Feb 2024 15:13:03 +0000 Taylor R Campbell wrote ---
> > Date: Sun, 25 Feb 2024 15:05:01 +0000 (UTC)
> > From: Robert Elz kre@munnari.OZ.AU>
> >
> > But that would still be wrong, the old addresses should expire,
> > but weren't (and apparently didn't for Taylor either).
>
> Just to clarify: the issue I observed is that the old DHCPv6-assigned
> addresses weren't removed.
Why should they be? They obey the same rules as RA addresses in that removal is either manual or a zero lifetime is emitted.
Now IA options can include a status code such as NotOnLink, but there is no indication in the RFC that any existing address should be removed.
Infact it's recommendation to enter discovery mode and find a new DHCPv6 server.
Is the lifetime of the address continously decreasing or does it ever increase?
> (and kept getting used as default source
> addresses).
I find this statement odd because DHCPv6 addresses are not tied to any route.
Infact there is zero routing in DHCPv6 because RA covers it nicely.
> The SLAAC addresses with the old prefix, in contrast, did
> get removed and replaced by SLAAC addresses with the new prefix.
The error in dhcpcd is that when a prefix derived route changes it's address the source address in the route is not changed.
But I don't think this is what you are seeing.
From: Roy Marples <roy@marples.name>
To: "Robert Elz" <kre@munnari.OZ.AU>
Cc: "gnats-bugs" <gnats-bugs@netbsd.org>,
"gnats-admin" <gnats-admin@netbsd.org>,
"netbsd-bugs" <netbsd-bugs@netbsd.org>,
"campbell+netbsd" <campbell+netbsd@mumble.net>
Subject: Re: bin/57957: dhcpcd fails to unconfigure addresses from released
lease
Date: Sun, 25 Feb 2024 16:31:10 +0000
---- On Sun, 25 Feb 2024 14:57:57 +0000 Robert Elz wrote ---
> Date: Sun, 25 Feb 2024 12:18:03 +0000
> From: Roy Marples roy@marples.name>
> Message-ID: .1370758253774638123@marples.name>
>
> | If dhcpcd received an address via DHCPv6 or RA with a lifetime
> | of zero then it is removed from the interface.
>
> Yes, that part is/was working fine - the normal case caused no
> issues at all. (At one time I was getting addr changes every
> day - now they're reduced to one a week I think, so the chances
> to observe this have decreases a lot, and my general net stability
> is much better now, so things rarely get lost).
>
> | Otherwise dhcpcd leaves it alone and the address will naturally expire.
>
> But that one didn't seem to. There was no apparent expiration happening
> at all (and at times this was after many hours).
>
> | This behaviour is pretty well documented in then RFCs.
>
> Yes, what should happen is clear - the question is more what
> does happen.
>
> | So the only issue left is one of source address selection.
>
> Well, yes, and no. If the old addr was just left sitting there,
> and the replacement used for the source addr, then probably no-one
> would really ever notice (though if the old addresses built up over
> time it would start getting annoying, eventually).
>
> But that would still be wrong, the old addresses should expire,
> but weren't (and apparently didn't for Taylor either).
So is the address lifetime continously decreasing or does it ever increase?
Maybe it's just a long lifetime of months.
No-one has posted yet what their expected address lifetimes are and what they see from ifconfig.
Roy
From: Roy Marples <roy@marples.name>
To: "Robert Elz" <kre@munnari.OZ.AU>
Cc: "gnats-bugs" <gnats-bugs@netbsd.org>,
"gnats-admin" <gnats-admin@netbsd.org>,
"netbsd-bugs" <netbsd-bugs@netbsd.org>,
"campbell+netbsd" <campbell+netbsd@mumble.net>
Subject: Re: bin/57957: dhcpcd fails to unconfigure addresses from released
lease
Date: Sun, 25 Feb 2024 16:31:10 +0000
---- On Sun, 25 Feb 2024 14:57:57 +0000 Robert Elz wrote ---
> Date: Sun, 25 Feb 2024 12:18:03 +0000
> From: Roy Marples roy@marples.name>
> Message-ID: .1370758253774638123@marples.name>
>
> | If dhcpcd received an address via DHCPv6 or RA with a lifetime
> | of zero then it is removed from the interface.
>
> Yes, that part is/was working fine - the normal case caused no
> issues at all. (At one time I was getting addr changes every
> day - now they're reduced to one a week I think, so the chances
> to observe this have decreases a lot, and my general net stability
> is much better now, so things rarely get lost).
>
> | Otherwise dhcpcd leaves it alone and the address will naturally expire.
>
> But that one didn't seem to. There was no apparent expiration happening
> at all (and at times this was after many hours).
>
> | This behaviour is pretty well documented in then RFCs.
>
> Yes, what should happen is clear - the question is more what
> does happen.
>
> | So the only issue left is one of source address selection.
>
> Well, yes, and no. If the old addr was just left sitting there,
> and the replacement used for the source addr, then probably no-one
> would really ever notice (though if the old addresses built up over
> time it would start getting annoying, eventually).
>
> But that would still be wrong, the old addresses should expire,
> but weren't (and apparently didn't for Taylor either).
So is the address lifetime continously decreasing or does it ever increase?
Maybe it's just a long lifetime of months.
No-one has posted yet what their expected address lifetimes are and what they see from ifconfig.
Roy
From: Taylor R Campbell <riastradh@NetBSD.org>
To: Roy Marples <roy@marples.name>
Cc: "Robert Elz" <kre@munnari.OZ.AU>,
gnats-bugs@NetBSD.org,
gnats-admin@NetBSD.org,
netbsd-bugs@NetBSD.org
Subject: Re: bin/57957: dhcpcd fails to unconfigure addresses from released
lease
Date: Sun, 25 Feb 2024 19:58:54 +0000
> Date: Sun, 25 Feb 2024 16:31:10 +0000
> From: Roy Marples <roy@marples.name>
>
> No-one has posted yet what their expected address lifetimes are and
> what they see from ifconfig.
The main point of this PR is that even after I asked dhcpcd to release
_everything_ with `dhcpcd -k' (no interface argument, because of
https://gnats.NetBSD.org/57956), it deleted all addresses _except_ for
the old DHCPv6 addresses.
That is, before all this, ifconfig showed something like this, with
everything under the old prefix 2001:db8:0:1111::/64 (sorry,
this is from memory, I don't still have the console transcript):
ure0: ...
...
inet6 fe80::${URE_LLA}%ure0/64 flags 0 scopeid 0x2
inet6 2001:db8:0:1111:${URE_SLAAC_GUA}/64 flags 0x40<AUTOCONF>
inet6 fd...${URE_SLAAC_ULA}/64 flags 0x40<AUTOCONF>
inet6 2001:db8:0:1111::1234/128 flags 0
inet6 fd...:1234/128 flags 0
...
dhcpcd -n successfully added the new address 2001:db8:0:2220::1234, so
it looked something like this (not sure of the order of addresses, not
100% sure it excluded the old SLAAC addresses):
ure0: ...
...
inet6 fe80::${URE_LLA}%ure0/64 flags 0 scopeid 0x2
inet6 2001:db8:0:2220:${URE_SLAAC_GUA}/64 flags 0x40<AUTOCONF>
inet6 fd...${URE_SLAAC_ULA}/64 flags 0x40<AUTOCONF>
inet6 2001:db8:0:1111::1234/128 flags 0
inet6 2001:db8:0:2220::1234/128 flags 0
inet6 fd...:1234/128 flags 0
...
Then when I ran dhcpcd -k, ifconfig showed this:
ure0: ...
...
inet6 fe80::${URE_LLA}%ure0/64 flags 0 scopeid 0x2
inet6 2001:db8:0:1111::1234/128 flags 0
...
(I forget whether it also had fd...:1234 left, sorry. But it
definitely didn't have any SLAAC addresses, or any DHCPv6 addresses
with the new prefix.)
At this point, dhcpcd was not running, and yet ure0 still had the old
address. But it did not have the new address.
So `dhcpcd -k' succeeded in removing the new address, but failed to
remove the old address. I expected it would remove _all_ addresses
that dhcpcd had configured for the interface.
(Now, another issue is why did the system continue to use the old
address as a source address, losing my IPv6 connectivity? Also to be
investigated, and perhaps related -- perhaps dhcpcd forgot about it
and treated it as a manually configured address and somehow that led
to its being the default source address -- but let's keep this PR
focussed on how `dhcpcd -k' failed to unconfigure addresses it had
previously configured.)
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.47 2022/09/11 19:34:41 kim Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2024
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home