NetBSD Problem Report #48945
From www@NetBSD.org Tue Jun 24 09:02:05 2014
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
by mollari.NetBSD.org (Postfix) with ESMTPS id 132B0A5D59
for <gnats-bugs@gnats.NetBSD.org>; Tue, 24 Jun 2014 09:02:05 +0000 (UTC)
Message-Id: <20140624090203.81DABA6546@mollari.NetBSD.org>
Date: Tue, 24 Jun 2014 09:02:03 +0000 (UTC)
From: netbsd@seirios.org
Reply-To: netbsd@seirios.org
To: gnats-bugs@NetBSD.org
Subject: CARP preempt is not working
X-Send-Pr-Version: www-1.0
>Number: 48945
>Category: kern
>Synopsis: CARP preempt is not working
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: kern-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Tue Jun 24 09:05:00 +0000 2014
>Last-Modified: Fri Jul 04 08:50:00 +0000 2014
>Originator: HEO SeonMeyong
>Release: NetBSD 6.1 stable
>Organization:
>Environment:
NetBSD slb001 6.1_STABLE NetBSD 6.1_STABLE (XEN3_DOMU) #0: Tue Jun 24 15:16:34 JST 2014 seirios@dev001:/export/NetBSD/src-rel/sys/arch/amd64/compile/XEN3_DOMU amd64
>Description:
- CARP preempt is not working.
1. Environment
There are two NetBSD router (on XenServer)
- Kernel is XEN3_DOMU + with followings.
pseudo-device carp # Common Address Redundancy Protocol
options BRIDGE_IPF # bridge uses IP/IPv6 pfil hooks too
pseudo-device pf # PF packet filter
pseudo-device pflog # PF log if
The Router has two Interfaces connect to Net-A and Net-B
Router-A(rt-A)'s xennet0 and Router-B(rt-B)'s xennet0 is connect to Net-A
Router-A(rt-A)'s xennet1 and Router-B(rt-B)'s xennet1 is connect to Net-B
2. Configuration
Both Router is configured followings.
sysctl net.inet.ip.forwarding=1
sysctl net.inet.carp.allow=1
sysctl net.inet.carp.preempt=1
sysctl net.inet.carp.log=1
rt-A
ifconfig xennet0 inet 192.0.2.2/24
ifconfig carp0 vhid 200 advbase 1 advskew 100 pass testTEST 192.0.2.1/24
ifconfig xennet1 inet 198.51.100.2/24
ifconfig carp1 vhid 201 advbase 1 advskew 100 pass TESTtest 198.51.100.1/24
rt-B
ifconfig xennet0 inet 192.0.2.3/24
ifconfig carp0 vhid 200 advbase 1 advskew 100 pass testTEST 192.0.2.1/24
ifconfig xennet1 inet 198.51.100.3/24
ifconfig carp1 vhid 201 advbase 1 advskew 100 pass TESTtest 198.51.100.1/24
3. How NOT works
- Initial rt-A's carp0 and carp1 is MASTER / rt-B's carp0 and carp1 is BACKUP
- run "ifconfig carp0 down" on rt-A
- rt-A's carp0's state change to INIT but carp1's state is still MASTER
- rt-B's carp0's state change to MASTER but carp1's state is still MASTER
I think this behavior is bug. rt-A's carp1's state is change to BACKUP when CARP preempt=1
>How-To-Repeat:
See Full description.
>Fix:
>Audit-Trail:
From: Manuel Bouyer <bouyer@antioche.eu.org>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@NetBSD.org, gnats-admin@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: kern/48945: CARP preempt is not working
Date: Tue, 24 Jun 2014 12:12:00 +0200
On Tue, Jun 24, 2014 at 09:05:00AM +0000, netbsd@seirios.org wrote:
> >Number: 48945
> >Category: kern
> >Synopsis: CARP preempt is not working
> >Confidential: no
> >Severity: serious
> >Priority: high
> >Responsible: kern-bug-people
> >State: open
> >Class: sw-bug
> >Submitter-Id: net
> >Arrival-Date: Tue Jun 24 09:05:00 +0000 2014
> >Originator: HEO SeonMeyong
> >Release: NetBSD 6.1 stable
> >Organization:
> >Environment:
> NetBSD slb001 6.1_STABLE NetBSD 6.1_STABLE (XEN3_DOMU) #0: Tue Jun 24 15:16:34 JST 2014 seirios@dev001:/export/NetBSD/src-rel/sys/arch/amd64/compile/XEN3_DOMU amd64
> >Description:
> - CARP preempt is not working.
>
> 1. Environment
> There are two NetBSD router (on XenServer)
> - Kernel is XEN3_DOMU + with followings.
> pseudo-device carp # Common Address Redundancy Protocol
> options BRIDGE_IPF # bridge uses IP/IPv6 pfil hooks too
> pseudo-device pf # PF packet filter
> pseudo-device pflog # PF log if
> The Router has two Interfaces connect to Net-A and Net-B
> Router-A(rt-A)'s xennet0 and Router-B(rt-B)'s xennet0 is connect to Net-A
> Router-A(rt-A)'s xennet1 and Router-B(rt-B)'s xennet1 is connect to Net-B
>
> 2. Configuration
> Both Router is configured followings.
> sysctl net.inet.ip.forwarding=1
> sysctl net.inet.carp.allow=1
> sysctl net.inet.carp.preempt=1
> sysctl net.inet.carp.log=1
>
> rt-A
> ifconfig xennet0 inet 192.0.2.2/24
> ifconfig carp0 vhid 200 advbase 1 advskew 100 pass testTEST 192.0.2.1/24
> ifconfig xennet1 inet 198.51.100.2/24
> ifconfig carp1 vhid 201 advbase 1 advskew 100 pass TESTtest 198.51.100.1/24
>
> rt-B
> ifconfig xennet0 inet 192.0.2.3/24
> ifconfig carp0 vhid 200 advbase 1 advskew 100 pass testTEST 192.0.2.1/24
> ifconfig xennet1 inet 198.51.100.3/24
> ifconfig carp1 vhid 201 advbase 1 advskew 100 pass TESTtest 198.51.100.1/24
>
> 3. How NOT works
> - Initial rt-A's carp0 and carp1 is MASTER / rt-B's carp0 and carp1 is BACKUP
> - run "ifconfig carp0 down" on rt-A
> - rt-A's carp0's state change to INIT but carp1's state is still MASTER
> - rt-B's carp0's state change to MASTER but carp1's state is still MASTER
>
> I think this behavior is bug. rt-A's carp1's state is change to BACKUP when CARP preempt=1
No, I think it's correct. Preemption does indeed work (I have a similiar setup,
but with ~30 interfaces) and the backup router does take the traffic
if one of the master's interface goes down.
rt-A's carp1 is still master because the CARP protocol says it should
still be master (carp0 would remain master too if you disconnected the
interface from the brdige in dom0 instead of taking it down). preemp
only force backup->master transition but not the other way round
(that would be dangerous, you could end up with all interfaces in backup state
on both routers).
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
From: HEO SeonMeyong <netbsd@seirios.org>
To: gnats-bugs@NetBSD.org, bouyer@antioche.eu.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, netbsd@seirios.org
Subject: Re: kern/48945: CARP preempt is not working
Date: Wed, 25 Jun 2014 12:16:40 +0900 (JST)
Hi bouyer.
Thank you very much for your review.
bouyer> No, I think it's correct. Preemption does indeed work (I have a similiar setup,
bouyer> but with ~30 interfaces) and the backup router does take the traffic
bouyer> if one of the master's interface goes down.
bouyer> rt-A's carp1 is still master because the CARP protocol says it should
bouyer> still be master (carp0 would remain master too if you disconnected the
bouyer> interface from the brdige in dom0 instead of taking it down). preemp
bouyer> only force backup->master transition but not the other way round
I understand. I thought that CARP watchs CARP state and Interface
state. but you sey CARP watches tied Interface state and not CARP
I/F state.
bouyer> (that would be dangerous, you could end up with all interfaces in backup state
bouyer> on both routers).
Followings are maybe off topic, sorry.
I want to this works. I wrote rt-A/rt-B is a router, but in my real
environment, rt-A and rt-B is router with Firewall(pf) and
IDS(snort).
So if rt-A and rt-B is asynmetric, pf and snort works limited
because (for ex) Incomming traffic is pass through rt-A and outgoing
traffic is pass through rt-B.
I think(or hope) pfsync is avoidance of this limitation, but snort
has no avoidance method.
I thought it is very few situation that Physical I/F is up and carp
I/F is down. But Operator can down CARP I/F, so I want force
preemption method.
Anyway, thank you very much for your review, again.
HEO
From: HEO SeonMeyong <netbsd@seirios.org>
To: gnats-bugs@NetBSD.org, bouyer@antioche.eu.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, netbsd@seirios.org
Subject: Re: kern/48945: CARP preempt is not working
Date: Wed, 25 Jun 2014 12:16:40 +0900 (JST)
Hi bouyer.
Thank you very much for your review.
bouyer> No, I think it's correct. Preemption does indeed work (I have a similiar setup,
bouyer> but with ~30 interfaces) and the backup router does take the traffic
bouyer> if one of the master's interface goes down.
bouyer> rt-A's carp1 is still master because the CARP protocol says it should
bouyer> still be master (carp0 would remain master too if you disconnected the
bouyer> interface from the brdige in dom0 instead of taking it down). preemp
bouyer> only force backup->master transition but not the other way round
I understand. I thought that CARP watchs CARP state and Interface
state. but you sey CARP watches tied Interface state and not CARP
I/F state.
bouyer> (that would be dangerous, you could end up with all interfaces in backup state
bouyer> on both routers).
Followings are maybe off topic, sorry.
I want to this works. I wrote rt-A/rt-B is a router, but in my real
environment, rt-A and rt-B is router with Firewall(pf) and
IDS(snort).
So if rt-A and rt-B is asynmetric, pf and snort works limited
because (for ex) Incomming traffic is pass through rt-A and outgoing
traffic is pass through rt-B.
I think(or hope) pfsync is avoidance of this limitation, but snort
has no avoidance method.
I thought it is very few situation that Physical I/F is up and carp
I/F is down. But Operator can down CARP I/F, so I want force
preemption method.
Anyway, thank you very much for your review, again.
HEO
From: Manuel Bouyer <bouyer@antioche.eu.org>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@NetBSD.org, gnats-admin@NetBSD.org, netbsd-bugs@NetBSD.org,
netbsd@seirios.org
Subject: Re: kern/48945: CARP preempt is not working
Date: Wed, 25 Jun 2014 09:49:20 +0200
On Wed, Jun 25, 2014 at 03:25:01AM +0000, HEO SeonMeyong wrote:
> [...]
> bouyer> (that would be dangerous, you could end up with all interfaces in backup state
> bouyer> on both routers).
>
> Followings are maybe off topic, sorry.
>
> I want to this works. I wrote rt-A/rt-B is a router, but in my real
> environment, rt-A and rt-B is router with Firewall(pf) and
> IDS(snort).
> So if rt-A and rt-B is asynmetric, pf and snort works limited
> because (for ex) Incomming traffic is pass through rt-A and outgoing
> traffic is pass through rt-B.
this is what I don't get; why would traffic go to rt-B if rt-A is up ?
And if rt-A is down, traffic won't go to it (there may be some time before
the traffic switches from A to B while the switch's commutation table is
updated).
I have a setup similar to yours, and AFAIK if an interface on rt-A goes
down, all traffic is redirected to rt-B.
> I think(or hope) pfsync is avoidance of this limitation, but snort
> has no avoidance method.
in my setup both routers are stateless: ipf rules are stateless
(well, almost, there's some state for some UDP traffic but it's not a
big deal to loose a few packets here) and they don't do anything else.
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
From: HEO SeonMeyong <netbsd@seirios.org>
To: bouyer@antioche.eu.org
Cc: gnats-bugs@NetBSD.org, kern-bug-people@NetBSD.org,
gnats-admin@NetBSD.org, netbsd-bugs@NetBSD.org, netbsd@seirios.org
Subject: Re: kern/48945: CARP preempt is not working
Date: Fri, 04 Jul 2014 17:27:07 +0900 (JST)
Hi bouyer.
I'm very thanks for your reply, and sorry for late reply.
bouyer> > bouyer> (that would be dangerous, you could end up with all interfaces in backup state
bouyer> > bouyer> on both routers).
bouyer> >
bouyer> > Followings are maybe off topic, sorry.
bouyer> >
bouyer> > I want to this works. I wrote rt-A/rt-B is a router, but in my real
bouyer> > environment, rt-A and rt-B is router with Firewall(pf) and
bouyer> > IDS(snort).
bouyer> > So if rt-A and rt-B is asynmetric, pf and snort works limited
bouyer> > because (for ex) Incomming traffic is pass through rt-A and outgoing
bouyer> > traffic is pass through rt-B.
bouyer>
bouyer> this is what I don't get; why would traffic go to rt-B if rt-A is up ?
bouyer> And if rt-A is down, traffic won't go to it (there may be some time before
bouyer> the traffic switches from A to B while the switch's commutation table is
bouyer> updated).
bouyer> I have a setup similar to yours, and AFAIK if an interface on rt-A goes
bouyer> down, all traffic is redirected to rt-B.
I found following scenario.
0. Topology
carp0 carp1
xennet0 xennet1
+--- rt-A ---+
Term-A ---+ +--- Term-B
+--- rt-B ---+
Term-A: 10.0.0.10/24 default route is 10.0.0.1(carp0's Addr)
carp0 : 10.0.0.1/24
rt-A : xennet0: 10.0.0.2/24
rt-B : xennet0: 10.0.0.3/24
Term-B: 10.1.1.10/24 default route is 10.1.1.1(carp1's Addr)
carp0 : 10.1.1.1/24
rt-A : xennet1: 10.1.1.2/24
rt-B : xennet1: 10.1.1.3/24
1. rt-A's carp0: advskew 100, tied to xennet0 -> MASTER has 10.0.0.1
carp1: advskew 100, tied to xennet1 -> MASTER has 10.1.1.1
rt-B's carp0: advskew 150, tied to xennet0 -> BACKUP
carp1: advskew 150, tied to xennet1 -> BACKUP
2. I did followings on rt-A
ifconfig carp0 down
then state is followings
rt-A's carp0: down -> BACKUP
carp1: up -> MASTER : has 10.0.0.1
rt-B's carp0: up -> MASTER : has 10.1.1.1
carp1: up -> BACKUP
3. Ping from Term-A to Term-B
for go: Term-A -> rt-B's carp0 -> rt-B's xennet1 -> Term-B
go back: Term-B -> rt-A's carp1 -> rt-A's xennet0 -> Term-A
This is because rt-A's carp0 is down but carp1 is up.
Of cource this is worst case because this is rare case that xennet0
has no trouble but carp0 is down.
As you say if an interdace on rt-A goes down, all traffic is
redirected to rt-B because if Physical interface(or interface which
tied to carp interface) is down, preempting is work.
"Now" I thought too much worried of my senario, but it will be able
to happen.
Thanks again.
HEO
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home