NetBSD Problem Report #11916

Received: (qmail 16047 invoked from network); 8 Jan 2001 06:14:08 -0000
Message-Id: <200101080614.f086E7T04160@charm.icompute.com>
Date: Mon, 8 Jan 2001 00:14:07 -0600 (CST)
From: sysadmin@icompute.com
Reply-To: sysadmin@icompute.com
To: gnats-bugs@gnats.netbsd.org
Subject: PPPD fails when re-connecting broken connection
X-Send-Pr-Version: 3.95

>Number:         11916
>Category:       misc
>Synopsis:       PPPD fails to set up routes/interface when re-connecting
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    itojun
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Jan 08 06:15:00 +0000 2001
>Closed-Date:    Mon Apr 30 07:50:18 +0000 2001
>Last-Modified:  Mon Apr 30 07:50:18 +0000 2001
>Originator:     Donald Lee
>Release:        NetBSD 1.3.x, 1.4.x, and 1.5
>Organization:
	icompute.com - small ISP
>Environment:
	Many.  I can reproduce this problem on MacPPC and 68K Macs
	and under 1.3.x, 1.4.x, and 1.5 NetBSD.
System: NetBSD charm 1.5 NetBSD 1.5 (try1) #16: Sun Jan 7 20:28:22 CST 2001 donlee@charm:/usr/src/sys/arch/macppc/compile/try1 macppc


>Description:
	When using pppd in a dial-in mode, pppd needs to set up both
	the pppx interface and a proxy-arp for the ethernet side
	of the connection.

	If the PPP connection is broken for some reason while
	a socket is active, the proxy arp entry will persist
	(probably because the remote connection is still trying to
	use the socket) and this will prevent re-connection of
	the PPP connection on the next dial-up.

	This might seem fairly minor, but is really a pain
	if you are trying to provide a reliable dialup service.
	Any user who rudely disconnects his PPP session leaves
	behind this problem, and there is no way I can find to
	clean it up.

	I've tried connect and disconnect scripts, and they don't do
	the job.
>How-To-Repeat:
	The easiest way to reproduce this is to dial up to NetBSD with
	PPP.  (PPPD doing the serving, of course)  Start an FTP
	transfer to a third machine (some random ftp server that is
	not the same as the pppd host).  In the middle of the transfer,
	disconnect from the ppp server.  (hang up the phone)

	When you try to re-connect (re-dial), you will notice that the
	stock pppd will let you connect, and it will appear to
	work, but you can't route packets between ppp and ethernet.
	Notice that netstat -r shows the remote "PPP" address
	listed as a "link #1" on the ethernet interface, and that
	arp -a lists this address as "incomplete".  (it should be 
	a permanent published  proxy-only arp)

	YOu can restore the dialup to health by killing the
	disconnected ftpd on the remote machine, and by doing
	an "arp -d PPPADDR" on the ppp host.  (and maybe a
	"route delete PPPADDR")
>Fix:
	I have struggled with this for some time, and now have a
	"fix" for it by doing a "system("arp -d HISADDR");" in
	sifaddr() in pppd/sys-bsd.c, but the real fix is
	surely something better.

	It may be that this is *the* fix, but I looked at the
	source to arp, and it looked like too much work to steal
	the code. ;->
>Release-Note:
>Audit-Trail:

From: Jun-ichiro itojun Hagino <itojun@iijlab.net>
To: sysadmin@icompute.com
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: misc/11916
Date: Wed, 10 Jan 2001 13:58:32 +0900

 	since it is pppd which makes proxy ARP entry, i believe it
 	pppd's responsibility to remove proxy ARP entry on failures.
 	you may want to call cifproxyarp() instead of doing system().

 	we need to do more about cleaning up ARP entries that are
 	generated automatically, but that is different topic...

 itojun

From: itojun@iijlab.net
To: gnats-bugs@gnats.netbsd.org
Cc:  
Subject: Re: misc/11916
Date: Sat, 13 Jan 2001 13:14:47 +0900

 ------- Forwarded Messages

 Return-Path: donlee_ppc@icompute.com
 Return-Path: <donlee_ppc@icompute.com>
 Received: from sh1.iijlab.net (sh1.iijlab.net [202.232.15.98])
 	by coconut.itojun.org (8.9.3+3.2W/3.7W) with ESMTP id OAA09287
 	for <itojun@itojun.org>; Wed, 10 Jan 2001 14:24:25 +0900 (JST)
 Received: from icompute.com (mailhost.icompute.com [209.46.51.194])
 	by sh1.iijlab.net (8.9.1+3.1W/3.7W) with ESMTP id OAA19027
 	for <itojun@iijlab.net>; Wed, 10 Jan 2001 14:37:41 +0900 (JST)
 Received: from [209.46.51.197] (209.46.51.197) by icompute.com with ESMTP
  (Eudora Internet Mail Server 2.2.2); Tue, 9 Jan 2001 23:24:30 -0600
 X-Sender: donlee_ppc (Unverified)
 Message-Id: <v03130300b6819e65d9f8@[209.46.51.197]>
 In-Reply-To: <20010110045832.AFBAD7E66@starfruit.itojun.org>
 Mime-Version: 1.0
 Content-Type: text/plain; charset="us-ascii"
 Date: Tue, 9 Jan 2001 23:24:01 -0600
 To: Jun-ichiro itojun Hagino <itojun@iijlab.net>
 From: Donald Lee <donlee_ppc@icompute.com>
 Subject: Re: misc/11916
 Cc: donlee_ppc@icompute.com
 X-Filter: mailagent [version 3.0 PL68] for itojun@itojun.org

 >	since it is pppd which makes proxy ARP entry, i believe it
 >	pppd's responsibility to remove proxy ARP entry on failures.
 >	you may want to call cifproxyarp() instead of doing system().
 >
 >	we need to do more about cleaning up ARP entries that are
 >	generated automatically, but that is different topic...
 >
 >itojun

 Howdy,

 pppd *does* remove its proxy ARP entries.

 The system() is removing an ARP entry that was entered "automatically"
 by the system in response to external network activity between
 ppp sessions.  For
 example, an ftp socket, left open, will keep trying to talk to the
 disconnected ppp interface, and when it does, it seems to create the
 ARP entry that's the problem.

 The problem is that if a (bogus) ARP entry is present for the IP address
 being attached to pppx, then sifaddr will fail, leaving the connection up,
 but unable to pass data.  pppd should be more robust
 in the face of this condition.

 In NetBSD 1.5, the ioctl() that sets up the interface
 in sifaddr() fails *silently* (no error stat, and no log message).
 In 1.3 and 1.4, I modified
 sifaddr() so that if the ioctl(sock, SIOCAIFADDR, xxx) fails, sifaddr
 will return and cause a disconnect.  (this is better, I think)  In 1.5,
 I can't do this, as I can't tell if the call succeeds.  (pr 11924)

 - -dgl-



 ------- Message 2

 To: Donald Lee <donlee_ppc@icompute.com>
 In-reply-to: donlee_ppc's message of Tue, 09 Jan 2001 23:24:01 CST.
       <v03130300b6819e65d9f8@[209.46.51.197]>
 X-Template-Reply-To: itojun@itojun.org
 X-Template-Return-Receipt-To: itojun@itojun.org
 X-PGP-Fingerprint: F8 24 B4 2C 8C 98 57 FD  90 5F B4 60 79 54 16 E2
 Subject: Re: misc/11916
 From: itojun@iijlab.net
 Date: Wed, 10 Jan 2001 15:01:25 +0900
 Message-ID: <9764.979106485@coconut.itojun.org>
 Sender: itojun@coconut.itojun.org

 >pppd *does* remove its proxy ARP entries.
 >
 >The system() is removing an ARP entry that was entered "automatically"
 >by the system in response to external network activity between
 >ppp sessions.  For
 >example, an ftp socket, left open, will keep trying to talk to the
 >disconnected ppp interface, and when it does, it seems to create the
 >ARP entry that's the problem.

 	let me figure it out better.
 	- your ethernet is x.y.z.0/24
 	- ppp router is using x.y.z.1
 	- ppp client A is using x.y.z.2, it gets proxy-arp'ed by ppp router

 	==+=== x.y.z.0/24
 	  |1
 	ppp router
 	  |
 	  |2
 	ppp client A

 >The problem is that if a (bogus) ARP entry is present for the IP address
 >being attached to pppx, then sifaddr will fail, leaving the connection up,
 >but unable to pass data.  pppd should be more robust
 >in the face of this condition.

 	- ppp client A disconnects without running proper steps, arp entry
 	  for x.y.z.2 (on ppp router) will be left
 	- ppp client B comes in, x.y.z.2 gets assigned

 	now, what is the problem?  if x.y.z.2 is again served by ppp router,
 	proxy arp entry should exactly be the same.

 itojun

 ------- Message 3

 Return-Path: donlee_ppc@icompute.com
 Return-Path: <donlee_ppc@icompute.com>
 Received: from sh1.iijlab.net (sh1.iijlab.net [202.232.15.98])
 	by coconut.itojun.org (8.9.3+3.2W/3.7W) with ESMTP id JAA22106
 	for <itojun@itojun.org>; Sat, 13 Jan 2001 09:52:03 +0900 (JST)
 Received: from icompute.com (mailhost.icompute.com [209.46.51.194])
 	by sh1.iijlab.net (8.9.1+3.1W/3.7W) with ESMTP id JAA02457
 	for <itojun@iijlab.net>; Sat, 13 Jan 2001 09:52:12 +0900 (JST)
 Received: from [209.46.51.197] (209.46.51.197) by icompute.com with ESMTP
  (Eudora Internet Mail Server 2.2.2); Fri, 12 Jan 2001 18:52:09 -0600
 X-Sender: donlee_ppc (Unverified)
 Message-Id: <v03130300b6855306f5d1@[209.46.51.197]>
 In-Reply-To: <9764.979106485@coconut.itojun.org>
 References: donlee_ppc's message of Tue, 09 Jan 2001 23:24:01 CST.     
  <v03130300b6819e65d9f8@[209.46.51.197]>
 Mime-Version: 1.0
 Content-Type: text/plain; charset="us-ascii"
 Date: Fri, 12 Jan 2001 18:51:47 -0600
 To: itojun@iijlab.net
 From: Donald Lee <donlee_ppc@icompute.com>
 Subject: Re: misc/11916
 Cc: donlee@icompute.com
 X-Filter: mailagent [version 3.0 PL68] for itojun@itojun.org

 Sir,

 Let me explain:

 >>pppd *does* remove its proxy ARP entries.
 >>
 >>The system() is removing an ARP entry that was entered "automatically"
 >>by the system in response to external network activity between
 >>ppp sessions.  For
 >>example, an ftp socket, left open, will keep trying to talk to the
 >>disconnected ppp interface, and when it does, it seems to create the
 >>ARP entry that's the problem.
 >
 >	let me figure it out better.
 >	- your ethernet is x.y.z.0/24
 >	- ppp router is using x.y.z.1
 >	- ppp client A is using x.y.z.2, it gets proxy-arp'ed by ppp router
 >
 >	==+=== x.y.z.0/24
 >	  |1
 >	ppp router
 >	  |
 >	  |2
 >	ppp client A

 Yes, but we need a third machine server C, at address x.y.z.3 to complete
 the picture.  Ppp client A has connected to server C and is doing a long
 ftp download of 10 Mb of data.

 >>The problem is that if a (bogus) ARP entry is present for the IP address
 >>being attached to pppx, then sifaddr will fail, leaving the connection up,
 >>but unable to pass data.  pppd should be more robust
 >>in the face of this condition.
 >
 >	- ppp client A disconnects without running proper steps, arp entry
 >	  for x.y.z.2 (on ppp router) will be left
 >	- ppp client B comes in, x.y.z.2 gets assigned
 >
 >	now, what is the problem?  if x.y.z.2 is again served by ppp router,
 >	proxy arp entry should exactly be the same.
 >

 No, this is not quite right.

 The ppp client A disconnects, and follows the proper steps.  The proxy ARP is
 removed on ppp router.  All is well.

 The problem is that server C still has a socket open to ppp client A
 (or at least, it *thinks* it does.)  You can see this because ftpd
 on server is still running, and netstat -a still shows open socket
 to ppp client A.  I don't know if this is proper behavior for the ftpd
 server or not, but that's a separate issue.

 I'm speculating on why this happens, but what I _see_ is that a new ARP
 entry is created a few seconds after this happens, and if I manually remove
 that ARP entry, it will be re-created every few seconds, unless I go and kill
 the ftpd process on server C.  My speculation is that server C continues
 to retry packets to ppp client A at MAC address of ppp router because it has
 no way of knowing that the ppp client has disconnected.  When ppp router
 gets the packets at its MAC address, it appears to automatically assume
 that it should have a corresponding ARP entry, and creates it.  However,
 this ARP entry no longer uses pppx interface, but uses ethernet interface.

 For example, if you ping an address from the ppp router, any address in
 x.y.z.*, you'll get a new ARP entry that looks like this (netsat -r output)
 (dialin01 is the culprit, also grace.icompute.com, and my broadcast address:
 209.46.51.207)  All of these routes/ARPs were created simply by
 doing a ping to an address that does not respond.

 Internet:
 Destination        Gateway            Flags     Refs     Use    Mtu  Interface
 default            valor              UGS         3     4620   1500  mc0
 loopback           localhost          UGRS        0        0  33228  lo0
 localhost          localhost          UH          2       12  33228  lo0
 209.46.51.192/28   link#1             UC          0        0   1500  mc0
 dialin01           link#1             UHL         0        1   1500  mc0
 grace.icompute.com link#1             UHL         0        2   1500  mc0
 209.46.51.207      link#1             UHL         0        3   1500  mc0

 charm$ arp -a
 valor (209.46.51.193) at 00:e0:d0:01:06:3c
 hope (209.46.51.194) at 00:00:a5:92:1d:00
 charm.icompute.com (209.46.51.195) at 08:00:07:bc:e7:10
 humor.icompute.com (209.46.51.196) at 00:00:94:b4:8b:61
 joy.icompute.com (209.46.51.197) at 00:05:02:0f:ff:a9
 mercy (209.46.51.198) at 00:a0:59:01:79:cb permanent
 p75.icompute.com (209.46.51.199) at 00:05:02:a0:0a:70
 zest.icompute.com (209.46.51.201) at 02:07:01:0b:ef:88
 dialin01 (209.46.51.202) at (incomplete)
 grace.icompute.com (209.46.51.206) at (incomplete)
 ? (209.46.51.207) at (incomplete)

 This all makes little difference until you try to dial-up with pppd.  The
 problem is that this _bogus_ ARP entry prevents the ioctl() in sifaddr()
 (in sys-bsd.c) from succeeding, which prevents correct operation.

 A correct ARP entry for the ppp client would be proxy only, and
 a correct route has pppx as the interface (and different flags)

 By adding the system("arp -d IP"); right up front in sifaddr(), I've
 worked around this problem.

 Does this make it clearer?

 Thanks for your help,

 - -dgl-



 ------- Message 4

 To: Donald Lee <donlee_ppc@icompute.com>
 cc: donlee@icompute.com
 In-reply-to: donlee_ppc's message of Fri, 12 Jan 2001 18:51:47 CST.
       <v03130300b6855306f5d1@[209.46.51.197]>
 X-Template-Reply-To: itojun@itojun.org
 X-Template-Return-Receipt-To: itojun@itojun.org
 X-PGP-Fingerprint: F8 24 B4 2C 8C 98 57 FD  90 5F B4 60 79 54 16 E2
 Subject: Re: misc/11916
 From: itojun@iijlab.net
 Date: Sat, 13 Jan 2001 13:13:32 +0900
 Message-ID: <23583.979359212@coconut.itojun.org>
 Sender: itojun@coconut.itojun.org

 >This all makes little difference until you try to dial-up with pppd.  The
 >problem is that this _bogus_ ARP entry prevents the ioctl() in sifaddr()
 >(in sys-bsd.c) from succeeding, which prevents correct operation.
 >
 >A correct ARP entry for the ppp client would be proxy only, and
 >a correct route has pppx as the interface (and different flags)
 >
 >By adding the system("arp -d IP"); right up front in sifaddr(), I've
 >worked around this problem.
 >
 >Does this make it clearer?

 	i understood.  the above "incomplete" ARP entries are necessary
 	for ppp router box.  it indicates that there's outgoing IP packet
 	pending on this box (needs to wait for ARP resolution to complete).
 	i believe sys-bsd.c should do better when putting the proxy ARP entry,
 	that is the only way.  the ARP cache entry on the machine C (on
 	ethernet) only triggers the problem, C is not the bad guy.

 	(i'll archive the mail exchange on bug database)

 itojun

 ------- End of Forwarded Messages

State-Changed-From-To: open->feedback 
State-Changed-By: itojun 
State-Changed-When: Fri Jan 26 20:55:41 PST 2001 
State-Changed-Why:  
with latest netbsd-current kernel, it should be okay 
(explicit route setting can override cloned routes - incl. ARP routes) 
Responsible-Changed-From-To: misc-bug-people->itojun 
Responsible-Changed-By: itojun 
Responsible-Changed-When: Fri Jan 26 20:57:12 PST 2001 
Responsible-Changed-Why:  
mine 

From: itojun@itojun.org (Jun-ichiro itojun Hagino)
To: gnats-bugs@gnats.netbsd.org
Cc: sysadmin@icompute.com
Subject: Re: misc/11916
Date: Wed, 18 Apr 2001 11:30:35 +0900 (JST)

 	please let me know if you still have the problem or not.
 	http://www.NetBSD.org/cgi-bin/query-pr-single.pl?number=11916

 itojun
State-Changed-From-To: feedback->closed 
State-Changed-By: itojun 
State-Changed-When: Mon Apr 30 00:50:05 PDT 2001 
State-Changed-Why:  
fixed 
>Unformatted:

Home	PR Database Search