NetBSD Problem Report #27164
Received: (qmail 28607 invoked by uid 605); 6 Oct 2004 15:06:55 -0000
Message-Id: <200410061506.i96F6boW001727@heiligenberg.nt.e-technik.tu-darmstadt.de>
Date: Wed, 6 Oct 2004 17:06:37 +0200 (CEST)
From: Hauke Fath <hf@spg.tu-darmstadt.de>
Sender: gnats-bugs-owner@NetBSD.org
Reply-To: Hauke Fath <hf@spg.tu-darmstadt.de>
To: gnats-bugs@gnats.NetBSD.org
Cc: Hauke Fath <hf@spg.tu-darmstadt.de>
Subject: ipfilter 'keep state' blocking legal connections
X-Send-Pr-Version: 3.95
>Number: 27164
>Category: kern
>Synopsis: ipfilter 'keep state' blocking legal connections
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: ipf-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Wed Oct 06 15:07:01 +0000 2004
>Closed-Date:
>Last-Modified: Mon Dec 23 17:45:25 +0000 2013
>Originator: Hauke Fath <hf@spg.tu-darmstadt.de>
>Release: NetBSD 1.6ZG
>Organization:
--
/~\ The ASCII Ribbon Campaign Hauke Fath
\ / No HTML/RTF in email Institut für Nachrichtentechnik
X No Word docs in email TU Darmstadt
/ \ Respect for open standards Ruf +49-6151-16-3281
>Environment:
System: NetBSD fifi 1.6ZG NetBSD 1.6ZG (FIFI) #0: Mon Dec 22 14:41:32 CET 2003 hf@heiligenberg:/var/obj/netbsd-builds/i386/obj/sys/arch/i386/compile/FIFI i386
Architecture: i386
Machine: i386
>Description:
Given the setup (4 interfaces, 4 subnets)
< DMZ >------+
|
|
<clients>-------------< ipf router >------------------/INTERNET/
|
|
<Internal Servers>------+
where <clients> can access a machine in <internal servers> ("IS") for NIS
services and NFS shares.
The NFS server is running NetBSD 2.0_beta, the clients are NetBSD
1.6.2 (upgraded since), NetBSD 2.0_beta, stock RedHat 8 (linux
2.4.18-27.8.0) + 9 (linux 2.4.20-8 / 2.4.27) installations.
The <clients> can access IS via rules like
pass in log quick on wm1 all keep frags head 100
pass in quick proto tcp from <clients> to any flags S keep state keep frags group 100
pass in quick proto udp from <clients> to any keep state keep frags group 100
pass in quick proto icmp from <clients> to any keep state keep frags group 100
<clients> access to IS works fine for YP. NetBSD clients work fine with the IS nfs server.
The five RedHat <clients>, OTOH, work fine *most of the time*. Then,
often after a user login and associated amd mount, a RH machine will
lock up. The ipfilter log on the router will show lots of entries
like
00:03:06.688586 wm0 @0:7 b SERVER[130.83.xx.yy],nfs -> CLIENT[130.83.uu.vv],800 PR tcp len 20 52 -A IN
for nfs3/tcp connections, and
18:01:32.254120 2x wm1 @0:5 b SERVER[130.83.xx.yy] -> CLIENT[130.83.uu.vv] PR udp len 20 (1500) frag +1480@1480 OUT
18:01:32.254470 13x wm0 @0:7 b SERVER[130.83.xx.yy] -> CLIENT[130.83.uu.vv] PR udp len 20 (1500) frag +1480@1480 IN
for nfs2/udp mounts. The @0:[57] rules are "block {in,out} log quick"
style head rules.
At this point, rebooting client or server machine
does not change anything. Leaving the client powered off for a night,
OTOH, re-enables client access - until eventually the problem kicks in
again.
Monitoring the connection on the client-side switch (ethereal) shows
repeated attempts to contact the server (nfs3/tcp) that are never
acknowledged (tcpdump protocol available).
A typical ipfstate on the router today gives
IPv6 packets: in 50 out 62
input packets: blocked 1381539 passed 600800307 nomatch 50 counted 143525206 short 0
output packets: blocked 14689 passed 600802869 nomatch 62 counted 192702497 short 0
input packets logged: blocked 1368453 passed 8678
output packets logged: blocked 14689 passed 9978
packets logged: input 0 output 0
log failures: input 37670 output 2937
fragment state(in): kept 422488 lost 122032
fragment state(out): kept 422130 lost 125298
packet state(in): kept 5311041 lost 16173
packet state(out): kept 2562221 lost 23665
ICMP replies: 92878 TCP RSTs sent: 742704
Invalid source(in): 0
Result cache hits(in): 9550076 (out): 9273810
IN Pullups succeeded: 0 failed: 0
OUT Pullups succeeded: 0 failed: 0
Fastroute successes: 835582 failures: 0
TCP cksum fails(in): 4 (out): 32
Packet log flags set: (0)
none
Apart from this nfs problem, both netbsd-netbsd nfs connections and
any non-nfs connections involving RH machines have been working fine
during the nine months that the router is running now.
>How-To-Repeat:
Set up a NetBSD 2 nfs/nis server and RedHat 8/9 client
machines on two sides of an ipfilter based router like
pictured above.
Find that eventually the RH clients wil hang, and rebooting
them won't help. Find a lot of log entries on the router.
>Fix:
The workaround I found was to set up vanilla 'pass
all' rules (2 directions times 2 interfaces) *without* keeping
state. Anything *with* keep state between RH nfs clients and
the NetBSD nfs server would eventually trigger the problem.
While this kludge works for me in the current situation, it is
not going to suffice in a security-sensitive context.
>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->feedback
State-Changed-By: jdolecek
State-Changed-When: Wed Oct 6 15:29:08 UTC 2004
State-Changed-Why:
Please update to 2.0H first, then report if the problem persists.
From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: gnats-bugs@gnats.netbsd.org
Cc: hf@spg.tu-darmstadt.de, jdolecek@netbsd.org, kern-bug-people@netbsd.org
Subject: Re: kern/27164
Date: Thu, 7 Oct 2004 10:39:09 +0200
Am 06.10.2004 um 15:30 Uhr +0000 schrieb jdolecek@netbsd.org:
>Please update to 2.0H first, then report if the problem persists.
Automated reply? 8->
No offense, intended, but...
<rant>
I have spent a weekend and the better part of the following week on
this (significantly helped by the fact that half of the group
including the local chieftains were on holiday), and a few more hours
on collecting facts for the PR. If there were any possibility of
trying something as simple as "drop in a current kernel and retry", I
might already have thought of it.
</rant>
This is a production machine, and a bug report against ipfilter 3.xx
which is most likely relevant for the netbsd-1-6 branch.
-current as well as netbsd-2-0 ship with ipfilter 4.xx which, alas,
"is not there, yet" as a steady trickle of PRs confirms.
Additionally, "of course" a -current kernel won't work with the
ipfilter 3.xx userland tools. Hell, even some 3.xx *rulesets* fail
with 4.xx.
What I can offer besides the tcpdump file mentioned is move one of
the clients in question back to the stateful ruleset and do further
tests. No random "upgrading" possible, though.
HTH,
hauke
--
/~\ The ASCII Ribbon Campaign Hauke Fath
\ / No HTML/RTF in email Institut für Nachrichtentechnik
X No Word docs in email TU Darmstadt
/ \ Respect for open standards Ruf +49-6151-16-3281
Responsible-Changed-From-To: kern-bug-people->darrenr
Responsible-Changed-By: darrenr
Responsible-Changed-When: Sat Oct 30 09:12:57 UTC 2004
Responsible-Changed-Why:
From: darrenr@www.netbsd.org (Darren Reed)
To: hf@spg.tu-darmstadt.de
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/27164
Date: Sat, 30 Oct 2004 09:15:04 +0000 (UTC)
I'm not sure that "keep frags" works with Linux because (for some versions
of it, at least) they send the fragments in the reverse order to that with
which ipfilter works.
Can you investigate with tcpdump and find out whether or not the
Linux boxes are sending the last fragment first?
Cheers,
Darren
From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: darrenr@www.netbsd.org (Darren Reed)
Cc: hf@spg.tu-darmstadt.de, gnats-bugs@netbsd.org
Subject: Re: kern/27164
Date: Sat, 30 Oct 2004 23:50:54 +0200
Am 30.10.2004 um 9:15 Uhr +0000 schrieb Darren Reed:
>I'm not sure that "keep frags" works with Linux because (for some versions
>of it, at least) they send the fragments in the reverse order to that with
>which ipfilter works.
>
>Can you investigate with tcpdump and find out whether or not the
>Linux boxes are sending the last fragment first?
Give me a fortnight. I'm on the eurobsdcon ATM, and then on holiday
during the next week.
I've got an ethereal dump file around that shows the problem, and I
can tcpdump a RH9 machine's traffic, if needed.
But I actually came across the question - some discussion on the ipf
mailing list a while back. And I'm pretty sure I tried nfs v2 / udp /
1k packets to check this, with and without "keep frags", and it
didn't make a difference.
The stateless rules that are currently in place, OTOH, have "keep frags".
So, while the reverse order issue may be part of the picture, it's
not everything.
I'll get back to you with the tcpdump.
hauke
--
/~\ The ASCII Ribbon Campaign Hauke Fath
\ / No HTML/RTF in email Institut für Nachrichtentechnik
X No Word docs in email TU Darmstadt
/ \ Respect for open standards Ruf +49-6151-16-3281
State-Changed-From-To: feedback->closed
State-Changed-By: wiz@netbsd.org
State-Changed-When: Wed, 14 Sep 2005 12:51:39 +0000
State-Changed-Why:
Closed on request by the submitter (not reproducible).
From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: Hauke Fath <hf@spg.tu-darmstadt.de>
Cc: Darren Reed <darrenr@netbsd.org>, kern-bug-people@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/27164
Date: Wed, 14 Sep 2005 12:14:34 +0200
Hauke Fath wrote:
> [Mail from 2005-04-13 re-sent;
> Darren, did you ever get around to download the tarball? I get regular
> GNATS feedback reminders, please reset the PR.
> Thanks, hauke]
>
>
>> Am 30.10.2004 um 23:50 Uhr +0200 schrieb Hauke Fath:
>>
>>> Am 30.10.2004 um 9:15 Uhr +0000 schrieb Darren Reed:
>>>
>>>> I'm not sure that "keep frags" works with Linux because (for some
>>>> versions
>>>> of it, at least) they send the fragments in the reverse order to
>>>> that with
>>>> which ipfilter works.
>>>>
>>>> Can you investigate with tcpdump and find out whether or not the
>>>> Linux boxes are sending the last fragment first?
>>>
>>>
>>> Give me a fortnight. I'm on the eurobsdcon ATM, and then on holiday
>>> during the next week.
>>>
>>> I've got an ethereal dump file around that shows the problem, and I
>>> can tcpdump a RH9 machine's traffic, if needed.
>>>
>>> But I actually came across the question - some discussion on the ipf
>>> mailing list a while back. And I'm pretty sure I tried nfs v2 / udp /
>>> 1k packets to check this, with and without "keep frags", and it
>>> didn't make a difference.
>>>
>>> The stateless rules that are currently in place, OTOH, have "keep
>>> frags".
>>>
>>> So, while the reverse order issue may be part of the picture, it's
>>> not everything.
>>>
>>> I'll get back to you with the tcpdump.
>>
>>
>> Hi Darren,
>>
>> I sent the URL some months ago, but never heard back from you - the
>> issue may have been lost during your move to China.
>>
>> I have put a 1.3 MByte ethereal trace on
>> http://www.spg.tu-darmstadt.de/~hf/XXXXXXXXXXXXXXXXXXXXXXX .
>>
>> It contains the traffic between a RH 9 machine and a pre-2.0 NetBSD
>> NFS server via a filtering router (specs and RCS ids in the PR).
>>
>> Since there may be security sensitive data, I'd rather not submit the
>> URL to the Gnats db. Nevertheless, can you please update the PR state?
>>
>> Regards,
>> hauke
The router in question runs netbsd-3 now (the PR is against ipfilter
1.3.29 as found in netbsd-1-6). The RedHat machines that triggered the
bug are gone from our network, so I have no means any more of
testing/reproducing the problem.
Since Darren seems too busy to pick up the tcpdump tarball, or even
update the status of the pr, I ask for this PR to be closed.
hauke
[http://www.netbsd.org/Misc/query-pr.html tells me ther is no PR
#27164?! Something's badly broken there...]
--
/~\ The ASCII Ribbon Campaign Hauke Fath
\ / No HTML/RTF in email Institut für Nachrichtentechnik
X No Word docs in email TU Darmstadt
/ \ Respect for open standards Ruf +49-6151-16-3281
From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: wiz@netbsd.org
Cc: darrenr@netbsd.org, netbsd-bugs@netbsd.org,
gnats-admin@netbsd.org, Hauke Fath <hf@spg.tu-darmstadt.de>
Subject: Re: kern/27164
Date: Wed, 4 Oct 2006 16:37:09 +0200
Am 14.09.2005 um 12:51 Uhr +0000 schrieb wiz@netbsd.org:
>Synopsis: ipfilter 'keep state' blocking legal connections
>
>State-Changed-From-To: feedback->closed
>State-Changed-By: wiz@netbsd.org
>State-Changed-When: Wed, 14 Sep 2005 12:51:39 +0000
>State-Changed-Why:
>Closed on request by the submitter (not reproducible).
I have just encountered the problem again with a Debian Linux client
(Linux 2.6.16-2).
The router runs a NetBSD 3_STABLE build of 2006-03-30. Since a
decision has been made to move our NetBSD based clients to a Linux OS
by the end of the year, I need to resolve the issue one way or other.
Please re-open the pr.
hauke
--
/~\ The ASCII Ribbon Campaign Hauke Fath
\ / No HTML/RTF in email Institut für Nachrichtentechnik
X No Word docs in email TU Darmstadt
/ \ Respect for open standards Ruf +49-6151-16-3281
From: "Darren Reed (NetBSD)" <darrenr@netbsd.org>
To: Hauke Fath <hf@spg.tu-darmstadt.de>
Cc: wiz@netbsd.org, netbsd-bugs@netbsd.org, gnats-admin@netbsd.org
Subject: Re: kern/27164
Date: Thu, 05 Oct 2006 01:05:15 +1000
Hauke Fath wrote:
> Am 14.09.2005 um 12:51 Uhr +0000 schrieb wiz@netbsd.org:
>> Synopsis: ipfilter 'keep state' blocking legal connections
>>
>> State-Changed-From-To: feedback->closed
>> State-Changed-By: wiz@netbsd.org
>> State-Changed-When: Wed, 14 Sep 2005 12:51:39 +0000
>> State-Changed-Why:
>> Closed on request by the submitter (not reproducible).
>
> I have just encountered the problem again with a Debian Linux client
> (Linux 2.6.16-2).
>
> The router runs a NetBSD 3_STABLE build of 2006-03-30. Since a
> decision has been made to move our NetBSD based clients to a Linux OS
> by the end of the year, I need to resolve the issue one way or other.
>
> Please re-open the pr.
The obvious question that should have been asked: is your state table
filling to the point where new state entries are rejected?
Darren
State-Changed-From-To: closed->feedback
State-Changed-By: wiz@netbsd.org
State-Changed-When: Wed, 04 Oct 2006 17:34:41 +0000
State-Changed-Why:
Problem reappeared, darrenr asked for feedback.
From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: "Darren Reed (NetBSD)" <darrenr@netbsd.org>
Cc: Hauke Fath <hf@spg.tu-darmstadt.de>, netbsd-bugs@netbsd.org,
gnats-admin@netbsd.org
Subject: Re: kern/27164
Date: Wed, 4 Oct 2006 17:53:05 +0200
Am 05.10.2006 um 1:05 Uhr +1000 schrieb Darren Reed (NetBSD):
>The obvious question that should have been asked: is your state table
>filling to the point where new state entries are rejected?
Doesn't look like it:
IP states added:
933126 TCP
3894060 UDP
3402 ICMP
1147944587 hits
650176372 misses
927 maximum
0 no memory
0 max bucket
927 maximum
0 no memory
479 bkts in use
506 active
0 expired
150709 closed
State logging enabled
State table bucket statistics:
479 in use
8.35% bucket usage
0 minimal length
2 maximal length
1.056 average length
All other clients are unharmed.
hauke
--
/~\ The ASCII Ribbon Campaign Hauke Fath
\ / No HTML/RTF in email Institut für Nachrichtentechnik
X No Word docs in email TU Darmstadt
/ \ Respect for open standards Ruf +49-6151-16-3281
State-Changed-From-To: feedback->open
State-Changed-By: darrenr@netbsd.org
State-Changed-When: Wed, 25 Oct 2006 07:20:06 +0000
State-Changed-Why:
From: Mail Delivery Subsystem <MAILER-DAEMON@epita.fr>
To: undisclosed-recipients: ;
Cc:
Subject: Re: Re: kern/27164 (ipfilter 'keep state' blocking legal connections)
Date: Wed, 25 Oct 2006 09:20:14 +0200 (CEST)
Désolé,
le compte de epibug est actuellement indisponible.
Votre e-mail a été délivré mais le destinataire risque temporairement
de ne pas pouvoir y acceder.
Sorry,
epibug's account is currently unavailable.
Your email has been delivered. The recipient may temporarily
be unable to read it.
From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: gnats-bugs@NetBSD.org
Cc: gnats-admin@NetBSD.org
Subject: Re: kern/27164
Date: Fri, 9 Feb 2007 19:20:02 +0100
Am 04.10.2006 um 14:55 Uhr +0000 schrieb Hauke Fath:
> I have just encountered the problem again with a Debian Linux client
> (Linux 2.6.16-2).
>
> The router runs a NetBSD 3_STABLE build of 2006-03-30. Since a
> decision has been made to move our NetBSD based clients to a Linux OS
> by the end of the year, I need to resolve the issue one way or other.
In case it matters (and anybody other than me cares), the problem is
still present after a router upgrade to NetBSD 4beta2 as of
2007-02-07.
The client will work for a while, and then nfs accesses freeze. The
ipfilter log shows blocked tcp packets to the fileserver's port 2049
(nfs). A reboot doen't change anything; powering the client machine
off for half a day makes it work - for a while.
hauke
--
The ASCII Ribbon Campaign Hauke Fath
() No HTML/RTF in email Institut für Nachrichtentechnik
/\ No Word docs in email TU Darmstadt
Respect for open standards Ruf +49-6151-16-3281
Responsible-Changed-From-To: darrenr->kern-bug-people
Responsible-Changed-By: wiz@NetBSD.org
Responsible-Changed-When: Mon, 23 Dec 2013 11:32:16 +0000
Responsible-Changed-Why:
resigned, back to role account
From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@NetBSD.org, darrenr@NetBSD.org, netbsd-bugs@NetBSD.org,
gnats-admin@NetBSD.org, wiz@NetBSD.org
Subject: Re: kern/27164 (ipfilter 'keep state' blocking legal connections)
Date: Mon, 23 Dec 2013 12:41:36 +0100
On Mon, 23 Dec 2013 11:32:16 +0000 (UTC), wiz@NetBSD.org wrote:
> 27164
>=20
> resigned, back to role account
Just FTR, I moved the setup (as well as my home server, for different=20
reasons) to pf(4) years ago, and haven't seen a reason for touching=20
ipfilter since.
hauke
--=20
The ASCII Ribbon Campaign Hauke Fath
() No HTML/RTF in email Institut f=FCr Nachrichtentechnik
/\ No Word docs in email TU Darmstadt
Respect for open standards Ruf +49-6151-16-3281
From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: Thomas Klausner <wiz@NetBSD.org>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/27164 (ipfilter 'keep state' blocking legal connections)
Date: Mon, 23 Dec 2013 13:02:05 +0100
On Mon, 23 Dec 2013 12:58:09 +0100, Thomas Klausner wrote:
> So should I close the PR?
I have no reason to believe that the reported bug has been fixed. =3D8>
But it certainly looks as if nobody cares (or runs NFS over a NetBSD=20
router), so from my POV, you may as well close the PR.
Cheerio,
hauke
--=20
The ASCII Ribbon Campaign Hauke Fath
() No HTML/RTF in email Institut f=FCr Nachrichtentechnik
/\ No Word docs in email TU Darmstadt
Respect for open standards Ruf +49-6151-16-3281
Responsible-Changed-From-To: kern-bug-people->ipf-bug-people
Responsible-Changed-By: dholland@NetBSD.org
Responsible-Changed-When: Mon, 23 Dec 2013 17:45:25 +0000
Responsible-Changed-Why:
ipf role account
>Unformatted:
>Quarter:
>Keywords:
>Date-Required:
ipfilter version: # ipf -V
ipf: IP Filter: v3.4.29 (336)
Kernel: IP Filter: v3.4.29
[hf@fifi] /sbin # ident ipf
ipf:
$IPFilter: parse.c,v 2.8 1999/12/28 10:49:46 darrenr Exp $
$IPFilter: parse.c,v 2.8 1999/12/28 10:49:46 darrenr Exp $
$NetBSD: crt0.c,v 1.13 2003/07/26 19:24:27 salo Exp $
[hf@fifi] /sbin # ident /netbsd
/netbsd:
$NetBSD: GENERIC,v 1.583 2003/11/20 13:32:41 fvdl Exp $
$Revision: 1.1 $
$NetBSD: std.i386,v 1.24 2003/02/26 21:33:36 fvdl Exp $
[...]
$NetBSD: fil.c,v 1.59 2003/08/07 16:33:07 agc Exp $
$NetBSD: ip_auth.c,v 1.32 2003/08/22 21:53:03 itojun Exp $
$NetBSD: ip_fil.c,v 1.95 2003/08/22 22:11:44 itojun Exp $
$NetBSD: ip_frag.c,v 1.34 2002/09/19 08:12:49 martti Exp $
$NetBSD: ip_log.c,v 1.23 2002/09/25 06:43:21 martti Exp $
$NetBSD: ip_nat.c,v 1.55 2003/12/04 15:32:01 christos Exp $
$NetBSD: ip_proxy.c,v 1.36 2002/09/19 08:12:54 martti Exp $
$NetBSD: ip_ftp_pxy.c,v 1.26 2002/09/19 08:12:50 martti Exp $
$NetBSD: ip_rcmd_pxy.c,v 1.9 2002/01/24 08:23:45 martti Exp $
$NetBSD: ip_raudio_pxy.c,v 1.8 2002/01/24 08:23:14 martti Exp $
$NetBSD: ip_netbios_pxy.c,v 1.4 2002/09/19 08:12:53 martti Exp $
$NetBSD: ip_h323_pxy.c,v 1.8 2003/06/23 15:20:57 martin Exp $
$NetBSD: ip_ipsec_pxy.c,v 1.2 2002/04/01 16:47:46 jdolecek Exp $
$NetBSD: ip_state.c,v 1.42 2002/09/19 08:12:54 martti Exp $
[...]
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.