NetBSD Problem Report #27164

Received: (qmail 28607 invoked by uid 605); 6 Oct 2004 15:06:55 -0000
Message-Id: <200410061506.i96F6boW001727@heiligenberg.nt.e-technik.tu-darmstadt.de>
Date: Wed, 6 Oct 2004 17:06:37 +0200 (CEST)
From: Hauke Fath <hf@spg.tu-darmstadt.de>
Sender: gnats-bugs-owner@NetBSD.org
Reply-To: Hauke Fath <hf@spg.tu-darmstadt.de>
To: gnats-bugs@gnats.NetBSD.org
Cc: Hauke Fath <hf@spg.tu-darmstadt.de>
Subject: ipfilter 'keep state' blocking legal connections
X-Send-Pr-Version: 3.95

>Number:         27164
>Category:       kern
>Synopsis:       ipfilter 'keep state' blocking legal connections
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    ipf-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Oct 06 15:07:01 +0000 2004
>Closed-Date:    
>Last-Modified:  Mon Dec 23 17:45:25 +0000 2013
>Originator:     Hauke Fath <hf@spg.tu-darmstadt.de>
>Release:        NetBSD 1.6ZG
>Organization:
-- 
/~\  The ASCII Ribbon Campaign                    Hauke Fath
\ /    No HTML/RTF in email	        Institut fr Nachrichtentechnik
 X     No Word docs in email	                  TU Darmstadt
/ \  Respect for open standards              Ruf +49-6151-16-3281
>Environment:


System: NetBSD fifi 1.6ZG NetBSD 1.6ZG (FIFI) #0: Mon Dec 22 14:41:32 CET 2003  hf@heiligenberg:/var/obj/netbsd-builds/i386/obj/sys/arch/i386/compile/FIFI i386

Architecture: i386
Machine: i386

>Description:

Given the setup (4 interfaces, 4 subnets)


                 < DMZ >------+
                              |
                              | 
   <clients>-------------< ipf router >------------------/INTERNET/
                              |
                              |
      <Internal Servers>------+


where <clients> can access a machine in <internal servers> ("IS") for NIS
services and NFS shares.

The NFS server is running NetBSD 2.0_beta, the clients are NetBSD
1.6.2 (upgraded since), NetBSD 2.0_beta, stock RedHat 8 (linux
2.4.18-27.8.0) + 9 (linux 2.4.20-8 / 2.4.27) installations.

The <clients> can access IS via rules like

pass in log quick on wm1 all keep frags head 100
	pass in quick proto tcp from <clients> to any flags S keep state keep frags group 100
	pass in quick proto udp from <clients> to any keep state keep frags group 100
	pass in quick proto icmp from <clients> to any keep state keep frags group 100

<clients> access to IS works fine for YP. NetBSD clients work fine with the IS nfs server.

The five RedHat <clients>, OTOH, work fine *most of the time*. Then,
often after a user login and associated amd mount, a RH machine will
lock up. The ipfilter log on the router will show lots of entries
like

00:03:06.688586 wm0 @0:7 b SERVER[130.83.xx.yy],nfs -> CLIENT[130.83.uu.vv],800 PR tcp len 20 52 -A IN

for nfs3/tcp connections, and

18:01:32.254120 2x wm1 @0:5 b SERVER[130.83.xx.yy] -> CLIENT[130.83.uu.vv] PR udp len 20 (1500) frag +1480@1480 OUT 
18:01:32.254470 13x wm0 @0:7 b SERVER[130.83.xx.yy] -> CLIENT[130.83.uu.vv] PR udp len 20 (1500) frag +1480@1480 IN 

for nfs2/udp mounts. The @0:[57] rules are "block {in,out} log quick"
style head rules.

At this point, rebooting client or server machine
does not change anything. Leaving the client powered off for a night,
OTOH, re-enables client access - until eventually the problem kicks in
again.

Monitoring the connection on the client-side switch (ethereal) shows
repeated attempts to contact the server (nfs3/tcp) that are never
acknowledged (tcpdump protocol available).

A typical ipfstate on the router today gives

 IPv6 packets:          in 50 out 62
 input packets:         blocked 1381539 passed 600800307 nomatch 50 counted 143525206 short 0
output packets:         blocked 14689 passed 600802869 nomatch 62 counted 192702497 short 0
 input packets logged:  blocked 1368453 passed 8678
output packets logged:  blocked 14689 passed 9978
 packets logged:        input 0 output 0
 log failures:          input 37670 output 2937
fragment state(in):     kept 422488     lost 122032
fragment state(out):    kept 422130     lost 125298
packet state(in):       kept 5311041    lost 16173
packet state(out):      kept 2562221    lost 23665
ICMP replies:   92878   TCP RSTs sent:  742704
Invalid source(in):     0
Result cache hits(in):  9550076 (out):  9273810
IN Pullups succeeded:   0       failed: 0
OUT Pullups succeeded:  0       failed: 0
Fastroute successes:    835582  failures:       0
TCP cksum fails(in):    4       (out):  32
Packet log flags set: (0)
        none


Apart from this nfs problem, both netbsd-netbsd nfs connections and
any non-nfs connections involving RH machines have been working fine
during the nine months that the router is running now.

>How-To-Repeat:

	Set up a NetBSD 2 nfs/nis server and RedHat 8/9 client
	machines on two sides of an ipfilter based router like
	pictured above.

	Find that eventually the RH clients wil hang, and rebooting
	them won't help. Find a lot of log entries on the router.

>Fix:

	The workaround I found was to set up vanilla 'pass
	all' rules (2 directions times 2 interfaces) *without* keeping
	state. Anything *with* keep state between RH nfs clients and
	the NetBSD nfs server would eventually trigger the problem.

	While this kludge works for me in the current situation, it is
	not going to suffice in a security-sensitive context.
>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->feedback 
State-Changed-By: jdolecek 
State-Changed-When: Wed Oct 6 15:29:08 UTC 2004 
State-Changed-Why:  
Please update to 2.0H first, then report if the problem persists. 

From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: gnats-bugs@gnats.netbsd.org
Cc: hf@spg.tu-darmstadt.de, jdolecek@netbsd.org, kern-bug-people@netbsd.org
Subject: Re: kern/27164
Date: Thu, 7 Oct 2004 10:39:09 +0200

 Am 06.10.2004 um 15:30 Uhr +0000 schrieb jdolecek@netbsd.org:
 >Please update to 2.0H first, then report if the problem persists.

 Automated reply?  8->

 No offense, intended, but...

 <rant>
 I have spent a weekend and the better part of the following week on 
 this (significantly helped by the fact that half of the group 
 including the local chieftains were on holiday), and a few more hours 
 on collecting facts for the PR. If there were any possibility of 
 trying something as simple as "drop in a current kernel and retry", I 
 might already have thought of it.
 </rant>

 This is a production machine, and a bug report against ipfilter 3.xx 
 which is most likely relevant for the netbsd-1-6 branch.

 -current as well as netbsd-2-0 ship with ipfilter 4.xx which, alas, 
 "is not there, yet" as a steady trickle of PRs confirms. 
 Additionally, "of course" a -current kernel won't work with the 
 ipfilter 3.xx userland tools. Hell, even some 3.xx *rulesets* fail 
 with 4.xx.

 What I can offer besides the tcpdump file mentioned is move one of 
 the clients in question back to the stateful ruleset and do further 
 tests. No random "upgrading" possible, though.

 HTH,
 	hauke

 -- 
 /~\  The ASCII Ribbon Campaign                    Hauke Fath
 \ /    No HTML/RTF in email	        Institut fr Nachrichtentechnik
   X     No Word docs in email	                  TU Darmstadt
 / \  Respect for open standards              Ruf +49-6151-16-3281
Responsible-Changed-From-To: kern-bug-people->darrenr 
Responsible-Changed-By: darrenr 
Responsible-Changed-When: Sat Oct 30 09:12:57 UTC 2004 
Responsible-Changed-Why:  

From: darrenr@www.netbsd.org (Darren Reed)
To: hf@spg.tu-darmstadt.de
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/27164
Date: Sat, 30 Oct 2004 09:15:04 +0000 (UTC)

 I'm not sure that "keep frags" works with Linux because (for some versions
 of it, at least) they send the fragments in the reverse order to that with
 which ipfilter works.

 Can you investigate with tcpdump and find out whether or not the
 Linux boxes are sending the last fragment first?

 Cheers,
 Darren

From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: darrenr@www.netbsd.org (Darren Reed)
Cc: hf@spg.tu-darmstadt.de, gnats-bugs@netbsd.org
Subject: Re: kern/27164
Date: Sat, 30 Oct 2004 23:50:54 +0200

 Am 30.10.2004 um 9:15 Uhr +0000 schrieb Darren Reed:
 >I'm not sure that "keep frags" works with Linux because (for some versions
 >of it, at least) they send the fragments in the reverse order to that with
 >which ipfilter works.
 >
 >Can you investigate with tcpdump and find out whether or not the
 >Linux boxes are sending the last fragment first?

 Give me a fortnight. I'm on the eurobsdcon ATM, and then on holiday 
 during the next week.

 I've got an ethereal dump file around that shows the problem, and I 
 can tcpdump a RH9 machine's traffic, if needed.

 But I actually came across the question - some discussion on the ipf 
 mailing list a while back. And I'm pretty sure I tried nfs v2 / udp / 
 1k packets to check this, with and without "keep frags", and it 
 didn't make a difference.

 The stateless rules that are currently in place, OTOH, have "keep frags".

 So, while the reverse order issue may be part of the picture, it's 
 not everything.

 I'll get back to you with the tcpdump.

 	hauke

 -- 
 /~\  The ASCII Ribbon Campaign                    Hauke Fath
 \ /    No HTML/RTF in email	        Institut fr Nachrichtentechnik
   X     No Word docs in email	                  TU Darmstadt
 / \  Respect for open standards              Ruf +49-6151-16-3281
State-Changed-From-To: feedback->closed
State-Changed-By: wiz@netbsd.org
State-Changed-When: Wed, 14 Sep 2005 12:51:39 +0000
State-Changed-Why:
Closed on request by the submitter (not reproducible).


From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: Hauke Fath <hf@spg.tu-darmstadt.de>
Cc: Darren Reed <darrenr@netbsd.org>, kern-bug-people@netbsd.org,
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/27164
Date: Wed, 14 Sep 2005 12:14:34 +0200

 Hauke Fath wrote:
 > [Mail from 2005-04-13 re-sent;
 > Darren, did you ever get around to download the tarball? I get regular 
 > GNATS feedback reminders, please reset the PR.
 > Thanks, hauke]
 > 
 > 
 >> Am 30.10.2004 um 23:50 Uhr +0200 schrieb Hauke Fath:
 >>
 >>> Am 30.10.2004 um 9:15 Uhr +0000 schrieb Darren Reed:
 >>>
 >>>> I'm not sure that "keep frags" works with Linux because (for some 
 >>>> versions
 >>>> of it, at least) they send the fragments in the reverse order to 
 >>>> that with
 >>>> which ipfilter works.
 >>>>
 >>>> Can you investigate with tcpdump and find out whether or not the
 >>>> Linux boxes are sending the last fragment first?
 >>>
 >>>
 >>> Give me a fortnight. I'm on the eurobsdcon ATM, and then on holiday 
 >>> during the next week.
 >>>
 >>> I've got an ethereal dump file around that shows the problem, and I 
 >>> can tcpdump a RH9 machine's traffic, if needed.
 >>>
 >>> But I actually came across the question - some discussion on the ipf 
 >>> mailing list a while back. And I'm pretty sure I tried nfs v2 / udp / 
 >>> 1k packets to check this, with and without "keep frags", and it 
 >>> didn't make a difference.
 >>>
 >>> The stateless rules that are currently in place, OTOH, have "keep 
 >>> frags".
 >>>
 >>> So, while the reverse order issue may be part of the picture, it's 
 >>> not everything.
 >>>
 >>> I'll get back to you with the tcpdump.
 >>
 >>
 >> Hi Darren,
 >>
 >> I sent the URL some months ago, but never heard back from you - the 
 >> issue may have been lost during your move to China.
 >>
 >> I have put a 1.3 MByte ethereal trace on 
 >> http://www.spg.tu-darmstadt.de/~hf/XXXXXXXXXXXXXXXXXXXXXXX .
 >>
 >> It contains the traffic between a RH 9 machine and a pre-2.0 NetBSD 
 >> NFS server via a filtering router (specs and RCS ids in the PR).
 >>
 >> Since there may be security sensitive data, I'd rather not submit the 
 >> URL to the Gnats db. Nevertheless, can you please update the PR state?
 >>
 >> Regards,
 >>     hauke

 The router in question runs netbsd-3 now (the PR is against ipfilter 
 1.3.29 as found in netbsd-1-6). The RedHat machines that triggered the 
 bug are gone from our network, so I have no means any more of 
 testing/reproducing the problem.

 Since Darren seems too busy to pick up the tcpdump tarball, or even 
 update the status of the pr, I ask for this PR to be closed.

 	hauke


 [http://www.netbsd.org/Misc/query-pr.html tells me ther is no PR 
 #27164?! Something's badly broken there...]

 -- 
 /~\  The ASCII Ribbon Campaign                    Hauke Fath
 \ /    No HTML/RTF in email	        Institut fr Nachrichtentechnik
   X     No Word docs in email	                  TU Darmstadt
 / \  Respect for open standards              Ruf +49-6151-16-3281

From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: wiz@netbsd.org
Cc: darrenr@netbsd.org, netbsd-bugs@netbsd.org,
	gnats-admin@netbsd.org, Hauke Fath <hf@spg.tu-darmstadt.de>
Subject: Re: kern/27164
Date: Wed, 4 Oct 2006 16:37:09 +0200

 Am 14.09.2005 um 12:51 Uhr +0000 schrieb wiz@netbsd.org:
 >Synopsis: ipfilter 'keep state' blocking legal connections
 >
 >State-Changed-From-To: feedback->closed
 >State-Changed-By: wiz@netbsd.org
 >State-Changed-When: Wed, 14 Sep 2005 12:51:39 +0000
 >State-Changed-Why:
 >Closed on request by the submitter (not reproducible).

 I have just encountered the problem again with a Debian Linux client 
 (Linux 2.6.16-2).

 The router runs a NetBSD 3_STABLE build of 2006-03-30. Since a 
 decision has been made to move our NetBSD based clients to a Linux OS 
 by the end of the year, I need to resolve the issue one way or other.

 Please re-open the pr.

 	hauke

 -- 
 /~\  The ASCII Ribbon Campaign                    Hauke Fath
 \ /    No HTML/RTF in email	        Institut fr Nachrichtentechnik
   X     No Word docs in email	                  TU Darmstadt
 / \  Respect for open standards              Ruf +49-6151-16-3281

From: "Darren Reed (NetBSD)" <darrenr@netbsd.org>
To: Hauke Fath <hf@spg.tu-darmstadt.de>
Cc: wiz@netbsd.org, netbsd-bugs@netbsd.org, gnats-admin@netbsd.org
Subject: Re: kern/27164
Date: Thu, 05 Oct 2006 01:05:15 +1000

 Hauke Fath wrote:
 > Am 14.09.2005 um 12:51 Uhr +0000 schrieb wiz@netbsd.org:
 >> Synopsis: ipfilter 'keep state' blocking legal connections
 >>
 >> State-Changed-From-To: feedback->closed
 >> State-Changed-By: wiz@netbsd.org
 >> State-Changed-When: Wed, 14 Sep 2005 12:51:39 +0000
 >> State-Changed-Why:
 >> Closed on request by the submitter (not reproducible).
 >
 > I have just encountered the problem again with a Debian Linux client 
 > (Linux 2.6.16-2).
 >
 > The router runs a NetBSD 3_STABLE build of 2006-03-30. Since a 
 > decision has been made to move our NetBSD based clients to a Linux OS 
 > by the end of the year, I need to resolve the issue one way or other.
 >
 > Please re-open the pr.

 The obvious question that should have been asked: is your state table
 filling to the point where new state entries are rejected?

 Darren

State-Changed-From-To: closed->feedback
State-Changed-By: wiz@netbsd.org
State-Changed-When: Wed, 04 Oct 2006 17:34:41 +0000
State-Changed-Why:
Problem reappeared, darrenr asked for feedback.


From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: "Darren Reed (NetBSD)" <darrenr@netbsd.org>
Cc: Hauke Fath <hf@spg.tu-darmstadt.de>, netbsd-bugs@netbsd.org,
	gnats-admin@netbsd.org
Subject: Re: kern/27164
Date: Wed, 4 Oct 2006 17:53:05 +0200

 Am 05.10.2006 um 1:05 Uhr +1000 schrieb Darren Reed (NetBSD):
 >The obvious question that should have been asked: is your state table
 >filling to the point where new state entries are rejected?

 Doesn't look like it:

 IP states added:
          933126 TCP
          3894060 UDP
          3402 ICMP
          1147944587 hits
          650176372 misses
          927 maximum
          0 no memory
          0 max bucket
          927 maximum
          0 no memory
          479 bkts in use
          506 active
          0 expired
          150709 closed
 State logging enabled

 State table bucket statistics:
          479 in use
          8.35% bucket usage
          0 minimal length
          2 maximal length
          1.056 average length

 All other clients are unharmed.

 	hauke


 -- 
 /~\  The ASCII Ribbon Campaign                    Hauke Fath
 \ /    No HTML/RTF in email	        Institut fr Nachrichtentechnik
   X     No Word docs in email	                  TU Darmstadt
 / \  Respect for open standards              Ruf +49-6151-16-3281

State-Changed-From-To: feedback->open
State-Changed-By: darrenr@netbsd.org
State-Changed-When: Wed, 25 Oct 2006 07:20:06 +0000
State-Changed-Why:


From: Mail Delivery Subsystem <MAILER-DAEMON@epita.fr>
To: undisclosed-recipients: ;
Cc: 
Subject: Re: Re: kern/27164 (ipfilter 'keep state' blocking legal connections)
Date: Wed, 25 Oct 2006 09:20:14 +0200 (CEST)

 	Dsol, 
         le compte de epibug est actuellement indisponible.
 	Votre e-mail a t dlivr mais le destinataire risque temporairement 
         de ne  pas pouvoir y acceder.

         Sorry,
         epibug's account is currently unavailable.
         Your email has been delivered. The recipient may temporarily
         be unable to read it.

From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: gnats-bugs@NetBSD.org
Cc: gnats-admin@NetBSD.org
Subject: Re: kern/27164
Date: Fri, 9 Feb 2007 19:20:02 +0100

 Am 04.10.2006 um 14:55 Uhr +0000 schrieb Hauke Fath:
 >  I have just encountered the problem again with a Debian Linux client
 >  (Linux 2.6.16-2).
 >
 >  The router runs a NetBSD 3_STABLE build of 2006-03-30. Since a
 >  decision has been made to move our NetBSD based clients to a Linux OS
 >  by the end of the year, I need to resolve the issue one way or other.

 In case it matters (and anybody other than me cares), the problem is 
 still present after a router upgrade to NetBSD 4beta2 as of 
 2007-02-07.

 The client will work for a while, and then nfs accesses freeze. The 
 ipfilter log shows blocked tcp packets to the fileserver's port 2049 
 (nfs). A reboot doen't change anything; powering the client machine 
 off for half a day makes it work - for a while.

 	hauke

 -- 
       The ASCII Ribbon Campaign                    Hauke Fath
 ()     No HTML/RTF in email            Institut fr Nachrichtentechnik
 /\     No Word docs in email                     TU Darmstadt
       Respect for open standards              Ruf +49-6151-16-3281

Responsible-Changed-From-To: darrenr->kern-bug-people
Responsible-Changed-By: wiz@NetBSD.org
Responsible-Changed-When: Mon, 23 Dec 2013 11:32:16 +0000
Responsible-Changed-Why:
resigned, back to role account


From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@NetBSD.org, darrenr@NetBSD.org, netbsd-bugs@NetBSD.org,
        gnats-admin@NetBSD.org, wiz@NetBSD.org
Subject: Re: kern/27164 (ipfilter 'keep state' blocking legal connections)
Date: Mon, 23 Dec 2013 12:41:36 +0100

 On Mon, 23 Dec 2013 11:32:16 +0000 (UTC), wiz@NetBSD.org wrote:
 > 27164
 >=20
 > resigned, back to role account

 Just FTR, I moved the setup (as well as my home server, for different=20
 reasons) to pf(4) years ago, and haven't seen a reason for touching=20
 ipfilter since.

 hauke

 --=20
      The ASCII Ribbon Campaign                    Hauke Fath
 ()     No HTML/RTF in email            Institut f=FCr Nachrichtentechnik
 /\     No Word docs in email                     TU Darmstadt
      Respect for open standards              Ruf +49-6151-16-3281

From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: Thomas Klausner <wiz@NetBSD.org>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/27164 (ipfilter 'keep state' blocking legal connections)
Date: Mon, 23 Dec 2013 13:02:05 +0100

 On Mon, 23 Dec 2013 12:58:09 +0100, Thomas Klausner wrote:
 > So should I close the PR?

 I have no reason to believe that the reported bug has been fixed. =3D8>

 But it certainly looks as if nobody cares (or runs NFS over a NetBSD=20
 router), so from my POV, you may as well close the PR.

 Cheerio,
 hauke

 --=20
      The ASCII Ribbon Campaign                    Hauke Fath
 ()     No HTML/RTF in email            Institut f=FCr Nachrichtentechnik
 /\     No Word docs in email                     TU Darmstadt
      Respect for open standards              Ruf +49-6151-16-3281

Responsible-Changed-From-To: kern-bug-people->ipf-bug-people
Responsible-Changed-By: dholland@NetBSD.org
Responsible-Changed-When: Mon, 23 Dec 2013 17:45:25 +0000
Responsible-Changed-Why:
ipf role account


>Unformatted:
 >Quarter:        
 >Keywords:       
 >Date-Required:  

 ipfilter version: # ipf -V
 ipf: IP Filter: v3.4.29 (336)
 Kernel: IP Filter: v3.4.29 

 [hf@fifi] /sbin # ident ipf
 ipf:
      $IPFilter: parse.c,v 2.8 1999/12/28 10:49:46 darrenr Exp $
      $IPFilter: parse.c,v 2.8 1999/12/28 10:49:46 darrenr Exp $
      $NetBSD: crt0.c,v 1.13 2003/07/26 19:24:27 salo Exp $

 [hf@fifi] /sbin # ident /netbsd 
 /netbsd:
      $NetBSD: GENERIC,v 1.583 2003/11/20 13:32:41 fvdl Exp $
      $Revision: 1.1 $
      $NetBSD: std.i386,v 1.24 2003/02/26 21:33:36 fvdl Exp $
 [...]
      $NetBSD: fil.c,v 1.59 2003/08/07 16:33:07 agc Exp $
      $NetBSD: ip_auth.c,v 1.32 2003/08/22 21:53:03 itojun Exp $
      $NetBSD: ip_fil.c,v 1.95 2003/08/22 22:11:44 itojun Exp $
      $NetBSD: ip_frag.c,v 1.34 2002/09/19 08:12:49 martti Exp $
      $NetBSD: ip_log.c,v 1.23 2002/09/25 06:43:21 martti Exp $
      $NetBSD: ip_nat.c,v 1.55 2003/12/04 15:32:01 christos Exp $
      $NetBSD: ip_proxy.c,v 1.36 2002/09/19 08:12:54 martti Exp $
      $NetBSD: ip_ftp_pxy.c,v 1.26 2002/09/19 08:12:50 martti Exp $
      $NetBSD: ip_rcmd_pxy.c,v 1.9 2002/01/24 08:23:45 martti Exp $
      $NetBSD: ip_raudio_pxy.c,v 1.8 2002/01/24 08:23:14 martti Exp $
      $NetBSD: ip_netbios_pxy.c,v 1.4 2002/09/19 08:12:53 martti Exp $
      $NetBSD: ip_h323_pxy.c,v 1.8 2003/06/23 15:20:57 martin Exp $
      $NetBSD: ip_ipsec_pxy.c,v 1.2 2002/04/01 16:47:46 jdolecek Exp $
      $NetBSD: ip_state.c,v 1.42 2002/09/19 08:12:54 martti Exp $
 [...]

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.