NetBSD Problem Report #34799

From www@NetBSD.org  Thu Oct 12 17:41:32 2006
Return-Path: <www@NetBSD.org>
Received: by narn.NetBSD.org (Postfix, from userid 31301)
	id 6BCD763B874; Thu, 12 Oct 2006 17:41:32 +0000 (UTC)
Message-Id: <20061012174132.6BCD763B874@narn.NetBSD.org>
Date: Thu, 12 Oct 2006 17:41:32 +0000 (UTC)
From: dhgutteridge@sympatico.ca
Reply-To: dhgutteridge@sympatico.ca
To: gnats-bugs@NetBSD.org
Subject: IP Filter does not work correctly with gem(4) when hardware checksumming is enabled
X-Send-Pr-Version: www-1.0

>Number:         34799
>Category:       kern
>Synopsis:       IP Filter does not work correctly with gem(4) when hardware checksumming is enabled
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    jdc
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Oct 12 17:45:00 +0000 2006
>Closed-Date:    Sat May 24 20:44:06 +0000 2008
>Last-Modified:  Sat May 24 20:44:06 +0000 2008
>Originator:     David H. Gutteridge
>Release:        3.0 series on macppc and i386
>Organization:
>Environment:
>Description:
When hardware checksumming is enabled for gem(4), IP Filter does not work correctly with outbound connections.  ipfstat(8) does not report any checksum failures, but ipmon(8) logs the affected outbound packets, marking them "bad".

I cannot duplicate this problem on another machine running the same NetBSD release with an ex(4) card with hardware checksumming enabled.  The latter machine has no problems with outbound packets through IP Filter.

I also cannot duplicate this problem on the machine with gem(4) when using OpenBSD's pf, only with IP Filter, so it appears not to be an issue with the driver.

Another user reported the same problem back in April.  (http://mail-index.netbsd.org/port-macppc/2006/04/02/0000.html)  I thought this had already been PRed, but apparently not.
>How-To-Repeat:
Enable TCP and UDP checksumming with gem(4) and use IP Filter.
>Fix:

>Release-Note:

>Audit-Trail:
From: john heasley <heas@shrubbery.net>
To: gnats-bugs@NetBSD.org, dhgutteridge@sympatico.ca
Cc: 
Subject: Re: kern/34799: IP Filter does not work correctly with gem(4) when hardware checksumming is enabled
Date: Mon, 16 Oct 2006 17:13:05 -0700

 I'm having trouble reproducing this bug on my sparc64.  I assume I've made
 an error in mbuf handling.  Could you share your ipf.conf?

From: "David H. GUTTERIDGE" <dhgutteridge@sympatico.ca>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/34799: IP Filter does not work correctly with gem(4) when hardware chec
Date: Wed, 18 Oct 2006 02:02:05 +0000

 >  I'm having trouble reproducing this bug on my sparc64.  I assume I've 
 >made
 >  an error in mbuf handling.  Could you share your ipf.conf?

 Here's my ipf.conf:

 pass in quick on lo0 all
 pass out quick on lo0 all

 block return-rst in log quick proto tcp all
 block in log quick proto udp all
 block in log quick proto icmp all

 block out log quick all head 1
   pass out proto tcp from any to any flags S keep state keep frags group 1
   pass out proto udp from any to any keep state keep frags group 1
   pass out proto icmp from any to any keep state keep frags group 1
   block out log quick from any to 127.0.0.0/8 group 1
   block out log quick from any to 172.16.0.0/12 group 1
   block out log quick from any to 10.0.0.0/8 group 1
   block out log quick from any to 255.255.255.255/32 group 1
   block out log quick from any to 0.0.0.0/8 group 1
   block out log quick from any to 169.254.0.0/16 group 1
   block out log quick from any to 192.0.2.0/24 group 1
   block out log quick from any to 204.152.64.0/23 group 1
   block out log quick from any to 224.0.0.0/3 group 1

 Relevant ifconfig details are:

 gem0: flags=8863<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST> mtu 1500
         capabilities=66<TCP4CSUM,UDP4CSUM,TCP4CSUM_Rx,UDP4CSUM_Rx>
         enabled=6<TCP4CSUM,UDP4CSUM>

 Here is a transcript of a sample test I did:

 [root@arcusv:root]# ifconfig gem0 tcp4csum
 [root@arcusv:root]# ifconfig gem0 udp4csum
 [root@arcusv:root]# date
 Tue Oct 17 21:20:43 EDT 2006
 [root@arcusv:root]# ping www.onetbsd.org
 ping: Cannot resolve "www.onetbsd.org" (Host name lookup failure)
 [root@arcusv:root]# tail -1 /var/log/messages
 Oct 17 21:21:14 arcusv ipmon[255]: 21:21:13.891345 gem0 @0:3 b 
 arcus0.nonus-porta.net[192.168.1.6],domain -> 
 arcusv.nonus-porta.net[192.168.1.5],65532 PR udp len 20 150 IN bad
 [root@arcusv:root]# ifconfig gem0 -tcp4csum
 [root@arcusv:root]# ifconfig gem0 -udp4csum
 [root@arcusv:root]# ping www.onetbsd.org
 PING www.onetbsd.org (213.28.202.226): 56 data bytes
 64 bytes from 213.28.202.226: icmp_seq=0 ttl=236 time=322.561 ms
 64 bytes from 213.28.202.226: icmp_seq=1 ttl=236 time=165.242 ms
 64 bytes from 213.28.202.226: icmp_seq=2 ttl=236 time=164.031 ms
 ^C
 ----www.onetbsd.org PING Statistics----
 3 packets transmitted, 3 packets received, 0.0% packet loss
 round-trip min/avg/max/stddev = 164.031/217.278/322.561/91.180 ms

 Dave


From: "David H. Gutteridge" <dhgutteridge@sympatico.ca>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/34799: IP Filter does not work correctly with gem(4) when hardware chec
Date: Wed, 12 Dec 2007 12:49:00 -0500

 As an update, this issue is still present with 4.0 RC5/macppc.


From: Julian Coleman <jdc@coris.org.uk>
To: gnats-bugs@NetBSD.org
Cc: jdc@NetBSD.org
Subject: Re: kern/34799: IP Filter does not work correctly with gem(4) when hardware chec
Date: Mon, 17 Dec 2007 09:32:51 +0000

 Hi,

 >  As an update, this issue is still present with 4.0 RC5/macppc.

 I've been looking at the gem documentation that Sun recently released, in
 order to make the fibre gem cards work.  I see that the gem card only supports
 TCP checksums, so we should not enable UDP checksums.  Does turning off UDP
 checksums only make things work?

 Thanks,

 J

 -- 
   My other computer also runs NetBSD    /        Sailing at Newbiggin
         http://www.netbsd.org/        /   http://www.newbigginsailingclub.org/

From: "David H. Gutteridge" <dhgutteridge@sympatico.ca>
To: gnats-bugs@NetBSD.org, jdc@coris.org.uk
Cc: 
Subject: Re: kern/34799: IP Filter does not work correctly with gem(4) when hardware chec
Date: Tue, 18 Dec 2007 21:55:10 -0500

 Julian Coleman wrote:
 >I've been looking at the gem documentation that Sun recently released, in
 >order to make the fibre gem cards work.  I see that the gem card only 
 >supports
 >TCP checksums, so we should not enable UDP checksums.  Does turning off UDP
 >checksums only make things work?

 The problem occurs with both UDP and TCP.  If I enable only TCP
 checksums, I still see the problem with TCP packets, e.g.:

 [root@arcusv:root]# ifconfig gem0

 gem0: flags=8863<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST> mtu 1500
         capabilities=3c00<TCP4CSUM_Rx,TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx>
         enabled=c00<TCP4CSUM_Rx,TCP4CSUM_Tx>
 ...
 [root@arcusv:root]# telnet www.google.ca 80

 Trying 72.14.253.103...
 ^C
 [root@arcusv:root]# tail -1 /var/log/messages
 Dec 18 21:39:33 arcusv ipmon[303]: 21:39:30.159629 gem0 @0:2 b 
 po-in-f103.google.com[72.14.253.103],http -> 
 arcusv.nonus-porta.net[192.168.1.5],65534 PR tcp len 20 52 -AS IN bad

 [root@arcusv:root]# ifconfig gem0 -tcp4csum

 [root@arcusv:root]# ifconfig gem0

 gem0: flags=8863<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST> mtu 1500
         capabilities=3c00<TCP4CSUM_Rx,TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx>
         enabled=0
 ...

 [root@arcusv:root]# telnet www.google.ca 80

 Trying 72.14.253.147...
 Connected to www.l.google.com.

 ...

 The problem apparently affects (or affected) the same chip on Solaris,
 judging from the IP Filter FAQ entry I found on the subject:
 http://www.phildev.net/ipf/IPFsolaris.html#solaris15

 This problem only happens with IP Filter.  If I use OpenBSD's PF, UDP
 and TCP packets pass out the gem interface without issue.

 Dave


From: "David H. Gutteridge" <dhgutteridge@sympatico.ca>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/34799: IP Filter does not work correctly with gem(4) when hardware chec
Date: Tue, 18 Dec 2007 23:27:41 -0500

 I can duplicate this with with HEAD sources (4.99.42) as well.


From: Julian Coleman <jdc@coris.org.uk>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/34799: IP Filter does not work correctly with gem(4) when hardware chec
Date: Wed, 19 Dec 2007 10:05:39 +0000

 Hi,

 >  The problem occurs with both UDP and TCP.  If I enable only TCP
 >  checksums, I still see the problem with TCP packets, e.g.:

 >  The problem apparently affects (or affected) the same chip on Solaris,
 >  judging from the IP Filter FAQ entry I found on the subject:
 >  http://www.phildev.net/ipf/IPFsolaris.html#solaris15
 >  
 >  This problem only happens with IP Filter.  If I use OpenBSD's PF, UDP
 >  and TCP packets pass out the gem interface without issue.

 Thanks for the extra information.  I wondered if it was only UDP, as the
 name lookup failyre:

   ping: Cannot resolve "www.onetbsd.org" (Host name lookup failure)

 would be a symptom of UDP problems.

 I'm close to getting my fibre card working, so I'll see if I can reproduce
 and/or spot the problem here.

 Thanks,

 J

 -- 
   My other computer also runs NetBSD    /        Sailing at Newbiggin
         http://www.netbsd.org/        /   http://www.newbigginsailingclub.org/

From: Julian Coleman <jdc@coris.org.uk>
To: gnats-admin@NetBSD.org, dhgutteridge@sympatico.ca
Cc: 
Subject: Re: kern/34799: IP Filter does not work correctly with gem(4) when hardware chec
Date: Tue, 1 Jan 2008 15:16:25 +0000

 Hi,

 > I'm close to getting my fibre card working, so I'll see if I can reproduce
 > and/or spot the problem here.

 I have working gem interfaces, but I'm unable to reproduce the problem here.
 Looking at the documentation, we have to pre-program the offset at which the
 card starts generating the TCP checksum.  We set this to:

         v |= (ETHER_HDR_LEN + sizeof(struct ip) +
               ((sc->sc_ethercom.ec_capenable & ETHERCAP_VLAN_MTU) ?
                 ETHER_VLAN_ENCAP_LEN : 0)) << GEM_RX_CONFIG_CXM_START_SHFT;

 in gem_init().  So, if you have a vlan configured on that interface and
 the packet is received directly on the interface, then the pre-programmed
 receive checksum will be incorrect.  Is this the case here?

 Could you send a dump of the received packet that is blocked?  Using:

   tcpdump -vv -s 1514 -x -X -c 1 host www.google.ca and src port 80

 should catch just the return packet (from `telnet www.google.ca 80`).

 > I can duplicate this with with HEAD sources (4.99.42) as well.

 Yes.  There isn't much change in the driver between 4.0 and HEAD.

 Thanks,

 J

 -- 
   My other computer also runs NetBSD    /        Sailing at Newbiggin
         http://www.netbsd.org/        /   http://www.newbigginsailingclub.org/

From: "David H. Gutteridge" <dhgutteridge@sympatico.ca>
To: jdc@coris.org.uk, gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/34799: IP Filter does not work correctly with gem(4) when hardware chec
Date: Tue, 01 Jan 2008 18:55:30 -0500

 Julian Coleman wrote:
 >I have working gem interfaces, but I'm unable to reproduce the problem 
 >here.
 >Looking at the documentation, we have to pre-program the offset at which 
 >the
 >card starts generating the TCP checksum.  We set this to:
 >
 >        v |= (ETHER_HDR_LEN + sizeof(struct ip) +
 >              ((sc->sc_ethercom.ec_capenable & ETHERCAP_VLAN_MTU) ?
 >                ETHER_VLAN_ENCAP_LEN : 0)) << GEM_RX_CONFIG_CXM_START_SHFT;
 >
 >in gem_init().  So, if you have a vlan configured on that interface and
 >the packet is received directly on the interface, then the pre-programmed
 >receive checksum will be incorrect.  Is this the case here?

 No, it's just on a conventional LAN.  (The packets go from the host in
 question to a switch, then to a machine doing NAT and firewalling and
 then out to the internet.)

 >Could you send a dump of the received packet that is blocked?  Using:
 >
 >  tcpdump -vv -s 1514 -x -X -c 1 host www.google.ca and src port 80
 >
 >should catch just the return packet (from `telnet www.google.ca 80`).

 Okay, here's the tcpdump output when TCP checksumming isn't enabled:

 tcpdump: listening on gem0, link-type EN10MB (Ethernet), capture size 1514 
 bytes
 18:37:56.541838 IP (tos 0x0, ttl  52, id 2885, offset 0, flags [none], 
 length: 52) py-in-f104.google.com.http > arcusv.nonus-porta.net.65354: S 
 [tcp sum ok] 1236678017:1236678017(0) ack 1361609820 win 5720 <mss 
 1430,nop,nop,sackOK,nop,wscale 6>
         0x0000:  0030 656b e3f2 0001 022c 0f58 0800 4500  .0ek.....,.X..E.
         0x0010:  0034 0b45 0000 3406 d180 40e9 a768 c0a8  .4.E..4...@..h..
         0x0020:  0105 0050 ff4a 49b6 3581 5128 845c 8012  ...P.JI.5.Q(.\..
         0x0030:  1658 5a72 0000 0204 0596 0101 0402 0103  .XZr............
         0x0040:  0306                                     ..
 1 packets captured
 9 packets received by filter
 0 packets dropped by kernel

 And here's the output when TCP checksumming is enabled:

 tcpdump: listening on gem0, link-type EN10MB (Ethernet), capture size 1514 
 bytes
 18:38:47.750683 IP (tos 0x0, ttl  52, id 29483, offset 0, flags [none], 
 length: 52) py-in-f104.google.com.http > arcusv.nonus-porta.net.65353: S 
 [tcp sum ok] 1424794231:1424794231(0) ack 3091734025 win 5720 <mss 
 1430,nop,nop,sackOK,nop,wscale 0>
         0x0000:  0030 656b e3f2 0001 022c 0f58 0800 4500  .0ek.....,.X..E.
         0x0010:  0034 732b 0000 3406 699a 40e9 a768 c0a8  .4s+..4.i.@..h..
         0x0020:  0105 0050 ff49 54ec a277 b848 1e09 8012  ...P.IT..w.H....
         0x0030:  1658 e17f 0000 0204 0596 0101 0402 0103  .X..............
         0x0040:  0300                                     ..
 1 packets captured
 11 packets received by filter
 0 packets dropped by kernel

 Regards,

 Dave


From: Julian Coleman <jdc@coris.org.uk>
To: gnats-bugs@NetBSD.org
Cc: dhgutteridge@sympatico.ca
Subject: Re: kern/34799: IP Filter does not work correctly with gem(4) when hardware chec
Date: Sun, 27 Jan 2008 17:47:42 +0000

 --0F1p//8PRICkK4MW
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline

 Hi,

 >  Okay, here's the tcpdump output when TCP checksumming isn't enabled:

 >  And here's the output when TCP checksumming is enabled:

 Two perfectly fine TCP packets.  I had a chance to look at this again (it's
 bugging me that I can't reproduce it).  I'm fairly sure that ipfilter is
 correct here and that the hardware checksum is not correct.  However, I'm
 not sure why this is only picked up by ipf.  Also, for some reason, the ipf
 check must not be enabled on sparc64 (I didn't immediately spot why).

 I starting instrumenting the receive checksum code, and I see that the
 bytes that are meant to contain the checksum always contain a value related
 to the packet length and the remote IP address.  For example:

 	IP		csum - length
 	81.2.110.35	7e36
 	81.2.110.36	7e37
 	 ...
 	81.2.110.47	7e42

 	204.152.190.12	49b6
 	204.152.190.13	49b7

 If you try the attached patch (which prints out the received packet flags
 from the hardware), I imagine that you will see a constant for each site
 you connect to.  (It will print a line on the console for each TCP packet
 received when tcp4csum_rx is enabled.)

 So, either receive checksums are always broken (seems odd to me), or we
 are setting up the card incorrectly.   I'll see if I can spot where.

 Thanks,

 J

 PS.  The patch is against -current, but it should apply against 4.0 with an
 offset.

 -- 
   My other computer also runs NetBSD    /        Sailing at Newbiggin
         http://www.netbsd.org/        /   http://www.newbigginsailingclub.org/

 --0F1p//8PRICkK4MW
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: attachment; filename="gem.diffs"

 --- gem.c.dist	2008-01-27 17:26:05.000000000 +0000
 +++ gem.c	2008-01-27 16:53:57.000000000 +0000
 @@ -1860,16 +1860,17 @@
  					~((~m->m_pkthdr.csum_data) - ~optsum);

  				while (m->m_pkthdr.csum_data >> 16)
  					m->m_pkthdr.csum_data =
  						(m->m_pkthdr.csum_data >> 16) +
  						(m->m_pkthdr.csum_data &
  						 0xffff);
  			}
 +printf("%s: rxstat = 0x%016lx: len=%4d (%03x); csum=%04x; c - l=%04x\n", sc->sc_dev.dv_xname, rxstat, (int) ((rxstat & GEM_RD_BUFSIZE) >> GEM_RD_BUFSHIFT), (int) ((rxstat & GEM_RD_BUFSIZE) >> GEM_RD_BUFSHIFT), (int) (rxstat & GEM_RD_CHECKSUM), (int) ((rxstat & GEM_RD_CHECKSUM) - ((rxstat & GEM_RD_BUFSIZE) >> GEM_RD_BUFSHIFT)));

  			m->m_pkthdr.csum_flags |= M_CSUM_DATA |
  						  M_CSUM_NO_PSEUDOHDR;
  		} else
  swcsum:
  			m->m_pkthdr.csum_flags = 0;
  #endif
  		/* Pass it on. */

 --0F1p//8PRICkK4MW--

From: Julian Coleman <jdc@coris.org.uk>
To: gnats-bugs@NetBSD.org
Cc: dhgutteridge@sympatico.ca
Subject: Re: kern/34799: IP Filter does not work correctly with gem(4) when hardware chec
Date: Mon, 28 Jan 2008 09:35:21 +0000

 Hi,

 >  So, either receive checksums are always broken (seems odd to me), or we
 >  are setting up the card incorrectly.   I'll see if I can spot where.

 Looking at this further, I thought that we should be setting the receive
 checksum offset to the end of the ethernet header, not the end of the IP
 header.  However, when I do this, the checksum part of the received data
 is still the same, but the hash value part changed.

 I looked at the OpenSolaris and Linux sources to see what they were doing.
 OpenSolaris sets the offset to the end of the ethernet header (so they must
 see the same bug that we're seeing) and Linux appears to ignore receive
 checksums entirely.

 I'm tempted to do the latter, as then we will at least work out of the box,
 rather than breaking with ipfilter.  Also, I'm pretty much out of ideas at
 this point.

 Thanks,

 J

 -- 
   My other computer also runs NetBSD    /        Sailing at Newbiggin
         http://www.netbsd.org/        /   http://www.newbigginsailingclub.org/

From: "David H. Gutteridge" <dhgutteridge@sympatico.ca>
To: jdc@coris.org.uk, gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/34799: IP Filter does not work correctly with gem(4) when hardware chec
Date: Mon, 28 Jan 2008 20:59:53 -0500

 >Two perfectly fine TCP packets.  I had a chance to look at this again (it's
 >bugging me that I can't reproduce it).  I'm fairly sure that ipfilter is
 >correct here and that the hardware checksum is not correct.  However, I'm
 >not sure why this is only picked up by ipf.  Also, for some reason, the ipf
 >check must not be enabled on sparc64 (I didn't immediately spot why).
 >
 >I starting instrumenting the receive checksum code, and I see that the
 >bytes that are meant to contain the checksum always contain a value related
 >to the packet length and the remote IP address.  For example:
 >
 >	IP		csum - length
 >	81.2.110.35	7e36
 >	81.2.110.36	7e37
 >	 ...
 >	81.2.110.47	7e42
 >
 >	204.152.190.12	49b6
 >	204.152.190.13	49b7
 >
 >If you try the attached patch (which prints out the received packet flags
 >from the hardware), I imagine that you will see a constant for each site
 >you connect to.  (It will print a line on the console for each TCP packet
 >received when tcp4csum_rx is enabled.)
 >
 >So, either receive checksums are always broken (seems odd to me), or we
 >are setting up the card incorrectly.   I'll see if I can spot where.
 >
 >Thanks,
 >
 >J
 >
 >PS.  The patch is against -current, but it should apply against 4.0 with an
 >offset.

 I built a -current kernel and tested, a sample result set is below:

 Jan 28 20:30:51 arcusv /netbsd: gem0: rxstat = 0x000000000042aa25: len=  66 
 (042); csum=aa25; c - l=a9e3
 Jan 28 20:30:52 arcusv ipmon[179]: 20:30:51.823636 gem0 @0:2 b 
 py-in-f104.google.com[64.233.167.104],http -> 
 arcusv.nonus-porta.net[192.168.1.5],65534 PR tcp len 20 52 -AS IN bad
 Jan 28 20:31:03 arcusv /netbsd: gem0: rxstat = 0x00000000004e4c85: len=  78 
 (04e); csum=4c85; c - l=4c37
 Jan 28 20:31:04 arcusv ipmon[179]: 20:31:03.167841 gem0 @0:2 b 
 www.netbsd.org[204.152.190.12],http -> 
 arcusv.nonus-porta.net[192.168.1.5],65532 PR tcp len 20 64 -AS IN bad
 Jan 28 20:31:20 arcusv /netbsd: gem0: rxstat = 0x0000000000429155: len=  66 
 (042); csum=9155; c - l=9113
 Jan 28 20:31:20 arcusv ipmon[179]: 20:31:20.356135 gem0 @0:2 b 
 96.f6.ead8.static.theplanet.com[216.234.246.150],telnet -> 
 arcusv.nonus-porta.net[192.168.1.5],65531 PR tcp len 20 52 -AS IN bad
 Jan 28 20:31:53 arcusv /netbsd: gem0: rxstat = 0x0000000000429913: len=  66 
 (042); csum=9913; c - l=98d1
 Jan 28 20:31:55 arcusv ipmon[179]: 20:31:53.680198 gem0 @0:2 b 
 81.52.134.11,http -> arcusv.nonus-porta.net[192.168.1.5],65530 PR tcp len 20 
 52 -AS IN bad
 Jan 28 20:31:57 arcusv ipmon[179]: 20:31:56.954812 gem0 @0:2 b 
 81.52.134.11,http -> arcusv.nonus-porta.net[192.168.1.5],65530 PR tcp len 20 
 52 -AS IN bad
 Jan 28 20:32:00 arcusv ipmon[179]: 20:31:59.681054 gem0 @0:2 b 
 81.52.134.11,http -> arcusv.nonus-porta.net[192.168.1.5],65530 PR tcp len 20 
 52 -AS IN bad
 Jan 28 20:32:03 arcusv ipmon[179]: 20:32:02.952896 gem0 @0:2 b 
 81.52.134.11,http -> arcusv.nonus-porta.net[192.168.1.5],65530 PR tcp len 20 
 52 -AS IN bad
 Jan 28 20:32:15 arcusv ipmon[179]: 20:32:14.950542 gem0 @0:2 b 
 81.52.134.11,http -> arcusv.nonus-porta.net[192.168.1.5],65530 PR tcp len 20 
 52 -AS IN bad
 Jan 28 20:32:39 arcusv last message repeated 5 times
 Jan 28 20:32:39 arcusv ipmon[179]: 20:32:39.146851 gem0 @0:2 b 
 81.52.134.11,http -> arcusv.nonus-porta.net[192.168.1.5],65530 PR tcp len 20 
 52 -AS IN bad
 Jan 28 20:32:46 arcusv /netbsd: gem0: rxstat = 0x00000000004e24b2: len=  78 
 (04e); csum=24b2; c - l=2464
 Jan 28 20:32:46 arcusv ipmon[179]: 20:32:46.552125 gem0 @0:2 b 
 pyry.gw.com[204.80.150.129],http -> 
 arcusv.nonus-porta.net[192.168.1.5],65529 PR tcp len 20 64 -AS IN bad
 Jan 28 20:33:27 arcusv /netbsd: gem0: rxstat = 0x0000000000429913: len=  66 
 (042); csum=9913; c - l=98d1
 Jan 28 20:33:27 arcusv ipmon[179]: 20:33:27.338335 gem0 @0:2 b 
 81.52.134.11,http -> arcusv.nonus-porta.net[192.168.1.5],65530 PR tcp len 20 
 52 -AS IN bad

 Regards,

 Dave


From: "David H. Gutteridge" <dhgutteridge@sympatico.ca>
To: jdc@coris.org.uk, gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/34799: IP Filter does not work correctly with gem(4) when hardware chec
Date: Mon, 28 Jan 2008 21:16:47 -0500

 >Looking at this further, I thought that we should be setting the receive
 >checksum offset to the end of the ethernet header, not the end of the IP
 >header.  However, when I do this, the checksum part of the received data
 >is still the same, but the hash value part changed.
 >
 >I looked at the OpenSolaris and Linux sources to see what they were doing.
 >OpenSolaris sets the offset to the end of the ethernet header (so they must
 >see the same bug that we're seeing) and Linux appears to ignore receive
 >checksums entirely.
 >
 >I'm tempted to do the latter, as then we will at least work out of the box,
 >rather than breaking with ipfilter.  Also, I'm pretty much out of ideas at
 >this point.

 Hi,

 I did some more poking about on the web, and found this reference
 from Darwin developers' correspondence:

 http://lists.apple.com/archives/darwin-kernel/2002/Jan/msg00036.html

 I think this except is telling:

 "> Also, why does darwin not allow csum offloading for received ip
 >fragments using the CSUM_TCP_SUM16 method?

 The CSUM_TCP_SUM16 flag is used to indicate that the GMAC's checksum is
 being used. The GMAC hardware checksum has some limitations. There are a lot
 of checks for the various limitations to avoid cases where the hardware does
 the wrong thing. I believe IP fragments may be one of the cases where the
 chipset does the wrong thing."

 Could it be that this problem only appears on macppc because specific
 GMAC chips are defective?  I don't know anything about GMAC, but I
 wonder if there were bugs present in the chips shipped in (some) Apple
 machines, but perhaps not in the ones used by Sun?  (Though there was
 that old IPFilter FAQ entry about the eri driver under Solaris not
 working with hardware checksumming too...  Maybe it's the same problem,
 and it affects very specific Sun hardware as well.)

 My gem card shows it's revision 0:
 gem0 at pci2 dev 15 function 0: Apple Computer GMAC Ethernet (rev. 0x00)

 My guess is that revision 1 GMAC chips behave differently in this
 regard.

 Regards,

 Dave

 PS Having tested the GENERIC kernel from HEAD, it seems that gem no
 longer supports UDP checksumming?  It did with 4.0.


From: Julian Coleman <jdc@coris.org.uk>
To: "David H. Gutteridge" <dhgutteridge@sympatico.ca>
Cc: gnats-bugs@netbsd.org
Subject: Re: kern/34799: IP Filter does not work correctly with gem(4) when hardware chec
Date: Tue, 29 Jan 2008 19:48:31 +0000

 --rwEMma7ioTxnRzrJ
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline

 Hi,

 > The CSUM_TCP_SUM16 flag is used to indicate that the GMAC's checksum is
 > being used. The GMAC hardware checksum has some limitations. There are a lot
 > of checks for the various limitations to avoid cases where the hardware does
 > the wrong thing. I believe IP fragments may be one of the cases where the
 > chipset does the wrong thing."

 It looks like other people have problems here - which is good news in that
 it is the hardware that's broken, not our driver (well, not here anyway ...).

 > Could it be that this problem only appears on macppc because specific
 > GMAC chips are defective?  I don't know anything about GMAC, but I
 > wonder if there were bugs present in the chips shipped in (some) Apple
 > machines, but perhaps not in the ones used by Sun?  (Though there was
 > that old IPFilter FAQ entry about the eri driver under Solaris not
 > working with hardware checksumming too...  Maybe it's the same problem,
 > and it affects very specific Sun hardware as well.)
 > 
 > My gem card shows it's revision 0:
 > gem0 at pci2 dev 15 function 0: Apple Computer GMAC Ethernet (rev. 0x00)
 > 
 > My guess is that revision 1 GMAC chips behave differently in this
 > regard.

 It could be.  The cards that I have are both rev. 0x01 and they exhibit
 the problem, but they are Sun cards (one is ERI and one is GEM).  So, I
 think that disabling HW receive checksums for Apple rev. 0 and for Sun
 rev. 1 chips is the way to go.

 > PS Having tested the GENERIC kernel from HEAD, it seems that gem no
 > longer supports UDP checksumming?  It did with 4.0.

 Yes.  I took that out after reading comments in the FreeBSD version of the
 driver - it notes that the hardware checksum doesn't compensate for the UDP
 case where the checksum ends up being all 0.  They did leave the ability to
 enable UDP checksums with `ifconfig gem0 link0`, but I don't see the
 usefulness of that, so I didn't put it in our driver.  So, we are left with
 just TCP hardware checksums, and only outgoing on the buggy revisions.

 If you could try the attached patch, that would be great.  If it's OK, I'll
 check it in and close the PR (if that's OK with you?).

 Thanks,

 J

 PS.  Would you have any interest in the newer gem code being in the 4.0
 branch?  My test machine is running that, so I have the changes that I've
 made to the driver working on the branch.  It'll just be a case of sorting
 out the appropriate revisions for the pull up request.

 -- 
   My other computer also runs NetBSD    /        Sailing at Newbiggin
         http://www.netbsd.org/        /   http://www.newbigginsailingclub.org/

 --rwEMma7ioTxnRzrJ
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: attachment; filename="gem.diffs"

 Index: sys/dev/ic/gem.c
 ===================================================================
 RCS file: /cvsroot/src/sys/dev/ic/gem.c,v
 retrieving revision 1.72
 diff -u -r1.72 gem.c
 --- sys/dev/ic/gem.c	22 Jan 2008 23:19:14 -0000	1.72
 +++ sys/dev/ic/gem.c	29 Jan 2008 19:34:09 -0000
 @@ -416,9 +416,15 @@
  	ifp->if_flags =
  	    IFF_BROADCAST | IFF_SIMPLEX | IFF_NOTRAILERS | IFF_MULTICAST;
  	sc->sc_if_flags = ifp->if_flags;
 -	/* The GEM hardware supports basic TCP checksum offloading only. */
 -	ifp->if_capabilities |=
 -	    IFCAP_CSUM_TCPv4_Tx | IFCAP_CSUM_TCPv4_Rx;
 +	/*
 +	 * The GEM hardware supports basic TCP checksum offloading only.
 +	 * Sun revision 1 and Apple revision 0 (at least) have bugs in the
 +	 * receive checksum, so don't enable it for them.
 +	 */
 +	ifp->if_capabilities |= IFCAP_CSUM_TCPv4_Tx;
 +	if ((GEM_IS_SUN(sc) && sc->sc_chiprev != 1) ||
 +	    (GEM_IS_APPLE(sc) && sc->sc_chiprev != 0))
 +		ifp->if_capabilities |= IFCAP_CSUM_TCPv4_Rx;
  	ifp->if_start = gem_start;
  	ifp->if_ioctl = gem_ioctl;
  	ifp->if_watchdog = gem_watchdog;
 @@ -2144,9 +2150,6 @@
  	}
  	if (status & GEM_INTR_RX_MAC) {
  		int rxstat = bus_space_read_4(t, h, GEM_MAC_RX_STATUS);
 -		if (rxstat & ~GEM_MAC_RX_DONE)
 -			printf("%s: MAC rx fault, status %x\n",
 -			    sc->sc_dev.dv_xname, rxstat);
  		/*
  		 * At least with GEM_SUN_GEM and some GEM_SUN_ERI
  		 * revisions GEM_MAC_RX_OVERFLOW happen often due to a
 @@ -2157,7 +2160,7 @@
  			ifp->if_ierrors++;
  			gem_reset_rxdma(sc);
  		} else if (rxstat & ~(GEM_MAC_RX_DONE | GEM_MAC_RX_FRAME_CNT))
 -			printf("%s: MAC rx fault, status %x\n",
 +			printf("%s: MAC rx fault, status 0x%02x\n",
  			    sc->sc_dev.dv_xname, rxstat);
  	}
  	if (status & GEM_INTR_PCS) {
 Index: sys/dev/ic/gemvar.h
 ===================================================================
 RCS file: /cvsroot/src/sys/dev/ic/gemvar.h,v
 retrieving revision 1.17
 diff -u -r1.17 gemvar.h
 --- sys/dev/ic/gemvar.h	19 Jan 2008 22:20:42 -0000	1.17
 +++ sys/dev/ic/gemvar.h	29 Jan 2008 19:34:14 -0000
 @@ -139,10 +139,15 @@
  #define	GEM_APPLE_GMAC		3	/* Apple GMAC variant */
  #define GEM_APPLE_K2_GMAC	4	/* Apple K2 GMAC */

 +#define GEM_IS_SUN(sc) \
 +	((sc)->sc_variant == GEM_SUN_GEM || \
 +	(sc)->sc_variant == GEM_SUN_ERI)
  #define	GEM_IS_APPLE(sc) \
  	((sc)->sc_variant == GEM_APPLE_GMAC || \
  	(sc)->sc_variant == GEM_APPLE_K2_GMAC)

 +	int		sc_chiprev;	/* hardware revision */
 +
  	u_int		sc_flags;	/* */
  	short		sc_if_flags;	/* copy of ifp->if_flags */
  #define	GEM_GIGABIT		0x0001	/* has a gigabit PHY */
 Index: sys/dev/pci/if_gem_pci.c
 ===================================================================
 RCS file: /cvsroot/src/sys/dev/pci/if_gem_pci.c,v
 retrieving revision 1.25
 diff -u -r1.25 if_gem_pci.c
 --- sys/dev/pci/if_gem_pci.c	5 Jan 2008 20:30:22 -0000	1.25
 +++ sys/dev/pci/if_gem_pci.c	29 Jan 2008 19:34:19 -0000
 @@ -194,8 +194,8 @@
  	aprint_naive(": Ethernet controller\n");

  	pci_devinfo(pa->pa_id, pa->pa_class, 0, devinfo, sizeof(devinfo));
 -	aprint_normal(": %s (rev. 0x%02x)\n", devinfo,
 -	    PCI_REVISION(pa->pa_class));
 +	sc->sc_chiprev = PCI_REVISION(pa->pa_class);
 +	aprint_normal(": %s (rev. 0x%02x)\n", devinfo, sc->sc_chiprev);

  	/*
  	 * Some Sun GEMs/ERIs do have their intpin register bogusly set to 0,

 --rwEMma7ioTxnRzrJ--

From: "David H. Gutteridge" <dhgutteridge@sympatico.ca>
To: jdc@coris.org.uk, gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/34799: IP Filter does not work correctly with gem(4) when hardware chec
Date: Tue, 29 Jan 2008 21:55:50 -0500

 Hi,

 > > My guess is that revision 1 GMAC chips behave differently in this
 > > regard.
 >
 >It could be.  The cards that I have are both rev. 0x01 and they exhibit
 >the problem, but they are Sun cards (one is ERI and one is GEM).  So, I
 >think that disabling HW receive checksums for Apple rev. 0 and for Sun
 >rev. 1 chips is the way to go.

 Oh, I thought only the Apple hardware was showing the problem.  It
 might be safer to assume then that none of the variations work
 properly.  The original report about the issue was from another macppc
 user, and having tracked down his email, his was on a Mac Mini, which
 is considerably newer than my machine.  (Some) PowerPC Mac Minis
 apparently report this:

 gem0 at pci2 dev 15 function 0: Apple Computer GMAC Ethernet (rev. 0x80)

 Not sure about the original reporter's specifics, I couldn't get at the
 dmesg he'd posted, but it seems my revision theory is wrong for Apples.

 > > PS Having tested the GENERIC kernel from HEAD, it seems that gem no
 > > longer supports UDP checksumming?  It did with 4.0.
 >
 >Yes.  I took that out after reading comments in the FreeBSD version of the
 >driver - it notes that the hardware checksum doesn't compensate for the UDP
 >case where the checksum ends up being all 0.  They did leave the ability to
 >enable UDP checksums with `ifconfig gem0 link0`, but I don't see the
 >usefulness of that, so I didn't put it in our driver.  So, we are left with
 >just TCP hardware checksums, and only outgoing on the buggy revisions.

 Ah.  I think I may disable all hardware checksumming on my gem, it
 sounds pretty dodgy.  (But of course, I will test patches.)

 >If you could try the attached patch, that would be great.  If it's OK, I'll
 >check it in and close the PR (if that's OK with you?).

 I'll give it a try tomorrow.

 >
 >Thanks,
 >
 >J
 >
 >PS.  Would you have any interest in the newer gem code being in the 4.0
 >branch?  My test machine is running that, so I have the changes that I've
 >made to the driver working on the branch.  It'll just be a case of sorting
 >out the appropriate revisions for the pull up request.

 Sounds good to me, though I might not be making use of the particular
 code we're talking about.


From: "David H. Gutteridge" <dhgutteridge@sympatico.ca>
To: jdc@coris.org.uk, gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/34799: IP Filter does not work correctly with gem(4) when hardware chec
Date: Wed, 30 Jan 2008 18:27:53 -0500

 Hello,

 >>Oh, I thought only the Apple hardware was showing the problem.  It
 >>might be safer to assume then that none of the variations work
 >>properly.

 >This seems like a good idea to me too.  The attached patch does that.
 >If you could test it instead of the previous one, that would be great!

 It worked fine for me with a -current kernel again.

 >>Ah.  I think I may disable all hardware checksumming on my gem, it
 >>sounds pretty dodgy.  (But of course, I will test patches.)

 >Well, it seems the transmit TCP checksum is OK ;-)  Thanks.

 Yes, that seemed to work fine for me in conjunction with ipf in this
 latest test.

 >>Sounds good to me, though I might not be making use of the particular
 >>code we're talking about.

 >;-)  I'll sort out a pull up request when the latest code has been in
 >the tree for a couple of weeks (to make sure I haven't broken anything).

 Perhaps something should be added to the "bugs" section of the gem man
 page noting why there's no UDP or TCP RX hardware checksumming
 available?  (I see you removed the reference to UDP being available
 already.)

 Regards,

 Dave


Responsible-Changed-From-To: kern-bug-people->jdc
Responsible-Changed-By: jdc@narn.netbsd.org
Responsible-Changed-When: Fri, 01 Feb 2008 13:04:24 +0000
Responsible-Changed-Why:
I've been looking at this.


State-Changed-From-To: open->pending-pullups
State-Changed-By: jdc@narn.netbsd.org
State-Changed-When: Fri, 01 Feb 2008 13:04:24 +0000
State-Changed-Why:
Fixed, but will pull up to netbsd-4.


State-Changed-From-To: pending-pullups->closed
State-Changed-By: jdc@NetBSD.org
State-Changed-When: Sat, 24 May 2008 20:44:06 +0000
State-Changed-Why:
Pulled up to netbsd-4


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.