NetBSD Problem Report #29978

From www@netbsd.org  Fri Apr 15 11:12:48 2005
Return-Path: <www@netbsd.org>
Received: by narn.netbsd.org (Postfix, from userid 31301)
	id C725B63B117; Fri, 15 Apr 2005 11:12:48 +0000 (UTC)
Message-Id: <20050415111248.C725B63B117@narn.netbsd.org>
Date: Fri, 15 Apr 2005 11:12:48 +0000 (UTC)
From: tino@wildenhain.de
Reply-To: tino@wildenhain.de
To: gnats-bugs@netbsd.org
Subject: tlp driver bad performance (transmit underrun) especially ANA-6944A
X-Send-Pr-Version: www-1.0

>Number:         29978
>Category:       kern
>Synopsis:       tlp driver bad performance (transmit underrun) especially ANA-6944A
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Apr 15 11:13:00 +0000 2005
>Last-Modified:  Sun Apr 17 14:32:01 +0000 2005
>Originator:     Tino Wildenhain
>Release:        NetBSD 3.0_BETA
>Organization:
>Environment:
NetBSD fw.bln.thecampus.de 3.0_BETA NetBSD 3.0_BETA (FW_IDE) #0: Fri Apr 15 08:29:48 UTC 2005  root@fw.bln.thecampus.de:/usr/src/sys/arch/i386/compile/FW_IDE i386
>Description:
after boot, beginning use of the network interfaces causes
these messages:
tlp0: transmit underrun; new threshold: 96/256 bytes
tlp0: transmit underrun; new threshold: 128/512 bytes
tlp0: transmit underrun; new threshold: 160/1024 bytes

this is immediately after few packets, ttcp measures
about 5MB/s

after some MB of traffic:
tlp0: transmit underrun; new threshold: store and forward mode

now performance drops to around 4MB/s

I compared with linux2.6.8 on that hardware, where 
no problem appears in the log (may depend on the
driver) and performance is around 8-9MB/s
>How-To-Repeat:
dmesg output:

tlp0 at pci2 dev 4 function 0: DECchip 21140A Ethernet, pass 2.2
tlp0: interrupting at irq 5
tlp0: Adaptec ANA-6944A, Ethernet address 00:00:d1:1f:09:54
nsphy0 at tlp0 phy 1: DP83840 10/100 media interface, rev. 1
nsphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
tlp1 at pci2 dev 5 function 0: DECchip 21140A Ethernet, pass 2.2
tlp1: sharing interrupt with tlp0
tlp1: Adaptec ANA-6944A, Ethernet address 00:00:d1:1f:09:55
nsphy1 at tlp1 phy 1: DP83840 10/100 media interface, rev. 1
nsphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
tlp2 at pci2 dev 6 function 0: DECchip 21140A Ethernet, pass 2.2
tlp2: sharing interrupt with tlp0
tlp2: Adaptec ANA-6944A, Ethernet address 00:00:d1:1f:09:56
nsphy2 at tlp2 phy 1: DP83840 10/100 media interface, rev. 1
nsphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
tlp3 at pci2 dev 7 function 0: DECchip 21140A Ethernet, pass 2.2
tlp3: sharing interrupt with tlp0
tlp3: Adaptec ANA-6944A, Ethernet address 00:00:d1:1f:09:57
nsphy3 at tlp3 phy 1: DP83840 10/100 media interface, rev. 1
nsphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto

NB: "dribbling bits" were noticed with other tlp-cards
as well.
>Fix:
may be ideas can be ripped from linux driver? 

>Audit-Trail:
From: Manuel Bouyer <bouyer@antioche.lip6.fr>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@NetBSD.org, gnats-admin@NetBSD.org,
	netbsd-bugs@NetBSD.org
Subject: Re: kern/29978: tlp driver bad performance (transmit underrun) especially ANA-6944A
Date: Fri, 15 Apr 2005 17:57:23 +0200

 On Fri, Apr 15, 2005 at 11:13:00AM +0000, tino@wildenhain.de wrote:
 > after boot, beginning use of the network interfaces causes
 > these messages:
 > tlp0: transmit underrun; new threshold: 96/256 bytes
 > tlp0: transmit underrun; new threshold: 128/512 bytes
 > tlp0: transmit underrun; new threshold: 160/1024 bytes
 > 
 > this is immediately after few packets, ttcp measures
 > about 5MB/s
 > 
 > after some MB of traffic:
 > tlp0: transmit underrun; new threshold: store and forward mode
 > 
 > now performance drops to around 4MB/s
 > 
 > I compared with linux2.6.8 on that hardware, where 
 > no problem appears in the log (may depend on the
 > driver) and performance is around 8-9MB/s

 FWIW, the tlp driver always end in store and forward mode on my hardware.
 But I see decent network performances anyway.
 tlp0 at pci0 dev 10 function 0: DECchip 21140A Ethernet, pass 2.2
 tlp0: interrupting at physical irq 10, irq 6
 tlp0: Ethernet address 00:80:c8:7f:d4:06
 lxtphy0 at tlp0 phy 0: LXT970 10/100 media interface, rev. 0

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

From: Tino Wildenhain <tino@wildenhain.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/29978: tlp driver bad performance (transmit underrun)
	especially ANA-6944A
Date: Sun, 17 Apr 2005 15:14:36 +0200

 Am Freitag, den 15.04.2005, 11:13 +0000 schrieb gnats-admin@netbsd.org:
 > Thank you very much for your problem report.
 > It has the internal identification `kern/29978'.
 > The individual assigned to look at your
 > report is: kern-bug-people. 
 > 
 > >Category:       kern
 > >Responsible:    kern-bug-people
 > >Synopsis:       tlp driver bad performance (transmit underrun) especially ANA-6944A
 > >Arrival-Date:   Fri Apr 15 11:13:00 +0000 2005

 Additional informations: 

 I added a 3com single card and 2 tlp single cards to the system
 to compare the drivers. Here is the buslayout (with linux)

 lspci -tv
 -[00]-+-00.0  Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX Host bridge
       +-01.0-[01]----00.0  ATI Technologies Inc Rage 128 PF/PRO AGP 4x
 TMDS
       +-07.0  Intel Corp. 82371AB/EB/MB PIIX4 ISA
       +-07.1  Intel Corp. 82371AB/EB/MB PIIX4 IDE
       +-07.2  Intel Corp. 82371AB/EB/MB PIIX4 USB
       +-07.3  Intel Corp. 82371AB/EB/MB PIIX4 ACPI
       +-08.0  3Com Corporation 3c905 100BaseTX [Boomerang]
       +-09.0-[02]--+-04.0  Digital Equipment Corporation DECchip 21140
 [FasterNet]
       |            +-05.0  Digital Equipment Corporation DECchip 21140
 [FasterNet]
       |            +-06.0  Digital Equipment Corporation DECchip 21140
 [FasterNet]
       |            \-07.0  Digital Equipment Corporation DECchip 21140
 [FasterNet]
       +-0a.0  Digital Equipment Corporation DECchip 21140 [FasterNet]
       \-0b.0  Digital Equipment Corporation DECchip 21140 [FasterNet]


 (in all tests I run only console, no gfx)

 relevant dmesg output:

 ex0 at pci0 dev 8 function 0: 3Com 3c905-TX 10/100 Ethernet (rev. 0x0)
 ex0: interrupting at irq 11
 ex0: MAC address 00:60:97:83:ab:6d
 nsphy0 at ex0 phy 24: DP83840 10/100 media interface, rev. 1
 nsphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 ppb1 at pci0 dev 9 function 0: Digital Equipment DC21152 PCI-PCI Bridge
 (rev. 0x03)
 pci2 at ppb1 bus 2
 pci2: i/o space, memory space enabled, rd/line, wr/inv ok
 tlp0 at pci2 dev 4 function 0: DECchip 21140A Ethernet, pass 2.2
 tlp0: interrupting at irq 5
 tlp0: Adaptec ANA-6944A, Ethernet address 00:00:d1:1f:09:54
 nsphy1 at tlp0 phy 1: DP83840 10/100 media interface, rev. 1
 nsphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 tlp1 at pci2 dev 5 function 0: DECchip 21140A Ethernet, pass 2.2
 tlp1: sharing interrupt with tlp0
 tlp1: Adaptec ANA-6944A, Ethernet address 00:00:d1:1f:09:55
 nsphy2 at tlp1 phy 1: DP83840 10/100 media interface, rev. 1
 nsphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 tlp2 at pci2 dev 6 function 0: DECchip 21140A Ethernet, pass 2.2
 tlp2: sharing interrupt with tlp0
 tlp2: Adaptec ANA-6944A, Ethernet address 00:00:d1:1f:09:56
 nsphy3 at tlp2 phy 1: DP83840 10/100 media interface, rev. 1
 nsphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 tlp3 at pci2 dev 7 function 0: DECchip 21140A Ethernet, pass 2.2
 tlp3: sharing interrupt with tlp0
 tlp3: Adaptec ANA-6944A, Ethernet address 00:00:d1:1f:09:57
 nsphy4 at tlp3 phy 1: DP83840 10/100 media interface, rev. 1
 nsphy4: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 tlp4 at pci0 dev 10 function 0: DECchip 21140A Ethernet, pass 2.2
 tlp4: interrupting at irq 9
 tlp4: Ethernet address 00:80:c8:f6:c9:b3
 lxtphy0 at tlp4 phy 0: LXT970 10/100 media interface, rev. 1
 lxtphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 tlp5 at pci0 dev 11 function 0: DECchip 21140A Ethernet, pass 2.2
 tlp5: interrupting at irq 10
 tlp5: Ethernet address 00:80:c8:f6:46:bd
 lxtphy1 at tlp5 phy 0: LXT970 10/100 media interface, rev. 1
 lxtphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto

 =======================================================================
 ttcp test with 3com (ex0):
 -----------------------------------------------------------------------
 ttcp-t: buflen=8192, nbuf=16384, align=16384/0, port=5001ttcp-t: socket
 ttcp-t: connect
 ttcp-t: 134217728 bytes in 12.02 real seconds = 10905.64 KB/sec +++
 ttcp-t: 134217728 bytes in 5.29 CPU seconds = 24769.54 KB/cpu sec
 ttcp-t: 16384 I/O calls, msec/call = 0.75, calls/sec = 1363.21
 ttcp-t: 0.0user 5.2sys 0:12real 43% 0i+0d 0maxrss 0+17228pf 28096+33csw
 ttcp-t: buffer address 0x8050000

 =======================================================================
 ttcp test with AHA-6944A 1. Port (tlp0):
 -----------------------------------------------------------------------
 ttcp-t: buflen=8192, nbuf=16384, align=16384/0, port=5001ttcp-t: socket
 ttcp-t: connect
 ttcp-t: 134217728 bytes in 30.47 real seconds = 4301.84 KB/sec +++
 ttcp-t: 134217728 bytes in 6.28 CPU seconds = 20873.48 KB/cpu sec
 ttcp-t: 16384 I/O calls, msec/call = 1.90, calls/sec = 537.73
 ttcp-t: 0.0user 6.1sys 0:30real 20% 0i+0d 0maxrss 0+14937pf 38361+17csw
 ttcp-t: buffer address 0x8050000

 =======================================================================
 ttcp test with AHA-6944A 4. Port (tlp3):
 -----------------------------------------------------------------------
 ttcp-t: buflen=8192, nbuf=16384, align=16384/0, port=5001ttcp-t: socket
 ttcp-t: connect
 ttcp-t: 134217728 bytes in 30.45 real seconds = 4304.97 KB/sec +++
 ttcp-t: 134217728 bytes in 6.15 CPU seconds = 21312.09 KB/cpu sec
 ttcp-t: 16384 I/O calls, msec/call = 1.90, calls/sec = 538.12
 ttcp-t: 0.0user 6.0sys 0:30real 20% 0i+0d 0maxrss 0+14828pf 38473+18csw
 ttcp-t: buffer address 0x8050000

 =======================================================================
 ttcp test with single tlp-Card (tlp4):
 -----------------------------------------------------------------------
 ttcp-t: buflen=8192, nbuf=16384, align=16384/0, port=5001ttcp-t: socket
 ttcp-t: connect
 ttcp-t: 134217728 bytes in 12.30 real seconds = 10654.03 KB/sec +++
 ttcp-t: 134217728 bytes in 5.96 CPU seconds = 22000.53 KB/cpu sec
 ttcp-t: 16384 I/O calls, msec/call = 0.77, calls/sec = 1331.75
 ttcp-t: 0.0user 5.9sys 0:12real 48% 0i+0d 0maxrss 0+17449pf 22722+42csw
 ttcp-t: buffer address 0x8050000


 We can see the performance with the single tlp card is much better.










From: Manuel Bouyer <bouyer@antioche.eu.org>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@NetBSD.org, gnats-admin@NetBSD.org,
	netbsd-bugs@NetBSD.org
Subject: Re: kern/29978: tlp driver bad performance (transmit underrun) especially ANA-6944A
Date: Sun, 17 Apr 2005 16:31:34 +0200

 On Sun, Apr 17, 2005 at 01:40:02PM +0000, Tino Wildenhain wrote:
 >  Additional informations: 
 >  
 >  I added a 3com single card and 2 tlp single cards to the system
 >  to compare the drivers. Here is the buslayout (with linux)
 >  
 >  lspci -tv
 >  -[00]-+-00.0  Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX Host bridge
 >        +-01.0-[01]----00.0  ATI Technologies Inc Rage 128 PF/PRO AGP 4x
 >  TMDS
 >        +-07.0  Intel Corp. 82371AB/EB/MB PIIX4 ISA
 >        +-07.1  Intel Corp. 82371AB/EB/MB PIIX4 IDE
 >        +-07.2  Intel Corp. 82371AB/EB/MB PIIX4 USB
 >        +-07.3  Intel Corp. 82371AB/EB/MB PIIX4 ACPI
 >        +-08.0  3Com Corporation 3c905 100BaseTX [Boomerang]
 >        +-09.0-[02]--+-04.0  Digital Equipment Corporation DECchip 21140
 >  [FasterNet]
 >        |            +-05.0  Digital Equipment Corporation DECchip 21140
 >  [FasterNet]
 >        |            +-06.0  Digital Equipment Corporation DECchip 21140
 >  [FasterNet]
 >        |            \-07.0  Digital Equipment Corporation DECchip 21140
 >  [FasterNet]
 >        +-0a.0  Digital Equipment Corporation DECchip 21140 [FasterNet]
 >        \-0b.0  Digital Equipment Corporation DECchip 21140 [FasterNet]
 >  
 >  
 >  (in all tests I run only console, no gfx)
 >  
 >  relevant dmesg output:
 >  
 >  ex0 at pci0 dev 8 function 0: 3Com 3c905-TX 10/100 Ethernet (rev. 0x0)
 >  ex0: interrupting at irq 11
 >  ex0: MAC address 00:60:97:83:ab:6d
 >  nsphy0 at ex0 phy 24: DP83840 10/100 media interface, rev. 1
 >  nsphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 >  ppb1 at pci0 dev 9 function 0: Digital Equipment DC21152 PCI-PCI Bridge
 >  (rev. 0x03)
 >  pci2 at ppb1 bus 2
 >  pci2: i/o space, memory space enabled, rd/line, wr/inv ok
 >  tlp0 at pci2 dev 4 function 0: DECchip 21140A Ethernet, pass 2.2
 >  tlp0: interrupting at irq 5
 >  tlp0: Adaptec ANA-6944A, Ethernet address 00:00:d1:1f:09:54
 >  nsphy1 at tlp0 phy 1: DP83840 10/100 media interface, rev. 1
 >  nsphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 >  tlp1 at pci2 dev 5 function 0: DECchip 21140A Ethernet, pass 2.2
 >  tlp1: sharing interrupt with tlp0
 >  tlp1: Adaptec ANA-6944A, Ethernet address 00:00:d1:1f:09:55
 >  nsphy2 at tlp1 phy 1: DP83840 10/100 media interface, rev. 1
 >  nsphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 >  tlp2 at pci2 dev 6 function 0: DECchip 21140A Ethernet, pass 2.2
 >  tlp2: sharing interrupt with tlp0
 >  tlp2: Adaptec ANA-6944A, Ethernet address 00:00:d1:1f:09:56
 >  nsphy3 at tlp2 phy 1: DP83840 10/100 media interface, rev. 1
 >  nsphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 >  tlp3 at pci2 dev 7 function 0: DECchip 21140A Ethernet, pass 2.2
 >  tlp3: sharing interrupt with tlp0
 >  tlp3: Adaptec ANA-6944A, Ethernet address 00:00:d1:1f:09:57
 >  nsphy4 at tlp3 phy 1: DP83840 10/100 media interface, rev. 1
 >  nsphy4: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 >  tlp4 at pci0 dev 10 function 0: DECchip 21140A Ethernet, pass 2.2
 >  tlp4: interrupting at irq 9
 >  tlp4: Ethernet address 00:80:c8:f6:c9:b3
 >  lxtphy0 at tlp4 phy 0: LXT970 10/100 media interface, rev. 1
 >  lxtphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 >  tlp5 at pci0 dev 11 function 0: DECchip 21140A Ethernet, pass 2.2
 >  tlp5: interrupting at irq 10
 >  tlp5: Ethernet address 00:80:c8:f6:46:bd
 >  lxtphy1 at tlp5 phy 0: LXT970 10/100 media interface, rev. 1
 >  lxtphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 >  
 >  =======================================================================
 >  ttcp test with 3com (ex0):
 >  -----------------------------------------------------------------------
 >  ttcp-t: buflen=8192, nbuf=16384, align=16384/0, port=5001ttcp-t: socket
 >  ttcp-t: connect
 >  ttcp-t: 134217728 bytes in 12.02 real seconds = 10905.64 KB/sec +++
 >  ttcp-t: 134217728 bytes in 5.29 CPU seconds = 24769.54 KB/cpu sec
 >  ttcp-t: 16384 I/O calls, msec/call = 0.75, calls/sec = 1363.21
 >  ttcp-t: 0.0user 5.2sys 0:12real 43% 0i+0d 0maxrss 0+17228pf 28096+33csw
 >  ttcp-t: buffer address 0x8050000
 >  
 >  =======================================================================
 >  ttcp test with AHA-6944A 1. Port (tlp0):
 >  -----------------------------------------------------------------------
 >  ttcp-t: buflen=8192, nbuf=16384, align=16384/0, port=5001ttcp-t: socket
 >  ttcp-t: connect
 >  ttcp-t: 134217728 bytes in 30.47 real seconds = 4301.84 KB/sec +++
 >  ttcp-t: 134217728 bytes in 6.28 CPU seconds = 20873.48 KB/cpu sec
 >  ttcp-t: 16384 I/O calls, msec/call = 1.90, calls/sec = 537.73
 >  ttcp-t: 0.0user 6.1sys 0:30real 20% 0i+0d 0maxrss 0+14937pf 38361+17csw
 >  ttcp-t: buffer address 0x8050000
 >  
 >  =======================================================================
 >  ttcp test with AHA-6944A 4. Port (tlp3):
 >  -----------------------------------------------------------------------
 >  ttcp-t: buflen=8192, nbuf=16384, align=16384/0, port=5001ttcp-t: socket
 >  ttcp-t: connect
 >  ttcp-t: 134217728 bytes in 30.45 real seconds = 4304.97 KB/sec +++
 >  ttcp-t: 134217728 bytes in 6.15 CPU seconds = 21312.09 KB/cpu sec
 >  ttcp-t: 16384 I/O calls, msec/call = 1.90, calls/sec = 538.12
 >  ttcp-t: 0.0user 6.0sys 0:30real 20% 0i+0d 0maxrss 0+14828pf 38473+18csw
 >  ttcp-t: buffer address 0x8050000
 >  
 >  =======================================================================
 >  ttcp test with single tlp-Card (tlp4):
 >  -----------------------------------------------------------------------
 >  ttcp-t: buflen=8192, nbuf=16384, align=16384/0, port=5001ttcp-t: socket
 >  ttcp-t: connect
 >  ttcp-t: 134217728 bytes in 12.30 real seconds = 10654.03 KB/sec +++
 >  ttcp-t: 134217728 bytes in 5.96 CPU seconds = 22000.53 KB/cpu sec
 >  ttcp-t: 16384 I/O calls, msec/call = 0.77, calls/sec = 1331.75
 >  ttcp-t: 0.0user 5.9sys 0:12real 48% 0i+0d 0maxrss 0+17449pf 22722+42csw
 >  ttcp-t: buffer address 0x8050000
 >  
 >  
 >  We can see the performance with the single tlp card is much better.

 It may be an issue with the pci-pci bridge on the adaptec adapter.
 It would be interesting to see if the linux driver has something special about
 it.

 Is there a way to dump the PCI registers on linux ? If would be interesting
 to compare the values between linux and NetBSD for the pci-pci bridge.

 You can also compare the values for the tlp devices on the adaptec adapter and
 the single device (with pcictl dump)

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

Home	PR Database Search