NetBSD Problem Report #20390

Received: (qmail 16765 invoked by uid 605); 17 Feb 2003 15:55:04 -0000
Message-Id: <20030217155500.09281292C2@smistad.uninett.no>
Date: Mon, 17 Feb 2003 16:55:00 +0100 (CET)
From: he@uninett.no
Sender: gnats-bugs-owner@netbsd.org
Reply-To: he@uninett.no
To: gnats-bugs@gnats.netbsd.org
Subject: callwheel corruption under higher network loads
X-Send-Pr-Version: 3.95

>Number:         20390
>Category:       kern
>Synopsis:       callwheel corruption under higher network loads
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    he
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Feb 17 15:56:00 +0000 2003
>Closed-Date:    Fri Oct 06 22:28:55 +0000 2006
>Last-Modified:  Fri Oct 06 22:28:55 +0000 2006
>Originator:     Havard Eidnes
>Release:        NetBSD 1.6.1_RC1, Feb 11 2003 vintage
>Organization:
	UNINETT AS
>Environment:
System: NetBSD black-3.sth.netnod.se 1.6.1_RC1 NetBSD 1.6.1_RC1 (BLACKHOLE) #5: Sat Feb 15 18:46:52 MET 2003     he@black-3.sth.netnod.se:/usr/src/sys/arch/i386/compile/BLACKHOLE i386
Architecture: i386
Machine: i386
>Description:

	I'm involved in debugging NetBSD on a system which sees a fair
	amount of TCP-based DNS traffic; the network interface reports
	about 1500-3000 interupts per second.

	This system is plagued with problems where the TCP connection
	table after a somewhat indeterminate time (sometimes days)
	grows without bounds, ending with exhaustion of kernel
	resources and typically a hang.

	To be able to catch it in the act, I've upped the NMBCLUSTERS
	limit:

options         NMBCLUSTERS=4096

	Recently I've noticed that when this happens, operations such
	as running "tset" or "sleep 2" never terminate.  I've seen
	output from "netstat -f inet -n" of 37000 lines...

	To enable further debugging, I've also enabled:

options         QUEUEDEBUG
options         CALLWHEEL_STATS

	This has recently landed me a couple of panics in the
	QUEUEDEBUG code, one of which is reported here.  While
	inspecting the callwheel on the running kernel, I found
	several entries whose time had come and gone, and which were
	lingering around (that's before I had CALLWHEEL_STATS turned
	on), and a code inspection of sys/kern_clock.c reveals that
	indeed, "late" events will never be processed.  To fix that, I
	added this patch:

------------------------------
Index: kern_clock.c
===================================================================
RCS file: /cvsroot/src/sys/kern/kern_clock.c,v
retrieving revision 1.79
diff -u -r1.79 kern_clock.c
--- kern_clock.c	2002/03/17 11:10:43	1.79
+++ kern_clock.c	2003/02/17 15:24:42
@@ -372,6 +372,7 @@
 struct evcnt callwheel_softchecks;	/* # checks per softclock() */
 struct evcnt callwheel_softempty;	/* # empty buckets seen */
 struct evcnt callwheel_hintworked;	/* # times hint saved scan */
+struct evcnt callwheel_toolate;		/* # times time passed us by */
 #endif /* CALLWHEEL_STATS */

 /*
@@ -963,7 +964,7 @@
 #ifdef CALLWHEEL_STATS
 			callwheel_softchecks.ev_count++;
 #endif
-			if (c->c_time != softclock_ticks) {
+			if (c->c_time > softclock_ticks) {
 				if (c->c_time < bucket->cq_hint)
 					bucket->cq_hint = c->c_time;
 				c = TAILQ_NEXT(c, c_link);
@@ -976,6 +977,10 @@
 					steps = 0;
 				}
 			} else {
+#ifdef CALLWHEEL_STATS
+				if (c->c_time < softclock_ticks)
+					callwheel_toolate.ev_count++;
+#endif
 				nextsoftcheck = TAILQ_NEXT(c, c_link);
 				TAILQ_REMOVE(&bucket->cq_q, c, c_link);
 #ifdef CALLWHEEL_STATS
@@ -1055,6 +1060,8 @@
 	    NULL, "callwheel", "softempty");
 	evcnt_attach_dynamic(&callwheel_hintworked, EVCNT_TYPE_MISC,
 	    NULL, "callwheel", "hintworked");
+	evcnt_attach_dynamic(&callwheel_toolate, EVCNT_TYPE_MISC,
+	    NULL, "callwheel", "toolate");
 #endif /* CALLWHEEL_STATS */
 }

@@ -1206,6 +1213,8 @@
 		    (long long) callwheel_changed.ev_count);
 	printf("\tCallouts that fired: %llu\n",
 	    (long long) callwheel_fired.ev_count);
+	printf("\tCallouts fired late: %llu\n",
+	    (long long) callwheel_toolate.ev_count);
 	printf("\tNumber of buckets: %d\n", callwheelsize);
 	printf("\tNumber of hash collisions: %llu\n",
 	    (long long) callwheel_collisions.ev_count);
------------------------------

	However, so far, these events have not occurred before the
	various panics I've seen.

	Parts of the debug session for this latest panic follows
	below; here we have apparent corruption of the callwheel,
	where an entry appears to be removed (tqe_next == tqe_prev ==
	0x1), ref. the QUEUEDEBUG_TAILQ_POSTREMOVE() macro, but which
	is still sitting on the callwheel, and this triggers the panic
	when we later come to remove the next element in the chain.

	I'm not certain, but it does look as if something is
	manipulating the data structure without proper locking(?).

	The salient parts of the debug session (DDB and GDB, yes, I
	have a kernel core dump) follows:


panic: TAILQ_* back 0xc08af8ec /sys/arch/i386/compile/BLACKHOLE/../../../../kern/kern_clock.c:1160
Stopped at      cpu_Debugger+0x4:       leave
db> show reg
ds                0x10
es                0x10
fs                0x10
gs                0x10
edi              0x100
esi         0xc025ebf0  copyright+0x4d0
ebp         0xd2ec0b0c
ebx         0xd2ec0b38
edx         0xc02612f3  fmt.192+0x82e
ecx             0x7620
eax               0x67
eip         0xc01f19fc  cpu_Debugger+0x4
cs                 0x8
eflags           0x202
esp         0xd2ec0b0c
ss                0x10
cpu_Debugger+0x4:       leave
db>  
db> trace
cpu_Debugger(c545c830,e,10,c08dea6e,0) at cpu_Debugger+0x4
panic(c025ebf0,c08af8ec,c025eb80,488,c0000000) at panic+0xad
callout_stop_locked(c08af8ec,10,4,c01b87b2) at callout_stop_locked+0x80
callout_stop(c08af8ec,1,c01c0310,c08af8e0,130afc0) at callout_stop+0x26
tcp_input(c08c0c00,14,6,1,c08c0c00) at tcp_input+0x22e6
ip_input(c08c0c00,c0224c11,d2ec0d60,0) at ip_input+0x5f2
ipintr(10,10,10,10,d2be5900) at ipintr+0x6b
Xsoftnet() at Xsoftnet+0x2c
--- interrupt ---
idle(d2be5900,2c,c0164c2c,d2be5900) at idle+0x1b
bpendtsleep(c02c80cc,118,c0261500,2c,0) at bpendtsleep
sys_select(d2be5900,d2ec0f80,d2ec0f78,c0799038) at sys_select+0x300
syscall_plain(1f,1f,1f,1f,bfbfdb18) at syscall_plain+0xa7
db>  
db> x 0xc08af8ec ,4
0xc08af8ec:     c0968244    c09604b0    52a983      0
db> 
db> x 0xc09604b0 ,4
0xc09604b0:     1           1           52a984      0
db>  
db> x/x softclock_ticks ,2
softclock_ticks:        52a736      0
db> x/x hardclock_ticks ,2
hardclock_ticks:        52a736      0
db>  

...


(gdb) up
#10 0xc0158b54 in callout_stop_locked (c=0xc08af8ec)
    at /sys/arch/i386/compile/BLACKHOLE/../../../../kern/kern_clock.c:1160
1160            TAILQ_REMOVE(&bucket->cq_q, c, c_link);
(gdb) l
1155    
1156            if (nextsoftcheck == c)
1157                    nextsoftcheck = TAILQ_NEXT(c, c_link);
1158    
1159            bucket = &callwheel[c->c_time & callwheelmask];
1160            TAILQ_REMOVE(&bucket->cq_q, c, c_link);
1161            if (TAILQ_EMPTY(&bucket->cq_q))
1162                    bucket->cq_hint = UQUAD_MAX;
1163    #ifdef CALLWHEEL_STATS
1164            callwheel_count.ev_count--;
(gdb) p c
$1 = (struct callout *) 0x0

(bah, optimizer?!?)

(gdb) p 0xc08af8ec
$2 = 3230333164
(gdb) p (struct callout *)$
$3 = (struct callout *) 0xc08af8ec
(gdb) p *$
$4 = {c_link = {tqe_next = 0xc0968244, tqe_prev = 0xc09604b0}, 
  c_time = 5417347, c_arg = 0xc08af8e0, c_func = 0xc01c0058 <tcp_timer_rexmt>, 
  c_flags = 0}
(gdb) p $.c_link.tqe_prev
$5 = (struct callout **) 0xc09604b0
(gdb) p *$
$6 = (struct callout *) 0x1
(gdb) 
(gdb) p callwheel_count
$7 = {ev_count = 1049, ev_list = {tqe_next = 0xc02c59c0, 
    tqe_prev = 0xc02c5a48}, ev_type = 0 '\000', ev_grouplen = 9 '\t', 
  ev_namelen = 5 '\005', ev_pad1 = 0 '\000', ev_parent = 0x0, 
  ev_group = 0xc025ec11 "callwheel", ev_name = 0xc025ec25 "count"}
(gdb) 

... created gdb script:

define cwdump
        set $i = 0
        while ($i < callwheelsize)
                printf "Callwheel head %d\n", $i
                cwchain callwheel[$i].cq_q.tqh_first
                set $i++
        end
end     

define cwchain
        set $co = (struct callout*)$arg0
        while ($co)
                print *$co
                set $co = $co->c_link.tqe_next
        end
end

...

(gdb) cwdump
...
Callwheel head 387
$364 = {c_link = {tqe_next = 0x1, tqe_prev = 0x1}, c_time = 5417348, 
  c_arg = 0xc09604a4, c_func = 0, c_flags = 0}
can not access 0x1, invalid translation (invalid PDE)
can not access 0x1, invalid translation (invalid PDE)
Cannot access memory at address 0x1
(gdb) p $364
$365 = {c_link = {tqe_next = 0x1, tqe_prev = 0x1}, c_time = 5417348, 
  c_arg = 0xc09604a4, c_func = 0, c_flags = 0}
(gdb)  
(gdb) p &$
$366 = (struct callout *) 0xc09604b0
(gdb) 
(gdb) p softclock_ticks
$367 = 5416758
(gdb) 

(gdb) p *(struct callout*)0xc08af8ec
$369 = {c_link = {tqe_next = 0xc0968244, tqe_prev = 0xc09604b0}, 
  c_time = 5417347, c_arg = 0xc08af8e0, c_func = 0xc01c0058 <tcp_timer_rexmt>, 
  c_flags = 0}
(gdb) p *$.c_link.tqe_next
$370 = {c_link = {tqe_next = 0x0, tqe_prev = 0xc08af8ec}, c_time = 5417347, 
  c_arg = 0xc0968238, c_func = 0xc01c0058 <tcp_timer_rexmt>, c_flags = 3}
(gdb) 

	BTW, the activity levels for the callwheel are presently as
	follows:

% vmstat -v -e
event                                         total     rate type
callwheel collisions                       17059969     2391 misc
callwheel maxlength                              42        0 misc
callwheel count                                 916        0 misc
callwheel established                      20032006     2808 misc
callwheel fired                              291793       40 misc
callwheel disestablished                   19739297     2767 misc
callwheel changed                           5629534      789 misc
callwheel softclocks                         489552       68 misc
callwheel softempty                               2        0 misc
callwheel hintworked                            279        0 misc
callwheel toolate                                 0        0 misc

	The kernel configuration is as follows:

------------------------------	
# $NetBSD: GENERIC,v 1.491.4.3 2002/10/24 09:34:22 lukem Exp $
#
# GENERIC machine description file
# 
# This machine description file is used to generate the default NetBSD
# kernel.  The generic kernel does not include all options, subsystems
# and device drivers, but should be useful for most applications.
#
# The machine description file can be customised for your specific
# machine to reduce the kernel size and improve its performance.
#
# For further information on compiling NetBSD kernels, see the config(8)
# man page.
#
# For further information on hardware support for this architecture, see
# the intro(4) man page.  For further information about kernel options
# for this architecture, see the options(4) man page.  For an explanation
# of each device driver in this file see the section 4 man page for the
# device.

include 	"arch/i386/conf/std.i386"

options 	INCLUDE_CONFIG_FILE	# embed config file in kernel binary

#ident 		"GENERIC-$Revision: 1.491.4.3 $"

maxusers	32		# estimated number of users

#-----

# Increase mbuf clusters above GATEWAY=2048 setting
options 	NMBCLUSTERS=4096
options 	QUEUEDEBUG
options 	CALLWHEEL_STATS

#-----

# CPU support.  At least one is REQUIRED.
#options 	I386_CPU
#options 	I486_CPU
options 	I586_CPU
options 	I686_CPU

# CPU-related options.
#options 	MATH_EMULATE	# floating point emulation
#options 	VM86		# virtual 8086 emulation
#options 	USER_LDT	# user-settable LDT; used by WINE
# eliminate delay no-ops in I/O; recommended on all but very old machines
#options 	DUMMY_NOPS

#options 	MTRR		# memory-type range register syscall support
options 	PERFCTRS	# performance-monitoring counters support

# delay between "rebooting ..." message and hardware reset, in milliseconds
#options 	CPURESET_DELAY=2000

# This option allows you to force a serial console at the specified
# I/O address.   see console(4) for details.
#options 	CONSDEVNAME="\"com\"",CONADDR=0x2f8,CONSPEED=57600
#	you don't want the option below ON iff you are using the
#	serial console option of the new boot strap code.
#options 	CONS_OVERRIDE	# Always use above! independent of boot info

# The following options override the memory sizes passed in from the boot
# block.  Use them *only* if the boot block is unable to determine the correct
# values.  Note that the BIOS may *correctly* report less than 640k of base
# memory if the extended BIOS data area is located at the top of base memory
# (as is the case on most recent systems).
#options 	REALBASEMEM=639		# size of base memory (in KB)
#options 	REALEXTMEM=15360	# size of extended memory (in KB)

# Standard system options

#options 	UCONSOLE	# users can use TIOCCONS (for xconsole)
#options 	INSECURE	# disable kernel security levels

options 	RTC_OFFSET=0	# hardware clock is this many mins. west of GMT
options 	NTP		# NTP phase/frequency locked loop

options 	KTRACE		# system call tracing via ktrace(1)

options 	SYSVMSG		# System V-like message queues
options 	SYSVSEM		# System V-like semaphores
#options 	SEMMNI=10	# number of semaphore identifiers
#options 	SEMMNS=60	# number of semaphores in system
#options 	SEMUME=10	# max number of undo entries per process
#options 	SEMMNU=30	# number of undo structures in system
options 	SYSVSHM		# System V-like memory sharing
#options 	SHMMAXPGS=1024	# 1024 pages is the default

#options 	LKM		# loadable kernel modules

#options 	USERCONF	# userconf(4) support
#options 	PIPE_SOCKETPAIR	# smaller, but slower pipe(2)

# Diagnostic/debugging support options
#options 	DIAGNOSTIC	# expensive kernel consistency checks
#options 	DEBUG		# expensive debugging checks/support
options 	KMEMSTATS	# kernel memory statistics (vmstat -m)
options 	DDB		# in-kernel debugger
options 	DDB_HISTORY_SIZE=512	# enable history editing in DDB
#options 	KGDB		# remote debugger
#options 	KGDB_DEVNAME="\"com\"",KGDB_DEVADDR=0x3f8,KGDB_DEVRATE=9600
makeoptions	DEBUG="-g"	# compile full symbol table

# Compatibility options
#options 	COMPAT_NOMID	# compatibility with 386BSD, BSDI, NetBSD 0.8,
#options 	COMPAT_09	# NetBSD 0.9,
#options 	COMPAT_10	# NetBSD 1.0,
#options 	COMPAT_11	# NetBSD 1.1,
#options 	COMPAT_12	# NetBSD 1.2,
#options 	COMPAT_13	# NetBSD 1.3,
#options 	COMPAT_14	# NetBSD 1.4,
options 	COMPAT_15	# NetBSD 1.5,
options 	COMPAT_43	# and 4.3BSD
options 	COMPAT_386BSD_MBRPART # recognize old partition ID
#options 	TCP_COMPAT_42	# 4.2BSD TCP/IP bug compat. Not recommended.

#options 	COMPAT_SVR4	# binary compatibility with SVR4
#options 	COMPAT_IBCS2	# binary compatibility with SCO and ISC
#options 	COMPAT_LINUX	# binary compatibility with Linux
#options 	COMPAT_FREEBSD	# binary compatibility with FreeBSD
#options 	COMPAT_MACH	# binary compatibility with Mach binaries
#options 	EXEC_MACHO	# exec MACH-O binaries
#options 	COMPAT_PECOFF	# kernel support to run Win32 apps

# File systems
file-system 	FFS		# UFS
#file-system 	EXT2FS		# second extended file system (linux)
#file-system 	LFS		# log-structured file system
file-system 	MFS		# memory file system
#file-system 	NFS		# Network File System client
#file-system 	NTFS		# Windows/NT file system (experimental)
file-system 	CD9660		# ISO 9660 + Rock Ridge file system
file-system 	MSDOSFS		# MS-DOS file system
#file-system 	FDESC		# /dev/fd
file-system 	KERNFS		# /kern
#file-system 	NULLFS		# loopback file system
#file-system 	OVERLAY		# overlay file system
#file-system 	PORTAL		# portal filesystem (still experimental)
#file-system 	PROCFS		# /proc
#file-system 	UMAPFS		# NULLFS + uid and gid remapping
#file-system 	UNION		# union file system
#file-system	CODA		# Coda File System; also needs vcoda (below)

# File system options
#options 	QUOTA		# UFS quotas
#options 	FFS_EI		# FFS Endian Independent support
options 	SOFTDEP		# FFS soft updates support.
#options 	NFSSERVER	# Network File System server
#options 	EXT2FS_SYSTEM_FLAGS # makes ext2fs file flags (append and
				# immutable) behave as system flags.

# Networking options
#options 	GATEWAY		# packet forwarding
options 	INET		# IP + ICMP + TCP + UDP
options 	INET6		# IPV6
#options 	IPSEC		# IP security
#options 	IPSEC_ESP	# IP security (encryption part; define w/IPSEC)
#options 	IPSEC_DEBUG	# debug for IP security
#options 	MROUTING	# IP multicast routing
#options 	NS		# XNS
#options 	NSIP		# XNS tunneling over IP
#options 	ISO,TPIP	# OSI
#options 	EON		# OSI tunneling over IP
#options 	CCITT,LLC,HDLC	# X.25
#options 	NETATALK	# AppleTalk networking protocols
#options 	PPP_BSDCOMP	# BSD-Compress compression support for PPP
#options 	PPP_DEFLATE	# Deflate compression support for PPP
#options 	PPP_FILTER	# Active filter support for PPP (requires bpf)
#options 	PFIL_HOOKS	# pfil(9) packet filter hooks
#options 	IPFILTER_LOG	# ipmon(8) log support
#options 	IPFILTER_DEFAULT_BLOCK	# block all packets by default
#options 	TCP_DEBUG	# Record last TCP_NDEBUG packets with SO_DEBUG

# These options enable verbose messages for several subsystems.
# Warning, these may compile large string tables into the kernel!
#options 	EISAVERBOSE	# verbose EISA device autoconfig messages
options 	MIIVERBOSE	# verbose PHY autoconfig messages
options 	PCIVERBOSE	# verbose PCI device autoconfig messages
#options 	PCI_CONFIG_DUMP	# verbosely dump PCI config space
#options 	PCMCIAVERBOSE	# verbose PCMCIA configuration messages
options 	SCSIVERBOSE	# human readable SCSI error messages
options 	USBVERBOSE	# verbose USB device autoconfig messages
#options 	PNPBIOSVERBOSE	# verbose PnP BIOS messages
#options 	PNPBIOSDEBUG	# more fulsome PnP BIOS debugging messages
#options 	I2OVERBOSE	# verbose I2O driver messages
#options 	MCAVERBOSE	# verbose MCA device autoconfig messages

#options 	NFS_BOOT_DHCP,NFS_BOOT_BOOTPARAM

#
# wscons options
#
# builtin terminal emulations
#options 	WSEMUL_SUN		# sun terminal emulation
options 	WSEMUL_VT100		# VT100 / VT220 emulation
# different kernel output - see dev/wscons/wsdisplayvar.h
options 	WS_KERNEL_FG=WSCOL_GREEN
#options 	WS_KERNEL_BG=WSCOL_BLACK
# compatibility to other console drivers
options 	WSDISPLAY_COMPAT_PCVT		# emulate some ioctls
options 	WSDISPLAY_COMPAT_SYSCONS	# emulate some ioctls
options 	WSDISPLAY_COMPAT_USL		# VT handling
options 	WSDISPLAY_COMPAT_RAWKBD		# can get raw scancodes
# see dev/pckbc/wskbdmap_mfii.c for implemented layouts
#options 	PCKBD_LAYOUT="(KB_DE | KB_NODEAD)"
# allocate a number of virtual screens at autoconfiguration time
#options 	WSDISPLAY_DEFAULTSCREENS=4
# use a large software cursor that doesn't blink
options 	PCDISPLAY_SOFTCURSOR
# modify the screen type of the console; defaults to "80x25"
#options 	VGA_CONSOLE_SCREENTYPE="\"80x24\""

# Kernel root file system and dump configuration.
config		netbsd	root on ? type ?
#config		netbsd	root on sd0a type ffs
#config		netbsd	root on ? type nfs

#
# Device configuration
#

mainbus0 at root

#apm0	at mainbus0			# Advanced power management

# Tuning for power management, see apm(4) for more details.
#options 	APM_NO_IDLE		# Don't call BIOS CPU idle function
#options 	APM_V10_ONLY		# Use only the APM 1.0 calls
#options 	APM_NO_POWEROFF		# Don't power off on halt(8)
#options 	APM_POWER_PRINT		# Print stats on the console
#options 	APM_DISABLE_INTERRUPTS=0 # Don't disable interrupts


# Basic Bus Support

# Plug-and-Play BIOS and attached devices

#pnpbios*	at mainbus?

# mainboard audio chips
#ess*		at pnpbios? index ?	# ESS AudioDrive
#sb*		at pnpbios? index ?	# NeoMagic 256AV in sb mode
#wss*		at pnpbios? index ?	# NeoMagic 256AV in wss mode
#ym*		at pnpbios? index ?	# OPL3-SA3

# com port
# If enabled, consider changing "com0", "com1", and "com2" under "ISA Serial
# Interfaces" to "com*", otherwise com2 will attach at pnpbios? and there
# will be no com0.  A side effect is pcmcia (and other) com? previously
# starting at com3 may attach as com1 or com2.
#com*		at pnpbios? index ?	# serial ports

# parallel port
# The above "com*" comments apply, cf. "lpt0" under "ISA parallel
# "printer interfaces".
#lpt*		at pnpbios? index ?	# parallel ports

#pckbc*		at pnpbios? index ?	# PC keyboard/mouse controller
#fdc*		at pnpbios? index ?	# floppy controller

# IDE controller on Toshiba Portege 3000 series (crippled PCI device)
#pciide*	at pnpbios? index ?

# PCI bus support
pci*	at mainbus? bus ?
pci*	at pchb? bus ?
pci*	at ppb? bus ?

# Configure PCI using BIOS information
#options 	PCIBIOS			# PCI BIOS support
#options 	PCIBIOSVERBOSE		# PCI BIOS verbose info
#options 	PCIBIOS_ADDR_FIXUP	# fixup PCI I/O addresses
#options 	PCIBIOS_BUS_FIXUP	# fixup PCI bus numbering
#options 	PCIBIOS_INTR_FIXUP	# fixup PCI interrupt routing
#options 	PCIBIOS_IRQS_HINT=0x0a00 # PCI interrupts hint. IRQ 9 or 11
#options 	PCIBIOS_INTR_GUESS	# see pcibios(4)
#options 	PCIINTR_DEBUG		# super-verbose PCI interrupt fixup

# PCI bridges
pchb*	at pci? dev ? function ?	# PCI-Host bridges
#pceb*	at pci? dev ? function ?	# PCI-EISA bridges
pcib*	at pci? dev ? function ?	# PCI-ISA bridges
#pcmb*	at pci? dev ? function ?	# PCI-MCA bridges
ppb*	at pci? dev ? function ?	# PCI-PCI bridges
# XXX 'puc's aren't really bridges, but there's no better place for them here
puc*	at pci? dev ? function ?	# PCI "universal" comm. cards

agp* 	at pchb?

# EISA bus support
#eisa0	at mainbus?
#eisa0	at pceb?

# ISA bus support
isa0	at mainbus?
#isa0	at pceb?
isa0	at pcib?

# PCMCIA bus support
#pcmcia*	at pcic? controller ? socket ?
#pcmcia*	at tcic? controller ? socket ?

# MCA bus support
#mca0	at mainbus?

# ISA PCMCIA controllers
#pcic0	at isa? port 0x3e0 iomem 0xd0000 iosiz 0x10000
#pcic1	at isa? port 0x3e2 iomem 0xe0000 iosiz 0x4000
#pcic2	at isa? port 0x3e4 iomem 0xe0000 iosiz 0x4000
#tcic0	at isa? port 0x240 iomem 0xd0000 iosiz 0x10000

# PCI PCMCIA controllers
#pcic0	at pci? dev? function ?

# ISA Plug-and-Play bus support
isapnp0	at isa?

# ISA Plug-and-Play PCMCIA controllers
#pcic*	at isapnp?

# CardBus bridge support
#cbb*		at pci? dev ? function ?
#cardslot*	at cbb?

# CardBus bus support
#cardbus*	at cardslot?
#pcmcia* 	at cardslot?

# Coprocessor Support

# Math Coprocessor support
npx0	at isa? port 0xf0 irq 13	# x86 math coprocessor


# Console Devices

# ISA console
#pc0	at isa? port 0x60 irq 1		# pccons generic PC console driver
# Keyboard layout configuration for pccons
#options 	FRENCH_KBD
#options 	FINNISH_KBD
#options 	GERMAN_KBD
#options 	NORWEGIAN_KBD
# pccons-specific options:
#options 	XSERVER_DDB	# PF12 gets you into DDB when X is running
#options 	XSERVER		# X server support


# wscons
pckbc0		at isa?			# pc keyboard controller
pckbd*		at pckbc?		# PC keyboard
pms*		at pckbc?		# PS/2 mouse for wsmouse
#options 	PMS_DISABLE_POWERHOOK
vga0		at isa?
vga*		at pci? dev ? function ?
pcdisplay0	at isa?			# CGA, MDA, EGA, HGA
wsdisplay*	at vga? console ?
wsdisplay*	at pcdisplay? console ?
wskbd* 		at pckbd? console ?
wsmouse*	at pms? mux 0

pcppi0		at isa?
sysbeep0	at pcppi?

# Serial Devices

# PCI serial interfaces
com*	at puc? port ?			# 16x50s on "universal" comm boards
#cy*	at pci? dev ? function ?	# Cyclades Cyclom-Y serial boards
#cz*	at pci? dev ? function ?	# Cyclades-Z multi-port serial boards

# ISA Plug-and-Play serial interfaces
#com*	at isapnp?			# Modems and serial boards

# PCMCIA serial interfaces
#com*	at pcmcia? function ?		# Modems and serial cards

#pcmcom*	at pcmcia? function ?		# PCMCIA multi-port serial cards
#com*	at pcmcom? slave ?		# ...and the slave devices

# CardBus serial interfaces
#com*	at cardbus? dev ? function ?	# Modems and serial cards

# ISA serial interfaces
#options 	COM_HAYESP		# adds Hayes ESP serial board support
com0	at isa? port 0x3f8 irq 4	# Standard PC serial ports
com1	at isa? port 0x2f8 irq 3
com2	at isa? port 0x3e8 irq 5
#com3	at isa? port 0x2e8 irq 9
#ast0	at isa? port 0x1a0 irq 5	# AST 4-port serial cards
#com*	at ast? slave ?
#boca0	at isa? port 0x100 irq 5	# BOCA 8-port serial cards
#boca0	at isa? port 0x100 irq 5	# BOCA 16-port serial cards (BB2016)
#boca1	at isa? port 0x140 irq 5	# this line is also needed for BB2016
#com*	at boca? slave ?
#tcom0	at isa? port 0x100 irq 7	# TC-800 8-port serial cards
#com*	at tcom? slave ?
#rtfps0	at isa? port 0x1230 irq 10	# RT 4-port serial cards
#com*	at rtfps? slave ?
#cy0	at isa? iomem 0xd4000 irq 12	# Cyclades serial cards
#addcom0 at isa? port 0x108 irq 5	# Addonics FlexPort 8S
#com*	at addcom? slave ?
#moxa0	at isa?	port 0x100 irq 5	# MOXA C168H serial card (experimental)
#com*	at moxa? slave ?

# MCA serial interfaces
#com*	at mca? slot ?			# 16x50s on comm boards

# Parallel Printer Interfaces

# PCI parallel printer interfaces
lpt*	at puc? port ?			# || ports on "universal" comm boards

# ISA parallel printer interfaces
lpt0	at isa? port 0x378 irq 7	# standard PC parallel ports
lpt1	at isa? port 0x278
lpt2	at isa? port 0x3bc

# Hardware monitors

# LM7[89] and compatible hardware monitors
#lm0	at isa?	port 0x290		# other common ports: 0x280, 0x310

# VIA VT82C686A hardware monitor
#viapm*	at pci? dev ? function ?
#viaenv* at viapm?


# I2O devices
#iop*	at pci? dev ? function ?	# I/O processor
#iopsp*	at iop? tid ?			# SCSI/FC-AL ports
#ld*	at iop? tid ?			# block devices
#dpti*	at iop? tid 0			# DPT/Adaptec control interface


# SCSI Controllers and Devices

# PCI SCSI controllers
#adv*	at pci? dev ? function ?	# AdvanSys 1200[A,B], 9xx[U,UA] SCSI
#adw*	at pci? dev ? function ?	# AdvanSys 9x0UW[D], 3940U[2,3]W SCSI
#ahc*	at pci? dev ? function ?	# Adaptec [23]94x, aic78x0 SCSI
#bha*	at pci? dev ? function ?	# BusLogic 9xx SCSI
#dpt*	at pci? dev ? function ?	# DPT SmartCache/SmartRAID
#iha*	at pci? dev ? function ?	# Initio INIC-940/950 SCSI
#isp*	at pci? dev ? function ?	# Qlogic ISP [12]0x0 SCSI/FibreChannel
#mly*	at pci? dev ? function ?	# Mylex AcceleRAID and eXtremeRAID
#pcscp*	at pci? dev ? function ?	# AMD 53c974 PCscsi-PCI SCSI
#siop*	at pci? dev ? function ?	# Symbios 53c8xx SCSI
#options 	SIOP_SYMLED		# drive the act. LED in software
#trm*	at pci? dev ? function ?	# Tekram DC-395U/UW/F, DC-315/U SCSI

# EISA SCSI controllers
#ahb*	at eisa? slot ?			# Adaptec 174[02] SCSI
#ahc*	at eisa? slot ?			# Adaptec 274x, aic7770 SCSI
#bha*	at eisa? slot ?			# BusLogic 7xx SCSI
#dpt*	at eisa? slot ?			# DPT EATA SCSI
#uha*	at eisa? slot ?			# UltraStor 24f SCSI

# PCMCIA SCSI controllers
#aic*	at pcmcia? function ?		# Adaptec APA-1460 SCSI
#esp*	at pcmcia? function ?		# Qlogic ESP406/FAS408 SCSI

# ISA Plug-and-Play SCSI controllers
#aha*	at isapnp? 			# Adaptec AHA-154[02
#aic*	at isapnp?			# Adaptec AHA-1520B

# ISA SCSI controllers
#adv0	at isa? port ? irq ? drq ?	# AdvanSys APB-514[02] SCSI
#aha0	at isa? port 0x330 irq ? drq ?	# Adaptec 154[02] SCSI
#aha1	at isa? port 0x334 irq ? drq ?
#ahc0	at isa? port ? irq ?		# Adaptec 284x SCSI
#aic0	at isa? port 0x340 irq 11	# Adaptec 152[02] SCSI
#bha0	at isa? port 0x330 irq ? drq ?	# BusLogic [457]4X SCSI
#bha1	at isa? port 0x334 irq ? drq ?
# The "nca" and "dpt" probes might give false hits or hang your machine.
#dpt0	at isa? port 0x170 irq ? drq ?	# DPT SmartCache/SmartRAID
#nca0	at isa? port 0x360 irq 15	# Port-mapped NCR 53C80 controller
#nca1	at isa? iomem 0xd8000 irq 5	# Memory-mapped controller (T128, etc.)
#sea0	at isa? iomem 0xc8000 irq 5	# Seagate/Future Domain SCSI
#uha0	at isa? port 0x330 irq ? drq ?	# UltraStor [13]4f SCSI
#uha1	at isa? port 0x340 irq ? drq ?
#wds0	at isa? port 0x350 irq 15 drq 6	# WD7000 and TMC-7000 controllers
#wds1	at isa? port 0x358 irq 11 drq 5

# CardBus SCSI cards
#adv*	at cardbus? dev ? function ?	# AdvanSys 1200[A,B], 9xx[U,UA] SCSI
#ahc*	at cardbus? dev ? function ?	# Adaptec ADP-1480

# MCA SCSI cards
#aha*	at mca? slot ?			# Adaptec AHA-1640

# SCSI bus support
#scsibus* at adv?
#scsibus* at adw?
#scsibus* at aha?
#scsibus* at ahb?
#scsibus* at ahc?
#scsibus* at aic?
#scsibus* at bha?
#scsibus* at dpt?
#scsibus* at esp?
#scsibus* at iha?
#scsibus* at iopsp?
#scsibus* at isp?
#scsibus* at mly?
#scsibus* at nca?
#scsibus* at pcscp?
#scsibus* at sea?
#scsibus* at siop?
#scsibus* at trm?
#scsibus* at uha?
#scsibus* at wds?

# SCSI devices
#sd*	at scsibus? target ? lun ?	# SCSI disk drives
#st*	at scsibus? target ? lun ?	# SCSI tape drives
#cd*	at scsibus? target ? lun ?	# SCSI CD-ROM drives
#ch*	at scsibus? target ? lun ?	# SCSI autochangers
#ses*	at scsibus? target ? lun ?	# SCSI Enclosure Services devices
#ss*	at scsibus? target ? lun ?	# SCSI scanners
#uk*	at scsibus? target ? lun ?	# SCSI unknown


# RAID controllers and devices
#aac*	at pci? dev ? function ?	# Adaptec AAC family
#cac*	at eisa? slot ?			# Compaq EISA array controllers
#cac*	at pci? dev ? function ?	# Compaq PCI array controllers
#icp*	at pci? dev ? function ?	# ICP-Vortex GDT & Intel RAID
#mlx*	at pci? dev ? function ?	# Mylex DAC960 & DEC SWXCR family
#mlx*	at eisa? slot ?			# Mylex DAC960 & DEC SWXCR family
#twe*	at pci? dev ? function ?	# 3ware Escalade RAID controllers

#ld*	at aac? unit ?			# logical disk devices
#ld*	at cac? unit ?
#ld*	at icp? unit ?
#ld*	at twe? unit ?
#ld*	at mlx? unit ?

#icpsp*	at icp? unit ?			# SCSI pass-through
#scsibus* at icpsp?

# IDE and related devices
# PCI IDE controllers - see pciide(4) for supported hardware.
# The 0x0001 flag force the driver to use DMA, even if the driver doesn't know
# how to set up DMA modes for this chip. This may work, or may cause
# a machine hang with some controllers.
pciide* at pci? dev ? function ? flags 0x0000

# ISA Plug-and-Play IDE controllers
#wdc*	at isapnp?

# PCMCIA IDE controllers
#wdc*	at pcmcia? function ?

# ISA ST506, ESDI, and IDE controllers
# Use flags 0x01 if you want to try to use 32bits data I/O (the driver will
# fall back to 16bits I/O if 32bits I/O are not functional).
# Some controllers pass the initial 32bit test, but will fail later.
#wdc0	at isa? port 0x1f0 irq 14 flags 0x00
#wdc1	at isa? port 0x170 irq 15 flags 0x00

# IDE drives
# Flags are used only with controllers that support DMA operations
# and mode settings (e.g. some pciide controllers)
# The lowest order four bits (rightmost digit) of the flags define the PIO
# mode to use, the next set of four bits the DMA mode and the third set the
# UltraDMA mode. For each set of four bits, the 3 lower bits define the mode
# to use, and the last bit must be 1 for this setting to be used.
# For DMA and UDMA, 0xf (1111) means 'disable'.
# 0x0fac means 'use PIO mode 4, DMA mode 2, disable UltraDMA'.
# (0xc=1100, 0xa=1010, 0xf=1111)
# 0x0000 means "use whatever the drive claims to support".
#wd*	at wdc? channel ? drive ? flags 0x0000
wd*	at pciide? channel ? drive ? flags 0x0000

# ATAPI bus support
#atapibus* at wdc? channel ?
atapibus* at pciide? channel ?

# ATAPI devices
# flags have the same meaning as for IDE drives.
cd*	at atapibus? drive ? flags 0x0000	# ATAPI CD-ROM drives
sd*	at atapibus? drive ? flags 0x0000	# ATAPI disk drives
st*	at atapibus? drive ? flags 0x0000	# ATAPI tape drives
uk*	at atapibus? drive ? flags 0x0000	# ATAPI unknown


# Miscellaneous mass storage devices

# ISA floppy
fdc0	at isa? port 0x3f0 irq 6 drq 2	# standard PC floppy controllers
#fdc1	at isa? port 0x370 irq ? drq ?
fd*	at fdc? drive ?			# the drives themselves
# some machines need you to do this instead of fd*
#fd0	at fdc0 drive 0

# ISA CD-ROM devices
#mcd0	at isa? port 0x300 irq 10	# Mitsumi CD-ROM drives

# ISA tape devices
# note: the wt driver conflicts unpleasantly with SMC boards at the
# same I/O address. The probe reprograms their EEPROMs. Don't
# uncomment it unless you are actually using it.
#wt0	at isa? port 0x308 irq 5 drq 1	# Archive and Wangtek QIC tape drives

# MCA ESDI devices
#edc*	at mca? slot ?			# IBM ESDI Disk Controllers
#ed*	at edc?


# Network Interfaces

# PCI network interfaces
#an*	at pci? dev ? function ?	# Aironet PC4500/PC4800 (802.11)
#en*	at pci? dev ? function ?	# ENI/Adaptec ATM
#ep*	at pci? dev ? function ?	# 3Com 3c59x
#epic*	at pci? dev ? function ?	# SMC EPIC/100 Ethernet
#esh*	at pci? dev ? function ?	# Essential HIPPI card
ex*	at pci? dev ? function ?	# 3Com 90x[BC]
#fpa*	at pci? dev ? function ?	# DEC DEFPA FDDI
fxp*	at pci? dev ? function ?	# Intel EtherExpress PRO 10+/100B
#gsip*	at pci? dev ? function ?	# NS83820 Gigabit Ethernet
#hme*	at pci? dev ? function ?	# Sun Microelectronics STP2002-STQ
#le*	at pci? dev ? function ?	# PCnet-PCI Ethernet
#lmc*	at pci? dev ? function ?	# Lan Media Corp SSI/HSSI/DS3
#ne*	at pci? dev ? function ?	# NE2000-compatible Ethernet
#ntwoc*	at pci? dev ? function ?	# Riscom/N2 PCI Sync Serial
#pcn*	at pci? dev ? function ?	# AMD PCnet-PCI Ethernet
#rtk*	at pci? dev ? function ?	# Realtek 8129/8139
#sf*	at pci? dev ? function ?	# Adaptec AIC-6915 Ethernet
#sip*	at pci? dev ? function ?	# SiS 900/DP83815 Ethernet
#ste*	at pci? dev ? function ?	# Sundance ST-201 Ethernet
#stge*	at pci? dev ? function ?	# Sundance/Tamarack TC9021 Gigabit
#ti*	at pci? dev ? function ?	# Alteon ACEnic gigabit Ethernet
#tl*	at pci? dev ? function ?	# ThunderLAN-based Ethernet
tlp*	at pci? dev ? function ?	# DECchip 21x4x and clones
#vr*	at pci? dev ? function ?	# VIA Rhine Fast Ethernet
#wi*	at pci? dev ? function ?	# Intersil Prism Mini-PCI (802.11b)
#wm*	at pci? dev ? function ?	# Intel 82543/82544 gigabit

# EISA network interfaces
#ep*	at eisa? slot ?			# 3Com 3c579 Ethernet
#fea*	at eisa? slot ?			# DEC DEFEA FDDI
#tlp*	at eisa? slot ?			# DEC DE-425 Ethernet

# ISA Plug-and-Play network interfaces
#an*	at isapnp?			# Aironet 802.11
#ep*	at isapnp?			# 3Com 3c509 Ethernet
#ne*	at isapnp?			# NE2000-compatible Ethernet
#tr*	at isapnp?			# IBM/3COM TROPIC Token-Ring

# PCMCIA network interfaces
#an*	at pcmcia? function ?		# Aironet PC4500/PC4800 (802.11)
#awi*	at pcmcia? function ?		# BayStack 650/660 (802.11FH/DS)
#cnw*	at pcmcia? function ?		# Xircom/Netwave AirSurfer
#ep*	at pcmcia? function ?		# 3Com 3c589 and 3c562 Ethernet
#mbe*	at pcmcia? function ?		# MB8696x based Ethernet
#ne*	at pcmcia? function ?		# NE2000-compatible Ethernet
#ray*	at pcmcia? function ?		# Raytheon Raylink (802.11)
#sm*	at pcmcia? function ?		# Megahertz Ethernet
# tr at pcmcia has problems with Cardbus bridges
#tr*	at pcmcia? function ?		# TROPIC based Token-Ring
#wi*	at pcmcia? function ?		# Lucent WaveLan IEEE (802.11)
#xi*	at pcmcia? function ?		# Xircom CreditCard Ethernet

#mhzc*	at pcmcia? function ?		# Megahertz Ethernet/Modem combo cards
#com*	at mhzc?
#sm*	at mhzc?

# ISA network interfaces
#ate0	at isa? port 0x2a0 irq ?		# AT1700
#cs0	at isa? port 0x300 iomem ? irq ? drq ?	# CS8900 Ethernet
#ec0	at isa? port 0x250 iomem 0xd8000 irq 9	# 3Com 3c503 Ethernet
#eg0	at isa? port 0x280 irq 9		# 3C505 ethernet cards
#el0	at isa? port 0x300 irq 9		# 3C501 ethernet cards
#ep*	at isa? port ? irq ?			# 3C509 ethernet cards
#ef0	at isa? port 0x360 iomem 0xd0000 irq 7	# 3C507
#ai0	at isa? port 0x360 iomem 0xd0000 irq 7	# StarLAN
#fmv0	at isa? port 0x2a0 irq ?		# FMV-180 series
#ix0	at isa? port 0x300 irq 10		# EtherExpress/16
#iy0	at isa? port 0x360 irq ?		# EtherExpress PRO 10 ISA
#lc0	at isa? port 0x320 iomem ? irq ?	# DEC EtherWORKS III (LEMAC)
#depca0	at isa? port 0x300 iomem 0xc8000 iosiz 0x8000 irq 5	# DEPCA
#le*	at depca?
#nele0	at isa? port 0x320 irq 9 drq 7		# NE2100
#le*	at nele?
#ntwoc0	at isa? port 0x300 irq 5 iomem 0xc8000 flags 1	# Riscom/N2 sync serial
#bicc0	at isa? port 0x320 irq 10 drq 7		# BICC IsoLan
#le*	at bicc?
#ne0	at isa? port 0x280 irq 9		# NE[12]000 ethernet cards
#ne1	at isa? port 0x300 irq 10
#sm0	at isa? port 0x300 irq 10		# SMC91C9x Ethernet
#tr0	at isa? port 0xa20 iomem 0xd8000 irq ?	# IBM TROPIC based Token-Ring
#tr1	at isa? port 0xa24 iomem 0xd0000 irq ?	# IBM TROPIC based Token-Ring
#tr*	at isa? port ? irq ?			# 3COM TROPIC based Token-Ring
#we0	at isa? port 0x280 iomem 0xd0000 irq 9	# WD/SMC Ethernet
#we1	at isa? port 0x300 iomem 0xcc000 irq 10

# CardBus network cards
#ex*	at cardbus? dev ? function ?	# 3Com 3C575TX
#fxp*	at cardbus? dev ? function ?	# Intel i8255x
#rtk*	at cardbus? dev ? function ?	# Realtek 8129/8139
#tlp*	at cardbus? dev ? function ?	# DECchip 21143

# MCA network cards
#elmc*	at mca? slot ?			# 3Com EtherLink/MC (3c523)
#ep*	at mca? slot ?			# 3Com EtherLink III (3c529)
#we*	at mca? slot ?			# WD/SMC Ethernet
#ate*	at mca? slot ?			# Allied Telesis AT1720
#ne*	at mca? slot ?			# Novell NE/2 and clones
#tr*	at mca? slot ?			# IBM Token Ring adapter
#le*	at mca? slot ?			# SKNET Personal/MC2+

# MII/PHY support
#acphy*	at mii? phy ?			# DAltima AC101 and AMD Am79c874 PHYs
#amhphy* at mii? phy ?			# AMD 79c901 Ethernet PHYs
#bmtphy* at mii? phy ?			# Broadcom BCM5201 and BCM5202 PHYs
#brgphy* at mii? phy ?			# Broadcom BCM5400-family PHYs
#dmphy*	at mii? phy ?			# Davicom DM9101 PHYs
#exphy*	at mii? phy ?			# 3Com internal PHYs
#glxtphy* at mii? phy ?			# Level One LXT-1000 PHYs
#gphyter* at mii? phy ?			# NS83861 Gig-E PHY
#icsphy* at mii? phy ?			# Integrated Circuit Systems ICS189x
inphy*	at mii? phy ?			# Intel 82555 PHYs
#iophy*	at mii? phy ?			# Intel 82553 PHYs
#lxtphy* at mii? phy ?			# Level One LXT-970 PHYs
#makphy* at mii? phy ?			# Marvell Semiconductor 88E1000 PHYs
#nsphy*	at mii? phy ?			# NS83840 PHYs
#nsphyter* at mii? phy ?			# NS83843 PHYs
#pnaphy* at mii? phy ?			# generic HomePNA PHYs
#qsphy*	at mii? phy ?			# Quality Semiconductor QS6612 PHYs
#sqphy*	at mii? phy ?			# Seeq 80220/80221/80223 PHYs
#tlphy*	at mii? phy ?			# ThunderLAN PHYs
#tqphy*	at mii? phy ?			# TDK Semiconductor PHYs
ukphy*	at mii? phy ?			# generic unknown PHYs
#urlphy* at mii? phy ?			# Realtek RTL8150L internal PHYs


# USB Controller and Devices

# PCI USB controllers
# The EHCI is not ready for prime time.
#ehci*	at pci?	dev ? function ?	# Enhanced Host Controller
ohci*	at pci?	dev ? function ?	# Open Host Controller
uhci*	at pci?	dev ? function ?	# Universal Host Controller (Intel)

# CardBus USB controllers
#ehci*	at cardbus? dev ? function ?	# Enhanced Host Controller
#ohci*	at cardbus? dev ? function ?	# Open Host Controller

# USB bus support
#usb*	at ehci?
usb*	at ohci?
usb*	at uhci?

# USB Hubs
uhub*	at usb?
uhub*	at uhub? port ? configuration ? interface ?

# USB HID device
uhidev*	at uhub? port ? configuration ? interface ?

# USB Mice
ums*	at uhidev? reportid ?
wsmouse* at ums? mux 0

# USB Keyboards
ukbd*	at uhidev? reportid ?
wskbd*	at ukbd? console ? mux 1

# USB Generic HID devices
uhid*	at uhidev? reportid ?

# USB Printer
#ulpt*	at uhub? port ? configuration ? interface ?

# USB Modem
umodem*	at uhub? port ? configuration ?
ucom*	at umodem?

# USB Mass Storage
umass*	at uhub? port ? configuration ? interface ?
atapibus* at umass? channel ?
scsibus* at umass? channel ?
#wd* at umass?

# USB audio
#uaudio*	at uhub? port ? configuration ?

# USB MIDI
#umidi* at uhub? port ? configuration ?

# USB IrDA
# USB-IrDA bridge spec
#uirda* at uhub? port ? configuration ? interface ?
#irframe* at uirda?

# SigmaTel STIr4200 USB/IrDA Bridge
#ustir* at uhub? port ?
#irframe* at ustir?

# USB Ethernet adapters
#aue*	at uhub? port ?		# ADMtek AN986 Pegasus based adapters
#cue*	at uhub? port ?		# CATC USB-EL1201A based adapters
#kue*	at uhub? port ?		# Kawasaki LSI KL5KUSB101B based adapters
#url*	at uhub? port ?		# Realtek RTL8150L based adapters

# Prolific PL2301/PL2302 host-to-host adapter
#upl*	at uhub? port ?

# Serial adapters
#uftdi*	at uhub? port ?		# FTDI FT8U100AX serial adapter
#ucom*	at uftdi? portno ?

#umct*	at uhub? port ?		# MCT USB-RS232 serial adapter
#ucom*	at umct? portno ?

#uplcom*	at uhub? port ?		# I/O DATA USB-RSAQ2 serial adapter
#ucom*	at uplcom? portno ?

#uvscom*	at uhub? port ?		# SUNTAC Slipper U VS-10U serial adapter
#ucom*	at uvscom? portno ?

# Diamond Multimedia Rio 500
#urio*	at uhub? port ?

# USB Handspring Visor
#uvisor*	at uhub? port ?
#ucom*	at uvisor?

# USB scanners
#uscanner* at uhub? port ?

# USB scanners that use SCSI emulation, e.g., HP5300
#usscanner* at uhub? port ?
#scsibus* at usscanner? channel ?

# Y@P firmware loader
#uyap* at uhub? port ?

# D-Link DSB-R100 USB radio
#udsbr*	at uhub? port ?
#radio*	at udsbr?

# USB Generic driver
#ugen*	at uhub? port ?


# IrDA and Consumer Ir devices

# Toshiba Oboe
#oboe* 	at pci? dev ? function ?
#irframe* at oboe?


# Audio Devices

# PCI audio devices
#auich*	at pci? dev ? function ?	# Intel ICH integrated AC'97 Audio
#autri*	at pci? dev ? function ?	# Trident 4DWAVE based AC'97 Audio
auvia*	at pci? dev ? function ?	# VIA VT82C686A integrated AC'97 Audio
#clcs*	at pci? dev ? function ?	# Cirrus Logic CS4280
#clct*	at pci? dev ? function ?	# Cirrus Logic CS4281
#cmpci*	at pci? dev ? function ?	# C-Media CMI8338/8738
#eap*	at pci? dev ? function ?	# Ensoniq AudioPCI
#emuxki*	at pci? dev ? function ?	# Creative SBLive! and PCI512
#esa*	at pci? dev ? function ?	# ESS Allegro-1 / Maestro-3 PCI Audio
#esm*	at pci? dev ? function ?	# ESS Maestro-1/2/2e PCI Audio Accelerator
#eso*	at pci? dev ? function ?	# ESS Solo-1 PCI AudioDrive
#fms*	at pci? dev ? function ?	# Forte Media FM801
#neo*	at pci? dev ? function ?	# NeoMagic 256 AC'97 Audio
#sv*	at pci? dev ? function ?	# S3 SonicVibes
#yds*	at pci? dev ? function ?	# Yamaha DS-1 PCI Audio

# ISA Plug-and-Play audio devices
#ess*	at isapnp?			# ESS Tech ES1887, ES1888, ES888 audio
#guspnp*	at isapnp?			# Gravis Ultra Sound PnP audio
#sb*	at isapnp?			# SoundBlaster-compatible audio
#wss*	at isapnp?			# Windows Sound System
#ym*	at isapnp?			# Yamaha OPL3-SA3 audio

# ISA audio devices
# the "aria" probe might give false hits
#aria0	at isa? port 0x290 irq 10 		# Aria
#ess0	at isa? port 0x220 irq 5 drq 1 drq2 5	# ESS 18XX
#gus0	at isa? port 0x220 irq 7 drq 1 drq2 6	# Gravis Ultra Sound
#pas0	at isa? port 0x220 irq 7 drq 1		# ProAudio Spectrum
#pss0	at isa? port 0x220 irq 7 drq 6		# Personal Sound System
#sp0	at pss0 port 0x530 irq 10 drq 0		# 	sound port driver
#sb0	at isa? port 0x220 irq 5 drq 1 drq2 5	# SoundBlaster
#wss0	at isa? port 0x530 irq 10 drq 0	drq2 1	# Windows Sound System

# PCMCIA audio devices
#esl*	at pcmcia? function ?			# ESS 1688 AudioDrive

#cms0	at isa? port 0x220			# Creative Music System

# OPL[23] FM synthesizers
#opl0	at isa? port 0x388	# use only if not attached to sound card
#opl*	at cmpci? flags 1
#opl*	at esl?
#opl*	at eso?
#opl*	at ess?
#opl*	at fms?
#opl*	at sb?
#opl*	at sv?
#opl*	at wss?
#opl*	at yds?
#opl*	at ym?

# Audio support
#audio*	at aria?
#audio*	at auich?
#audio*	at autri?
audio*	at auvia?
#audio*	at clcs?
#audio*	at clct?
#audio*	at cmpci?
#audio*	at eap?
#audio*	at emuxki?
#audio*	at esa?
#audio*	at esl?
#audio*	at esm?
#audio*	at eso?
#audio*	at ess?
#audio*	at fms?
#audio*	at gus?
#audio*	at guspnp?
#audio*	at neo?
#audio*	at pas?
#audio*	at sb?
#audio*	at sp?
#audio*	at sv?
#audio*	at uaudio?
#audio*	at wss?
#audio*	at yds?
#audio*	at ym?

# MPU 401 UARTs
#mpu*	at isa? port 0x330 irq 9	# MPU401 or compatible card
#mpu*	at cmpci?
#mpu*	at eso?
#mpu*	at fms?
#mpu*	at sb?
#mpu*	at yds?
#mpu*	at ym?

# MIDI support
#midi*	at autri?		# Trident 4DWAVE MIDI port
#midi*	at clcs?		# Cirrus Logic CS4280 MIDI port
#midi*	at clct?		# Cirrus Logic CS4281 MIDI port
#midi*	at cms?			# Creative Music System
#midi*	at eap?			# 137[01] MIDI port
#midi*	at mpu?			# MPU 401
#midi*	at opl?			# OPL FM synth
#midi*	at pcppi?		# MIDI interface to the PC speaker
#midi*	at sb?			# SB1 MIDI port
#midi*	at umidi?		# USB MIDI

# The spkr driver provides a simple tone interface to the built in speaker.
#spkr0	at pcppi?		# PC speaker


# FM-Radio devices
# ISA radio devices
#az0	at isa? port 0x350      # Aztech/PackardBell FM Radio Card
#az1	at isa? port 0x358
#rt0	at isa? port 0x20c      # AIMS Lab Radiotrack & compatible
#rt1	at isa? port 0x284
#rt2	at isa? port 0x30c
#rt3	at isa? port 0x384
#rtii0	at isa? port 0x20c      # AIMS Lab Radiotrack II FM Radio Card
#rtii1	at isa? port 0x30c
#sf2r0	at isa? port 0x384      # SoundForte RadioLink SF16-FMR2 FM Radio Card

# PCI radio devices
#mr*	at pci? dev ? function ? # Guillemot Maxi Radio FM 2000 Radio Card

# Radio support
#radio*	at az?
#radio*	at mr?
#radio*	at rt?
#radio*	at rtii?
#radio*	at sf2r?


# TV cards

# Brooktree 848/849/878/879 based TV cards
#bktr* at pci? dev ? function ?
#radio* at bktr?


# Mice

# ISA busmice
#lms0	at isa? port 0x23c irq 5	# Logitech bus mouse
#lms1	at isa? port 0x238 irq 5
#mms0	at isa? port 0x23c irq 5	# Microsoft InPort mouse
#mms1	at isa? port 0x238 irq 5
#wsmouse*	at lms? mux 0
#wsmouse*	at mms? mux 0
# backwards compatible versions
#olms0	at isa? port 0x23c irq 5	# Logitech bus mouse
#olms1	at isa? port 0x238 irq 5
#omms0	at isa? port 0x23c irq 5	# Microsoft InPort mouse
#omms1	at isa? port 0x238 irq 5


# Joysticks

#joy*		at pnpbios? index ?	# Game port

# ISA Plug-and-Play joysticks
#joy*	at isapnp?			# Game ports (usually on audio cards)

# PCI joysticks
#joy*	at pci?				# Game ports (usually on audio cards)
#joy*	at eso?				# ESS Solo-1 on-board joystick

# ISA joysticks. Probe is a little strange; add only if you have one.
#joy0	at isa? port 0x201


# Miscellaneous Devices

# Planetconnect Satellite receiver driver.
#satlink0 at isa? port 0x300 drq 1

# Middle Digital, Inc. PCI-Weasel serial console board control
# devices (watchdog timer, etc.)
weasel* at pci?

# Pull in optional local configuration
include	"arch/i386/conf/GENERIC.local"


# Pseudo-Devices

# disk/mass storage pseudo-devices
#pseudo-device	ccd		4	# concatenated/striped disk devices
#pseudo-device	raid		8	# RAIDframe disk driver
#options 	RAID_AUTOCONFIG		# auto-configuration of RAID components
#Options to enable various other RAIDframe RAID types.
#options 	RF_INCLUDE_EVENODD=1
#options 	RF_INCLUDE_RAID5_RS=1
#options 	RF_INCLUDE_PARITYLOGGING=1
#options 	RF_INCLUDE_CHAINDECLUSTER=1
#options 	RF_INCLUDE_INTERDECLUSTER=1
#options 	RF_INCLUDE_PARITY_DECLUSTERING=1
#options 	RF_INCLUDE_PARITY_DECLUSTERING_DS=1
pseudo-device	md		1	# memory disk device (ramdisk)
pseudo-device	vnd		4	# disk-like interface to files

# network pseudo-devices
pseudo-device	bpfilter	8	# Berkeley packet filter
pseudo-device	ipfilter		# IP filter (firewall) and NAT
pseudo-device	loop			# network loopback
#pseudo-device	ppp		2	# Point-to-Point Protocol
#pseudo-device	pppoe			# PPP over Ethernet (RFC 2516)
#pseudo-device	sl		2	# Serial Line IP
#pseudo-device	strip		2	# Starmode Radio IP (Metricom)
#pseudo-device	irframetty		# IrDA frame line discipline
#pseudo-device	tun		2	# network tunneling over tty
#pseudo-device	gre		2	# generic L3 over IP tunnel
#pseudo-device	gif		4	# IPv[46] over IPv[46] tunnel (RFC1933)
#pseudo-device	faith		1	# IPv[46] tcp relay translation i/f
#pseudo-device	stf		1	# 6to4 IPv6 over IPv4 encapsulation
#pseudo-device	vlan			# IEEE 802.1q encapsulation
#pseudo-device	bridge			# simple inter-network bridging

# miscellaneous pseudo-devices
pseudo-device	pty			# pseudo-terminals
#pseudo-device	tb		1	# tablet line discipline
#pseudo-device	sequencer	1	# MIDI sequencer
# rnd works; RND_COM does not on port i386 yet.
pseudo-device	rnd			# /dev/random and in-kernel generator
#options 	RND_COM			# use "com" randomness as well (BROKEN)
pseudo-device	clockctl		# user control of clock subsystem

# a pseudo device needed for Coda	# also needs CODA (above)
#pseudo-device	vcoda		4	# coda minicache <-> venus comm.

# wscons pseudo-devices
pseudo-device	wsmux			# mouse & keyboard multiplexor
pseudo-device	wsfont
------------------------------

	And the dmesg is as follows:

NetBSD 1.6.1_RC1 (BLACKHOLE) #5: Sat Feb 15 18:46:52 MET 2003
    he@black-3.sth.netnod.se:/usr/src/sys/arch/i386/compile/BLACKHOLE
cpu0: AMD Duron (686-class), 807.25 MHz
cpu0: I-cache 64 KB 64b/line 2-way, D-cache 64 KB 64b/line 2-way
cpu0: L2 cache 64 KB 64b/line 16-way
cpu0: features 183f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR>
cpu0: features 183f9ff<PGE,MCA,CMOV,FGPAT,PSE36,MMX>
cpu0: features 183f9ff<FXSR>
total memory = 239 MB
avail memory = 219 MB
using 3092 buffers containing 12368 KB of memory
BIOS32 rev. 0 found at 0xf0e70
mainbus0 (root)
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: VIA Technologies VT8363 KT133 System Controller (rev. 0x81)
agp0 at pchb0: aperture at 0xf8000000, size 0x10000000
ppb0 at pci0 dev 1 function 0: VIA Technologies VT8363 KT133 PCI to AGP Bridge (rev. 0x00)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled
S3 product 0x8a26 (VGA display, revision 0x03) at pci1 dev 0 function 0 not configured
pcib0 at pci0 dev 7 function 0
pcib0: VIA Technologies VT82C686A (Apollo KX133) PCI-ISA Bridge (rev. 0x40)
pciide0 at pci0 dev 7 function 1: VIA Technologies VT82C686A (Apollo KX133) ATA100 controller
pciide0: bus-master DMA support present
pciide0: primary channel configured to compatibility mode
wd0 at pciide0 channel 0 drive 0: <IC35L020AVER07-0>
wd0: drive supports 16-sector PIO transfers, LBA addressing
wd0: 19623 MB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 40188960 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
pciide0: primary channel interrupting at irq 14
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5 (Ultra/100) (using DMA data transfers)
pciide0: secondary channel configured to compatibility mode
atapibus0 at pciide0 channel 1: 2 targets
cd0 at atapibus0 drive 0: <CD-540E, , 1.0A> type 5 cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33)
pciide0: secondary channel interrupting at irq 15
cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2 (Ultra/33) (using DMA data transfers)
uhci0 at pci0 dev 7 function 2: VIA Technologies VT83C572 USB Controller (rev. 0x16)
uhci0: interrupting at irq 5
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: VIA Technologie UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 7 function 3: VIA Technologies VT83C572 USB Controller (rev. 0x16)
uhci1: interrupting at irq 5
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: VIA Technologie UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
VIA Technologies VT82C686A SMBus Controller (miscellaneous bridge, revision 0x40) at pci0 dev 7 function 4 not configured
auvia0 at pci0 dev 7 function 5: VIA VT82C686A AC'97 Audio (rev 0x50)
auvia0: interrupting at irq 12
auvia0: ALG16 codec; headphone, 18 bit DAC, 18 bit ADC, Realtek 3D
audio0 at auvia0: full duplex, mmap, independent
fxp0 at pci0 dev 13 function 0: i82550 Ethernet, rev 12
fxp0: interrupting at irq 11
fxp0: Ethernet address 00:02:b3:a0:b6:33
inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 4
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp1 at pci0 dev 14 function 0: i82550 Ethernet, rev 12
fxp1: interrupting at irq 10
fxp1: Ethernet address 00:02:b3:89:3f:05
inphy1 at fxp1 phy 1: i82555 10/100 media interface, rev. 4
inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vga1 at pci0 dev 15 function 0: Middle Digital, Inc. Weasel Virtual VGA (rev. 0x47)
wsdisplay0 at vga1 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
puc0 at pci0 dev 15 function 1: Middle Digital, Inc. Weasel serial port (com)
com3 at puc0 port 0: interrupting at irq 12
com3: ns16550a, working fifo
weasel0 at pci0 dev 15 function 2: PCI-Weasel watchdog timer
weasel0: Weasel didn't respond to SHOW_CONFIG
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
lpt0 at isa0 port 0x378-0x37b irq 7
pcppi0 at isa0 port 0x61
sysbeep0 at pcppi0
isapnp0 at isa0 port 0x279: ISA Plug 'n Play device support
npx0 at isa0 port 0xf0-0xff: using exception 16
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
isapnp0: no ISA Plug 'n Play devices found
biomask e365 netmask ef65 ttymask efe7
uhub0: port error, restarting port 1
uhub0: port error, restarting port 2
uhub1: port error, restarting port 1
uhub1: port error, restarting port 2
boot device: wd0
root on wd0a dumps on wd0b
root file system type: ffs
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)

>How-To-Repeat:
	Subject a netbsd-1-6-running host to high levels of "random"
	short-lived TCP sessions, and watch the machine "randomly" lose.

>Fix:
	Sorry, don't know; hints for further debugging gratefully
	accepted.
>Release-Note:
>Audit-Trail:

From: Havard Eidnes <he@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: kern-bug-people@netbsd.org
Subject: Re: kern/20390: callwheel corruption under higher network loads
Date: Fri, 28 Feb 2003 15:41:42 +0100 (CET)

 Hi,

 here is a brief "formal" update on this case, after inserting
 booby-traps in the code in several places related to handling of
 callouts.

 The problem is apparently caused by what can happen when a callout
 function is invoked.  Just before that happens, priority level is
 lowered (with CALLWHEEL_UNLOCK() which uses splx()), which will allow
 other code to run before the callout function is run.  Sometimes, the
 code which is run in the meantime can cause the storage containing the
 callout structure whose callout is now waiting to be run to be released.
 The callout function can even get to reactivating itself (e.g. to cause
 repeated TCP retransmits), using the now-released storage.  Sometimes
 this storage is even reallocated, causing Bad Things to happen to the
 callout data structures when that container structure is initialized/
 zeroed before being taken into use.

 This has been observed to be happening with both syn-cache entries and
 TCP control blocks.

 I'm presently running with code which delays the release of the storage
 until the callout function is invoked, and the "delayed release" has
 been triggered a few times already (I added a couple of TCP statistics
 counters to monitor this):

 black-3: {46} netstat -s | egrep delay
                 5028963 ack-only packets (4889736 delayed)
 -->     9 delayed frees of tcpcb
 -->     0 double delayed frees of tcpcb
 -->             5 delayed pool_put()s
 black-3: {47} 

 The latter counter is for syn-cache entries.

 Since this is all with netbsd-1-6 code, I'll need to take a look at
 whether this also applies to NetBSD-current, but chances are that it
 does.

 Regards,

 - Havard
Responsible-Changed-From-To: kern-bug-people->he 
Responsible-Changed-By: perry 
Responsible-Changed-When: Sat Apr 5 11:31:51 PST 2003 
Responsible-Changed-Why:  
Havard appears to be actively debugging this. 

From: Havard Eidnes <he@netbsd.org>
To: gnats-bugs@gnats.netbsd.org
Cc:  
Subject: Re: kern/20390: callwheel corruption at higher network loads
Date: Thu, 17 Jul 2003 11:07:00 +0200 (CEST)

 ----Next_Part(Thu_Jul_17_11:07:00_2003_251)--
 Content-Type: Text/Plain; charset=iso-8859-1
 Content-Transfer-Encoding: quoted-printable

 Hi,

 here is the current set of fixes which I use with -current which works
 around this problem.

 A summary of the private feedback I have received so far is:

  o This is not MP-safe, in that a timer could fire between the test
    and the action on the test.  This is true.  However, it is my
    understanding that in the present implementation of SMP in NetBSD,
    spl() provides the required mutual exclusion, also from interrupts
    on other CPUs, so if I've understood things correctly, this
    workaround does not *in the present state* of NetBSD introduce any
    *new* problems.

  o This is actually a generic problem related to callouts, and the fix
    is a special-case fix for the netinet code.

    Some allusions were made to the fact that this problem might
    actually be hitting us in other parts of the code, but no concrete
    references were given.

    However, I have tried to take a brief look at the other uses of the
    callout code in NetBSD, and it does not appear that the other uses
    are as eager to use dynamic storage allocation of structures
    containing callout structures as the netinet code, and this
    workaround does provably improve the robustness of NetBSD.

  o One comment indicated that a "RUNNING" flag for the callout code
    would be preferable to the "INVOKING" flag, and I understood that
    the RUNNING flag should be kept active as long as possible to cover
    the entire time the callout function is running.  This poses
    problems because some of the timer functions will want to call
    tcp_close(), and it tests the flag to decide whether to release the
    storage or "mark as dead", and with the flag still set, the storage
    would never be released.  Also, several of the timer functions have
    several exits, making it necessary to sprinkle callout_ack() calls
    in several more places.

    As long as the spl() mechanism provides mutual exclusion, the
    critical section which needs protection is the span between the
    lowering of the priority in the callout code and the raising of the
    priority in the spl() calls in the callout handlers, so here I have
    decided to keep the "INVOKING" flag and associated naming.

  o The callout code could be changed so that callout handlers have to
    be registered with a priority level, and instead of dropping all
    priority before invoking the callout handler, the callout code
    could drop priority directly to the desired priority level, thereby
    avoiding this problem altogether.

    However, this will possibly require MD changes to the priority
    handling code, and this would be a project exceeding my current
    abilities, and noone else has taken up this ball.

  o The callout code and API could be redesigned so as to avoid this
    problem; references were made to the Solaris timeout code.

    Again, a full redesign of the callout code exceeds my abilities,
    and noone else has taken up the ball and run with it in this
    direction.

 Attached below is the current version of the patch I'm using with
 NetBSD-current.

 BTW, this problem is also present on the netbsd-1-6 branch, and some
 of the suggestions above would IMHO be inappropriate to pull up to the
 netbsd-1-6 branch.  A large motivation on my part with this workaround
 is to get it in place on the netbsd-1-6 branch, in other words get it
 out there as soon as possible to improve the stability of our
 "released code".

 Regards,

 - H=E5vard

 ----Next_Part(Thu_Jul_17_11:07:00_2003_251)--
 Content-Type: Text/Plain; charset=us-ascii
 Content-Transfer-Encoding: 7bit

 Index: sys/sys/callout.h
 ===================================================================
 RCS file: /usr/users/he/nbcvs/netbsd/src/sys/sys/callout.h,v
 retrieving revision 1.17
 diff -u -r1.17 callout.h
 --- sys/sys/callout.h	4 Feb 2003 01:21:06 -0000	1.17
 +++ sys/sys/callout.h	17 Jul 2003 07:42:27 -0000
 @@ -83,6 +83,7 @@

  #define	CALLOUT_PENDING		0x0002	/* callout is on the queue */
  #define	CALLOUT_FIRED		0x0004	/* callout has fired */
 +#define	CALLOUT_INVOKING	0x0008	/* callout function is being invoked */

  #define	CALLOUT_INITIALIZER_SETFUNC(func, arg)				\
  				{ { NULL, NULL }, func, arg, 0, 0 }
 @@ -100,6 +101,8 @@

  #define	callout_pending(c)	((c)->c_flags & CALLOUT_PENDING)
  #define	callout_expired(c)	((c)->c_flags & CALLOUT_FIRED)
 +#define	callout_invoking(c)	((c)->c_flags & CALLOUT_INVOKING)
 +#define	callout_ack(c)		((c)->c_flags &= ~CALLOUT_INVOKING)
  #endif /* _KERNEL */

  #endif /* !_SYS_CALLOUT_H_ */
 Index: sys/kern/kern_timeout.c
 ===================================================================
 RCS file: /usr/users/he/nbcvs/netbsd/src/sys/kern/kern_timeout.c,v
 retrieving revision 1.7
 diff -u -r1.7 kern_timeout.c
 --- sys/kern/kern_timeout.c	14 Jul 2003 14:59:01 -0000	1.7
 +++ sys/kern/kern_timeout.c	17 Jul 2003 07:43:20 -0000
 @@ -258,7 +258,7 @@
  	/* Initialize the time here, it won't change. */
  	old_time = c->c_time;
  	c->c_time = to_ticks + hardclock_ticks;
 -	c->c_flags &= ~CALLOUT_FIRED;
 +	c->c_flags &= ~(CALLOUT_FIRED|CALLOUT_INVOKING);

  	c->c_func = func;
  	c->c_arg = arg;
 @@ -299,7 +299,7 @@
  	/* Initialize the time here, it won't change. */
  	old_time = c->c_time;
  	c->c_time = to_ticks + hardclock_ticks;
 -	c->c_flags &= ~CALLOUT_FIRED;
 +	c->c_flags &= ~(CALLOUT_FIRED|CALLOUT_INVOKING);

  	/*
  	 * If this timeout is already scheduled and now is moved
 @@ -334,7 +334,7 @@
  	if (callout_pending(c))
  		CIRCQ_REMOVE(&c->c_list);

 -	c->c_flags &= ~(CALLOUT_PENDING|CALLOUT_FIRED);
 +	c->c_flags &= ~(CALLOUT_PENDING|CALLOUT_FIRED|CALLOUT_INVOKING);

  	CALLOUT_UNLOCK(s);
  }
 @@ -393,7 +393,7 @@
  				callout_ev_late.ev_count++;
  #endif
  			c->c_flags = (c->c_flags  & ~CALLOUT_PENDING) |
 -			    CALLOUT_FIRED;
 +			    (CALLOUT_FIRED|CALLOUT_INVOKING);

  			func = c->c_func;
  			arg = c->c_arg;
 Index: sys/netinet/tcp_input.c
 ===================================================================
 RCS file: /usr/users/he/nbcvs/netbsd/src/sys/netinet/tcp_input.c,v
 retrieving revision 1.172
 diff -u -r1.172 tcp_input.c
 --- sys/netinet/tcp_input.c	2 Jul 2003 19:33:20 -0000	1.172
 +++ sys/netinet/tcp_input.c	17 Jul 2003 07:45:37 -0000
 @@ -2799,7 +2799,10 @@
  		(void) m_free((sc)->sc_ipopts);				\
  	if ((sc)->sc_route4.ro_rt != NULL)				\
  		RTFREE((sc)->sc_route4.ro_rt);				\
 -	pool_put(&syn_cache_pool, (sc));				\
 +	if (callout_invoking(&(sc)->sc_timer))				\
 +		(sc)->sc_flags |= SCF_DEAD;				\
 +	else								\
 +		pool_put(&syn_cache_pool, (sc));			\
  } while (/*CONSTCOND*/0)

  struct pool syn_cache_pool;
 @@ -2946,6 +2949,14 @@
  	int s;

  	s = splsoftnet();
 +	callout_ack(&sc->sc_timer);
 +
 +	if (__predict_false(sc->sc_flags & SCF_DEAD)) {
 +		tcpstat.tcps_sc_delayed_free++;
 +		pool_put(&syn_cache_pool, sc);
 +		splx(s);
 +		return;
 +	}

  	if (__predict_false(sc->sc_rxtshift == TCP_MAXRXTSHIFT)) {
  		/* Drop it -- too many retransmissions. */
 Index: sys/netinet/tcp_subr.c
 ===================================================================
 RCS file: /usr/users/he/nbcvs/netbsd/src/sys/netinet/tcp_subr.c,v
 retrieving revision 1.143
 diff -u -r1.143 tcp_subr.c
 --- sys/netinet/tcp_subr.c	3 Jul 2003 08:28:16 -0000	1.143
 +++ sys/netinet/tcp_subr.c	17 Jul 2003 07:48:14 -0000
 @@ -1031,6 +1031,32 @@
  }

  /*
 + * Return whether this tcpcb is marked as dead, indicating
 + * to the calling timer function that no further action should
 + * be taken, as we are about to release this tcpcb.  The release
 + * of the storage will be done if this is the last timer running.
 + *
 + * This is typically called from the callout handler function before
 + * callout_ack() is done, therefore we need to test the number of
 + * running timer functions against 1 below, not 0.
 + */
 +int
 +tcp_isdead(tp)
 +	struct tcpcb *tp;
 +{
 +	int dead = (tp->t_flags & TF_DEAD);
 +
 +	if (__predict_false(dead)) {
 +		if (tcp_timers_invoking(tp) > 1)
 +				/* not quite there yet -- count separately? */
 +			return dead;
 +		tcpstat.tcps_delayed_free++;
 +		pool_put(&tcpcb_pool, tp);
 +	}
 +	return dead;
 +}
 +
 +/*
   * Close a TCP control block:
   *	discard all space held by the tcp
   *	discard internet protocol block
 @@ -1148,7 +1174,11 @@
  		m_free(tp->t_template);
  		tp->t_template = NULL;
  	}
 -	pool_put(&tcpcb_pool, tp);
 +	if (tcp_timers_invoking(tp))
 +		tp->t_flags |= TF_DEAD;
 +	else
 +		pool_put(&tcpcb_pool, tp);
 +
  	if (inp) {
  		inp->inp_ppcb = 0;
  		soisdisconnected(so);
 Index: sys/netinet/tcp_timer.c
 ===================================================================
 RCS file: /usr/users/he/nbcvs/netbsd/src/sys/netinet/tcp_timer.c,v
 retrieving revision 1.62
 diff -u -r1.62 tcp_timer.c
 --- sys/netinet/tcp_timer.c	3 Feb 2003 23:51:04 -0000	1.62
 +++ sys/netinet/tcp_timer.c	17 Jul 2003 08:22:41 -0000
 @@ -196,6 +196,24 @@
  }

  /*
 + * Return how many timers are currently being invoked.
 + */
 +int
 +tcp_timers_invoking(struct tcpcb *tp)
 +{
 +	int i;
 +	int count = 0;
 +
 +	for (i = 0; i < TCPT_NTIMERS; i++)
 +		if (callout_invoking(&tp->t_timer[i]))
 +			count++;
 +	if (callout_invoking(&tp->t_delack_ch))
 +		count++;
 +
 +	return count;
 +}
 +
 +/*
   * Callout to process delayed ACKs for a TCPCB.
   */
  void
 @@ -211,6 +229,12 @@
  	 */

  	s = splsoftnet();
 +	callout_ack(&tp->t_delack_ch);
 +	if (tcp_isdead(tp)) {
 +		splx(s);
 +		return;
 +	}
 +
  	tp->t_flags |= TF_ACKNOW;
  	(void) tcp_output(tp);
  	splx(s);
 @@ -267,6 +291,11 @@
  #endif

  	s = splsoftnet();
 +	callout_ack(&tp->t_timer[TCPT_KEEP]);
 +	if (tcp_isdead(tp)) {
 +		splx(s);
 +		return;
 +	}

  #ifdef TCP_DEBUG
  #ifdef INET
 @@ -414,6 +443,11 @@
  #endif

  	s = splsoftnet();
 +	callout_ack(&tp->t_timer[TCPT_PERSIST]);
 +	if (tcp_isdead(tp)) {
 +		splx(s);
 +		return;
 +	}

  #ifdef TCP_DEBUG
  #ifdef INET
 @@ -476,6 +510,11 @@
  #endif

  	s = splsoftnet();
 +	callout_ack(&tp->t_timer[TCPT_KEEP]);
 +	if (tcp_isdead(tp)) {
 +		splx(s);
 +		return;
 +	}

  #ifdef TCP_DEBUG
  	ostate = tp->t_state;
 @@ -558,6 +597,11 @@
  #endif

  	s = splsoftnet();
 +	callout_ack(&tp->t_timer[TCPT_2MSL]);
 +	if (tcp_isdead(tp)) {
 +		splx(s);
 +		return;
 +	}

  #ifdef TCP_DEBUG
  #ifdef INET
 Index: sys/netinet/tcp_var.h
 ===================================================================
 RCS file: /usr/users/he/nbcvs/netbsd/src/sys/netinet/tcp_var.h,v
 retrieving revision 1.102
 diff -u -r1.102 tcp_var.h
 --- sys/netinet/tcp_var.h	29 Jun 2003 22:32:00 -0000	1.102
 +++ sys/netinet/tcp_var.h	17 Jul 2003 07:51:57 -0000
 @@ -185,6 +185,7 @@
  #define	TF_CANT_TXSACK	0x1000		/* other side said I could not SACK */
  #define	TF_IGNR_RXSACK	0x2000		/* ignore received SACK blocks */
  #define	TF_REASSEMBLING	0x4000		/* we're busy reassembling */
 +#define	TF_DEAD		0x8000		/* dead and to-be-released */


  	struct	mbuf *t_template;	/* skeletal packet for transmit */
 @@ -417,6 +418,7 @@

  #define	SCF_UNREACH		0x0001		/* we've had an unreach error */
  #define	SCF_TIMESTAMP		0x0002		/* peer will do timestamps */
 +#define	SCF_DEAD		0x0004		/* this entry to be released */

  	struct mbuf *sc_ipopts;			/* IP options */
  	u_int16_t sc_peermaxseg;
 @@ -547,6 +549,7 @@
  	u_quad_t tcps_noport;		/* no socket on port */
  	u_quad_t tcps_badsyn;		/* received ack for which we have
  					   no SYN in compressed state */
 +	u_quad_t tcps_delayed_free;	/* delayed pool_put() of tcpcb */

  	/* These statistics deal with the SYN cache. */
  	u_quad_t tcps_sc_added;		/* # of entries added */
 @@ -561,6 +564,7 @@
  	u_quad_t tcps_sc_dropped;	/* # of SYNs dropped (no route/mem) */
  	u_quad_t tcps_sc_collisions;	/* # of hash collisions */
  	u_quad_t tcps_sc_retransmitted;	/* # of retransmissions */
 +	u_quad_t tcps_sc_delayed_free;	/* # of delayed pool_put()s */

  	u_quad_t tcps_selfquench;	/* # of ENOBUFS we get on output */
  };
 @@ -706,8 +710,10 @@

  int	 tcp_attach __P((struct socket *));
  void	 tcp_canceltimers __P((struct tcpcb *));
 +int	 tcp_timers_invoking __P((struct tcpcb*));
  struct tcpcb *
  	 tcp_close __P((struct tcpcb *));
 +int	 tcp_isdead __P((struct tcpcb *));
  #ifdef INET6
  void	 tcp6_ctlinput __P((int, struct sockaddr *, void *));
  #endif
 Index: usr.bin/netstat/inet.c
 ===================================================================
 RCS file: /usr/users/he/nbcvs/netbsd/src/usr.bin/netstat/inet.c,v
 retrieving revision 1.56
 diff -u -r1.56 inet.c
 --- usr.bin/netstat/inet.c	12 Jul 2003 13:39:23 -0000	1.56
 +++ usr.bin/netstat/inet.c	13 Jul 2003 15:15:24 -0000
 @@ -263,6 +263,7 @@
  	p2(tcps_closed, tcps_drops,
  		"\t%llu connection%s closed (including %llu drop%s)\n");
  	p(tcps_conndrops, "\t%llu embryonic connection%s dropped\n");
 +	p(tcps_delayed_free, "\t%llu delayed free%s of tcpcb\n");
  	p2(tcps_rttupdated, tcps_segstimed,
  		"\t%llu segment%s updated rtt (of %llu attempt%s)\n");
  	p(tcps_rexmttimeo, "\t%llu retransmit timeout%s\n");
 @@ -292,6 +293,8 @@
  	ps(tcps_sc_bucketoverflow, "\t\t%llu dropped due to bucket overflow\n");
  	ps(tcps_sc_reset, "\t\t%llu dropped due to RST\n");
  	ps(tcps_sc_unreach, "\t\t%llu dropped due to ICMP unreachable\n");
 +	ps(tcps_sc_delayed_free, "\t\t%llu delayed free of SYN cache "
 +		"entries\n");
  	p(tcps_sc_retransmitted, "\t%llu SYN,ACK%s retransmitted\n");
  	p(tcps_sc_dupesyn, "\t%llu duplicate SYN%s received for entries "
  		"already in the cache\n");
 Index: share/man/man9/callout.9
 ===================================================================
 RCS file: /usr/users/he/nbcvs/netbsd/src/share/man/man9/callout.9,v
 retrieving revision 1.9
 diff -u -r1.9 callout.9
 --- share/man/man9/callout.9	16 Apr 2003 13:35:25 -0000	1.9
 +++ share/man/man9/callout.9	17 Jul 2003 07:54:41 -0000
 @@ -43,6 +43,9 @@
  .Nm callout_schedule ,
  .Nm callout_setfunc ,
  .Nm callout_stop ,
 +.Nm callout_expired ,
 +.Nm callout_invoking ,
 +.Nm callout_ack ,
  .Nm CALLOUT_INITIALIZER ,
  .Nm CALLOUT_INITIALIZER_SETFUNC
  .Nd execute a function after a specified length of time
 @@ -63,6 +66,10 @@
  .Fn "callout_pending" "struct callout *c"
  .Ft int
  .Fn "callout_expired" "struct callout *c"
 +.Ft int
 +.Fn "callout_invoking" "struct callout *c"
 +.Ft void
 +.Fn "callout_ack" "struct callout *c"
  .Fd CALLOUT_INITIALIZER
  .Pp
  .Fd CALLOUT_INITIALIZER_SETFUNC(func, arg)
 @@ -117,8 +124,10 @@
  Once the timer is started, the callout handle is marked as
  .Em PENDING .
  Once the timer expires,
 -the handle is marked at
 +the handle is marked as
  .Em EXPIRED
 +and
 +.Em RUNNING
  and the
  .Em PENDING
  status is cleared.
 @@ -153,11 +162,11 @@
  function stops the timer associated the callout handle
  .Fa c .
  The
 -.Em PENDING
 +.Em PENDING ,
 +.Em EXPIRED ,
 +and
 +.Em RUNNING
  status for the callout handle is cleared.
 -The
 -.Em EXPIRED
 -status is not affected.
  It is safe to call
  .Fn callout_stop
  on a callout handle that is not pending, so long as it is initialized.
 @@ -183,6 +192,42 @@
  .Fn callout_expired
  function tests to see if the callout's timer has expired and its
  function called.
 +.Pp
 +The
 +.Fn callout_invoking
 +function tests the
 +.Em INVOKING
 +status of the callout handle
 +.Fa c .
 +This flag is set just before a callout's function is being called.
 +Since the priority level is lowered prior to invocation of the
 +callout function, other pending higher-priority code may run before
 +the callout function is allowed to run.
 +This may create a race condition if this higher-priority code
 +deallocates storage containing one or more callout structures whose
 +callout functions are about to be run.
 +In such cases, one technique to prevent references to deallocated
 +storage would be to test whether any callout functions are in the
 +.Em INVOKING
 +state using
 +.Fn callout_invoking ,
 +and if so, to mark the data structure and defer storage
 +deallocation until the callout function is allowed to run.
 +For this handshake protocol to work, the callout function will
 +have to use the
 +.Fn callout_ack
 +function to clear this flag.
 +.Pp
 +The
 +.Fn callout_ack
 +function clears the
 +.Em INVOKING
 +state in the callout handle
 +.Fa c .
 +This is used in situations where it is necessary to protect against
 +the race condition described under
 +.Fn callout_invoking .
 +The
  .Sh SEE ALSO
  .Xr hz 9
  .Sh HISTORY

 ----Next_Part(Thu_Jul_17_11:07:00_2003_251)----

From: Havard Eidnes <he@netbsd.org>
To: gnats-bugs@gnats.netbsd.org
Cc:  
Subject: Re: kern/20390: callwheel corruption at higher network loads
Date: Thu, 17 Jul 2003 13:55:47 +0200 (CEST)

 Someone asked about a further explanation of this problem, and here is
 for posterity a copy of my reply:

 > Am I right in thinking that this whole problem is down to calling
 > callout_stop() in the window of:
 > 	CALLOUT_UNLOCK(s);
 > 	(*func)(arg);
 > 	CALLOUT_LOCK(s);
 > (Since one presumes that the calling code is remembering that it
 > has the timer pending.)

 The way I understand this, no.  The way I understand this what can
 happen in the context of the TCP code is as follows:

 1) a tcp session sets one (or several) timers for a given tcpcb
 2) one of these timers fires, and the callout code lowers priority
    prior to calling the callout handler with a pointer to the tcpcb as
    argument.
 3) while the callout code has been running at elevated priority (at
    splsched()), network traffic has arrived, and when the priority is
    lowered, network interrupts are processed, typically at splnet().
 4) the code at splnet() will schedule soft network interrupts to be
    processed, and they also get invoked, still before the callout
    handler is invoked.
 5) some of this "process incoming network data" may cause tcp_close() to
    be called.  This will in turn call tcp_canceltimers(), which does
    callout_stop().  This is not problematical; CALLOUT_PENDING has
    already been cleared in the c_flags field in the callout struct by
    softclock(), so callout_stop() will not try to remove it from the
    queue, but remember, the callout handler has not yet been invoked,
    but is still waiting for us further down on the stack!  tcp_close()
    will however, as the code currently stands, do a pool_put() of the
    tcp control block.  (You probably now realize where this is headed.)
 6) when the incoming network traffic has been processed, we return from
    (hard and soft) interrupt context, and the callout handler gets to be
    invoked with a pointer to the now-released tcpcb as argument.  The
    callout handler may decide on it's own to reschedule a timeout using
    the released tcpcb (I'm a little fuzzy on this point).  This will put
    the callout struct inside the now-released tcpcb back on the callout
    queues (this has been observed as happening by adding sanity checks).
 7) when the storage for the tcpcb, which contains the callout structs
    is reallocated, the contents is reinitialized, causing the queue
    pointers in the callout structs to be reinitialized, causing
    corruption of the callwheel data structure (this has also been
    observed as happening by adding sanity checks).

 The end result is as already described.

 > The whole business of callouts is definitely riddled with timing errors.
 > NetBSD can't use some of the solutions because it uses caller-supplied
 > data areas which it is useful to allow the timeout routine to invalidate.

 Yes; the caller-supplied data areas and the fact that the callout code
 relinquishes ownership of the callout struct at the moment the callout
 handler is invoked makes some otherwise conceivable simple solutions
 impossible.  This is, e.g. what triggered the need for the code which
 uses the callout code to cooperate by using callout_ack() in my patch.

 However, redoing the callout API to fix this design flaw is most
 probably going to be too big of a change for the netbsd-1-6 branch.

 > I know some systems make callout_stop() loop/sleep until the routine
 > finishes (for MP sanity).
 >
 > I think it might be possible to fix things with only minor changes:
 > 1) Make callout_stop() tell the user whether the callout was active
 >    - maybe return the old value of c_flags.
 > 2) Don't clear CALLOUT_FIRED in callout_stop().

 As I said above, the problem appears to be a bit more insidious than
 that, so I'm unconvinced that this will fix the problem.

 Regards,

 - Havard

From: Havard Eidnes <he@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:  
Subject: pr/20390 CVS commit: src
Date: Sun, 20 Jul 2003 16:25:59 +0000 (UTC)

 Module Name:	src
 Committed By:	he
 Date:		Sun Jul 20 16:25:59 UTC 2003

 Modified Files:
 	src/share/man/man9: callout.9
 	src/sys/kern: kern_timeout.c
 	src/sys/sys: callout.h

 Log Message:
 Temporarily introduce CALLOUT_INVOKING, callout_invoking() and callout_ack()
 to make users of the callout facility able to cooperate to work around the
 race caused by the callout code lowering interrupt priority level when
 invoking callout handlers, something which allows other code to run before
 the callout handler gets to it's spl*() call.

 This is to enable the workaround for the TCP code found in PR#20390 to be
 applied.

 This should be backed out once a more comprehensive fix can be put in
 place.


 To generate a diff of this commit:
 cvs rdiff -r1.9 -r1.10 src/share/man/man9/callout.9
 cvs rdiff -r1.7 -r1.8 src/sys/kern/kern_timeout.c
 cvs rdiff -r1.17 -r1.18 src/sys/sys/callout.h

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.


From: Havard Eidnes <he@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:  
Subject: pr/20390 CVS commit: src
Date: Sun, 20 Jul 2003 16:35:12 +0000 (UTC)

 Module Name:	src
 Committed By:	he
 Date:		Sun Jul 20 16:35:12 UTC 2003

 Modified Files:
 	src/sys/netinet: tcp_input.c tcp_subr.c tcp_timer.c tcp_var.h
 	src/usr.bin/netstat: inet.c

 Log Message:
 As a temporary workaround, apply the fix from PR#20390, thereby
 cooperating with the callout code in working around the race
 condition caused by the TCP code's use of the callout facility.

 Instead of unconditionally releasing memory in tcp_close() and
 SYN_CACHE_PUT(), check whether any of the related callout handlers
 are about to be invoked (but have not yet done callout_ack()), and
 if so, just mark the associated data structure (tcpcb or syn cache
 entry) as "dead", and test for this (and release storage) in the
 callout handler functions.


 To generate a diff of this commit:
 cvs rdiff -r1.172 -r1.173 src/sys/netinet/tcp_input.c
 cvs rdiff -r1.143 -r1.144 src/sys/netinet/tcp_subr.c
 cvs rdiff -r1.62 -r1.63 src/sys/netinet/tcp_timer.c
 cvs rdiff -r1.102 -r1.103 src/sys/netinet/tcp_var.h
 cvs rdiff -r1.56 -r1.57 src/usr.bin/netstat/inet.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.


From: James Chacon <jmc@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:  
Subject: pr/20390 CVS commit: [netbsd-1-6] src/sys/netinet
Date: Wed, 22 Oct 2003 06:05:47 +0000 (UTC)

 Module Name:	src
 Committed By:	jmc
 Date:		Wed Oct 22 06:05:47 UTC 2003

 Modified Files:
 	src/sys/netinet [netbsd-1-6]: tcp_input.c

 Log Message:
 Pullup rev 1.173 (requested by he in ticket #1530)

         Introduce a new INVOKING status for callouts, and use it to close
         a race condition in the TCP code.  Fixes PR#20390.


 To generate a diff of this commit:
 cvs rdiff -r1.141.4.6 -r1.141.4.7 src/sys/netinet/tcp_input.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.


From: James Chacon <jmc@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:  
Subject: pr/20390 CVS commit: [netbsd-1-6] src/sys/netinet
Date: Wed, 22 Oct 2003 06:06:05 +0000 (UTC)

 Module Name:	src
 Committed By:	jmc
 Date:		Wed Oct 22 06:06:05 UTC 2003

 Modified Files:
 	src/sys/netinet [netbsd-1-6]: tcp_timer.c

 Log Message:
 Pullup rev 1.63 (requested by he in ticket #1530)

         Introduce a new INVOKING status for callouts, and use it to close
         a race condition in the TCP code.  Fixes PR#20390.


 To generate a diff of this commit:
 cvs rdiff -r1.57.10.1 -r1.57.10.2 src/sys/netinet/tcp_timer.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.


From: James Chacon <jmc@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:  
Subject: pr/20390 CVS commit: [netbsd-1-6] src/sys/netinet
Date: Wed, 22 Oct 2003 06:05:57 +0000 (UTC)

 Module Name:	src
 Committed By:	jmc
 Date:		Wed Oct 22 06:05:57 UTC 2003

 Modified Files:
 	src/sys/netinet [netbsd-1-6]: tcp_subr.c

 Log Message:
 Pullup rev 1.144 (requested by he in ticket #1530)

         Introduce a new INVOKING status for callouts, and use it to close
         a race condition in the TCP code.  Fixes PR#20390.


 To generate a diff of this commit:
 cvs rdiff -r1.127.4.3 -r1.127.4.4 src/sys/netinet/tcp_subr.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.


From: James Chacon <jmc@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:  
Subject: pr/20390 CVS commit: [netbsd-1-6] src/sys/netinet
Date: Wed, 22 Oct 2003 06:06:13 +0000 (UTC)

 Module Name:	src
 Committed By:	jmc
 Date:		Wed Oct 22 06:06:13 UTC 2003

 Modified Files:
 	src/sys/netinet [netbsd-1-6]: tcp_var.h

 Log Message:
 Pullup rev 1.03 (requested by he in ticket #1530)

         Introduce a new INVOKING status for callouts, and use it to close
         a race condition in the TCP code.  Fixes PR#20390.


 To generate a diff of this commit:
 cvs rdiff -r1.90.4.1 -r1.90.4.2 src/sys/netinet/tcp_var.h

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.


From: James Chacon <jmc@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:  
Subject: pr/20390 CVS commit: [netbsd-1-6] src/usr.bin/netstat
Date: Wed, 22 Oct 2003 06:06:23 +0000 (UTC)

 Module Name:	src
 Committed By:	jmc
 Date:		Wed Oct 22 06:06:23 UTC 2003

 Modified Files:
 	src/usr.bin/netstat [netbsd-1-6]: inet.c

 Log Message:
 Pullup rev 1.57 (requested by he in ticket #1530)

         Introduce a new INVOKING status for callouts, and use it to close
         a race condition in the TCP code.  Fixes PR#20390.


 To generate a diff of this commit:
 cvs rdiff -r1.51.2.1 -r1.51.2.2 src/usr.bin/netstat/inet.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.


From: James Chacon <jmc@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:  
Subject: pr/20390 CVS commit: [netbsd-1-6] src
Date: Wed, 22 Oct 2003 06:06:50 +0000 (UTC)

 Module Name:	src
 Committed By:	jmc
 Date:		Wed Oct 22 06:06:50 UTC 2003

 Modified Files:
 	src/share/man/man9 [netbsd-1-6]: callout.9
 	src/sys/kern [netbsd-1-6]: kern_clock.c
 	src/sys/sys [netbsd-1-6]: callout.h

 Log Message:
 Pullup via patch  (requested by he in ticket #1530)

         Introduce a new INVOKING status for callouts, and use it to close
         a race condition in the TCP code.  Fixes PR#20390.


 To generate a diff of this commit:
 cvs rdiff -r1.6 -r1.6.2.1 src/share/man/man9/callout.9
 cvs rdiff -r1.79 -r1.79.6.1 src/sys/kern/kern_clock.c
 cvs rdiff -r1.16 -r1.16.12.1 src/sys/sys/callout.h

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->feedback 
State-Changed-By: mycroft 
State-Changed-When: Thu Jul 8 20:28:45 UTC 2004 
State-Changed-Why:  
Someone should update the status of this PR.  IF more work needs to be done, 
that should be mentioned in the PR. 
State-Changed-From-To: feedback->closed
State-Changed-By: elad@netbsd.org
State-Changed-When: Fri, 06 Oct 2006 22:28:55 +0000
State-Changed-Why:
fixed, feedback timeout


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.