NetBSD Problem Report #20390
Received: (qmail 16765 invoked by uid 605); 17 Feb 2003 15:55:04 -0000
Message-Id: <20030217155500.09281292C2@smistad.uninett.no>
Date: Mon, 17 Feb 2003 16:55:00 +0100 (CET)
From: he@uninett.no
Sender: gnats-bugs-owner@netbsd.org
Reply-To: he@uninett.no
To: gnats-bugs@gnats.netbsd.org
Subject: callwheel corruption under higher network loads
X-Send-Pr-Version: 3.95
>Number: 20390
>Category: kern
>Synopsis: callwheel corruption under higher network loads
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: he
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Mon Feb 17 15:56:00 +0000 2003
>Closed-Date: Fri Oct 06 22:28:55 +0000 2006
>Last-Modified: Fri Oct 06 22:28:55 +0000 2006
>Originator: Havard Eidnes
>Release: NetBSD 1.6.1_RC1, Feb 11 2003 vintage
>Organization:
UNINETT AS
>Environment:
System: NetBSD black-3.sth.netnod.se 1.6.1_RC1 NetBSD 1.6.1_RC1 (BLACKHOLE) #5: Sat Feb 15 18:46:52 MET 2003 he@black-3.sth.netnod.se:/usr/src/sys/arch/i386/compile/BLACKHOLE i386
Architecture: i386
Machine: i386
>Description:
I'm involved in debugging NetBSD on a system which sees a fair
amount of TCP-based DNS traffic; the network interface reports
about 1500-3000 interupts per second.
This system is plagued with problems where the TCP connection
table after a somewhat indeterminate time (sometimes days)
grows without bounds, ending with exhaustion of kernel
resources and typically a hang.
To be able to catch it in the act, I've upped the NMBCLUSTERS
limit:
options NMBCLUSTERS=4096
Recently I've noticed that when this happens, operations such
as running "tset" or "sleep 2" never terminate. I've seen
output from "netstat -f inet -n" of 37000 lines...
To enable further debugging, I've also enabled:
options QUEUEDEBUG
options CALLWHEEL_STATS
This has recently landed me a couple of panics in the
QUEUEDEBUG code, one of which is reported here. While
inspecting the callwheel on the running kernel, I found
several entries whose time had come and gone, and which were
lingering around (that's before I had CALLWHEEL_STATS turned
on), and a code inspection of sys/kern_clock.c reveals that
indeed, "late" events will never be processed. To fix that, I
added this patch:
------------------------------
Index: kern_clock.c
===================================================================
RCS file: /cvsroot/src/sys/kern/kern_clock.c,v
retrieving revision 1.79
diff -u -r1.79 kern_clock.c
--- kern_clock.c 2002/03/17 11:10:43 1.79
+++ kern_clock.c 2003/02/17 15:24:42
@@ -372,6 +372,7 @@
struct evcnt callwheel_softchecks; /* # checks per softclock() */
struct evcnt callwheel_softempty; /* # empty buckets seen */
struct evcnt callwheel_hintworked; /* # times hint saved scan */
+struct evcnt callwheel_toolate; /* # times time passed us by */
#endif /* CALLWHEEL_STATS */
/*
@@ -963,7 +964,7 @@
#ifdef CALLWHEEL_STATS
callwheel_softchecks.ev_count++;
#endif
- if (c->c_time != softclock_ticks) {
+ if (c->c_time > softclock_ticks) {
if (c->c_time < bucket->cq_hint)
bucket->cq_hint = c->c_time;
c = TAILQ_NEXT(c, c_link);
@@ -976,6 +977,10 @@
steps = 0;
}
} else {
+#ifdef CALLWHEEL_STATS
+ if (c->c_time < softclock_ticks)
+ callwheel_toolate.ev_count++;
+#endif
nextsoftcheck = TAILQ_NEXT(c, c_link);
TAILQ_REMOVE(&bucket->cq_q, c, c_link);
#ifdef CALLWHEEL_STATS
@@ -1055,6 +1060,8 @@
NULL, "callwheel", "softempty");
evcnt_attach_dynamic(&callwheel_hintworked, EVCNT_TYPE_MISC,
NULL, "callwheel", "hintworked");
+ evcnt_attach_dynamic(&callwheel_toolate, EVCNT_TYPE_MISC,
+ NULL, "callwheel", "toolate");
#endif /* CALLWHEEL_STATS */
}
@@ -1206,6 +1213,8 @@
(long long) callwheel_changed.ev_count);
printf("\tCallouts that fired: %llu\n",
(long long) callwheel_fired.ev_count);
+ printf("\tCallouts fired late: %llu\n",
+ (long long) callwheel_toolate.ev_count);
printf("\tNumber of buckets: %d\n", callwheelsize);
printf("\tNumber of hash collisions: %llu\n",
(long long) callwheel_collisions.ev_count);
------------------------------
However, so far, these events have not occurred before the
various panics I've seen.
Parts of the debug session for this latest panic follows
below; here we have apparent corruption of the callwheel,
where an entry appears to be removed (tqe_next == tqe_prev ==
0x1), ref. the QUEUEDEBUG_TAILQ_POSTREMOVE() macro, but which
is still sitting on the callwheel, and this triggers the panic
when we later come to remove the next element in the chain.
I'm not certain, but it does look as if something is
manipulating the data structure without proper locking(?).
The salient parts of the debug session (DDB and GDB, yes, I
have a kernel core dump) follows:
panic: TAILQ_* back 0xc08af8ec /sys/arch/i386/compile/BLACKHOLE/../../../../kern/kern_clock.c:1160
Stopped at cpu_Debugger+0x4: leave
db> show reg
ds 0x10
es 0x10
fs 0x10
gs 0x10
edi 0x100
esi 0xc025ebf0 copyright+0x4d0
ebp 0xd2ec0b0c
ebx 0xd2ec0b38
edx 0xc02612f3 fmt.192+0x82e
ecx 0x7620
eax 0x67
eip 0xc01f19fc cpu_Debugger+0x4
cs 0x8
eflags 0x202
esp 0xd2ec0b0c
ss 0x10
cpu_Debugger+0x4: leave
db>
db> trace
cpu_Debugger(c545c830,e,10,c08dea6e,0) at cpu_Debugger+0x4
panic(c025ebf0,c08af8ec,c025eb80,488,c0000000) at panic+0xad
callout_stop_locked(c08af8ec,10,4,c01b87b2) at callout_stop_locked+0x80
callout_stop(c08af8ec,1,c01c0310,c08af8e0,130afc0) at callout_stop+0x26
tcp_input(c08c0c00,14,6,1,c08c0c00) at tcp_input+0x22e6
ip_input(c08c0c00,c0224c11,d2ec0d60,0) at ip_input+0x5f2
ipintr(10,10,10,10,d2be5900) at ipintr+0x6b
Xsoftnet() at Xsoftnet+0x2c
--- interrupt ---
idle(d2be5900,2c,c0164c2c,d2be5900) at idle+0x1b
bpendtsleep(c02c80cc,118,c0261500,2c,0) at bpendtsleep
sys_select(d2be5900,d2ec0f80,d2ec0f78,c0799038) at sys_select+0x300
syscall_plain(1f,1f,1f,1f,bfbfdb18) at syscall_plain+0xa7
db>
db> x 0xc08af8ec ,4
0xc08af8ec: c0968244 c09604b0 52a983 0
db>
db> x 0xc09604b0 ,4
0xc09604b0: 1 1 52a984 0
db>
db> x/x softclock_ticks ,2
softclock_ticks: 52a736 0
db> x/x hardclock_ticks ,2
hardclock_ticks: 52a736 0
db>
...
(gdb) up
#10 0xc0158b54 in callout_stop_locked (c=0xc08af8ec)
at /sys/arch/i386/compile/BLACKHOLE/../../../../kern/kern_clock.c:1160
1160 TAILQ_REMOVE(&bucket->cq_q, c, c_link);
(gdb) l
1155
1156 if (nextsoftcheck == c)
1157 nextsoftcheck = TAILQ_NEXT(c, c_link);
1158
1159 bucket = &callwheel[c->c_time & callwheelmask];
1160 TAILQ_REMOVE(&bucket->cq_q, c, c_link);
1161 if (TAILQ_EMPTY(&bucket->cq_q))
1162 bucket->cq_hint = UQUAD_MAX;
1163 #ifdef CALLWHEEL_STATS
1164 callwheel_count.ev_count--;
(gdb) p c
$1 = (struct callout *) 0x0
(bah, optimizer?!?)
(gdb) p 0xc08af8ec
$2 = 3230333164
(gdb) p (struct callout *)$
$3 = (struct callout *) 0xc08af8ec
(gdb) p *$
$4 = {c_link = {tqe_next = 0xc0968244, tqe_prev = 0xc09604b0},
c_time = 5417347, c_arg = 0xc08af8e0, c_func = 0xc01c0058 <tcp_timer_rexmt>,
c_flags = 0}
(gdb) p $.c_link.tqe_prev
$5 = (struct callout **) 0xc09604b0
(gdb) p *$
$6 = (struct callout *) 0x1
(gdb)
(gdb) p callwheel_count
$7 = {ev_count = 1049, ev_list = {tqe_next = 0xc02c59c0,
tqe_prev = 0xc02c5a48}, ev_type = 0 '\000', ev_grouplen = 9 '\t',
ev_namelen = 5 '\005', ev_pad1 = 0 '\000', ev_parent = 0x0,
ev_group = 0xc025ec11 "callwheel", ev_name = 0xc025ec25 "count"}
(gdb)
... created gdb script:
define cwdump
set $i = 0
while ($i < callwheelsize)
printf "Callwheel head %d\n", $i
cwchain callwheel[$i].cq_q.tqh_first
set $i++
end
end
define cwchain
set $co = (struct callout*)$arg0
while ($co)
print *$co
set $co = $co->c_link.tqe_next
end
end
...
(gdb) cwdump
...
Callwheel head 387
$364 = {c_link = {tqe_next = 0x1, tqe_prev = 0x1}, c_time = 5417348,
c_arg = 0xc09604a4, c_func = 0, c_flags = 0}
can not access 0x1, invalid translation (invalid PDE)
can not access 0x1, invalid translation (invalid PDE)
Cannot access memory at address 0x1
(gdb) p $364
$365 = {c_link = {tqe_next = 0x1, tqe_prev = 0x1}, c_time = 5417348,
c_arg = 0xc09604a4, c_func = 0, c_flags = 0}
(gdb)
(gdb) p &$
$366 = (struct callout *) 0xc09604b0
(gdb)
(gdb) p softclock_ticks
$367 = 5416758
(gdb)
(gdb) p *(struct callout*)0xc08af8ec
$369 = {c_link = {tqe_next = 0xc0968244, tqe_prev = 0xc09604b0},
c_time = 5417347, c_arg = 0xc08af8e0, c_func = 0xc01c0058 <tcp_timer_rexmt>,
c_flags = 0}
(gdb) p *$.c_link.tqe_next
$370 = {c_link = {tqe_next = 0x0, tqe_prev = 0xc08af8ec}, c_time = 5417347,
c_arg = 0xc0968238, c_func = 0xc01c0058 <tcp_timer_rexmt>, c_flags = 3}
(gdb)
BTW, the activity levels for the callwheel are presently as
follows:
% vmstat -v -e
event total rate type
callwheel collisions 17059969 2391 misc
callwheel maxlength 42 0 misc
callwheel count 916 0 misc
callwheel established 20032006 2808 misc
callwheel fired 291793 40 misc
callwheel disestablished 19739297 2767 misc
callwheel changed 5629534 789 misc
callwheel softclocks 489552 68 misc
callwheel softempty 2 0 misc
callwheel hintworked 279 0 misc
callwheel toolate 0 0 misc
The kernel configuration is as follows:
------------------------------
# $NetBSD: GENERIC,v 1.491.4.3 2002/10/24 09:34:22 lukem Exp $
#
# GENERIC machine description file
#
# This machine description file is used to generate the default NetBSD
# kernel. The generic kernel does not include all options, subsystems
# and device drivers, but should be useful for most applications.
#
# The machine description file can be customised for your specific
# machine to reduce the kernel size and improve its performance.
#
# For further information on compiling NetBSD kernels, see the config(8)
# man page.
#
# For further information on hardware support for this architecture, see
# the intro(4) man page. For further information about kernel options
# for this architecture, see the options(4) man page. For an explanation
# of each device driver in this file see the section 4 man page for the
# device.
include "arch/i386/conf/std.i386"
options INCLUDE_CONFIG_FILE # embed config file in kernel binary
#ident "GENERIC-$Revision: 1.491.4.3 $"
maxusers 32 # estimated number of users
#-----
# Increase mbuf clusters above GATEWAY=2048 setting
options NMBCLUSTERS=4096
options QUEUEDEBUG
options CALLWHEEL_STATS
#-----
# CPU support. At least one is REQUIRED.
#options I386_CPU
#options I486_CPU
options I586_CPU
options I686_CPU
# CPU-related options.
#options MATH_EMULATE # floating point emulation
#options VM86 # virtual 8086 emulation
#options USER_LDT # user-settable LDT; used by WINE
# eliminate delay no-ops in I/O; recommended on all but very old machines
#options DUMMY_NOPS
#options MTRR # memory-type range register syscall support
options PERFCTRS # performance-monitoring counters support
# delay between "rebooting ..." message and hardware reset, in milliseconds
#options CPURESET_DELAY=2000
# This option allows you to force a serial console at the specified
# I/O address. see console(4) for details.
#options CONSDEVNAME="\"com\"",CONADDR=0x2f8,CONSPEED=57600
# you don't want the option below ON iff you are using the
# serial console option of the new boot strap code.
#options CONS_OVERRIDE # Always use above! independent of boot info
# The following options override the memory sizes passed in from the boot
# block. Use them *only* if the boot block is unable to determine the correct
# values. Note that the BIOS may *correctly* report less than 640k of base
# memory if the extended BIOS data area is located at the top of base memory
# (as is the case on most recent systems).
#options REALBASEMEM=639 # size of base memory (in KB)
#options REALEXTMEM=15360 # size of extended memory (in KB)
# Standard system options
#options UCONSOLE # users can use TIOCCONS (for xconsole)
#options INSECURE # disable kernel security levels
options RTC_OFFSET=0 # hardware clock is this many mins. west of GMT
options NTP # NTP phase/frequency locked loop
options KTRACE # system call tracing via ktrace(1)
options SYSVMSG # System V-like message queues
options SYSVSEM # System V-like semaphores
#options SEMMNI=10 # number of semaphore identifiers
#options SEMMNS=60 # number of semaphores in system
#options SEMUME=10 # max number of undo entries per process
#options SEMMNU=30 # number of undo structures in system
options SYSVSHM # System V-like memory sharing
#options SHMMAXPGS=1024 # 1024 pages is the default
#options LKM # loadable kernel modules
#options USERCONF # userconf(4) support
#options PIPE_SOCKETPAIR # smaller, but slower pipe(2)
# Diagnostic/debugging support options
#options DIAGNOSTIC # expensive kernel consistency checks
#options DEBUG # expensive debugging checks/support
options KMEMSTATS # kernel memory statistics (vmstat -m)
options DDB # in-kernel debugger
options DDB_HISTORY_SIZE=512 # enable history editing in DDB
#options KGDB # remote debugger
#options KGDB_DEVNAME="\"com\"",KGDB_DEVADDR=0x3f8,KGDB_DEVRATE=9600
makeoptions DEBUG="-g" # compile full symbol table
# Compatibility options
#options COMPAT_NOMID # compatibility with 386BSD, BSDI, NetBSD 0.8,
#options COMPAT_09 # NetBSD 0.9,
#options COMPAT_10 # NetBSD 1.0,
#options COMPAT_11 # NetBSD 1.1,
#options COMPAT_12 # NetBSD 1.2,
#options COMPAT_13 # NetBSD 1.3,
#options COMPAT_14 # NetBSD 1.4,
options COMPAT_15 # NetBSD 1.5,
options COMPAT_43 # and 4.3BSD
options COMPAT_386BSD_MBRPART # recognize old partition ID
#options TCP_COMPAT_42 # 4.2BSD TCP/IP bug compat. Not recommended.
#options COMPAT_SVR4 # binary compatibility with SVR4
#options COMPAT_IBCS2 # binary compatibility with SCO and ISC
#options COMPAT_LINUX # binary compatibility with Linux
#options COMPAT_FREEBSD # binary compatibility with FreeBSD
#options COMPAT_MACH # binary compatibility with Mach binaries
#options EXEC_MACHO # exec MACH-O binaries
#options COMPAT_PECOFF # kernel support to run Win32 apps
# File systems
file-system FFS # UFS
#file-system EXT2FS # second extended file system (linux)
#file-system LFS # log-structured file system
file-system MFS # memory file system
#file-system NFS # Network File System client
#file-system NTFS # Windows/NT file system (experimental)
file-system CD9660 # ISO 9660 + Rock Ridge file system
file-system MSDOSFS # MS-DOS file system
#file-system FDESC # /dev/fd
file-system KERNFS # /kern
#file-system NULLFS # loopback file system
#file-system OVERLAY # overlay file system
#file-system PORTAL # portal filesystem (still experimental)
#file-system PROCFS # /proc
#file-system UMAPFS # NULLFS + uid and gid remapping
#file-system UNION # union file system
#file-system CODA # Coda File System; also needs vcoda (below)
# File system options
#options QUOTA # UFS quotas
#options FFS_EI # FFS Endian Independent support
options SOFTDEP # FFS soft updates support.
#options NFSSERVER # Network File System server
#options EXT2FS_SYSTEM_FLAGS # makes ext2fs file flags (append and
# immutable) behave as system flags.
# Networking options
#options GATEWAY # packet forwarding
options INET # IP + ICMP + TCP + UDP
options INET6 # IPV6
#options IPSEC # IP security
#options IPSEC_ESP # IP security (encryption part; define w/IPSEC)
#options IPSEC_DEBUG # debug for IP security
#options MROUTING # IP multicast routing
#options NS # XNS
#options NSIP # XNS tunneling over IP
#options ISO,TPIP # OSI
#options EON # OSI tunneling over IP
#options CCITT,LLC,HDLC # X.25
#options NETATALK # AppleTalk networking protocols
#options PPP_BSDCOMP # BSD-Compress compression support for PPP
#options PPP_DEFLATE # Deflate compression support for PPP
#options PPP_FILTER # Active filter support for PPP (requires bpf)
#options PFIL_HOOKS # pfil(9) packet filter hooks
#options IPFILTER_LOG # ipmon(8) log support
#options IPFILTER_DEFAULT_BLOCK # block all packets by default
#options TCP_DEBUG # Record last TCP_NDEBUG packets with SO_DEBUG
# These options enable verbose messages for several subsystems.
# Warning, these may compile large string tables into the kernel!
#options EISAVERBOSE # verbose EISA device autoconfig messages
options MIIVERBOSE # verbose PHY autoconfig messages
options PCIVERBOSE # verbose PCI device autoconfig messages
#options PCI_CONFIG_DUMP # verbosely dump PCI config space
#options PCMCIAVERBOSE # verbose PCMCIA configuration messages
options SCSIVERBOSE # human readable SCSI error messages
options USBVERBOSE # verbose USB device autoconfig messages
#options PNPBIOSVERBOSE # verbose PnP BIOS messages
#options PNPBIOSDEBUG # more fulsome PnP BIOS debugging messages
#options I2OVERBOSE # verbose I2O driver messages
#options MCAVERBOSE # verbose MCA device autoconfig messages
#options NFS_BOOT_DHCP,NFS_BOOT_BOOTPARAM
#
# wscons options
#
# builtin terminal emulations
#options WSEMUL_SUN # sun terminal emulation
options WSEMUL_VT100 # VT100 / VT220 emulation
# different kernel output - see dev/wscons/wsdisplayvar.h
options WS_KERNEL_FG=WSCOL_GREEN
#options WS_KERNEL_BG=WSCOL_BLACK
# compatibility to other console drivers
options WSDISPLAY_COMPAT_PCVT # emulate some ioctls
options WSDISPLAY_COMPAT_SYSCONS # emulate some ioctls
options WSDISPLAY_COMPAT_USL # VT handling
options WSDISPLAY_COMPAT_RAWKBD # can get raw scancodes
# see dev/pckbc/wskbdmap_mfii.c for implemented layouts
#options PCKBD_LAYOUT="(KB_DE | KB_NODEAD)"
# allocate a number of virtual screens at autoconfiguration time
#options WSDISPLAY_DEFAULTSCREENS=4
# use a large software cursor that doesn't blink
options PCDISPLAY_SOFTCURSOR
# modify the screen type of the console; defaults to "80x25"
#options VGA_CONSOLE_SCREENTYPE="\"80x24\""
# Kernel root file system and dump configuration.
config netbsd root on ? type ?
#config netbsd root on sd0a type ffs
#config netbsd root on ? type nfs
#
# Device configuration
#
mainbus0 at root
#apm0 at mainbus0 # Advanced power management
# Tuning for power management, see apm(4) for more details.
#options APM_NO_IDLE # Don't call BIOS CPU idle function
#options APM_V10_ONLY # Use only the APM 1.0 calls
#options APM_NO_POWEROFF # Don't power off on halt(8)
#options APM_POWER_PRINT # Print stats on the console
#options APM_DISABLE_INTERRUPTS=0 # Don't disable interrupts
# Basic Bus Support
# Plug-and-Play BIOS and attached devices
#pnpbios* at mainbus?
# mainboard audio chips
#ess* at pnpbios? index ? # ESS AudioDrive
#sb* at pnpbios? index ? # NeoMagic 256AV in sb mode
#wss* at pnpbios? index ? # NeoMagic 256AV in wss mode
#ym* at pnpbios? index ? # OPL3-SA3
# com port
# If enabled, consider changing "com0", "com1", and "com2" under "ISA Serial
# Interfaces" to "com*", otherwise com2 will attach at pnpbios? and there
# will be no com0. A side effect is pcmcia (and other) com? previously
# starting at com3 may attach as com1 or com2.
#com* at pnpbios? index ? # serial ports
# parallel port
# The above "com*" comments apply, cf. "lpt0" under "ISA parallel
# "printer interfaces".
#lpt* at pnpbios? index ? # parallel ports
#pckbc* at pnpbios? index ? # PC keyboard/mouse controller
#fdc* at pnpbios? index ? # floppy controller
# IDE controller on Toshiba Portege 3000 series (crippled PCI device)
#pciide* at pnpbios? index ?
# PCI bus support
pci* at mainbus? bus ?
pci* at pchb? bus ?
pci* at ppb? bus ?
# Configure PCI using BIOS information
#options PCIBIOS # PCI BIOS support
#options PCIBIOSVERBOSE # PCI BIOS verbose info
#options PCIBIOS_ADDR_FIXUP # fixup PCI I/O addresses
#options PCIBIOS_BUS_FIXUP # fixup PCI bus numbering
#options PCIBIOS_INTR_FIXUP # fixup PCI interrupt routing
#options PCIBIOS_IRQS_HINT=0x0a00 # PCI interrupts hint. IRQ 9 or 11
#options PCIBIOS_INTR_GUESS # see pcibios(4)
#options PCIINTR_DEBUG # super-verbose PCI interrupt fixup
# PCI bridges
pchb* at pci? dev ? function ? # PCI-Host bridges
#pceb* at pci? dev ? function ? # PCI-EISA bridges
pcib* at pci? dev ? function ? # PCI-ISA bridges
#pcmb* at pci? dev ? function ? # PCI-MCA bridges
ppb* at pci? dev ? function ? # PCI-PCI bridges
# XXX 'puc's aren't really bridges, but there's no better place for them here
puc* at pci? dev ? function ? # PCI "universal" comm. cards
agp* at pchb?
# EISA bus support
#eisa0 at mainbus?
#eisa0 at pceb?
# ISA bus support
isa0 at mainbus?
#isa0 at pceb?
isa0 at pcib?
# PCMCIA bus support
#pcmcia* at pcic? controller ? socket ?
#pcmcia* at tcic? controller ? socket ?
# MCA bus support
#mca0 at mainbus?
# ISA PCMCIA controllers
#pcic0 at isa? port 0x3e0 iomem 0xd0000 iosiz 0x10000
#pcic1 at isa? port 0x3e2 iomem 0xe0000 iosiz 0x4000
#pcic2 at isa? port 0x3e4 iomem 0xe0000 iosiz 0x4000
#tcic0 at isa? port 0x240 iomem 0xd0000 iosiz 0x10000
# PCI PCMCIA controllers
#pcic0 at pci? dev? function ?
# ISA Plug-and-Play bus support
isapnp0 at isa?
# ISA Plug-and-Play PCMCIA controllers
#pcic* at isapnp?
# CardBus bridge support
#cbb* at pci? dev ? function ?
#cardslot* at cbb?
# CardBus bus support
#cardbus* at cardslot?
#pcmcia* at cardslot?
# Coprocessor Support
# Math Coprocessor support
npx0 at isa? port 0xf0 irq 13 # x86 math coprocessor
# Console Devices
# ISA console
#pc0 at isa? port 0x60 irq 1 # pccons generic PC console driver
# Keyboard layout configuration for pccons
#options FRENCH_KBD
#options FINNISH_KBD
#options GERMAN_KBD
#options NORWEGIAN_KBD
# pccons-specific options:
#options XSERVER_DDB # PF12 gets you into DDB when X is running
#options XSERVER # X server support
# wscons
pckbc0 at isa? # pc keyboard controller
pckbd* at pckbc? # PC keyboard
pms* at pckbc? # PS/2 mouse for wsmouse
#options PMS_DISABLE_POWERHOOK
vga0 at isa?
vga* at pci? dev ? function ?
pcdisplay0 at isa? # CGA, MDA, EGA, HGA
wsdisplay* at vga? console ?
wsdisplay* at pcdisplay? console ?
wskbd* at pckbd? console ?
wsmouse* at pms? mux 0
pcppi0 at isa?
sysbeep0 at pcppi?
# Serial Devices
# PCI serial interfaces
com* at puc? port ? # 16x50s on "universal" comm boards
#cy* at pci? dev ? function ? # Cyclades Cyclom-Y serial boards
#cz* at pci? dev ? function ? # Cyclades-Z multi-port serial boards
# ISA Plug-and-Play serial interfaces
#com* at isapnp? # Modems and serial boards
# PCMCIA serial interfaces
#com* at pcmcia? function ? # Modems and serial cards
#pcmcom* at pcmcia? function ? # PCMCIA multi-port serial cards
#com* at pcmcom? slave ? # ...and the slave devices
# CardBus serial interfaces
#com* at cardbus? dev ? function ? # Modems and serial cards
# ISA serial interfaces
#options COM_HAYESP # adds Hayes ESP serial board support
com0 at isa? port 0x3f8 irq 4 # Standard PC serial ports
com1 at isa? port 0x2f8 irq 3
com2 at isa? port 0x3e8 irq 5
#com3 at isa? port 0x2e8 irq 9
#ast0 at isa? port 0x1a0 irq 5 # AST 4-port serial cards
#com* at ast? slave ?
#boca0 at isa? port 0x100 irq 5 # BOCA 8-port serial cards
#boca0 at isa? port 0x100 irq 5 # BOCA 16-port serial cards (BB2016)
#boca1 at isa? port 0x140 irq 5 # this line is also needed for BB2016
#com* at boca? slave ?
#tcom0 at isa? port 0x100 irq 7 # TC-800 8-port serial cards
#com* at tcom? slave ?
#rtfps0 at isa? port 0x1230 irq 10 # RT 4-port serial cards
#com* at rtfps? slave ?
#cy0 at isa? iomem 0xd4000 irq 12 # Cyclades serial cards
#addcom0 at isa? port 0x108 irq 5 # Addonics FlexPort 8S
#com* at addcom? slave ?
#moxa0 at isa? port 0x100 irq 5 # MOXA C168H serial card (experimental)
#com* at moxa? slave ?
# MCA serial interfaces
#com* at mca? slot ? # 16x50s on comm boards
# Parallel Printer Interfaces
# PCI parallel printer interfaces
lpt* at puc? port ? # || ports on "universal" comm boards
# ISA parallel printer interfaces
lpt0 at isa? port 0x378 irq 7 # standard PC parallel ports
lpt1 at isa? port 0x278
lpt2 at isa? port 0x3bc
# Hardware monitors
# LM7[89] and compatible hardware monitors
#lm0 at isa? port 0x290 # other common ports: 0x280, 0x310
# VIA VT82C686A hardware monitor
#viapm* at pci? dev ? function ?
#viaenv* at viapm?
# I2O devices
#iop* at pci? dev ? function ? # I/O processor
#iopsp* at iop? tid ? # SCSI/FC-AL ports
#ld* at iop? tid ? # block devices
#dpti* at iop? tid 0 # DPT/Adaptec control interface
# SCSI Controllers and Devices
# PCI SCSI controllers
#adv* at pci? dev ? function ? # AdvanSys 1200[A,B], 9xx[U,UA] SCSI
#adw* at pci? dev ? function ? # AdvanSys 9x0UW[D], 3940U[2,3]W SCSI
#ahc* at pci? dev ? function ? # Adaptec [23]94x, aic78x0 SCSI
#bha* at pci? dev ? function ? # BusLogic 9xx SCSI
#dpt* at pci? dev ? function ? # DPT SmartCache/SmartRAID
#iha* at pci? dev ? function ? # Initio INIC-940/950 SCSI
#isp* at pci? dev ? function ? # Qlogic ISP [12]0x0 SCSI/FibreChannel
#mly* at pci? dev ? function ? # Mylex AcceleRAID and eXtremeRAID
#pcscp* at pci? dev ? function ? # AMD 53c974 PCscsi-PCI SCSI
#siop* at pci? dev ? function ? # Symbios 53c8xx SCSI
#options SIOP_SYMLED # drive the act. LED in software
#trm* at pci? dev ? function ? # Tekram DC-395U/UW/F, DC-315/U SCSI
# EISA SCSI controllers
#ahb* at eisa? slot ? # Adaptec 174[02] SCSI
#ahc* at eisa? slot ? # Adaptec 274x, aic7770 SCSI
#bha* at eisa? slot ? # BusLogic 7xx SCSI
#dpt* at eisa? slot ? # DPT EATA SCSI
#uha* at eisa? slot ? # UltraStor 24f SCSI
# PCMCIA SCSI controllers
#aic* at pcmcia? function ? # Adaptec APA-1460 SCSI
#esp* at pcmcia? function ? # Qlogic ESP406/FAS408 SCSI
# ISA Plug-and-Play SCSI controllers
#aha* at isapnp? # Adaptec AHA-154[02
#aic* at isapnp? # Adaptec AHA-1520B
# ISA SCSI controllers
#adv0 at isa? port ? irq ? drq ? # AdvanSys APB-514[02] SCSI
#aha0 at isa? port 0x330 irq ? drq ? # Adaptec 154[02] SCSI
#aha1 at isa? port 0x334 irq ? drq ?
#ahc0 at isa? port ? irq ? # Adaptec 284x SCSI
#aic0 at isa? port 0x340 irq 11 # Adaptec 152[02] SCSI
#bha0 at isa? port 0x330 irq ? drq ? # BusLogic [457]4X SCSI
#bha1 at isa? port 0x334 irq ? drq ?
# The "nca" and "dpt" probes might give false hits or hang your machine.
#dpt0 at isa? port 0x170 irq ? drq ? # DPT SmartCache/SmartRAID
#nca0 at isa? port 0x360 irq 15 # Port-mapped NCR 53C80 controller
#nca1 at isa? iomem 0xd8000 irq 5 # Memory-mapped controller (T128, etc.)
#sea0 at isa? iomem 0xc8000 irq 5 # Seagate/Future Domain SCSI
#uha0 at isa? port 0x330 irq ? drq ? # UltraStor [13]4f SCSI
#uha1 at isa? port 0x340 irq ? drq ?
#wds0 at isa? port 0x350 irq 15 drq 6 # WD7000 and TMC-7000 controllers
#wds1 at isa? port 0x358 irq 11 drq 5
# CardBus SCSI cards
#adv* at cardbus? dev ? function ? # AdvanSys 1200[A,B], 9xx[U,UA] SCSI
#ahc* at cardbus? dev ? function ? # Adaptec ADP-1480
# MCA SCSI cards
#aha* at mca? slot ? # Adaptec AHA-1640
# SCSI bus support
#scsibus* at adv?
#scsibus* at adw?
#scsibus* at aha?
#scsibus* at ahb?
#scsibus* at ahc?
#scsibus* at aic?
#scsibus* at bha?
#scsibus* at dpt?
#scsibus* at esp?
#scsibus* at iha?
#scsibus* at iopsp?
#scsibus* at isp?
#scsibus* at mly?
#scsibus* at nca?
#scsibus* at pcscp?
#scsibus* at sea?
#scsibus* at siop?
#scsibus* at trm?
#scsibus* at uha?
#scsibus* at wds?
# SCSI devices
#sd* at scsibus? target ? lun ? # SCSI disk drives
#st* at scsibus? target ? lun ? # SCSI tape drives
#cd* at scsibus? target ? lun ? # SCSI CD-ROM drives
#ch* at scsibus? target ? lun ? # SCSI autochangers
#ses* at scsibus? target ? lun ? # SCSI Enclosure Services devices
#ss* at scsibus? target ? lun ? # SCSI scanners
#uk* at scsibus? target ? lun ? # SCSI unknown
# RAID controllers and devices
#aac* at pci? dev ? function ? # Adaptec AAC family
#cac* at eisa? slot ? # Compaq EISA array controllers
#cac* at pci? dev ? function ? # Compaq PCI array controllers
#icp* at pci? dev ? function ? # ICP-Vortex GDT & Intel RAID
#mlx* at pci? dev ? function ? # Mylex DAC960 & DEC SWXCR family
#mlx* at eisa? slot ? # Mylex DAC960 & DEC SWXCR family
#twe* at pci? dev ? function ? # 3ware Escalade RAID controllers
#ld* at aac? unit ? # logical disk devices
#ld* at cac? unit ?
#ld* at icp? unit ?
#ld* at twe? unit ?
#ld* at mlx? unit ?
#icpsp* at icp? unit ? # SCSI pass-through
#scsibus* at icpsp?
# IDE and related devices
# PCI IDE controllers - see pciide(4) for supported hardware.
# The 0x0001 flag force the driver to use DMA, even if the driver doesn't know
# how to set up DMA modes for this chip. This may work, or may cause
# a machine hang with some controllers.
pciide* at pci? dev ? function ? flags 0x0000
# ISA Plug-and-Play IDE controllers
#wdc* at isapnp?
# PCMCIA IDE controllers
#wdc* at pcmcia? function ?
# ISA ST506, ESDI, and IDE controllers
# Use flags 0x01 if you want to try to use 32bits data I/O (the driver will
# fall back to 16bits I/O if 32bits I/O are not functional).
# Some controllers pass the initial 32bit test, but will fail later.
#wdc0 at isa? port 0x1f0 irq 14 flags 0x00
#wdc1 at isa? port 0x170 irq 15 flags 0x00
# IDE drives
# Flags are used only with controllers that support DMA operations
# and mode settings (e.g. some pciide controllers)
# The lowest order four bits (rightmost digit) of the flags define the PIO
# mode to use, the next set of four bits the DMA mode and the third set the
# UltraDMA mode. For each set of four bits, the 3 lower bits define the mode
# to use, and the last bit must be 1 for this setting to be used.
# For DMA and UDMA, 0xf (1111) means 'disable'.
# 0x0fac means 'use PIO mode 4, DMA mode 2, disable UltraDMA'.
# (0xc=1100, 0xa=1010, 0xf=1111)
# 0x0000 means "use whatever the drive claims to support".
#wd* at wdc? channel ? drive ? flags 0x0000
wd* at pciide? channel ? drive ? flags 0x0000
# ATAPI bus support
#atapibus* at wdc? channel ?
atapibus* at pciide? channel ?
# ATAPI devices
# flags have the same meaning as for IDE drives.
cd* at atapibus? drive ? flags 0x0000 # ATAPI CD-ROM drives
sd* at atapibus? drive ? flags 0x0000 # ATAPI disk drives
st* at atapibus? drive ? flags 0x0000 # ATAPI tape drives
uk* at atapibus? drive ? flags 0x0000 # ATAPI unknown
# Miscellaneous mass storage devices
# ISA floppy
fdc0 at isa? port 0x3f0 irq 6 drq 2 # standard PC floppy controllers
#fdc1 at isa? port 0x370 irq ? drq ?
fd* at fdc? drive ? # the drives themselves
# some machines need you to do this instead of fd*
#fd0 at fdc0 drive 0
# ISA CD-ROM devices
#mcd0 at isa? port 0x300 irq 10 # Mitsumi CD-ROM drives
# ISA tape devices
# note: the wt driver conflicts unpleasantly with SMC boards at the
# same I/O address. The probe reprograms their EEPROMs. Don't
# uncomment it unless you are actually using it.
#wt0 at isa? port 0x308 irq 5 drq 1 # Archive and Wangtek QIC tape drives
# MCA ESDI devices
#edc* at mca? slot ? # IBM ESDI Disk Controllers
#ed* at edc?
# Network Interfaces
# PCI network interfaces
#an* at pci? dev ? function ? # Aironet PC4500/PC4800 (802.11)
#en* at pci? dev ? function ? # ENI/Adaptec ATM
#ep* at pci? dev ? function ? # 3Com 3c59x
#epic* at pci? dev ? function ? # SMC EPIC/100 Ethernet
#esh* at pci? dev ? function ? # Essential HIPPI card
ex* at pci? dev ? function ? # 3Com 90x[BC]
#fpa* at pci? dev ? function ? # DEC DEFPA FDDI
fxp* at pci? dev ? function ? # Intel EtherExpress PRO 10+/100B
#gsip* at pci? dev ? function ? # NS83820 Gigabit Ethernet
#hme* at pci? dev ? function ? # Sun Microelectronics STP2002-STQ
#le* at pci? dev ? function ? # PCnet-PCI Ethernet
#lmc* at pci? dev ? function ? # Lan Media Corp SSI/HSSI/DS3
#ne* at pci? dev ? function ? # NE2000-compatible Ethernet
#ntwoc* at pci? dev ? function ? # Riscom/N2 PCI Sync Serial
#pcn* at pci? dev ? function ? # AMD PCnet-PCI Ethernet
#rtk* at pci? dev ? function ? # Realtek 8129/8139
#sf* at pci? dev ? function ? # Adaptec AIC-6915 Ethernet
#sip* at pci? dev ? function ? # SiS 900/DP83815 Ethernet
#ste* at pci? dev ? function ? # Sundance ST-201 Ethernet
#stge* at pci? dev ? function ? # Sundance/Tamarack TC9021 Gigabit
#ti* at pci? dev ? function ? # Alteon ACEnic gigabit Ethernet
#tl* at pci? dev ? function ? # ThunderLAN-based Ethernet
tlp* at pci? dev ? function ? # DECchip 21x4x and clones
#vr* at pci? dev ? function ? # VIA Rhine Fast Ethernet
#wi* at pci? dev ? function ? # Intersil Prism Mini-PCI (802.11b)
#wm* at pci? dev ? function ? # Intel 82543/82544 gigabit
# EISA network interfaces
#ep* at eisa? slot ? # 3Com 3c579 Ethernet
#fea* at eisa? slot ? # DEC DEFEA FDDI
#tlp* at eisa? slot ? # DEC DE-425 Ethernet
# ISA Plug-and-Play network interfaces
#an* at isapnp? # Aironet 802.11
#ep* at isapnp? # 3Com 3c509 Ethernet
#ne* at isapnp? # NE2000-compatible Ethernet
#tr* at isapnp? # IBM/3COM TROPIC Token-Ring
# PCMCIA network interfaces
#an* at pcmcia? function ? # Aironet PC4500/PC4800 (802.11)
#awi* at pcmcia? function ? # BayStack 650/660 (802.11FH/DS)
#cnw* at pcmcia? function ? # Xircom/Netwave AirSurfer
#ep* at pcmcia? function ? # 3Com 3c589 and 3c562 Ethernet
#mbe* at pcmcia? function ? # MB8696x based Ethernet
#ne* at pcmcia? function ? # NE2000-compatible Ethernet
#ray* at pcmcia? function ? # Raytheon Raylink (802.11)
#sm* at pcmcia? function ? # Megahertz Ethernet
# tr at pcmcia has problems with Cardbus bridges
#tr* at pcmcia? function ? # TROPIC based Token-Ring
#wi* at pcmcia? function ? # Lucent WaveLan IEEE (802.11)
#xi* at pcmcia? function ? # Xircom CreditCard Ethernet
#mhzc* at pcmcia? function ? # Megahertz Ethernet/Modem combo cards
#com* at mhzc?
#sm* at mhzc?
# ISA network interfaces
#ate0 at isa? port 0x2a0 irq ? # AT1700
#cs0 at isa? port 0x300 iomem ? irq ? drq ? # CS8900 Ethernet
#ec0 at isa? port 0x250 iomem 0xd8000 irq 9 # 3Com 3c503 Ethernet
#eg0 at isa? port 0x280 irq 9 # 3C505 ethernet cards
#el0 at isa? port 0x300 irq 9 # 3C501 ethernet cards
#ep* at isa? port ? irq ? # 3C509 ethernet cards
#ef0 at isa? port 0x360 iomem 0xd0000 irq 7 # 3C507
#ai0 at isa? port 0x360 iomem 0xd0000 irq 7 # StarLAN
#fmv0 at isa? port 0x2a0 irq ? # FMV-180 series
#ix0 at isa? port 0x300 irq 10 # EtherExpress/16
#iy0 at isa? port 0x360 irq ? # EtherExpress PRO 10 ISA
#lc0 at isa? port 0x320 iomem ? irq ? # DEC EtherWORKS III (LEMAC)
#depca0 at isa? port 0x300 iomem 0xc8000 iosiz 0x8000 irq 5 # DEPCA
#le* at depca?
#nele0 at isa? port 0x320 irq 9 drq 7 # NE2100
#le* at nele?
#ntwoc0 at isa? port 0x300 irq 5 iomem 0xc8000 flags 1 # Riscom/N2 sync serial
#bicc0 at isa? port 0x320 irq 10 drq 7 # BICC IsoLan
#le* at bicc?
#ne0 at isa? port 0x280 irq 9 # NE[12]000 ethernet cards
#ne1 at isa? port 0x300 irq 10
#sm0 at isa? port 0x300 irq 10 # SMC91C9x Ethernet
#tr0 at isa? port 0xa20 iomem 0xd8000 irq ? # IBM TROPIC based Token-Ring
#tr1 at isa? port 0xa24 iomem 0xd0000 irq ? # IBM TROPIC based Token-Ring
#tr* at isa? port ? irq ? # 3COM TROPIC based Token-Ring
#we0 at isa? port 0x280 iomem 0xd0000 irq 9 # WD/SMC Ethernet
#we1 at isa? port 0x300 iomem 0xcc000 irq 10
# CardBus network cards
#ex* at cardbus? dev ? function ? # 3Com 3C575TX
#fxp* at cardbus? dev ? function ? # Intel i8255x
#rtk* at cardbus? dev ? function ? # Realtek 8129/8139
#tlp* at cardbus? dev ? function ? # DECchip 21143
# MCA network cards
#elmc* at mca? slot ? # 3Com EtherLink/MC (3c523)
#ep* at mca? slot ? # 3Com EtherLink III (3c529)
#we* at mca? slot ? # WD/SMC Ethernet
#ate* at mca? slot ? # Allied Telesis AT1720
#ne* at mca? slot ? # Novell NE/2 and clones
#tr* at mca? slot ? # IBM Token Ring adapter
#le* at mca? slot ? # SKNET Personal/MC2+
# MII/PHY support
#acphy* at mii? phy ? # DAltima AC101 and AMD Am79c874 PHYs
#amhphy* at mii? phy ? # AMD 79c901 Ethernet PHYs
#bmtphy* at mii? phy ? # Broadcom BCM5201 and BCM5202 PHYs
#brgphy* at mii? phy ? # Broadcom BCM5400-family PHYs
#dmphy* at mii? phy ? # Davicom DM9101 PHYs
#exphy* at mii? phy ? # 3Com internal PHYs
#glxtphy* at mii? phy ? # Level One LXT-1000 PHYs
#gphyter* at mii? phy ? # NS83861 Gig-E PHY
#icsphy* at mii? phy ? # Integrated Circuit Systems ICS189x
inphy* at mii? phy ? # Intel 82555 PHYs
#iophy* at mii? phy ? # Intel 82553 PHYs
#lxtphy* at mii? phy ? # Level One LXT-970 PHYs
#makphy* at mii? phy ? # Marvell Semiconductor 88E1000 PHYs
#nsphy* at mii? phy ? # NS83840 PHYs
#nsphyter* at mii? phy ? # NS83843 PHYs
#pnaphy* at mii? phy ? # generic HomePNA PHYs
#qsphy* at mii? phy ? # Quality Semiconductor QS6612 PHYs
#sqphy* at mii? phy ? # Seeq 80220/80221/80223 PHYs
#tlphy* at mii? phy ? # ThunderLAN PHYs
#tqphy* at mii? phy ? # TDK Semiconductor PHYs
ukphy* at mii? phy ? # generic unknown PHYs
#urlphy* at mii? phy ? # Realtek RTL8150L internal PHYs
# USB Controller and Devices
# PCI USB controllers
# The EHCI is not ready for prime time.
#ehci* at pci? dev ? function ? # Enhanced Host Controller
ohci* at pci? dev ? function ? # Open Host Controller
uhci* at pci? dev ? function ? # Universal Host Controller (Intel)
# CardBus USB controllers
#ehci* at cardbus? dev ? function ? # Enhanced Host Controller
#ohci* at cardbus? dev ? function ? # Open Host Controller
# USB bus support
#usb* at ehci?
usb* at ohci?
usb* at uhci?
# USB Hubs
uhub* at usb?
uhub* at uhub? port ? configuration ? interface ?
# USB HID device
uhidev* at uhub? port ? configuration ? interface ?
# USB Mice
ums* at uhidev? reportid ?
wsmouse* at ums? mux 0
# USB Keyboards
ukbd* at uhidev? reportid ?
wskbd* at ukbd? console ? mux 1
# USB Generic HID devices
uhid* at uhidev? reportid ?
# USB Printer
#ulpt* at uhub? port ? configuration ? interface ?
# USB Modem
umodem* at uhub? port ? configuration ?
ucom* at umodem?
# USB Mass Storage
umass* at uhub? port ? configuration ? interface ?
atapibus* at umass? channel ?
scsibus* at umass? channel ?
#wd* at umass?
# USB audio
#uaudio* at uhub? port ? configuration ?
# USB MIDI
#umidi* at uhub? port ? configuration ?
# USB IrDA
# USB-IrDA bridge spec
#uirda* at uhub? port ? configuration ? interface ?
#irframe* at uirda?
# SigmaTel STIr4200 USB/IrDA Bridge
#ustir* at uhub? port ?
#irframe* at ustir?
# USB Ethernet adapters
#aue* at uhub? port ? # ADMtek AN986 Pegasus based adapters
#cue* at uhub? port ? # CATC USB-EL1201A based adapters
#kue* at uhub? port ? # Kawasaki LSI KL5KUSB101B based adapters
#url* at uhub? port ? # Realtek RTL8150L based adapters
# Prolific PL2301/PL2302 host-to-host adapter
#upl* at uhub? port ?
# Serial adapters
#uftdi* at uhub? port ? # FTDI FT8U100AX serial adapter
#ucom* at uftdi? portno ?
#umct* at uhub? port ? # MCT USB-RS232 serial adapter
#ucom* at umct? portno ?
#uplcom* at uhub? port ? # I/O DATA USB-RSAQ2 serial adapter
#ucom* at uplcom? portno ?
#uvscom* at uhub? port ? # SUNTAC Slipper U VS-10U serial adapter
#ucom* at uvscom? portno ?
# Diamond Multimedia Rio 500
#urio* at uhub? port ?
# USB Handspring Visor
#uvisor* at uhub? port ?
#ucom* at uvisor?
# USB scanners
#uscanner* at uhub? port ?
# USB scanners that use SCSI emulation, e.g., HP5300
#usscanner* at uhub? port ?
#scsibus* at usscanner? channel ?
# Y@P firmware loader
#uyap* at uhub? port ?
# D-Link DSB-R100 USB radio
#udsbr* at uhub? port ?
#radio* at udsbr?
# USB Generic driver
#ugen* at uhub? port ?
# IrDA and Consumer Ir devices
# Toshiba Oboe
#oboe* at pci? dev ? function ?
#irframe* at oboe?
# Audio Devices
# PCI audio devices
#auich* at pci? dev ? function ? # Intel ICH integrated AC'97 Audio
#autri* at pci? dev ? function ? # Trident 4DWAVE based AC'97 Audio
auvia* at pci? dev ? function ? # VIA VT82C686A integrated AC'97 Audio
#clcs* at pci? dev ? function ? # Cirrus Logic CS4280
#clct* at pci? dev ? function ? # Cirrus Logic CS4281
#cmpci* at pci? dev ? function ? # C-Media CMI8338/8738
#eap* at pci? dev ? function ? # Ensoniq AudioPCI
#emuxki* at pci? dev ? function ? # Creative SBLive! and PCI512
#esa* at pci? dev ? function ? # ESS Allegro-1 / Maestro-3 PCI Audio
#esm* at pci? dev ? function ? # ESS Maestro-1/2/2e PCI Audio Accelerator
#eso* at pci? dev ? function ? # ESS Solo-1 PCI AudioDrive
#fms* at pci? dev ? function ? # Forte Media FM801
#neo* at pci? dev ? function ? # NeoMagic 256 AC'97 Audio
#sv* at pci? dev ? function ? # S3 SonicVibes
#yds* at pci? dev ? function ? # Yamaha DS-1 PCI Audio
# ISA Plug-and-Play audio devices
#ess* at isapnp? # ESS Tech ES1887, ES1888, ES888 audio
#guspnp* at isapnp? # Gravis Ultra Sound PnP audio
#sb* at isapnp? # SoundBlaster-compatible audio
#wss* at isapnp? # Windows Sound System
#ym* at isapnp? # Yamaha OPL3-SA3 audio
# ISA audio devices
# the "aria" probe might give false hits
#aria0 at isa? port 0x290 irq 10 # Aria
#ess0 at isa? port 0x220 irq 5 drq 1 drq2 5 # ESS 18XX
#gus0 at isa? port 0x220 irq 7 drq 1 drq2 6 # Gravis Ultra Sound
#pas0 at isa? port 0x220 irq 7 drq 1 # ProAudio Spectrum
#pss0 at isa? port 0x220 irq 7 drq 6 # Personal Sound System
#sp0 at pss0 port 0x530 irq 10 drq 0 # sound port driver
#sb0 at isa? port 0x220 irq 5 drq 1 drq2 5 # SoundBlaster
#wss0 at isa? port 0x530 irq 10 drq 0 drq2 1 # Windows Sound System
# PCMCIA audio devices
#esl* at pcmcia? function ? # ESS 1688 AudioDrive
#cms0 at isa? port 0x220 # Creative Music System
# OPL[23] FM synthesizers
#opl0 at isa? port 0x388 # use only if not attached to sound card
#opl* at cmpci? flags 1
#opl* at esl?
#opl* at eso?
#opl* at ess?
#opl* at fms?
#opl* at sb?
#opl* at sv?
#opl* at wss?
#opl* at yds?
#opl* at ym?
# Audio support
#audio* at aria?
#audio* at auich?
#audio* at autri?
audio* at auvia?
#audio* at clcs?
#audio* at clct?
#audio* at cmpci?
#audio* at eap?
#audio* at emuxki?
#audio* at esa?
#audio* at esl?
#audio* at esm?
#audio* at eso?
#audio* at ess?
#audio* at fms?
#audio* at gus?
#audio* at guspnp?
#audio* at neo?
#audio* at pas?
#audio* at sb?
#audio* at sp?
#audio* at sv?
#audio* at uaudio?
#audio* at wss?
#audio* at yds?
#audio* at ym?
# MPU 401 UARTs
#mpu* at isa? port 0x330 irq 9 # MPU401 or compatible card
#mpu* at cmpci?
#mpu* at eso?
#mpu* at fms?
#mpu* at sb?
#mpu* at yds?
#mpu* at ym?
# MIDI support
#midi* at autri? # Trident 4DWAVE MIDI port
#midi* at clcs? # Cirrus Logic CS4280 MIDI port
#midi* at clct? # Cirrus Logic CS4281 MIDI port
#midi* at cms? # Creative Music System
#midi* at eap? # 137[01] MIDI port
#midi* at mpu? # MPU 401
#midi* at opl? # OPL FM synth
#midi* at pcppi? # MIDI interface to the PC speaker
#midi* at sb? # SB1 MIDI port
#midi* at umidi? # USB MIDI
# The spkr driver provides a simple tone interface to the built in speaker.
#spkr0 at pcppi? # PC speaker
# FM-Radio devices
# ISA radio devices
#az0 at isa? port 0x350 # Aztech/PackardBell FM Radio Card
#az1 at isa? port 0x358
#rt0 at isa? port 0x20c # AIMS Lab Radiotrack & compatible
#rt1 at isa? port 0x284
#rt2 at isa? port 0x30c
#rt3 at isa? port 0x384
#rtii0 at isa? port 0x20c # AIMS Lab Radiotrack II FM Radio Card
#rtii1 at isa? port 0x30c
#sf2r0 at isa? port 0x384 # SoundForte RadioLink SF16-FMR2 FM Radio Card
# PCI radio devices
#mr* at pci? dev ? function ? # Guillemot Maxi Radio FM 2000 Radio Card
# Radio support
#radio* at az?
#radio* at mr?
#radio* at rt?
#radio* at rtii?
#radio* at sf2r?
# TV cards
# Brooktree 848/849/878/879 based TV cards
#bktr* at pci? dev ? function ?
#radio* at bktr?
# Mice
# ISA busmice
#lms0 at isa? port 0x23c irq 5 # Logitech bus mouse
#lms1 at isa? port 0x238 irq 5
#mms0 at isa? port 0x23c irq 5 # Microsoft InPort mouse
#mms1 at isa? port 0x238 irq 5
#wsmouse* at lms? mux 0
#wsmouse* at mms? mux 0
# backwards compatible versions
#olms0 at isa? port 0x23c irq 5 # Logitech bus mouse
#olms1 at isa? port 0x238 irq 5
#omms0 at isa? port 0x23c irq 5 # Microsoft InPort mouse
#omms1 at isa? port 0x238 irq 5
# Joysticks
#joy* at pnpbios? index ? # Game port
# ISA Plug-and-Play joysticks
#joy* at isapnp? # Game ports (usually on audio cards)
# PCI joysticks
#joy* at pci? # Game ports (usually on audio cards)
#joy* at eso? # ESS Solo-1 on-board joystick
# ISA joysticks. Probe is a little strange; add only if you have one.
#joy0 at isa? port 0x201
# Miscellaneous Devices
# Planetconnect Satellite receiver driver.
#satlink0 at isa? port 0x300 drq 1
# Middle Digital, Inc. PCI-Weasel serial console board control
# devices (watchdog timer, etc.)
weasel* at pci?
# Pull in optional local configuration
include "arch/i386/conf/GENERIC.local"
# Pseudo-Devices
# disk/mass storage pseudo-devices
#pseudo-device ccd 4 # concatenated/striped disk devices
#pseudo-device raid 8 # RAIDframe disk driver
#options RAID_AUTOCONFIG # auto-configuration of RAID components
#Options to enable various other RAIDframe RAID types.
#options RF_INCLUDE_EVENODD=1
#options RF_INCLUDE_RAID5_RS=1
#options RF_INCLUDE_PARITYLOGGING=1
#options RF_INCLUDE_CHAINDECLUSTER=1
#options RF_INCLUDE_INTERDECLUSTER=1
#options RF_INCLUDE_PARITY_DECLUSTERING=1
#options RF_INCLUDE_PARITY_DECLUSTERING_DS=1
pseudo-device md 1 # memory disk device (ramdisk)
pseudo-device vnd 4 # disk-like interface to files
# network pseudo-devices
pseudo-device bpfilter 8 # Berkeley packet filter
pseudo-device ipfilter # IP filter (firewall) and NAT
pseudo-device loop # network loopback
#pseudo-device ppp 2 # Point-to-Point Protocol
#pseudo-device pppoe # PPP over Ethernet (RFC 2516)
#pseudo-device sl 2 # Serial Line IP
#pseudo-device strip 2 # Starmode Radio IP (Metricom)
#pseudo-device irframetty # IrDA frame line discipline
#pseudo-device tun 2 # network tunneling over tty
#pseudo-device gre 2 # generic L3 over IP tunnel
#pseudo-device gif 4 # IPv[46] over IPv[46] tunnel (RFC1933)
#pseudo-device faith 1 # IPv[46] tcp relay translation i/f
#pseudo-device stf 1 # 6to4 IPv6 over IPv4 encapsulation
#pseudo-device vlan # IEEE 802.1q encapsulation
#pseudo-device bridge # simple inter-network bridging
# miscellaneous pseudo-devices
pseudo-device pty # pseudo-terminals
#pseudo-device tb 1 # tablet line discipline
#pseudo-device sequencer 1 # MIDI sequencer
# rnd works; RND_COM does not on port i386 yet.
pseudo-device rnd # /dev/random and in-kernel generator
#options RND_COM # use "com" randomness as well (BROKEN)
pseudo-device clockctl # user control of clock subsystem
# a pseudo device needed for Coda # also needs CODA (above)
#pseudo-device vcoda 4 # coda minicache <-> venus comm.
# wscons pseudo-devices
pseudo-device wsmux # mouse & keyboard multiplexor
pseudo-device wsfont
------------------------------
And the dmesg is as follows:
NetBSD 1.6.1_RC1 (BLACKHOLE) #5: Sat Feb 15 18:46:52 MET 2003
he@black-3.sth.netnod.se:/usr/src/sys/arch/i386/compile/BLACKHOLE
cpu0: AMD Duron (686-class), 807.25 MHz
cpu0: I-cache 64 KB 64b/line 2-way, D-cache 64 KB 64b/line 2-way
cpu0: L2 cache 64 KB 64b/line 16-way
cpu0: features 183f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR>
cpu0: features 183f9ff<PGE,MCA,CMOV,FGPAT,PSE36,MMX>
cpu0: features 183f9ff<FXSR>
total memory = 239 MB
avail memory = 219 MB
using 3092 buffers containing 12368 KB of memory
BIOS32 rev. 0 found at 0xf0e70
mainbus0 (root)
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: VIA Technologies VT8363 KT133 System Controller (rev. 0x81)
agp0 at pchb0: aperture at 0xf8000000, size 0x10000000
ppb0 at pci0 dev 1 function 0: VIA Technologies VT8363 KT133 PCI to AGP Bridge (rev. 0x00)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled
S3 product 0x8a26 (VGA display, revision 0x03) at pci1 dev 0 function 0 not configured
pcib0 at pci0 dev 7 function 0
pcib0: VIA Technologies VT82C686A (Apollo KX133) PCI-ISA Bridge (rev. 0x40)
pciide0 at pci0 dev 7 function 1: VIA Technologies VT82C686A (Apollo KX133) ATA100 controller
pciide0: bus-master DMA support present
pciide0: primary channel configured to compatibility mode
wd0 at pciide0 channel 0 drive 0: <IC35L020AVER07-0>
wd0: drive supports 16-sector PIO transfers, LBA addressing
wd0: 19623 MB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 40188960 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
pciide0: primary channel interrupting at irq 14
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5 (Ultra/100) (using DMA data transfers)
pciide0: secondary channel configured to compatibility mode
atapibus0 at pciide0 channel 1: 2 targets
cd0 at atapibus0 drive 0: <CD-540E, , 1.0A> type 5 cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33)
pciide0: secondary channel interrupting at irq 15
cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2 (Ultra/33) (using DMA data transfers)
uhci0 at pci0 dev 7 function 2: VIA Technologies VT83C572 USB Controller (rev. 0x16)
uhci0: interrupting at irq 5
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: VIA Technologie UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 7 function 3: VIA Technologies VT83C572 USB Controller (rev. 0x16)
uhci1: interrupting at irq 5
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: VIA Technologie UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
VIA Technologies VT82C686A SMBus Controller (miscellaneous bridge, revision 0x40) at pci0 dev 7 function 4 not configured
auvia0 at pci0 dev 7 function 5: VIA VT82C686A AC'97 Audio (rev 0x50)
auvia0: interrupting at irq 12
auvia0: ALG16 codec; headphone, 18 bit DAC, 18 bit ADC, Realtek 3D
audio0 at auvia0: full duplex, mmap, independent
fxp0 at pci0 dev 13 function 0: i82550 Ethernet, rev 12
fxp0: interrupting at irq 11
fxp0: Ethernet address 00:02:b3:a0:b6:33
inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 4
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp1 at pci0 dev 14 function 0: i82550 Ethernet, rev 12
fxp1: interrupting at irq 10
fxp1: Ethernet address 00:02:b3:89:3f:05
inphy1 at fxp1 phy 1: i82555 10/100 media interface, rev. 4
inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vga1 at pci0 dev 15 function 0: Middle Digital, Inc. Weasel Virtual VGA (rev. 0x47)
wsdisplay0 at vga1 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
puc0 at pci0 dev 15 function 1: Middle Digital, Inc. Weasel serial port (com)
com3 at puc0 port 0: interrupting at irq 12
com3: ns16550a, working fifo
weasel0 at pci0 dev 15 function 2: PCI-Weasel watchdog timer
weasel0: Weasel didn't respond to SHOW_CONFIG
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
lpt0 at isa0 port 0x378-0x37b irq 7
pcppi0 at isa0 port 0x61
sysbeep0 at pcppi0
isapnp0 at isa0 port 0x279: ISA Plug 'n Play device support
npx0 at isa0 port 0xf0-0xff: using exception 16
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
isapnp0: no ISA Plug 'n Play devices found
biomask e365 netmask ef65 ttymask efe7
uhub0: port error, restarting port 1
uhub0: port error, restarting port 2
uhub1: port error, restarting port 1
uhub1: port error, restarting port 2
boot device: wd0
root on wd0a dumps on wd0b
root file system type: ffs
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)
>How-To-Repeat:
Subject a netbsd-1-6-running host to high levels of "random"
short-lived TCP sessions, and watch the machine "randomly" lose.
>Fix:
Sorry, don't know; hints for further debugging gratefully
accepted.
>Release-Note:
>Audit-Trail:
From: Havard Eidnes <he@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: kern-bug-people@netbsd.org
Subject: Re: kern/20390: callwheel corruption under higher network loads
Date: Fri, 28 Feb 2003 15:41:42 +0100 (CET)
Hi,
here is a brief "formal" update on this case, after inserting
booby-traps in the code in several places related to handling of
callouts.
The problem is apparently caused by what can happen when a callout
function is invoked. Just before that happens, priority level is
lowered (with CALLWHEEL_UNLOCK() which uses splx()), which will allow
other code to run before the callout function is run. Sometimes, the
code which is run in the meantime can cause the storage containing the
callout structure whose callout is now waiting to be run to be released.
The callout function can even get to reactivating itself (e.g. to cause
repeated TCP retransmits), using the now-released storage. Sometimes
this storage is even reallocated, causing Bad Things to happen to the
callout data structures when that container structure is initialized/
zeroed before being taken into use.
This has been observed to be happening with both syn-cache entries and
TCP control blocks.
I'm presently running with code which delays the release of the storage
until the callout function is invoked, and the "delayed release" has
been triggered a few times already (I added a couple of TCP statistics
counters to monitor this):
black-3: {46} netstat -s | egrep delay
5028963 ack-only packets (4889736 delayed)
--> 9 delayed frees of tcpcb
--> 0 double delayed frees of tcpcb
--> 5 delayed pool_put()s
black-3: {47}
The latter counter is for syn-cache entries.
Since this is all with netbsd-1-6 code, I'll need to take a look at
whether this also applies to NetBSD-current, but chances are that it
does.
Regards,
- Havard
Responsible-Changed-From-To: kern-bug-people->he
Responsible-Changed-By: perry
Responsible-Changed-When: Sat Apr 5 11:31:51 PST 2003
Responsible-Changed-Why:
Havard appears to be actively debugging this.
From: Havard Eidnes <he@netbsd.org>
To: gnats-bugs@gnats.netbsd.org
Cc:
Subject: Re: kern/20390: callwheel corruption at higher network loads
Date: Thu, 17 Jul 2003 11:07:00 +0200 (CEST)
----Next_Part(Thu_Jul_17_11:07:00_2003_251)--
Content-Type: Text/Plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
Hi,
here is the current set of fixes which I use with -current which works
around this problem.
A summary of the private feedback I have received so far is:
o This is not MP-safe, in that a timer could fire between the test
and the action on the test. This is true. However, it is my
understanding that in the present implementation of SMP in NetBSD,
spl() provides the required mutual exclusion, also from interrupts
on other CPUs, so if I've understood things correctly, this
workaround does not *in the present state* of NetBSD introduce any
*new* problems.
o This is actually a generic problem related to callouts, and the fix
is a special-case fix for the netinet code.
Some allusions were made to the fact that this problem might
actually be hitting us in other parts of the code, but no concrete
references were given.
However, I have tried to take a brief look at the other uses of the
callout code in NetBSD, and it does not appear that the other uses
are as eager to use dynamic storage allocation of structures
containing callout structures as the netinet code, and this
workaround does provably improve the robustness of NetBSD.
o One comment indicated that a "RUNNING" flag for the callout code
would be preferable to the "INVOKING" flag, and I understood that
the RUNNING flag should be kept active as long as possible to cover
the entire time the callout function is running. This poses
problems because some of the timer functions will want to call
tcp_close(), and it tests the flag to decide whether to release the
storage or "mark as dead", and with the flag still set, the storage
would never be released. Also, several of the timer functions have
several exits, making it necessary to sprinkle callout_ack() calls
in several more places.
As long as the spl() mechanism provides mutual exclusion, the
critical section which needs protection is the span between the
lowering of the priority in the callout code and the raising of the
priority in the spl() calls in the callout handlers, so here I have
decided to keep the "INVOKING" flag and associated naming.
o The callout code could be changed so that callout handlers have to
be registered with a priority level, and instead of dropping all
priority before invoking the callout handler, the callout code
could drop priority directly to the desired priority level, thereby
avoiding this problem altogether.
However, this will possibly require MD changes to the priority
handling code, and this would be a project exceeding my current
abilities, and noone else has taken up this ball.
o The callout code and API could be redesigned so as to avoid this
problem; references were made to the Solaris timeout code.
Again, a full redesign of the callout code exceeds my abilities,
and noone else has taken up the ball and run with it in this
direction.
Attached below is the current version of the patch I'm using with
NetBSD-current.
BTW, this problem is also present on the netbsd-1-6 branch, and some
of the suggestions above would IMHO be inappropriate to pull up to the
netbsd-1-6 branch. A large motivation on my part with this workaround
is to get it in place on the netbsd-1-6 branch, in other words get it
out there as soon as possible to improve the stability of our
"released code".
Regards,
- H=E5vard
----Next_Part(Thu_Jul_17_11:07:00_2003_251)--
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Index: sys/sys/callout.h
===================================================================
RCS file: /usr/users/he/nbcvs/netbsd/src/sys/sys/callout.h,v
retrieving revision 1.17
diff -u -r1.17 callout.h
--- sys/sys/callout.h 4 Feb 2003 01:21:06 -0000 1.17
+++ sys/sys/callout.h 17 Jul 2003 07:42:27 -0000
@@ -83,6 +83,7 @@
#define CALLOUT_PENDING 0x0002 /* callout is on the queue */
#define CALLOUT_FIRED 0x0004 /* callout has fired */
+#define CALLOUT_INVOKING 0x0008 /* callout function is being invoked */
#define CALLOUT_INITIALIZER_SETFUNC(func, arg) \
{ { NULL, NULL }, func, arg, 0, 0 }
@@ -100,6 +101,8 @@
#define callout_pending(c) ((c)->c_flags & CALLOUT_PENDING)
#define callout_expired(c) ((c)->c_flags & CALLOUT_FIRED)
+#define callout_invoking(c) ((c)->c_flags & CALLOUT_INVOKING)
+#define callout_ack(c) ((c)->c_flags &= ~CALLOUT_INVOKING)
#endif /* _KERNEL */
#endif /* !_SYS_CALLOUT_H_ */
Index: sys/kern/kern_timeout.c
===================================================================
RCS file: /usr/users/he/nbcvs/netbsd/src/sys/kern/kern_timeout.c,v
retrieving revision 1.7
diff -u -r1.7 kern_timeout.c
--- sys/kern/kern_timeout.c 14 Jul 2003 14:59:01 -0000 1.7
+++ sys/kern/kern_timeout.c 17 Jul 2003 07:43:20 -0000
@@ -258,7 +258,7 @@
/* Initialize the time here, it won't change. */
old_time = c->c_time;
c->c_time = to_ticks + hardclock_ticks;
- c->c_flags &= ~CALLOUT_FIRED;
+ c->c_flags &= ~(CALLOUT_FIRED|CALLOUT_INVOKING);
c->c_func = func;
c->c_arg = arg;
@@ -299,7 +299,7 @@
/* Initialize the time here, it won't change. */
old_time = c->c_time;
c->c_time = to_ticks + hardclock_ticks;
- c->c_flags &= ~CALLOUT_FIRED;
+ c->c_flags &= ~(CALLOUT_FIRED|CALLOUT_INVOKING);
/*
* If this timeout is already scheduled and now is moved
@@ -334,7 +334,7 @@
if (callout_pending(c))
CIRCQ_REMOVE(&c->c_list);
- c->c_flags &= ~(CALLOUT_PENDING|CALLOUT_FIRED);
+ c->c_flags &= ~(CALLOUT_PENDING|CALLOUT_FIRED|CALLOUT_INVOKING);
CALLOUT_UNLOCK(s);
}
@@ -393,7 +393,7 @@
callout_ev_late.ev_count++;
#endif
c->c_flags = (c->c_flags & ~CALLOUT_PENDING) |
- CALLOUT_FIRED;
+ (CALLOUT_FIRED|CALLOUT_INVOKING);
func = c->c_func;
arg = c->c_arg;
Index: sys/netinet/tcp_input.c
===================================================================
RCS file: /usr/users/he/nbcvs/netbsd/src/sys/netinet/tcp_input.c,v
retrieving revision 1.172
diff -u -r1.172 tcp_input.c
--- sys/netinet/tcp_input.c 2 Jul 2003 19:33:20 -0000 1.172
+++ sys/netinet/tcp_input.c 17 Jul 2003 07:45:37 -0000
@@ -2799,7 +2799,10 @@
(void) m_free((sc)->sc_ipopts); \
if ((sc)->sc_route4.ro_rt != NULL) \
RTFREE((sc)->sc_route4.ro_rt); \
- pool_put(&syn_cache_pool, (sc)); \
+ if (callout_invoking(&(sc)->sc_timer)) \
+ (sc)->sc_flags |= SCF_DEAD; \
+ else \
+ pool_put(&syn_cache_pool, (sc)); \
} while (/*CONSTCOND*/0)
struct pool syn_cache_pool;
@@ -2946,6 +2949,14 @@
int s;
s = splsoftnet();
+ callout_ack(&sc->sc_timer);
+
+ if (__predict_false(sc->sc_flags & SCF_DEAD)) {
+ tcpstat.tcps_sc_delayed_free++;
+ pool_put(&syn_cache_pool, sc);
+ splx(s);
+ return;
+ }
if (__predict_false(sc->sc_rxtshift == TCP_MAXRXTSHIFT)) {
/* Drop it -- too many retransmissions. */
Index: sys/netinet/tcp_subr.c
===================================================================
RCS file: /usr/users/he/nbcvs/netbsd/src/sys/netinet/tcp_subr.c,v
retrieving revision 1.143
diff -u -r1.143 tcp_subr.c
--- sys/netinet/tcp_subr.c 3 Jul 2003 08:28:16 -0000 1.143
+++ sys/netinet/tcp_subr.c 17 Jul 2003 07:48:14 -0000
@@ -1031,6 +1031,32 @@
}
/*
+ * Return whether this tcpcb is marked as dead, indicating
+ * to the calling timer function that no further action should
+ * be taken, as we are about to release this tcpcb. The release
+ * of the storage will be done if this is the last timer running.
+ *
+ * This is typically called from the callout handler function before
+ * callout_ack() is done, therefore we need to test the number of
+ * running timer functions against 1 below, not 0.
+ */
+int
+tcp_isdead(tp)
+ struct tcpcb *tp;
+{
+ int dead = (tp->t_flags & TF_DEAD);
+
+ if (__predict_false(dead)) {
+ if (tcp_timers_invoking(tp) > 1)
+ /* not quite there yet -- count separately? */
+ return dead;
+ tcpstat.tcps_delayed_free++;
+ pool_put(&tcpcb_pool, tp);
+ }
+ return dead;
+}
+
+/*
* Close a TCP control block:
* discard all space held by the tcp
* discard internet protocol block
@@ -1148,7 +1174,11 @@
m_free(tp->t_template);
tp->t_template = NULL;
}
- pool_put(&tcpcb_pool, tp);
+ if (tcp_timers_invoking(tp))
+ tp->t_flags |= TF_DEAD;
+ else
+ pool_put(&tcpcb_pool, tp);
+
if (inp) {
inp->inp_ppcb = 0;
soisdisconnected(so);
Index: sys/netinet/tcp_timer.c
===================================================================
RCS file: /usr/users/he/nbcvs/netbsd/src/sys/netinet/tcp_timer.c,v
retrieving revision 1.62
diff -u -r1.62 tcp_timer.c
--- sys/netinet/tcp_timer.c 3 Feb 2003 23:51:04 -0000 1.62
+++ sys/netinet/tcp_timer.c 17 Jul 2003 08:22:41 -0000
@@ -196,6 +196,24 @@
}
/*
+ * Return how many timers are currently being invoked.
+ */
+int
+tcp_timers_invoking(struct tcpcb *tp)
+{
+ int i;
+ int count = 0;
+
+ for (i = 0; i < TCPT_NTIMERS; i++)
+ if (callout_invoking(&tp->t_timer[i]))
+ count++;
+ if (callout_invoking(&tp->t_delack_ch))
+ count++;
+
+ return count;
+}
+
+/*
* Callout to process delayed ACKs for a TCPCB.
*/
void
@@ -211,6 +229,12 @@
*/
s = splsoftnet();
+ callout_ack(&tp->t_delack_ch);
+ if (tcp_isdead(tp)) {
+ splx(s);
+ return;
+ }
+
tp->t_flags |= TF_ACKNOW;
(void) tcp_output(tp);
splx(s);
@@ -267,6 +291,11 @@
#endif
s = splsoftnet();
+ callout_ack(&tp->t_timer[TCPT_KEEP]);
+ if (tcp_isdead(tp)) {
+ splx(s);
+ return;
+ }
#ifdef TCP_DEBUG
#ifdef INET
@@ -414,6 +443,11 @@
#endif
s = splsoftnet();
+ callout_ack(&tp->t_timer[TCPT_PERSIST]);
+ if (tcp_isdead(tp)) {
+ splx(s);
+ return;
+ }
#ifdef TCP_DEBUG
#ifdef INET
@@ -476,6 +510,11 @@
#endif
s = splsoftnet();
+ callout_ack(&tp->t_timer[TCPT_KEEP]);
+ if (tcp_isdead(tp)) {
+ splx(s);
+ return;
+ }
#ifdef TCP_DEBUG
ostate = tp->t_state;
@@ -558,6 +597,11 @@
#endif
s = splsoftnet();
+ callout_ack(&tp->t_timer[TCPT_2MSL]);
+ if (tcp_isdead(tp)) {
+ splx(s);
+ return;
+ }
#ifdef TCP_DEBUG
#ifdef INET
Index: sys/netinet/tcp_var.h
===================================================================
RCS file: /usr/users/he/nbcvs/netbsd/src/sys/netinet/tcp_var.h,v
retrieving revision 1.102
diff -u -r1.102 tcp_var.h
--- sys/netinet/tcp_var.h 29 Jun 2003 22:32:00 -0000 1.102
+++ sys/netinet/tcp_var.h 17 Jul 2003 07:51:57 -0000
@@ -185,6 +185,7 @@
#define TF_CANT_TXSACK 0x1000 /* other side said I could not SACK */
#define TF_IGNR_RXSACK 0x2000 /* ignore received SACK blocks */
#define TF_REASSEMBLING 0x4000 /* we're busy reassembling */
+#define TF_DEAD 0x8000 /* dead and to-be-released */
struct mbuf *t_template; /* skeletal packet for transmit */
@@ -417,6 +418,7 @@
#define SCF_UNREACH 0x0001 /* we've had an unreach error */
#define SCF_TIMESTAMP 0x0002 /* peer will do timestamps */
+#define SCF_DEAD 0x0004 /* this entry to be released */
struct mbuf *sc_ipopts; /* IP options */
u_int16_t sc_peermaxseg;
@@ -547,6 +549,7 @@
u_quad_t tcps_noport; /* no socket on port */
u_quad_t tcps_badsyn; /* received ack for which we have
no SYN in compressed state */
+ u_quad_t tcps_delayed_free; /* delayed pool_put() of tcpcb */
/* These statistics deal with the SYN cache. */
u_quad_t tcps_sc_added; /* # of entries added */
@@ -561,6 +564,7 @@
u_quad_t tcps_sc_dropped; /* # of SYNs dropped (no route/mem) */
u_quad_t tcps_sc_collisions; /* # of hash collisions */
u_quad_t tcps_sc_retransmitted; /* # of retransmissions */
+ u_quad_t tcps_sc_delayed_free; /* # of delayed pool_put()s */
u_quad_t tcps_selfquench; /* # of ENOBUFS we get on output */
};
@@ -706,8 +710,10 @@
int tcp_attach __P((struct socket *));
void tcp_canceltimers __P((struct tcpcb *));
+int tcp_timers_invoking __P((struct tcpcb*));
struct tcpcb *
tcp_close __P((struct tcpcb *));
+int tcp_isdead __P((struct tcpcb *));
#ifdef INET6
void tcp6_ctlinput __P((int, struct sockaddr *, void *));
#endif
Index: usr.bin/netstat/inet.c
===================================================================
RCS file: /usr/users/he/nbcvs/netbsd/src/usr.bin/netstat/inet.c,v
retrieving revision 1.56
diff -u -r1.56 inet.c
--- usr.bin/netstat/inet.c 12 Jul 2003 13:39:23 -0000 1.56
+++ usr.bin/netstat/inet.c 13 Jul 2003 15:15:24 -0000
@@ -263,6 +263,7 @@
p2(tcps_closed, tcps_drops,
"\t%llu connection%s closed (including %llu drop%s)\n");
p(tcps_conndrops, "\t%llu embryonic connection%s dropped\n");
+ p(tcps_delayed_free, "\t%llu delayed free%s of tcpcb\n");
p2(tcps_rttupdated, tcps_segstimed,
"\t%llu segment%s updated rtt (of %llu attempt%s)\n");
p(tcps_rexmttimeo, "\t%llu retransmit timeout%s\n");
@@ -292,6 +293,8 @@
ps(tcps_sc_bucketoverflow, "\t\t%llu dropped due to bucket overflow\n");
ps(tcps_sc_reset, "\t\t%llu dropped due to RST\n");
ps(tcps_sc_unreach, "\t\t%llu dropped due to ICMP unreachable\n");
+ ps(tcps_sc_delayed_free, "\t\t%llu delayed free of SYN cache "
+ "entries\n");
p(tcps_sc_retransmitted, "\t%llu SYN,ACK%s retransmitted\n");
p(tcps_sc_dupesyn, "\t%llu duplicate SYN%s received for entries "
"already in the cache\n");
Index: share/man/man9/callout.9
===================================================================
RCS file: /usr/users/he/nbcvs/netbsd/src/share/man/man9/callout.9,v
retrieving revision 1.9
diff -u -r1.9 callout.9
--- share/man/man9/callout.9 16 Apr 2003 13:35:25 -0000 1.9
+++ share/man/man9/callout.9 17 Jul 2003 07:54:41 -0000
@@ -43,6 +43,9 @@
.Nm callout_schedule ,
.Nm callout_setfunc ,
.Nm callout_stop ,
+.Nm callout_expired ,
+.Nm callout_invoking ,
+.Nm callout_ack ,
.Nm CALLOUT_INITIALIZER ,
.Nm CALLOUT_INITIALIZER_SETFUNC
.Nd execute a function after a specified length of time
@@ -63,6 +66,10 @@
.Fn "callout_pending" "struct callout *c"
.Ft int
.Fn "callout_expired" "struct callout *c"
+.Ft int
+.Fn "callout_invoking" "struct callout *c"
+.Ft void
+.Fn "callout_ack" "struct callout *c"
.Fd CALLOUT_INITIALIZER
.Pp
.Fd CALLOUT_INITIALIZER_SETFUNC(func, arg)
@@ -117,8 +124,10 @@
Once the timer is started, the callout handle is marked as
.Em PENDING .
Once the timer expires,
-the handle is marked at
+the handle is marked as
.Em EXPIRED
+and
+.Em RUNNING
and the
.Em PENDING
status is cleared.
@@ -153,11 +162,11 @@
function stops the timer associated the callout handle
.Fa c .
The
-.Em PENDING
+.Em PENDING ,
+.Em EXPIRED ,
+and
+.Em RUNNING
status for the callout handle is cleared.
-The
-.Em EXPIRED
-status is not affected.
It is safe to call
.Fn callout_stop
on a callout handle that is not pending, so long as it is initialized.
@@ -183,6 +192,42 @@
.Fn callout_expired
function tests to see if the callout's timer has expired and its
function called.
+.Pp
+The
+.Fn callout_invoking
+function tests the
+.Em INVOKING
+status of the callout handle
+.Fa c .
+This flag is set just before a callout's function is being called.
+Since the priority level is lowered prior to invocation of the
+callout function, other pending higher-priority code may run before
+the callout function is allowed to run.
+This may create a race condition if this higher-priority code
+deallocates storage containing one or more callout structures whose
+callout functions are about to be run.
+In such cases, one technique to prevent references to deallocated
+storage would be to test whether any callout functions are in the
+.Em INVOKING
+state using
+.Fn callout_invoking ,
+and if so, to mark the data structure and defer storage
+deallocation until the callout function is allowed to run.
+For this handshake protocol to work, the callout function will
+have to use the
+.Fn callout_ack
+function to clear this flag.
+.Pp
+The
+.Fn callout_ack
+function clears the
+.Em INVOKING
+state in the callout handle
+.Fa c .
+This is used in situations where it is necessary to protect against
+the race condition described under
+.Fn callout_invoking .
+The
.Sh SEE ALSO
.Xr hz 9
.Sh HISTORY
----Next_Part(Thu_Jul_17_11:07:00_2003_251)----
From: Havard Eidnes <he@netbsd.org>
To: gnats-bugs@gnats.netbsd.org
Cc:
Subject: Re: kern/20390: callwheel corruption at higher network loads
Date: Thu, 17 Jul 2003 13:55:47 +0200 (CEST)
Someone asked about a further explanation of this problem, and here is
for posterity a copy of my reply:
> Am I right in thinking that this whole problem is down to calling
> callout_stop() in the window of:
> CALLOUT_UNLOCK(s);
> (*func)(arg);
> CALLOUT_LOCK(s);
> (Since one presumes that the calling code is remembering that it
> has the timer pending.)
The way I understand this, no. The way I understand this what can
happen in the context of the TCP code is as follows:
1) a tcp session sets one (or several) timers for a given tcpcb
2) one of these timers fires, and the callout code lowers priority
prior to calling the callout handler with a pointer to the tcpcb as
argument.
3) while the callout code has been running at elevated priority (at
splsched()), network traffic has arrived, and when the priority is
lowered, network interrupts are processed, typically at splnet().
4) the code at splnet() will schedule soft network interrupts to be
processed, and they also get invoked, still before the callout
handler is invoked.
5) some of this "process incoming network data" may cause tcp_close() to
be called. This will in turn call tcp_canceltimers(), which does
callout_stop(). This is not problematical; CALLOUT_PENDING has
already been cleared in the c_flags field in the callout struct by
softclock(), so callout_stop() will not try to remove it from the
queue, but remember, the callout handler has not yet been invoked,
but is still waiting for us further down on the stack! tcp_close()
will however, as the code currently stands, do a pool_put() of the
tcp control block. (You probably now realize where this is headed.)
6) when the incoming network traffic has been processed, we return from
(hard and soft) interrupt context, and the callout handler gets to be
invoked with a pointer to the now-released tcpcb as argument. The
callout handler may decide on it's own to reschedule a timeout using
the released tcpcb (I'm a little fuzzy on this point). This will put
the callout struct inside the now-released tcpcb back on the callout
queues (this has been observed as happening by adding sanity checks).
7) when the storage for the tcpcb, which contains the callout structs
is reallocated, the contents is reinitialized, causing the queue
pointers in the callout structs to be reinitialized, causing
corruption of the callwheel data structure (this has also been
observed as happening by adding sanity checks).
The end result is as already described.
> The whole business of callouts is definitely riddled with timing errors.
> NetBSD can't use some of the solutions because it uses caller-supplied
> data areas which it is useful to allow the timeout routine to invalidate.
Yes; the caller-supplied data areas and the fact that the callout code
relinquishes ownership of the callout struct at the moment the callout
handler is invoked makes some otherwise conceivable simple solutions
impossible. This is, e.g. what triggered the need for the code which
uses the callout code to cooperate by using callout_ack() in my patch.
However, redoing the callout API to fix this design flaw is most
probably going to be too big of a change for the netbsd-1-6 branch.
> I know some systems make callout_stop() loop/sleep until the routine
> finishes (for MP sanity).
>
> I think it might be possible to fix things with only minor changes:
> 1) Make callout_stop() tell the user whether the callout was active
> - maybe return the old value of c_flags.
> 2) Don't clear CALLOUT_FIRED in callout_stop().
As I said above, the problem appears to be a bit more insidious than
that, so I'm unconvinced that this will fix the problem.
Regards,
- Havard
From: Havard Eidnes <he@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: pr/20390 CVS commit: src
Date: Sun, 20 Jul 2003 16:25:59 +0000 (UTC)
Module Name: src
Committed By: he
Date: Sun Jul 20 16:25:59 UTC 2003
Modified Files:
src/share/man/man9: callout.9
src/sys/kern: kern_timeout.c
src/sys/sys: callout.h
Log Message:
Temporarily introduce CALLOUT_INVOKING, callout_invoking() and callout_ack()
to make users of the callout facility able to cooperate to work around the
race caused by the callout code lowering interrupt priority level when
invoking callout handlers, something which allows other code to run before
the callout handler gets to it's spl*() call.
This is to enable the workaround for the TCP code found in PR#20390 to be
applied.
This should be backed out once a more comprehensive fix can be put in
place.
To generate a diff of this commit:
cvs rdiff -r1.9 -r1.10 src/share/man/man9/callout.9
cvs rdiff -r1.7 -r1.8 src/sys/kern/kern_timeout.c
cvs rdiff -r1.17 -r1.18 src/sys/sys/callout.h
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Havard Eidnes <he@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: pr/20390 CVS commit: src
Date: Sun, 20 Jul 2003 16:35:12 +0000 (UTC)
Module Name: src
Committed By: he
Date: Sun Jul 20 16:35:12 UTC 2003
Modified Files:
src/sys/netinet: tcp_input.c tcp_subr.c tcp_timer.c tcp_var.h
src/usr.bin/netstat: inet.c
Log Message:
As a temporary workaround, apply the fix from PR#20390, thereby
cooperating with the callout code in working around the race
condition caused by the TCP code's use of the callout facility.
Instead of unconditionally releasing memory in tcp_close() and
SYN_CACHE_PUT(), check whether any of the related callout handlers
are about to be invoked (but have not yet done callout_ack()), and
if so, just mark the associated data structure (tcpcb or syn cache
entry) as "dead", and test for this (and release storage) in the
callout handler functions.
To generate a diff of this commit:
cvs rdiff -r1.172 -r1.173 src/sys/netinet/tcp_input.c
cvs rdiff -r1.143 -r1.144 src/sys/netinet/tcp_subr.c
cvs rdiff -r1.62 -r1.63 src/sys/netinet/tcp_timer.c
cvs rdiff -r1.102 -r1.103 src/sys/netinet/tcp_var.h
cvs rdiff -r1.56 -r1.57 src/usr.bin/netstat/inet.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: James Chacon <jmc@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: pr/20390 CVS commit: [netbsd-1-6] src/sys/netinet
Date: Wed, 22 Oct 2003 06:05:47 +0000 (UTC)
Module Name: src
Committed By: jmc
Date: Wed Oct 22 06:05:47 UTC 2003
Modified Files:
src/sys/netinet [netbsd-1-6]: tcp_input.c
Log Message:
Pullup rev 1.173 (requested by he in ticket #1530)
Introduce a new INVOKING status for callouts, and use it to close
a race condition in the TCP code. Fixes PR#20390.
To generate a diff of this commit:
cvs rdiff -r1.141.4.6 -r1.141.4.7 src/sys/netinet/tcp_input.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: James Chacon <jmc@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: pr/20390 CVS commit: [netbsd-1-6] src/sys/netinet
Date: Wed, 22 Oct 2003 06:06:05 +0000 (UTC)
Module Name: src
Committed By: jmc
Date: Wed Oct 22 06:06:05 UTC 2003
Modified Files:
src/sys/netinet [netbsd-1-6]: tcp_timer.c
Log Message:
Pullup rev 1.63 (requested by he in ticket #1530)
Introduce a new INVOKING status for callouts, and use it to close
a race condition in the TCP code. Fixes PR#20390.
To generate a diff of this commit:
cvs rdiff -r1.57.10.1 -r1.57.10.2 src/sys/netinet/tcp_timer.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: James Chacon <jmc@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: pr/20390 CVS commit: [netbsd-1-6] src/sys/netinet
Date: Wed, 22 Oct 2003 06:05:57 +0000 (UTC)
Module Name: src
Committed By: jmc
Date: Wed Oct 22 06:05:57 UTC 2003
Modified Files:
src/sys/netinet [netbsd-1-6]: tcp_subr.c
Log Message:
Pullup rev 1.144 (requested by he in ticket #1530)
Introduce a new INVOKING status for callouts, and use it to close
a race condition in the TCP code. Fixes PR#20390.
To generate a diff of this commit:
cvs rdiff -r1.127.4.3 -r1.127.4.4 src/sys/netinet/tcp_subr.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: James Chacon <jmc@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: pr/20390 CVS commit: [netbsd-1-6] src/sys/netinet
Date: Wed, 22 Oct 2003 06:06:13 +0000 (UTC)
Module Name: src
Committed By: jmc
Date: Wed Oct 22 06:06:13 UTC 2003
Modified Files:
src/sys/netinet [netbsd-1-6]: tcp_var.h
Log Message:
Pullup rev 1.03 (requested by he in ticket #1530)
Introduce a new INVOKING status for callouts, and use it to close
a race condition in the TCP code. Fixes PR#20390.
To generate a diff of this commit:
cvs rdiff -r1.90.4.1 -r1.90.4.2 src/sys/netinet/tcp_var.h
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: James Chacon <jmc@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: pr/20390 CVS commit: [netbsd-1-6] src/usr.bin/netstat
Date: Wed, 22 Oct 2003 06:06:23 +0000 (UTC)
Module Name: src
Committed By: jmc
Date: Wed Oct 22 06:06:23 UTC 2003
Modified Files:
src/usr.bin/netstat [netbsd-1-6]: inet.c
Log Message:
Pullup rev 1.57 (requested by he in ticket #1530)
Introduce a new INVOKING status for callouts, and use it to close
a race condition in the TCP code. Fixes PR#20390.
To generate a diff of this commit:
cvs rdiff -r1.51.2.1 -r1.51.2.2 src/usr.bin/netstat/inet.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: James Chacon <jmc@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: pr/20390 CVS commit: [netbsd-1-6] src
Date: Wed, 22 Oct 2003 06:06:50 +0000 (UTC)
Module Name: src
Committed By: jmc
Date: Wed Oct 22 06:06:50 UTC 2003
Modified Files:
src/share/man/man9 [netbsd-1-6]: callout.9
src/sys/kern [netbsd-1-6]: kern_clock.c
src/sys/sys [netbsd-1-6]: callout.h
Log Message:
Pullup via patch (requested by he in ticket #1530)
Introduce a new INVOKING status for callouts, and use it to close
a race condition in the TCP code. Fixes PR#20390.
To generate a diff of this commit:
cvs rdiff -r1.6 -r1.6.2.1 src/share/man/man9/callout.9
cvs rdiff -r1.79 -r1.79.6.1 src/sys/kern/kern_clock.c
cvs rdiff -r1.16 -r1.16.12.1 src/sys/sys/callout.h
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->feedback
State-Changed-By: mycroft
State-Changed-When: Thu Jul 8 20:28:45 UTC 2004
State-Changed-Why:
Someone should update the status of this PR. IF more work needs to be done,
that should be mentioned in the PR.
State-Changed-From-To: feedback->closed
State-Changed-By: elad@netbsd.org
State-Changed-When: Fri, 06 Oct 2006 22:28:55 +0000
State-Changed-Why:
fixed, feedback timeout
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.