NetBSD Problem Report #38927
From gcw@primenet.com.au Tue Jun 10 04:02:51 2008
Return-Path: <gcw@primenet.com.au>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by narn.NetBSD.org (Postfix) with ESMTP id 8DFE563B8BC
for <gnats-bugs@gnats.NetBSD.org>; Tue, 10 Jun 2008 04:02:51 +0000 (UTC)
Message-Id: <20080610040246.809.qmail@g.primenet.com.au>
Date: 10 Jun 2008 14:02:46 +1000
From: gcw@primenet.com.au
Reply-To: gcw@primenet.com.au
To: gnats-bugs@gnats.NetBSD.org
Subject: processes getting stuck in uvm_map (cv_timedwait), hanging machine
X-Send-Pr-Version: 3.95
>Number: 38927
>Category: kern
>Synopsis: processes getting stuck in uvm_map (cv_timedwait)
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: kern-bug-people
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Tue Jun 10 04:05:00 +0000 2008
>Closed-Date: Sun Dec 12 01:06:27 +0000 2010
>Last-Modified: Sun Dec 12 01:06:27 +0000 2010
>Originator: Geoff C. Wing
>Release: NetBSD 4.99.64
>Organization:
>Environment:
System: NetBSD g.primenet.com.au 4.99.64 NetBSD 4.99.64 (G) #0: Tue Jun 10 10:32:35 EST 2008 gcw@g.primenet.com.au:/usr/netbsd/src/sys/arch/i386/compile/G i386
Architecture: i386
Machine: i386
>Description:
On an MP i386 (Core2Duo), a heavily loaded machine will
quickly have multiple processes getting stuck in uvm_map.
This has been happening around a month for me.
Running /etc/security always does it for me with "find" getting
stuck (first?) then most other things waiting.
A backtrace of "find" on the running system:
#0 0xc02c789e in mi_switch (l=0xd142bc40) at ../../../../kern/kern_synch.c:745
#1 0xc02c408d in sleepq_block (timo=100, catch=false) at ../../../../kern/kern_sleepq.c:254
#2 0xc02a274a in cv_timedwait (cv=0xc05befb4, mtx=0xc05befb0, timo=100) at ../../../../kern/kern_condvar.c:243
#3 0xc02694de in uvm_map_prepare (map=0xc05befa0, start=3221225472, size=131072, uobj=0x0, uoffset=-1, align=131072, flags=25171751, args=0xd12cf7c0)
at ../../../../uvm/uvm_map.c:1185
#4 0xc026d4c6 in uvm_map (map=0xc05befa0, startp=0xd12cf824, size=131072, uobj=0x0, uoffset=-1, align=131072, flags=25171751)
at ../../../../uvm/uvm_map.c:1084
#5 0xc0264a64 in km_vacache_alloc (pp=0xc05bf068, flags=1) at ../../../../uvm/uvm_km.c:186
#6 0xc02de6b6 in pool_grow (pp=0xc05bf068, flags=1) at ../../../../kern/subr_pool.c:2816
#7 0xc02de004 in pool_get (pp=0xc05bf068, flags=1) at ../../../../kern/subr_pool.c:1072
#8 0xc0265483 in uvm_km_alloc_poolpage_cache (map=0xc05befa0, waitok=true) at ../../../../uvm/uvm_km.c:695
#9 0xc02de6b6 in pool_grow (pp=0xcc045000, flags=1) at ../../../../kern/subr_pool.c:2816
#10 0xc02de004 in pool_get (pp=0xcc045000, flags=1) at ../../../../kern/subr_pool.c:1072
#11 0xc02e0426 in pool_cache_get_slow (cc=<value optimized out>, s=0xd12cf9a8, objectp=0xd12cf9ac, pap=0x0, flags=1) at ../../../../kern/subr_pool.c:2456
#12 0xc02e0c23 in pool_cache_get_paddr (pc=0xcc045000, flags=1, pap=0x0) at ../../../../kern/subr_pool.c:2538
#13 0xc0315f53 in cache_enter (dvp=0xffbfe3a4, vp=0xd10112ec, cnp=0xd12cfc28) at ../../../../kern/vfs_cache.c:585
#14 0xc025546f in ufs_lookup (v=0xd12cfae4) at ../../../../ufs/ufs/ufs_lookup.c:637
#15 0xc0328414 in VOP_LOOKUP (dvp=0xffbfe3a4, vpp=0xd12cfc14, cnp=0xd12cfc28) at ../../../../kern/vnode_if.c:131
#16 0xc0318364 in lookup (ndp=0xd12cfc00) at ../../../../kern/vfs_lookup.c:696
#17 0xc0318b2f in namei (ndp=0xd12cfc00) at ../../../../kern/vfs_lookup.c:332
#18 0xc0320768 in do_sys_stat (path=0xbb92be8c <Error reading address 0xbb92be8c: Bad address>, nd_flags=0, sb=0xd12cfc70)
at ../../../../kern/vfs_syscalls.c:2443
#19 0xc03207b9 in sys___lstat30 (l=0xd142bc40, uap=0xd12cfd00, retval=0xd12cfd28) at ../../../../kern/vfs_syscalls.c:2485
#20 0xc0374c32 in syscall (frame=0xd12cfd48) at ../../../../arch/i386/i386/syscall.c:102
#21 0xc010055d in syscall1 ()
Similarly, a bt of one of my shells:
#0 0xc02c789e in mi_switch (l=0xd142b060) at ../../../../kern/kern_synch.c:745
#1 0xc02c408d in sleepq_block (timo=100, catch=false) at ../../../../kern/kern_sleepq.c:254
#2 0xc02a274a in cv_timedwait (cv=0xc05befb4, mtx=0xc05befb0, timo=100) at ../../../../kern/kern_condvar.c:243
#3 0xc02694de in uvm_map_prepare (map=0xc05befa0, start=3221225472, size=131072, uobj=0x0, uoffset=-1, align=131072, flags=25171751, args=0xd144f980)
at ../../../../uvm/uvm_map.c:1185
#4 0xc026d4c6 in uvm_map (map=0xc05befa0, startp=0xd144f9e4, size=131072, uobj=0x0, uoffset=-1, align=131072, flags=25171751)
at ../../../../uvm/uvm_map.c:1084
#5 0xc0264a64 in km_vacache_alloc (pp=0xc05bf068, flags=1) at ../../../../uvm/uvm_km.c:186
#6 0xc02de6b6 in pool_grow (pp=0xc05bf068, flags=1) at ../../../../kern/subr_pool.c:2816
#7 0xc02de004 in pool_get (pp=0xc05bf068, flags=1) at ../../../../kern/subr_pool.c:1072
#8 0xc0265483 in uvm_km_alloc_poolpage_cache (map=0xc05befa0, waitok=true) at ../../../../uvm/uvm_km.c:695
#9 0xc02de6b6 in pool_grow (pp=0xc05c5180, flags=1) at ../../../../kern/subr_pool.c:2816
#10 0xc02de004 in pool_get (pp=0xc05c5180, flags=1) at ../../../../kern/subr_pool.c:1072
#11 0xc02e0426 in pool_cache_get_slow (cc=<value optimized out>, s=0xd144fb68, objectp=0xd144fb6c, pap=0x0, flags=1) at ../../../../kern/subr_pool.c:2456
#12 0xc02e0c23 in pool_cache_get_paddr (pc=0xc05c5180, flags=1, pap=0x0) at ../../../../kern/subr_pool.c:2538
#13 0xc035f150 in pmap_create () at ../../../../arch/x86/x86/pmap.c:2169
#14 0xc0267ac5 in uvmspace_init (vm=0xd0fcd25c, pmap=0x0, vmin=0, vmax=3217031168) at ../../../../uvm/uvm_map.c:3952
#15 0xc0267af7 in uvmspace_alloc (vmin=0, vmax=3217031168) at ../../../../uvm/uvm_map.c:3927
#16 0xc026ae74 in uvmspace_fork (vm1=0xd11167d8) at ../../../../uvm/uvm_map.c:4169
#17 0xc0263da1 in uvm_proc_fork (p1=0xd142fa4c, p2=0xd4650018, shared=false) at ../../../../uvm/uvm_glue.c:208
#18 0xc02ac227 in fork1 (l1=0xd142b060, flags=<value optimized out>, exitsig=20, stack=0x0, stacksize=0, func=0, arg=0x0, retval=0xd144fd28, rnewprocp=0x0)
at ../../../../kern/kern_fork.c:426
#19 0xc02ac881 in sys_fork (l=0xd142b060, v=0xd144fd00, retval=0xd144fd28) at ../../../../kern/kern_fork.c:110
#20 0xc0374c32 in syscall (frame=0xd144fd48) at ../../../../arch/i386/i386/syscall.c:102
#21 0xc010055d in syscall1 ()
Which, if any, of the structures here is useful to examine?
>How-To-Repeat:
.
>Fix:
?
>Release-Note:
>Audit-Trail:
From: Andrew Doran <ad@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/38927: processes getting stuck in uvm_map (cv_timedwait), hanging machine
Date: Tue, 10 Jun 2008 12:31:26 +0100
On Tue, Jun 10, 2008 at 04:05:00AM +0000, gcw@primenet.com.au wrote:
> Which, if any, of the structures here is useful to examine?
A good start would be to "set print pretty on" and print the out the uvm_map
structure being passed to one of the uvm_map() functions.
Thanks,
Andrew
From: Geoff Wing <gcw@pobox.com>
To: gnats-bugs@NetBSD.org
Cc: ad@netbsd.org
Subject: Re: kern/38927: processes getting stuck in uvm_map (cv_timedwait),
hanging machine
Date: Wed, 11 Jun 2008 16:16:35 +1000
On Tuesday 2008-06-10 11:35 +0000, Andrew Doran output:
: A good start would be to "set print pretty on" and print the out the uvm_map
: structure being passed to one of the uvm_map() functions.
OK, doesn't mean much to me :-)
BTW, machine has 2GB memory (I chucked on the dmesg at the bottom in case any
of it may be relevant)
Regards,
Geoff
$2 = {
pmap = 0xc0603ea0,
lock = {
rw_owner = 8
},
busy = 0x0,
mutex = {
u = {
mtxa_owner = 4
}
},
misc_lock = {
u = {
mtxa_owner = 4
}
},
cv = {
cv_opaque = {0xd135e520, 0xd135e5e0},
cv_wmesg = 0xc04ff94c "vm_map"
},
flags = 385,
rbhead = {
rbh_root = 0xd13deb14
},
header = {
rb_entry = {
rbe_left = 0x0,
rbe_right = 0x0,
rbe_parent = 0x0,
rbe_color = 0
},
ownspace = 0,
space = 0,
prev = 0xd1cdb06c,
next = 0xc05bf1a0,
start = 4290772992,
end = 3221225472,
object = {
uvm_obj = 0x0,
sub_map = 0x0
},
offset = 0,
etype = 0,
protection = 0,
max_protection = 0,
inheritance = 0,
wired_count = 0,
aref = {
ar_pageoff = 0,
ar_amap = 0x0
},
advice = 0,
flags = 0 '\0'
},
nentries = 3220,
size = 999698432,
ref_count = 1,
hint = 0xc05bf1a0,
first_free = 0xcc03f014,
timestamp = 34370
}
------------------------------------------------------------
NetBSD 4.99.64 (G) #0: Tue Jun 10 10:32:35 EST 2008
gcw@g.primenet.com.au:/usr/netbsd/src/sys/arch/i386/compile/G
total memory = 2047 MB
avail memory = 2003 MB
timecounter: Timecounters tick every 10.000 msec
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
System manufacturer P5K3 Deluxe (System Version)
mainbus0 (root)
cpu0 at mainbus0 apid 0: Intel 686-class, 3005MHz, id 0x6fb
cpu1 at mainbus0 apid 1: Intel 686-class, 3005MHz, id 0x6fb
ioapic0 at mainbus0 apid 2: pa 0xfec00000, version 20, 24 pins
acpi0 at mainbus0: Intel ACPICA 20080321
acpi0: X/RSDT: OemId <A_M_I_,OEMXSDT ,12000721>, AslId <MSFT,00000097>
acpi0: SCI interrupting at int 9
acpi0: fixed-feature power button present
timecounter: Timecounter "ACPI-Fast" frequency 3579545 Hz quality 1000
ACPI-Fast 24-bit timer
attimer0 at acpi0 (TMR, PNP0100): AT Timer
attimer0: io 0x40-0x43 irq 0
pcppi0 at acpi0 (SPKR, PNP0800)
pcppi0: io 0x61
midi0 at pcppi0: PC speaker (CPU-intensive output)
spkr0 at pcppi0
sysbeep0 at pcppi0
npx0 at acpi0 (COPR, PNP0C04)
npx0: io 0xf0-0xff irq 13
npx0: reported by CPUID; using exception 16
fdc0 at acpi0 (FDC, PNP0700)
fdc0: io 0x3f0-0x3f5,0x3f7 irq 6 drq 2
fdc0: expected BUFFER, got 4
aiboost0 at acpi0 (ASOC, ATK0110-16843024)
aiboost0: ASUS AI Boost Hardware monitor
hpet0 at acpi0 (HPET, PNP0103)
hpet0: mem 0xfed00000-0xfed003ff
timecounter: Timecounter "hpet0" frequency 14318179 Hz quality 2000
com0 at acpi0 (UAR1, PNP0501-1)
com0: io 0x3f8-0x3ff irq 4
com0: ns16550a, working fifo
pckbc0 at acpi0 (PS2K, PNP0303): kbd port
pckbc0: io 0x60,0x64 irq 1
MBT4 (PNP0C32) at acpi0 not configured
acpibut0 at acpi0 (PWRB, PNP0C0C-170): ACPI Power Button
attimer0: attached to pcppi0
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard
pmsprobe: reset error 5
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: Intel 82G33/P35 Host Bridge (rev. 0x02)
agp0 at pchb0: can't find internal VGA device config space
ppb0 at pci0 dev 1 function 0: Intel 82G33 PCI Express Port (rev. 0x02)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
vga0 at pci1 dev 0 function 0: NVIDIA product 0x0402 (rev. 0xa1)
vga0: WARNING: ignoring 64-bit BAR @ 0x14
vga0: WARNING: ignoring 64-bit BAR @ 0x1c
wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation), using wskbd0
wsmux1: connecting to wsdisplay0
drm at vga0 not configured
uhci0 at pci0 dev 26 function 0: Intel 82801I USB UHCI Controller (rev. 0x02)
uhci0: interrupting at ioapic0 pin 16
usb0 at uhci0: USB revision 1.0
uhci1 at pci0 dev 26 function 1: Intel 82801I USB UHCI Controller (rev. 0x02)
uhci1: interrupting at ioapic0 pin 21
usb1 at uhci1: USB revision 1.0
uhci2 at pci0 dev 26 function 2: Intel 82801I USB UHCI Controller (rev. 0x02)
uhci2: interrupting at ioapic0 pin 18
usb2 at uhci2: USB revision 1.0
ehci0 at pci0 dev 26 function 7: Intel 82801I USB EHCI Controller (rev. 0x02)
ehci0: interrupting at ioapic0 pin 18
ehci0: EHCI version 1.0
ehci0: companion controllers, 2 ports each: uhci0 uhci1 uhci2
usb3 at ehci0: USB revision 2.0
azalia0 at pci0 dev 27 function 0: Generic High Definition Audio Controller
azalia0: interrupting at ioapic0 pin 22
azalia0: host: Intel 82801I High Definition Audio Controller (rev. 2), HDA rev. 1.0
ppb1 at pci0 dev 28 function 0: Intel 82801I PCI Express Port #1 (rev. 0x02)
pci2 at ppb1 bus 4
pci2: memory space enabled, rd/line, wr/inv ok
ppb2 at pci0 dev 28 function 4: Intel 82801I PCI Express Port #5 (rev. 0x02)
pci3 at ppb2 bus 3
pci3: i/o space, memory space enabled, rd/line, wr/inv ok
jmide0 at pci3 dev 0 function 0: JMicron Technology JMB363 SATA/PATA controller
jmide0: 1 PATA port, 2 SATA ports
jmide0: interrupting at ioapic0 pin 16
ahcisata0 at jmide0
ahcisata0: AHCI revision 1.0, 2 ports, 32 command slots, features 0xc722e000
atabus0 at ahcisata0 channel 0
atabus1 at ahcisata0 channel 1
jmide0: PCI IDE interface used
jmide0: bus-master DMA support present
jmide0: primary channel wired to native-PCI mode
jmide0: primary channel is unused
jmide0: secondary channel wired to native-PCI mode
jmide0: secondary channel is PATA
atabus2 at jmide0 channel 1
ppb3 at pci0 dev 28 function 5: Intel 82801I PCI Express Port #6 (rev. 0x02)
pci4 at ppb3 bus 2
pci4: i/o space, memory space enabled, rd/line, wr/inv ok
mskc0 at pci4 dev 0 function 0mskc0: interrupt moderation is 0 us
, Yukon-2 EC Ultra rev. A1 (0x2): ioapic0 pin 17
msk0 at mskc0 port A: Ethernet address 00:1b:fc:22:d7:43
ukphy0 at msk0 phy 0: Generic IEEE 802.3u media interface
ukphy0: OUI 0x000ac2, model 0x000b, rev. 1
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
uhci3 at pci0 dev 29 function 0: Intel 82801I USB UHCI Controller (rev. 0x02)
uhci3: interrupting at ioapic0 pin 23
usb4 at uhci3: USB revision 1.0
uhci4 at pci0 dev 29 function 1: Intel 82801I USB UHCI Controller (rev. 0x02)
uhci4: interrupting at ioapic0 pin 19
usb5 at uhci4: USB revision 1.0
uhci5 at pci0 dev 29 function 2: Intel 82801I USB UHCI Controller (rev. 0x02)
uhci5: interrupting at ioapic0 pin 18
usb6 at uhci5: USB revision 1.0
ehci1 at pci0 dev 29 function 7: Intel 82801I USB EHCI Controller (rev. 0x02)
ehci1: interrupting at ioapic0 pin 23
ehci1: EHCI version 1.0
ehci1: companion controllers, 2 ports each: uhci3 uhci4 uhci5
usb7 at ehci1: USB revision 2.0
ppb4 at pci0 dev 30 function 0: Intel 82801BA Hub-PCI Bridge (rev. 0x92)
pci5 at ppb4 bus 5
pci5: i/o space, memory space enabled
ex0 at pci5 dev 2 function 0: 3Com 3c905C-TX 10/100 Ethernet with mngmt (rev. 0x78)
ex0: interrupting at ioapic0 pin 18
ex0: MAC address 00:04:75:e5:71:c5
exphy0 at ex0 phy 24: 3Com internal media interface
exphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
Lucent Technologies FW322/323 IEEE 1394 Host Controller (Firewire serial bus, interface 0x10, revision 0x70) at pci5 dev 3 function 0 not configured
re0 at pci5 dev 4 function 0: RealTek 8169SC/8110SC Single-chip Gigabit Ethernet (rev. 0x10)
re0: interrupting at ioapic0 pin 16
re0: Ethernet address 00:1b:fc:22:d6:f9
re0: using 256 tx descriptors
rgephy0 at re0 phy 7: RTL8169S/8110S/8211 1000BASE-T media interface, rev. 2
rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
ichlpcib0 at pci0 dev 31 function 0
ichlpcib0: Intel 82801IR LPC Interface Bridge (rev. 0x02)
timecounter: Timecounter "ichlpcib0" frequency 3579545 Hz quality 1000
ichlpcib0: 24-bit timer
ichlpcib0: TCO (watchdog) timer configured.
ahcisata1 at pci0 dev 31 function 2: Intel 82801I AHCI SATA Controller with 6 ports
ahcisata1: interrupting at ioapic0 pin 22
ahcisata1: AHCI revision 1.2, 6 ports, 32 command slots, features 0xef22e0e0
atabus3 at ahcisata1 channel 0
atabus4 at ahcisata1 channel 1
atabus5 at ahcisata1 channel 2
atabus6 at ahcisata1 channel 3
atabus7 at ahcisata1 channel 4
atabus8 at ahcisata1 channel 5
ichsmb0 at pci0 dev 31 function 3: Intel 82801I SMBus Controller (rev. 0x02)
ichsmb0: interrupting at ioapic0 pin 18
iic0 at ichsmb0: I2C bus
isa0 at ichlpcib0
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
timecounter: Timecounter "TSC" frequency 3005655750 Hz quality 3000
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
fd1 at fdc0 drive 1: density unknown
azalia0: codec[0]: Analog Devices AD1988B (rev. 3.0), HDA rev. 1.0
audio0 at azalia0: full duplex, independent
uhub0 at usb0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhub1 at usb1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhub2 at usb2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhub3 at usb3: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub3: 6 ports with 6 removable, self powered
uhub4 at usb4: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub4: 2 ports with 2 removable, self powered
uhub5 at usb5: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub5: 2 ports with 2 removable, self powered
uhub6 at usb6: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub6: 2 ports with 2 removable, self powered
uhub7 at usb7: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub7: 6 ports with 6 removable, self powered
ehci1: handing over low speed device on port 4 to uhci4
ugen0 at uhub3 port 3
ugen0: Realtek RTL8187_Wireless, rev 2.00/1.00, addr 2
ugen1 at uhub5 port 2
ugen1: American Power Conversion Back-UPS RS 1500 FW:8.g9a.I USB FW:g9a, rev 1.10/1.06, addr 2
wd0 at atabus2 drive 0: <ST380021A>
wd0: drive supports 16-sector PIO transfers, LBA addressing
wd0: 76319 MB, 155061 cyl, 16 head, 63 sec, 512 bytes/sect x 156301488 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
wd0(jmide0:1:0): using PIO mode 4, Ultra-DMA mode 5 (Ultra/100) (using DMA)
boot device: wd0
root on wd0a dumps on wd0b
dump_misc_init: max_paddr = 0x7ff80000
mountroot: trying lfs...
mountroot: trying ffs...
root file system type: ffs
init: copying out path `/sbin/init' 11
mskc0: interrupt moderation is 1000 us
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)
Accounting started
From: Andrew Doran <ad@netbsd.org>
To: Geoff Wing <gcw@pobox.com>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/38927: processes getting stuck in uvm_map (cv_timedwait), hanging machine
Date: Wed, 11 Jun 2008 11:49:41 +0100
On Wed, Jun 11, 2008 at 04:16:35PM +1000, Geoff Wing wrote:
> nentries = 3220,
> size = 999698432,
> ref_count = 1,
It looks like something has exhausted the virtual address space available
to the kernel (1GB). What is the machine doing?
Andrew
From: Geoff Wing <gcw@pobox.com>
To: Andrew Doran <ad@netbsd.org>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/38927: processes getting stuck in uvm_map (cv_timedwait),
hanging machine
Date: Thu, 12 Jun 2008 20:18:56 +1000
On Wednesday 2008-06-11 11:49 +0100, Andrew Doran output:
:On Wed, Jun 11, 2008 at 04:16:35PM +1000, Geoff Wing wrote:
:> nentries = 3220,
:> size = 999698432,
:> ref_count = 1,
:It looks like something has exhausted the virtual address space available
:to the kernel (1GB). What is the machine doing?
sh /etc/security
It's doing a "find" for setuid/gid files.
If this is relevant, I have kern.maxvnodes=1048576
and "pstat -T" shows it make vnode use to around 532686
Also, this only appeared around a month ago.
Regards,
Geoff
From: Andrew Doran <ad@netbsd.org>
To: Geoff Wing <gcw@pobox.com>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/38927: processes getting stuck in uvm_map (cv_timedwait), hanging machine
Date: Thu, 12 Jun 2008 12:22:18 +0000
On Thu, Jun 12, 2008 at 08:18:56PM +1000, Geoff Wing wrote:
> On Wednesday 2008-06-11 11:49 +0100, Andrew Doran output:
> :On Wed, Jun 11, 2008 at 04:16:35PM +1000, Geoff Wing wrote:
> :> nentries = 3220,
> :> size = 999698432,
> :> ref_count = 1,
> :It looks like something has exhausted the virtual address space available
> :to the kernel (1GB). What is the machine doing?
>
> sh /etc/security
>
> It's doing a "find" for setuid/gid files.
>
> If this is relevant, I have kern.maxvnodes=1048576
> and "pstat -T" shows it make vnode use to around 532686
That's why your machine is hanging: the kernel address space is so full that
the system can no longer map anything into it. It should really do a better
job of reclaiming space but that's difficult, and perhaps for now we should
just limit the number of vnodes based on the available space.
Can you 'print uvmexp' if you have a moment?
Andrew
From: Geoff Wing <gcw@pobox.com>
To: Andrew Doran <ad@netbsd.org>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/38927: processes getting stuck in uvm_map (cv_timedwait),
hanging machine
Date: Fri, 13 Jun 2008 13:54:02 +1000
On Thursday 2008-06-12 12:25 +0000, Andrew Doran output:
: Can you 'print uvmexp' if you have a moment?
$6 = {
pagesize = 4096,
pagemask = 4095,
pageshift = 12,
npages = 513077,
free = 288505,
paging = 0,
wired = 3514,
ncolors = 64,
colormask = 63,
zeropages = 287868,
reserve_pagedaemon = 1,
reserve_kernel = 5,
anonpages = 19082,
filepages = 7320,
execpages = 3282,
freemin = 64,
freetarg = 85,
wiredmax = 171025,
nswapdev = 1,
swpages = 528383,
swpgavail = 528383,
swpginuse = 0,
swpgonly = 0,
nswget = 0,
faults = 99592,
traps = 96025,
intrs = 84409,
swtch = 202399,
softs = 215403,
syscalls = 3401135,
pageins = 0,
swapins = 0,
swapouts = 0,
pgswapin = 0,
pgswapout = 0,
forks = 661,
forks_ppwait = 162,
forks_sharevm = 162,
pga_zerohit = 37252,
pga_zeromiss = 208,
zeroaborts = 64,
colorhit = 275497,
colormiss = 0,
cpuhit = 155524,
cpumiss = 119973,
fltnoram = 0,
fltnoanon = 0,
fltpgwait = 0,
fltpgrele = 0,
fltrelck = 1599,
fltrelckok = 1598,
fltanget = 25255,
fltanretry = 0,
fltamcopy = 10424,
fltnamap = 20891,
fltnomap = 135585,
fltlget = 37061,
fltget = 1599,
flt_anon = 17680,
flt_acow = 7568,
flt_obj = 31093,
flt_prcopy = 5967,
flt_przero = 33275,
pdwoke = 0,
pdrevs = 0,
pdswout = 0,
pdfreed = 0,
pdscans = 0,
pdanscan = 0,
pdobscan = 0,
pdreact = 0,
pdbusy = 0,
pdpageouts = 0,
pdpending = 0,
pddeact = 0,
pdreanon = 0,
pdrefile = 0,
pdreexec = 0
}
Regards,
Geoff
From: Andrew Doran <ad@netbsd.org>
To: Geoff Wing <gcw@pobox.com>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/38927: processes getting stuck in uvm_map (cv_timedwait), hanging machine
Date: Mon, 16 Jun 2008 10:50:05 +0000
On Thu, Jun 12, 2008 at 08:18:56PM +1000, Geoff Wing wrote:
> On Wednesday 2008-06-11 11:49 +0100, Andrew Doran output:
> :On Wed, Jun 11, 2008 at 04:16:35PM +1000, Geoff Wing wrote:
> :> nentries = 3220,
> :> size = 999698432,
> :> ref_count = 1,
> :It looks like something has exhausted the virtual address space available
> :to the kernel (1GB). What is the machine doing?
>
> sh /etc/security
>
> It's doing a "find" for setuid/gid files.
What types of file system are in use on the machine?
Andrew
From: Andrew Doran <ad@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/38927 CVS commit: src/sys
Date: Mon, 16 Jun 2008 11:26:28 +0000 (UTC)
Module Name: src
Committed By: ad
Date: Mon Jun 16 11:26:28 UTC 2008
Modified Files:
src/sys/kern: init_main.c init_sysctl.c
src/sys/sys: param.h
Log Message:
PR kern/38927: processes getting stuck in uvm_map (cv_timedwait), hanging
machine
Assume that a vnode (and associated data structures) costs 2kB in the
worst imaginable case. Don't allow sysctl to set desiredvnodes to a
value that would use more than 75% of KVA or 75% of physical memory.
To generate a diff of this commit:
cvs rdiff -r1.358 -r1.359 src/sys/kern/init_main.c
cvs rdiff -r1.141 -r1.142 src/sys/kern/init_sysctl.c
cvs rdiff -r1.319 -r1.320 src/sys/sys/param.h
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->closed
State-Changed-By: chs@NetBSD.org
State-Changed-When: Sun, 12 Dec 2010 01:06:27 +0000
State-Changed-Why:
the code to change the limit on the number of vnodes
now has a sanity check to avoid this kind of hang.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.