NetBSD Problem Report #53155
From he@smistad.uninett.no Wed Apr 4 13:36:34 2018
Return-Path: <he@smistad.uninett.no>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 2F4A97A172
for <gnats-bugs@gnats.NetBSD.org>; Wed, 4 Apr 2018 13:36:34 +0000 (UTC)
Message-Id: <20180404133555.42D0043F11C@smistad.uninett.no>
Date: Wed, 4 Apr 2018 15:35:55 +0200 (CEST)
From: he@NetBSD.org
Reply-To: he@NetBSD.org
To: gnats-bugs@NetBSD.org
Subject: OS wedges after <12h uptime when >2 bnx network interfaces in use
X-Send-Pr-Version: 3.95
>Number: 53155
>Category: port-amd64
>Synopsis: Wedge after <12h uptime when >2 bnx network interfaces in use
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: port-amd64-maintainer
>State: feedback
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Wed Apr 04 13:40:00 +0000 2018
>Closed-Date:
>Last-Modified: Sat Jul 24 20:38:17 +0000 2021
>Originator: Havard Eidnes
>Release: NetBSD 8.0_BETA
>Organization:
I try...
>Environment:
System: NetBSD xx.xx.xxx 8.0_BETA NetBSD 8.0_BETA (GENERIC.201803261630Z) amd64
Architecture: x86_64
Machine: amd64
>Description:
This particular system appears to wedge at unpredictable times
with less than 12 hours uptime when more than two network
interfaces are configured and in use. We have tried with both
bnx and wm interfaces, and the mix doesn't appear to make any
difference.
The symptoms when it wedges are that the console (VGA with USB
keyboard, or physical serial port) is unresponsive, and the
host no longer responds to network traffic on any of its
network interfaces.
I should perhaps mention that this host has been tuned to
allow up to 1MB socket buffers, both for ipv4 and ipv6, and is
using the "cubic" congestion control algorithm, via the
following /etc/sysctl.conf entries:
kern.sbmax=1048576
net.inet6.tcp6.sendbuf_max=1048576
net.inet6.tcp6.recvbuf_max=1048576
net.inet.tcp.sendbuf_max=1048576
net.inet.tcp.recvbuf_max=1048576
net.inet.tcp.congctl.selected=cubic
and in normal use it has at least ~170 active TCP sessions,
most if not all all related to running conserver, so normally
carries very little traffic. The amount of memory allocated
to mbuf cluseters is (defaulted to):
kern.mbuf.nmbclusters = 261730
On the host I've rigged it to run "vmstat -m" every 100
seconds or so, but the last it logged didn't look strange, it
ended in
In use 92189K, total allocated 96140K; utilization 95.9%
and the mbpl and mclpl usage isn't anywhere near their
high-water marks if I read this correctly:
Memory resource pool statistics
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
...
mbpl 512 2176 0 0 274 0 274 274 2 inf 2
mclpl 2048 1953 0 0 981 0 981 981 4 130865 4
...
After I got rigged up a serial console, I managed to break
into DDB, and below you'll find an edited part of the DDB
session. What I observe is:
- Only one of the 8 CPUs is doing anything (running expect in
user-mode), but is apparently busy doing interrupt
processing. However, DDB apparently isn't able to decipher
the stack through the interrupt frames (any hints for
fixing that?)
- There is plenty of free memory available
- A number of processes have status "tstile", possibly
indicating a locking problem or contention of some sort,
but my ability to debug that sort of stuff isn't there, and
we're not good at documenting what info to collect (hint,
hint), and "tstile" doesn't really point towards the lock
which is contested, only indicates "some lock somewhere is
contested".
- My gut says this may be an interrupt storm, but none of the
event counters seems to back that up
I beleive this problem is the same one which was reported
against NetBSD/amd64 6.0 in PR#47229, in fact it probably is
the exact same host system.
Here's the edited DDB session (with "greater-than" replaced
with ":" in order to not confuse gnats):
:DDB break sequence:
fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0xffff80009db2
3000 ilevel 0x8 rsp 0xffff80008f69bcf8
curlwp 0xfffffe822f6fd440 pid 0.6 lowest kstack 0xffff80008f6972c0
Stopped in pid 0.6 (system) at netbsd:breakpoint+0x5: leave
db{0}: trace
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
DDB lost frame for netbsd:Xdoreti, trying 0xffff80008f69bd70
Xdoreti() at netbsd:Xdoreti
--- interrupt ---
Bad frame pointer: 0xffff800000000000
206:
db{0}: show uvmexp
Current UVM status:
pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12, ncolors=8
2028595 VM pages: 1037330 active, 0 inactive, 1805 wired, 923564 free
pages 54144 anon, 979303 file, 5688 exec
freemin=2048, free-target=2730, wired-max=676198
resv-pg=1, resv-kernel=40, zeropages=820585
cpu0:
faults=5107500, traps=2516760, intrs=5678897, ctxswitch=3206198
softint=6171203, syscalls=10897655
cpu1:
faults=601410, traps=562379, intrs=0, ctxswitch=407170
softint=64702, syscalls=2203289
cpu2:
faults=607483, traps=592645, intrs=0, ctxswitch=357226
softint=127481, syscalls=1962203
cpu3:
faults=585411, traps=548343, intrs=0, ctxswitch=348991
softint=88312, syscalls=2002413
cpu4:
faults=843615, traps=531147, intrs=0, ctxswitch=538383
softint=104299, syscalls=1896227
cpu5:
faults=909578, traps=884707, intrs=0, ctxswitch=2440981
softint=103484, syscalls=8171151
cpu6:
faults=700436, traps=693435, intrs=0, ctxswitch=1017896
softint=139013, syscalls=4260226
cpu7:
faults=804685, traps=791518, intrs=0, ctxswitch=877603
softint=156355, syscalls=3733242
fault counts:
noram=0, noanon=0, pgwait=0, pgrele=0
ok relocks(total)=21248(21250), anget(retrys)=4091394(0), amapcopy=457812
neighbor anon/obj pg=406830/8542654, gets(lock/unlock)=3654322/21250
cases: anon=3893186, anoncow=198224, obj=2917328, prcopy=737081, przero=1448638
daemon and swap counts:
woke=0, revs=0, scans=0, obscans=0, anscans=0
busy=0, freed=0, reactivate=0, deactivate=0
pageouts=0, pending=0, nswget=0
nswapdev=1, swpgavail=32767
swpages=32767, swpginuse=0, swpgonly=0, paging=0
db{0}: machine cpu 0
using CPU 0
db{0}: trace
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
DDB lost frame for netbsd:Xdoreti, trying 0xffff80008f69bd70
Xdoreti() at netbsd:Xdoreti
--- interrupt ---
Bad frame pointer: 0xffff800000000000
206:
db{0}: machine cpu 1
using CPU 1
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: machine cpu 2
using CPU 2
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: machine cpu 3
using CPU 3
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: machine cpu 4
using CPU 4
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: machine cpu 5
using CPU 5
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: machine cpu 6
using CPU 6
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: machine cpu 7
using CPU 7
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: help show
db{0}: show event
evcnt type 0: bus_dma bounces = 8814235
evcnt type 0: bus_dma loads = 9314511
evcnt type 0: bus_dma nbouncebufs = 3211
evcnt type 0: vmcmd kills = 12944
evcnt type 0: vmcmd extends = 12219
evcnt type 0: vmcmd calls = 138643
evcnt type 0: vmem static_bt_inuse = 200
evcnt type 0: vmem static_bt_count = 200
evcnt type 0: rndpseudo open soft = 4559
evcnt type 1: TLB shootdown = 2192282
evcnt type 0: pserialize exclusive access = 18952
evcnt type 0: cpu0 runqueue pull = 34799
evcnt type 0: cpu0 runqueue push = 1042
evcnt type 0: cpu0 runqueue stay = 1725421
evcnt type 0: cpu0 runqueue localize = 8211
evcnt type 0: softint net/0 = 5191013
evcnt type 0: softint net block/0 = 1497
evcnt type 0: softint bio/0 = 487407
evcnt type 0: softint bio block/0 = 2270
evcnt type 0: softint clk/0 = 427502
evcnt type 0: softint clk block/0 = 12
evcnt type 0: softint ser/0 = 65281
evcnt type 0: callout late/0 = 13
evcnt type 0: callout wait/0 = 1
evcnt type 0: crosscall unicast = 1560
evcnt type 0: crosscall broadcast = 17
evcnt type 0: namecache entries collected = 10484
evcnt type 0: namecache under scan target = 12858
evcnt type 1: cpu0 timer = 1285733
evcnt type 0: cpu0 generic IPI = 437
evcnt type 0: cpu0 FPU synch IPI = 364
evcnt type 0: cpu0 xcall IPI = 26520
evcnt type 0: cpu0 kpreempt IPI = 61
evcnt type 0: cpu1 runqueue pull = 4176
evcnt type 0: cpu1 runqueue push = 1154
evcnt type 0: cpu1 runqueue stay = 217239
evcnt type 0: cpu1 runqueue localize = 188
evcnt type 0: softint net/1 = 2
evcnt type 0: softint net block/1 = 1
evcnt type 0: softint clk/1 = 27146
evcnt type 0: softint ser/1 = 37554
evcnt type 0: softint ser block/1 = 1
evcnt type 0: cpu1 timer = 1285906
evcnt type 0: cpu1 FPU synch IPI = 325
evcnt type 0: cpu1 xcall IPI = 34579
evcnt type 0: cpu2 runqueue pull = 3811
evcnt type 0: cpu2 runqueue push = 968
evcnt type 0: cpu2 runqueue stay = 193225
evcnt type 0: cpu2 runqueue localize = 300
evcnt type 0: softint clk/2 = 90104
evcnt type 0: softint ser/2 = 37377
evcnt type 0: cpu2 timer = 1285894
evcnt type 0: cpu2 FPU synch IPI = 291
evcnt type 0: cpu2 xcall IPI = 33748
evcnt type 0: cpu3 runqueue pull = 4132
evcnt type 0: cpu3 runqueue push = 1000
evcnt type 0: cpu3 runqueue stay = 188850
evcnt type 0: cpu3 runqueue localize = 487
evcnt type 0: softint net/3 = 1
evcnt type 0: softint clk/3 = 50860
evcnt type 0: softint ser/3 = 37451
evcnt type 0: cpu3 timer = 1285909
evcnt type 0: cpu3 FPU synch IPI = 276
evcnt type 0: cpu3 xcall IPI = 34139
evcnt type 0: cpu4 runqueue pull = 22245
evcnt type 0: cpu4 runqueue push = 1481
evcnt type 0: cpu4 runqueue stay = 281034
evcnt type 0: cpu4 runqueue localize = 298
evcnt type 0: softint clk/4 = 66905
evcnt type 0: softint ser/4 = 37394
evcnt type 0: cpu4 timer = 1285902
evcnt type 0: cpu4 FPU synch IPI = 290
evcnt type 0: cpu4 xcall IPI = 34811
evcnt type 0: cpu4 kpreempt IPI = 1
evcnt type 0: cpu5 runqueue pull = 7977
evcnt type 0: cpu5 runqueue push = 1309
evcnt type 0: cpu5 runqueue stay = 1249669
evcnt type 0: cpu5 runqueue localize = 1286
evcnt type 0: softint net/5 = 1
evcnt type 0: softint clk/5 = 65774
evcnt type 0: softint clk block/5 = 1
evcnt type 0: softint ser/5 = 37709
evcnt type 0: softint ser block/5 = 1
evcnt type 0: cpu5 timer = 1285895
evcnt type 0: cpu5 FPU synch IPI = 485
evcnt type 0: cpu5 xcall IPI = 31308
evcnt type 0: cpu6 runqueue pull = 5535
evcnt type 0: cpu6 runqueue push = 1795
evcnt type 0: cpu6 runqueue stay = 536359
evcnt type 0: cpu6 runqueue localize = 575
evcnt type 0: softint clk/6 = 101534
evcnt type 0: softint ser/6 = 37479
evcnt type 0: cpu6 timer = 1285900
evcnt type 0: cpu6 FPU synch IPI = 357
evcnt type 0: cpu6 xcall IPI = 34247
evcnt type 0: cpu7 runqueue pull = 7441
evcnt type 0: cpu7 runqueue push = 3442
evcnt type 0: cpu7 runqueue stay = 469377
evcnt type 0: cpu7 runqueue localize = 727
evcnt type 0: softint clk/7 = 118815
evcnt type 0: softint clk block/7 = 3
evcnt type 0: softint ser/7 = 37540
evcnt type 0: callout late/7 = 2
evcnt type 0: cpu7 timer = 1285900
evcnt type 0: cpu7 FPU synch IPI = 436
evcnt type 0: cpu7 xcall IPI = 31790
evcnt type 1: ioapic1 pin 4 = 5153711
evcnt type 1: ioapic1 pin 0 = 29577
evcnt type 1: ioapic1 pin 10 = 7447
evcnt type 1: ioapic0 pin 17 = 20
evcnt type 1: ioapic0 pin 19 = 21
evcnt type 1: ioapic0 pin 16 = 487731
evcnt type 1: ioapic0 pin 21 = 29
evcnt type 1: ioapic0 pin 23 = 12
evcnt type 1: ioapic0 pin 4 = 349
evcnt type 0: acpicpu0 C1 (HALT) = 6993829
evcnt type 0: acpicpu1 C1 (HALT) = 3158420
evcnt type 0: acpicpu2 C1 (HALT) = 3155060
evcnt type 0: acpicpu3 C1 (HALT) = 3049008
evcnt type 0: acpicpu4 C1 (HALT) = 3255477
evcnt type 0: acpicpu5 C1 (HALT) = 3851591
evcnt type 0: acpicpu6 C1 (HALT) = 3459820
evcnt type 0: acpicpu7 C1 (HALT) = 3405818
evcnt type 0: kpreempt defer: critical section = 48
evcnt type 0: kpreempt defer: kernel_lock = 60
evcnt type 0: kpreempt immediate = 3524
db{0}: ps
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
20105 1 3 1 0 fffffe8224d7d780 cron tstile
26131 1 3 1 0 fffffe822281d620 cron wait
23319 1 3 1 0 fffffe8224850b60 cron tstile
25811 1 3 7 0 fffffe82036d0740 cron tstile
22533 1 3 7 0 fffffe820a1115c0 cron wait
24189 1 3 1 0 fffffe8221441880 cron wait
26325 1 3 5 0 fffffe8226e1e900 cron tstile
18295 1 3 1 0 fffffe8229822360 cron tstile
24699 1 3 6 0 fffffe822571b100 cron tstile
20160 1 3 7 0 fffffe821430ea20 cron tstile
24819 1 3 5 0 fffffe821e31f2a0 cron wait
18159 1 3 6 0 fffffe81fdf214a0 cron wait
20826 1 3 7 0 fffffe8215beeaa0 cron wait
19785 1 3 1 0 fffffe822a5968c0 cron wait
19192 1 3 1 0 fffffe822da91540 cron tstile
24053 1 3 7 0 fffffe822d9c9140 cron tstile
22204 1 3 7 0 fffffe8224bf4000 cron wait
19906 1 3 1 0 fffffe82260d4760 cron wait
25501 1 3 4 0 fffffe821e715260 bash tstile
16391 1 3 1 0 fffffe822b3fb8a0 cron tstile
22719 1 3 1 0 fffffe8215236740 cron wait
27003 1 3 2 80 fffffe820a90e6e0 pickup kqueue
25066 1 3 5 0 fffffe8212edf620 ssh xchicv
21812 2 3 5 80 fffffe822a960740 expect select
21812 1 3 4 80 fffffe8226255060 expect wait
22983 1 3 5 80 fffffe8211b16780 sh wait
24598 1 3 5 80 fffffe821ed7e200 perl wait
23813 1 3 5 80 fffffe8209d819a0 sh wait
20606 1 3 0 80 fffffe821430e600 ssh select
14822 1 3 6 80 fffffe821f1f05c0 sh wait
29431 1 3 0 80 fffffe815c71a4c0 ssh select
25005 1 3 4 80 fffffe82140376c0 sh wait
23260 1 3 3 0 fffffe8212bd1520 ssh tstile
22933 1 3 4 80 fffffe82142275a0 sh wait
23169 1 3 0 80 fffffe821431d120 ssh select
20146 2 3 6 0 fffffe820f2f5b20 expect tstile
20146 1 3 6 0 fffffe822b7e3ae0 expect fdclose
27429 1 3 7 80 fffffe82200e3860 sh wait
23732 1 3 7 80 fffffe82257e8120 perl pipe_rd
19502 1 3 7 80 fffffe82107e9020 sh wait
23886 1 3 5 80 fffffe820f982220 ssh select
20022 2 3 4 0 fffffe821e467a00 expect tstile
20022 1 3 5 0 fffffe8210e401e0 expect fdclose
20799 1 3 0 80 fffffe82114f39c0 sh wait
21173 1 3 5 80 fffffe82036d0b60 perl wait
21675 1 3 6 80 fffffe82284e8260 sh wait
18171 1 3 0 80 fffffe8209d81160 ssh select
19295 2 3 4 80 fffffe82107e9860 expect select
19295: 1 7 0 0 fffffe82288daa80 expect
21354 1 3 5 80 fffffe8206f1a260 sh wait
26421 1 3 5 80 fffffe820e5634c0 perl pipe_rd
24533 1 3 5 80 fffffe82273078e0 sh wait
22388 1 3 0 80 fffffe8225d0d760 ssh select
20900 2 3 7 0 fffffe822a596080 expect tstile
20900 1 3 7 0 fffffe8212a4d080 expect fdclose
20540 1 3 0 80 fffffe822a3a9500 sh wait
22103 1 3 7 80 fffffe822571b940 perl pipe_rd
24457 1 3 5 80 fffffe8222326aa0 sh wait
23314 1 3 0 80 fffffe8215f8d840 ssh select
23095 1 3 0 80 fffffe82200e3020 sh wait
18806 1 3 0 80 fffffe821f97e980 ssh select
17372 1 3 4 80 fffffe820e5638e0 sh wait
23989 1 3 0 80 fffffe82201ad520 ssh select
23534 1 3 3 80 fffffe820d6e9440 sh wait
19407 1 3 2 80 fffffe821eb7e0e0 ssh select
25841 1 3 6 80 fffffe8222107a80 sh wait
26243 1 3 0 80 fffffe820fe60720 ssh select
17183 1 3 6 80 fffffe820acaa0e0 sh wait
24185 1 3 5 80 fffffe822a3a90e0 ssh select
19244 1 3 4 80 fffffe821dc670c0 sh wait
18735 1 3 0 80 fffffe822acb2460 ssh select
21395 1 3 6 80 fffffe8210521340 sh wait
26341 1 3 0 80 fffffe822604b700 ssh select
20515 1 3 0 80 fffffe822a5964a0 sh wait
24911 1 3 0 80 fffffe820de6a240 ssh select
19983 1 3 1 80 fffffe822da9c580 sh wait
18877 1 3 0 80 fffffe822281d200 ssh select
23537 1 3 7 80 fffffe82273074c0 sh wait
21663 1 3 0 80 fffffe821fad9420 ssh select
25421 1 3 1 80 fffffe82218a3ac0 sh wait
24440 1 3 0 80 fffffe8215f8d420 ssh select
26449 1 3 0 80 fffffe82253a8b40 sh wait
22382 1 3 0 80 fffffe820b0600c0 ssh select
21078 1 3 1 80 fffffe8166096900 sh wait
23069 1 3 5 80 fffffe821c068100 ssh select
20686 1 3 5 80 fffffe8220479080 sh wait
19346 1 3 0 80 fffffe820f2f5700 ssh select
21254 1 3 6 80 fffffe8212a4d8c0 sh wait
23663 1 3 0 80 fffffe81fdf21080 ssh select
22711 1 3 1 80 fffffe816482f960 sh wait
22549 1 3 0 80 fffffe821eb7e500 ssh select
12635 1 3 7 80 fffffe821de42120 sh wait
22248 1 3 0 80 fffffe822a3a9920 ssh select
16468 1 3 4 80 fffffe8228cb76e0 sh wait
17532 1 3 0 80 fffffe82111a0280 ssh select
24374 1 3 0 80 fffffe821f05d240 sh wait
23230 1 3 0 80 fffffe822b7e36c0 ssh select
18702 1 3 0 80 fffffe820a19ba80 sh wait
23591 1 3 0 80 fffffe822b6b6200 ssh select
24665 1 3 7 80 fffffe820de6a660 sh wait
20189 1 3 0 80 fffffe822a025a60 ssh select
20252 1 3 0 80 fffffe822d9c9980 ssh select
23080 1 3 0 80 fffffe82172c4b80 sh wait
20740 1 3 7 80 fffffe820a19b660 sh wait
24958 1 3 0 80 fffffe8210e40600 ssh select
22951 1 3 5 80 fffffe82204794a0 sh wait
21698 1 3 0 80 fffffe82143bbba0 ssh select
17032 1 3 2 80 fffffe821fad9840 sh wait
24314 1 3 0 80 fffffe821eb7e920 ssh select
20886 1 3 7 80 fffffe820e5630a0 sh wait
22406 1 3 0 80 fffffe821d814140 ssh select
23192 1 3 2 80 fffffe8215475420 sh wait
17303 1 3 0 80 fffffe822a960320 ssh select
24518 1 3 7 80 fffffe820e350b40 sh wait
21932 1 3 0 80 fffffe821227e040 ssh select
22174 1 3 6 80 fffffe822b7e32a0 sh wait
20504 1 3 0 80 fffffe821d855720 ssh select
19436 1 3 7 80 fffffe820ef105e0 sh wait
23481 1 3 0 80 fffffe820d6e9860 ssh select
22354 1 3 0 80 fffffe821f05d660 ssh select
21891 1 3 3 80 fffffe8159f125a0 sh wait
17000 1 3 6 80 fffffe8229822780 sh wait
21472 1 3 0 80 fffffe8221af7ac0 ssh select
24341 1 3 7 80 fffffe8212536220 sh wait
24131 1 3 0 80 fffffe821ed7e620 ssh select
22981 1 3 0 80 fffffe816482f540 ssh select
15043 1 3 4 80 fffffe820f982640 sh wait
21177 1 3 0 80 fffffe815dcc1160 sh wait
21503 1 3 0 80 fffffe820c092060 ssh select
25824 1 3 1 80 fffffe821e40d880 sh wait
17304 1 3 0 80 fffffe8225b3a540 ssh select
19360 1 3 6 80 fffffe820f2f52e0 sh wait
16456 1 3 5 80 fffffe821548d2e0 par pause
16577 1 3 0 80 fffffe8166b46500 sh wait
21790 1 3 0 80 fffffe82200e3440 sh wait
18068 1 3 1 80 fffffe821431d540 sh wait
22385 1 3 1 80 fffffe821d855300 sh wait
17963 1 3 1 80 fffffe81660964e0 cron pipe_rd
5078 1 3 0 80 fffffe820c092480 ssh select
7768 1 3 3 80 fffffe8164637140 sh wait
7239 1 3 0 80 fffffe8212edf200 ssh select
10982 1 3 5 80 fffffe8210521b80 sh wait
5126 1 3 0 80 fffffe820f982a60 ssh select
8692 1 3 2 80 fffffe821c1f80a0 sh wait
9318 1 3 0 80 fffffe82098348a0 ssh select
9227 1 3 2 80 fffffe821f97e140 sh wait
10068 1 3 0 80 fffffe821dfe99a0 ssh select
8697 1 3 0 80 fffffe821e69f5c0 sh wait
10511 1 3 0 80 fffffe82036d0320 ssh select
11284 1 3 2 80 fffffe82201ad100 sh wait
9755 1 3 0 80 fffffe82289472a0 ssh select
9929 1 3 5 80 fffffe821fad9000 sh wait
4650 1 3 0 80 fffffe820de6aa80 ssh select
9232 1 3 1 80 fffffe8215475000 sh wait
11976 1 3 0 80 fffffe820acaa920 ssh select
8477 1 3 4 80 fffffe815de49520 sh wait
5558 1 3 0 80 fffffe820d891420 ssh select
3911 1 3 3 80 fffffe82111a0ac0 sh wait
9329 1 3 0 80 fffffe815de49100 ssh select
9759 1 3 6 80 fffffe8221af76a0 sh wait
11080 1 3 0 80 fffffe82172c4760 ssh select
12525 1 3 5 80 fffffe820b060900 sh wait
10080 1 3 0 80 fffffe8209834480 ssh select
12878 1 3 7 80 fffffe822b3fb480 sh wait
14096 1 3 0 80 fffffe82260d4340 ssh select
9034 1 3 7 80 fffffe8221e7ba60 sh wait
8694 1 3 0 80 fffffe8164637560 ssh select
4956 1 3 7 80 fffffe8226e1e0c0 sh wait
8298 1 3 0 80 fffffe8215bee680 ssh select
8048 1 3 7 80 fffffe820e8582c0 sh wait
6948 1 3 0 80 fffffe8214227180 ssh select
10692 1 3 7 80 fffffe821df299c0 sh wait
3628 1 3 0 80 fffffe821227e460 ssh select
4171 1 3 6 80 fffffe822da91960 sh wait
11767 1 3 0 80 fffffe8215236b60 ssh select
11095 1 3 0 80 fffffe82172c4340 ssh select
7282 1 3 5 80 fffffe822995e1e0 sh wait
9931 1 3 6 80 fffffe82204798c0 sh wait
11290 1 3 0 80 fffffe821b623ac0 ssh select
13455 1 3 0 80 fffffe8215236320 ssh select
12282 1 3 6 80 fffffe822acb2880 sh wait
9284 1 3 7 80 fffffe821ba38640 sh wait
7897 1 3 0 80 fffffe821e31fae0 ssh select
10861 1 3 0 80 fffffe8206f1aaa0 ssh select
9946 1 3 6 80 fffffe8224850320 sh wait
4820 1 3 7 80 fffffe821e9dc880 sh wait
8676 1 3 0 80 fffffe8228cb7b00 ssh select
11199 1 3 6 80 fffffe821e40d040 sh wait
10988 1 3 0 80 fffffe8221af7280 ssh select
10711 1 3 0 80 fffffe8211d1f5e0 sh wait
9326 1 3 0 80 fffffe820d891840 ssh select
6637 1 3 2 80 fffffe820fe60b40 sh wait
5607 1 3 0 80 fffffe82143bb780 ssh select
6223 1 3 7 80 fffffe8218c8f620 sh wait
6174 1 3 0 80 fffffe82273070a0 ssh select
3660 1 3 0 80 fffffe820a90e2c0 sh wait
3691 1 3 0 80 fffffe821e9dc040 ssh select
13059 1 3 0 80 fffffe8211b16ba0 sh wait
8242 1 3 0 80 fffffe822c00eb20 ssh select
10272 1 3 3 80 fffffe821dc674e0 sh wait
9712 1 3 7 80 fffffe816482f120 bash select
8589 1 3 0 80 fffffe8229aeda00 sshd select
10949 1 3 1 80 fffffe8224d7d360 conserver select
7765 1 3 4 80 fffffe821f05da80 sshd select
11101 1 3 7 0 fffffe820e858b00 conserver tstile
11558 1 3 6 0 fffffe820b0604e0 conserver tstile
8956 1 3 5 80 fffffe8212bd1940 conserver select
10634 1 3 7 80 fffffe820e350300 conserver select
9547 1 3 0 80 fffffe820d6e9020 conserver select
7586 1 3 4 0 fffffe8212536a60 conserver tstile
8569 1 3 6 0 fffffe8218c8f200 conserver tstile
11721 1 3 6 0 fffffe8225b3a120 conserver tstile
9474 1 3 5 80 fffffe821c068940 conserver select
9068 1 3 0 80 fffffe8212a4d4a0 conserver select
8864 1 3 0 80 fffffe82101bc860 conserver select
6021 1 3 0 80 fffffe82238ef020 getty ttyraw
747 1 3 4 80 fffffe821e715680 bash wait
1233 1 3 5 80 fffffe82285c76a0 ssh select
336 1 3 0 80 fffffe821ba38a60 sh wait
1101 1 3 4 80 fffffe821ed7ea40 ssh select
1228 1 3 0 80 fffffe821bf101e0 sh wait
459 1 3 3 80 fffffe821bf10600 ssh select
458 1 3 0 80 fffffe821bf10a20 sh wait
836 1 3 5 80 fffffe82285c7ac0 ssh select
322 1 3 5 80 fffffe821e69f9e0 sh wait
321 1 3 0 80 fffffe821df29180 ssh select
320 1 3 0 80 fffffe821df295a0 sh wait
1470 1 3 7 80 fffffe8222107660 ssh select
829 1 3 7 80 fffffe821e4a1160 sh wait
956 1 3 4 80 fffffe821e4a1580 ssh select
1211 1 3 0 80 fffffe821e4a19a0 sh wait
1203 1 3 6 80 fffffe821de42540 ssh select
306 1 3 0 80 fffffe821de42960 sh wait
1455 1 3 6 80 fffffe82201ad940 ssh select
1326 1 3 0 80 fffffe821f1730e0 sh wait
813 1 3 2 80 fffffe821f173500 ssh select
1196 1 3 0 80 fffffe821f173920 sh wait
299 1 3 2 80 fffffe82201860c0 ssh select
298 1 3 5 80 fffffe82201864e0 sh wait
297 1 3 5 80 fffffe8220186900 ssh select
296 1 3 5 80 fffffe821eca10a0 sh wait
1191 1 3 7 80 fffffe821eca14c0 ssh select
1062 1 3 0 80 fffffe821eca18e0 sh wait
1445 1 3 6 80 fffffe8220596080 ssh select
1188 1 3 0 80 fffffe82205964a0 sh wait
1443 1 3 5 80 fffffe82205968c0 ssh select
290 1 3 0 80 fffffe821f131060 sh wait
289 1 3 4 80 fffffe821f131480 ssh select
288 1 3 0 80 fffffe821f1318a0 sh wait
1439 1 3 4 80 fffffe8221441040 ssh select
1438 1 3 0 80 fffffe8221441460 sh wait
1181 1 3 7 80 fffffe8222326260 ssh select
540 1 3 0 80 fffffe82209cb780 sh wait
1422 1 3 2 80 fffffe822d8b55a0 ssh select
1165 1 3 5 80 fffffe82209cb360 sh wait
1164 1 3 6 80 fffffe822c5039e0 ssh select
267 1 3 4 80 fffffe822da9c160 sh wait
264 1 3 5 80 fffffe82209cbba0 ssh select
775 1 3 3 80 fffffe8222326680 sh wait
1156 1 3 7 80 fffffe8222107240 ssh select
386 1 3 7 80 fffffe8220131340 sh wait
385 1 3 5 80 fffffe8220131760 ssh select
384 1 3 0 80 fffffe8220131b80 sh wait
780 1 3 6 80 fffffe8221a55320 ssh select
144 1 3 0 80 fffffe8221a55740 sh wait
146 1 3 5 80 fffffe82253a8300 ssh select
1279 1 3 7 80 fffffe8221a55b60 sh wait
1278 1 3 4 80 fffffe822105b300 ssh select
893 1 3 0 80 fffffe822105b720 sh wait
1019 1 3 3 80 fffffe822105bb40 ssh select
1146 1 3 0 80 fffffe8221dc52e0 sh wait
1080 1 3 6 80 fffffe8221dc5700 ssh select
694 1 3 0 80 fffffe8221dc5b20 sh wait
370 1 3 6 80 fffffe8220ab16e0 ssh select
1073 1 3 6 80 fffffe8220ab1b00 sh wait
176 1 3 3 80 fffffe8220d512a0 ssh select
1071 1 3 0 80 fffffe8220d516c0 sh wait
1198 1 3 0 80 fffffe8220d51ae0 ssh select
877 1 3 0 80 fffffe82218a3280 sh wait
795 1 3 3 80 fffffe822227b1e0 ssh select
986 1 3 0 80 fffffe822227b600 sh wait
985 1 3 4 80 fffffe822227ba20 ssh select
1240 1 3 0 80 fffffe82223491c0 sh wait
727 1 3 0 80 fffffe82223495e0 ssh select
790 1 3 0 80 fffffe8222349a00 sh wait
1173 1 3 1 80 fffffe82232021a0 ssh select
980 1 3 1 80 fffffe82232025c0 sh wait
659 1 3 1 80 fffffe82232029e0 ssh select
210 1 3 1 80 fffffe822a57f180 sh wait
913 1 3 5 80 fffffe822a57f5a0 ssh select
208 1 3 1 80 fffffe822a57f9c0 sh wait
1231 1 3 7 80 fffffe822bd73160 ssh select
1038 1 3 1 80 fffffe822bd73580 sh wait
973 1 3 2 80 fffffe822bd739a0 ssh select
844 1 3 1 80 fffffe8223fb7140 sh wait
139 1 3 1 80 fffffe8223fb7560 ssh select
138 1 3 1 80 fffffe8223fb7980 sh wait
508 1 3 1 80 fffffe82238ef440 cron nanoslp
827 1 3 1 80 fffffe82238ef860 python2.7 select
112 1 3 0 80 fffffe822d8b5180 inetd kqueue
743 1 3 0 80 fffffe8224bf4420 nginx kqueue
870 1 3 0 80 fffffe8224bf4840 nginx pause
974 1 3 1 80 fffffe822604bb20 qmgr kqueue
991 1 3 2 80 fffffe8224850740 master kqueue
647 1 3 4 80 fffffe822da9c9a0 sshd select
701 1 3 2 80 fffffe822995ea20 tac_plus select
685 1 3 1 80 fffffe822d9c9560 powerd kqueue
630 1 3 0 80 fffffe82289476c0 ntpd pause
595 4 3 6 80 fffffe8228947ae0 dhcpd parked
595 3 3 0 80 fffffe82284e8680 dhcpd kqueue
595 2 3 6 80 fffffe82284e8aa0 dhcpd parked
595 1 3 0 80 fffffe82288da660 dhcpd parked
439 1 3 7 0 fffffe82288da240 syslogd tstile
1 1 3 2 80 fffffe810f9b20a0 init wait
0 123 3 5 200 fffffe8229aed1c0 npfgc-0 xchicv
0 122 3 0 200 fffffe822d8b59c0 ipmi0 ipmi0
0 121 3 0 200 fffffe822ddb5500 physiod physiod
0 120 3 0 200 fffffe822dd44520 aiodoned aiodoned
0 119 3 6 200 fffffe822dd44940 ioflush biolock
0 118 3 0 200 fffffe822ddb50e0 pgdaemon pgdaemon
0 115 3 0 280 fffffe810f41e0c0 spkr1 bellcv
0 114 3 5 280 fffffe810f41e4e0 audiomix play
0 113 3 5 280 fffffe810ed322a0 audiorec record
0 112 3 5 200 fffffe810f2c9300 atapibus0 sccomp
0 110 3 3 200 fffffe810f2e0b60 usb5 usbevt
0 109 3 5 200 fffffe810f2c9b40 usb4 usbevt
0 108 3 0 200 fffffe810f1fc2c0 usb1 usbevt
0 107 3 0 200 fffffe810f2c9720 usb0 usbevt
0 106 3 5 200 fffffe810f2ba2e0 usb3 usbevt
0 105 3 5 200 fffffe810f41e900 usb2 usbevt
0 104 3 0 200 fffffe810f9b24c0 rt_free rt_free
0 103 3 0 200 fffffe810f9b28e0 unpgc unpgc
0 102 3 2 200 fffffe810f47b080 key_timehandler key_timehandler
0 101 3 7 200 fffffe810f47b4a0 icmp6_wqinput/7 icmp6_wqinput
0 100 3 6 200 fffffe810f47b8c0 icmp6_wqinput/6 icmp6_wqinput
0 99 3 5 200 fffffe810f3b4060 icmp6_wqinput/5 icmp6_wqinput
0 98 3 4 200 fffffe810f3b4480 icmp6_wqinput/4 icmp6_wqinput
0 97 3 3 200 fffffe810f3b48a0 icmp6_wqinput/3 icmp6_wqinput
0 96 3 2 200 fffffe810f435040 icmp6_wqinput/2 icmp6_wqinput
0 95 3 1 200 fffffe810f435460 icmp6_wqinput/1 icmp6_wqinput
0 94 3 0 200 fffffe810f435880 icmp6_wqinput/0 icmp6_wqinput
0 93 3 0 200 fffffe810f34e020 nd6_timer nd6_timer
0 92 3 7 200 fffffe810f34e440 icmp_wqinput/7 icmp_wqinput
0 91 3 6 200 fffffe810f34e860 icmp_wqinput/6 icmp_wqinput
0 90 3 5 200 fffffe810f337000 icmp_wqinput/5 icmp_wqinput
0 89 3 4 200 fffffe810f337420 icmp_wqinput/4 icmp_wqinput
0 88 3 3 200 fffffe810f337840 icmp_wqinput/3 icmp_wqinput
0 87 3 2 200 fffffe810f360360 icmp_wqinput/2 icmp_wqinput
0 86 3 1 200 fffffe810f360780 icmp_wqinput/1 icmp_wqinput
0 85 3 0 200 fffffe810f360ba0 icmp_wqinput/0 icmp_wqinput
0 84 3 5 200 fffffe810f2ba700 rt_timer rt_timer
0 83 3 5 200 fffffe810f2bab20 vmem_rehash vmem_rehash
0 82 3 3 200 fffffe810f2d7340 coretemp3 coretemp3
0 81 3 6 200 fffffe810f2d7760 coretemp2 coretemp2
0 80 3 1 200 fffffe810f2d7b80 coretemp1 coretemp1
0 79 3 7 200 fffffe810f2e0320 coretemp0 coretemp0
0 78 3 6 200 fffffe810f2e0740 mfi0 mfi0
0 69 3 0 200 fffffe810f1fc6e0 atabus1 atath
0 68 3 0 200 fffffe810f1fcb00 atabus0 atath
0 66 3 0 200 fffffe810ed326c0 scsibus0 sccomp
0 65 3 5 200 fffffe810ed32ae0 usbtask-dr usbtsk
0 64 3 0 200 fffffe810eca9280 usbtask-hc usbtsk
0 63 3 7 200 fffffe810eca96a0 bnx3 bnx3
0 62 3 4 200 fffffe810eca9ac0 bnx2 bnx2
0 61 3 0 200 fffffe810eb60260 bnx1 bnx1
0 60 3 7 200 fffffe810eb60680 bnx0 bnx0
0 59 3 5 200 fffffe810eb60aa0 ipmi ipmipoll
0 58 3 7 200 fffffe810ea73240 xcall/7 xcall
0 57 1 7 200 fffffe810ea73660 softser/7
0 56 1 7 200 fffffe810ea73a80 softclk/7
0 55 1 7 200 fffffe810ea54220 softbio/7
0 54 1 7 200 fffffe810ea54640 softnet/7
0 : 53 7 7 201 fffffe810ea54a60 idle/7
0 52 3 6 200 fffffe810ea3d200 xcall/6 xcall
0 51 1 6 200 fffffe810ea3d620 softser/6
0 50 1 6 200 fffffe810ea3da40 softclk/6
0 49 1 6 200 fffffe810ea161e0 softbio/6
0 48 1 6 200 fffffe810ea16600 softnet/6
0 : 47 7 6 201 fffffe810ea16a20 idle/6
0 46 3 5 200 fffffe810e9d71c0 xcall/5 xcall
0 45 1 5 200 fffffe810e9d75e0 softser/5
0 44 1 5 200 fffffe810e9d7a00 softclk/5
0 43 1 5 200 fffffe810e9d01a0 softbio/5
0 42 1 5 200 fffffe810e9d05c0 softnet/5
0 : 41 7 5 201 fffffe810e9d09e0 idle/5
0 40 3 4 200 fffffe810e9a1180 xcall/4 xcall
0 39 1 4 200 fffffe810e9a15a0 softser/4
0 38 1 4 200 fffffe810e9a19c0 softclk/4
0 37 1 4 200 fffffe810e97a160 softbio/4
0 36 1 4 200 fffffe810e97a580 softnet/4
0 : 35 7 4 201 fffffe810e97a9a0 idle/4
0 34 3 3 200 fffffe810e943140 xcall/3 xcall
0 33 1 3 200 fffffe810e943560 softser/3
0 32 1 3 200 fffffe810e943980 softclk/3
0 31 1 3 200 fffffe810e934120 softbio/3
0 30 1 3 200 fffffe810e934540 softnet/3
0 : 29 7 3 201 fffffe810e934960 idle/3
0 28 3 2 200 fffffe810e90d100 xcall/2 xcall
0 27 1 2 200 fffffe810e90d520 softser/2
0 26 1 2 200 fffffe810e90d940 softclk/2
0 25 1 2 200 fffffe810e8f60e0 softbio/2
0 24 1 2 200 fffffe810e8f6500 softnet/2
0 : 23 7 2 201 fffffe810e8f6920 idle/2
0 22 3 1 200 fffffe810e8c50c0 xcall/1 xcall
0 21 1 1 200 fffffe810e8c54e0 softser/1
0 20 1 1 200 fffffe810e8c5900 softclk/1
0 19 1 1 200 fffffe810e8a90a0 softbio/1
0 18 1 1 200 fffffe810e8a94c0 softnet/1
0 : 17 7 1 201 fffffe810e8a98e0 idle/1
0 16 3 0 200 fffffe822de9e080 lnxsyswq lnxsyswq
0 15 3 5 200 fffffe822de9e4a0 sysmon smtaskq
0 14 3 7 200 fffffe822de9e8c0 pmfsuspend pmfsuspend
0 13 3 4 200 fffffe822e2c5060 pmfevent pmfevent
0 12 3 0 200 fffffe822e2c5480 sopendfree sopendfr
0 11 3 2 200 fffffe822e2c58a0 nfssilly nfssilly
0 10 3 6 200 fffffe822f6dc040 cachegc cachegc
0 9 3 6 200 fffffe822f6dc460 vdrain vdrain
0 8 3 1 200 fffffe822f6dc880 modunload mod_unld
0 7 3 0 200 fffffe822f6fd020 xcall/0 xcall
0 : 6 7 0 200 fffffe822f6fd440 softser/0
0 : 5 7 0 200 fffffe822f6fd860 softclk/0
0 4 1 0 200 fffffe822f719000 softbio/0
0 3 1 0 200 fffffe822f719420 softnet/0
0 2 1 0 201 fffffe822f719840 idle/0
0 1 3 3 200 ffffffff81481b20 swapper uvm
db{0}:
and here's the dmesg output for the host:
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
NetBSD 8.0_BETA (GENERIC.201803261630Z)
total memory = 8179 MB
avail memory = 7920 MB
timecounter: Timecounters tick every 10.000 msec
Kernelized RAIDframe activated
running cgd selftest aes-xts-256 aes-xts-512 done
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
Dell Inc. PowerEdge R610
mainbus0 (root)
ACPI: RSDP 0x00000000000F1150 000024 (v02 DELL )
ACPI: XSDT 0x00000000000F1254 000094 (v01 DELL PE_SC3 00000001 DELL 00000001)
ACPI: FACP 0x00000000CF3B3F9C 0000F4 (v03 DELL PE_SC3 00000001 DELL 00000001)
ACPI: DSDT 0x00000000CF38F000 003DD0 (v01 DELL PE_SC3 00000001 INTL 20050624)
ACPI: FACS 0x00000000CF3B6000 000040
ACPI: FACS 0x00000000CF3B6000 000040
ACPI: APIC 0x00000000CF3B3478 00015E (v01 DELL PE_SC3 00000001 DELL 00000001)
ACPI: SPCR 0x00000000CF3B35D8 000050 (v01 DELL PE_SC3 00000001 DELL 00000001)
ACPI: HPET 0x00000000CF3B362C 000038 (v01 DELL PE_SC3 00000001 DELL 00000001)
ACPI: DM__ 0x00000000CF3B3668 0001A8 (v01 DELL PE_SC3 00000001 DELL 00000001)
ACPI: MCFG 0x00000000CF3B38C4 00003C (v01 DELL PE_SC3 00000001 DELL 00000001)
ACPI: WD__ 0x00000000CF3B3904 000134 (v01 DELL PE_SC3 00000001 DELL 00000001)
ACPI: SLIC 0x00000000CF3B3A3C 000176 (v01 DELL PE_SC3 00000001 DELL 00000001)
ACPI: ERST 0x00000000CF392F70 000270 (v01 DELL PE_SC3 00000001 DELL 00000001)
ACPI: HEST 0x00000000CF3931E0 0003A8 (v01 DELL PE_SC3 00000001 DELL 00000001)
ACPI: BERT 0x00000000CF392DD0 000030 (v01 DELL PE_SC3 00000001 DELL 00000001)
ACPI: EINJ 0x00000000CF392E00 000170 (v01 DELL PE_SC3 00000001 DELL 00000001)
ACPI: SRAT 0x00000000CF3B3BC0 000370 (v01 DELL PE_SC3 00000001 DELL 00000001)
ACPI: TCPA 0x00000000CF3B3F34 000064 (v02 DELL PE_SC3 00000001 DELL 00000001)
ACPI: 1 ACPI AML tables successfully acquired and loaded
ioapic0 at mainbus0 apid 0: pa 0xfec00000, version 0x20, 24 pins
ioapic1 at mainbus0 apid 1: pa 0xfec80000, version 0x20, 24 pins
cpu0 at mainbus0 apid 32
cpu0: Intel(R) Xeon(R) CPU E5640 @ 2.67GHz, id 0x206c2
cpu0: package 1, core 0, smt 0
cpu1 at mainbus0 apid 34
cpu1: Intel(R) Xeon(R) CPU E5640 @ 2.67GHz, id 0x206c2
cpu1: package 1, core 1, smt 0
cpu2 at mainbus0 apid 50
cpu2: Intel(R) Xeon(R) CPU E5640 @ 2.67GHz, id 0x206c2
cpu2: package 1, core 9, smt 0
cpu3 at mainbus0 apid 52
cpu3: Intel(R) Xeon(R) CPU E5640 @ 2.67GHz, id 0x206c2
cpu3: package 1, core 10, smt 0
cpu4 at mainbus0 apid 33
cpu4: Intel(R) Xeon(R) CPU E5640 @ 2.67GHz, id 0x206c2
cpu4: package 1, core 0, smt 1
cpu5 at mainbus0 apid 35
cpu5: Intel(R) Xeon(R) CPU E5640 @ 2.67GHz, id 0x206c2
cpu5: package 1, core 1, smt 1
cpu6 at mainbus0 apid 51
cpu6: Intel(R) Xeon(R) CPU E5640 @ 2.67GHz, id 0x206c2
cpu6: package 1, core 9, smt 1
cpu7 at mainbus0 apid 53
cpu7: Intel(R) Xeon(R) CPU E5640 @ 2.67GHz, id 0x206c2
cpu7: package 1, core 10, smt 1
acpi0 at mainbus0: Intel ACPICA 20170303
acpi0: X/RSDT: OemId <DELL ,PE_SC3 ,00000001>, AslId <DELL,00000001>
acpi0: MCFG: segment 0, bus 0-255, address 0x00000000e0000000
acpi0: SCI interrupting at int 9
timecounter: Timecounter "ACPI-Fast" frequency 3579545 Hz quality 1000
hpet0 at acpi0: high precision event timer (mem 0xfed00000-0xfed00400)
timecounter: Timecounter "hpet0" frequency 14318180 Hz quality 2000
WHEA (PNP0C33) at acpi0 not configured
PMI0 (ACPI000D) at acpi0 not configured
SPK (PNP0C01) at acpi0 not configured
attimer1 at acpi0 (TMR, PNP0100): io 0x40-0x5f irq 0
COMA (PNP0501) at acpi0 not configured
COMB (PNP0501) at acpi0 not configured
MBIO (PNP0C01) at acpi0 not configured
NIPM (IPI0001) at acpi0 not configured
PEHB (PNP0C02) at acpi0 not configured
VTD (PNP0C02) at acpi0 not configured
acpiwmi0 at acpi0 (SRIO, PNP0C14-SRIO): ACPI WMI Interface
acpiwmibus at acpiwmi0 not configured
ACPI: Enabled 1 GPEs in block 00 to 3F
ipmi0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0: vendor 8086 product 3403 (rev. 0x13)
ppb0 at pci0 dev 1 function 0: vendor 8086 product 3408 (rev. 0x13)
ppb0: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x4 @ 5.0GT/s
ppb0: link is x4 @ 2.5GT/s
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
bnx0 at pci1 dev 0 function 0: Broadcom NetXtreme II BCM5709 1000Base-T
bnx0: Ethernet address f0:4d:a2:03:7b:f3
bnx0: interrupting at ioapic1 pin 4
bnx0: ASIC BCM5709 C0 (0x57092003)
bnx0: PCIe x4 2.5Gbps
bnx0: Coal (RX:6,6,18,18; TX:20,20,80,80)
brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bnx1 at pci1 dev 0 function 1: Broadcom NetXtreme II BCM5709 1000Base-T
bnx1: Ethernet address f0:4d:a2:03:7b:f5
bnx1: interrupting at ioapic1 pin 16
bnx1: ASIC BCM5709 C0 (0x57092003)
bnx1: PCIe x4 2.5Gbps
bnx1: Coal (RX:6,6,18,18; TX:20,20,80,80)
brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
ppb1 at pci0 dev 3 function 0: vendor 8086 product 340a (rev. 0x13)
ppb1: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x4 @ 5.0GT/s
ppb1: link is x4 @ 2.5GT/s
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
bnx2 at pci2 dev 0 function 0: Broadcom NetXtreme II BCM5709 1000Base-T
bnx2: Ethernet address f0:4d:a2:03:7b:f7
bnx2: interrupting at ioapic1 pin 0
bnx2: ASIC BCM5709 C0 (0x57092003)
bnx2: PCIe x4 2.5Gbps
bnx2: Coal (RX:6,6,18,18; TX:20,20,80,80)
brgphy2 at bnx2 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
brgphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bnx3 at pci2 dev 0 function 1: Broadcom NetXtreme II BCM5709 1000Base-T
bnx3: Ethernet address f0:4d:a2:03:7b:f9
bnx3: interrupting at ioapic1 pin 10
bnx3: ASIC BCM5709 C0 (0x57092003)
bnx3: PCIe x4 2.5Gbps
bnx3: Coal (RX:6,6,18,18; TX:20,20,80,80)
brgphy3 at bnx3 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
brgphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
ppb2 at pci0 dev 7 function 0: vendor 8086 product 340e (rev. 0x13)
ppb2: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x8 @ 5.0GT/s
pci3 at ppb2 bus 4
pci3: i/o space, memory space enabled, rd/line, wr/inv ok
ppb3 at pci0 dev 9 function 0: vendor 8086 product 3410 (rev. 0x13)
ppb3: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x8 @ 5.0GT/s
pci4 at ppb3 bus 5
pci4: i/o space, memory space enabled, rd/line, wr/inv ok
vendor 8086 product 342e (interrupt system, generic 8259, revision 0x13) at pci0 dev 20 function 0 not configured
vendor 8086 product 3422 (interrupt system, generic 8259, revision 0x13) at pci0 dev 20 function 1 not configured
vendor 8086 product 3423 (interrupt system, generic 8259, revision 0x13) at pci0 dev 20 function 2 not configured
uhci0 at pci0 dev 26 function 0: vendor 8086 product 2937 (rev. 0x02)
uhci0: interrupting at ioapic0 pin 17
usb0 at uhci0: USB revision 1.0
uhci1 at pci0 dev 26 function 1: vendor 8086 product 2938 (rev. 0x02)
uhci1: interrupting at ioapic0 pin 18
usb1 at uhci1: USB revision 1.0
ehci0 at pci0 dev 26 function 7: vendor 8086 product 293c (rev. 0x02)
ehci0: interrupting at ioapic0 pin 19
ehci0: BIOS has given up ownership
ehci0: EHCI version 1.0
ehci0: 2 companion controllers, 2 ports each: uhci0 uhci1
usb2 at ehci0: USB revision 2.0
ppb4 at pci0 dev 28 function 0: vendor 8086 product 2940 (rev. 0x02)
ppb4: PCI Express capability version 1 <Root Port of PCI-E Root Complex> x4 @ 2.5GT/s
pci5 at ppb4 bus 3
pci5: i/o space, memory space enabled, rd/line, wr/inv ok
mfi0 at pci5 dev 0 function 0: Dell PERC H700 Integrated
mfi0: interrupting at ioapic0 pin 16
mfi0: PERC H700 Integrated version 12.10.4-0001
mfi0: logical drives 2, 1024MB RAM, BBU type BBU, status good
scsibus0 at mfi0: 64 targets, 8 luns per target
uhci2 at pci0 dev 29 function 0: vendor 8086 product 2934 (rev. 0x02)
uhci2: interrupting at ioapic0 pin 21
usb3 at uhci2: USB revision 1.0
uhci3 at pci0 dev 29 function 1: vendor 8086 product 2935 (rev. 0x02)
uhci3: interrupting at ioapic0 pin 20
usb4 at uhci3: USB revision 1.0
ehci1 at pci0 dev 29 function 7: vendor 8086 product 293a (rev. 0x02)
ehci1: interrupting at ioapic0 pin 21
ehci1: EHCI version 1.0
ehci1: 2 companion controllers, 2 ports each: uhci2 uhci3
usb5 at ehci1: USB revision 2.0
ppb5 at pci0 dev 30 function 0: vendor 8086 product 244e (rev. 0x92)
pci6 at ppb5 bus 6
pci6: i/o space, memory space enabled
vga0 at pci6 dev 3 function 0: vendor 102b product 0532 (rev. 0x0a)
wsdisplay0 at vga0 kbdmux 1
wsmux1: connecting to wsdisplay0
drm at vga0 not configured
ichlpcib0 at pci0 dev 31 function 0: vendor 8086 product 2918 (rev. 0x02)
timecounter: Timecounter "ichlpcib0" frequency 3579545 Hz quality 1000
ichlpcib0: 24-bit timer
tco0 at ichlpcib0: TCO (watchdog) timer configured.
tco0: Min/Max interval 1/367 seconds
piixide0 at pci0 dev 31 function 2: Intel 82801I Serial ATA Controller (ICH9) (rev. 0x02)
piixide0: bus-master DMA support present
piixide0: primary channel configured to native-PCI mode
piixide0: using ioapic0 pin 23 for native-PCI interrupt
atabus0 at piixide0 channel 0
piixide0: secondary channel configured to native-PCI mode
atabus1 at piixide0 channel 1
isa0 at ichlpcib0
tpm0 at isa0 iomem 0xfed40000-0xfed44fff irq 7: device 0x0000104a rev 0x4e
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com0: console
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pcppi0 at isa0 port 0x61
spkr0 at pcppi0: PC Speaker
midi0 at pcppi0: PC speaker
sysbeep0 at pcppi0
attimer1: attached to pcppi0
acpicpu0 at cpu0: ACPI CPU
acpicpu0: C1: HLT, lat 0 us, pow 0 mW
coretemp0 at cpu0: thermal sensor, 1 C resolution, Tjmax=95
acpicpu1 at cpu1: ACPI CPU
coretemp1 at cpu1: thermal sensor, 1 C resolution, Tjmax=95
acpicpu2 at cpu2: ACPI CPU
coretemp2 at cpu2: thermal sensor, 1 C resolution, Tjmax=95
acpicpu3 at cpu3: ACPI CPU
coretemp3 at cpu3: thermal sensor, 1 C resolution, Tjmax=95
acpicpu4 at cpu4: ACPI CPU
acpicpu5 at cpu5: ACPI CPU
acpicpu6 at cpu6: ACPI CPU
acpicpu7 at cpu7: ACPI CPU
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
timecounter: Timecounter "TSC" frequency 2660187410 Hz quality 3000
scsibus0: waiting 2 seconds for devices to settle...
IPsec: Initialized Security Association Processing.
uhub0 at usb4: vendor 8086 (0x8086) UHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhub1 at usb1: vendor 8086 (0x8086) UHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhub2 at usb3: vendor 8086 (0x8086) UHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhub3 at usb2: vendor 8086 (0x8086) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub3: 4 ports with 4 removable, self powered
uhub4 at usb0: vendor 8086 (0x8086) UHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub4: 2 ports with 2 removable, self powered
uhub5 at usb5: vendor 8086 (0x8086) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub5: 4 ports with 4 removable, self powered
ehci0: handing over low speed device on port 1 to uhci0
ehci1: handing over full speed device on port 2 to uhci2
sd0 at scsibus0 target 0 lun 0: <DELL, PERC H700, 2.10> disk fixed
sd0: fabricating a geometry
sd0: 69376 MB, 69376 cyl, 64 head, 32 sec, 512 bytes/sect x 142082048 sectors
sd0: fabricating a geometry
sd0: tagged queueing
sd1 at scsibus0 target 1 lun 0: <DELL, PERC H700, 2.10> disk fixed
sd1: fabricating a geometry
sd1: 558 GB, 571776 cyl, 64 head, 32 sec, 512 bytes/sect x 1170997248 sectors
sd1: fabricating a geometry
sd1: tagged queueing
uhub6 at uhub3 port 3: vendor 0424 (0x424) product 2514 (0x2514), class 9/0, rev 2.00/0.00, addr 2
uhub6: multiple transaction translators
uhub6: 3 ports with 3 removable, self powered
uhidev0 at uhub4 port 1 configuration 1 interface 0
uhidev0: LITE-ON Technology (0x413c) Dell USB/PS2 Combo Rackmount Keyboard (0x2103), rev 1.10/1.00, addr 2, iclass 3/1
ukbd0 at uhidev0: 8 Variable keys, 6 Array codes
uhidev1 at uhub2 port 2 configuration 1 interface 0
uhidev1: Avocent (0x624) USB Composite Device-0 (0x248), rev 1.10/0.00, addr 2, iclass 3/1
ukbd1 at uhidev1: 8 Variable keys, 6 Array codes
wskbd0 at ukbd0 mux 1
wskbd0: connecting to wsdisplay0
uhidev2 at uhub4 port 1 configuration 1 interface 1
uhidev2: LITE-ON Technology (0x413c) Dell USB/PS2 Combo Rackmount Keyboard (0x2103), rev 1.10/1.00, addr 2, iclass 3/1
ums0 at uhidev2: 2 buttons
wsmouse0 at ums0 mux 0
wskbd1 at ukbd1 mux 1
wskbd1: connecting to wsdisplay0
uhidev3 at uhub2 port 2 configuration 1 interface 1
uhidev3: Avocent (0x624) USB Composite Device-0 (0x248), rev 1.10/0.00, addr 2, iclass 3/1
ums1 at uhidev3: 3 buttons and Z dir
wsmouse1 at ums1 mux 0
atapibus0 at atabus0: 2 targets
cd0 at atapibus0 drive 0: <TEAC DVD-ROM DV-28SW, 10092412144820, R.2A> cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
cd0(piixide0:0:0): using PIO mode 4, Ultra-DMA mode 5 (Ultra/100) (using DMA)
pad0: outputs: 44100Hz, 16-bit, stereo
audio0 at pad0: half duplex, playback, capture, mmap
pad0: Virtual format configured - Format SLINEAR, precision 16, channels 2, frequency 44100
pad0: Latency: 139 milliseconds
spkr1 at audio0: PC Speaker (synthesized)
boot device: sd0
root on sd0a dumps on sd0b
root file system type: ffs
kern.module.path=/stand/amd64/8.0/modules
mfi0: normal state on 'mfi0:0' (online)
mfi0: normal state on 'mfi0:1' (online)
ipmi0: version 2.0 interface KCS iobase 0xca8/8 spacing 4
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)
>How-To-Repeat:
Configure a host with 3 active interfaces, use it moderately,
watch it wedge after less than a day.
(We have other identical systems which only use one network
interface and which don't wedge.)
>Fix:
Sorry, don't know.
I'm asking for more information about what to do next to zero
in on the root cause of this problem.
A kernel with DEBUG? DIAGNOSTIC? LOCKDEBUG? All of them? Or
only two?
What other information needs to be collected next time it
wedges to get closer to the root cause of the problem?
>Release-Note:
>Audit-Trail:
From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Wed, 04 Apr 2018 16:24:58 +0200 (CEST)
Hi,
a follow-up with a new hang with similar symptoms.
This time fewer processes in "tstile" and a few in biowait.
Doing "c" once and waiting a few seconds and collecting new info
appears to indicate it's not making any progress.
Again, edited DDB console log follows.
fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0x767a01b11cb0
ilevel 0x8 rsp 0xffff80008f69bc88
curlwp 0xfffffe822f6fd440 pid 0.6 lowest kstack 0xffff80008f6972c0
Stopped in pid 0.6 (system) at netbsd:breakpoint+0x5: leave
db{0}: trace
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
bus_space_read_2() at netbsd:bus_space_read_2+0x10
uhci_intr() at netbsd:uhci_intr+0x8f
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}: c
fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0x767a01b11cb0
ilevel 0x8 rsp 0xffff80008f69bc88
curlwp 0xfffffe822f6fd440 pid 0.6 lowest kstack 0xffff80008f6972c0
Stopped in pid 0.6 (system) at netbsd:breakpoint+0x5: leave
db{0}: tra
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
bus_space_read_2() at netbsd:bus_space_read_2+0x10
uhci_intr() at netbsd:uhci_intr+0x8f
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}: ps
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
5868 1 3 5 0 fffffe8215553960 postdrop biowait
11951 1 3 5 80 fffffe81da398080 sendmail netio
6001 1 3 7 0 fffffe82180044a0 vmstat biowait
9372 1 3 2 0 fffffe81f0125420 tail tstile
8217 1 3 6 0 fffffe81efc4d320 tail tstile
9862 1 3 3 80 fffffe81d8509060 cron wait
4632 1 3 0 80 fffffe822b3bb080 ssh select
4405 2 3 4 80 fffffe8224c45b40 expect select
4405 1 3 0 80 fffffe821f14e540 expect parked
4495 1 3 0 80 fffffe8226a1e8a0 sh wait
9356 1 3 5 80 fffffe821b5d0200 perl pipe_rd
6548 1 3 5 80 fffffe8221863a60 sh wait
10228: 2 7 0 0 fffffe81d9f12440 expect
10228 1 3 7 80 fffffe820e8f9160 expect parked
7679 1 3 2 80 fffffe822c143460 sh wait
12514 1 3 2 80 fffffe821daaa100 perl pipe_rd
10055 1 3 2 80 fffffe82140c9140 sh wait
11571 1 3 0 80 fffffe8221215460 ssh select
7806 1 3 4 80 fffffe82205b9280 sh wait
9266 1 3 0 80 fffffe8221215880 ssh select
10356 1 3 5 80 fffffe8229b8e760 sh wait
7934 1 3 0 80 fffffe8225e1f6e0 ssh select
10592 1 3 1 80 fffffe82170dc500 sh wait
9681 1 3 2 80 fffffe820e8f9580 perl wait
9305 1 3 1 80 fffffe8219f67b60 sh wait
9908 1 3 0 80 fffffe8225fad840 ssh select
8779 1 3 0 80 fffffe82205b96a0 sh wait
6856 1 3 0 80 fffffe8215553120 ssh select
8803 1 3 1 80 fffffe81eb91a620 sh wait
8994 1 3 0 80 fffffe81d9f12860 ssh select
10470 1 3 6 80 fffffe8225fad420 sh wait
9662 1 3 6 80 fffffe821a15d8e0 perl wait
5076 1 3 5 80 fffffe82140c9980 sh wait
7123 1 3 7 0 fffffe81f997ca80 perl biowait
8956 1 3 5 80 fffffe8225f1c920 sh wait
9975 1 3 0 80 fffffe81f0b356e0 ssh select
5270 1 3 2 80 fffffe822b3bb4a0 sh wait
8794 1 3 0 80 fffffe822b9d6760 ssh select
5168 1 3 2 80 fffffe8215553540 sh wait
8565 1 3 0 80 fffffe821951e020 ssh select
8522 1 3 0 80 fffffe822a8f4100 sh wait
8946 1 3 0 80 fffffe820dddda20 ssh select
8677 1 3 5 80 fffffe822d94a560 sh wait
6875 1 3 0 80 fffffe821951e440 ssh select
8644 1 3 7 80 fffffe821d24b660 sh wait
7694 1 3 0 80 fffffe82251dfb20 ssh select
7939 1 3 3 80 fffffe820354a5a0 sh wait
5953 1 3 5 80 fffffe81f997c240 ssh select
5952 1 3 6 80 fffffe8224c45720 sh wait
7773 1 3 0 80 fffffe822140b680 ssh select
6643 1 3 7 80 fffffe822c5ab680 sh wait
6757 1 3 0 80 fffffe81f0125840 ssh select
8943 1 3 4 80 fffffe81fc3e5640 sh wait
7329 1 3 0 80 fffffe822d946780 ssh select
6222 1 3 0 80 fffffe820af7d1a0 sh wait
6930 1 3 0 80 fffffe821a2012a0 ssh select
6708 1 3 6 80 fffffe821a15d0a0 sh wait
5857 1 3 6 80 fffffe8223ee1320 ssh select
7463 1 3 2 80 fffffe821d4af220 sh wait
5817 1 3 0 80 fffffe8222cfd540 ssh select
6855 1 3 5 80 fffffe8225a53780 sh wait
7144 1 3 0 80 fffffe821b5d0620 ssh select
7222 1 3 5 80 fffffe8221863640 sh wait
6589 1 3 0 80 fffffe820af7d5c0 ssh select
7157 1 3 3 80 fffffe822d94a140 sh wait
6877 1 3 2 80 fffffe82039c65e0 ssh select
6057 1 3 7 80 fffffe821d4af640 sh wait
6392 1 3 5 80 fffffe822d71a6e0 ssh select
4924 1 3 4 80 fffffe8222a64100 sh wait
5490 1 3 0 80 fffffe8218a57000 ssh select
5061 1 3 6 80 fffffe81fbd2daa0 sh wait
6508 1 3 0 80 fffffe82251df2e0 ssh select
4832 1 3 6 80 fffffe81fbd2d680 sh wait
5990 1 3 0 80 fffffe8225e1f2c0 ssh select
5243 1 3 2 80 fffffe820dddd600 sh wait
3871 1 3 5 80 fffffe8221476b20 par pause
2696 1 3 5 80 fffffe822aad3900 sh wait
4142 1 3 0 80 fffffe821a62c060 sh wait
4763 1 3 0 80 fffffe82299ed1a0 sh wait
5220 1 3 0 80 fffffe8222a64520 sh wait
4858 1 3 3 80 fffffe821daba920 cron pipe_rd
3551 1 3 0 80 fffffe821de8b9e0 ssh select
4159 1 3 5 80 fffffe8226fc48e0 sh wait
5091 1 3 0 80 fffffe8221863220 ssh select
3848 1 3 0 80 fffffe822c143880 sh wait
4429 1 3 0 80 fffffe82140c9560 ssh select
3178 1 3 0 80 fffffe81eb91aa40 sh wait
4329 1 3 0 80 fffffe820354a180 ssh select
4193 1 3 1 80 fffffe820dddd1e0 sh wait
3408 1 3 0 80 fffffe82251df700 ssh select
4951 1 3 0 80 fffffe8221ea5660 bash wait
4422 1 3 5 80 fffffe820af7d9e0 sshd select
3919 1 3 0 80 fffffe82299aa6a0 sshd select
3312 1 3 0 80 fffffe8229b8eb80 ssh select
3607 1 3 7 80 fffffe8225a53ba0 bash wait
4021 1 3 0 80 fffffe821e8b72e0 sshd select
3876 1 3 0 80 fffffe821daba0e0 sshd select
3633 1 3 1 80 fffffe822aad34e0 ssh select
3679 1 3 7 80 fffffe822a8f4520 bash wait
3250 1 3 0 80 fffffe8224c45300 sshd select
3676 1 3 0 80 fffffe821ffb8200 sshd select
3306 1 3 0 80 fffffe82253fc020 ssh select
3556 1 3 2 80 fffffe821951e860 sh wait
2753 1 3 0 80 fffffe8221215040 ssh select
2752 1 3 0 80 fffffe821a201ae0 bash wait
3789 1 3 0 80 fffffe8228146640 sshd select
2376 1 3 5 80 fffffe820e8f99a0 sshd select
1466 1 3 7 80 fffffe8221476700 bash wait
1782 1 3 0 80 fffffe821daba500 tcsh ttyraw
2030 1 3 2 80 fffffe821ffb8620 sshd select
1515 1 3 5 80 fffffe822c5abaa0 sshd select
1238 1 3 0 80 fffffe821d4afa60 ssh select
853 1 3 3 80 fffffe822d772580 sh wait
1740 1 3 0 80 fffffe821e6ec1e0 ssh select
459 1 3 7 80 fffffe821e6ec600 sh wait
458 1 3 0 80 fffffe821daaa940 ssh select
457 1 3 1 80 fffffe821e6eca20 sh wait
1222 1 3 0 80 fffffe821dd2b1c0 ssh select
1093 1 3 0 80 fffffe821dd2b5e0 sh wait
1092 1 3 0 80 fffffe821dd2ba00 ssh select
835 1 3 0 80 fffffe821de8b1a0 sh wait
1212 1 3 0 80 fffffe821d19a9c0 ssh select
1467 1 3 0 80 fffffe821e0db160 conserver nanoslp
1082 1 3 5 80 fffffe821e0db580 sh wait
1081 1 3 0 80 fffffe821e0db9a0 ssh select
824 1 3 5 80 fffffe821ef36140 sh wait
1079 1 3 0 80 fffffe821ef36560 ssh select
1206 1 3 5 80 fffffe821ef36980 sh wait
1203 1 3 6 80 fffffe821f14e960 conserver select
1324 1 3 0 80 fffffe821df920c0 ssh select
299 1 3 0 80 fffffe821df924e0 sh wait
298 1 3 0 80 fffffe821df92900 ssh select
297 1 3 0 80 fffffe821e2910a0 sh wait
296 1 3 0 80 fffffe821e2914c0 ssh select
1703 1 3 0 80 fffffe821e2918e0 sh wait
1190 1 3 0 80 fffffe821e442080 ssh select
1061 1 3 0 80 fffffe821e4424a0 sh wait
1188 1 3 0 80 fffffe821e4428c0 ssh select
1187 1 3 0 80 fffffe821e6e9060 sh wait
674 1 3 0 80 fffffe821e6e9480 ssh select
289 1 3 0 80 fffffe821e6e98a0 sh wait
288 1 3 0 80 fffffe821ff14040 ssh select
927 1 3 0 80 fffffe821ff14460 sh wait
798 1 3 0 80 fffffe821ff14880 ssh select
1181 1 3 0 80 fffffe822129d020 sh wait
1052 1 3 0 80 fffffe822129d440 ssh select
1051 1 3 0 80 fffffe822dafd120 sh wait
1178 1 3 0 80 fffffe821f670280 ssh select
1177 1 3 0 80 fffffe82283bd680 sh wait
1176 1 3 0 80 fffffe8225e1fb00 ssh select
1431 1 3 7 80 fffffe822129d860 sh wait
790 1 3 0 80 fffffe821edea000 conserver select
1173 1 3 0 80 fffffe821edea420 conserver nanoslp
1044 1 3 0 80 fffffe82299aaac0 ssh select
402 1 3 7 80 fffffe822140b260 sh wait
400 1 3 0 80 fffffe82214d7360 conserver select
1293 1 3 0 80 fffffe822d1cc5a0 ssh select
1292 1 3 0 80 fffffe821f670ac0 sh wait
267 1 3 0 80 fffffe82214d7780 ssh select
266 1 3 4 80 fffffe82214d7ba0 sh wait
1415 1 3 5 80 fffffe821f5f2760 conserver select
1030 1 3 0 80 fffffe821f5f2b80 ssh select
1157 1 3 3 80 fffffe82299ed9e0 sh wait
1156 1 3 0 80 fffffe82299ed5c0 ssh select
1155 1 3 0 80 fffffe8229e40ae0 sh wait
386 1 3 0 80 fffffe821f9a1320 ssh select
385 1 3 7 80 fffffe821f9a1740 sh wait
384 1 3 0 80 fffffe821f9a1b60 ssh select
447 1 3 7 80 fffffe82211cb300 sh wait
830 1 3 0 80 fffffe82211cb720 ssh select
1021 1 3 7 80 fffffe82299aa280 sh wait
888 1 3 0 80 fffffe82211cbb40 ssh select
1143 1 3 1 80 fffffe82214762e0 sh wait
176 1 3 0 80 fffffe8220dab2c0 ssh select
815 1 3 0 80 fffffe8220dab6e0 sh wait
878 1 3 0 80 fffffe8220dabb00 ssh select
1069 1 3 0 80 fffffe8220f432a0 sh wait
1196 1 3 7 80 fffffe8220f436c0 ssh select
363 1 3 5 80 fffffe8220f43ae0 sh wait
361 1 3 0 80 fffffe821f6706a0 conserver select
1115 1 3 0 80 fffffe8221cb31e0 ssh select
986 1 3 5 80 fffffe8221cb3600 sh wait
985 1 3 0 80 fffffe8221cb3a20 ssh select
984 1 3 5 80 fffffe822261e1c0 sh wait
1239 1 3 0 80 fffffe822261e5e0 ssh select
854 1 3 5 80 fffffe822261ea00 sh wait
789 1 3 0 80 fffffe822a8ba1a0 ssh select
1108 1 3 5 80 fffffe822a8ba5c0 sh wait
851 1 3 0 80 fffffe822a8ba9e0 ssh select
210 1 3 5 80 fffffe8220918180 sh wait
913 1 3 0 80 fffffe82209185a0 ssh select
208 1 3 5 80 fffffe82209189c0 sh wait
783 1 3 0 80 fffffe8222d0a160 ssh select
974 1 3 0 80 fffffe8222d0a580 sh wait
1165 1 3 0 80 fffffe8222d0a9a0 ssh select
1356 1 3 0 80 fffffe822287f140 sh wait
139 1 3 0 80 fffffe822287f560 ssh select
138 1 3 0 80 fffffe822287f980 sh wait
137 1 3 5 80 fffffe8222cfd120 conserver select
168 1 3 1 80 fffffe822dafd960 conserver select
1127 1 3 5 80 fffffe8228f78620 conserver select
933 1 3 5 80 fffffe822dda2920 getty ttyraw
1031 1 3 0 80 fffffe822dd49100 conserver select
797 1 3 5 80 fffffe8225a53360 conserver select
829 1 3 3 80 fffffe82283bd260 cron nanoslp
1019 1 3 3 0 fffffe822bcd0a00 python2.7 biowait
112 1 3 0 80 fffffe8229b8e340 inetd kqueue
772 1 3 0 80 fffffe8223ee1740 nginx kqueue
803 1 3 0 80 fffffe8223ee1b60 nginx pause
773 1 3 5 80 fffffe8229e406c0 qmgr kqueue
951 1 3 5 80 fffffe8228146a60 pickup kqueue
794 1 3 5 80 fffffe8229e402a0 master kqueue
696 1 3 0 80 fffffe822d1cc180 sshd select
725 1 3 3 80 fffffe822d772160 tac_plus select
686 1 3 6 80 fffffe822806d240 powerd kqueue
659 1 3 0 80 fffffe822806d660 ntpd pause
606 4 3 6 80 fffffe822806da80 dhcpd parked
606 3 3 6 80 fffffe8228146220 dhcpd kqueue
606 2 3 6 80 fffffe8228f78200 dhcpd parked
606 1 3 0 80 fffffe822986b1e0 dhcpd parked
383 : 1 7 7 0 fffffe8228f78a40 syslogd
1 1 3 3 80 fffffe810f45a0a0 init wait
0 123 3 3 200 fffffe822986ba20 npfgc-0 xchicv
0 122 3 5 200 fffffe822d1cc9c0 ipmi0 ipmi0
0 121 3 0 200 fffffe822dda2500 physiod physiod
0 120 3 0 200 fffffe822dd49520 aiodoned aiodoned
0 119 3 0 200 fffffe822dd49940 ioflush syncer
0 118 3 0 200 fffffe822dda20e0 pgdaemon pgdaemon
0 115 3 0 280 fffffe810f4260c0 spkr1 bellcv
0 114 3 4 280 fffffe810f4264e0 audiomix play
0 113 3 4 280 fffffe810ed322a0 audiorec record
0 112 3 5 200 fffffe810f2c9720 atapibus0 sccomp
0 110 3 0 200 fffffe810f2ba700 usb2 usbevt
0 109 3 0 200 fffffe810f2ba2e0 usb1 usbevt
0 108 3 0 200 fffffe810f2bab20 usb3 usbevt
0 107 3 0 200 fffffe810f2c9300 usb0 usbevt
0 106 3 5 200 fffffe810f2e0b60 usb4 usbevt
0 105 3 5 200 fffffe810f426900 usb5 usbevt
0 104 3 0 200 fffffe810f45a4c0 rt_free rt_free
0 103 3 0 200 fffffe810f45a8e0 unpgc unpgc
0 102 3 3 200 fffffe810f463080 key_timehandler key_timehandler
0 101 3 7 200 fffffe810f4634a0 icmp6_wqinput/7 icmp6_wqinput
0 100 3 6 200 fffffe810f4638c0 icmp6_wqinput/6 icmp6_wqinput
0 99 3 5 200 fffffe810f3ec060 icmp6_wqinput/5 icmp6_wqinput
0 98 3 4 200 fffffe810f3ec480 icmp6_wqinput/4 icmp6_wqinput
0 97 3 3 200 fffffe810f3ec8a0 icmp6_wqinput/3 icmp6_wqinput
0 96 3 2 200 fffffe810f38d040 icmp6_wqinput/2 icmp6_wqinput
0 95 3 1 200 fffffe810f38d460 icmp6_wqinput/1 icmp6_wqinput
0 94 3 0 200 fffffe810f2c9b40 icmp6_wqinput/0 icmp6_wqinput
0 93 3 0 200 fffffe810f38d880 nd6_timer nd6_timer
0 92 3 7 200 fffffe810f34e020 icmp_wqinput/7 icmp_wqinput
0 91 3 6 200 fffffe810f34e440 icmp_wqinput/6 icmp_wqinput
0 90 3 5 200 fffffe810f34e860 icmp_wqinput/5 icmp_wqinput
0 89 3 4 200 fffffe810f337000 icmp_wqinput/4 icmp_wqinput
0 88 3 3 200 fffffe810f337420 icmp_wqinput/3 icmp_wqinput
0 87 3 2 200 fffffe810f337840 icmp_wqinput/2 icmp_wqinput
0 86 3 1 200 fffffe810f33d360 icmp_wqinput/1 icmp_wqinput
0 85 3 0 200 fffffe810f33d780 icmp_wqinput/0 icmp_wqinput
0 84 3 6 200 fffffe810f33dba0 rt_timer rt_timer
0 83 3 7 200 fffffe810f1fc2c0 vmem_rehash vmem_rehash
0 82 3 3 200 fffffe810f2d7340 coretemp3 coretemp3
0 81 3 7 200 fffffe810f2d7760 coretemp2 coretemp2
0 80 3 1 200 fffffe810f2d7b80 coretemp1 coretemp1
0 79 3 6 200 fffffe810f2e0320 coretemp0 coretemp0
0 78 3 4 200 fffffe810f2e0740 mfi0 mfi0
0 69 3 0 200 fffffe810f1fc6e0 atabus1 atath
0 68 3 0 200 fffffe810f1fcb00 atabus0 atath
0 66 3 0 200 fffffe810ed326c0 scsibus0 sccomp
0 65 3 5 200 fffffe810ed32ae0 usbtask-dr usbtsk
0 64 3 0 200 fffffe810eca9280 usbtask-hc usbtsk
0 63 3 7 200 fffffe810eca96a0 bnx3 bnx3
0 62 3 1 200 fffffe810eca9ac0 bnx2 bnx2
0 61 3 5 200 fffffe810eb60260 bnx1 bnx1
0 60 3 4 200 fffffe810eb60680 bnx0 bnx0
0 59 3 5 200 fffffe810eb60aa0 ipmi ipmipoll
0 58 3 7 200 fffffe810ea73240 xcall/7 xcall
0 57 1 7 200 fffffe810ea73660 softser/7
0 56 1 7 200 fffffe810ea73a80 softclk/7
0 55 1 7 200 fffffe810ea54220 softbio/7
0 54 1 7 200 fffffe810ea54640 softnet/7
0 53 1 7 201 fffffe810ea54a60 idle/7
0 52 3 6 200 fffffe810ea3d200 xcall/6 xcall
0 51 1 6 200 fffffe810ea3d620 softser/6
0 50 1 6 200 fffffe810ea3da40 softclk/6
0 49 1 6 200 fffffe810ea161e0 softbio/6
0 48 1 6 200 fffffe810ea16600 softnet/6
0 : 47 7 6 201 fffffe810ea16a20 idle/6
0 46 3 5 200 fffffe810e9d71c0 xcall/5 xcall
0 45 1 5 200 fffffe810e9d75e0 softser/5
0 44 1 5 200 fffffe810e9d7a00 softclk/5
0 43 1 5 200 fffffe810e9d01a0 softbio/5
0 42 1 5 200 fffffe810e9d05c0 softnet/5
0 : 41 7 5 201 fffffe810e9d09e0 idle/5
0 40 3 4 200 fffffe810e9a1180 xcall/4 xcall
0 39 1 4 200 fffffe810e9a15a0 softser/4
0 38 1 4 200 fffffe810e9a19c0 softclk/4
0 37 1 4 200 fffffe810e97a160 softbio/4
0 36 1 4 200 fffffe810e97a580 softnet/4
0 : 35 7 4 201 fffffe810e97a9a0 idle/4
0 34 3 3 200 fffffe810e943140 xcall/3 xcall
0 33 1 3 200 fffffe810e943560 softser/3
0 32 1 3 200 fffffe810e943980 softclk/3
0 31 1 3 200 fffffe810e934120 softbio/3
0 30 1 3 200 fffffe810e934540 softnet/3
0 : 29 7 3 201 fffffe810e934960 idle/3
0 28 3 2 200 fffffe810e90d100 xcall/2 xcall
0 27 1 2 200 fffffe810e90d520 softser/2
0 26 1 2 200 fffffe810e90d940 softclk/2
0 25 1 2 200 fffffe810e8f60e0 softbio/2
0 24 1 2 200 fffffe810e8f6500 softnet/2
0 : 23 7 2 201 fffffe810e8f6920 idle/2
0 22 3 1 200 fffffe810e8c50c0 xcall/1 xcall
0 21 1 1 200 fffffe810e8c54e0 softser/1
0 20 1 1 200 fffffe810e8c5900 softclk/1
0 19 1 1 200 fffffe810e8a90a0 softbio/1
0 18 1 1 200 fffffe810e8a94c0 softnet/1
0 : 17 7 1 201 fffffe810e8a98e0 idle/1
0 16 3 5 200 fffffe822de9e080 lnxsyswq lnxsyswq
0 15 3 5 200 fffffe822de9e4a0 sysmon smtaskq
0 14 3 2 200 fffffe822de9e8c0 pmfsuspend pmfsuspend
0 13 3 4 200 fffffe822e2c5060 pmfevent pmfevent
0 12 3 0 200 fffffe822e2c5480 sopendfree sopendfr
0 11 3 6 200 fffffe822e2c58a0 nfssilly nfssilly
0 10 3 3 200 fffffe822f6dc040 cachegc cachegc
0 9 3 3 200 fffffe822f6dc460 vdrain vdrain
0 8 3 3 200 fffffe822f6dc880 modunload mod_unld
0 7 3 0 200 fffffe822f6fd020 xcall/0 xcall
0 : 6 7 0 200 fffffe822f6fd440 softser/0
0 : 5 7 0 200 fffffe822f6fd860 softclk/0
0 4 1 0 200 fffffe822f719000 softbio/0
0 3 1 0 200 fffffe822f719420 softnet/0
0 2 1 0 201 fffffe822f719840 idle/0
0 1 3 2 200 ffffffff81481b20 swapper uvm
db{0}: trace/t 0t9372
trace: pid 9372 lid 1 at 0xffff80009f6f78e0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
rw_vector_enter() at netbsd:rw_vector_enter+0x3e8
genfs_lock() at netbsd:genfs_lock+0x3c
VOP_LOCK() at netbsd:VOP_LOCK+0x67
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
781169a3e2ca:
db{0}: trace/t 0t8217
trace: pid 8217 lid 1 at 0xffff80009f6be8e0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
rw_vector_enter() at netbsd:rw_vector_enter+0x3e8
genfs_lock() at netbsd:genfs_lock+0x3c
VOP_LOCK() at netbsd:VOP_LOCK+0x67
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7ac9b2a3e2ca:
db{0}: trace/t 0t6001
trace: pid 6001 lid 1 at 0xffff80009eff7950
sleepq_block() at netbsd:sleepq_block+0x97
cv_wait() at netbsd:cv_wait+0xfb
biowait() at netbsd:biowait+0x5d
bwrite() at netbsd:bwrite+0x153
ffs_balloc() at netbsd:ffs_balloc+0x1b06
ufs_gop_alloc() at netbsd:ufs_gop_alloc+0xa1
ufs_balloc_range() at netbsd:ufs_balloc_range+0x136
ffs_write() at netbsd:ffs_write+0x4d7
VOP_WRITE() at netbsd:VOP_WRITE+0x37
vn_write() at netbsd:vn_write+0xec
dofilewrite() at netbsd:dofilewrite+0x97
sys_write() at netbsd:sys_write+0x5f
syscall() at netbsd:syscall+0x1d8
--- syscall (number 4) ---
79de3d23e02a:
db{0}: trace/t 0t5868
trace: pid 5868 lid 1 at 0xffff80009f0c1a50
sleepq_block() at netbsd:sleepq_block+0x97
cv_wait() at netbsd:cv_wait+0xfb
biowait() at netbsd:biowait+0x5d
bwrite() at netbsd:bwrite+0x153
ffs_update.part.3() at netbsd:ffs_update.part.3+0x292
ufs_makeinode() at netbsd:ufs_makeinode+0x134
ufs_create() at netbsd:ufs_create+0x31
VOP_CREATE() at netbsd:VOP_CREATE+0x3d
vn_open() at netbsd:vn_open+0x351
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7d9dfe63e2ca:
db{0}: trace/t 0t11951
trace: pid 11951 lid 1 at 0xffff80009f7a1c80
sleepq_block() at netbsd:sleepq_block+0x97
cv_timedwait_sig() at netbsd:cv_timedwait_sig+0x107
sbwait() at netbsd:sbwait+0x69
soreceive() at netbsd:soreceive+0x1233
dofileread() at netbsd:dofileread+0x8f
sys_read() at netbsd:sys_read+0x5f
syscall() at netbsd:syscall+0x1d8
--- syscall (number 3) ---
6fd303c3e22a:
db{0}: trace/t 0t9862
trace: pid 9862 lid 1 at 0xffff80009f772b60
sleepq_block() at netbsd:sleepq_block+0x97
cv_wait_sig() at netbsd:cv_wait_sig+0xfe
do_sys_waitid() at netbsd:do_sys_waitid+0x44e
do_sys_wait() at netbsd:do_sys_wait+0x8f
sys___wait450() at netbsd:sys___wait450+0x42
syscall() at netbsd:syscall+0x1d8
--- syscall (number 449) ---
7cc8fa83e06a:
db{0}: trace/t 0t10228
trace: pid 10228 lid 2 at 0xffff80009f71de40
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1d8
--- syscall (number 417) ---
767a01c3e10a:
db{0}: trace/t 0t7123
trace: pid 7123 lid 1 at 0xffff80009f5d9c10
sleepq_block() at netbsd:sleepq_block+0x97
cv_wait() at netbsd:cv_wait+0xfb
biowait() at netbsd:biowait+0x5d
bwrite() at netbsd:bwrite+0x153
VOP_BWRITE() at netbsd:VOP_BWRITE+0x51
ufs_dirremove() at netbsd:ufs_dirremove+0xd8
ufs_remove() at netbsd:ufs_remove+0x142
VOP_REMOVE() at netbsd:VOP_REMOVE+0x37
do_sys_unlinkat() at netbsd:do_sys_unlinkat+0x237
syscall() at netbsd:syscall+0x1d8
--- syscall (number 10) ---
7973de4fb2aa:
db{0}: trace/t 0t9662
trace: pid 9662 lid 1 at 0xffff80009f074b60
sleepq_block() at netbsd:sleepq_block+0x97
cv_wait_sig() at netbsd:cv_wait_sig+0xfe
do_sys_waitid() at netbsd:do_sys_waitid+0x44e
do_sys_wait() at netbsd:do_sys_wait+0x8f
sys___wait450() at netbsd:sys___wait450+0x42
syscall() at netbsd:syscall+0x1d8
--- syscall (number 449) ---
72c419a3e06a:
db{0}: trace/t 0t9681
trace: pid 9681 lid 1 at 0xffff80009f0fdb60
sleepq_block() at netbsd:sleepq_block+0x97
cv_wait_sig() at netbsd:cv_wait_sig+0xfe
do_sys_waitid() at netbsd:do_sys_waitid+0x44e
do_sys_wait() at netbsd:do_sys_wait+0x8f
sys___wait450() at netbsd:sys___wait450+0x42
syscall() at netbsd:syscall+0x1d8
--- syscall (number 449) ---
74946003e06a:
db{0}: trace/t 0t1019
trace: pid 1019 lid 1 at 0xffff80009de6e8a0
sleepq_block() at netbsd:sleepq_block+0x97
cv_wait() at netbsd:cv_wait+0xfb
biowait() at netbsd:biowait+0x5d
bwrite() at netbsd:bwrite+0x153
ffs_update.part.3() at netbsd:ffs_update.part.3+0x292
ffs_truncate() at netbsd:ffs_truncate+0x1046
ufs_truncate_retry() at netbsd:ufs_truncate_retry+0x79
ufs_setattr() at netbsd:ufs_setattr+0x44c
VOP_SETATTR() at netbsd:VOP_SETATTR+0x34
vn_open() at netbsd:vn_open+0x26e
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
79538a23e2ca:
db{0}: trace/t 0t383
trace: pid 383 lid 1 at 0x1
sleepq_block() at netbsd:sleepq_block+0x110
fatal page fault in supervisor mode
trap type 6 code 0 rip 0xffffffff80238eaa cs 0x8 rflags 0x10246 cr2 0x9 ilevel 0x8 rsp 0xffff80008f69afe0
curlwp 0xfffffe822f6fd440 pid 0.6 lowest kstack 0xffff80008f6972c0
kernel: page fault trap, code=0
Faulted in DDB; continuing...
db{0}: c
fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0x9 ilevel 0x8 rsp 0xffff80008f69bcf8
curlwp 0xfffffe822f6fd440 pid 0.6 lowest kstack 0xffff80008f6972c0
Stopped in pid 0.6 (system) at netbsd:breakpoint+0x5: leave
db{0}: ps
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
5868 1 3 5 0 fffffe8215553960 postdrop biowait
11951 1 3 5 80 fffffe81da398080 sendmail netio
6001 1 3 7 0 fffffe82180044a0 vmstat biowait
9372 1 3 2 0 fffffe81f0125420 tail tstile
8217 1 3 6 0 fffffe81efc4d320 tail tstile
9862 1 3 3 80 fffffe81d8509060 cron wait
4632 1 3 0 80 fffffe822b3bb080 ssh select
4405 2 3 4 80 fffffe8224c45b40 expect select
4405 1 3 0 80 fffffe821f14e540 expect parked
4495 1 3 0 80 fffffe8226a1e8a0 sh wait
9356 1 3 5 80 fffffe821b5d0200 perl pipe_rd
6548 1 3 5 80 fffffe8221863a60 sh wait
10228: 2 7 0 0 fffffe81d9f12440 expect
10228 1 3 7 80 fffffe820e8f9160 expect parked
7679 1 3 2 80 fffffe822c143460 sh wait
12514 1 3 2 80 fffffe821daaa100 perl pipe_rd
10055 1 3 2 80 fffffe82140c9140 sh wait
11571 1 3 0 80 fffffe8221215460 ssh select
7806 1 3 4 80 fffffe82205b9280 sh wait
9266 1 3 0 80 fffffe8221215880 ssh select
10356 1 3 5 80 fffffe8229b8e760 sh wait
7934 1 3 0 80 fffffe8225e1f6e0 ssh select
10592 1 3 1 80 fffffe82170dc500 sh wait
9681 1 3 2 80 fffffe820e8f9580 perl wait
9305 1 3 1 80 fffffe8219f67b60 sh wait
9908 1 3 0 80 fffffe8225fad840 ssh select
8779 1 3 0 80 fffffe82205b96a0 sh wait
6856 1 3 0 80 fffffe8215553120 ssh select
8803 1 3 1 80 fffffe81eb91a620 sh wait
8994 1 3 0 80 fffffe81d9f12860 ssh select
10470 1 3 6 80 fffffe8225fad420 sh wait
9662 1 3 6 80 fffffe821a15d8e0 perl wait
5076 1 3 5 80 fffffe82140c9980 sh wait
7123 1 3 7 0 fffffe81f997ca80 perl biowait
8956 1 3 5 80 fffffe8225f1c920 sh wait
9975 1 3 0 80 fffffe81f0b356e0 ssh select
5270 1 3 2 80 fffffe822b3bb4a0 sh wait
8794 1 3 0 80 fffffe822b9d6760 ssh select
5168 1 3 2 80 fffffe8215553540 sh wait
8565 1 3 0 80 fffffe821951e020 ssh select
8522 1 3 0 80 fffffe822a8f4100 sh wait
8946 1 3 0 80 fffffe820dddda20 ssh select
8677 1 3 5 80 fffffe822d94a560 sh wait
6875 1 3 0 80 fffffe821951e440 ssh select
8644 1 3 7 80 fffffe821d24b660 sh wait
7694 1 3 0 80 fffffe82251dfb20 ssh select
7939 1 3 3 80 fffffe820354a5a0 sh wait
5953 1 3 5 80 fffffe81f997c240 ssh select
5952 1 3 6 80 fffffe8224c45720 sh wait
7773 1 3 0 80 fffffe822140b680 ssh select
6643 1 3 7 80 fffffe822c5ab680 sh wait
6757 1 3 0 80 fffffe81f0125840 ssh select
8943 1 3 4 80 fffffe81fc3e5640 sh wait
7329 1 3 0 80 fffffe822d946780 ssh select
6222 1 3 0 80 fffffe820af7d1a0 sh wait
6930 1 3 0 80 fffffe821a2012a0 ssh select
6708 1 3 6 80 fffffe821a15d0a0 sh wait
5857 1 3 6 80 fffffe8223ee1320 ssh select
7463 1 3 2 80 fffffe821d4af220 sh wait
5817 1 3 0 80 fffffe8222cfd540 ssh select
6855 1 3 5 80 fffffe8225a53780 sh wait
7144 1 3 0 80 fffffe821b5d0620 ssh select
7222 1 3 5 80 fffffe8221863640 sh wait
6589 1 3 0 80 fffffe820af7d5c0 ssh select
7157 1 3 3 80 fffffe822d94a140 sh wait
6877 1 3 2 80 fffffe82039c65e0 ssh select
6057 1 3 7 80 fffffe821d4af640 sh wait
6392 1 3 5 80 fffffe822d71a6e0 ssh select
4924 1 3 4 80 fffffe8222a64100 sh wait
5490 1 3 0 80 fffffe8218a57000 ssh select
5061 1 3 6 80 fffffe81fbd2daa0 sh wait
6508 1 3 0 80 fffffe82251df2e0 ssh select
4832 1 3 6 80 fffffe81fbd2d680 sh wait
5990 1 3 0 80 fffffe8225e1f2c0 ssh select
5243 1 3 2 80 fffffe820dddd600 sh wait
3871 1 3 5 80 fffffe8221476b20 par pause
2696 1 3 5 80 fffffe822aad3900 sh wait
4142 1 3 0 80 fffffe821a62c060 sh wait
4763 1 3 0 80 fffffe82299ed1a0 sh wait
5220 1 3 0 80 fffffe8222a64520 sh wait
4858 1 3 3 80 fffffe821daba920 cron pipe_rd
3551 1 3 0 80 fffffe821de8b9e0 ssh select
4159 1 3 5 80 fffffe8226fc48e0 sh wait
5091 1 3 0 80 fffffe8221863220 ssh select
3848 1 3 0 80 fffffe822c143880 sh wait
4429 1 3 0 80 fffffe82140c9560 ssh select
3178 1 3 0 80 fffffe81eb91aa40 sh wait
4329 1 3 0 80 fffffe820354a180 ssh select
4193 1 3 1 80 fffffe820dddd1e0 sh wait
3408 1 3 0 80 fffffe82251df700 ssh select
4951 1 3 0 80 fffffe8221ea5660 bash wait
4422 1 3 5 80 fffffe820af7d9e0 sshd select
3919 1 3 0 80 fffffe82299aa6a0 sshd select
3312 1 3 0 80 fffffe8229b8eb80 ssh select
3607 1 3 7 80 fffffe8225a53ba0 bash wait
4021 1 3 0 80 fffffe821e8b72e0 sshd select
3876 1 3 0 80 fffffe821daba0e0 sshd select
3633 1 3 1 80 fffffe822aad34e0 ssh select
3679 1 3 7 80 fffffe822a8f4520 bash wait
3250 1 3 0 80 fffffe8224c45300 sshd select
3676 1 3 0 80 fffffe821ffb8200 sshd select
3306 1 3 0 80 fffffe82253fc020 ssh select
3556 1 3 2 80 fffffe821951e860 sh wait
2753 1 3 0 80 fffffe8221215040 ssh select
2752 1 3 0 80 fffffe821a201ae0 bash wait
3789 1 3 0 80 fffffe8228146640 sshd select
2376 1 3 5 80 fffffe820e8f99a0 sshd select
1466 1 3 7 80 fffffe8221476700 bash wait
1782 1 3 0 80 fffffe821daba500 tcsh ttyraw
2030 1 3 2 80 fffffe821ffb8620 sshd select
1515 1 3 5 80 fffffe822c5abaa0 sshd select
1238 1 3 0 80 fffffe821d4afa60 ssh select
853 1 3 3 80 fffffe822d772580 sh wait
1740 1 3 0 80 fffffe821e6ec1e0 ssh select
459 1 3 7 80 fffffe821e6ec600 sh wait
458 1 3 0 80 fffffe821daaa940 ssh select
457 1 3 1 80 fffffe821e6eca20 sh wait
1222 1 3 0 80 fffffe821dd2b1c0 ssh select
1093 1 3 0 80 fffffe821dd2b5e0 sh wait
1092 1 3 0 80 fffffe821dd2ba00 ssh select
835 1 3 0 80 fffffe821de8b1a0 sh wait
1212 1 3 0 80 fffffe821d19a9c0 ssh select
1467 1 3 0 80 fffffe821e0db160 conserver nanoslp
1082 1 3 5 80 fffffe821e0db580 sh wait
1081 1 3 0 80 fffffe821e0db9a0 ssh select
824 1 3 5 80 fffffe821ef36140 sh wait
1079 1 3 0 80 fffffe821ef36560 ssh select
1206 1 3 5 80 fffffe821ef36980 sh wait
1203 1 3 6 80 fffffe821f14e960 conserver select
1324 1 3 0 80 fffffe821df920c0 ssh select
299 1 3 0 80 fffffe821df924e0 sh wait
298 1 3 0 80 fffffe821df92900 ssh select
297 1 3 0 80 fffffe821e2910a0 sh wait
296 1 3 0 80 fffffe821e2914c0 ssh select
1703 1 3 0 80 fffffe821e2918e0 sh wait
1190 1 3 0 80 fffffe821e442080 ssh select
1061 1 3 0 80 fffffe821e4424a0 sh wait
1188 1 3 0 80 fffffe821e4428c0 ssh select
1187 1 3 0 80 fffffe821e6e9060 sh wait
674 1 3 0 80 fffffe821e6e9480 ssh select
289 1 3 0 80 fffffe821e6e98a0 sh wait
288 1 3 0 80 fffffe821ff14040 ssh select
927 1 3 0 80 fffffe821ff14460 sh wait
798 1 3 0 80 fffffe821ff14880 ssh select
1181 1 3 0 80 fffffe822129d020 sh wait
1052 1 3 0 80 fffffe822129d440 ssh select
1051 1 3 0 80 fffffe822dafd120 sh wait
1178 1 3 0 80 fffffe821f670280 ssh select
1177 1 3 0 80 fffffe82283bd680 sh wait
1176 1 3 0 80 fffffe8225e1fb00 ssh select
1431 1 3 7 80 fffffe822129d860 sh wait
790 1 3 0 80 fffffe821edea000 conserver select
1173 1 3 0 80 fffffe821edea420 conserver nanoslp
1044 1 3 0 80 fffffe82299aaac0 ssh select
402 1 3 7 80 fffffe822140b260 sh wait
400 1 3 0 80 fffffe82214d7360 conserver select
1293 1 3 0 80 fffffe822d1cc5a0 ssh select
1292 1 3 0 80 fffffe821f670ac0 sh wait
267 1 3 0 80 fffffe82214d7780 ssh select
266 1 3 4 80 fffffe82214d7ba0 sh wait
1415 1 3 5 80 fffffe821f5f2760 conserver select
1030 1 3 0 80 fffffe821f5f2b80 ssh select
1157 1 3 3 80 fffffe82299ed9e0 sh wait
1156 1 3 0 80 fffffe82299ed5c0 ssh select
1155 1 3 0 80 fffffe8229e40ae0 sh wait
386 1 3 0 80 fffffe821f9a1320 ssh select
385 1 3 7 80 fffffe821f9a1740 sh wait
384 1 3 0 80 fffffe821f9a1b60 ssh select
447 1 3 7 80 fffffe82211cb300 sh wait
830 1 3 0 80 fffffe82211cb720 ssh select
1021 1 3 7 80 fffffe82299aa280 sh wait
888 1 3 0 80 fffffe82211cbb40 ssh select
1143 1 3 1 80 fffffe82214762e0 sh wait
176 1 3 0 80 fffffe8220dab2c0 ssh select
815 1 3 0 80 fffffe8220dab6e0 sh wait
878 1 3 0 80 fffffe8220dabb00 ssh select
1069 1 3 0 80 fffffe8220f432a0 sh wait
1196 1 3 7 80 fffffe8220f436c0 ssh select
363 1 3 5 80 fffffe8220f43ae0 sh wait
361 1 3 0 80 fffffe821f6706a0 conserver select
1115 1 3 0 80 fffffe8221cb31e0 ssh select
986 1 3 5 80 fffffe8221cb3600 sh wait
985 1 3 0 80 fffffe8221cb3a20 ssh select
984 1 3 5 80 fffffe822261e1c0 sh wait
1239 1 3 0 80 fffffe822261e5e0 ssh select
854 1 3 5 80 fffffe822261ea00 sh wait
789 1 3 0 80 fffffe822a8ba1a0 ssh select
1108 1 3 5 80 fffffe822a8ba5c0 sh wait
851 1 3 0 80 fffffe822a8ba9e0 ssh select
210 1 3 5 80 fffffe8220918180 sh wait
913 1 3 0 80 fffffe82209185a0 ssh select
208 1 3 5 80 fffffe82209189c0 sh wait
783 1 3 0 80 fffffe8222d0a160 ssh select
974 1 3 0 80 fffffe8222d0a580 sh wait
1165 1 3 0 80 fffffe8222d0a9a0 ssh select
1356 1 3 0 80 fffffe822287f140 sh wait
139 1 3 0 80 fffffe822287f560 ssh select
138 1 3 0 80 fffffe822287f980 sh wait
137 1 3 5 80 fffffe8222cfd120 conserver select
168 1 3 1 80 fffffe822dafd960 conserver select
1127 1 3 5 80 fffffe8228f78620 conserver select
933 1 3 5 80 fffffe822dda2920 getty ttyraw
1031 1 3 0 80 fffffe822dd49100 conserver select
797 1 3 5 80 fffffe8225a53360 conserver select
829 1 3 3 80 fffffe82283bd260 cron nanoslp
1019 1 3 3 0 fffffe822bcd0a00 python2.7 biowait
112 1 3 0 80 fffffe8229b8e340 inetd kqueue
772 1 3 0 80 fffffe8223ee1740 nginx kqueue
803 1 3 0 80 fffffe8223ee1b60 nginx pause
773 1 3 5 80 fffffe8229e406c0 qmgr kqueue
951 1 3 5 80 fffffe8228146a60 pickup kqueue
794 1 3 5 80 fffffe8229e402a0 master kqueue
696 1 3 0 80 fffffe822d1cc180 sshd select
725 1 3 3 80 fffffe822d772160 tac_plus select
686 1 3 6 80 fffffe822806d240 powerd kqueue
659 1 3 0 80 fffffe822806d660 ntpd pause
606 4 3 6 80 fffffe822806da80 dhcpd parked
606 3 3 6 80 fffffe8228146220 dhcpd kqueue
606 2 3 6 80 fffffe8228f78200 dhcpd parked
606 1 3 0 80 fffffe822986b1e0 dhcpd parked
383 1 2 7 0 fffffe8228f78a40 syslogd
1 1 3 3 80 fffffe810f45a0a0 init wait
0 123 3 3 200 fffffe822986ba20 npfgc-0 xchicv
0 122 3 5 200 fffffe822d1cc9c0 ipmi0 ipmi0
0 121 3 0 200 fffffe822dda2500 physiod physiod
0 120 3 0 200 fffffe822dd49520 aiodoned aiodoned
0 119 3 0 200 fffffe822dd49940 ioflush syncer
0 118 3 0 200 fffffe822dda20e0 pgdaemon pgdaemon
0 115 3 0 280 fffffe810f4260c0 spkr1 bellcv
0 114 3 4 280 fffffe810f4264e0 audiomix play
0 113 3 4 280 fffffe810ed322a0 audiorec record
0 112 3 5 200 fffffe810f2c9720 atapibus0 sccomp
0 110 3 0 200 fffffe810f2ba700 usb2 usbevt
0 109 3 0 200 fffffe810f2ba2e0 usb1 usbevt
0 108 3 0 200 fffffe810f2bab20 usb3 usbevt
0 107 3 0 200 fffffe810f2c9300 usb0 usbevt
0 106 3 5 200 fffffe810f2e0b60 usb4 usbevt
0 105 3 5 200 fffffe810f426900 usb5 usbevt
0 104 3 0 200 fffffe810f45a4c0 rt_free rt_free
0 103 3 0 200 fffffe810f45a8e0 unpgc unpgc
0 102 3 3 200 fffffe810f463080 key_timehandler key_timehandler
0 101 3 7 200 fffffe810f4634a0 icmp6_wqinput/7 icmp6_wqinput
0 100 3 6 200 fffffe810f4638c0 icmp6_wqinput/6 icmp6_wqinput
0 99 3 5 200 fffffe810f3ec060 icmp6_wqinput/5 icmp6_wqinput
0 98 3 4 200 fffffe810f3ec480 icmp6_wqinput/4 icmp6_wqinput
0 97 3 3 200 fffffe810f3ec8a0 icmp6_wqinput/3 icmp6_wqinput
0 96 3 2 200 fffffe810f38d040 icmp6_wqinput/2 icmp6_wqinput
0 95 3 1 200 fffffe810f38d460 icmp6_wqinput/1 icmp6_wqinput
0 94 3 0 200 fffffe810f2c9b40 icmp6_wqinput/0 icmp6_wqinput
0 93 3 0 200 fffffe810f38d880 nd6_timer nd6_timer
0 92 3 7 200 fffffe810f34e020 icmp_wqinput/7 icmp_wqinput
0 91 3 6 200 fffffe810f34e440 icmp_wqinput/6 icmp_wqinput
0 90 3 5 200 fffffe810f34e860 icmp_wqinput/5 icmp_wqinput
0 89 3 4 200 fffffe810f337000 icmp_wqinput/4 icmp_wqinput
0 88 3 3 200 fffffe810f337420 icmp_wqinput/3 icmp_wqinput
0 87 3 2 200 fffffe810f337840 icmp_wqinput/2 icmp_wqinput
0 86 3 1 200 fffffe810f33d360 icmp_wqinput/1 icmp_wqinput
0 85 3 0 200 fffffe810f33d780 icmp_wqinput/0 icmp_wqinput
0 84 3 6 200 fffffe810f33dba0 rt_timer rt_timer
0 83 3 7 200 fffffe810f1fc2c0 vmem_rehash vmem_rehash
0 82 3 3 200 fffffe810f2d7340 coretemp3 coretemp3
0 81 3 7 200 fffffe810f2d7760 coretemp2 coretemp2
0 80 3 1 200 fffffe810f2d7b80 coretemp1 coretemp1
0 79 3 6 200 fffffe810f2e0320 coretemp0 coretemp0
0 78 3 4 200 fffffe810f2e0740 mfi0 mfi0
0 69 3 0 200 fffffe810f1fc6e0 atabus1 atath
0 68 3 0 200 fffffe810f1fcb00 atabus0 atath
0 66 3 0 200 fffffe810ed326c0 scsibus0 sccomp
0 65 3 5 200 fffffe810ed32ae0 usbtask-dr usbtsk
0 64 3 0 200 fffffe810eca9280 usbtask-hc usbtsk
0 63 3 7 200 fffffe810eca96a0 bnx3 bnx3
0 62 3 1 200 fffffe810eca9ac0 bnx2 bnx2
0 61 3 5 200 fffffe810eb60260 bnx1 bnx1
0 60 3 4 200 fffffe810eb60680 bnx0 bnx0
0 59 3 1 200 fffffe810eb60aa0 ipmi ipmipoll
0 58 3 7 200 fffffe810ea73240 xcall/7 xcall
0 57 1 7 200 fffffe810ea73660 softser/7
0 56 1 7 200 fffffe810ea73a80 softclk/7
0 55 1 7 200 fffffe810ea54220 softbio/7
0 54 1 7 200 fffffe810ea54640 softnet/7
0 : 53 7 7 201 fffffe810ea54a60 idle/7
0 52 3 6 200 fffffe810ea3d200 xcall/6 xcall
0 51 1 6 200 fffffe810ea3d620 softser/6
0 50 1 6 200 fffffe810ea3da40 softclk/6
0 49 1 6 200 fffffe810ea161e0 softbio/6
0 48 1 6 200 fffffe810ea16600 softnet/6
0 : 47 7 6 201 fffffe810ea16a20 idle/6
0 46 3 5 200 fffffe810e9d71c0 xcall/5 xcall
0 45 1 5 200 fffffe810e9d75e0 softser/5
0 44 1 5 200 fffffe810e9d7a00 softclk/5
0 43 1 5 200 fffffe810e9d01a0 softbio/5
0 42 1 5 200 fffffe810e9d05c0 softnet/5
0 : 41 7 5 201 fffffe810e9d09e0 idle/5
0 40 3 4 200 fffffe810e9a1180 xcall/4 xcall
0 39 1 4 200 fffffe810e9a15a0 softser/4
0 38 1 4 200 fffffe810e9a19c0 softclk/4
0 37 1 4 200 fffffe810e97a160 softbio/4
0 36 1 4 200 fffffe810e97a580 softnet/4
0 : 35 7 4 201 fffffe810e97a9a0 idle/4
0 34 3 3 200 fffffe810e943140 xcall/3 xcall
0 33 1 3 200 fffffe810e943560 softser/3
0 32 1 3 200 fffffe810e943980 softclk/3
0 31 1 3 200 fffffe810e934120 softbio/3
0 30 1 3 200 fffffe810e934540 softnet/3
0 : 29 7 3 201 fffffe810e934960 idle/3
0 28 3 2 200 fffffe810e90d100 xcall/2 xcall
0 27 1 2 200 fffffe810e90d520 softser/2
0 26 1 2 200 fffffe810e90d940 softclk/2
0 25 1 2 200 fffffe810e8f60e0 softbio/2
0 24 1 2 200 fffffe810e8f6500 softnet/2
0 : 23 7 2 201 fffffe810e8f6920 idle/2
0 22 3 1 200 fffffe810e8c50c0 xcall/1 xcall
0 21 1 1 200 fffffe810e8c54e0 softser/1
0 20 1 1 200 fffffe810e8c5900 softclk/1
0 19 1 1 200 fffffe810e8a90a0 softbio/1
0 18 1 1 200 fffffe810e8a94c0 softnet/1
0 : 17 7 1 201 fffffe810e8a98e0 idle/1
0 16 3 5 200 fffffe822de9e080 lnxsyswq lnxsyswq
0 15 3 5 200 fffffe822de9e4a0 sysmon smtaskq
0 14 3 2 200 fffffe822de9e8c0 pmfsuspend pmfsuspend
0 13 3 4 200 fffffe822e2c5060 pmfevent pmfevent
0 12 3 0 200 fffffe822e2c5480 sopendfree sopendfr
0 11 3 6 200 fffffe822e2c58a0 nfssilly nfssilly
0 10 3 3 200 fffffe822f6dc040 cachegc cachegc
0 9 3 3 200 fffffe822f6dc460 vdrain vdrain
0 8 3 3 200 fffffe822f6dc880 modunload mod_unld
0 7 3 0 200 fffffe822f6fd020 xcall/0 xcall
0 : 6 7 0 200 fffffe822f6fd440 softser/0
0 : 5 7 0 200 fffffe822f6fd860 softclk/0
0 4 1 0 200 fffffe822f719000 softbio/0
0 3 1 0 200 fffffe822f719420 softnet/0
0 2 1 0 201 fffffe822f719840 idle/0
0 1 3 2 200 ffffffff81481b20 swapper uvm
db{0}: trace/t 0t5868
trace: pid 5868 lid 1 at 0xffff80009f0c1a50
sleepq_block() at netbsd:sleepq_block+0x97
cv_wait() at netbsd:cv_wait+0xfb
biowait() at netbsd:biowait+0x5d
bwrite() at netbsd:bwrite+0x153
ffs_update.part.3() at netbsd:ffs_update.part.3+0x292
ufs_makeinode() at netbsd:ufs_makeinode+0x134
ufs_create() at netbsd:ufs_create+0x31
VOP_CREATE() at netbsd:VOP_CREATE+0x3d
vn_open() at netbsd:vn_open+0x351
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7d9dfe63e2ca:
db{0}: trace/t 0t6001
trace: pid 6001 lid 1 at 0xffff80009eff7950
sleepq_block() at netbsd:sleepq_block+0x97
cv_wait() at netbsd:cv_wait+0xfb
biowait() at netbsd:biowait+0x5d
bwrite() at netbsd:bwrite+0x153
ffs_balloc() at netbsd:ffs_balloc+0x1b06
ufs_gop_alloc() at netbsd:ufs_gop_alloc+0xa1
ufs_balloc_range() at netbsd:ufs_balloc_range+0x136
ffs_write() at netbsd:ffs_write+0x4d7
VOP_WRITE() at netbsd:VOP_WRITE+0x37
vn_write() at netbsd:vn_write+0xec
dofilewrite() at netbsd:dofilewrite+0x97
sys_write() at netbsd:sys_write+0x5f
syscall() at netbsd:syscall+0x1d8
--- syscall (number 4) ---
79de3d23e02a:
db{0}: trace/t 0t9372
trace: pid 9372 lid 1 at 0xffff80009f6f78e0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
rw_vector_enter() at netbsd:rw_vector_enter+0x3e8
genfs_lock() at netbsd:genfs_lock+0x3c
VOP_LOCK() at netbsd:VOP_LOCK+0x67
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
781169a3e2ca:
db{0}: trace/t 0t8217
trace: pid 8217 lid 1 at 0xffff80009f6be8e0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
rw_vector_enter() at netbsd:rw_vector_enter+0x3e8
genfs_lock() at netbsd:genfs_lock+0x3c
VOP_LOCK() at netbsd:VOP_LOCK+0x67
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7ac9b2a3e2ca:
db{0}: trace/t 0t10228
trace: pid 10228 lid 2 at 0xffff80009f71de40
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1d8
--- syscall (number 417) ---
767a01c3e10a:
db{0}: trace/t 0t7123
trace: pid 7123 lid 1 at 0xffff80009f5d9c10
sleepq_block() at netbsd:sleepq_block+0x97
cv_wait() at netbsd:cv_wait+0xfb
biowait() at netbsd:biowait+0x5d
bwrite() at netbsd:bwrite+0x153
VOP_BWRITE() at netbsd:VOP_BWRITE+0x51
ufs_dirremove() at netbsd:ufs_dirremove+0xd8
ufs_remove() at netbsd:ufs_remove+0x142
VOP_REMOVE() at netbsd:VOP_REMOVE+0x37
do_sys_unlinkat() at netbsd:do_sys_unlinkat+0x237
syscall() at netbsd:syscall+0x1d8
--- syscall (number 10) ---
7973de4fb2aa:
db{0}: trace/t 0t1019
trace: pid 1019 lid 1 at 0xffff80009de6e8a0
sleepq_block() at netbsd:sleepq_block+0x97
cv_wait() at netbsd:cv_wait+0xfb
biowait() at netbsd:biowait+0x5d
bwrite() at netbsd:bwrite+0x153
ffs_update.part.3() at netbsd:ffs_update.part.3+0x292
ffs_truncate() at netbsd:ffs_truncate+0x1046
ufs_truncate_retry() at netbsd:ufs_truncate_retry+0x79
ufs_setattr() at netbsd:ufs_setattr+0x44c
VOP_SETATTR() at netbsd:VOP_SETATTR+0x34
vn_open() at netbsd:vn_open+0x26e
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
79538a23e2ca:
db{0}: call cpu_reset
From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Wed, 04 Apr 2018 19:00:49 +0200 (CEST)
a new follow-up with a new hang with similar symptoms.
This time quite a number of processes in "tstile" and a few in
biowait.
Edited DDB console log follows.
login: [-- MARK -- Wed Apr 4 16:00:00 2018]
fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0x78a14a33a1e8
ilevel 0x8 rsp 0xffff80008f69bc78
curlwp 0xfffffe822f6fd440 pid 0.6 lowest kstack 0xffff80008f6972c0
Stopped in pid 0.6 (system) at netbsd:breakpoint+0x5: leave
db{0}: trace
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
bus_space_read_4() at netbsd:bus_space_read_4+0xa
mfi_intr() at netbsd:mfi_intr+0x1e
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
Xhandle_ioapic_level8() at netbsd:Xhandle_ioapic_level8+0x6a
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}: machine cpu 1
using CPU 1
db{0}: trace
x86_pause() at netbsd:x86_pause+0x2
kevent1() at netbsd:kevent1+0x648
sys___kevent50() at netbsd:sys___kevent50+0x33
syscall() at netbsd:syscall+0x1d8
--- syscall (number 435) ---
7ae67143e40a:
db{0}: machine cpu 2
using CPU 2
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: machine cpu 3
using CPU 3
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: machine cpu 4
using CPU 4
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: machine cpu 5
using CPU 5
db{0}: trace
_kernel_lock() at netbsd:_kernel_lock+0x96
nd6_timer_work() at netbsd:nd6_timer_work+0x49
workqueue_worker() at netbsd:workqueue_worker+0xcd
db{0}: machine cpu 6
using CPU 6
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: machine cpu 7
using CPU 7
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: ps
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
5640 1 3 3 0 fffffe817b781140 cron tstile
13063 1 3 2 0 fffffe82223452e0 cron tstile
8710 1 3 2 0 fffffe817b781560 cron wait
8453 1 3 3 0 fffffe817b781980 cron wait
7684 1 3 3 0 fffffe8221017120 cron tstile
1795 1 3 3 0 fffffe8221017540 cron wait
770 1 3 5 0 fffffe821f18a900 cron tstile
769 1 3 3 0 fffffe8221017960 cron tstile
768 1 3 7 0 fffffe820f1fd580 cron tstile
13308 1 3 1 0 fffffe821f786620 cron tstile
14541 1 3 2 0 fffffe821e8b42a0 cron tstile
14861 1 3 7 0 fffffe8180a91100 cron wait
13152 1 3 5 0 fffffe8180a91520 cron wait
12852 1 3 1 0 fffffe8180a91940 cron wait
11020 1 3 2 0 fffffe8188fa90e0 cron wait
17742 1 3 3 0 fffffe8188fa9500 cron wait
15140 1 3 3 0 fffffe8188fa9920 cron tstile
15067 1 3 3 0 fffffe81f7d541c0 cron wait
11390 1 3 3 0 fffffe821fed45a0 cron tstile
14640 1 3 2 0 fffffe822002c160 cron tstile
17396 1 3 2 0 fffffe821cf5aba0 cron wait
12021 1 3 3 0 fffffe82275b0280 cron wait
13774 1 3 5 0 fffffe8229b76a00 cron tstile
18260 1 3 3 0 fffffe81c991a4c0 cron tstile
16295 1 3 1 0 fffffe8226aecae0 cron tstile
18585 1 3 2 0 fffffe81e492e420 cron tstile
17773 1 3 5 0 fffffe822455d900 cron wait
15341 1 3 1 0 fffffe8210253120 cron wait
17117 1 3 2 0 fffffe8221604860 cron wait
14560 1 3 3 0 fffffe822178c000 cron wait
15183 1 3 3 0 fffffe821d68b6a0 cron tstile
17051 1 3 2 0 fffffe821e07b260 cron tstile
15464 1 3 2 0 fffffe81dda0ba60 cron wait
17435 1 3 3 0 fffffe822c6926c0 cron wait
12406 1 3 5 0 fffffe82260f6b40 master tstile
14315 1 3 3 0 fffffe822dba2640 cron tstile
15832 1 3 3 0 fffffe8225d154c0 cron wait
12474 1 3 1 0 fffffe81e492e000 expect xchicv
12734 1 3 1 80 fffffe8221604440 sh wait
14618 1 3 1 80 fffffe822984c1e0 perl pipe_rd
13905 1 3 1 80 fffffe81d2bdb8c0 sh wait
12785 1 3 1 80 fffffe82268cf560 ssh select
14284 2 3 0 80 fffffe82262548c0 expect select
14284 1 3 0 80 fffffe821f700760 expect parked
14946 1 3 6 80 fffffe821a5a60a0 sh wait
12207 1 3 6 80 fffffe82233941e0 perl pipe_rd
13422 1 3 6 80 fffffe81e289f480 sh wait
12051 1 3 0 80 fffffe8220655540 ssh select
9942 2 3 6 80 fffffe81e289f8a0 expect select
9942 1 3 0 80 fffffe81d4261ba0 expect parked
10880 1 3 0 80 fffffe81e9aad860 sh wait
11122 1 3 0 80 fffffe822dba2220 perl pipe_rd
10951 1 3 5 80 fffffe821e07b680 sh wait
17641 1 3 0 80 fffffe820d9cc5a0 ssh select
15330 1 3 4 80 fffffe822ab49b80 sh wait
16497 1 3 0 80 fffffe822da7a980 ssh select
12422 1 3 0 80 fffffe8220e1b780 sh wait
12129 1 3 0 80 fffffe81d06814e0 ssh select
13232 1 3 0 80 fffffe8204535740 sh wait
13244 1 3 0 80 fffffe82221b3240 ssh select
13911 2 3 7 0 fffffe821df300e0 expect tstile
13911 1 3 7 0 fffffe8223394600 expect fdclose
15831 1 3 7 80 fffffe822178c420 sh wait
14617 1 3 7 80 fffffe8222bc08c0 perl pipe_rd
10185 1 3 7 80 fffffe81d2bdb080 sh wait
19345 1 3 0 80 fffffe82278df940 ssh select
15377 2 3 5 80 fffffe82221b3660 expect select
15377 1 3 0 80 fffffe8222bc0080 expect parked
15019 1 3 3 80 fffffe82235045e0 sh wait
12888 1 3 5 80 fffffe82278df520 perl pipe_rd
14017 1 3 3 80 fffffe81d2bdb4a0 sh wait
13129 1 3 0 80 fffffe822366d4e0 ssh select
15179 1 3 6 80 fffffe81e9aad020 sh wait
10490 1 3 0 80 fffffe821d9fa600 ssh select
11328 1 3 6 80 fffffe822dba2a60 sh wait
13321 1 3 0 80 fffffe81d9ca9680 ssh select
13102 1 3 6 80 fffffe81f7d54a00 sh wait
12611 1 3 0 80 fffffe81fb83e300 ssh select
16274 1 3 0 80 fffffe8225d158e0 sh wait
12373 1 3 0 80 fffffe81fecc5240 ssh select
15573 1 3 6 80 fffffe8223a0b480 sh wait
11298 1 3 0 80 fffffe81fa035a20 ssh select
16055 1 3 5 80 fffffe8221e59040 sh wait
15194 1 3 0 80 fffffe8221ffeb40 ssh select
15688 1 3 5 80 fffffe810f2bab20 sh wait
11884 1 3 0 80 fffffe8225a112c0 ssh select
13930 1 3 6 80 fffffe8223efd5c0 sh wait
12554 1 3 0 80 fffffe8210253960 ssh select
13250 1 3 3 80 fffffe822c6922a0 sh wait
9817 1 3 0 80 fffffe8225a11b00 ssh select
16604 1 3 4 80 fffffe822143e460 sh wait
12461 1 3 0 80 fffffe81f2f55a40 ssh select
16683 1 3 0 80 fffffe8223209500 sh wait
10467 1 3 0 80 fffffe81fa0351e0 ssh select
13535 1 3 4 80 fffffe821e2fdb40 sh wait
11893 1 3 0 80 fffffe81ef0c0ac0 ssh select
13544 1 3 0 80 fffffe822455d4e0 sh wait
10768 1 3 0 80 fffffe822984c600 ssh select
11064 1 3 5 80 fffffe822366d900 sh wait
13043 1 3 0 80 fffffe8226254080 ssh select
13228 1 3 3 80 fffffe8222345b20 sh wait
11709 1 3 0 80 fffffe822cfa6360 ssh select
13659 1 3 5 80 fffffe822cc325a0 sh wait
13760 1 3 0 80 fffffe82268cf980 ssh select
13053 1 3 4 80 fffffe82267ce4c0 sh wait
17050 1 3 5 80 fffffe821fcf51c0 ssh select
12595 1 3 0 80 fffffe8223373960 ssh select
9355 1 3 4 80 fffffe822cdda320 sh wait
12295 1 3 1 80 fffffe8222866280 sh wait
11395 1 3 0 80 fffffe822002c9a0 ssh select
13163 1 3 6 80 fffffe82267ce0a0 sh wait
8207 1 3 1 80 fffffe821c608240 par pause
10163 1 3 0 80 fffffe821fcf55e0 sh wait
9328 1 3 0 80 fffffe821d68b280 sh wait
8257 1 3 3 80 fffffe822366d0c0 sh wait
8479 1 3 3 80 fffffe821d74a220 sh wait
10636 1 3 3 80 fffffe8220eaab80 cron pipe_rd
10908 1 3 7 80 fffffe821fed4180 mtr pipe_rd
9031 1 3 0 80 fffffe8209c5b140 mtr-packet select
9787 1 3 7 0 fffffe820c3bb100 mtr tstile
10625 1 3 7 80 fffffe821cf5a360 bash wait
10526 1 3 5 80 fffffe82278df100 sshd select
9327 1 3 0 80 fffffe82214b92e0 sshd select
7813 1 3 0 80 fffffe81f7ac7b20 ssh select
9090 1 3 3 80 fffffe821f18a0c0 sh wait
8423 1 3 0 80 fffffe82228666a0 ssh select
11985 1 3 6 80 fffffe8221fbb260 sh wait
8121 1 3 0 80 fffffe81fb83eb40 ssh select
6432 1 3 2 80 fffffe8220244560 sh wait
8170 1 3 0 80 fffffe8219548060 ssh select
8983 1 3 7 80 fffffe81dda0b220 sh wait
7376 1 3 0 80 fffffe82221b3a80 ssh select
7594 1 3 7 80 fffffe8221fbb680 sh wait
8867 1 3 0 80 fffffe822d95d580 ssh select
6776 1 3 7 80 fffffe8223504a00 sh wait
8541 1 3 0 80 fffffe8205fa79e0 ssh select
7823 1 3 7 80 fffffe81e6eee2c0 sh wait
10488 1 3 0 80 fffffe822178c840 ssh select
6625 1 3 7 80 fffffe822cfa6ba0 sh wait
13028 1 3 0 80 fffffe8226947b20 ssh select
13357 1 3 0 80 fffffe822ab49760 sh wait
7685 1 3 0 80 fffffe821fcf5a00 ssh select
8454 1 3 2 80 fffffe81d2c38460 sh wait
8547 1 3 0 80 fffffe81fb83e720 ssh select
7077 1 3 2 80 fffffe821c608a80 sh wait
7593 1 3 0 80 fffffe822143e040 ssh select
9009 1 3 4 80 fffffe81dda0b640 sh wait
9463 1 3 0 80 fffffe822cbca040 ssh select
8402 1 3 4 80 fffffe822c3905c0 sh wait
8739 1 3 0 80 fffffe8222345700 ssh select
7028 1 3 4 80 fffffe821e07baa0 sh wait
8548 1 3 0 80 fffffe8220ec9b00 ssh select
8630 1 3 7 80 fffffe82214b9700 sh wait
8138 1 3 0 80 fffffe822617c9a0 ssh select
8911 1 3 7 80 fffffe821f18a4e0 sh wait
6100 1 3 0 80 fffffe81d81a3b80 ssh select
8531 1 3 7 80 fffffe8218492440 sh wait
9062 1 3 0 80 fffffe82260f6300 ssh select
6442 1 3 1 80 fffffe8223a0b060 sh wait
7332 1 3 0 80 fffffe8210253540 ssh select
6557 1 3 1 80 fffffe8222c04500 sh wait
9950 1 3 0 80 fffffe822c692ae0 ssh select
11071 1 3 5 80 fffffe81f2f55620 sh wait
8468 1 3 0 80 fffffe821c34b740 ssh select
9075 1 3 7 80 fffffe821a5a64c0 sh wait
5832 1 3 0 80 fffffe8220553480 ssh select
9604 1 3 7 80 fffffe822dd1a100 sh wait
8606 1 3 0 80 fffffe81d82e86c0 ssh select
6015 1 3 2 80 fffffe820d9cc180 sh wait
4455 1 3 0 80 fffffe81d4261780 ssh select
7799 1 3 6 80 fffffe822b379420 sh wait
6867 1 3 0 80 fffffe81fecc5a80 ssh select
4749 1 3 6 80 fffffe8220244140 sh wait
5686 1 3 0 80 fffffe8221e59880 ssh select
9353 1 3 6 80 fffffe8221604020 sh wait
5470 1 3 0 80 fffffe82297c7a60 ssh select
6083 1 3 2 80 fffffe8220eaa340 sh wait
5782 1 3 0 80 fffffe8227899aa0 ssh select
7547 1 3 3 80 fffffe821fcbe5c0 sh wait
5968 1 3 0 80 fffffe8220655120 ssh select
8516 1 3 3 80 fffffe821fed49c0 sh wait
7154 1 3 0 80 fffffe822086d520 ssh select
6185 1 3 6 80 fffffe821c34bb60 sh wait
10365 1 3 0 80 fffffe822617c160 ssh select
4986 1 3 6 80 fffffe8223373540 sh wait
6230 1 3 0 80 fffffe820f1fd9a0 ssh select
4380 1 3 4 80 fffffe822da7a560 sh wait
10536 1 3 0 80 fffffe81f2f55200 ssh select
7189 1 3 6 80 fffffe8204535320 sh wait
9292 1 3 0 80 fffffe8222bc04a0 ssh select
4786 1 3 4 80 fffffe822086d100 sh wait
6470 1 3 0 80 fffffe8209c5b560 ssh select
6162 1 3 6 80 fffffe8204535b60 sh wait
6032 1 3 0 80 fffffe8227899260 ssh select
6221 1 3 6 80 fffffe822455d0c0 sh wait
7423 1 3 0 80 fffffe821d68bac0 ssh select
5297 1 3 4 80 fffffe821e8b46c0 sh wait
6168 1 3 0 80 fffffe821c34b320 ssh select
8225 1 3 6 80 fffffe8220eaa760 sh wait
6670 1 3 0 80 fffffe8229b765e0 ssh select
6562 1 3 4 80 fffffe821f700340 sh wait
4001 1 3 0 80 fffffe822daed540 ssh select
7269 1 3 6 80 fffffe82205538a0 sh wait
5690 1 3 0 80 fffffe822c3909e0 ssh select
6859 1 3 1 80 fffffe82249b2740 sh wait
6538 1 3 0 80 fffffe8219548480 ssh select
7428 1 3 6 80 fffffe822617c580 sh wait
5384 1 3 0 80 fffffe8223dd0420 ssh select
5593 1 3 0 80 fffffe822daed120 sh wait
4848 1 3 0 80 fffffe81d82e82a0 ssh select
5676 1 3 2 80 fffffe82231fc620 sh wait
7822 1 3 0 80 fffffe81e6eeeb00 ssh select
10050 1 3 4 80 fffffe81d2c38880 sh wait
5573 1 3 0 80 fffffe8223373120 ssh select
7342 1 3 4 80 fffffe821e8b4ae0 sh wait
4518 1 3 0 80 fffffe8223efd9e0 ssh select
4760 1 3 0 80 fffffe81d9ca9aa0 sh wait
2041 1 3 0 80 fffffe820c3bb940 tcsh ttyraw
2022 1 3 5 80 fffffe82262544a0 tcsh pause
1821 1 3 6 80 fffffe8223394a20 sshd select
2038 1 3 0 80 fffffe821d9faa20 sshd select
1081 1 3 5 80 fffffe822002c580 conserver select
306 1 3 3 0 fffffe821fcbe9e0 conserver tstile
1431 1 3 6 0 fffffe82267ce8e0 conserver tstile
790 1 3 5 80 fffffe82210a4080 conserver select
1173 1 3 0 80 fffffe822143e880 ssh select
1172 1 3 7 80 fffffe8226947700 sh wait
1171 1 3 5 0 fffffe82210a44a0 conserver uvn_fp2
267 1 3 6 80 fffffe8220553060 conserver select
361 1 3 4 0 fffffe822247eb00 conserver tstile
136 1 3 5 0 fffffe8224f80180 conserver tstile
168 1 3 3 0 fffffe822cbca460 conserver tstile
551 1 3 5 80 fffffe82297c7640 conserver select
1027 1 3 0 80 fffffe8228d9ea80 getty ttyraw
828 1 3 0 80 fffffe8229796a40 conserver select
1022 1 3 5 80 fffffe822cbca880 conserver select
1019 1 3 3 80 fffffe82249b2b60 cron nanoslp
889 1 3 7 0 fffffe8226aec2a0 python2.7 biowait
751 1 3 5 80 fffffe822a4fa020 inetd kqueue
806 1 3 0 80 fffffe822a4fa440 nginx kqueue
837 1 3 0 80 fffffe822a4fa860 nginx pause
805 1 3 6 80 fffffe82269472e0 qmgr kqueue
1006 1 3 5 80 fffffe82260f6720 master kqueue
696 1 3 7 80 fffffe82275b06a0 sshd select
710 1 3 7 80 fffffe8229b761c0 tac_plus select
474 1 3 6 80 fffffe8227899680 powerd kqueue
693 1 3 0 80 fffffe8228d9e660 ntpd pause
607 4 3 3 80 fffffe8228d9e240 dhcpd parked
607 3 3 6 80 fffffe822d95d160 dhcpd kqueue
607 2 3 3 0 fffffe822984ca20 dhcpd biowait
607 1 3 7 80 fffffe8229796620 dhcpd parked
391 : 1 7 1 0 fffffe8229796200 syslogd
1 1 3 7 80 fffffe810f599900 init wait
0 123 3 5 200 fffffe822cc32180 npfgc-0 xchicv
0 122 3 7 200 fffffe822cc329c0 ipmi0 ipmi0
0 121 3 0 200 fffffe810f2e0b60 physiod physiod
0 120 3 0 200 fffffe822dd1a520 aiodoned aiodoned
0 119 3 7 200 fffffe822dd1a940 ioflush biolock
0 118 3 0 200 fffffe822dd800e0 pgdaemon pgdaemon
0 115 3 5 280 fffffe822dd80500 spkr1 bellcv
0 114 3 0 280 fffffe822dd80920 audiomix play
0 113 3 0 280 fffffe810ed322a0 audiorec record
0 112 3 7 200 fffffe810f5990c0 atapibus0 sccomp
0 110 3 7 200 fffffe810f2c9300 usb4 usbevt
0 109 3 0 200 fffffe810f2c9720 usb2 usbevt
0 108 3 0 200 fffffe810f2c9b40 usb3 usbevt
0 107 3 0 200 fffffe810f2ba700 usb0 usbevt
0 106 3 7 200 fffffe810f2ba2e0 usb5 usbevt
0 105 3 7 200 fffffe810f5994e0 usb1 usbevt
0 104 3 7 200 fffffe810f3a20a0 rt_free rt_free
0 103 3 7 200 fffffe810f3a24c0 unpgc unpgc
0 102 3 0 200 fffffe810f3a28e0 key_timehandler key_timehandler
0 101 3 7 200 fffffe810f3ab080 icmp6_wqinput/7 icmp6_wqinput
0 100 3 6 200 fffffe810f3ab4a0 icmp6_wqinput/6 icmp6_wqinput
0 99 3 5 200 fffffe810f3ab8c0 icmp6_wqinput/5 icmp6_wqinput
0 98 3 4 200 fffffe810f3c4060 icmp6_wqinput/4 icmp6_wqinput
0 97 3 3 200 fffffe810f3c4480 icmp6_wqinput/3 icmp6_wqinput
0 96 3 2 200 fffffe810f3c48a0 icmp6_wqinput/2 icmp6_wqinput
0 95 3 1 200 fffffe810f395040 icmp6_wqinput/1 icmp6_wqinput
0 94 3 0 200 fffffe810f395460 icmp6_wqinput/0 icmp6_wqinput
0 : 93 7 5 200 fffffe810f395880 nd6_timer
0 92 3 7 200 fffffe810f34e020 icmp_wqinput/7 icmp_wqinput
0 91 3 6 200 fffffe810f34e440 icmp_wqinput/6 icmp_wqinput
0 90 3 5 200 fffffe810f34e860 icmp_wqinput/5 icmp_wqinput
0 89 3 4 200 fffffe810f337000 icmp_wqinput/4 icmp_wqinput
0 88 3 3 200 fffffe810f337420 icmp_wqinput/3 icmp_wqinput
0 87 3 2 200 fffffe810f337840 icmp_wqinput/2 icmp_wqinput
0 86 3 1 200 fffffe810f345360 icmp_wqinput/1 icmp_wqinput
0 85 3 0 200 fffffe810f345780 icmp_wqinput/0 icmp_wqinput
0 84 3 0 200 fffffe810f345ba0 rt_timer rt_timer
0 83 3 5 200 fffffe810f1fc2c0 vmem_rehash vmem_rehash
0 82 3 3 200 fffffe810f2d7340 coretemp3 coretemp3
0 81 3 4 200 fffffe810f2d7760 coretemp2 coretemp2
0 80 3 5 200 fffffe810f2d7b80 coretemp1 coretemp1
0 79 3 2 200 fffffe810f2e0320 coretemp0 coretemp0
0 78 3 6 200 fffffe810f2e0740 mfi0 mfi0
0 69 3 5 200 fffffe810f1fc6e0 atabus1 atath
0 68 3 0 200 fffffe810f1fcb00 atabus0 atath
0 66 3 0 200 fffffe810ed326c0 scsibus0 sccomp
0 65 3 1 200 fffffe810ed32ae0 usbtask-dr usbtsk
0 64 3 0 200 fffffe810eca9280 usbtask-hc usbtsk
0 63 3 7 200 fffffe810eca96a0 bnx3 bnx3
0 62 3 5 200 fffffe810eca9ac0 bnx2 bnx2
0 61 3 6 200 fffffe810eb60260 bnx1 bnx1
0 60 3 1 200 fffffe810eb60680 bnx0 bnx0
0 59 3 0 200 fffffe810eb60aa0 ipmi ipmipoll
0 58 3 7 200 fffffe810ea73240 xcall/7 xcall
0 57 1 7 200 fffffe810ea73660 softser/7
0 56 3 7 200 fffffe810ea73a80 softclk/7 tstile
0 55 1 7 200 fffffe810ea54220 softbio/7
0 54 1 7 200 fffffe810ea54640 softnet/7
0 : 53 7 7 201 fffffe810ea54a60 idle/7
0 52 3 6 200 fffffe810ea3d200 xcall/6 xcall
0 51 1 6 200 fffffe810ea3d620 softser/6
0 50 1 6 200 fffffe810ea3da40 softclk/6
0 49 1 6 200 fffffe810ea161e0 softbio/6
0 48 1 6 200 fffffe810ea16600 softnet/6
0 : 47 7 6 201 fffffe810ea16a20 idle/6
0 46 3 5 200 fffffe810e9d71c0 xcall/5 xcall
0 45 1 5 200 fffffe810e9d75e0 softser/5
0 44 1 5 200 fffffe810e9d7a00 softclk/5
0 43 1 5 200 fffffe810e9d01a0 softbio/5
0 42 1 5 200 fffffe810e9d05c0 softnet/5
0 41 1 5 201 fffffe810e9d09e0 idle/5
0 40 3 4 200 fffffe810e9a1180 xcall/4 xcall
0 39 1 4 200 fffffe810e9a15a0 softser/4
0 38 1 4 200 fffffe810e9a19c0 softclk/4
0 37 1 4 200 fffffe810e97a160 softbio/4
0 36 1 4 200 fffffe810e97a580 softnet/4
0 : 35 7 4 201 fffffe810e97a9a0 idle/4
0 34 3 3 200 fffffe810e943140 xcall/3 xcall
0 33 1 3 200 fffffe810e943560 softser/3
0 32 1 3 200 fffffe810e943980 softclk/3
0 31 1 3 200 fffffe810e934120 softbio/3
0 30 1 3 200 fffffe810e934540 softnet/3
0 : 29 7 3 201 fffffe810e934960 idle/3
0 28 3 2 200 fffffe810e90d100 xcall/2 xcall
0 27 1 2 200 fffffe810e90d520 softser/2
0 26 1 2 200 fffffe810e90d940 softclk/2
0 25 1 2 200 fffffe810e8f60e0 softbio/2
0 24 1 2 200 fffffe810e8f6500 softnet/2
0 : 23 7 2 201 fffffe810e8f6920 idle/2
0 22 3 1 200 fffffe810e8c50c0 xcall/1 xcall
0 21 1 1 200 fffffe810e8c54e0 softser/1
0 20 1 1 200 fffffe810e8c5900 softclk/1
0 19 1 1 200 fffffe810e8a90a0 softbio/1
0 18 1 1 200 fffffe810e8a94c0 softnet/1
0 17 1 1 201 fffffe810e8a98e0 idle/1
0 16 3 1 200 fffffe822de9e080 lnxsyswq lnxsyswq
0 15 3 0 200 fffffe822de9e4a0 sysmon smtaskq
0 14 3 4 200 fffffe822de9e8c0 pmfsuspend pmfsuspend
0 13 3 7 200 fffffe822e2c5060 pmfevent pmfevent
0 12 3 0 200 fffffe822e2c5480 sopendfree sopendfr
0 11 3 2 200 fffffe822e2c58a0 nfssilly nfssilly
0 10 3 0 200 fffffe822f6dc040 cachegc cachegc
0 9 3 6 200 fffffe822f6dc460 vdrain vdrain
0 8 3 6 200 fffffe822f6dc880 modunload mod_unld
0 7 3 0 200 fffffe822f6fd020 xcall/0 xcall
0 : 6 7 0 200 fffffe822f6fd440 softser/0
0 5 1 0 200 fffffe822f6fd860 softclk/0
0 4 1 0 200 fffffe822f719000 softbio/0
0 3 1 0 200 fffffe822f719420 softnet/0
0 : 2 7 0 201 fffffe822f719840 idle/0
0 1 3 4 200 ffffffff81481b20 swapper uvm
db{0}: trace/t 0t5640
trace: pid 5640 lid 1 at 0xffff80009f9cf8f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t13063
trace: pid 13063 lid 1 at 0xffff80009eb148f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t7684
trace: pid 7684 lid 1 at 0xffff80009f9a88f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t770
trace: pid 770 lid 1 at 0xffff80009ec898f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t769
trace: pid 769 lid 1 at 0xffff80009f9968f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t768
trace: pid 768 lid 1 at 0xffff80009f1098f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t13308
trace: pid 13308 lid 1 at 0xffff80009edae8f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t14541
trace: pid 14541 lid 1 at 0xffff80009ee768f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t15140
trace: pid 15140 lid 1 at 0xffff80009f9688f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t11390
trace: pid 11390 lid 1 at 0xffff80009ed2e8f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t14640
trace: pid 14640 lid 1 at 0xffff80009ed248f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t13774
trace: pid 13774 lid 1 at 0xffff80009de6d8f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t18260
trace: pid 18260 lid 1 at 0xffff80009f6bd8f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t16295
trace: pid 16295 lid 1 at 0xffff80009e6568f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t18585
trace: pid 18585 lid 1 at 0xffff80009f66a8f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t15183
trace: pid 15183 lid 1 at 0xffff80009ee448f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t17051
trace: pid 17051 lid 1 at 0xffff80009ee048f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t12406
trace: pid 12406 lid 1 at 0xffff80009e6b97a0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
check_exec() at netbsd:check_exec+0x4f
execve_loadvm() at netbsd:execve_loadvm+0x247
execve1() at netbsd:execve1+0x28
syscall() at netbsd:syscall+0x1d8
--- syscall (number 59) ---
71a7b203e4aa:
db{0}: trace/t 0t14315
trace: pid 14315 lid 1 at 0xffff80009ea7d8f0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x460
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7344c183e2ca:
db{0}: trace/t 0t13911
trace: pid 13911 lid 2 at 0xffff80009ecc4ab0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x74d
VOP_POLL() at netbsd:VOP_POLL+0x37
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1d8
--- syscall (number 417) ---
713dd9e3e10a:
db{0}: trace/t 0t9787
trace: pid 9787 lid 1 at 0xffff80009ef7cbe0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x74d
VOP_LOCK() at netbsd:VOP_LOCK+0x4d
vn_lock() at netbsd:vn_lock+0x90
vn_write() at netbsd:vn_write+0x9b
dofilewrite() at netbsd:dofilewrite+0x97
sys_write() at netbsd:sys_write+0x5f
syscall() at netbsd:syscall+0x1d8
--- syscall (number 4) ---
6fa9c5e3e02a:
db{0}: trace/t 0t306
trace: pid 306 lid 1 at 0xffff80009ed38ab0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x74d
VOP_POLL() at netbsd:VOP_POLL+0x37
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1d8
--- syscall (number 417) ---
7b059de3e10a:
db{0}: trace/t 0t1431
trace: pid 1431 lid 1 at 0xffff80009ec29ab0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x74d
VOP_POLL() at netbsd:VOP_POLL+0x37
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1d8
--- syscall (number 417) ---
7b059de3e10a:
db{0}: trace/t 0t1171
trace: pid 1171 lid 1 at 0xffff80009ebce8c0
sleepq_block() at netbsd:sleepq_block+0x97
mtsleep() at netbsd:mtsleep+0x102
uvn_findpage() at netbsd:uvn_findpage+0xaf
uvn_findpages() at netbsd:uvn_findpages+0x78
genfs_getpages() at netbsd:genfs_getpages+0x5f9
VOP_GETPAGES() at netbsd:VOP_GETPAGES+0x4f
ufs_balloc_range() at netbsd:ufs_balloc_range+0xf8
ffs_write() at netbsd:ffs_write+0x4d7
VOP_WRITE() at netbsd:VOP_WRITE+0x37
vn_write() at netbsd:vn_write+0xec
dofilewrite() at netbsd:dofilewrite+0x97
sys_write() at netbsd:sys_write+0x5f
syscall() at netbsd:syscall+0x1d8
--- syscall (number 4) ---
7b059de3e02a:
db{0}: trace/t 0t361
trace: pid 361 lid 1 at 0xffff80009eae9ab0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x74d
VOP_POLL() at netbsd:VOP_POLL+0x37
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1d8
--- syscall (number 417) ---
7b059de3e10a:
db{0}: trace/t 0t136
trace: pid 136 lid 1 at 0xffff80009e9f3ab0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x74d
VOP_POLL() at netbsd:VOP_POLL+0x37
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1d8
--- syscall (number 417) ---
7b059de3e10a:
db{0}: trace/t 0t168
trace: pid 168 lid 1 at 0xffff80009e86dab0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
fstrans_start() at netbsd:fstrans_start+0x74d
VOP_POLL() at netbsd:VOP_POLL+0x37
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1d8
--- syscall (number 417) ---
7b059de3e10a:
db{0}: trace/t 0t607
trace: pid 607 lid 4 at 0xffff80009e0d9e10
sleepq_block() at netbsd:sleepq_block+0x106
lwp_park() at netbsd:lwp_park+0x107
sys____lwp_park60() at netbsd:sys____lwp_park60+0x52
syscall() at netbsd:syscall+0x1d8
--- syscall (number 478) ---
74d67e09998a:
db{0}: trace fffffe810f2e0740
fffffe810ea0cd18:
db{0}: trace/a fffffe810f2e0740
trace: pid 0 lid 78 at 0xffff80009db7ce60
sleepq_block() at netbsd:sleepq_block+0x97
cv_wait() at netbsd:cv_wait+0xfb
workqueue_worker() at netbsd:workqueue_worker+0x8f
db{0}: trace/t 0t889
trace: pid 889 lid 1 at 0xffff80009e6788a0
sleepq_block() at netbsd:sleepq_block+0x97
cv_wait() at netbsd:cv_wait+0xfb
biowait() at netbsd:biowait+0x5d
bwrite() at netbsd:bwrite+0x153
ffs_update.part.3() at netbsd:ffs_update.part.3+0x292
ffs_truncate() at netbsd:ffs_truncate+0x1046
ufs_truncate_retry() at netbsd:ufs_truncate_retry+0x79
ufs_setattr() at netbsd:ufs_setattr+0x44c
VOP_SETATTR() at netbsd:VOP_SETATTR+0x34
vn_open() at netbsd:vn_open+0x26e
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1d8
--- syscall (number 5) ---
7444cbe3e2ca:
db{0}: trace/a fffffe810ea73a80
trace: pid 0 lid 56 at 0xffff80008f7e2e20
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x35c
mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
ip_slowtimo() at netbsd:ip_slowtimo+0x10
pfslowtimo() at netbsd:pfslowtimo+0x31
callout_softclock() at netbsd:callout_softclock+0x201
softint_dispatch() at netbsd:softint_dispatch+0xd3
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f7e2ff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}:
db{0}: call cpu_reset
From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Thu, 05 Apr 2018 09:20:14 +0200 (CEST)
Another morning, another wedge:
fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0x7641663b8000 ilevel 0x8 rsp 0xffff80008f69bcf8
curlwp 0xfffffe822f6eb440 pid 0.6 lowest kstack 0xffff80008f6972c0
Stopped in pid 0.6 (system) at netbsd:breakpoint+0x5: leave
db{0}: trace
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
DDB lost frame for netbsd:Xdoreti, trying 0xffff80008f69bd70
Xdoreti() at netbsd:Xdoreti
--- interrupt ---
Bad frame pointer: 0xffff800000000000
202:
db{0}: machine cpu 1
using CPU 1
db{0}: tra
x86_pause() at netbsd:x86_pause+0x2
cdev_open() at netbsd:cdev_open+0xaf
spec_open() at netbsd:spec_open+0x260
VOP_OPEN() at netbsd:VOP_OPEN+0x33
vn_open() at netbsd:vn_open+0x28b
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1ed
--- syscall (number 5) ---
799d4323e2ca:
db{0}: machine cpu 2
using CPU 2
db{0}: tra
x86_pause() at netbsd:x86_pause+0x2
nd6_timer_work() at netbsd:nd6_timer_work+0x49
workqueue_worker() at netbsd:workqueue_worker+0xcd
db{0}: machine cpu 3
using CPU 3
db{0}: tra
x86_pause() at netbsd:x86_pause
_kernel_lock() at netbsd:_kernel_lock+0x183
kevent1() at netbsd:kevent1+0x648
sys___kevent50() at netbsd:sys___kevent50+0x33
syscall() at netbsd:syscall+0x1ed
--- syscall (number 435) ---
79a32843e40a:
db{0}: machine cpu 4
using CPU 4
db{0}: tra
_kernel_lock() at netbsd:_kernel_lock+0x1a1
kqueue_register() at netbsd:kqueue_register+0x431
kevent1() at netbsd:kevent1+0x150
sys___kevent50() at netbsd:sys___kevent50+0x33
syscall() at netbsd:syscall+0x1ed
--- syscall (number 435) ---
7aec56c3e40a:
db{0}: machine cpu 5
using CPU 5
db{0}: tra
x86_pause() at netbsd:x86_pause+0x2
sleepq_block() at netbsd:sleepq_block+0x1c5
cv_timedwait() at netbsd:cv_timedwait+0x131
ipmi_thread() at netbsd:ipmi_thread+0x2f4
db{0}: machine cpu 6
using CPU 6
db{0}: tra
x86_pause() at netbsd:x86_pause
_kernel_lock() at netbsd:_kernel_lock+0x183
VOP_POLL() at netbsd:VOP_POLL+0x7a
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
776367e3e10a:
db{0}: machine cpu 7
using CPU 7
db{0}: tra
x86_pause() at netbsd:x86_pause
_kernel_lock() at netbsd:_kernel_lock+0x183
VFS_ROOT() at netbsd:VFS_ROOT+0x49
lookup_once() at netbsd:lookup_once+0x36d
namei_tryemulroot() at netbsd:namei_tryemulroot+0x413
namei() at netbsd:namei+0x29
vn_open() at netbsd:vn_open+0x92
do_open() at netbsd:do_open+0x112
do_sys_openat() at netbsd:do_sys_openat+0x68
sys_open() at netbsd:sys_open+0x24
syscall() at netbsd:syscall+0x1ed
--- syscall (number 5) ---
74ec32c3e2ca:
db{0}: show uvmexp
Current UVM status:
pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12, ncolors=8
2028615 VM pages: 413947 active, 1908 inactive, 1814 wired, 1282873 free
pages 43514 anon, 367957 file, 6198 exec
freemin=2048, free-target=2730, wired-max=676205
resv-pg=1, resv-kernel=40, zeropages=925073
cpu0:
faults=3009472, traps=3008330, intrs=6915999, ctxswitch=2672825
softint=8300569, syscalls=14016898
cpu1:
faults=1117777, traps=1117078, intrs=0, ctxswitch=1020296
softint=340527, syscalls=5307734
cpu2:
faults=1518940, traps=1517481, intrs=0, ctxswitch=944964
softint=347368, syscalls=4438330
cpu3:
faults=975558, traps=974861, intrs=0, ctxswitch=894789
softint=135439, syscalls=4914018
cpu4:
faults=1339014, traps=1337920, intrs=0, ctxswitch=989824
softint=301182, syscalls=5570074
cpu5:
faults=2229358, traps=2229534, intrs=0, ctxswitch=2649435
softint=222674, syscalls=12788873
cpu6:
faults=1292311, traps=1291923, intrs=0, ctxswitch=1382918
softint=408535, syscalls=6133777
cpu7:
faults=1189005, traps=1189014, intrs=0, ctxswitch=1574424
softint=308728, syscalls=6615055
fault counts:
noram=0, noanon=0, pgwait=0, pgrele=0
ok relocks(total)=12242(12242), anget(retrys)=3183917(0), amapcopy=992662
neighbor anon/obj pg=1091862/15499112, gets(lock/unlock)=5789978/12242
cases: anon=2567450, anoncow=616481, obj=4466484, prcopy=1323525, przero=310
4330
daemon and swap counts:
woke=0, revs=0, scans=0, obscans=0, anscans=0
busy=0, freed=0, reactivate=0, deactivate=0
pageouts=0, pending=0, nswget=0
nswapdev=1, swpgavail=32767
swpages=32767, swpginuse=0, swpgonly=0, paging=0
db{0}: ps
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
6864 : 1 7 1 0 fffffe821e9b4a20 expect
5145 1 3 1 80 fffffe821e6ae220 sh wait
23761: 1 7 7 0 fffffe8220dabac0 sh
11818 1 3 7 0 fffffe820e438340 sh wait
23101 1 3 7 80 fffffe8224f626e0 expect wait
18472 1 3 7 80 fffffe822be434a0 expect pipe_rd
21360 1 3 1 80 fffffe821654d260 perl pipe_rd
9311 1 3 0 80 fffffe8220696420 sh wait
27159 1 3 5 80 fffffe821b40a540 sh wait
21823 1 3 0 80 fffffe821ef7d1a0 perl pipe_rd
12285 1 3 5 80 fffffe821f2c3060 sh wait
21866 1 3 4 80 fffffe820d167520 ssh select
3553 2 3 4 80 fffffe821b54a680 expect select
3553 1 2 0 0 fffffe822dce5540 expect
22723 1 3 5 80 fffffe8205c49600 sh wait
1399 1 3 0 80 fffffe82197b9660 perl pipe_rd
16437 1 3 5 80 fffffe8220ff2200 sh wait
20095 1 3 5 80 fffffe821cfbe180 ssh select
24470 2 3 5 80 fffffe821eb6b900 expect select
24470 1 2 0 0 fffffe822be438c0 expect
26012 1 3 5 80 fffffe821cfbe9c0 sh wait
21306 1 3 3 80 fffffe821ce608c0 perl pipe_rd
637 1 3 5 80 fffffe8222a46140 sh wait
11788: 1 7 0 0 fffffe8210b32640 ssh
13950: 2 7 6 0 fffffe821cf46780 expect
13950 1 2 7 0 fffffe8221bfc660 expect
29647 1 3 1 80 fffffe8227944720 sh wait
5400 1 3 3 80 fffffe820d9fa020 perl pipe_rd
9552 1 3 5 80 fffffe820e2f4580 sh wait
28864 1 3 5 80 fffffe8221d871e0 par pause
15339 1 3 6 80 fffffe820a9eb9c0 sh wait
11851 1 3 2 80 fffffe820a3ca060 sh wait
26008 1 3 2 80 fffffe820e8d7680 sh wait
12131 1 3 2 80 fffffe821cfd7100 sh wait
24961 1 3 2 80 fffffe821cfbe5a0 cron pipe_rd
13841 1 3 4 80 fffffe8206c524c0 bash select
12019 1 3 0 80 fffffe8222d370a0 sshd select
9136 1 3 5 80 fffffe8210b32a60 sshd select
6190 1 3 4 80 fffffe822a6372a0 bash select
22250 1 3 5 80 fffffe821f50a1c0 sshd select
14356 1 3 5 80 fffffe82214a21c0 sshd select
18494 1 3 5 80 fffffe820d717540 ssh select
13452 1 3 4 80 fffffe821e843940 ssh select
5613 1 3 4 80 fffffe82197b9a80 bash wait
4436 1 3 5 80 fffffe82229e34a0 sshd select
12189 1 3 5 80 fffffe8220ff2a40 sshd select
16044 1 3 7 80 fffffe820b365660 bash wait
16554 1 3 3 80 fffffe81ecddf2a0 sshd select
9867 1 3 1 80 fffffe821eb6b0c0 sshd select
13669 1 3 2 80 fffffe8229b70260 pickup kqueue
27835 1 3 5 80 fffffe821eb6b4e0 ssh select
23377 1 3 5 80 fffffe821ce60080 sh wait
14548 1 3 0 80 fffffe822042c860 ssh select
12025 1 3 6 80 fffffe820d717960 sh wait
13955 1 3 2 80 fffffe821a2e3600 ssh select
10145 1 3 5 80 fffffe8207ee9080 ssh select
10991 1 3 6 80 fffffe82246f4420 sh wait
10919 1 3 5 80 fffffe8220ec90c0 sh wait
8354 1 3 5 80 fffffe821b54aaa0 ssh select
12457 1 3 0 80 fffffe822d0b3a40 sh wait
12262 1 3 4 80 fffffe821b67b2e0 ssh select
8647 1 3 1 80 fffffe821cc3a020 sh wait
12615 1 3 6 80 fffffe82209b1a60 ssh select
6504 1 3 6 80 fffffe821e835200 sh wait
10077 1 3 5 80 fffffe821f50a5e0 ssh select
9097 1 3 7 80 fffffe8210032000 sh wait
12776 1 3 3 80 fffffe821e835a40 ssh select
8975 1 3 0 80 fffffe821d71a160 sh wait
15272 1 3 4 80 fffffe820a3ca8a0 ssh select
9186 1 3 1 80 fffffe821ce604a0 sh wait
11697 1 3 0 80 fffffe821b67bb20 ssh select
11201 1 3 6 80 fffffe8227f4c2e0 sh wait
11492 1 3 1 80 fffffe82205f59e0 ssh select
7480 1 3 1 80 fffffe8227944b40 sh wait
10626 1 3 0 80 fffffe8213be7580 ssh select
9457 1 3 6 80 fffffe822776db00 sh wait
10478 1 3 1 80 fffffe820d167100 ssh select
11749 1 3 1 80 fffffe821f762500 sh wait
10715 1 3 0 80 fffffe822373c020 ssh select
11482 1 3 0 80 fffffe8224f622c0 sh wait
10165 1 3 0 80 fffffe820675a0c0 ssh select
8333 1 3 1 80 fffffe8210032840 sh wait
13105 1 3 0 80 fffffe8206c528e0 ssh select
10194 1 3 4 80 fffffe821ec3f920 sh wait
8968 1 3 6 80 fffffe820675a4e0 ssh select
11039 1 3 5 80 fffffe8207ee98c0 sh wait
12997 1 3 6 80 fffffe82214252e0 ssh select
10428 1 3 6 80 fffffe82228c5b80 sh wait
13132 1 3 5 80 fffffe820deaaae0 ssh select
12249 1 3 7 80 fffffe821654daa0 sh wait
12547 1 3 0 80 fffffe8220696840 ssh select
10612 1 3 1 80 fffffe8223545880 sh wait
10685 1 3 0 80 fffffe82164555e0 ssh select
12205 1 3 1 80 fffffe820bb17a40 sh wait
9717 1 3 0 80 fffffe82104fb280 ssh select
8576 1 3 6 80 fffffe821f5e1340 sh wait
9263 1 3 2 80 fffffe82290a06a0 ssh select
8236 1 3 2 80 fffffe821fc844c0 sh wait
7629 1 3 7 80 fffffe821cfd7940 ssh select
10678 1 3 5 80 fffffe8222b97040 sh wait
11663 1 3 0 80 fffffe8222d378e0 ssh select
9517 1 3 6 80 fffffe8229b70aa0 sh wait
8203 1 3 0 80 fffffe8228128ae0 ssh select
10074 1 3 5 80 fffffe821d71a9a0 sh wait
13318 1 3 6 80 fffffe8221d87600 ssh select
8850 1 3 0 80 fffffe820fedeac0 sh wait
13166 1 3 0 80 fffffe822042c020 ssh select
11230 1 3 0 80 fffffe822d8079c0 sh wait
10197 1 3 0 80 fffffe822207a900 ssh select
9056 1 3 1 80 fffffe821b7545c0 sh wait
12554 1 3 7 80 fffffe82130ff360 ssh select
9510 1 3 0 80 fffffe8220964060 sh wait
9456 1 3 7 80 fffffe822d1c45e0 ssh select
11296 1 3 0 80 fffffe821d71a580 sh wait
10743 1 3 0 80 fffffe822a637ae0 ssh select
13144 1 3 0 80 fffffe8226369ba0 sh wait
10605 1 3 4 80 fffffe8213be79a0 ssh select
11156 1 3 0 80 fffffe822aecd580 sh wait
9831 1 3 0 80 fffffe821ef7d9e0 ssh select
10696 1 3 1 80 fffffe8222a46560 sh wait
11143 1 3 2 80 fffffe8210b7aa40 ssh select
10167 1 3 0 80 fffffe82242529c0 sh wait
10843 1 3 3 80 fffffe822d807180 ssh select
9171 1 3 5 80 fffffe8227456760 sh wait
9017 1 3 0 80 fffffe8227456b80 ssh select
8264 1 3 0 80 fffffe822207a0c0 sh wait
7803 1 3 7 80 fffffe82228c5760 ssh select
9042 1 3 0 80 fffffe82203e5300 sh wait
8337 1 3 1 80 fffffe822aeefa80 ssh select
8918 1 3 1 80 fffffe8221ab5680 sh wait
9259 1 3 0 80 fffffe8222d5b480 ssh select
9894 1 3 1 80 fffffe8228895720 sh wait
10382 1 3 5 80 fffffe820deaa2a0 ssh select
12603 1 3 6 80 fffffe822d9d8980 sh wait
10391 1 3 6 80 fffffe8221425700 ssh select
10053 1 3 0 80 fffffe821e7ea4c0 sh wait
8569 1 3 5 80 fffffe820675a900 ssh select
9513 1 3 5 80 fffffe821e7ea0a0 sh wait
7732 1 3 2 80 fffffe821e9b41e0 ssh select
7288 1 3 5 80 fffffe810f2da300 sh wait
7615 1 3 3 80 fffffe8210032420 ssh select
11516 1 3 4 80 fffffe8221bfc240 sh wait
6244 1 3 0 80 fffffe8227944300 ssh select
8987 1 3 0 80 fffffe8221d87a20 sh wait
10342 1 3 3 80 fffffe82203e5b40 ssh select
9118 1 3 0 80 fffffe820a3ca480 sh wait
10063 1 3 7 80 fffffe822bc5da60 ssh select
14307 1 3 7 80 fffffe8220dcd540 sh wait
8607 1 3 4 80 fffffe82146c4140 ssh select
8875 1 3 4 80 fffffe8220ff2620 sh wait
7285 1 3 3 80 fffffe82229e38c0 ssh select
6689 1 3 4 80 fffffe82067c8920 sh wait
11095 1 3 4 80 fffffe821fc840a0 ssh select
9536 1 3 5 80 fffffe821f762920 sh wait
8030 1 3 4 80 fffffe821f50aa00 ssh select
11495 1 3 5 80 fffffe82130ffba0 sh wait
10208 1 3 2 80 fffffe820ed99040 ssh select
8352 1 3 2 80 fffffe820e8d7260 sh wait
8859 1 3 5 80 fffffe8220964480 ssh select
7714 1 3 2 80 fffffe822207c000 sh wait
9647 1 3 0 80 fffffe8207ee94a0 ssh select
10611 1 3 0 80 fffffe8227b8d560 sh wait
7484 1 3 7 80 fffffe82228c5340 ssh select
8137 1 3 7 80 fffffe8227b8d140 sh wait
9308 1 3 7 80 fffffe8222b97460 ssh select
7474 1 3 7 80 fffffe8227456340 sh wait
12342 1 3 0 80 fffffe822a2b5120 ssh select
10235 1 3 0 80 fffffe821e843100 sh wait
8851 1 3 7 80 fffffe82201ea2a0 ssh select
13638 1 3 0 80 fffffe821f2c3480 sh wait
8869 1 3 7 80 fffffe820e438760 ssh select
7630 1 3 7 80 fffffe82281282a0 sh wait
8531 1 3 5 80 fffffe8205c49a20 ssh select
8642 1 3 7 80 fffffe8216455a00 sh wait
8310 1 3 5 80 fffffe822d9bd580 ssh select
11193 1 3 7 80 fffffe822a2b5540 sh wait
6579 1 3 0 80 fffffe822d9d8140 ssh select
12466 1 3 0 80 fffffe8214a1db00 sh wait
7694 1 3 0 80 fffffe8222b97880 ssh select
10187 1 3 0 80 fffffe82146c4980 sh wait
9246 1 3 0 80 fffffe8220696000 ssh select
10052 1 3 0 80 fffffe8221bfca80 sh wait
9722 1 3 3 80 fffffe8210b32220 ssh select
11654 1 3 3 80 fffffe8223ab30e0 sh wait
9180 1 3 2 80 fffffe82229e3080 ssh select
7647 1 3 3 80 fffffe822a6376c0 sh wait
10535 1 3 0 80 fffffe821fb6d240 ssh select
8965 1 3 0 80 fffffe82214a25e0 sh wait
10010 1 3 1 80 fffffe821cddf280 ssh select
8138 1 3 7 80 fffffe8221ab5aa0 sh wait
6940 1 3 0 80 fffffe8210b7a620 ssh select
9251 1 3 0 80 fffffe82104fb6a0 sh wait
8556 1 3 7 80 fffffe822776d2c0 tcsh ttyraw
8477 1 3 0 80 fffffe8222d5b060 sshd select
7657 1 3 1 80 fffffe822aecd160 sshd select
322 1 3 3 0 fffffe8220727b60 conserver tstile
563 1 3 3 0 fffffe8220727740 conserver tstile
534 1 3 3 0 fffffe821ffd7360 conserver tstile
531 1 3 7 80 fffffe82201eaae0 conserver select
400 1 3 3 0 fffffe821ffd7ba0 conserver tstile
900 1 3 3 0 fffffe8220a7f720 conserver tstile
361 1 3 3 0 fffffe82209b1640 conserver tstile
136 1 3 7 80 fffffe8223ab3500 conserver select
549 1 3 4 0 fffffe822d9d8560 conserver tstile
516 1 3 3 0 fffffe82246f4840 conserver tstile
931 1 3 5 80 fffffe822dd780e0 getty ttyraw
857 1 3 2 80 fffffe8226369360 conserver select
511 1 3 5 80 fffffe82286af2c0 conserver select
892 1 3 2 80 fffffe8226369780 cron nanoslp
890 1 3 2 80 fffffe822d0b3200 python2.7 select
107 1 3 4 80 fffffe82290a0280 inetd kqueue
98 1 3 6 80 fffffe822778c320 nginx kqueue
97 1 3 0 80 fffffe822778c740 nginx pause
589 : 1 7 3 0 fffffe82286afb00 qmgr
775 : 1 7 4 0 fffffe82286af6e0 master
471 1 3 5 80 fffffe8229b70680 sshd select
460 1 3 3 0 fffffe822aeef240 tac_plus tstile
430 1 3 7 80 fffffe822d9bd160 powerd kqueue
420 1 3 6 80 fffffe822d0b3620 ntpd pause
342 4 3 3 80 fffffe822d7bf1a0 dhcpd parked
342 3 3 0 80 fffffe810f2dab40 dhcpd kqueue
342 2 3 6 80 fffffe822d9bd9a0 dhcpd parked
342 1 3 6 80 fffffe822d8075a0 dhcpd parked
221 1 2 1 0 fffffe822d1c4a00 syslogd
1 1 3 3 80 fffffe810f51a900 init wait
0 123 3 4 200 fffffe822d1c41c0 npfgc-0 xchicv
0 122 3 7 200 fffffe822d7bf9e0 ipmi0 ipmi0
0 121 3 7 200 fffffe822dcee940 physiod physiod
0 120 3 6 200 fffffe822dce5960 aiodoned aiodoned
0 119 3 0 200 fffffe822dcee100 ioflush syncer
0 118 3 0 200 fffffe822dcee520 pgdaemon pgdaemon
0 115 3 0 280 fffffe822dd78500 spkr1 bellcv
0 114 3 5 280 fffffe822dd78920 audiomix play
0 113 3 5 280 fffffe810ed3b2a0 audiorec record
0 112 3 0 200 fffffe810f51a0c0 atapibus0 sccomp
0 110 3 7 200 fffffe810f20d2c0 usb5 usbevt
0 109 3 7 200 fffffe810f2cb2e0 usb0 usbevt
0 108 2 3 200 fffffe810f2cb700 usb4
0 107 2 3 200 fffffe810f2cbb20 usb1
0 106 2 3 200 fffffe810f2da720 usb3
0 105 2 3 200 fffffe810f51a4e0 usb2
0 104 3 0 200 fffffe810f43b0a0 rt_free rt_free
0 103 3 0 200 fffffe810f43b4c0 unpgc unpgc
0 102 2 4 200 fffffe810f43b8e0 key_timehandler
0 101 3 7 200 fffffe810f45c080 icmp6_wqinput/7 icmp6_wqinput
0 100 3 6 200 fffffe810f45c4a0 icmp6_wqinput/6 icmp6_wqinput
0 99 3 5 200 fffffe810f45c8c0 icmp6_wqinput/5 icmp6_wqinput
0 98 3 4 200 fffffe810f585060 icmp6_wqinput/4 icmp6_wqinput
0 97 3 3 200 fffffe810f585480 icmp6_wqinput/3 icmp6_wqinput
0 96 3 2 200 fffffe810f5858a0 icmp6_wqinput/2 icmp6_wqinput
0 95 3 1 200 fffffe810f38e040 icmp6_wqinput/1 icmp6_wqinput
0 94 3 0 200 fffffe810f38e460 icmp6_wqinput/0 icmp6_wqinput
0 : 93 7 2 200 fffffe810f38e880 nd6_timer
0 92 3 7 200 fffffe810f35f020 icmp_wqinput/7 icmp_wqinput
0 91 3 6 200 fffffe810f35f440 icmp_wqinput/6 icmp_wqinput
0 90 3 5 200 fffffe810f35f860 icmp_wqinput/5 icmp_wqinput
0 89 3 4 200 fffffe810f370000 icmp_wqinput/4 icmp_wqinput
0 88 3 3 200 fffffe810f370420 icmp_wqinput/3 icmp_wqinput
0 87 3 2 200 fffffe810f370840 icmp_wqinput/2 icmp_wqinput
0 86 3 1 200 fffffe810f33e360 icmp_wqinput/1 icmp_wqinput
0 85 3 0 200 fffffe810f33e780 icmp_wqinput/0 icmp_wqinput
0 84 2 3 200 fffffe810f33eba0 rt_timer
0 83 3 7 200 fffffe810f2f1b60 vmem_rehash vmem_rehash
0 82 3 4 200 fffffe810f2f0340 coretemp3 coretemp3
0 81 3 6 200 fffffe810f2f0760 coretemp2 coretemp2
0 80 3 7 200 fffffe810f2f0b80 coretemp1 coretemp1
0 79 3 1 200 fffffe810f2f1320 coretemp0 coretemp0
0 78 3 1 200 fffffe810f2f1740 mfi0 mfi0
0 69 3 5 200 fffffe810f20d6e0 atabus1 atath
0 68 3 4 200 fffffe810f20db00 atabus0 atath
0 66 3 0 200 fffffe810ed3b6c0 scsibus0 sccomp
0 65 3 7 200 fffffe810ed3bae0 usbtask-dr usbtsk
0 64 3 0 200 fffffe810ecaa280 usbtask-hc usbtsk
0 63 3 7 200 fffffe810ecaa6a0 bnx3 bnx3
0 62 3 5 200 fffffe810ecaaac0 bnx2 bnx2
0 61 3 7 200 fffffe810eb61260 bnx1 bnx1
0 60 3 7 200 fffffe810eb61680 bnx0 bnx0
0 : 59 7 5 200 fffffe810eb61aa0 ipmi
0 58 3 7 200 fffffe810ea7c240 xcall/7 xcall
0 57 1 7 200 fffffe810ea7c660 softser/7
0 56 3 7 200 fffffe810ea7ca80 softclk/7 tstile
0 55 1 7 200 fffffe810ea5d220 softbio/7
0 54 1 7 200 fffffe810ea5d640 softnet/7
0 53 1 7 201 fffffe810ea5da60 idle/7
0 52 3 6 200 fffffe810ea36200 xcall/6 xcall
0 51 1 6 200 fffffe810ea36620 softser/6
0 50 1 6 200 fffffe810ea36a40 softclk/6
0 49 1 6 200 fffffe810ea0f1e0 softbio/6
0 48 1 6 200 fffffe810ea0f600 softnet/6
0 47 1 6 201 fffffe810ea0fa20 idle/6
0 46 3 5 200 fffffe810e9e01c0 xcall/5 xcall
0 45 1 5 200 fffffe810e9e05e0 softser/5
0 44 1 5 200 fffffe810e9e0a00 softclk/5
0 43 1 5 200 fffffe810e9d11a0 softbio/5
0 42 1 5 200 fffffe810e9d15c0 softnet/5
0 41 1 5 201 fffffe810e9d19e0 idle/5
0 40 3 4 200 fffffe810e9a2180 xcall/4 xcall
0 39 1 4 200 fffffe810e9a25a0 softser/4
0 38 1 4 200 fffffe810e9a29c0 softclk/4
0 37 1 4 200 fffffe810e973160 softbio/4
0 36 1 4 200 fffffe810e973580 softnet/4
0 35 1 4 201 fffffe810e9739a0 idle/4
0 34 3 3 200 fffffe810e94c140 xcall/3 xcall
0 33 1 3 200 fffffe810e94c560 softser/3
0 32 1 3 200 fffffe810e94c980 softclk/3
0 31 1 3 200 fffffe810e93d120 softbio/3
0 30 1 3 200 fffffe810e93d540 softnet/3
0 29 1 3 201 fffffe810e93d960 idle/3
0 28 3 2 200 fffffe810e906100 xcall/2 xcall
0 27 1 2 200 fffffe810e906520 softser/2
0 26 3 2 200 fffffe810e906940 softclk/2 tstile
0 25 1 2 200 fffffe810e8ef0e0 softbio/2
0 24 1 2 200 fffffe810e8ef500 softnet/2
0 23 1 2 201 fffffe810e8ef920 idle/2
0 22 3 1 200 fffffe810e8c50c0 xcall/1 xcall
0 21 1 1 200 fffffe810e8c54e0 softser/1
0 20 1 1 200 fffffe810e8c5900 softclk/1
0 19 1 1 200 fffffe810e8b10a0 softbio/1
0 18 1 1 200 fffffe810e8b14c0 softnet/1
0 17 1 1 201 fffffe810e8b18e0 idle/1
0 16 3 0 200 fffffe822de92080 lnxsyswq lnxsyswq
0 15 3 0 200 fffffe822de924a0 sysmon smtaskq
0 14 3 2 200 fffffe822de928c0 pmfsuspend pmfsuspend
0 13 3 1 200 fffffe822e2b1060 pmfevent pmfevent
0 12 3 0 200 fffffe822e2b1480 sopendfree sopendfr
0 11 3 4 200 fffffe822e2b18a0 nfssilly nfssilly
0 10 2 3 200 fffffe822f6d0040 cachegc
0 9 3 1 200 fffffe822f6d0460 vdrain vdrain
0 8 3 0 200 fffffe822f6d0880 modunload mod_unld
0 7 3 0 200 fffffe822f6eb020 xcall/0 xcall
0 : 6 7 0 200 fffffe822f6eb440 softser/0
0 : 5 7 0 200 fffffe822f6eb860 softclk/0
0 4 1 0 200 fffffe822f707000 softbio/0
0 3 1 0 200 fffffe822f707420 softnet/0
0 2 1 0 201 fffffe822f707840 idle/0
0 1 2 3 200 ffffffff81481b20 swapper
db{0}: trace/t 0t322
trace: pid 322 lid 1 at 0xffff80009eb99b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7af9a8e3e10a:
db{0}: trace/t 0t563
trace: pid 563 lid 1 at 0xffff80009eb9eb00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7af9a8e3e10a:
db{0}: trace/t 0t534
trace: pid 534 lid 1 at 0xffff80009ec64b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7af9a8e3e10a:
db{0}: trace/t 0t400
trace: pid 400 lid 1 at 0xffff80009ebc9b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7af9a8e3e10a:
db{0}: trace/t 0t900
trace: pid 900 lid 1 at 0xffff80009eb94b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7af9a8e3e10a:
db{0}: trace/t 0t361
trace: pid 361 lid 1 at 0xffff80009eadcb00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7af9a8e3e10a:
db{0}: trace/t 0t549
trace: pid 549 lid 1 at 0xffff80009dd72b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7af9a8e3e10a:
db{0}: trace/t 0t516
trace: pid 516 lid 1 at 0xffff80009e8a9b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7af9a8e3e10a:
db{0}: trace/t 0t460
trace: pid 460 lid 1 at 0xffff80009e12eb20
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x176
pollcommon.part.3() at netbsd:pollcommon.part.3+0x8d
sys_poll() at netbsd:sys_poll+0x9b
syscall() at netbsd:syscall+0x1ed
--- syscall (number 209) ---
7dba6203e28a:
db{0}: trace/a fffffe810ea7ca80
trace: pid 0 lid 56 at 0xffff80008f7e3e40
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
tcp_slowtimo() at netbsd:tcp_slowtimo+0x10
callout_softclock() at netbsd:callout_softclock+0x201
softint_dispatch() at netbsd:softint_dispatch+0xd3
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f7e3ff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}: trace/a fffffe810e906940
trace: pid 0 lid 26 at 0xffff80008f725e00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
tcp_timer_rexmt() at netbsd:tcp_timer_rexmt+0x27
callout_softclock() at netbsd:callout_softclock+0x201
softint_dispatch() at netbsd:softint_dispatch+0xd3
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f725ff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}: call cpu_reset
From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Thu, 05 Apr 2018 09:36:04 +0200 (CEST)
BTW, the last update is a kernel which is GENERIC + DIAGNOSTIC +
LOCKDEBUG and with 4x default NMBCLUSTERS.
- H=E5vard
From: Ryota Ozaki <ozaki-r@netbsd.org>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc: port-amd64-maintainer@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, Havard Eidnes <he@netbsd.org>
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network
interfaces in use
Date: Thu, 5 Apr 2018 16:48:54 +0900
On Thu, Apr 5, 2018 at 4:40 PM, Havard Eidnes <he@netbsd.org> wrote:
> The following reply was made to PR port-amd64/53155; it has been noted by GNATS.
>
> From: Havard Eidnes <he@NetBSD.org>
> To: gnats-bugs@NetBSD.org
> Cc:
> Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
> network interfaces in use
> Date: Thu, 05 Apr 2018 09:36:04 +0200 (CEST)
>
> BTW, the last update is a kernel which is GENERIC + DIAGNOSTIC +
> LOCKDEBUG and with 4x default NMBCLUSTERS.
If you're using latest source code of netbsd-8, you can use a new ddb command:
show all locks [/t], which may help for locking issues.
ozaki-r
From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Thu, 05 Apr 2018 14:22:17 +0200 (CEST)
Following ozaki-r's hints, here's more information with the locks
dumped on another wedge event.
"DDB break sequence keyed in"
fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0xffff80009da69000 ilevel 0x8 rsp 0xffff80008f69bcf8
curlwp 0xfffffe822f6eb440 pid 0.6 lowest kstack 0xffff80008f6972c0
Stopped in pid 0.6 (system) at netbsd:breakpoint+0x5: leave
db{0}: show all locks
[Locks tracked through LWPs]
Locks held by an LWP (expect):
Lock 0 (initialized at fstrans_init)
lock address : 0xffffffff815cb9b8 type : sleep/adaptive
initialized : 0xffffffff809fe690
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 3
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe821e439080
last locked* : 0xffffffff80a00523 unlocked : 0xffffffff80a0315f
owner field : 0xfffffe821e439080 wait/spin: 0/0
Turnstile chain at 0xffffffff814c5130.
=: No active turnstile for this lock.
Lock 1 (initialized at fstrans_init)
lock address : 0xffffffff815cb9b0 type : sleep/adaptive
initialized : 0xffffffff809fe6a3
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 3
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe821e439080
last locked* : 0xffffffff809ffb6d unlocked : 0xffffffff809fecc4
owner field : 0xfffffe821e439080 wait/spin: 0/0
Turnstile chain at 0xffffffff814c5120.
=: No active turnstile for this lock.
Locks held by an LWP (pickup):
Lock 0 (initialized at fork1)
lock address : 0xfffffe821dce4a30 type : sleep/adaptive
initialized : 0xffffffff809599ab
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 2
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe81b102da80
last locked* : 0xffffffff809580c1 unlocked : 0xffffffff80879889
owner/count : 0xfffffe81b102da80 flags : 0x0000000000000004
Turnstile chain at 0xffffffff814c4e20.
=: No active turnstile for this lock.
Lock 1 (initialized at filedesc_ctor)
lock address : 0xfffffe821f62f880 type : sleep/adaptive
initialized : 0xffffffff8094d11f
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 2
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe81b102da80
last locked* : 0xffffffff80953c76 unlocked : 0xffffffff8094e4a5
owner field : 0xfffffe81b102da80 wait/spin: 0/0
Turnstile chain at 0xffffffff814c4ec0.
=: No active turnstile for this lock.
Locks held by an LWP (master):
Lock 0 (initialized at filedesc_ctor)
lock address : 0xfffffe822870adc0 type : sleep/adaptive
initialized : 0xffffffff8094d11f
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 1
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe822962bac0
last locked* : 0xffffffff809522e8 unlocked : 0xffffffff8094ec84
owner field : 0xfffffe822962bac0 wait/spin: 0/0
Turnstile chain at 0xffffffff814c5140.
=: No active turnstile for this lock.
Locks held by an LWP (ioflush):
Lock 0 (initialized at vcache_alloc)
lock address : 0xfffffe8061cb6fa0 type : sleep/adaptive
initialized : 0xffffffff80a01652
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 7
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe822dce8100
last locked* : 0xffffffff80a17d69 unlocked : 0xffffffff80a17db4
owner/count : 0xfffffe822dce8100 flags : 0x0000000000000004
Turnstile chain at 0xffffffff814c5100.
=: No active turnstile for this lock.
Locks held by an LWP (softnet/0):
Lock 0 (initialized at soinit)
lock address : 0xfffffe822fb97f40 type : sleep/adaptive
initialized : 0xffffffff809db0cb
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 14
current cpu : 0 last held: 0
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe822f707420
last locked* : 0xffffffff806d7add unlocked : 0xffffffff80951f2c
owner field : 0xfffffe822f707420 wait/spin: 1/0
Turnstile chain at 0xffffffff814c5040.
=: Turnstile at 0xfffffe810f381d20 (wrq=0xfffffe810f381d40, rdq=0xfffffe810f381d
50).
=: 0 waiting readers:
=: 14 waiting writers: 0xfffffe810f396880 0xfffffe810e9a29c0 0xfffffe810e9e0a00 0xfffffe810e94c980 0xfffffe82212fc700 0xfffffe82228e28e0 0xfffffe8222950b60 0xfffffe82212fcb20 0xfffffe822962b6a0 0xfffffe822c341200 0xfffffe8220e621e0 0xfffffe821ea02260 0xfffffe821f0986c0 0xfffffe810f2da300
[Locks tracked through CPUs]
Locks held on CPU 0:
Lock 0 (initialized at main)
lock address : 0xffffffff814c3840 type : spin
initialized : 0xffffffff80d02d91
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 6
current cpu : 0 last held: 0
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe822f707420
last locked* : 0xffffffff806d7ae7 unlocked : 0xffffffff80a3a00f
curcpu holds : 1 wanted by: 000000000000000000
Lock 1 (initialized at com_attach_subr)
lock address : 0xfffffe810f20b1e0 type : spin
initialized : 0xffffffff805ce644
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 0
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe822f6eb440
last locked* : 0xffffffff805cdba0 unlocked : 0xffffffff805cdd7f
owner field : 0x0000000000010800 wait/spin: 0/1
db{0}: ps
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
6110 : 1 7 3 0 fffffe821e439080 expect
16029 1 3 5 80 fffffe8223e0d680 sh wait
17269 1 3 5 80 fffffe821e4398c0 perl pipe_rd
6258 1 3 5 80 fffffe8220e10960 sh wait
6160 1 3 4 80 fffffe821db62940 ssh select
9045 2 3 2 80 fffffe8207c29900 expect select
9045 1 3 5 80 fffffe821d8e9120 expect parked
8628 1 3 2 80 fffffe8217495860 sh wait
6162 1 3 0 80 fffffe821baa01a0 perl pipe_rd
10236 1 3 4 80 fffffe8220439aa0 sh wait
10536 1 3 4 40000 fffffe821ea02260 ssh tstile
16657 2 3 4 80 fffffe822d304a00 expect select
16657 1 3 0 80 fffffe822244f160 expect parked
1568 1 3 2 80 fffffe821c6c44c0 sh wait
621 1 3 3 80 fffffe821eb29980 perl pipe_rd
1714 1 3 5 80 fffffe82220db9a0 sh wait
20537 1 3 7 80 fffffe8220e62600 ssh select
855 2 3 3 80 fffffe8220c66920 expect select
855 : 1 7 6 0 fffffe820e54e100 expect
2913 1 3 6 80 fffffe8220439680 sh wait
10978 1 3 6 80 fffffe821d9671e0 ssh select
26960 1 3 6 80 fffffe821f312a60 perl pipe_rd
13813 1 3 0 80 fffffe820e54e940 sh wait
4350 2 3 6 80 fffffe8222d32880 expect select
4350 1 3 0 80 fffffe82220e8b80 expect parked
22724 1 3 7 80 fffffe822a7bc960 sh wait
5198 1 3 0 80 fffffe822301aba0 perl pipe_rd
3944 1 3 5 80 fffffe821fe156e0 sh wait
6695 1 3 5 80 fffffe8221b59560 par pause
1700 1 3 5 80 fffffe81dc6c72a0 sh wait
9549 1 3 1 80 fffffe821c9580e0 sh wait
6024 1 3 1 80 fffffe8220975780 sh wait
7673 1 3 1 80 fffffe822244f580 sh wait
10638 1 3 1 80 fffffe821ed0c9c0 cron pipe_rd
4839 1 3 6 80 fffffe820e0aa200 sshd select
3563 1 3 6 80 fffffe821f7e8a60 sshd select
11359 1 3 1 80 fffffe821f4f2b40 ssh select
1612 1 3 1 80 fffffe81dc6c7ae0 sh wait
16815 1 3 1 80 fffffe821ea02aa0 bash select
5195 1 3 0 80 fffffe820f859840 sshd select
19300 1 3 0 80 fffffe8220e62a20 sshd select
6945 1 3 1 80 fffffe821d1d9340 bash wait
5497 1 3 2 80 fffffe8226609b40 sshd select
3611 1 3 2 80 fffffe8226a51b20 sshd select
18181 1 3 6 80 fffffe82228e20a0 ssh select
29538 1 3 6 80 fffffe82174aa600 sh wait
1782 1 3 2 40000 fffffe821f0986c0 ping tstile
23170 1 3 4 80 fffffe822345f900 ssh select
25239 1 3 6 80 fffffe8220e10540 sh wait
22752 1 3 2 80 fffffe8223e968c0 ssh select
21643 1 3 6 80 fffffe8221723ac0 sh wait
22354 1 3 1 80 fffffe822301a780 ssh select
12292 1 3 6 80 fffffe8221b59140 sh wait
440 1 3 1 80 fffffe821f2f9760 bash select
16624 1 3 4 80 fffffe821dd60320 sshd select
24686 1 3 3 80 fffffe82219b8480 sshd select
17072: 1 7 2 0 fffffe81b102da80 pickup
17333 1 3 3 80 fffffe821e378ba0 bash wait
12287 1 3 5 80 fffffe8223e0d260 sshd select
28431 1 3 3 80 fffffe821f62b300 sshd select
25668 1 3 5 80 fffffe821db62100 ssh select
18777 1 3 7 80 fffffe821d548460 sh wait
23483 1 3 6 80 fffffe821dfed5e0 ssh select
24114 1 3 1 80 fffffe82219b8060 sh wait
18841 1 3 2 80 fffffe821db62520 ssh select
8541 1 3 1 80 fffffe821c347020 sh wait
20854 1 3 3 80 fffffe8220a11420 ssh select
28316 1 3 7 80 fffffe821d449760 sh wait
21952 1 3 6 80 fffffe8223e964a0 ssh select
21821 1 3 7 80 fffffe81b0264680 sh wait
22659 1 3 7 80 fffffe82220db580 ssh select
15239 1 3 7 80 fffffe8200e69640 sh wait
20787 1 3 0 80 fffffe8222093620 ssh select
25294 1 3 7 80 fffffe821e959b20 sh wait
20182 1 3 4 80 fffffe82205cf440 ssh select
21140 1 3 1 80 fffffe822301a360 sh wait
10348 1 3 1 80 fffffe822084f900 ssh select
19979 1 3 6 80 fffffe8219d9d9e0 sh wait
20748 1 3 0 80 fffffe81b102d240 ssh select
29870 1 3 6 80 fffffe821ea84a60 sh wait
28154 1 3 3 80 fffffe821e6f1060 ssh select
21713 1 3 1 80 fffffe82215225a0 sh wait
22666 1 3 5 80 fffffe821fe912a0 ssh select
19782 1 3 3 80 fffffe8228f16ae0 sh wait
23668 1 3 4 80 fffffe822325b440 ssh select
24824 1 3 3 80 fffffe821c347440 sh wait
21452 1 3 1 80 fffffe821e6f18a0 ssh select
18115 1 3 6 80 fffffe8223c84b20 sh wait
23734 1 3 3 80 fffffe82101970e0 ssh select
23642 1 3 3 80 fffffe821b7f01c0 sh wait
15721 1 3 2 80 fffffe822d3045e0 ssh select
24741 1 3 6 80 fffffe8222d32040 sh wait
25785 1 3 4 80 fffffe820e91e8c0 ssh select
29353 1 3 6 80 fffffe822d96d580 sh wait
17591 1 3 3 80 fffffe8211ded180 ssh select
22243 1 3 6 80 fffffe821e378360 sh wait
11016 1 3 7 80 fffffe820d036160 ssh select
23915 1 3 6 80 fffffe822d9f1560 sh wait
23755 1 3 2 80 fffffe82220e8760 ssh select
28517 1 3 3 80 fffffe821d967600 sh wait
21937 1 3 1 80 fffffe8222b59420 ssh select
24531 1 3 3 80 fffffe8222093a40 sh wait
21091 1 3 5 80 fffffe82101cf480 ssh select
24050 1 3 6 80 fffffe82174aaa20 sh wait
21377 1 3 7 80 fffffe821dfed1c0 ssh select
25968 1 3 6 80 fffffe822d9f1140 sh wait
19956 1 3 5 80 fffffe8215f17980 ssh select
22513 1 3 1 80 fffffe821d1d9b80 sh wait
23875 1 3 1 80 fffffe821e2da1e0 ssh select
20549 1 3 1 80 fffffe821eb29560 sh wait
21480 1 3 6 80 fffffe822281b940 ssh select
19370 1 3 6 80 fffffe8221343200 sh wait
27941 1 3 3 80 fffffe821d548040 ssh select
1308 1 3 3 80 fffffe821abcc300 sh wait
16930 1 3 5 80 fffffe821e2a7160 ssh select
20374 1 3 6 80 fffffe8221b59980 sh wait
17114 1 3 4 80 fffffe82219c66e0 ssh select
19866 1 3 6 80 fffffe820f859000 sh wait
26412 1 3 6 80 fffffe821b800780 ssh select
21138 1 3 1 80 fffffe821c0755a0 sh wait
19764 1 3 7 80 fffffe8220e10120 ssh select
19992 1 3 6 80 fffffe82219c62c0 sh wait
8561 1 3 4 80 fffffe822c0c4660 ssh select
22282 1 3 5 80 fffffe820a38b0a0 sh wait
21455 1 3 5 80 fffffe82205446c0 ssh select
19503 1 3 7 80 fffffe822a7bc540 sh wait
20237 1 3 2 80 fffffe820f859420 ssh select
23846 1 3 7 80 fffffe821b7f05e0 sh wait
23390 1 3 0 80 fffffe821f62b720 ssh select
6312 1 3 3 80 fffffe822325b860 sh wait
22688 1 3 6 80 fffffe8200e69220 ssh select
26232 1 3 6 80 fffffe8220975360 sh wait
25797 1 3 1 80 fffffe82217cba00 ssh select
26094 1 3 5 80 fffffe822d77e5c0 sh wait
18349 1 3 2 80 fffffe82224e1920 ssh select
18457 1 3 5 80 fffffe821b800360 sh wait
4150 1 3 6 80 fffffe8222ec34a0 ssh select
17097 1 3 1 80 fffffe821dfeda00 sh wait
21786 1 3 2 80 fffffe81b0264260 ssh select
21190 1 3 5 80 fffffe821e959700 sh wait
21970 1 3 4 80 fffffe8226a51700 ssh select
19183 1 3 5 80 fffffe821f2f9b80 sh wait
18769 1 3 2 80 fffffe821db454e0 ssh select
23677 1 3 1 80 fffffe821c958500 sh wait
21567 1 3 1 80 fffffe822232c060 ssh select
27597 1 3 2 80 fffffe821ed0c5a0 sh wait
23189 1 3 0 80 fffffe8217495440 ssh select
24479 1 3 0 80 fffffe82205cf860 sh wait
19747 1 3 5 80 fffffe820e91e080 ssh select
22977 1 3 0 80 fffffe822d7a19c0 sh wait
19741 1 3 6 80 fffffe8211524880 ssh select
21217 1 3 5 80 fffffe820a38b8e0 sh wait
17887 1 3 1 80 fffffe821f7e8640 ssh select
20134 1 3 7 80 fffffe821f62bb40 sh wait
17673 1 3 3 80 fffffe8222ec3080 ssh select
16512 1 3 5 80 fffffe8222d32460 sh wait
22179 1 3 7 80 fffffe8220c66500 ssh select
23863 1 3 3 80 fffffe821ea84220 sh wait
15669 1 3 5 80 fffffe820e91e4a0 ssh select
19089 1 3 3 80 fffffe821f4f2300 sh wait
22480 1 3 2 80 fffffe8222b59000 ssh select
15576 1 3 6 80 fffffe8220439260 sh wait
16125 1 3 2 80 fffffe821f098ae0 ssh select
14729 1 3 6 80 fffffe810f2f1b60 sh wait
20188 1 3 2 80 fffffe822d77e1a0 ssh select
23671 1 3 6 80 fffffe8219d9d1a0 sh wait
2378 1 3 4 80 fffffe822244f9a0 ssh select
13692 1 3 3 80 fffffe8226a512e0 sh wait
16255 1 3 6 80 fffffe8224cb4940 ssh select
13845 1 3 3 80 fffffe822084f4e0 sh wait
21973 1 3 3 80 fffffe821c958920 ssh select
13867 1 3 3 80 fffffe821e6f1480 sh wait
13804 1 3 4 80 fffffe82212fc2e0 ssh select
20798 1 3 6 80 fffffe822325b020 sh wait
17218 1 3 2 80 fffffe8229c95960 ssh select
3258 1 3 0 80 fffffe821fe152c0 sh wait
21575 1 3 1 80 fffffe82200a1660 ssh select
18008 1 3 6 80 fffffe81b102d660 sh wait
19955 1 3 6 80 fffffe821db45900 ssh select
19174 1 3 6 80 fffffe822dcc7120 sh wait
16916 1 3 2 80 fffffe822962b280 ssh select
2248 1 3 6 80 fffffe82224e10e0 sh wait
16945 1 3 0 80 fffffe8222b59840 ssh select
21163 1 3 2 80 fffffe8211524460 sh wait
22509 1 3 2 80 fffffe8223c84700 ssh select
18626 1 3 2 80 fffffe82220e8340 sh wait
18828 1 3 6 80 fffffe821de0a8e0 ssh select
15805 1 3 7 80 fffffe8221343a40 sh wait
20827 1 3 4 80 fffffe822c0c4a80 ssh select
14384 1 3 0 80 fffffe821e4a6a40 sh wait
20713 1 3 6 80 fffffe82164eab60 bash select
9812 1 3 3 80 fffffe82280a41a0 sshd select
13062 1 3 3 80 fffffe8220975ba0 sshd select
17386 1 3 6 80 fffffe820e0aaa40 ssh select
11169 1 3 2 80 fffffe82101cf060 sh wait
1442 1 3 0 80 fffffe821c075180 sshd select
458 1 3 0 80 fffffe821d8e9540 conserver select
1083 1 3 4 0 fffffe822c341200 conserver tstile
1182 1 3 4 0 fffffe82228e28e0 conserver tstile
1179 1 3 4 0 fffffe82212fcb20 conserver tstile
534 1 3 0 80 fffffe8220e93660 conserver select
900 1 3 2 0 fffffe82212fc700 conserver tstile
935 1 3 4 0 fffffe8220e621e0 conserver tstile
775 1 3 4 80 fffffe822345f0c0 conserver select
871 1 3 4 80 fffffe8222950740 conserver select
902 1 3 4 0 fffffe8222950b60 conserver tstile
724 1 3 7 80 fffffe822dd720e0 getty ttyraw
799 1 3 4 0 fffffe822962b6a0 conserver tstile
894 1 3 4 80 fffffe8226609300 conserver select
891 1 3 4 80 fffffe8226609720 cron nanoslp
570 1 3 4 80 fffffe822d7a15a0 python2.7 select
558 1 3 4 80 fffffe8228f166c0 inetd kqueue
485 1 3 7 80 fffffe822c0c4240 nginx kqueue
548 1 3 7 80 fffffe8228f162a0 nginx pause
397 1 3 6 80 fffffe822d405a20 qmgr kqueue
759 : 1 7 1 0 fffffe822962bac0 master
473 1 3 0 80 fffffe822a7ffaa0 tac_plus select
428 1 3 2 80 fffffe822d3041c0 powerd kqueue
408 1 3 4 80 fffffe822c9dfa60 ntpd pause
342 4 3 3 80 fffffe822c9df220 dhcpd parked
342 3 3 7 80 fffffe822c9df640 dhcpd kqueue
342 : 2 7 0 0 fffffe822d7a1180 dhcpd
342 1 3 1 80 fffffe822c341a40 dhcpd parked
83 1 3 2 0 fffffe810f2da300 syslogd tstile
1 1 3 7 80 fffffe810f512900 init wait
0 123 3 0 200 fffffe822d77e9e0 npfgc-0 npfgccv
0 122 3 5 200 fffffe822d405600 ipmi0 ipmi0
0 121 3 7 200 fffffe822dce8940 physiod physiod
0 120 3 0 200 fffffe822dcc7960 aiodoned aiodoned
0 : 119 7 7 200 fffffe822dce8100 ioflush
0 118 3 0 200 fffffe822dce8520 pgdaemon pgdaemon
0 115 3 0 280 fffffe822dd72500 spkr1 bellcv
0 114 3 5 280 fffffe822dd72920 audiomix play
0 113 3 5 280 fffffe810ed3b2a0 audiorec record
0 112 3 0 200 fffffe810f5120c0 atapibus0 sccomp
0 110 3 4 200 fffffe810f20d2c0 usb4 usbevt
0 109 3 4 200 fffffe810f2cb2e0 usb0 usbevt
0 108 3 4 200 fffffe810f2dab40 usb5 usbevt
0 107 3 0 200 fffffe810f2cbb20 usb3 usbevt
0 106 3 4 200 fffffe810f2da720 usb2 usbevt
0 105 3 4 200 fffffe810f5124e0 usb1 usbevt
0 104 3 0 200 fffffe810f4a30a0 rt_free rt_free
0 103 3 0 200 fffffe810f4a34c0 unpgc unpgc
0 102 3 0 200 fffffe810f4a38e0 key_timehandler key_timehandler
0 101 3 7 200 fffffe810f474080 icmp6_wqinput/7 icmp6_wqinput
0 100 3 6 200 fffffe810f4744a0 icmp6_wqinput/6 icmp6_wqinput
0 99 3 5 200 fffffe810f4748c0 icmp6_wqinput/5 icmp6_wqinput
0 98 3 4 200 fffffe810f495060 icmp6_wqinput/4 icmp6_wqinput
0 97 3 3 200 fffffe810f495480 icmp6_wqinput/3 icmp6_wqinput
0 96 3 2 200 fffffe810f4958a0 icmp6_wqinput/2 icmp6_wqinput
0 95 3 1 200 fffffe810f396040 icmp6_wqinput/1 icmp6_wqinput
0 94 3 0 200 fffffe810f396460 icmp6_wqinput/0 icmp6_wqinput
0 93 3 1 40200 fffffe810f396880 nd6_timer tstile
0 92 3 7 200 fffffe810f35f020 icmp_wqinput/7 icmp_wqinput
0 91 3 6 200 fffffe810f35f440 icmp_wqinput/6 icmp_wqinput
0 90 3 5 200 fffffe810f35f860 icmp_wqinput/5 icmp_wqinput
0 89 3 4 200 fffffe810f370000 icmp_wqinput/4 icmp_wqinput
0 88 3 3 200 fffffe810f370420 icmp_wqinput/3 icmp_wqinput
0 87 3 2 200 fffffe810f370840 icmp_wqinput/2 icmp_wqinput
0 86 3 1 200 fffffe810f33e360 icmp_wqinput/1 icmp_wqinput
0 85 3 0 200 fffffe810f33e780 icmp_wqinput/0 icmp_wqinput
0 84 3 4 200 fffffe810f33eba0 rt_timer rt_timer
0 83 3 2 200 fffffe810f2cb700 vmem_rehash vmem_rehash
0 82 3 2 200 fffffe810f2f0340 coretemp3 coretemp3
0 81 3 3 200 fffffe810f2f0760 coretemp2 coretemp2
0 80 3 4 200 fffffe810f2f0b80 coretemp1 coretemp1
0 79 3 5 200 fffffe810f2f1320 coretemp0 coretemp0
0 78 3 5 200 fffffe810f2f1740 mfi0 mfi0
0 69 3 5 200 fffffe810f20d6e0 atabus1 atath
0 68 3 7 200 fffffe810f20db00 atabus0 atath
0 66 3 0 200 fffffe810ed3b6c0 scsibus0 sccomp
0 65 3 2 200 fffffe810ed3bae0 usbtask-dr usbtsk
0 64 3 0 200 fffffe810ecaa280 usbtask-hc usbtsk
0 63 3 7 200 fffffe810ecaa6a0 bnx3 bnx3
0 62 3 4 200 fffffe810ecaaac0 bnx2 bnx2
0 61 3 2 200 fffffe810eb61260 bnx1 bnx1
0 60 3 5 200 fffffe810eb61680 bnx0 bnx0
0 : 59 7 5 200 fffffe810eb61aa0 ipmi
0 58 3 7 200 fffffe810ea7c240 xcall/7 xcall
0 57 1 7 200 fffffe810ea7c660 softser/7
0 56 1 7 200 fffffe810ea7ca80 softclk/7
0 55 1 7 200 fffffe810ea5d220 softbio/7
0 54 1 7 200 fffffe810ea5d640 softnet/7
0 53 1 7 201 fffffe810ea5da60 idle/7
0 52 3 6 200 fffffe810ea36200 xcall/6 xcall
0 51 1 6 200 fffffe810ea36620 softser/6
0 50 1 6 200 fffffe810ea36a40 softclk/6
0 49 1 6 200 fffffe810ea0f1e0 softbio/6
0 48 1 6 200 fffffe810ea0f600 softnet/6
0 47 1 6 201 fffffe810ea0fa20 idle/6
0 46 3 5 200 fffffe810e9e01c0 xcall/5 xcall
0 45 1 5 200 fffffe810e9e05e0 softser/5
0 44 3 5 200 fffffe810e9e0a00 softclk/5 tstile
0 43 1 5 200 fffffe810e9d11a0 softbio/5
0 42 1 5 200 fffffe810e9d15c0 softnet/5
0 41 1 5 201 fffffe810e9d19e0 idle/5
0 40 3 4 200 fffffe810e9a2180 xcall/4 xcall
0 39 1 4 200 fffffe810e9a25a0 softser/4
0 38 3 4 200 fffffe810e9a29c0 softclk/4 tstile
0 37 1 4 200 fffffe810e973160 softbio/4
0 36 1 4 200 fffffe810e973580 softnet/4
0 : 35 7 4 201 fffffe810e9739a0 idle/4
0 34 3 3 200 fffffe810e94c140 xcall/3 xcall
0 33 1 3 200 fffffe810e94c560 softser/3
0 32 3 3 200 fffffe810e94c980 softclk/3 tstile
0 31 1 3 200 fffffe810e93d120 softbio/3
0 30 1 3 200 fffffe810e93d540 softnet/3
0 29 1 3 201 fffffe810e93d960 idle/3
0 28 3 2 200 fffffe810e906100 xcall/2 xcall
0 27 1 2 200 fffffe810e906520 softser/2
0 26 1 2 200 fffffe810e906940 softclk/2
0 25 1 2 200 fffffe810e8ef0e0 softbio/2
0 24 1 2 200 fffffe810e8ef500 softnet/2
0 23 1 2 201 fffffe810e8ef920 idle/2
0 22 3 1 200 fffffe810e8c50c0 xcall/1 xcall
0 21 1 1 200 fffffe810e8c54e0 softser/1
0 20 1 1 200 fffffe810e8c5900 softclk/1
0 19 1 1 200 fffffe810e8b10a0 softbio/1
0 18 1 1 200 fffffe810e8b14c0 softnet/1
0 17 1 1 201 fffffe810e8b18e0 idle/1
0 16 3 0 200 fffffe822de92080 lnxsyswq lnxsyswq
0 15 3 0 200 fffffe822de924a0 sysmon smtaskq
0 14 3 5 200 fffffe822de928c0 pmfsuspend pmfsuspend
0 13 3 7 200 fffffe822e2b1060 pmfevent pmfevent
0 12 3 0 200 fffffe822e2b1480 sopendfree sopendfr
0 11 3 1 200 fffffe822e2b18a0 nfssilly nfssilly
0 10 3 4 200 fffffe822f6d0040 cachegc cachegc
0 9 3 5 200 fffffe822f6d0460 vdrain vdrain
0 8 3 0 200 fffffe822f6d0880 modunload mod_unld
0 7 3 0 200 fffffe822f6eb020 xcall/0 xcall
0 : 6 7 0 200 fffffe822f6eb440 softser/0
0 5 1 0 200 fffffe822f6eb860 softclk/0
0 4 1 0 200 fffffe822f707000 softbio/0
0 : 3 7 0 200 fffffe822f707420 softnet/0
0 2 1 0 201 fffffe822f707840 idle/0
0 1 3 4 200 ffffffff81481b20 swapper uvm
db{0}: trace/t 0t10536
trace: pid 10536 lid 1 at 0xffff80009f171b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
739a9de3e10a:
db{0}: trace/t 0t1782
trace: pid 1782 lid 1 at 0xffff80009f18ad00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sosetopt() at netbsd:sosetopt+0x2e
sys_setsockopt() at netbsd:sys_setsockopt+0x91
syscall() at netbsd:syscall+0x1ed
--- syscall (number 105) ---
72968e6ba14a:
db{0}: trace/t 0t1083
trace: pid 1083 lid 1 at 0xffff80009e0b9b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7bc97bc3e10a:
db{0}: trace/t 0t1182
trace: pid 1182 lid 1 at 0xffff80009e9aeb00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7bc97bc3e10a:
db{0}: trace/t 0t1179
trace: pid 1179 lid 1 at 0xffff80009eb5db00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7bc97bc3e10a:
db{0}: trace/t 0t900
trace: pid 900 lid 1 at 0xffff80009eb74b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7bc97bc3e10a:
db{0}: trace/t 0t935
trace: pid 935 lid 1 at 0xffff80009eab8b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7bc97bc3e10a:
db{0}: trace/t 0t902
trace: pid 902 lid 1 at 0xffff80009e84eb00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7bc97bc3e10a:
db{0}: trace/t 0t799
trace: pid 799 lid 1 at 0xffff80009e517b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7bc97bc3e10a:
db{0}: trace/t 0t83
trace: pid 83 lid 1 at 0xffff80009db73ae0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sosend() at netbsd:sosend+0x48
do_sys_sendmsg_so() at netbsd:do_sys_sendmsg_so+0x272
do_sys_sendmsg() at netbsd:do_sys_sendmsg+0x85
sys_sendto() at netbsd:sys_sendto+0x5c
syscall() at netbsd:syscall+0x1ed
--- syscall (number 133) ---
7d35dd0eea7a:
db{0}: trace/a fffffe810f396880
trace: pid 0 lid 93 at 0xffff80009dbc5d60
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
nd6_timer_work() at netbsd:nd6_timer_work+0x3f
workqueue_worker() at netbsd:workqueue_worker+0xcd
db{0}: trace/a fffffe810e9e0a00
trace: pid 0 lid 44 at 0xffff80008f797e00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
tcp_timer_rexmt() at netbsd:tcp_timer_rexmt+0x27
callout_softclock() at netbsd:callout_softclock+0x201
softint_dispatch() at netbsd:softint_dispatch+0xd3
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f797ff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}: trace/a fffffe810e9a29c0
trace: pid 0 lid 38 at 0xffff80008f771e00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
tcp_timer_rexmt() at netbsd:tcp_timer_rexmt+0x27
callout_softclock() at netbsd:callout_softclock+0x201
softint_dispatch() at netbsd:softint_dispatch+0xd3
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f771ff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}:
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
DDB lost frame for netbsd:Xdoreti, trying 0xffff80008f69bd70
Xdoreti() at netbsd:Xdoreti
--- interrupt ---
Bad frame pointer: 0xffff800000000000
202:
db{0}: trace/a fffffe810e94c980
trace: pid 0 lid 32 at 0xffff80008f74be00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
tcp_timer_rexmt() at netbsd:tcp_timer_rexmt+0x27
callout_softclock() at netbsd:callout_softclock+0x201
softint_dispatch() at netbsd:softint_dispatch+0xd3
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f74bff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}: show all locks /t
[Locks tracked through LWPs]
Locks held by an LWP (expect):
Lock 0 (initialized at fstrans_init)
lock address : 0xffffffff815cb9b8 type : sleep/adaptive
initialized : 0xffffffff809fe690
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 3
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe821e439080
last locked* : 0xffffffff80a00523 unlocked : 0xffffffff80a0315f
owner field : 0xfffffe821e439080 wait/spin: 0/0
Turnstile chain at 0xffffffff814c5130.
=: No active turnstile for this lock.
Lock 1 (initialized at fstrans_init)
lock address : 0xffffffff815cb9b0 type : sleep/adaptive
initialized : 0xffffffff809fe6a3
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 3
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe821e439080
last locked* : 0xffffffff809ffb6d unlocked : 0xffffffff809fecc4
owner field : 0xfffffe821e439080 wait/spin: 0/0
Turnstile chain at 0xffffffff814c5120.
=: No active turnstile for this lock.
trace: pid 6110 lid 1 at 0xffff80009ed098a0
sleepq_block() at netbsd:sleepq_block+0x1c5
kpause() at netbsd:kpause+0xea
pserialize_perform() at netbsd:pserialize_perform+0xf4
fstrans_setstate() at netbsd:fstrans_setstate+0x42
genfs_suspendctl() at netbsd:genfs_suspendctl+0x57
VFS_SUSPENDCTL() at netbsd:VFS_SUSPENDCTL+0x1f
vfs_suspend() at netbsd:vfs_suspend+0x5b
vrevoke_suspend_next() at netbsd:vrevoke_suspend_next+0x2a
vrevoke() at netbsd:vrevoke+0x2b
genfs_revoke() at netbsd:genfs_revoke+0x13
VOP_REVOKE() at netbsd:VOP_REVOKE+0x54
pty_grant_slave() at netbsd:pty_grant_slave+0xa8
ptmioctl() at netbsd:ptmioctl+0x90
cdev_ioctl() at netbsd:cdev_ioctl+0x88
VOP_IOCTL() at netbsd:VOP_IOCTL+0x3b
vn_ioctl() at netbsd:vn_ioctl+0xa6
sys_ioctl() at netbsd:sys_ioctl+0x101
syscall() at netbsd:syscall+0x1ed
--- syscall (number 54) ---
74c471efeeaa:
Locks held by an LWP (pickup):
Lock 0 (initialized at fork1)
lock address : 0xfffffe821dce4a30 type : sleep/adaptive
initialized : 0xffffffff809599ab
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 2
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe81b102da80
last locked* : 0xffffffff809580c1 unlocked : 0xffffffff80879889
owner/count : 0xfffffe81b102da80 flags : 0x0000000000000004
Turnstile chain at 0xffffffff814c4e20.
=: No active turnstile for this lock.
Lock 1 (initialized at filedesc_ctor)
lock address : 0xfffffe821f62f880 type : sleep/adaptive
initialized : 0xffffffff8094d11f
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 2
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe81b102da80
last locked* : 0xffffffff80953c76 unlocked : 0xffffffff8094e4a5
owner field : 0xfffffe81b102da80 wait/spin: 0/0
Turnstile chain at 0xffffffff814c4ec0.
=: No active turnstile for this lock.
trace: pid 17072 lid 1 at 0xffff80009fa24ca0
knote_detach() at netbsd:knote_detach+0x66
knote_fdclose() at netbsd:knote_fdclose+0x44
fd_close() at netbsd:fd_close+0x388
fd_free() at netbsd:fd_free+0x17e
exit1() at netbsd:exit1+0x11d
sys_exit() at netbsd:sys_exit+0x3d
syscall() at netbsd:syscall+0x1ed
--- syscall (number 1) ---
7947772fe52a:
Locks held by an LWP (master):
Lock 0 (initialized at filedesc_ctor)
lock address : 0xfffffe822870adc0 type : sleep/adaptive
initialized : 0xffffffff8094d11f
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 1
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe822962bac0
last locked* : 0xffffffff809522e8 unlocked : 0xffffffff8094ec84
owner field : 0xfffffe822962bac0 wait/spin: 0/0
Turnstile chain at 0xffffffff814c5140.
=: No active turnstile for this lock.
trace: pid 759 lid 1 at 0x81051570
?() at 0
address 0x81051578 is invalid
address 0x81051570 is invalid
Locks held by an LWP (ioflush):
Lock 0 (initialized at vcache_alloc)
lock address : 0xfffffe8061cb6fa0 type : sleep/adaptive
initialized : 0xffffffff80a01652
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 7
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe822dce8100
last locked* : 0xffffffff80a17d69 unlocked : 0xffffffff80a17db4
owner/count : 0xfffffe822dce8100 flags : 0x0000000000000004
Turnstile chain at 0xffffffff814c5100.
=: No active turnstile for this lock.
trace: pid 0 lid 119 at 0xffff80009dce9790
scsipi_adapter_request() at netbsd:scsipi_adapter_request+0x52
scsipi_run_queue() at netbsd:scsipi_run_queue+0xe2
scsipi_execute_xs() at netbsd:scsipi_execute_xs+0x1bb
sd_diskstart() at netbsd:sd_diskstart+0x25b
dk_start() at netbsd:dk_start+0xde
spec_strategy() at netbsd:spec_strategy+0x9a
VOP_STRATEGY() at netbsd:VOP_STRATEGY+0x54
genfs_do_io() at netbsd:genfs_do_io+0x1b4
genfs_gop_write() at netbsd:genfs_gop_write+0x52
genfs_do_putpages() at netbsd:genfs_do_putpages+0xb9c
VOP_PUTPAGES() at netbsd:VOP_PUTPAGES+0x3a
ffs_full_fsync() at netbsd:ffs_full_fsync+0x116
ffs_fsync() at netbsd:ffs_fsync+0x3a
VOP_FSYNC() at netbsd:VOP_FSYNC+0x3e
sched_sync() at netbsd:sched_sync+0x198
Locks held by an LWP (softnet/0):
Lock 0 (initialized at soinit)
lock address : 0xfffffe822fb97f40 type : sleep/adaptive
initialized : 0xffffffff809db0cb
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 14
current cpu : 0 last held: 0
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe822f707420
last locked* : 0xffffffff806d7add unlocked : 0xffffffff80951f2c
owner field : 0xfffffe822f707420 wait/spin: 1/0
Turnstile chain at 0xffffffff814c5040.
=: Turnstile at 0xfffffe810f381d20 (wrq=0xfffffe810f381d40, rdq=0xfffffe810f381d50).
=: 0 waiting readers:
=: 14 waiting writers: 0xfffffe810f396880 0xfffffe810e9a29c0 0xfffffe810e9e0a00 0xfffffe810e94c980 0xfffffe82212fc700 0xfffffe82228e28e0 0xfffffe8222950b60 0xfffffe82212fcb20 0xfffffe822962b6a0 0xfffffe822c341200 0xfffffe8220e621e0 0xfffffe821ea02260 0xfffffe821f0986c0 0xfffffe810f2da300
trace: pid 0 lid 3 at 0xffff80008f68cc00
sowakeup() at netbsd:sowakeup+0x58
tcp_input() at netbsd:tcp_input+0x2f61
tcp6_input() at netbsd:tcp6_input+0x49
ip6_input() at netbsd:ip6_input+0x69e
ip6intr() at netbsd:ip6intr+0x7b
softint_dispatch() at netbsd:softint_dispatch+0xd3
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f68cff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
[Locks tracked through CPUs]
Locks held on CPU 0:
Lock 0 (initialized at main)
lock address : 0xffffffff814c3840 type : spin
initialized : 0xffffffff80d02d91
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 6
current cpu : 0 last held: 0
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe822f707420
last locked* : 0xffffffff806d7ae7 unlocked : 0xffffffff80a3a00f
curcpu holds : 1 wanted by: 000000000000000000
trace: pid 0 lid 6 at 0xffff80008f69bcf8
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
DDB lost frame for netbsd:Xdoreti, trying 0xffff80008f69bd70
Xdoreti() at netbsd:Xdoreti
--- interrupt ---
Bad frame pointer: 0xffff800000000000
202:
Lock 1 (initialized at com_attach_subr)
lock address : 0xfffffe810f20b1e0 type : spin
initialized : 0xffffffff805ce644
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 0
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe822f6eb440
last locked* : 0xffffffff805cdba0 unlocked : 0xffffffff805cdd7f
owner field : 0x0000000000010800 wait/spin: 0/1
trace: pid 0 lid 6 at 0xffff80008f69bcf8
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
DDB lost frame for netbsd:Xdoreti, trying 0xffff80008f69bd70
Xdoreti() at netbsd:Xdoreti
--- interrupt ---
Bad frame pointer: 0xffff800000000000
202:
db{0}: call cpu_reset
From: Ryota Ozaki <ozaki-r@netbsd.org>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc: port-amd64-maintainer@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, Havard Eidnes <he@netbsd.org>
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network
interfaces in use
Date: Fri, 6 Apr 2018 02:24:11 +0900
On Thu, Apr 5, 2018 at 9:25 PM, Havard Eidnes <he@netbsd.org> wrote:
> The following reply was made to PR port-amd64/53155; it has been noted by=
GNATS.
>
> From: Havard Eidnes <he@NetBSD.org>
> To: gnats-bugs@NetBSD.org
> Cc:
> Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
> network interfaces in use
> Date: Thu, 05 Apr 2018 14:22:17 +0200 (CEST)
>
> Following ozaki-r's hints, here's more information with the locks
> dumped on another wedge event.
I've tried to compose a resource dependency graph from the outputs
(if the graph has a loop, there is a deadlock) and this is a result:
[expect] vfs_suspend_lock, fstrans_lock =3D> pserialize_perform =3D> kpau=
se =3D>
[tcp_timer_rexmt (softclk/?)] softnet_lock =3D> [ip6intr (softnet/0)]
sowakeup =3D> ???
I don't know what is going on in sowakeup. If it gets stuck because of
vfs_suspend_lock or fstrans_lock, a loop appears.
vfs experts would find a missing link.
ozaki-r
From: Joerg Sonnenberger <joerg@bec.de>
To: gnats-bugs@NetBSD.org
Cc: port-amd64-maintainer@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, he@NetBSD.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Thu, 5 Apr 2018 21:03:46 +0200
On Thu, Apr 05, 2018 at 07:25:00AM +0000, Havard Eidnes wrote:
> --- interrupt ---
> Bad frame pointer: 0xffff800000000000
> 202:
> db{0}: machine cpu 1
> using CPU 1
> db{0}: tra
> x86_pause() at netbsd:x86_pause+0x2
> cdev_open() at netbsd:cdev_open+0xaf
> spec_open() at netbsd:spec_open+0x260
> VOP_OPEN() at netbsd:VOP_OPEN+0x33
> vn_open() at netbsd:vn_open+0x28b
> do_open() at netbsd:do_open+0x112
> do_sys_openat() at netbsd:do_sys_openat+0x68
> sys_open() at netbsd:sys_open+0x24
> syscall() at netbsd:syscall+0x1ed
Any chance to find out what device this is? I wonder due to the kernel
lock instances on the other cores...
Joerg
From: Havard Eidnes <he@NetBSD.org>
To: joerg@bec.de
Cc: gnats-bugs@NetBSD.org, port-amd64-maintainer@netbsd.org,
netbsd-bugs@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Fri, 06 Apr 2018 00:30:04 +0200 (CEST)
> On Thu, Apr 05, 2018 at 07:25:00AM +0000, Havard Eidnes wrote:
>> --- interrupt ---
>> Bad frame pointer: 0xffff800000000000
>> 202:
>> db{0}: machine cpu 1
>> using CPU 1
>> db{0}: tra
>> x86_pause() at netbsd:x86_pause+0x2
>> cdev_open() at netbsd:cdev_open+0xaf
>> spec_open() at netbsd:spec_open+0x260
>> VOP_OPEN() at netbsd:VOP_OPEN+0x33
>> vn_open() at netbsd:vn_open+0x28b
>> do_open() at netbsd:do_open+0x112
>> do_sys_openat() at netbsd:do_sys_openat+0x68
>> sys_open() at netbsd:sys_open+0x24
>> syscall() at netbsd:syscall+0x1ed
>
> Any chance to find out what device this is? I wonder due to the kerne=
l
> lock instances on the other cores...
For one, the machine is no longer sitting in DDB, it is a
semi-normal production machine so needs to be up most of the
time.
Secondly, how would I go about finding out?
I don't even know how to identify which process this is, perhaps
other than doing a backtrace of each and every process on the
system at the time of the wedge.
However, if I were going to give a guess, I would guess at one of
the expect processes, which typically want to open a pty pair,
which should match the cdev_open() use.
Regards,
- H=E5vard
From: mlelstv@serpens.de (Michael van Elst)
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network interfaces in use
Date: Fri, 6 Apr 2018 05:38:13 -0000 (UTC)
he@NetBSD.org (Havard Eidnes) writes:
>I don't even know how to identify which process this is, perhaps
>other than doing a backtrace of each and every process on the
>system at the time of the wedge.
The ps output has marked the processes running on a CPU:
> PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
> 6110 : 1 7 3 0 fffffe821e439080 expect
> 855 : 1 7 6 0 fffffe820e54e100 expect
> 759 : 1 7 1 0 fffffe822962bac0 master
> 342 : 2 7 0 0 fffffe822d7a1180 dhcpd
> 0 : 119 7 7 200 fffffe822dce8100 ioflush
> 0 : 59 7 5 200 fffffe810eb61aa0 ipmi
> 0 : 35 7 4 201 fffffe810e9739a0 idle/4
> 0 : 6 7 0 200 fffffe822f6eb440 softser/0
> 0 : 3 7 0 200 fffffe822f707420 softnet/0
^
|
>> db{0}: machine cpu 1
>> using CPU 1
>> db{0}: tra
>> x86_pause() at netbsd:x86_pause+0x2
>> cdev_open() at netbsd:cdev_open+0xaf
>> spec_open() at netbsd:spec_open+0x260
>> VOP_OPEN() at netbsd:VOP_OPEN+0x33
>> vn_open() at netbsd:vn_open+0x28b
>> do_open() at netbsd:do_open+0x112
>> do_sys_openat() at netbsd:do_sys_openat+0x68
>> sys_open() at netbsd:sys_open+0x24
>> syscall() at netbsd:syscall+0x1ed
Assuming both outputs are from the same state: CPU 1 is running PID 759 (master).
From "netbsd:cdev_open+0xaf" I would also think it's waiting on KERNEL_LOCK
which is held by:
> current lwp : 0xfffffe822f6eb440 last held: 0xfffffe822f6eb440
> 0 : 6 7 0 200 fffffe822f6eb440 softser/0
Unfortunately there is no backtrace of that LWP.
--
--
Michael van Elst
Internet: mlelstv@serpens.de
"A potential Snark may lurk in every tree."
From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org, mlelstv@serpens.de
Cc: netbsd-bugs@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Fri, 06 Apr 2018 08:47:08 +0200 (CEST)
> >I don't even know how to identify which process this is, perhaps
> >other than doing a backtrace of each and every process on the
> >system at the time of the wedge.
>
> The ps output has marked the processes running on a CPU:
Ah, of course, thanks.
> Assuming both outputs are from the same state: CPU 1 is running PID =
759 (master).
> From "netbsd:cdev_open+0xaf" I would also think it's waiting on KERN=
EL_LOCK
> which is held by:
>
> > current lwp : 0xfffffe822f6eb440 last held: 0xfffffe822f6eb440
>
> > 0 : 6 7 0 200 fffffe822f6eb440 softser/0=
>
> Unfortunately there is no backtrace of that LWP.
OK, we'll provoke another wedge and I'll make sure to collect
this information sometime later today (I hope).
Regards,
- H=E5vard
From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org, mlelstv@serpens.de
Cc: netbsd-bugs@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Fri, 06 Apr 2018 14:21:00 +0200 (CEST)
OK, now I should have at least the backtraces of all the PIDs
active on CPUs + those in tstile state.
"DDB key sequence typed"
fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0x75e5b2f26020 ilevel 0x8 rsp 0xffff80008f69bc30
curlwp 0xfffffe822f6eb440 pid 0.6 lowest kstack 0xffff80008f6972c0
Stopped in pid 0.6 (system) at netbsd:breakpoint+0x5: leave
db{0}: tra
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
uhci_intr() at netbsd:uhci_intr+0x19
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}: show all locks /t
[Locks tracked through LWPs]
Locks held by an LWP (pickup):
Lock 0 (initialized at filedesc_ctor)
lock address : 0xfffffe821eb37240 type : sleep/adaptive
initialized : 0xffffffff8094d11f
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 3
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe81711d9960
last locked* : 0xffffffff8095362b unlocked : 0xffffffff809536d4
owner field : 0xfffffe81711d9960 wait/spin: 0/0
Turnstile chain at 0xffffffff814c4e40.
=: No active turnstile for this lock.
trace: pid 22385 lid 1 at 0xffff80009ed64e70
kevent1() at netbsd:kevent1+0x648
sys___kevent50() at netbsd:sys___kevent50+0x33
syscall() at netbsd:syscall+0x1ed
--- syscall (number 435) ---
72a5bf03e40a:
Locks held by an LWP (master):
Lock 0 (initialized at filedesc_ctor)
lock address : 0xfffffe822bcb1a00 type : sleep/adaptive
initialized : 0xffffffff8094d11f
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 1
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe822d043a00
last locked* : 0xffffffff809522e8 unlocked : 0xffffffff8094ec84
owner field : 0xfffffe822d043a00 wait/spin: 0/0
Turnstile chain at 0xffffffff814c4dc0.
=: No active turnstile for this lock.
trace: pid 735 lid 1 at 0xffff80009dea1bd0
kqueue_register() at netbsd:kqueue_register+0x431
kevent1() at netbsd:kevent1+0x150
sys___kevent50() at netbsd:sys___kevent50+0x33
syscall() at netbsd:syscall+0x1ed
--- syscall (number 435) ---
74de5623e40a:
Locks held by an LWP (npfgc-0):
Lock 0 (initialized at npf_config_init)
lock address : 0xfffffe822dd5f4c8 type : sleep/adaptive
initialized : 0xffffffff8186d8cf
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 6
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe822d96f160
last locked* : 0xffffffff81872df3 unlocked : 0xffffffff81872e03
owner field : 0xfffffe822d96f160 wait/spin: 0/0
Turnstile chain at 0xffffffff814c4f50.
=: No active turnstile for this lock.
trace: pid 0 lid 123 at 0xffff80009dd9edb0
sleepq_block() at netbsd:sleepq_block+0x97
cv_wait() at netbsd:cv_wait+0x127
xc_wait() at netbsd:xc_wait+0x7c
pserialize_perform() at netbsd:pserialize_perform+0xdf
npf_conn_gc() at npf:npf_conn_gc+0x1b5
npf_worker() at npf:npf_worker+0x4b
Locks held by an LWP (ioflush):
Lock 0 (initialized at vcache_alloc)
lock address : 0xfffffe822db39528 type : sleep/adaptive
initialized : 0xffffffff80a01652
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 5
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe822dd08520
last locked* : 0xffffffff80a17d69 unlocked : 0xffffffff80a17db4
owner/count : 0xfffffe822dd08520 flags : 0x0000000000000004
Turnstile chain at 0xffffffff814c5010.
=: No active turnstile for this lock.
trace: pid 0 lid 119 at 0xffff80009dce4790
scsipi_adapter_request() at netbsd:scsipi_adapter_request+0x52
scsipi_run_queue() at netbsd:scsipi_run_queue+0xe2
scsipi_execute_xs() at netbsd:scsipi_execute_xs+0x1bb
sd_diskstart() at netbsd:sd_diskstart+0x25b
dk_start() at netbsd:dk_start+0xde
spec_strategy() at netbsd:spec_strategy+0x9a
VOP_STRATEGY() at netbsd:VOP_STRATEGY+0x54
genfs_do_io() at netbsd:genfs_do_io+0x1b4
genfs_gop_write() at netbsd:genfs_gop_write+0x52
genfs_do_putpages() at netbsd:genfs_do_putpages+0xb9c
VOP_PUTPAGES() at netbsd:VOP_PUTPAGES+0x3a
ffs_full_fsync() at netbsd:ffs_full_fsync+0x116
ffs_fsync() at netbsd:ffs_fsync+0x3a
VOP_FSYNC() at netbsd:VOP_FSYNC+0x3e
sched_sync() at netbsd:sched_sync+0x198
Locks held by an LWP (nd6_timer):
Lock 0 (initialized at soinit)
lock address : 0xfffffe822fb97f40 type : sleep/adaptive
initialized : 0xffffffff809db0cb
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 12
current cpu : 0 last held: 7
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe810f38e880
last locked* : 0xffffffff806e1987 unlocked : 0xffffffff809dce9a
owner field : 0xfffffe810f38e880 wait/spin: 1/0
Turnstile chain at 0xffffffff814c5040.
=: Turnstile at 0xfffffe810f3810c0 (wrq=0xfffffe810f3810e0, rdq=0xfffffe810f3810f0).
=: 0 waiting readers:
=: 12 waiting writers: 0xfffffe810e9a29c0 0xfffffe810f2da720 0xfffffe822d96f9a0 0xfffffe822a70dac0 0xfffffe8223245340 0xfffffe821f081780 0xfffffe822111e2e0 0xfffffe82216935e0 0xfffffe82235c9900 0xfffffe821e590b40 0xfffffe821f30e560 0xfffffe821dc19940
trace: pid 0 lid 93 at 0xffff80009dbc5ea0
nd6_timer_work() at netbsd:nd6_timer_work+0x49
workqueue_worker() at netbsd:workqueue_worker+0xcd
[Locks tracked through CPUs]
Locks held on CPU 0:
Lock 0 (initialized at main)
lock address : 0xffffffff814c3840 type : spin
initialized : 0xffffffff80d02d91
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 7
current cpu : 0 last held: 0
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe8175a02200
last locked* : 0xffffffff80a0ed67 unlocked : 0xffffffff80a0e3ea
curcpu holds : 2 wanted by: 000000000000000000
trace: pid 0 lid 6 at 0xffff80008f69bc30
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
uhci_intr() at netbsd:uhci_intr+0x19
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
Lock 1 (initialized at com_attach_subr)
lock address : 0xfffffe810f20b1e0 type : spin
initialized : 0xffffffff805ce644
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 0
current lwp : 0xfffffe822f6eb440 last held: 0xfffffe822f6eb440
last locked* : 0xffffffff805cdba0 unlocked : 0xffffffff805cdd7f
owner field : 0x0000000000010800 wait/spin: 0/1
trace: pid 0 lid 6 at 0xffff80008f69bc30
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
uhci_intr() at netbsd:uhci_intr+0x19
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}: ps
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
28757 1 3 2 80 fffffe821bb6f840 ssh select
19899 2 3 1 80 fffffe816ce52980 expect select
19899 1 2 1 0 fffffe82273c5b80 expect
26666 1 3 2 80 fffffe82274af2a0 sh wait
6662 1 3 7 80 fffffe8175e19760 perl pipe_rd
2392 1 3 7 80 fffffe8176c60700 sh wait
3520 1 3 1 80 fffffe8015ba2580 ssh select
6776 2 3 7 80 fffffe821bb6f420 expect select
6776 : 1 7 0 0 fffffe8175a02200 expect
26945 1 3 1 80 fffffe822cddb620 sh wait
2043 1 3 0 80 fffffe801faf7460 perl pipe_rd
14744 1 3 7 80 fffffe816f4a2460 sh wait
14052 1 3 3 80 fffffe821dc0b5a0 ssh select
5530 2 2 3 0 fffffe8175b4e060 expect
5530 : 1 7 4 0 fffffe805815d0e0 expect
7604 1 3 7 80 fffffe817c8655a0 sh wait
934 1 2 5 0 fffffe813db3fb00 perl
25713 1 3 7 80 fffffe821cf411c0 sh wait
15595 1 3 0 80 fffffe817197e9c0 ssh select
22126: 2 7 6 0 fffffe822aac6700 expect
22126: 1 7 2 0 fffffe81788f42a0 expect
4276 1 3 7 80 fffffe8174860660 sh wait
11831 1 3 4 80 fffffe822d0431c0 perl pipe_rd
9327 1 3 7 80 fffffe8057cc5520 sh wait
27698 1 3 1 80 fffffe8129e07ae0 ssh select
1953 2 3 2 80 fffffe8175aeeaa0 expect select
1953 1 3 0 80 fffffe8226268260 expect parked
17149 1 3 0 80 fffffe81749f7020 sh wait
4019 1 3 2 80 fffffe817bd8c5e0 perl pipe_rd
17099 1 3 0 80 fffffe821d640040 sh wait
26876 1 3 7 80 fffffe821f30e980 par pause
23287 1 3 7 80 fffffe8229ce8780 sh wait
1075 1 3 4 80 fffffe817197e180 sh wait
17073 1 3 4 80 fffffe817a8eb560 sh wait
28968 1 3 4 80 fffffe822047f500 sh wait
15804 1 3 4 80 fffffe817c7f59a0 cron pipe_rd
24695 1 3 3 80 fffffe80716ad540 ssh select
29259 1 3 7 80 fffffe8020710020 bash wait
4005 1 3 3 80 fffffe81787d0280 sshd select
1624 1 3 0 80 fffffe8216428060 sshd select
22385: 1 7 3 0 fffffe81711d9960 pickup
14283 1 3 0 80 fffffe8228a6a6e0 ssh select
67 1 3 0 80 fffffe8170e612c0 bash wait
7137 1 3 2 80 fffffe8133f6ab80 sshd select
22735 1 3 1 80 fffffe8118cd8b40 sshd select
25974 1 3 7 80 fffffe821e0b10c0 bash select
23420 1 3 0 80 fffffe8170e61b00 sshd select
29656 1 3 0 80 fffffe813db3f2c0 sshd select
28863 1 3 7 80 fffffe81797b3640 bash select
26749 1 3 5 80 fffffe8175e19b80 tcsh pause
27317 1 3 0 80 fffffe816f4a2880 xterm select
22374 1 3 0 80 fffffe821e93ea20 tcsh pause
28940 1 3 2 80 fffffe817197e5a0 sshd select
25310 1 3 0 80 fffffe816e7a8840 sshd select
26934 1 3 0 80 fffffe8174828360 sshd select
28572 1 3 0 80 fffffe82273c5340 sshd select
26471 1 3 0 80 fffffe817c8659c0 ssh select
12600 1 3 3 80 fffffe81797b3a60 bash wait
11468 1 3 6 80 fffffe821af06440 sshd select
14064 1 3 4 80 fffffe8229ce8360 sshd select
27603 1 3 0 80 fffffe8178ff7600 sshd select
19455 1 3 0 80 fffffe8226268680 ssh select
24504 1 3 2 80 fffffe8170e616e0 bash wait
23735 1 3 3 80 fffffe821e7a9060 sshd select
11768 1 3 3 80 fffffe813db3f6e0 sshd select
10560 1 3 0 80 fffffe82164408c0 sshd select
8508 1 3 7 80 fffffe81707941c0 ssh select
9649 1 3 0 80 fffffe816c812a40 bash wait
25381 1 3 0 80 fffffe822722d6a0 sshd select
7565 1 3 1 80 fffffe82197160a0 sshd select
2309 1 3 6 80 fffffe81745ad1a0 ssh select
26354 1 3 7 80 fffffe8173cfc720 bash wait
17405 1 3 5 80 fffffe81711d9120 sshd select
751 1 3 0 80 fffffe822a70d280 sshd select
27677 1 3 0 80 fffffe822722dac0 ssh select
357 1 3 7 80 fffffe816c812620 bash wait
25010 1 3 0 80 fffffe82273c5760 sshd select
1179 1 3 0 80 fffffe8174828ba0 sshd select
26922 1 3 0 80 fffffe821f13e9a0 ssh select
23611 1 3 0 80 fffffe813ceb9b20 bash wait
17211 1 3 3 80 fffffe822004d080 sshd select
19266 1 3 5 80 fffffe81754f6920 sshd select
4685 1 3 1 80 fffffe8118cd8720 ssh select
8078 1 3 1 80 fffffe8174860240 tcsh pause
16167 1 3 0 80 fffffe8173cfcb40 xterm select
10545 1 3 4 80 fffffe821e87f540 tcsh pause
13913 1 3 6 80 fffffe817c81d520 sshd select
16693 1 3 4 80 fffffe821f13e160 sshd select
25137 1 3 6 80 fffffe822b87a680 less ttyraw
4957 1 3 6 80 fffffe81711d9540 tcsh pause
5578 1 3 5 80 fffffe81787d0ac0 xterm select
5847 1 3 3 80 fffffe821bb6f000 tcsh pause
21995 1 3 7 80 fffffe8175a02a40 sshd select
17502 1 3 3 80 fffffe8175a02620 sshd select
7577 1 4 7 1000000 fffffe82164288a0 less
578 1 3 6 80 fffffe821d0d2640 ssh select
1125 1 3 3 80 fffffe821bb785c0 sh wait
22670 1 3 3 80 fffffe821da51a40 ssh select
20005 1 3 7 80 fffffe821e9f5240 sh wait
3107 1 3 7 80 fffffe822aac6b20 ssh select
1604 1 3 6 80 fffffe81794ca5c0 sh wait
19643 1 3 6 80 fffffe816ce52560 ssh select
19274 1 3 3 80 fffffe821cf415e0 sh wait
19209 1 3 5 80 fffffe817a8eb980 ssh select
1745 1 3 3 80 fffffe817beb7900 sh wait
20698 1 3 2 80 fffffe821cf41a00 ssh select
2093 1 3 1 80 fffffe821da51620 sh wait
22181 1 3 5 80 fffffe821f13e580 ssh select
1594 1 3 0 80 fffffe817c77d120 sh wait
9837 1 3 4 80 fffffe821e9f5660 ssh select
20910 1 3 5 80 fffffe821a83d040 sh wait
21449 1 3 1 80 fffffe817b1cfa80 ssh select
1321 1 3 5 80 fffffe81716b2100 sh wait
23060 1 3 5 80 fffffe821e9f5a80 ssh select
1702 1 3 0 80 fffffe8176c602e0 sh wait
1949 1 3 2 80 fffffe821dc19520 ssh select
18511 1 3 4 80 fffffe8219e58b40 sh wait
2206 1 3 3 80 fffffe82298eaae0 ssh select
1312 1 3 2 80 fffffe821bb19320 sh wait
23320 1 3 5 80 fffffe821f30e140 ssh select
19993 1 3 6 80 fffffe81716b2520 sh wait
555 1 3 3 80 fffffe822004d4a0 ssh select
22754 1 3 6 80 fffffe816ce52140 sh wait
545 1 3 6 80 fffffe821deca4c0 ssh select
20813 1 3 6 80 fffffe821fee3740 sh wait
22820 1 3 4 80 fffffe821deca8e0 ssh select
18960 1 3 7 80 fffffe82298ea2a0 sh wait
20254 1 3 6 80 fffffe821e0b14e0 ssh select
20453 1 3 4 80 fffffe81794ca1a0 sh wait
17259 1 3 6 80 fffffe8173ba50a0 ssh select
19827 1 3 4 80 fffffe8220756420 sh wait
1052 1 3 2 80 fffffe821eed1440 ssh select
12864 1 3 7 80 fffffe822bcd5220 sh wait
8700 1 3 3 80 fffffe821eed1020 ssh select
21994 1 3 2 80 fffffe82274afae0 sh wait
21369 1 3 2 80 fffffe816d83b1e0 ssh select
20846 1 3 5 80 fffffe8170794a00 sh wait
21204 1 3 7 80 fffffe8220756840 ssh select
21380 1 3 1 80 fffffe817bd8ca00 sh wait
20422 1 3 2 80 fffffe821f081360 ssh select
20223 1 3 2 80 fffffe8216440080 sh wait
12794 1 3 0 80 fffffe81790a6080 tcsh ttyraw
4984 1 3 7 80 fffffe822004d8c0 tcsh pause
12264 1 3 7 80 fffffe8173ba58e0 xterm select
10847 1 3 0 80 fffffe821d0d2220 tcsh pause
12000 1 3 6 80 fffffe817c77d960 sshd select
12914 1 3 0 80 fffffe8216428480 sshd select
2019 1 3 0 80 fffffe8225286340 getty ttyraw
1589 1 3 2 80 fffffe8227ca52c0 ssh select
816 1 3 2 80 fffffe8227ca5b00 sh wait
953 1 3 2 0 fffffe821f30e560 conserver tstile
306 1 3 4 0 fffffe821dc19940 conserver tstile
1047 1 3 6 0 fffffe821f081780 conserver tstile
534 1 3 6 0 fffffe822111e2e0 conserver tstile
1045 1 3 2 80 fffffe822111e700 ssh select
916 1 3 0 80 fffffe821f081ba0 sh wait
915 1 3 2 0 fffffe8223245340 conserver tstile
274 1 3 2 80 fffffe821fee3b60 ssh select
1041 1 3 5 80 fffffe8222d56580 sh wait
400 1 3 0 80 fffffe8221c54640 ssh select
527 1 3 0 80 fffffe8223245760 sh wait
782 1 3 4 80 fffffe8223245b80 ssh select
1165 1 3 1 80 fffffe822200f5c0 sh wait
908 1 3 3 80 fffffe822200f1a0 ssh select
267 1 3 0 80 fffffe821fee3320 sh wait
264 1 3 3 80 fffffe821e590300 ssh select
903 1 3 1 80 fffffe821e590720 sh wait
774 1 3 6 0 fffffe821e590b40 conserver tstile
1155 1 3 6 80 fffffe822111eb20 ssh select
386 1 3 6 80 fffffe8221c54220 sh wait
385 1 3 4 80 fffffe822047b2c0 ssh select
384 1 3 0 80 fffffe822a01e4a0 sh wait
146 1 3 0 80 fffffe822047b6e0 ssh select
959 1 3 7 80 fffffe822047bb00 sh wait
958 1 3 7 80 fffffe8220e372a0 ssh select
445 1 3 7 80 fffffe8220e376c0 sh wait
700 1 3 1 80 fffffe8220e37ae0 ssh select
635 1 3 0 80 fffffe8220f27280 sh wait
698 1 3 7 80 fffffe8220f276a0 ssh select
761 1 3 0 80 fffffe8220f27ac0 sh wait
696 1 3 2 80 fffffe822195e260 ssh select
951 1 3 0 80 fffffe822195e680 sh wait
566 1 3 4 80 fffffe822195eaa0 ssh select
1141 1 3 0 80 fffffe8220529240 sh wait
756 1 3 5 80 fffffe8220529660 ssh select
691 1 3 0 80 fffffe8220529a80 sh wait
368 1 3 1 80 fffffe8221c54a60 ssh select
559 1 3 0 80 fffffe821f738200 sh wait
814 1 3 4 80 fffffe821f738620 ssh select
685 1 3 1 80 fffffe821f738a40 sh wait
620 1 3 5 80 fffffe8221b2f1e0 ssh select
171 1 3 0 80 fffffe8221b2f600 sh wait
362 1 3 3 80 fffffe8221b2fa20 ssh select
361 1 3 0 80 fffffe82216931c0 sh wait
360 1 3 2 80 fffffe822a70d6a0 ssh select
679 1 3 6 0 fffffe82216935e0 conserver tstile
550 1 3 1 80 fffffe8221693a00 sh wait
1123 1 3 4 80 fffffe822200f9e0 ssh select
354 1 3 0 80 fffffe8222a36180 sh wait
161 1 3 5 80 fffffe8222c13480 ssh select
160 1 3 6 80 fffffe822a01e080 sh wait
799 1 3 0 80 fffffe8222a365a0 ssh select
990 1 3 2 80 fffffe8222c138a0 sh wait
285 1 3 7 80 fffffe822d8565a0 ssh select
668 1 3 1 80 fffffe8227206740 sh wait
539 1 3 4 80 fffffe8222a369c0 ssh select
538 1 3 2 80 fffffe8222d56160 sh wait
919 1 3 2 80 fffffe822d96b560 ssh select
406 1 3 1 80 fffffe822da69540 sh wait
917 1 3 1 80 fffffe8222d569a0 ssh select
596 1 3 5 80 fffffe8222213140 sh wait
723 1 3 5 80 fffffe8222213560 ssh select
210 1 3 0 80 fffffe8222213980 sh wait
1105 1 3 4 80 fffffe8220d58120 ssh select
208 1 3 0 80 fffffe8220d58540 sh wait
399 1 3 6 80 fffffe822cddba40 ssh select
846 1 3 6 80 fffffe8220d58960 sh wait
781 1 3 7 80 fffffe82232e6100 ssh select
524 1 3 0 80 fffffe82232e6520 sh wait
139 1 3 4 80 fffffe82232e6940 ssh select
138 1 3 0 80 fffffe8223fe90e0 sh wait
137 1 3 1 80 fffffe8223fe9500 ssh select
136 1 3 1 80 fffffe8223fe9920 sh wait
519 1 3 7 80 fffffe822dd92500 ssh select
646 1 3 2 80 fffffe82235c90c0 sh wait
709 1 3 5 80 fffffe82235c94e0 ssh select
708 1 3 6 0 fffffe82235c9900 conserver tstile
963 1 3 0 80 fffffe82231dc0a0 sh wait
194 1 3 2 80 fffffe82231dc4c0 ssh select
193 1 3 0 80 fffffe82231dc8e0 sh wait
73 1 3 6 80 fffffe822a01e8c0 ssh select
72 1 3 5 80 fffffe8222c13060 sh wait
604 1 3 0 80 fffffe8224f36040 ssh select
895 1 3 5 80 fffffe8224f36460 sh wait
636 1 3 3 80 fffffe8224f36880 ssh select
381 1 3 5 80 fffffe8227747700 sh wait
728 1 3 3 80 fffffe822d96b980 ssh select
634 1 3 5 80 fffffe822d6b49e0 sh wait
564 1 3 6 80 fffffe8223956020 ssh select
949 1 3 1 80 fffffe8223956440 sh wait
517 1 3 0 80 fffffe8223956860 ssh select
178 1 3 0 80 fffffe8223182000 sh wait
913 1 3 2 80 fffffe8223182420 ssh select
176 1 3 0 80 fffffe822d856180 sh wait
495 1 3 3 80 fffffe822d8569c0 ssh select
717 1 3 3 80 fffffe82277472e0 sh wait
169 1 3 5 80 fffffe822da69960 ssh select
170 1 3 0 80 fffffe8223182840 sh wait
168 1 3 1 80 fffffe822d96b140 ssh select
615 1 3 0 80 fffffe82234ca360 sh wait
582 1 3 1 80 fffffe82234ca780 ssh select
516 1 3 6 80 fffffe82234caba0 sh wait
931 1 3 4 80 fffffe822d96f580 conserver select
162 1 3 1 0 fffffe822d96f9a0 conserver tstile
853 1 3 0 80 fffffe8227206b60 conserver select
894 1 3 0 80 fffffe8225286760 conserver select
475 1 3 4 80 fffffe8225286b80 cron nanoslp
569 1 2 2 0 fffffe8227206320 python2.7
460 1 3 1 80 fffffe822bd2e660 inetd kqueue
97 1 3 5 80 fffffe8227747b20 nginx kqueue
96 1 3 5 80 fffffe8228a6a2c0 nginx pause
531 1 2 4 0 fffffe822d3cf1e0 qmgr
735 : 1 7 1 0 fffffe822d043a00 master
486 1 3 0 80 fffffe822b87a260 sshd select
473 1 3 3 0 fffffe822a70dac0 tac_plus tstile
428 1 3 5 80 fffffe822bd2ea80 powerd kqueue
420 1 3 3 1000000 fffffe810f2da720 ntpd tstile
342 4 3 3 80 fffffe822b87aaa0 dhcpd parked
342 3 3 3 80 fffffe822bd2e240 dhcpd kqueue
342 2 3 0 80 fffffe822bcd5a60 dhcpd parked
342 1 3 2 80 fffffe822bcd5640 dhcpd parked
221 1 2 0 0 fffffe822d0435e0 syslogd
1 1 3 2 80 fffffe810f4fa900 init wait
0 123 3 6 200 fffffe822d96f160 npfgc-0 xchicv
0 122 3 5 200 fffffe822d6b45c0 ipmi0 ipmi0
0 121 3 7 200 fffffe822dd920e0 physiod physiod
0 120 3 3 200 fffffe822dd08100 aiodoned aiodoned
0 : 119 7 5 200 fffffe822dd08520 ioflush
0 118 3 0 200 fffffe822dd08940 pgdaemon pgdaemon
0 115 3 0 280 fffffe822dd92920 spkr1 bellcv
0 114 3 4 280 fffffe810f4fa0c0 audiomix play
0 113 3 4 280 fffffe810ed3b2a0 audiorec record
0 112 3 0 200 fffffe810f2f1b60 atapibus0 sccomp
0 110 2 2 200 fffffe810f20d2c0 usb3
0 109 2 6 200 fffffe810f2da300 usb0
0 108 2 2 200 fffffe810f2cb2e0 usb2
0 107 2 2 200 fffffe810f2dab40 usb5
0 106 3 0 200 fffffe810f2cbb20 usb1 usbevt
0 105 2 2 200 fffffe810f4fa4e0 usb4
0 104 3 0 200 fffffe810f4b30a0 rt_free rt_free
0 103 3 0 200 fffffe810f4b34c0 unpgc unpgc
0 102 3 0 200 fffffe810f4b38e0 key_timehandler key_timehandler
0 101 3 7 200 fffffe810f474080 icmp6_wqinput/7 icmp6_wqinput
0 100 3 6 200 fffffe810f4744a0 icmp6_wqinput/6 icmp6_wqinput
0 99 3 5 200 fffffe810f4748c0 icmp6_wqinput/5 icmp6_wqinput
0 98 3 4 200 fffffe810f4b5060 icmp6_wqinput/4 icmp6_wqinput
0 97 3 3 200 fffffe810f4b5480 icmp6_wqinput/3 icmp6_wqinput
0 96 3 2 200 fffffe810f4b58a0 icmp6_wqinput/2 icmp6_wqinput
0 95 3 1 200 fffffe810f38e040 icmp6_wqinput/1 icmp6_wqinput
0 94 3 0 200 fffffe810f38e460 icmp6_wqinput/0 icmp6_wqinput
0 : 93 7 7 200 fffffe810f38e880 nd6_timer
0 92 3 7 200 fffffe810f35f020 icmp_wqinput/7 icmp_wqinput
0 91 3 6 200 fffffe810f35f440 icmp_wqinput/6 icmp_wqinput
0 90 3 5 200 fffffe810f35f860 icmp_wqinput/5 icmp_wqinput
0 89 3 4 200 fffffe810f378000 icmp_wqinput/4 icmp_wqinput
0 88 3 3 200 fffffe810f378420 icmp_wqinput/3 icmp_wqinput
0 87 3 2 200 fffffe810f378840 icmp_wqinput/2 icmp_wqinput
0 86 3 1 200 fffffe810f331360 icmp_wqinput/1 icmp_wqinput
0 85 3 0 200 fffffe810f331780 icmp_wqinput/0 icmp_wqinput
0 84 3 4 200 fffffe810f2cb700 rt_timer rt_timer
0 83 3 0 200 fffffe810f331ba0 vmem_rehash vmem_rehash
0 82 3 7 200 fffffe810f2f0340 coretemp3 coretemp3
0 81 3 6 200 fffffe810f2f0760 coretemp2 coretemp2
0 80 3 7 200 fffffe810f2f0b80 coretemp1 coretemp1
0 79 3 1 200 fffffe810f2f1320 coretemp0 coretemp0
0 78 3 1 200 fffffe810f2f1740 mfi0 mfi0
0 69 3 5 200 fffffe810f20d6e0 atabus1 atath
0 68 3 5 200 fffffe810f20db00 atabus0 atath
0 66 3 0 200 fffffe810ed3b6c0 scsibus0 sccomp
0 65 3 0 200 fffffe810ed3bae0 usbtask-dr usbtsk
0 64 3 0 200 fffffe810ecaa280 usbtask-hc usbtsk
0 63 3 0 200 fffffe810ecaa6a0 bnx3 bnx3
0 62 3 2 200 fffffe810ecaaac0 bnx2 bnx2
0 61 3 1 200 fffffe810eb61260 bnx1 bnx1
0 60 3 7 200 fffffe810eb61680 bnx0 bnx0
0 59 3 0 200 fffffe810eb61aa0 ipmi ipmipoll
0 58 3 7 200 fffffe810ea7c240 xcall/7 xcall
0 57 1 7 200 fffffe810ea7c660 softser/7
0 56 1 7 200 fffffe810ea7ca80 softclk/7
0 55 1 7 200 fffffe810ea5d220 softbio/7
0 54 1 7 200 fffffe810ea5d640 softnet/7
0 53 1 7 201 fffffe810ea5da60 idle/7
0 52 3 6 200 fffffe810ea36200 xcall/6 xcall
0 51 1 6 200 fffffe810ea36620 softser/6
0 50 1 6 200 fffffe810ea36a40 softclk/6
0 49 1 6 200 fffffe810ea0f1e0 softbio/6
0 48 1 6 200 fffffe810ea0f600 softnet/6
0 47 1 6 201 fffffe810ea0fa20 idle/6
0 46 3 5 200 fffffe810e9e01c0 xcall/5 xcall
0 45 1 5 200 fffffe810e9e05e0 softser/5
0 44 1 5 200 fffffe810e9e0a00 softclk/5
0 43 1 5 200 fffffe810e9d11a0 softbio/5
0 42 1 5 200 fffffe810e9d15c0 softnet/5
0 41 1 5 201 fffffe810e9d19e0 idle/5
0 40 3 4 200 fffffe810e9a2180 xcall/4 xcall
0 39 1 4 200 fffffe810e9a25a0 softser/4
0 38 3 4 200 fffffe810e9a29c0 softclk/4 tstile
0 37 1 4 200 fffffe810e973160 softbio/4
0 36 1 4 200 fffffe810e973580 softnet/4
0 35 1 4 201 fffffe810e9739a0 idle/4
0 34 3 3 200 fffffe810e94c140 xcall/3 xcall
0 33 1 3 200 fffffe810e94c560 softser/3
0 32 1 3 200 fffffe810e94c980 softclk/3
0 31 1 3 200 fffffe810e93d120 softbio/3
0 30 1 3 200 fffffe810e93d540 softnet/3
0 29 1 3 201 fffffe810e93d960 idle/3
0 28 3 2 200 fffffe810e906100 xcall/2 xcall
0 27 1 2 200 fffffe810e906520 softser/2
0 26 1 2 200 fffffe810e906940 softclk/2
0 25 1 2 200 fffffe810e8ef0e0 softbio/2
0 24 1 2 200 fffffe810e8ef500 softnet/2
0 23 1 2 201 fffffe810e8ef920 idle/2
0 22 3 1 200 fffffe810e8c50c0 xcall/1 xcall
0 21 1 1 200 fffffe810e8c54e0 softser/1
0 20 1 1 200 fffffe810e8c5900 softclk/1
0 19 1 1 200 fffffe810e8b10a0 softbio/1
0 18 1 1 200 fffffe810e8b14c0 softnet/1
0 17 1 1 201 fffffe810e8b18e0 idle/1
0 16 3 1 200 fffffe822de92080 lnxsyswq lnxsyswq
0 15 3 0 200 fffffe822de924a0 sysmon smtaskq
0 14 3 0 200 fffffe822de928c0 pmfsuspend pmfsuspend
0 13 3 5 200 fffffe822e2b1060 pmfevent pmfevent
0 12 3 0 200 fffffe822e2b1480 sopendfree sopendfr
0 11 3 7 200 fffffe822e2b18a0 nfssilly nfssilly
0 10 2 6 200 fffffe822f6d0040 cachegc
0 9 3 5 200 fffffe822f6d0460 vdrain vdrain
0 8 3 0 200 fffffe822f6d0880 modunload mod_unld
0 7 3 0 200 fffffe822f6eb020 xcall/0 xcall
0 : 6 7 0 200 fffffe822f6eb440 softser/0
0 : 5 7 0 200 fffffe822f6eb860 softclk/0
0 4 1 0 200 fffffe822f707000 softbio/0
0 3 1 0 200 fffffe822f707420 softnet/0
0 2 1 0 201 fffffe822f707840 idle/0
0 1 2 2 200 ffffffff81481b20 swapper
db{0}: trace/t 0t6776
trace: pid 6776 lid 2 at 0xffff80009f5d9c00
sleepq_block() at netbsd:sleepq_block+0x97
sel_do_scan() at netbsd:sel_do_scan+0x4d1
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7426b363e10a:
db{0}: trace/t 0t5530
trace: pid 5530 lid 2 at 0xffff80009f752c00
sleepq_block() at netbsd:sleepq_block+0x97
sel_do_scan() at netbsd:sel_do_scan+0x4d1
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
7237e283e10a:
db{0}: trace/t 0t22126
trace: pid 22126 lid 2 at 0xffff80009f826c00
VOP_POLL() at netbsd:VOP_POLL+0x7a
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
789c3e03e10a:
db{0}: trace/a fffffe822aac6700
trace: pid 22126 lid 2 at 0xffff80009f826c00
VOP_POLL() at netbsd:VOP_POLL+0x7a
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
789c3e03e10a:
db{0}: trace/a fffffe81788f42a0
trace: pid 22126 lid 1 at 0xffff80009f64bd80
VOP_FCNTL() at netbsd:VOP_FCNTL+0x4b
sys_fcntl() at netbsd:sys_fcntl+0x254
syscall() at netbsd:syscall+0x1ed
--- syscall (number 92) ---
789c3e03e48a:
db{0}: trace/t 0t22385
trace: pid 22385 lid 1 at 0xffff80009ed64e70
kevent1() at netbsd:kevent1+0x648
sys___kevent50() at netbsd:sys___kevent50+0x33
syscall() at netbsd:syscall+0x1ed
--- syscall (number 435) ---
72a5bf03e40a:
db{0}: trace/t 0t735
trace: pid 735 lid 1 at 0xffff80009dea1bd0
kqueue_register() at netbsd:kqueue_register+0x431
kevent1() at netbsd:kevent1+0x150
sys___kevent50() at netbsd:sys___kevent50+0x33
syscall() at netbsd:syscall+0x1ed
--- syscall (number 435) ---
74de5623e40a:
db{0}: trace/a fffffe822dd08520
trace: pid 0 lid 119 at 0xffff80009dce4790
scsipi_adapter_request() at netbsd:scsipi_adapter_request+0x52
scsipi_run_queue() at netbsd:scsipi_run_queue+0xe2
scsipi_execute_xs() at netbsd:scsipi_execute_xs+0x1bb
sd_diskstart() at netbsd:sd_diskstart+0x25b
dk_start() at netbsd:dk_start+0xde
spec_strategy() at netbsd:spec_strategy+0x9a
VOP_STRATEGY() at netbsd:VOP_STRATEGY+0x54
genfs_do_io() at netbsd:genfs_do_io+0x1b4
genfs_gop_write() at netbsd:genfs_gop_write+0x52
genfs_do_putpages() at netbsd:genfs_do_putpages+0xb9c
VOP_PUTPAGES() at netbsd:VOP_PUTPAGES+0x3a
ffs_full_fsync() at netbsd:ffs_full_fsync+0x116
ffs_fsync() at netbsd:ffs_fsync+0x3a
VOP_FSYNC() at netbsd:VOP_FSYNC+0x3e
sched_sync() at netbsd:sched_sync+0x198
db{0}: trace/a fffffe810f38e880
trace: pid 0 lid 93 at 0xffff80009dbc5ea0
nd6_timer_work() at netbsd:nd6_timer_work+0x49
workqueue_worker() at netbsd:workqueue_worker+0xcd
db{0}: trace/a fffffe822f6eb440
trace: pid 0 lid 6 at 0xffff80008f69bc30
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
uhci_intr() at netbsd:uhci_intr+0x19
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}: trace/a fffffe822f6eb860
trace: pid 0 lid 5 at 0xffff80009edadd60
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
ffff80009edadf00:
db{0}: machine cpu 0
using CPU 0
db{0}: tra
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
uhci_intr() at netbsd:uhci_intr+0x19
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}: machine cpu 1
using CPU 1
db{0}: tra
x86_pause() at netbsd:x86_pause
_kernel_lock() at netbsd:_kernel_lock+0x183
kqueue_register() at netbsd:kqueue_register+0x431
kevent1() at netbsd:kevent1+0x150
sys___kevent50() at netbsd:sys___kevent50+0x33
syscall() at netbsd:syscall+0x1ed
--- syscall (number 435) ---
74de5623e40a:
db{0}: machine cpu 2
using CPU 2
db{0}: tra
x86_pause() at netbsd:x86_pause+0x2
VOP_FCNTL() at netbsd:VOP_FCNTL+0x4b
sys_fcntl() at netbsd:sys_fcntl+0x254
syscall() at netbsd:syscall+0x1ed
--- syscall (number 92) ---
789c3e03e48a:
db{0}: machine cpu 3
using CPU 3
db{0}: tra
x86_pause() at netbsd:x86_pause+0x2
kevent1() at netbsd:kevent1+0x648
sys___kevent50() at netbsd:sys___kevent50+0x33
syscall() at netbsd:syscall+0x1ed
--- syscall (number 435) ---
72a5bf03e40a:
db{0}: machine cpu 4
using CPU 4
db{0}: tra
x86_pause() at netbsd:x86_pause+0x2
VOP_FCNTL() at netbsd:VOP_FCNTL+0x4b
sys_fcntl() at netbsd:sys_fcntl+0x254
syscall() at netbsd:syscall+0x1ed
--- syscall (number 92) ---
7237e283e48a:
db{0}: machine cpu 5
using CPU 5
db{0}: tra
x86_pause() at netbsd:x86_pause+0x2
scsipi_adapter_request() at netbsd:scsipi_adapter_request+0x52
scsipi_run_queue() at netbsd:scsipi_run_queue+0xe2
scsipi_execute_xs() at netbsd:scsipi_execute_xs+0x1bb
sd_diskstart() at netbsd:sd_diskstart+0x25b
dk_start() at netbsd:dk_start+0xde
spec_strategy() at netbsd:spec_strategy+0x9a
VOP_STRATEGY() at netbsd:VOP_STRATEGY+0x54
genfs_do_io() at netbsd:genfs_do_io+0x1b4
genfs_gop_write() at netbsd:genfs_gop_write+0x52
genfs_do_putpages() at netbsd:genfs_do_putpages+0xb9c
VOP_PUTPAGES() at netbsd:VOP_PUTPAGES+0x3a
ffs_full_fsync() at netbsd:ffs_full_fsync+0x116
ffs_fsync() at netbsd:ffs_fsync+0x3a
VOP_FSYNC() at netbsd:VOP_FSYNC+0x3e
sched_sync() at netbsd:sched_sync+0x198
db{0}: machine cpu 6
using CPU 6
db{0}: tra
_kernel_lock() at netbsd:_kernel_lock+0x17e
VOP_POLL() at netbsd:VOP_POLL+0x7a
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
789c3e03e10a:
db{0}: machine cpu 7
using CPU 7
db{0}: tra
_kernel_lock() at netbsd:_kernel_lock+0x1a1
nd6_timer_work() at netbsd:nd6_timer_work+0x49
workqueue_worker() at netbsd:workqueue_worker+0xcd
db{0}: trace/a fffffe82273c5b80
trace: pid 19899 lid 1 at 0xffff80009f20de10
sleepq_block() at netbsd:sleepq_block+0x106
lwp_park() at netbsd:lwp_park+0x107
sys____lwp_park60() at netbsd:sys____lwp_park60+0x52
syscall() at netbsd:syscall+0x1ed
--- syscall (number 478) ---
7b611f09998a:
db{0}: trace/t 0t953
trace: pid 953 lid 1 at 0xffff80009ed78b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
776936e3e10a:
db{0}: trace/t 0t306
trace: pid 306 lid 1 at 0xffff80009ed34b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
776936e3e10a:
db{0}: trace/t 0t1047
trace: pid 1047 lid 1 at 0xffff80009ec09b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
776936e3e10a:
db{0}: trace/t 0t534
trace: pid 534 lid 1 at 0xffff80009eb6eb00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
776936e3e10a:
db{0}: trace/t 0t915
trace: pid 915 lid 1 at 0xffff80009ebadb00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
776936e3e10a:
db{0}: trace/t 0t774
trace: pid 774 lid 1 at 0xffff80009eb73b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
776936e3e10a:
db{0}: trace/t 0t679
trace: pid 679 lid 1 at 0xffff80009ea98b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
776936e3e10a:
db{0}: trace/t 0t708
trace: pid 708 lid 1 at 0xffff80009e9c4b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
776936e3e10a:
db{0}: trace/t 0t162
trace: pid 162 lid 1 at 0xffff80009dd94b00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x324
selcommon.part.2() at netbsd:selcommon.part.2+0x18b
sys___select50() at netbsd:sys___select50+0x6f
syscall() at netbsd:syscall+0x1ed
--- syscall (number 417) ---
776936e3e10a:
db{0}: trace/t 0t473
trace: pid 473 lid 1 at 0xffff80009e179b20
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sopoll() at netbsd:sopoll+0x1c
sel_do_scan() at netbsd:sel_do_scan+0x176
pollcommon.part.3() at netbsd:pollcommon.part.3+0x8d
sys_poll() at netbsd:sys_poll+0x9b
syscall() at netbsd:syscall+0x1ed
--- syscall (number 209) ---
720c50e3e28a:
db{0}: trace/t 0t420
trace: pid 420 lid 1 at 0xffff80009db6eae0
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
sosend() at netbsd:sosend+0x48
do_sys_sendmsg_so() at netbsd:do_sys_sendmsg_so+0x272
do_sys_sendmsg() at netbsd:do_sys_sendmsg+0x85
sys_sendto() at netbsd:sys_sendto+0x5c
syscall() at netbsd:syscall+0x1ed
--- syscall (number 133) ---
7334786eea7a:
db{0}: trace/a fffffe810e9a29c0
trace: pid 0 lid 38 at 0xffff80008f771e00
sleepq_block() at netbsd:sleepq_block+0x97
turnstile_block() at netbsd:turnstile_block+0x3e5
mutex_enter() at netbsd:mutex_enter+0x36d
tcp_timer_rexmt() at netbsd:tcp_timer_rexmt+0x27
callout_softclock() at netbsd:callout_softclock+0x201
softint_dispatch() at netbsd:softint_dispatch+0xd3
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f771ff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}: call cpu_reset
From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Wed, 16 May 2018 11:44:47 +0200 (CEST)
Hi,
we provoked another wedge, and captured a kernel core dump from
the wedging machine. The kernel core dump and NetBSD images are
available for looking at.
using gdb and crash, here's a brief summary of the "interesting"
processes in the crash dump, and below that I include backtraces
of all the waiting processes.
There's lots of contention for fstrans_lock.
Offhand I don't see a deadlock which might explain the observed
behaviour (goes totally "deaf" on the network, i.e. doesn't even
respond to ping).
It doesn't look like gdb can trace through interrupt frames (?),
looking at proc 788 all I get is:
(gdb) kvm proc 0xfffffe8220b8e360
0xffffffff8021cfe0 in softintr_ret ()
(gdb) where
#0 0xffffffff8021cfe0 in softintr_ret ()
#1 0x0000000000000000 in ?? ()
(gdb) =
Crash manages to do this one, though, apparently (see below).
Furthermore, using the various gdb scripts in
/usr/src/sys/gdbscripts/ I can look at some of the locks.
It is quite possible that up'ing the interface in question causes
lots of activity for opening pty pairs, and that the root cause
of the issue is there rather than related to networking in itself(?)
Further hints?
Regards,
- H=E5vard
------------------------------
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
7052 1 3 1 8020000 fffffe8220c58540 cron tstile=
Wants fstrans_lock
9187 1 3 6 8020000 fffffe821faa60c0 expect xchicv=
Holds fstrans_lock, in pserialize_perform, waits on condition variabl=
e
after doing xc_broadcast(XC_HIGHPRI, nullop)
Doing (roughly) pty_grant_slave -> genfs_revoke -> vfs_suspend -> =
fstrans_setstate -> pserialize_perform -> xc_wait -> cv_wait
3794 1 3 7 8020000 fffffe82045da540 ssh tstile=
Wants fstrans_lock
2853 2 3 6 8020000 fffffe8216d35140 expect tstile=
Wants fstrans_lock
2853 1 3 7 8020000 fffffe82214bf2c0 expect fdclos=
e
Doing cv_wait(&ff->ff_closing, &fdp->fd_lock);
8327 2 3 3 8020000 fffffe8225709000 expect tstile=
Wants fstrans_lock
8327 1 3 1 8020000 fffffe820df34480 expect fdclos=
e
Doing cv_wait(&ff->ff_closing, &fdp->fd_lock);
5687 2 3 3 8020000 fffffe8207fd0360 expect tstile=
Wants fstrans_lock
5687 1 3 2 8020000 fffffe822346f080 expect fdclos=
e
Doing cv_wait(&ff->ff_closing, &fdp->fd_lock);
9671 2 3 5 8020000 fffffe821d987a40 expect tstile=
Wants fstrans_lock
9671 1 3 4 8020000 fffffe821e90c240 expect fdclos=
e
Doing cv_wait(&ff->ff_closing, &fdp->fd_lock);
704 1 3 6 8020000 fffffe821d362980 conserver tstile=
Wants rw_enter(&vip->vi_lock, op);, lock held by 788 apparently
564 1 3 1 8020000 fffffe8229b924c0 conserver tstile=
Wants fstrans_lock
788 > 1 7 0 8020000 fffffe8220b8e360 conserver
Doing softintr_ret(), gdb doesn't trace further (interrupt frame?)
Crash, though, says:
crash> trace/a 0xfffffe8220b8e360
trace: pid 788 lid 1 at 0xffff80009ebfdc40
ufs_post_write_update.isra.0() at ufs_post_write_update.isra.0+0x95
ffs_write() at ffs_write+0x56f
VOP_WRITE() at VOP_WRITE+0x37
vn_write() at vn_write+0xec
dofilewrite() at dofilewrite+0x97
sys_write() at sys_write+0x5f
syscall() at syscall+0x1ed
--- syscall (number 4) ---
764fb503e02a:
crash> =
275 1 3 5 8020000 fffffe8220b8e780 conserver tstile=
Wants rw_enter(&vip->vi_lock, op);, lock held by 788 apparently
878 1 3 2 8020000 fffffe8221e57600 conserver tstile=
Wants fstrans_lock
136 1 3 3 8020000 fffffe8222569900 conserver tstile=
Wants fstrans_lock
773 1 3 1 8020000 fffffe8225709840 conserver tstile=
Wants fstrans_lock
------------------------------
crash> ps
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
7052 1 3 1 8020000 fffffe8220c58540 cron tstile=
8587 1 3 1 8020000 fffffe821d362560 cron wait
9187 1 3 6 8020000 fffffe821faa60c0 expect xchicv=
5064 1 2 6 8020000 fffffe8219f93540 sh
7813 1 2 0 8020000 fffffe8220ecf860 perl
7354 1 2 0 8020000 fffffe8207aba240 sh
3794 1 3 7 8020000 fffffe82045da540 ssh tstile=
2853 2 3 6 8020000 fffffe8216d35140 expect tstile=
2853 1 3 7 8020000 fffffe82214bf2c0 expect fdclos=
e
6097 1 2 0 8020000 fffffe8221165720 sh
5738 1 2 0 8020000 fffffe821e90ca80 perl
4461 1 2 0 8020000 fffffe8207a90aa0 sh
4310 1 2 3 8020000 fffffe8220c2c5a0 ssh
8327 2 3 3 8020000 fffffe8225709000 expect tstile=
8327 1 3 1 8020000 fffffe820df34480 expect fdclos=
e
6086 1 2 5 8020000 fffffe821d2d9a60 sh
3214 1 2 0 8020000 fffffe821f5d8360 perl
5921 1 2 0 8020000 fffffe8207a90260 sh
2890 1 2 2 8020000 fffffe8228bacae0 ssh
3772 1 2 0 8020000 fffffe821a27f420 sh
4425 1 2 0 8020000 fffffe821f5d8780 ssh
5793 1 2 0 8020000 fffffe821d342340 sh
4833 1 2 0 8020000 fffffe82207f5460 ssh
7238 1 2 0 8020000 fffffe822123b640 sh
6312 1 2 4 8020000 fffffe8222a9b500 ssh
7003 1 2 0 8020000 fffffe822160e700 sh
6900 1 2 0 8020000 fffffe8221d82140 ssh
4139 1 2 0 8020000 fffffe822346f8c0 sh
3765 1 2 0 8020000 fffffe821f94f060 ssh
5386 1 2 1 8020000 fffffe8222a7a8c0 sh
4633 1 2 0 8020000 fffffe821f8969c0 ssh
9691 1 2 0 8020000 fffffe821928d180 sh
6528 1 2 7 8020000 fffffe8207f4b460 ssh
5816 1 2 0 8020000 fffffe8207f4b880 sh
6285 1 2 0 8020000 fffffe821e1e28c0 ssh
8366 1 2 0 8020000 fffffe820d267520 sh
8041 1 2 0 8020000 fffffe821f94f8a0 ssh
3407 1 2 0 8020000 fffffe82045da960 sh
2815 1 2 0 8020000 fffffe8225709420 ssh
5687 2 3 3 8020000 fffffe8207fd0360 expect tstile=
5687 1 3 2 8020000 fffffe822346f080 expect fdclos=
e
2928 1 2 0 8020000 fffffe8228bac2a0 sh
5127 1 2 0 8020000 fffffe821b498860 perl
5658 1 2 0 8020000 fffffe820ea5eac0 sh
6746 1 2 0 8020000 fffffe821d443520 ssh
9671 2 3 5 8020000 fffffe821d987a40 expect tstile=
9671 1 3 4 8020000 fffffe821e90c240 expect fdclos=
e
9105 1 2 0 8020000 fffffe8226ea8b40 sh
3990 1 2 0 8020000 fffffe8227069700 perl
4504 1 2 0 8020000 fffffe8207a90680 sh
9019 1 2 0 8020000 fffffe821f4c78e0 ssh
2825 1 2 0 8020000 fffffe821f896180 sh
5779 1 2 0 8020000 fffffe821a27f840 ssh
2846 1 2 0 8020000 fffffe822af57440 sh
3544 1 2 0 8020000 fffffe82216b1a40 ssh
6513 1 2 0 8020000 fffffe822728b460 sh
5965 1 2 0 8020000 fffffe821faa6900 ssh
6327 1 2 0 8020000 fffffe8221837a80 sh
2739 1 2 0 8020000 fffffe822708e740 ssh
6361 1 2 0 8020000 fffffe82239772a0 sh
3534 1 2 0 8020000 fffffe821f94f480 ssh
5988 1 2 0 8020000 fffffe8222a7a080 sh
7283 1 2 0 8020000 fffffe8222b99980 ssh
6596 1 2 0 8020000 fffffe820ea5e6a0 sh
3343 1 2 0 8020000 fffffe8223049740 ssh
7708 1 2 0 8020000 fffffe822da31140 sh
6381 1 2 0 8020000 fffffe8221d82980 ssh
7056 1 2 0 8020000 fffffe821a542500 sh
4161 1 2 0 8020000 fffffe821c708100 ssh
7144 1 2 0 8020000 fffffe8223049b60 sh
6276 1 2 0 8020000 fffffe821d2d9220 ssh
4460 1 2 0 8020000 fffffe821f36a040 sh
6845 1 2 0 8020000 fffffe8222a9b0e0 ssh
4669 1 2 0 8020000 fffffe821e1e24a0 sh
3652 1 2 0 8020000 fffffe821c708520 ssh
5131 1 2 0 8020000 fffffe820df34060 sh
5704 1 2 0 8020000 fffffe8220ce6760 ssh
6031 1 2 0 8020000 fffffe821ecaea00 sh
5875 1 2 0 8020000 fffffe82240071a0 ssh
5464 1 2 0 8020000 fffffe8205110580 sh
2852 1 2 0 8020000 fffffe82207f5040 ssh
3432 1 2 0 8020000 fffffe820dd5e6e0 sh
4917 1 2 0 8020000 fffffe8221317ac0 ssh
2850 1 2 0 8020000 fffffe8207fd0780 sh
3399 1 2 0 8020000 fffffe822d92e9c0 ssh
3497 1 2 0 8020000 fffffe821d987200 sh
3880 1 2 0 8020000 fffffe821d342b80 ssh
4728 1 2 0 8020000 fffffe822188a5a0 sh
3904 1 2 0 8020000 fffffe820df348a0 ssh
4847 1 2 0 8020000 fffffe8204072900 sh
3465 1 2 0 8020000 fffffe8207fd0ba0 ssh
3881 1 2 0 8020000 fffffe821a27f000 sh
4000 1 2 0 8020000 fffffe821928d9c0 ssh
4242 1 2 0 8020000 fffffe8221165300 sh
4074 1 2 0 8020000 fffffe82207f5880 ssh
3902 1 2 0 8020000 fffffe8205110160 sh
3293 1 2 0 8020000 fffffe8224d918a0 ssh
3228 1 2 0 8020000 fffffe821d362140 sh
4064 1 2 0 8020000 fffffe8224b0c760 ssh
4762 1 2 0 8020000 fffffe822037f1a0 sh
2903 1 2 0 8020000 fffffe8221dab1a0 ssh
2218 1 2 0 8020000 fffffe821fdf9b00 sh
2497 1 2 0 8020000 fffffe822728b880 ssh
3080 1 2 0 8020000 fffffe8223563b40 sh
2573 1 2 0 8020000 fffffe8206fea440 ssh
2635 1 2 0 8020000 fffffe822af57020 sh
2114 1 2 0 8020000 fffffe82220aa520 ssh
4141 1 2 0 8020000 fffffe82240079e0 sh
4477 1 2 0 8020000 fffffe822160eb20 ssh
3978 1 2 0 8020000 fffffe821f6f9b60 sh
3243 1 2 0 8020000 fffffe821f1b0680 ssh
2652 1 2 0 8020000 fffffe821c49f5e0 sh
3804 1 2 0 8020000 fffffe821f37d9a0 ssh
4348 1 2 0 8020000 fffffe822c1e7660 sh
4334 1 2 0 8020000 fffffe8220c2c9c0 ssh
3781 1 2 0 8020000 fffffe82214bfb00 sh
3422 1 2 0 8020000 fffffe82216b1200 ssh
3277 1 2 0 8020000 fffffe8221165b40 sh
3997 1 2 0 8020000 fffffe8206fea020 ssh
3863 1 2 0 8020000 fffffe82239776c0 sh
2950 1 2 0 8020000 fffffe821f6f9320 par
2644 1 2 0 8020000 fffffe822220f9a0 ssh
2418 1 2 0 8020000 fffffe822253d1c0 sh
2616 1 2 0 8020000 fffffe821e1e2080 ssh
2724 1 2 0 8020000 fffffe821b0306a0 sh
1968 1 2 0 8020000 fffffe8224d91060 sh
3042 1 2 0 8020000 fffffe8221dab5c0 sh
2259 1 2 0 8020000 fffffe821c4bb4e0 sh
2733 1 2 0 8020000 fffffe8223c97a40 sh
2608 1 2 0 8020000 fffffe821b498020 cron
2402 1 2 0 8020000 fffffe822123ba60 ssh
2346 1 2 0 8020000 fffffe821f6f9740 sh
3141 1 2 0 8020000 fffffe8221837240 ssh
1970 1 2 0 8020000 fffffe821f4c70a0 sh
2649 1 2 0 8020000 fffffe821a5f28e0 ssh
2129 1 2 0 8020000 fffffe8227c3cb00 sh
2384 1 2 0 8020000 fffffe8221afe480 ssh
2971 1 2 0 8020000 fffffe82225690c0 sh
2070 1 2 0 8020000 fffffe82212562e0 ssh
2098 1 2 0 8020000 fffffe82201c92a0 sh
2764 1 2 0 8020000 fffffe821bf22680 ssh
3322 1 2 0 8020000 fffffe8221256b20 sh
2511 1 2 0 8020000 fffffe821f37d160 ssh
2661 1 2 0 8020000 fffffe821ecae1c0 sh
3491 1 2 0 8020000 fffffe822220f160 ssh
3195 1 2 0 8020000 fffffe822188a9c0 sh
3305 1 2 0 8020000 fffffe821b800600 ssh
2197 1 2 0 8020000 fffffe821d2d9640 sh
3011 1 2 0 8020000 fffffe82214bf6e0 ssh
3056 1 2 0 8020000 fffffe82288f2ac0 sh
2350 1 2 0 8020000 fffffe8219f93960 ssh
2297 1 2 0 8020000 fffffe8221e571e0 sh
2638 1 2 0 8020000 fffffe821a5f20a0 ssh
2169 1 2 0 8020000 fffffe821c4bb0c0 sh
2550 1 2 0 8020000 fffffe82213176a0 ssh
2225 1 2 0 8020000 fffffe82220aa100 sh
2677 1 2 0 8020000 fffffe821e90c660 ssh
3183 1 2 0 8020000 fffffe8229af4260 sh
2214 1 2 0 8020000 fffffe821f232000 ssh
3259 1 2 0 8020000 fffffe8223563720 sh
2092 1 2 0 8020000 fffffe822220f580 ssh
2331 1 2 0 8020000 fffffe8223563300 sh
2750 1 2 0 8020000 fffffe8226888000 ssh
1921 1 2 0 8020000 fffffe821bf22aa0 sh
2636 1 2 0 8020000 fffffe822d818160 ssh
3223 1 2 0 8020000 fffffe821d443100 sh
2773 1 2 0 8020000 fffffe8226888420 ssh
2150 1 2 0 8020000 fffffe8223049320 sh
2310 1 2 0 8020000 fffffe8222b99560 ssh
1587 1 2 0 8020000 fffffe8220d110e0 sh
2901 1 2 0 8020000 fffffe822188a180 ssh
2627 1 2 0 8020000 fffffe821c4bb900 sh
2060 1 2 0 8020000 fffffe821f36a460 ssh
2599 1 2 0 8020000 fffffe821bf22260 sh
2422 1 2 0 8020000 fffffe821f8965a0 ssh
2164 1 2 0 8020000 fffffe821d443940 sh
2022 1 2 0 8020000 fffffe821f556960 ssh
2000 1 2 0 8020000 fffffe821a542920 sh
1593 1 2 0 8020000 fffffe822d481a40 bash
1914 1 2 0 8020000 fffffe821f5d8ba0 sshd
2167 1 2 0 8020000 fffffe822037f5c0 sshd
1821 1 2 0 8020000 fffffe8220d11500 sshd
1590 1 2 0 8020000 fffffe822dcff120 sshd
1041 1 2 0 8020000 fffffe821c708940 bash
594 1 2 0 8020000 fffffe822037f9e0 bash
895 1 2 0 8020000 fffffe822708eb60 sshd
1533 1 2 0 8020000 fffffe8227069b20 sshd
704 1 3 6 8020000 fffffe821d362980 conserver tstile=
564 1 3 1 8020000 fffffe8229b924c0 conserver tstile=
789 1 2 0 8020000 fffffe821f232840 conserver
788 > 1 7 0 8020000 fffffe8220b8e360 conserver
275 1 3 5 8020000 fffffe8220b8e780 conserver tstile=
772 1 2 0 8020000 fffffe8221256700 conserver
878 1 3 2 8020000 fffffe8221e57600 conserver tstile=
136 1 3 3 8020000 fffffe8222569900 conserver tstile=
773 1 3 1 8020000 fffffe8225709840 conserver tstile=
643 1 2 0 8020000 fffffe82251ce360 conserver
162 1 2 0 8020000 fffffe822dd620e0 getty
861 1 2 0 8020000 fffffe82251ce780 conserver
575 1 2 0 8020000 fffffe82251ceba0 conserver
892 1 2 0 8020000 fffffe8224b0c340 cron
570 1 2 0 8020000 fffffe8224b0cb80 python2.7
107 1 2 0 8020000 fffffe822d7de5c0 inetd
740 1 2 0 8020000 fffffe822cf91a00 nginx
97 1 2 0 8020000 fffffe822d4cd600 nginx
677 1 2 0 8020000 fffffe822d481200 qmgr
479 1 2 0 8020000 fffffe822c312a60 pickup
760 1 2 0 8020000 fffffe822c1e7240 master
436 1 2 0 8020000 fffffe822d818580 sshd
474 1 2 0 8020000 fffffe822d92e5a0 tac_plus
428 1 2 0 8020000 fffffe822cf915e0 powerd
431 1 2 0 8020000 fffffe822d8189a0 ntpd
342 4 2 4 8020000 fffffe822c1e7a80 dhcpd
342 3 2 0 8020000 fffffe822c312220 dhcpd
342 2 2 1 8020000 fffffe822c312640 dhcpd
342 1 2 3 8020000 fffffe822d481620 dhcpd
83 > 1 7 6 8020000 fffffe822cf911c0 syslogd
1 1 2 0 8020000 fffffe810f4fa900 init
0 123 3 7 200 fffffe822d4cd1e0 npfgc-0 xchicv=
0 122 3 7 200 fffffe822d4cda20 ipmi0 ipmi0
0 121 3 0 200 fffffe822dd10940 physiod physio=
d
0 120 3 3 200 fffffe822dcff960 aiodoned aiodon=
ed
0 119 3 5 200 fffffe822dd10100 ioflush biowai=
t
0 118 3 0 200 fffffe822dd10520 pgdaemon pgdaem=
on
0 115 3 0 280 fffffe822dd62500 spkr1 bellcv=
0 114 3 3 280 fffffe822dd62920 audiomix play
0 113 3 3 280 fffffe810ed3b2a0 audiorec record=
0 112 3 0 200 fffffe810f4fa0c0 atapibus0 sccomp=
0 110 3 6 200 fffffe810f2e9b60 usb1 usbevt=
0 109 3 7 200 fffffe810f2c3b20 usb3 usbevt=
0 108 3 6 200 fffffe810f20d2c0 usb5 usbevt=
0 107 3 0 200 fffffe810f2da300 usb0 usbevt=
0 106 3 0 200 fffffe810f2c32e0 usb4 usbevt=
0 105 3 0 200 fffffe810f4fa4e0 usb2 usbevt=
0 104 3 0 200 fffffe810f4a30a0 rt_free rt_fre=
e
0 103 3 0 200 fffffe810f4a34c0 unpgc unpgc
0 102 3 7 200 fffffe810f4a38e0 key_timehandler key_ti=
mehandler
0 101 3 7 200 fffffe810f45c080 icmp6_wqinput/7 icmp6_=
wqinput
0 100 3 6 200 fffffe810f45c4a0 icmp6_wqinput/6 icmp6_=
wqinput
0 99 3 5 200 fffffe810f45c8c0 icmp6_wqinput/5 icmp6_=
wqinput
0 98 3 4 200 fffffe810f4b5060 icmp6_wqinput/4 icmp6_=
wqinput
0 97 3 3 200 fffffe810f4b5480 icmp6_wqinput/3 icmp6_=
wqinput
0 96 3 2 200 fffffe810f4b58a0 icmp6_wqinput/2 icmp6_=
wqinput
0 95 3 1 200 fffffe810f396040 icmp6_wqinput/1 icmp6_=
wqinput
0 94 3 0 200 fffffe810f396460 icmp6_wqinput/0 icmp6_=
wqinput
0 93 3 2 200 fffffe810f396880 nd6_timer nd6_ti=
mer
0 92 3 7 200 fffffe810f367020 icmp_wqinput/7 icmp_w=
qinput
0 91 3 6 200 fffffe810f367440 icmp_wqinput/6 icmp_w=
qinput
0 90 3 5 200 fffffe810f367860 icmp_wqinput/5 icmp_w=
qinput
0 89 3 4 200 fffffe810f378000 icmp_wqinput/4 icmp_w=
qinput
0 88 3 3 200 fffffe810f378420 icmp_wqinput/3 icmp_w=
qinput
0 87 3 2 200 fffffe810f378840 icmp_wqinput/2 icmp_w=
qinput
0 86 3 1 200 fffffe810f329360 icmp_wqinput/1 icmp_w=
qinput
0 85 3 0 200 fffffe810f329780 icmp_wqinput/0 icmp_w=
qinput
0 84 3 0 200 fffffe810f2da720 rt_timer rt_tim=
er
0 83 3 7 200 fffffe810f329ba0 vmem_rehash vmem_r=
ehash
0 82 3 6 200 fffffe810f2f8340 coretemp3 corete=
mp3
0 81 3 7 200 fffffe810f2f8760 coretemp2 corete=
mp2
0 80 3 2 200 fffffe810f2f8b80 coretemp1 corete=
mp1
0 79 3 3 200 fffffe810f2e9320 coretemp0 corete=
mp0
0 78 3 3 200 fffffe810f2e9740 mfi0 mfi0
0 69 3 5 200 fffffe810f20d6e0 atabus1 atath
0 68 3 4 200 fffffe810f20db00 atabus0 atath
0 66 3 0 200 fffffe810ed3b6c0 scsibus0 sccomp=
0 65 3 0 200 fffffe810ed3bae0 usbtask-dr usbtsk=
0 64 3 0 200 fffffe810ecaa280 usbtask-hc usbtsk=
0 63 3 0 200 fffffe810ecaa6a0 bnx3 bnx3
0 62 3 0 200 fffffe810ecaaac0 bnx2 bnx2
0 61 3 0 200 fffffe810eb61260 bnx1 bnx1
0 60 3 7 200 fffffe810eb61680 bnx0 bnx0
0 > 59 7 2 200 fffffe810eb61aa0 ipmi
0 58 3 7 200 fffffe810ea7c240 xcall/7 xcall
0 57 1 7 200 fffffe810ea7c660 softser/7
0 56 1 7 200 fffffe810ea7ca80 softclk/7
0 55 1 7 200 fffffe810ea5d220 softbio/7
0 54 1 7 200 fffffe810ea5d640 softnet/7
0 > 53 7 7 201 fffffe810ea5da60 idle/7
0 52 3 6 200 fffffe810ea36200 xcall/6 xcall
0 51 1 6 200 fffffe810ea36620 softser/6
0 50 1 6 200 fffffe810ea36a40 softclk/6
0 49 1 6 200 fffffe810ea0f1e0 softbio/6
0 48 1 6 200 fffffe810ea0f600 softnet/6
0 47 1 6 201 fffffe810ea0fa20 idle/6
0 46 3 5 200 fffffe810e9e01c0 xcall/5 xcall
0 45 1 5 200 fffffe810e9e05e0 softser/5
0 44 1 5 200 fffffe810e9e0a00 softclk/5
0 43 1 5 200 fffffe810e9d11a0 softbio/5
0 42 1 5 200 fffffe810e9d15c0 softnet/5
0 > 41 7 5 201 fffffe810e9d19e0 idle/5
0 40 3 4 200 fffffe810e9a2180 xcall/4 xcall
0 39 1 4 200 fffffe810e9a25a0 softser/4
0 38 1 4 200 fffffe810e9a29c0 softclk/4
0 37 1 4 200 fffffe810e973160 softbio/4
0 36 1 4 200 fffffe810e973580 softnet/4
0 > 35 7 4 201 fffffe810e9739a0 idle/4
0 34 3 3 200 fffffe810e94c140 xcall/3 xcall
0 33 1 3 200 fffffe810e94c560 softser/3
0 32 1 3 200 fffffe810e94c980 softclk/3
0 31 1 3 200 fffffe810e93d120 softbio/3
0 30 1 3 200 fffffe810e93d540 softnet/3
0 > 29 7 3 201 fffffe810e93d960 idle/3
0 28 3 2 200 fffffe810e906100 xcall/2 xcall
0 27 1 2 200 fffffe810e906520 softser/2
0 26 1 2 200 fffffe810e906940 softclk/2
0 25 1 2 200 fffffe810e8ef0e0 softbio/2
0 24 1 2 200 fffffe810e8ef500 softnet/2
0 23 1 2 201 fffffe810e8ef920 idle/2
0 22 3 1 200 fffffe810e8c50c0 xcall/1 xcall
0 21 1 1 200 fffffe810e8c54e0 softser/1
0 20 1 1 200 fffffe810e8c5900 softclk/1
0 19 1 1 200 fffffe810e8b10a0 softbio/1
0 18 1 1 200 fffffe810e8b14c0 softnet/1
0 > 17 7 1 201 fffffe810e8b18e0 idle/1
0 16 3 0 200 fffffe822de92080 lnxsyswq lnxsys=
wq
0 15 3 0 200 fffffe822de924a0 sysmon smtask=
q
0 14 3 2 200 fffffe822de928c0 pmfsuspend pmfsus=
pend
0 13 3 7 200 fffffe822e2b1060 pmfevent pmfeve=
nt
0 12 3 0 200 fffffe822e2b1480 sopendfree sopend=
fr
0 11 3 2 200 fffffe822e2b18a0 nfssilly nfssil=
ly
0 10 3 2 200 fffffe822f6d0040 cachegc cacheg=
c
0 9 3 3 200 fffffe822f6d0460 vdrain vdrain=
0 8 3 0 200 fffffe822f6d0880 modunload mod_un=
ld
0 7 3 0 200 fffffe822f6eb020 xcall/0 xcall
0 > 6 7 0 200 fffffe822f6eb440 softser/0
0 > 5 7 0 200 fffffe822f6eb860 softclk/0
0 4 1 0 200 fffffe822f707000 softbio/0
0 3 1 0 200 fffffe822f707420 softnet/0
0 2 1 0 201 fffffe822f707840 idle/0
0 1 3 4 200 ffffffff81481b20 swapper uvm
crash> =
Backtraces for all the lwps which have non-zero "WAIT" strings:
crash> trace/a fffffe8220c58540
trace: pid 7052 lid 1 at 0xffff80009ea3d8f0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x460
VOP_LOCK() at VOP_LOCK+0x4d
vn_lock() at vn_lock+0x90
namei_tryemulroot() at namei_tryemulroot+0x193
namei() at namei+0x29
vn_open() at vn_open+0x92
do_open() at do_open+0x112
do_sys_openat() at do_sys_openat+0x68
sys_open() at sys_open+0x24
syscall() at syscall+0x1ed
--- syscall (number 5) ---
70efab63e2ca:
crash>
crash> trace/a fffffe821d362560
trace: pid 8587 lid 1 at 0xffff80009edb8d20
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
fork1() at fork1+0x774
sys___vfork14() at sys___vfork14+0x35
syscall() at syscall+0x1ed
--- syscall (number 282) ---
70efab68204c:
crash>
crash> trace/a fffffe821faa60c0
trace: pid 9187 lid 1 at 0xffff80009ed4e890
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_wait() at xc_wait+0x7c
pserialize_perform() at pserialize_perform+0xdf
fstrans_setstate() at fstrans_setstate+0x42
genfs_suspendctl() at genfs_suspendctl+0x57
VFS_SUSPENDCTL() at VFS_SUSPENDCTL+0x1f
vfs_suspend() at vfs_suspend+0x5b
vrevoke_suspend_next() at vrevoke_suspend_next+0x2a
vrevoke() at vrevoke+0x2b
genfs_revoke() at genfs_revoke+0x13
VOP_REVOKE() at VOP_REVOKE+0x54
pty_grant_slave() at pty_grant_slave+0xa8
ptmioctl() at ptmioctl+0x90
cdev_ioctl() at cdev_ioctl+0x88
VOP_IOCTL() at VOP_IOCTL+0x3b
vn_ioctl() at vn_ioctl+0xa6
sys_ioctl() at sys_ioctl+0x101
syscall() at syscall+0x1ed
--- syscall (number 54) ---
6f9ce08feeaa:
crash>
crash> trace/a fffffe82045da540
trace: pid 3794 lid 1 at 0xffff80009f6c6be0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x74d
VOP_LOCK() at VOP_LOCK+0x4d
vn_lock() at vn_lock+0x90
vn_write() at vn_write+0x9b
dofilewrite() at dofilewrite+0x97
sys_write() at sys_write+0x5f
syscall() at syscall+0x1ed
--- syscall (number 4) ---
73b7cfe3e02a:
crash>
crash> trace/a fffffe8216d35140
trace: pid 2853 lid 2 at 0xffff80009f6eaab0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x460
VOP_POLL() at VOP_POLL+0x37
sel_do_scan() at sel_do_scan+0x324
selcommon.part.2() at selcommon.part.2+0x18b
sys___select50() at sys___select50+0x6f
syscall() at syscall+0x1ed
--- syscall (number 417) ---
7b328823e10a:
crash>
crash> trace/a fffffe82214bf2c0
trace: pid 2853 lid 1 at 0xffff80009ef1ddb0
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
fd_close() at fd_close+0x3ba
sys_close() at sys_close+0x20
syscall() at syscall+0x1ed
--- syscall (number 6) ---
7b328823e4ea:
crash>
crash> trace/a fffffe8225709000
trace: pid 8327 lid 2 at 0xffff80009e90eab0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x74d
VOP_POLL() at VOP_POLL+0x37
sel_do_scan() at sel_do_scan+0x324
selcommon.part.2() at selcommon.part.2+0x18b
sys___select50() at sys___select50+0x6f
syscall() at syscall+0x1ed
--- syscall (number 417) ---
7aeb6723e10a:
crash>
crash> trace/a fffffe820df34480
trace: pid 8327 lid 1 at 0xffff80009f668db0
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
fd_close() at fd_close+0x3ba
sys_close() at sys_close+0x20
syscall() at syscall+0x1ed
--- syscall (number 6) ---
7aeb6723e4ea:
crash>
crash> trace/a fffffe8207fd0360
trace: pid 5687 lid 2 at 0xffff80009f5f9ab0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x74d
VOP_POLL() at VOP_POLL+0x37
sel_do_scan() at sel_do_scan+0x324
selcommon.part.2() at selcommon.part.2+0x18b
sys___select50() at sys___select50+0x6f
syscall() at syscall+0x1ed
--- syscall (number 417) ---
7d665343e10a:
crash>
crash> trace/a fffffe822346f080
trace: pid 5687 lid 1 at 0xffff80009efd4db0
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
fd_close() at fd_close+0x3ba
sys_close() at sys_close+0x20
syscall() at syscall+0x1ed
--- syscall (number 6) ---
7d665343e4ea:
crash>
crash> trace/a fffffe821d987a40
trace: pid 9671 lid 2 at 0xffff80009ee64ab0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x74d
VOP_POLL() at VOP_POLL+0x37
sel_do_scan() at sel_do_scan+0x324
selcommon.part.2() at selcommon.part.2+0x18b
sys___select50() at sys___select50+0x6f
syscall() at syscall+0x1ed
--- syscall (number 417) ---
7ea7fc43e10a:
crash>
crash> trace/a fffffe821e90c240
trace: pid 9671 lid 1 at 0xffff80009ee94db0
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
fd_close() at fd_close+0x3ba
sys_close() at sys_close+0x20
syscall() at syscall+0x1ed
--- syscall (number 6) ---
7ea7fc43e4ea:
crash>
crash> trace/a fffffe821d362980
trace: pid 704 lid 1 at 0xffff80009edaebc0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
rw_enter() at rw_enter+0x49d
genfs_lock() at genfs_lock+0x3c
VOP_LOCK() at VOP_LOCK+0x67
vn_lock() at vn_lock+0x90
vn_write() at vn_write+0x9b
dofilewrite() at dofilewrite+0x97
sys_write() at sys_write+0x5f
syscall() at syscall+0x1ed
--- syscall (number 4) ---
764fb503e02a:
crash>
crash> trace/a fffffe8229b924c0
trace: pid 564 lid 1 at 0xffff80009e9c3ab0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x74d
VOP_POLL() at VOP_POLL+0x37
sel_do_scan() at sel_do_scan+0x324
selcommon.part.2() at selcommon.part.2+0x18b
sys___select50() at sys___select50+0x6f
syscall() at syscall+0x1ed
--- syscall (number 417) ---
764fb503e10a:
crash>
crash> trace/a fffffe8220b8e780
trace: pid 275 lid 1 at 0xffff80009ebf8bc0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
rw_enter() at rw_enter+0x49d
genfs_lock() at genfs_lock+0x3c
VOP_LOCK() at VOP_LOCK+0x67
vn_lock() at vn_lock+0x90
vn_write() at vn_write+0x9b
dofilewrite() at dofilewrite+0x97
sys_write() at sys_write+0x5f
syscall() at syscall+0x1ed
--- syscall (number 4) ---
764fb503e02a:
crash>
crash> trace/a fffffe8221e57600
trace: pid 878 lid 1 at 0xffff80009eae8ab0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x74d
VOP_POLL() at VOP_POLL+0x37
sel_do_scan() at sel_do_scan+0x324
selcommon.part.2() at selcommon.part.2+0x18b
sys___select50() at sys___select50+0x6f
syscall() at syscall+0x1ed
--- syscall (number 417) ---
764fb503e10a:
crash>
crash> trace/a fffffe8222569900
trace: pid 136 lid 1 at 0xffff80009e9cdab0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x74d
VOP_POLL() at VOP_POLL+0x37
sel_do_scan() at sel_do_scan+0x324
selcommon.part.2() at selcommon.part.2+0x18b
sys___select50() at sys___select50+0x6f
syscall() at syscall+0x1ed
--- syscall (number 417) ---
764fb503e10a:
crash>
crash> trace/a fffffe8225709840
trace: pid 773 lid 1 at 0xffff80009e904ab0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x74d
VOP_POLL() at VOP_POLL+0x37
sel_do_scan() at sel_do_scan+0x324
selcommon.part.2() at selcommon.part.2+0x18b
sys___select50() at sys___select50+0x6f
syscall() at syscall+0x1ed
--- syscall (number 417) ---
764fb503e10a:
crash>
crash> trace/a fffffe822d4cd1e0
trace: pid 0 lid 123 at 0xffff80009dee1da0
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_broadcast() at xc_broadcast+0x17a
pserialize_perform() at pserialize_perform+0xd7
npf_conn_gc() at npf_conn_gc+0x1b5
npf_worker() at npf_worker+0x4b
crash>
crash> trace/a fffffe822d4cda20
trace: pid 0 lid 122 at 0xffff80009dec3e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe822dd10940
trace: pid 0 lid 121 at 0xffff80009ddc4e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe822dcff960
trace: pid 0 lid 120 at 0xffff80009dceee60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe822dd10100
trace: pid 0 lid 119 at 0xffff80009dce9bf0
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
biowait() at biowait+0x5d
wapbl_buffered_flush() at wapbl_buffered_flush+0xc6
wapbl_buffered_write() at wapbl_buffered_write+0x69
wapbl_circ_write() at wapbl_circ_write+0xa7
wapbl_flush() at wapbl_flush+0x354
ffs_sync() at ffs_sync+0x221
VFS_SYNC() at VFS_SYNC+0x1c
sched_sync() at sched_sync+0x91
crash>
crash> trace/a fffffe822dd10520
trace: pid 0 lid 118 at 0xffff80009dce4d80
sleepq_block() at sleepq_block+0x97
mtsleep() at mtsleep+0x102
uvm_pageout() at uvm_pageout+0x62e
crash>
crash> trace/a fffffe822dd62500
trace: pid 0 lid 115 at 0xffff80009dc8ee60
sleepq_block() at sleepq_block+0x97
cv_wait_sig() at cv_wait_sig+0x12b
bell_thread() at bell_thread+0x5f
crash>
crash> trace/a fffffe822dd62920
trace: pid 0 lid 114 at 0xffff80009db5fe80
sleepq_block() at sleepq_block+0x97
cv_wait_sig() at cv_wait_sig+0x12b
audio_play_thread() at audio_play_thread+0x55
crash>
crash> trace/a fffffe810ed3b2a0
trace: pid 0 lid 113 at 0xffff80009dc89e90
sleepq_block() at sleepq_block+0x97
cv_wait_sig() at cv_wait_sig+0x12b
audio_rec_thread() at audio_rec_thread+0x3a
crash>
crash> trace/a fffffe810f4fa0c0
trace: pid 0 lid 112 at 0xffff80009dc32e80
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
scsipi_completion_thread() at scsipi_completion_thread+0xf5
crash>
crash> trace/a fffffe810f2e9b60
trace: pid 0 lid 110 at 0xffff80009dc27e80
sleepq_block() at sleepq_block+0x106
cv_timedwait() at cv_timedwait+0x131
usb_event_thread() at usb_event_thread+0xa2
crash>
crash> trace/a fffffe810f2c3b20
trace: pid 0 lid 109 at 0xffff80009dc22e80
sleepq_block() at sleepq_block+0x106
cv_timedwait() at cv_timedwait+0x131
usb_event_thread() at usb_event_thread+0xa2
crash>
crash> trace/a fffffe810f20d2c0
trace: pid 0 lid 108 at 0xffff80009db64e80
sleepq_block() at sleepq_block+0x106
cv_timedwait() at cv_timedwait+0x131
usb_event_thread() at usb_event_thread+0xa2
crash>
crash> trace/a fffffe810f2da300
trace: pid 0 lid 107 at 0xffff80009dc1de80
sleepq_block() at sleepq_block+0x106
cv_timedwait() at cv_timedwait+0x131
usb_event_thread() at usb_event_thread+0xa2
crash>
crash> trace/a fffffe810f2c32e0
trace: pid 0 lid 106 at 0xffff80009dc18e80
sleepq_block() at sleepq_block+0x106
cv_timedwait() at cv_timedwait+0x131
usb_event_thread() at usb_event_thread+0xa2
crash>
crash> trace/a fffffe810f4fa4e0
trace: pid 0 lid 105 at 0xffff80009dc13e80
sleepq_block() at sleepq_block+0x106
cv_timedwait() at cv_timedwait+0x131
usb_event_thread() at usb_event_thread+0xa2
crash>
crash> trace/a fffffe810f4a30a0
trace: pid 0 lid 104 at 0xffff80009dbfce60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f4a34c0
trace: pid 0 lid 103 at 0xffff80009dbf7e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
unp_thread() at unp_thread+0x700
crash>
crash> trace/a fffffe810f4a38e0
trace: pid 0 lid 102 at 0xffff80009dbf2e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f45c080
trace: pid 0 lid 101 at 0xffff80009dbede60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f45c4a0
trace: pid 0 lid 100 at 0xffff80009dbe8e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f45c8c0
trace: pid 0 lid 99 at 0xffff80009dbe3e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f4b5060
trace: pid 0 lid 98 at 0xffff80009dbdee60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f4b5480
trace: pid 0 lid 97 at 0xffff80009dbd9e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f4b58a0
trace: pid 0 lid 96 at 0xffff80009dbd4e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f396040
trace: pid 0 lid 95 at 0xffff80009dbcfe60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f396460
trace: pid 0 lid 94 at 0xffff80009dbcae60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f396880
trace: pid 0 lid 93 at 0xffff80009dbc5e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f367020
trace: pid 0 lid 92 at 0xffff80009dbc0e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f367440
trace: pid 0 lid 91 at 0xffff80009dbbbe60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f367860
trace: pid 0 lid 90 at 0xffff80009dbb6e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash> =
crash> trace/a fffffe810f378000
trace: pid 0 lid 89 at 0xffff80009dbb1e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f378420
trace: pid 0 lid 88 at 0xffff80009dbace60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f378840
trace: pid 0 lid 87 at 0xffff80009dba7e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f329360
trace: pid 0 lid 86 at 0xffff80009dba2e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f329780
trace: pid 0 lid 85 at 0xffff80009db6ee60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f2da720
trace: pid 0 lid 84 at 0xffff80009db9de60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f329ba0
trace: pid 0 lid 83 at 0xffff80009db98e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f2f8340
trace: pid 0 lid 82 at 0xffff80009db91e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f2f8760
trace: pid 0 lid 81 at 0xffff80009db8ce60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f2f8b80
trace: pid 0 lid 80 at 0xffff80009db87e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f2e9320
trace: pid 0 lid 79 at 0xffff80009db82e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f2e9740
trace: pid 0 lid 78 at 0xffff80009db7de60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f20d6e0
trace: pid 0 lid 69 at 0xffff80009d349eb0
sleepq_block() at sleepq_block+0x97
atabus_thread() at atabus_thread+0xcc
crash>
crash> trace/a fffffe810f20db00
trace: pid 0 lid 68 at 0xffff80009d344eb0
sleepq_block() at sleepq_block+0x97
atabus_thread() at atabus_thread+0xcc
crash>
crash> trace/a fffffe810ed3b6c0
trace: pid 0 lid 66 at 0xffff80009d0ffe80
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
scsipi_completion_thread() at scsipi_completion_thread+0xf5
crash>
crash> trace/a fffffe810ed3bae0
trace: pid 0 lid 65 at 0xffff800099072e70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
usb_task_thread() at usb_task_thread+0x8a
crash>
crash> trace/a fffffe810ecaa280
trace: pid 0 lid 64 at 0xffff80009906de70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
usb_task_thread() at usb_task_thread+0x8a
crash>
crash> trace/a fffffe810ecaa6a0
trace: pid 0 lid 63 at 0xffff800099056e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810ecaaac0
trace: pid 0 lid 62 at 0xffff800096a41e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810eb61260
trace: pid 0 lid 61 at 0xffff80009442ae60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810eb61680
trace: pid 0 lid 60 at 0xffff800091e15e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810eb61aa0
trace: pid 0 lid 59 at 0xffff80008f7f8e30
sleepq_block() at sleepq_block+0x1c5
cv_timedwait() at cv_timedwait+0x131
ipmi_thread() at ipmi_thread+0x2f4
crash>
crash> trace/a fffffe810ea7c240
trace: pid 0 lid 58 at 0xffff80008f7ede70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_thread() at xc_thread+0x5a
crash>
crash> trace/a fffffe810ea36200
trace: pid 0 lid 52 at 0xffff80008f7c7e70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_thread() at xc_thread+0x5a
crash>
crash> trace/a fffffe810e9e01c0
trace: pid 0 lid 46 at 0xffff80008f7a1e70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_thread() at xc_thread+0x5a
crash>
crash> trace/a fffffe810e9a2180
trace: pid 0 lid 40 at 0xffff80008f77be70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_thread() at xc_thread+0x5a
crash>
crash> trace/a fffffe810e94c140
trace: pid 0 lid 34 at 0xffff80008f755e70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_thread() at xc_thread+0x5a
crash>
crash> trace/a fffffe810e906100
trace: pid 0 lid 28 at 0xffff80008f72fe70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_thread() at xc_thread+0x5a
crash>
crash> trace/a fffffe810e8c50c0
trace: pid 0 lid 22 at 0xffff80008f706e70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_thread() at xc_thread+0x5a
crash>
crash> trace/a fffffe822de92080
trace: pid 0 lid 16 at 0xffff80008f6cde60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe822de924a0
trace: pid 0 lid 15 at 0xffff80008f6c8e90
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
sysmon_task_queue_thread() at sysmon_task_queue_thread+0x81
crash>
crash> trace/a fffffe822de928c0
trace: pid 0 lid 14 at 0xffff80008f6c3e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe822e2b1060
trace: pid 0 lid 13 at 0xffff80008f6bee60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe822e2b1480
trace: pid 0 lid 12 at 0xffff80008f6b9e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
sopendfree_thread() at sopendfree_thread+0x1da
crash>
crash> trace/a fffffe822e2b18a0
trace: pid 0 lid 11 at 0xffff80008f6b4e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe822f6d0040
trace: pid 0 lid 10 at 0xffff80008f6afe80
sleepq_block() at sleepq_block+0x106
kpause() at kpause+0xea
cache_thread() at cache_thread+0x30
crash>
crash> trace/a fffffe822f6d0460
trace: pid 0 lid 9 at 0xffff80008f6aae30
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
vdrain_thread() at vdrain_thread+0x3f1
crash>
crash> trace/a fffffe822f6d0880
trace: pid 0 lid 8 at 0xffff80008f6a5e80
sleepq_block() at sleepq_block+0x97
cv_timedwait() at cv_timedwait+0x131
module_thread() at module_thread+0xe9
crash>
crash> trace/a fffffe822f6eb020
trace: pid 0 lid 7 at 0xffff80008f6a0e70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_thread() at xc_thread+0x5a
crash>
crash> trace/a ffffffff81481b20
trace: pid 0 lid 1 at 0xffffffff817dde20
sleepq_block() at sleepq_block+0x106
kpause() at kpause+0xea
uvm_scheduler() at uvm_scheduler+0x69
sysctl_alloc() at sysctl_alloc
crash>
------------------------------
(gdb) kvm proc 0xfffffe8220c58540
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8220c58540)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8220c58540)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =
ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =
obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =
sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fec40 in fstrans_alloc_lwp_info (mp=3D0xfffffe822dd7a0=
08)
at /usr/src/sys/kern/vfs_trans.c:274
#5 fstrans_get_lwp_info (do_alloc=3Dtrue, mp=3D0xfffffe822dd7a008)
at /usr/src/sys/kern/vfs_trans.c:331
#6 _fstrans_start (wait=3D1, lock_type=3DFSTRANS_SHARED, mp=3D<optimiz=
ed out>)
at /usr/src/sys/kern/vfs_trans.c:375
#7 fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
.c:414
#8 0xffffffff80a0ed1b in vop_pre (vp=3D0xfffffe810f2f2930, =
vp=3D0xfffffe810f2f2930, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
, =
mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#9 VOP_LOCK (vp=3Dvp@entry=3D0xfffffe810f2f2930, flags=3Dflags@entry=3D=
131074)
at /usr/src/sys/kern/vnode_if.c:1281
#10 0xffffffff80a05846 in vn_lock (vp=3Dvp@entry=3D0xfffffe810f2f2930, =
flags=3Dflags@entry=3D131074) at /usr/src/sys/kern/vfs_vnops.c:1036=
#11 0xffffffff809ef7c3 in namei_start (startdir_ret=3D0xffff80009ea3db2=
8, =
isnfsd=3D0, state=3D0xffff80009ea3dbd0) at /usr/src/sys/kern/vfs_lo=
okup.c:681
#12 namei_oneroot (isnfsd=3D0, inhibitmagic=3D0, neverfollow=3D0, =
state=3D0xffff80009ea3dbd0) at /usr/src/sys/kern/vfs_lookup.c:1156
#13 namei_tryemulroot (state=3Dstate@entry=3D0xffff80009ea3dbd0, =
neverfollow=3Dneverfollow@entry=3D0, inhibitmagic=3Dinhibitmagic@en=
try=3D0, =
isnfsd=3Disnfsd@entry=3D0) at /usr/src/sys/kern/vfs_lookup.c:1510
#14 0xffffffff809f1905 in namei (ndp=3Dndp@entry=3D0xffff80009ea3dd58)
at /usr/src/sys/kern/vfs_lookup.c:1546
#15 0xffffffff80a061fd in vn_open (ndp=3Dndp@entry=3D0xffff80009ea3dd58=
, =
fmode=3Dfmode@entry=3D4194305, cmode=3Dcmode@entry=3D3116)
at /usr/src/sys/kern/vfs_vnops.c:175
#16 0xffffffff809fb7c4 in do_open (l=3Dl@entry=3D0xfffffe8220c58540, dv=
p=3D0x0, =
pb=3D<optimized out>, open_flags=3Dopen_flags@entry=3D4194304, =
open_mode=3Dopen_mode@entry=3D774778414, fd=3Dfd@entry=3D0xffff8000=
9ea3de7c)
at /usr/src/sys/kern/vfs_syscalls.c:1576
#17 0xffffffff809fb915 in do_sys_openat (l=3D0xfffffe8220c58540, =
fdat=3Dfdat@entry=3D-100, path=3D<optimized out>, flags=3D4194304, =
mode=3D774778414, =
fd=3Dfd@entry=3D0xffff80009ea3de7c) at /usr/src/sys/kern/vfs_syscal=
ls.c:1656
#18 0xffffffff809fb9d3 in sys_open (l=3D<optimized out>, uap=3D<optimiz=
ed out>, =
retval=3D0xffff80009ea3deb0) at /usr/src/sys/kern/vfs_syscalls.c:16=
76
#19 0xffffffff80248dbd in sy_call (rval=3D0xffff80009ea3deb0, =
uap=3D0xffff80009ea3df00, l=3D0xfffffe8220c58540, =
sy=3D0xffffffff8147da58 <sysent+120>) at /usr/src/sys/sys/syscallva=
r.h:65
#20 sy_invoke (code=3D5, rval=3D0xffff80009ea3deb0, uap=3D0xffff80009ea=
3df00, =
l=3D0xfffffe8220c58540, sy=3D0xffffffff8147da58 <sysent+120>)
at /usr/src/sys/sys/syscallvar.h:94
#21 syscall (frame=3D0xffff80009ea3df00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#22 0xffffffff8020768d in handle_syscall ()
#23 0x000070efab549000 in ?? ()
#24 0x0000000000400000 in ?? ()
#25 0x2e2e2e2e2e2e2e2e in ?? ()
#26 0x0000000000000000 in ?? ()
(gdb)
Looking at fstrans_lock:
(gdb) lock 0xffffffff815cb9b0
lock address : 0xffffffff815cb9b0 type : sleep/adaptive
initialized : 0xffffffff809fe6a3
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 10
cpu last held: 6
current lwp : 000000000000000000 last held: 0xfffffe821faa60c0
last locked : 0xffffffff809ffb6d unlocked : 0xffffffff809fecc4
(gdb) =
What I don't understand is that "current lwp" is 0x0, even though
this one seems to be held at the moment, and there's lots of
contention for the lock judging by the "shares wanted, exclusive: 10".
"last held" appears to point to
9187 1 3 6 8020000 fffffe821faa60c0 expect xchicv=
(gdb) kvm proc 0xfffffe821faa60c0
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821faa60c0)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821faa60c0)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8094afca in cv_wait (
cv=3Dcv@entry=3D0xffffffff814c5c88 <xc_high_pri+8>, =
mtx=3Dmtx@entry=3D0xffffffff814c5c80 <xc_high_pri>)
at /usr/src/sys/kern/kern_condvar.c:217
#3 0xffffffff809b1293 in xc_wait (where=3D64992)
at /usr/src/sys/kern/subr_xcall.c:321
#4 0xffffffff809aa122 in pserialize_perform (psz=3D0xfffffe822fbd20e8)=
at /usr/src/sys/kern/subr_pserialize.c:187
#5 0xffffffff809ffb7c in fstrans_setstate (mp=3D0xfffffe822c993008, =
new_state=3Dnew_state@entry=3DFSTRANS_SUSPENDED)
at /usr/src/sys/kern/vfs_trans.c:530
#6 0xffffffff80a17729 in genfs_suspendctl (mp=3D0xfffffe822c993008, =
cmd=3D<optimized out>) at /usr/src/sys/miscfs/genfs/genfs_vfsops.c:=
86
#7 0xffffffff809f781e in VFS_SUSPENDCTL (mp=3Dmp@entry=3D0xfffffe822c9=
93008, =
a=3Da@entry=3D1) at /usr/src/sys/kern/vfs_subr.c:1478
#8 0xffffffff80a00530 in vfs_suspend (mp=3D<optimized out>, =
mp@entry=3D0xfffffe822c993008, nowait=3Dnowait@entry=3D0)
at /usr/src/sys/kern/vfs_trans.c:587
#9 0xffffffff80a01769 in vrevoke_suspend_next (lastmp=3D<optimized out=
>, =
thismp=3D0xfffffe822c993008) at /usr/src/sys/kern/vfs_vnode.c:978
#10 0xffffffff80a030ed in vrevoke (vp=3D0xfffffe8221e36690)
at /usr/src/sys/kern/vfs_vnode.c:1002
#11 0xffffffff80a17caa in genfs_revoke (v=3D<optimized out>)
at /usr/src/sys/miscfs/genfs/genfs_vnops.c:273
#12 0xffffffff80a0e4f7 in VOP_REVOKE (vp=3D0xfffffe8221e36690, =
flags=3Dflags@entry=3D1) at /usr/src/sys/kern/vnode_if.c:744
#13 0xffffffff809cfb00 in pty_grant_slave (l=3Dl@entry=3D0xfffffe821faa=
60c0, =
dev=3D<optimized out>, mp=3D<optimized out>) at /usr/src/sys/kern/t=
ty_ptm.c:245
#14 0xffffffff809cfd7b in ptmioctl (dev=3D<optimized out>, cmd=3D<optim=
ized out>, =
data=3D0xfffffe822d624008, flag=3D<optimized out>, l=3D0xfffffe821f=
aa60c0)
at /usr/src/sys/kern/tty_ptm.c:397
#15 0xffffffff80996a52 in cdev_ioctl (dev=3D42241, cmd=3D1208513606, =
data=3D0xfffffe822d624008, flag=3D3, l=3D0xfffffe821faa60c0)
at /usr/src/sys/kern/subr_devsw.c:938
#16 0xffffffff80a0e2b8 in VOP_IOCTL (vp=3Dvp@entry=3D0xfffffe822caabd20=
, =
command=3Dcommand@entry=3D1208513606, data=3Ddata@entry=3D0xfffffe8=
22d624008, =
fflag=3D<optimized out>, cred=3D<optimized out>)
at /usr/src/sys/kern/vnode_if.c:610
#17 0xffffffff80a05e1e in vn_ioctl (fp=3D0xfffffe821d011a40, com=3D1208=
513606, =
data=3D0xfffffe822d624008) at /usr/src/sys/kern/vfs_vnops.c:765
#18 0xffffffff809b4e81 in sys_ioctl (l=3D<optimized out>, =
uap=3D0xffff80009ed4ef00, retval=3D<optimized out>)
at /usr/src/sys/kern/sys_generic.c:671
#19 0xffffffff80248dbd in sy_call (rval=3D0xffff80009ed4eeb0, =
uap=3D0xffff80009ed4ef00, l=3D0xfffffe821faa60c0, =
sy=3D0xffffffff8147def0 <sysent+1296>) at /usr/src/sys/sys/syscallv=
ar.h:65
#20 sy_invoke (code=3D54, rval=3D0xffff80009ed4eeb0, uap=3D0xffff80009e=
d4ef00, =
l=3D0xfffffe821faa60c0, sy=3D0xffffffff8147def0 <sysent+1296>)
at /usr/src/sys/sys/syscallvar.h:94
#21 syscall (frame=3D0xffff80009ed4ef00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#22 0xffffffff8020768d in handle_syscall ()
#23 0x0000000000000003 in ?? ()
#24 0x0000000048087446 in ?? ()
#25 0x00007f7fff318360 in ?? ()
#26 0x0000000000000000 in ?? ()
(gdb)
The xc_high_pri looks like this:
(gdb) lock 0xffffffff814c5c80
lock address : 0xffffffff814c5c80 type : sleep/adaptive
initialized : 0xffffffff809b1053
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 0
cpu last held: 7
current lwp : 000000000000000000 last held: 000000000000000000
last locked : 0xffffffff809b17d1 unlocked : 0xffffffff8094afb7
(gdb) =
but that doesn't appear to lead anywhere.
------------------------------
Decode of the stack backtraces of the non-kernel lwps using gdb:
(gdb) kvm proc 0xfffffe8220c58540
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8220c58540)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8220c58540)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =
ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =
obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =
sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fec40 in fstrans_alloc_lwp_info (mp=3D0xfffffe822dd7a0=
08)
at /usr/src/sys/kern/vfs_trans.c:274
#5 fstrans_get_lwp_info (do_alloc=3Dtrue, mp=3D0xfffffe822dd7a008)
at /usr/src/sys/kern/vfs_trans.c:331
#6 _fstrans_start (wait=3D1, lock_type=3DFSTRANS_SHARED, mp=3D<optimiz=
ed out>)
at /usr/src/sys/kern/vfs_trans.c:375
#7 fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
.c:414
#8 0xffffffff80a0ed1b in vop_pre (vp=3D0xfffffe810f2f2930, =
vp=3D0xfffffe810f2f2930, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
, =
mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#9 VOP_LOCK (vp=3Dvp@entry=3D0xfffffe810f2f2930, flags=3Dflags@entry=3D=
131074)
at /usr/src/sys/kern/vnode_if.c:1281
#10 0xffffffff80a05846 in vn_lock (vp=3Dvp@entry=3D0xfffffe810f2f2930, =
flags=3Dflags@entry=3D131074) at /usr/src/sys/kern/vfs_vnops.c:1036=
#11 0xffffffff809ef7c3 in namei_start (startdir_ret=3D0xffff80009ea3db2=
8, =
isnfsd=3D0, state=3D0xffff80009ea3dbd0) at /usr/src/sys/kern/vfs_lo=
okup.c:681
#12 namei_oneroot (isnfsd=3D0, inhibitmagic=3D0, neverfollow=3D0, =
state=3D0xffff80009ea3dbd0) at /usr/src/sys/kern/vfs_lookup.c:1156
#13 namei_tryemulroot (state=3Dstate@entry=3D0xffff80009ea3dbd0, =
neverfollow=3Dneverfollow@entry=3D0, inhibitmagic=3Dinhibitmagic@en=
try=3D0, =
isnfsd=3Disnfsd@entry=3D0) at /usr/src/sys/kern/vfs_lookup.c:1510
#14 0xffffffff809f1905 in namei (ndp=3Dndp@entry=3D0xffff80009ea3dd58)
at /usr/src/sys/kern/vfs_lookup.c:1546
#15 0xffffffff80a061fd in vn_open (ndp=3Dndp@entry=3D0xffff80009ea3dd58=
, =
fmode=3Dfmode@entry=3D4194305, cmode=3Dcmode@entry=3D3116)
at /usr/src/sys/kern/vfs_vnops.c:175
#16 0xffffffff809fb7c4 in do_open (l=3Dl@entry=3D0xfffffe8220c58540, dv=
p=3D0x0, =
pb=3D<optimized out>, open_flags=3Dopen_flags@entry=3D4194304, =
open_mode=3Dopen_mode@entry=3D774778414, fd=3Dfd@entry=3D0xffff8000=
9ea3de7c)
at /usr/src/sys/kern/vfs_syscalls.c:1576
#17 0xffffffff809fb915 in do_sys_openat (l=3D0xfffffe8220c58540, =
fdat=3Dfdat@entry=3D-100, path=3D<optimized out>, flags=3D4194304, =
mode=3D774778414, =
fd=3Dfd@entry=3D0xffff80009ea3de7c) at /usr/src/sys/kern/vfs_syscal=
ls.c:1656
#18 0xffffffff809fb9d3 in sys_open (l=3D<optimized out>, uap=3D<optimiz=
ed out>, =
retval=3D0xffff80009ea3deb0) at /usr/src/sys/kern/vfs_syscalls.c:16=
76
#19 0xffffffff80248dbd in sy_call (rval=3D0xffff80009ea3deb0, =
uap=3D0xffff80009ea3df00, l=3D0xfffffe8220c58540, =
sy=3D0xffffffff8147da58 <sysent+120>) at /usr/src/sys/sys/syscallva=
r.h:65
#20 sy_invoke (code=3D5, rval=3D0xffff80009ea3deb0, uap=3D0xffff80009ea=
3df00, =
l=3D0xfffffe8220c58540, sy=3D0xffffffff8147da58 <sysent+120>)
at /usr/src/sys/sys/syscallvar.h:94
#21 syscall (frame=3D0xffff80009ea3df00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#22 0xffffffff8020768d in handle_syscall ()
#23 0x000070efab549000 in ?? ()
#24 0x0000000000400000 in ?? ()
#25 0x2e2e2e2e2e2e2e2e in ?? ()
#26 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe821d362560
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821d362560)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821d362560)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8094afca in cv_wait (cv=3Dcv@entry=3D0xfffffe8210396788, =
mtx=3D0xfffffe822fb97f00) at /usr/src/sys/kern/kern_condvar.c:217
#3 0xffffffff80959f24 in fork1 (l1=3D0xfffffe821d362560, flags=3Dflags=
@entry=3D3, =
exitsig=3Dexitsig@entry=3D20, stack=3Dstack@entry=3D0x0, =
stacksize=3Dstacksize@entry=3D0, func=3Dfunc@entry=3D0x0, arg=3Darg=
@entry=3D0x0, =
retval=3D0xffff80009edb8eb0, rnewprocp=3Drnewprocp@entry=3D0x0)
at /usr/src/sys/kern/kern_fork.c:592
#4 0xffffffff8095a424 in sys___vfork14 (l=3D<optimized out>, v=3D<opti=
mized out>, =
retval=3D<optimized out>) at /usr/src/sys/kern/kern_fork.c:143
#5 0xffffffff80248dbd in sy_call (rval=3D0xffff80009edb8eb0, =
uap=3D0xffff80009edb8f00, l=3D0xfffffe821d362560, =
sy=3D0xffffffff8147f450 <sysent+6768>) at /usr/src/sys/sys/syscallv=
ar.h:65
#6 sy_invoke (code=3D282, rval=3D0xffff80009edb8eb0, uap=3D0xffff80009=
edb8f00, =
l=3D0xfffffe821d362560, sy=3D0xffffffff8147f450 <sysent+6768>)
at /usr/src/sys/sys/syscallvar.h:94
#7 syscall (frame=3D0xffff80009edb8f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#8 0xffffffff8020768d in handle_syscall ()
#9 0x00007f7fff9fa3d8 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe821faa60c0
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821faa60c0)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821faa60c0)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8094afca in cv_wait (
cv=3Dcv@entry=3D0xffffffff814c5c88 <xc_high_pri+8>, =
mtx=3Dmtx@entry=3D0xffffffff814c5c80 <xc_high_pri>)
at /usr/src/sys/kern/kern_condvar.c:217
#3 0xffffffff809b1293 in xc_wait (where=3D64992)
at /usr/src/sys/kern/subr_xcall.c:321
#4 0xffffffff809aa122 in pserialize_perform (psz=3D0xfffffe822fbd20e8)=
at /usr/src/sys/kern/subr_pserialize.c:187
#5 0xffffffff809ffb7c in fstrans_setstate (mp=3D0xfffffe822c993008, =
new_state=3Dnew_state@entry=3DFSTRANS_SUSPENDED)
at /usr/src/sys/kern/vfs_trans.c:530
#6 0xffffffff80a17729 in genfs_suspendctl (mp=3D0xfffffe822c993008, =
cmd=3D<optimized out>) at /usr/src/sys/miscfs/genfs/genfs_vfsops.c:=
86
#7 0xffffffff809f781e in VFS_SUSPENDCTL (mp=3Dmp@entry=3D0xfffffe822c9=
93008, =
a=3Da@entry=3D1) at /usr/src/sys/kern/vfs_subr.c:1478
#8 0xffffffff80a00530 in vfs_suspend (mp=3D<optimized out>, =
mp@entry=3D0xfffffe822c993008, nowait=3Dnowait@entry=3D0)
at /usr/src/sys/kern/vfs_trans.c:587
#9 0xffffffff80a01769 in vrevoke_suspend_next (lastmp=3D<optimized out=
>, =
thismp=3D0xfffffe822c993008) at /usr/src/sys/kern/vfs_vnode.c:978
#10 0xffffffff80a030ed in vrevoke (vp=3D0xfffffe8221e36690)
at /usr/src/sys/kern/vfs_vnode.c:1002
#11 0xffffffff80a17caa in genfs_revoke (v=3D<optimized out>)
at /usr/src/sys/miscfs/genfs/genfs_vnops.c:273
#12 0xffffffff80a0e4f7 in VOP_REVOKE (vp=3D0xfffffe8221e36690, =
flags=3Dflags@entry=3D1) at /usr/src/sys/kern/vnode_if.c:744
#13 0xffffffff809cfb00 in pty_grant_slave (l=3Dl@entry=3D0xfffffe821faa=
60c0, =
dev=3D<optimized out>, mp=3D<optimized out>) at /usr/src/sys/kern/t=
ty_ptm.c:245
#14 0xffffffff809cfd7b in ptmioctl (dev=3D<optimized out>, cmd=3D<optim=
ized out>, =
data=3D0xfffffe822d624008, flag=3D<optimized out>, l=3D0xfffffe821f=
aa60c0)
at /usr/src/sys/kern/tty_ptm.c:397
#15 0xffffffff80996a52 in cdev_ioctl (dev=3D42241, cmd=3D1208513606, =
data=3D0xfffffe822d624008, flag=3D3, l=3D0xfffffe821faa60c0)
at /usr/src/sys/kern/subr_devsw.c:938
#16 0xffffffff80a0e2b8 in VOP_IOCTL (vp=3Dvp@entry=3D0xfffffe822caabd20=
, =
command=3Dcommand@entry=3D1208513606, data=3Ddata@entry=3D0xfffffe8=
22d624008, =
fflag=3D<optimized out>, cred=3D<optimized out>)
at /usr/src/sys/kern/vnode_if.c:610
#17 0xffffffff80a05e1e in vn_ioctl (fp=3D0xfffffe821d011a40, com=3D1208=
513606, =
data=3D0xfffffe822d624008) at /usr/src/sys/kern/vfs_vnops.c:765
#18 0xffffffff809b4e81 in sys_ioctl (l=3D<optimized out>, =
uap=3D0xffff80009ed4ef00, retval=3D<optimized out>)
at /usr/src/sys/kern/sys_generic.c:671
#19 0xffffffff80248dbd in sy_call (rval=3D0xffff80009ed4eeb0, =
uap=3D0xffff80009ed4ef00, l=3D0xfffffe821faa60c0, =
sy=3D0xffffffff8147def0 <sysent+1296>) at /usr/src/sys/sys/syscallv=
ar.h:65
#20 sy_invoke (code=3D54, rval=3D0xffff80009ed4eeb0, uap=3D0xffff80009e=
d4ef00, =
l=3D0xfffffe821faa60c0, sy=3D0xffffffff8147def0 <sysent+1296>)
at /usr/src/sys/sys/syscallvar.h:94
#21 syscall (frame=3D0xffff80009ed4ef00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#22 0xffffffff8020768d in handle_syscall ()
#23 0x0000000000000003 in ?? ()
#24 0x0000000048087446 in ?? ()
#25 0x00007f7fff318360 in ?? ()
#26 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe82045da540
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe82045da540)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe82045da540)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =
ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =
obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =
sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fef2d in _fstrans_start (wait=3D1, lock_type=3DFSTRANS=
_SHARED, =
mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
#5 fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
.c:414
#6 0xffffffff80a0ed1b in vop_pre (vp=3D0xfffffe821a9bc7e0, =
vp=3D0xfffffe821a9bc7e0, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
, =
mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#7 VOP_LOCK (vp=3Dvp@entry=3D0xfffffe821a9bc7e0, flags=3Dflags@entry=3D=
131074)
at /usr/src/sys/kern/vnode_if.c:1281
#8 0xffffffff80a05846 in vn_lock (vp=3Dvp@entry=3D0xfffffe821a9bc7e0, =
flags=3Dflags@entry=3D131074) at /usr/src/sys/kern/vfs_vnops.c:1036=
#9 0xffffffff80a0601d in vn_write (fp=3D<optimized out>, =
offset=3D0xfffffe821ed94300, uio=3D0xffff80009f6c6df0, =
cred=3D0xfffffe8220120e40, flags=3D1) at /usr/src/sys/kern/vfs_vnop=
s.c:592
#10 0xffffffff809b4c1d in dofilewrite (fd=3Dfd@entry=3D2, fp=3D0xfffffe=
821ed94300, =
buf=3D0x7f7fff0d1230, nbyte=3D102, offset=3D<optimized out>, =
flags=3Dflags@entry=3D1, retval=3Dretval@entry=3D0xffff80009f6c6eb0=
)
at /usr/src/sys/kern/sys_generic.c:350
#11 0xffffffff809b4d22 in sys_write (l=3D<optimized out>, =
uap=3D0xffff80009f6c6f00, retval=3D0xffff80009f6c6eb0)
at /usr/src/sys/kern/sys_generic.c:318
#12 0xffffffff80248dbd in sy_call (rval=3D0xffff80009f6c6eb0, =
uap=3D0xffff80009f6c6f00, l=3D0xfffffe82045da540, =
sy=3D0xffffffff8147da40 <sysent+96>) at /usr/src/sys/sys/syscallvar=
.h:65
#13 sy_invoke (code=3D4, rval=3D0xffff80009f6c6eb0, uap=3D0xffff80009f6=
c6f00, =
l=3D0xfffffe82045da540, sy=3D0xffffffff8147da40 <sysent+96>)
at /usr/src/sys/sys/syscallvar.h:94
#14 syscall (frame=3D0xffff80009f6c6f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#15 0xffffffff8020768d in handle_syscall ()
#16 0x0000000000000002 in ?? ()
#17 0x00007f7fff0d1230 in ?? ()
#18 0x0000000000000066 in ?? ()
#19 0x00007f7fff0d129a in ?? ()
#20 0x0101010101010101 in ?? ()
#21 0x00007f7fff0d128c in ?? ()
#22 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe8216d35140
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8216d35140)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8216d35140)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, ts@ent=
ry=3D0x0, =
q=3Dq@entry=3D1, obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock=
>, =
sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fec40 in fstrans_alloc_lwp_info (mp=3D0xfffffe822c9930=
08)
at /usr/src/sys/kern/vfs_trans.c:274
#5 fstrans_get_lwp_info (do_alloc=3Dtrue, mp=3D0xfffffe822c993008)
at /usr/src/sys/kern/vfs_trans.c:331
#6 _fstrans_start (wait=3D1, lock_type=3DFSTRANS_SHARED, mp=3D<optimiz=
ed out>)
at /usr/src/sys/kern/vfs_trans.c:375
#7 fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
.c:414
#8 0xffffffff80a0e399 in vop_pre (vp=3D0xfffffe82226253f0, =
vp=3D0xfffffe82226253f0, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
, =
mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#9 VOP_POLL (vp=3D0xfffffe82226253f0, events=3D<optimized out>)
at /usr/src/sys/kern/vnode_if.c:677
#10 0xffffffff809bd7e8 in selscan (retval=3D0xffff80009f6eaeb0, ni=3D0,=
nfd=3D5, =
bits=3D0xffff80009f6ead48 "\030") at /usr/src/sys/kern/sys_select.c=
:402
#11 sel_do_scan (op=3Dop@entry=3D1, fds=3Dfds@entry=3D0xffff80009f6ead4=
8, =
nf=3Dnf@entry=3D5, ni=3Dni@entry=3D4, ts=3Dts@entry=3D0x0, mask=3Dm=
ask@entry=3D0x0, =
retval=3Dretval@entry=3D0xffff80009f6eaeb0)
at /usr/src/sys/kern/sys_select.c:264
#12 0xffffffff809bdb96 in selcommon (retval=3Dretval@entry=3D0xffff8000=
9f6eaeb0, =
nd=3D5, u_in=3D0x7b32875fff40, u_ou=3D0x7b32875fff60, u_ex=3D0x7b32=
875fff80, =
ts=3D0x0, mask=3Dmask@entry=3D0x0) at /usr/src/sys/kern/sys_select.=
c:362
#13 0xffffffff809bdeab in selcommon (mask=3D0x0, ts=3D<optimized out>, =
u_ex=3D<optimized out>, u_ou=3D<optimized out>, u_in=3D<optimized o=
ut>, =
nd=3D<optimized out>, retval=3D0xffff80009f6eaeb0)
at /usr/src/sys/kern/sys_select.c:337
#14 sys___select50 (l=3D<optimized out>, uap=3D0xffff80009f6eaf00, =
retval=3D0xffff80009f6eaeb0) at /usr/src/sys/kern/sys_select.c:212
#15 0xffffffff80248dbd in sy_call (rval=3D0xffff80009f6eaeb0, =
uap=3D0xffff80009f6eaf00, l=3D0xfffffe8216d35140, =
sy=3D0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscall=
var.h:65
#16 sy_invoke (code=3D417, rval=3D0xffff80009f6eaeb0, uap=3D0xffff80009=
f6eaf00, =
l=3D0xfffffe8216d35140, sy=3D0xffffffff814800f8 <sysent+10008>)
at /usr/src/sys/sys/syscallvar.h:94
#17 syscall (frame=3D0xffff80009f6eaf00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#18 0xffffffff8020768d in handle_syscall ()
#19 0x0000000000000005 in ?? ()
#20 0x00007b32875fff40 in ?? ()
#21 0x00007b32875fff60 in ?? ()
#22 0x00007b32875fff80 in ?? ()
#23 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe82214bf2c0
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe82214bf2c0)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe82214bf2c0)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8094afca in cv_wait (cv=3Dcv@entry=3D0xfffffe8221620998, =
mtx=3Dmtx@entry=3D0xfffffe8221620a00) at /usr/src/sys/kern/kern_con=
dvar.c:217
#3 0xffffffff8094e4e5 in fd_close (fd=3D<optimized out>)
at /usr/src/sys/kern/kern_descrip.c:684
#4 0xffffffff809b3c88 in sys_close (l=3D0xfffffe82214bf2c0, =
uap=3D0xffff80009ef1df00, retval=3D<optimized out>)
at /usr/src/sys/kern/sys_descrip.c:488
#5 0xffffffff80248dbd in sy_call (rval=3D0xffff80009ef1deb0, =
uap=3D0xffff80009ef1df00, l=3D0xfffffe82214bf2c0, =
sy=3D0xffffffff8147da70 <sysent+144>) at /usr/src/sys/sys/syscallva=
r.h:65
#6 sy_invoke (code=3D6, rval=3D0xffff80009ef1deb0, uap=3D0xffff80009ef=
1df00, =
l=3D0xfffffe82214bf2c0, sy=3D0xffffffff8147da70 <sysent+144>)
at /usr/src/sys/sys/syscallvar.h:94
#7 syscall (frame=3D0xffff80009ef1df00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#8 0xffffffff8020768d in handle_syscall ()
#9 0x0000000000000004 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe8225709000
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8225709000)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8225709000)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =
ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =
obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =
sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fef2d in _fstrans_start (wait=3D1, lock_type=3DFSTRANS=
_SHARED, =
mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
#5 fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
.c:414
#6 0xffffffff80a0e399 in vop_pre (vp=3D0xfffffe82220a8690, =
vp=3D0xfffffe82220a8690, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
, =
mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#7 VOP_POLL (vp=3D0xfffffe82220a8690, events=3D<optimized out>)
at /usr/src/sys/kern/vnode_if.c:677
#8 0xffffffff809bd7e8 in selscan (retval=3D0xffff80009e90eeb0, ni=3D0,=
nfd=3D5, =
bits=3D0xffff80009e90ed48 "\030") at /usr/src/sys/kern/sys_select.c=
:402
#9 sel_do_scan (op=3Dop@entry=3D1, fds=3Dfds@entry=3D0xffff80009e90ed4=
8, =
nf=3Dnf@entry=3D5, ni=3Dni@entry=3D4, ts=3Dts@entry=3D0x0, mask=3Dm=
ask@entry=3D0x0, =
retval=3Dretval@entry=3D0xffff80009e90eeb0)
at /usr/src/sys/kern/sys_select.c:264
#10 0xffffffff809bdb96 in selcommon (retval=3Dretval@entry=3D0xffff8000=
9e90eeb0, =
nd=3D5, u_in=3D0x7aeb665fff40, u_ou=3D0x7aeb665fff60, u_ex=3D0x7aeb=
665fff80, =
ts=3D0x0, mask=3Dmask@entry=3D0x0) at /usr/src/sys/kern/sys_select.=
c:362
#11 0xffffffff809bdeab in selcommon (mask=3D0x0, ts=3D<optimized out>, =
u_ex=3D<optimized out>, u_ou=3D<optimized out>, u_in=3D<optimized o=
ut>, =
nd=3D<optimized out>, retval=3D0xffff80009e90eeb0)
at /usr/src/sys/kern/sys_select.c:337
#12 sys___select50 (l=3D<optimized out>, uap=3D0xffff80009e90ef00, =
retval=3D0xffff80009e90eeb0) at /usr/src/sys/kern/sys_select.c:212
#13 0xffffffff80248dbd in sy_call (rval=3D0xffff80009e90eeb0, =
uap=3D0xffff80009e90ef00, l=3D0xfffffe8225709000, =
sy=3D0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscall=
var.h:65
#14 sy_invoke (code=3D417, rval=3D0xffff80009e90eeb0, uap=3D0xffff80009=
e90ef00, =
l=3D0xfffffe8225709000, sy=3D0xffffffff814800f8 <sysent+10008>)
at /usr/src/sys/sys/syscallvar.h:94
#15 syscall (frame=3D0xffff80009e90ef00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#16 0xffffffff8020768d in handle_syscall ()
#17 0x0000000000000005 in ?? ()
#18 0x00007aeb665fff40 in ?? ()
#19 0x00007aeb665fff60 in ?? ()
#20 0x00007aeb665fff80 in ?? ()
#21 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe820df34480
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe820df34480)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe820df34480)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8094afca in cv_wait (cv=3Dcv@entry=3D0xfffffe8207461358, =
mtx=3Dmtx@entry=3D0xfffffe82074613c0) at /usr/src/sys/kern/kern_con=
dvar.c:217
#3 0xffffffff8094e4e5 in fd_close (fd=3D<optimized out>)
at /usr/src/sys/kern/kern_descrip.c:684
#4 0xffffffff809b3c88 in sys_close (l=3D0xfffffe820df34480, =
uap=3D0xffff80009f668f00, retval=3D<optimized out>)
at /usr/src/sys/kern/sys_descrip.c:488
#5 0xffffffff80248dbd in sy_call (rval=3D0xffff80009f668eb0, =
uap=3D0xffff80009f668f00, l=3D0xfffffe820df34480, =
sy=3D0xffffffff8147da70 <sysent+144>) at /usr/src/sys/sys/syscallva=
r.h:65
#6 sy_invoke (code=3D6, rval=3D0xffff80009f668eb0, uap=3D0xffff80009f6=
68f00, =
l=3D0xfffffe820df34480, sy=3D0xffffffff8147da70 <sysent+144>)
at /usr/src/sys/sys/syscallvar.h:94
#7 syscall (frame=3D0xffff80009f668f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#8 0xffffffff8020768d in handle_syscall ()
#9 0x0000000000000004 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe8207fd0360
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8207fd0360)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8207fd0360)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =
ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =
obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =
sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fef2d in _fstrans_start (wait=3D1, lock_type=3DFSTRANS=
_SHARED, =
mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
#5 fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
.c:414
#6 0xffffffff80a0e399 in vop_pre (vp=3D0xfffffe822122fa88, =
vp=3D0xfffffe822122fa88, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
, =
mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#7 VOP_POLL (vp=3D0xfffffe822122fa88, events=3D<optimized out>)
at /usr/src/sys/kern/vnode_if.c:677
#8 0xffffffff809bd7e8 in selscan (retval=3D0xffff80009f5f9eb0, ni=3D0,=
nfd=3D5, =
bits=3D0xffff80009f5f9d48 "\030") at /usr/src/sys/kern/sys_select.c=
:402
#9 sel_do_scan (op=3Dop@entry=3D1, fds=3Dfds@entry=3D0xffff80009f5f9d4=
8, =
nf=3Dnf@entry=3D5, ni=3Dni@entry=3D4, ts=3Dts@entry=3D0x0, mask=3Dm=
ask@entry=3D0x0, =
retval=3Dretval@entry=3D0xffff80009f5f9eb0)
at /usr/src/sys/kern/sys_select.c:264
#10 0xffffffff809bdb96 in selcommon (retval=3Dretval@entry=3D0xffff8000=
9f5f9eb0, =
nd=3D5, u_in=3D0x7d66527fff40, u_ou=3D0x7d66527fff60, u_ex=3D0x7d66=
527fff80, =
ts=3D0x0, mask=3Dmask@entry=3D0x0) at /usr/src/sys/kern/sys_select.=
c:362
#11 0xffffffff809bdeab in selcommon (mask=3D0x0, ts=3D<optimized out>, =
u_ex=3D<optimized out>, u_ou=3D<optimized out>, u_in=3D<optimized o=
ut>, =
nd=3D<optimized out>, retval=3D0xffff80009f5f9eb0)
at /usr/src/sys/kern/sys_select.c:337
#12 sys___select50 (l=3D<optimized out>, uap=3D0xffff80009f5f9f00, =
retval=3D0xffff80009f5f9eb0) at /usr/src/sys/kern/sys_select.c:212
#13 0xffffffff80248dbd in sy_call (rval=3D0xffff80009f5f9eb0, =
uap=3D0xffff80009f5f9f00, l=3D0xfffffe8207fd0360, =
sy=3D0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscall=
var.h:65
#14 sy_invoke (code=3D417, rval=3D0xffff80009f5f9eb0, uap=3D0xffff80009=
f5f9f00, =
l=3D0xfffffe8207fd0360, sy=3D0xffffffff814800f8 <sysent+10008>)
at /usr/src/sys/sys/syscallvar.h:94
#15 syscall (frame=3D0xffff80009f5f9f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#16 0xffffffff8020768d in handle_syscall ()
#17 0x0000000000000005 in ?? ()
#18 0x00007d66527fff40 in ?? ()
#19 0x00007d66527fff60 in ?? ()
#20 0x00007d66527fff80 in ?? ()
#21 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe822346f080
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe822346f080)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe822346f080)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8094afca in cv_wait (cv=3Dcv@entry=3D0xfffffe82243bfc18, =
mtx=3Dmtx@entry=3D0xfffffe82243bfc80) at /usr/src/sys/kern/kern_con=
dvar.c:217
#3 0xffffffff8094e4e5 in fd_close (fd=3D<optimized out>)
at /usr/src/sys/kern/kern_descrip.c:684
#4 0xffffffff809b3c88 in sys_close (l=3D0xfffffe822346f080, =
uap=3D0xffff80009efd4f00, retval=3D<optimized out>)
at /usr/src/sys/kern/sys_descrip.c:488
#5 0xffffffff80248dbd in sy_call (rval=3D0xffff80009efd4eb0, =
uap=3D0xffff80009efd4f00, l=3D0xfffffe822346f080, =
sy=3D0xffffffff8147da70 <sysent+144>) at /usr/src/sys/sys/syscallva=
r.h:65
#6 sy_invoke (code=3D6, rval=3D0xffff80009efd4eb0, uap=3D0xffff80009ef=
d4f00, =
l=3D0xfffffe822346f080, sy=3D0xffffffff8147da70 <sysent+144>)
at /usr/src/sys/sys/syscallvar.h:94
#7 syscall (frame=3D0xffff80009efd4f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#8 0xffffffff8020768d in handle_syscall ()
#9 0x0000000000000004 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe821d987a40
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821d987a40)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821d987a40)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =
ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =
obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =
sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fef2d in _fstrans_start (wait=3D1, lock_type=3DFSTRANS=
_SHARED, =
mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
#5 fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
.c:414
#6 0xffffffff80a0e399 in vop_pre (vp=3D0xfffffe820ecde930, =
vp=3D0xfffffe820ecde930, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
, =
mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#7 VOP_POLL (vp=3D0xfffffe820ecde930, events=3D<optimized out>)
at /usr/src/sys/kern/vnode_if.c:677
#8 0xffffffff809bd7e8 in selscan (retval=3D0xffff80009ee64eb0, ni=3D0,=
nfd=3D5, =
bits=3D0xffff80009ee64d48 "\030") at /usr/src/sys/kern/sys_select.c=
:402
#9 sel_do_scan (op=3Dop@entry=3D1, fds=3Dfds@entry=3D0xffff80009ee64d4=
8, =
nf=3Dnf@entry=3D5, ni=3Dni@entry=3D4, ts=3Dts@entry=3D0x0, mask=3Dm=
ask@entry=3D0x0, =
retval=3Dretval@entry=3D0xffff80009ee64eb0)
at /usr/src/sys/kern/sys_select.c:264
#10 0xffffffff809bdb96 in selcommon (retval=3Dretval@entry=3D0xffff8000=
9ee64eb0, =
nd=3D5, u_in=3D0x7ea7fb7fff40, u_ou=3D0x7ea7fb7fff60, u_ex=3D0x7ea7=
fb7fff80, =
ts=3D0x0, mask=3Dmask@entry=3D0x0) at /usr/src/sys/kern/sys_select.=
c:362
#11 0xffffffff809bdeab in selcommon (mask=3D0x0, ts=3D<optimized out>, =
u_ex=3D<optimized out>, u_ou=3D<optimized out>, u_in=3D<optimized o=
ut>, =
nd=3D<optimized out>, retval=3D0xffff80009ee64eb0)
at /usr/src/sys/kern/sys_select.c:337
#12 sys___select50 (l=3D<optimized out>, uap=3D0xffff80009ee64f00, =
retval=3D0xffff80009ee64eb0) at /usr/src/sys/kern/sys_select.c:212
#13 0xffffffff80248dbd in sy_call (rval=3D0xffff80009ee64eb0, =
uap=3D0xffff80009ee64f00, l=3D0xfffffe821d987a40, =
sy=3D0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscall=
var.h:65
#14 sy_invoke (code=3D417, rval=3D0xffff80009ee64eb0, uap=3D0xffff80009=
ee64f00, =
l=3D0xfffffe821d987a40, sy=3D0xffffffff814800f8 <sysent+10008>)
at /usr/src/sys/sys/syscallvar.h:94
#15 syscall (frame=3D0xffff80009ee64f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#16 0xffffffff8020768d in handle_syscall ()
#17 0x0000000000000005 in ?? ()
#18 0x00007ea7fb7fff40 in ?? ()
#19 0x00007ea7fb7fff60 in ?? ()
#20 0x00007ea7fb7fff80 in ?? ()
#21 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe821e90c240
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821e90c240)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821e90c240)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8094afca in cv_wait (cv=3Dcv@entry=3D0xfffffe822400e918, =
mtx=3Dmtx@entry=3D0xfffffe822400e980) at /usr/src/sys/kern/kern_con=
dvar.c:217
#3 0xffffffff8094e4e5 in fd_close (fd=3D<optimized out>)
at /usr/src/sys/kern/kern_descrip.c:684
#4 0xffffffff809b3c88 in sys_close (l=3D0xfffffe821e90c240, =
uap=3D0xffff80009ee94f00, retval=3D<optimized out>)
at /usr/src/sys/kern/sys_descrip.c:488
#5 0xffffffff80248dbd in sy_call (rval=3D0xffff80009ee94eb0, =
uap=3D0xffff80009ee94f00, l=3D0xfffffe821e90c240, =
sy=3D0xffffffff8147da70 <sysent+144>) at /usr/src/sys/sys/syscallva=
r.h:65
#6 sy_invoke (code=3D6, rval=3D0xffff80009ee94eb0, uap=3D0xffff80009ee=
94f00, =
l=3D0xfffffe821e90c240, sy=3D0xffffffff8147da70 <sysent+144>)
at /usr/src/sys/sys/syscallvar.h:94
#7 syscall (frame=3D0xffff80009ee94f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#8 0xffffffff8020768d in handle_syscall ()
#9 0x0000000000000004 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe821d362980
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821d362980)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821d362980)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =
ts@entry=3D0xfffffe821f9db480, q=3Dq@entry=3D1, =
obj=3Dobj@entry=3D0xfffffe8224a7b130, =
sobj=3Dsobj@entry=3D0xffffffff81482720 <rw_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80977cb3 in rw_vector_enter (rw=3D0xfffffe8224a7b130, op=3D=
RW_WRITER)
at /usr/src/sys/kern/kern_rwlock.c:395
#4 0xffffffff80a17d7e in genfs_lock (v=3D<optimized out>)
at /usr/src/sys/miscfs/genfs/genfs_vnops.c:343
#5 0xffffffff80a0ed35 in VOP_LOCK (vp=3Dvp@entry=3D0xfffffe8224a7b000,=
=
flags=3Dflags@entry=3D131074) at /usr/src/sys/kern/vnode_if.c:1284
#6 0xffffffff80a05846 in vn_lock (vp=3Dvp@entry=3D0xfffffe8224a7b000, =
flags=3Dflags@entry=3D131074) at /usr/src/sys/kern/vfs_vnops.c:1036=
#7 0xffffffff80a0601d in vn_write (fp=3D<optimized out>, =
offset=3D0xfffffe8225588b00, uio=3D0xffff80009edaedf0, =
cred=3D0xfffffe822f724f00, flags=3D1) at /usr/src/sys/kern/vfs_vnop=
s.c:592
#8 0xffffffff809b4c1d in dofilewrite (fd=3Dfd@entry=3D2, fp=3D0xfffffe=
8225588b00, =
buf=3D0x764fb6902000, nbyte=3D94, offset=3D<optimized out>, flags=3D=
flags@entry=3D1, =
retval=3Dretval@entry=3D0xffff80009edaeeb0)
at /usr/src/sys/kern/sys_generic.c:350
#9 0xffffffff809b4d22 in sys_write (l=3D<optimized out>, =
uap=3D0xffff80009edaef00, retval=3D0xffff80009edaeeb0)
at /usr/src/sys/kern/sys_generic.c:318
#10 0xffffffff80248dbd in sy_call (rval=3D0xffff80009edaeeb0, =
uap=3D0xffff80009edaef00, l=3D0xfffffe821d362980, =
sy=3D0xffffffff8147da40 <sysent+96>) at /usr/src/sys/sys/syscallvar=
.h:65
#11 sy_invoke (code=3D4, rval=3D0xffff80009edaeeb0, uap=3D0xffff80009ed=
aef00, =
l=3D0xfffffe821d362980, sy=3D0xffffffff8147da40 <sysent+96>)
at /usr/src/sys/sys/syscallvar.h:94
#12 syscall (frame=3D0xffff80009edaef00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#13 0xffffffff8020768d in handle_syscall ()
#14 0x0000000000000002 in ?? ()
#15 0x0000764fb6902000 in ?? ()
#16 0x000000000000005e in ?? ()
#17 0x0000764fb690205e in ?? ()
#18 0x000000000000001f in ?? ()
#19 0x0000000000000001 in ?? ()
#20 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe8229b924c0
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8229b924c0)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8229b924c0)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =
ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =
obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =
sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fef2d in _fstrans_start (wait=3D1, lock_type=3DFSTRANS=
_SHARED, =
mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
#5 fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
.c:414
#6 0xffffffff80a0e399 in vop_pre (vp=3D0xfffffe821d7d1690, =
vp=3D0xfffffe821d7d1690, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
, =
mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#7 VOP_POLL (vp=3D0xfffffe821d7d1690, events=3D<optimized out>)
at /usr/src/sys/kern/vnode_if.c:677
#8 0xffffffff809bd7e8 in selscan (retval=3D0xffff80009e9c3eb0, ni=3D36=
39083008, =
nfd=3D48, bits=3D0xffff80009e9c3d48 "\b@\350\330(")
at /usr/src/sys/kern/sys_select.c:402
#9 sel_do_scan (op=3Dop@entry=3D1, fds=3Dfds@entry=3D0xffff80009e9c3d4=
8, =
nf=3Dnf@entry=3D48, ni=3Dni@entry=3D8, ts=3Dts@entry=3D0xffff80009e=
9c3e70, =
mask=3Dmask@entry=3D0x0, retval=3Dretval@entry=3D0xffff80009e9c3eb0=
)
at /usr/src/sys/kern/sys_select.c:264
#10 0xffffffff809bdb96 in selcommon (retval=3Dretval@entry=3D0xffff8000=
9e9c3eb0, =
nd=3D48, u_in=3D0x7f7fff4c5a10, u_ou=3D0x7f7fff4c5a30, u_ex=3D0x0, =
ts=3D0xffff80009e9c3e70, mask=3Dmask@entry=3D0x0)
at /usr/src/sys/kern/sys_select.c:362
#11 0xffffffff809bdeab in selcommon (mask=3D0x0, ts=3D<optimized out>, =
u_ex=3D<optimized out>, u_ou=3D<optimized out>, u_in=3D<optimized o=
ut>, =
nd=3D<optimized out>, retval=3D0xffff80009e9c3eb0)
at /usr/src/sys/kern/sys_select.c:337
#12 sys___select50 (l=3D<optimized out>, uap=3D0xffff80009e9c3f00, =
retval=3D0xffff80009e9c3eb0) at /usr/src/sys/kern/sys_select.c:212
#13 0xffffffff80248dbd in sy_call (rval=3D0xffff80009e9c3eb0, =
uap=3D0xffff80009e9c3f00, l=3D0xfffffe8229b924c0, =
sy=3D0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscall=
var.h:65
#14 sy_invoke (code=3D417, rval=3D0xffff80009e9c3eb0, uap=3D0xffff80009=
e9c3f00, =
l=3D0xfffffe8229b924c0, sy=3D0xffffffff814800f8 <sysent+10008>)
at /usr/src/sys/sys/syscallvar.h:94
#15 syscall (frame=3D0xffff80009e9c3f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#16 0xffffffff8020768d in handle_syscall ()
#17 0x0000000000000030 in ?? ()
#18 0x00007f7fff4c5a10 in ?? ()
#19 0x00007f7fff4c5a30 in ?? ()
#20 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe8220b8e780
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8220b8e780)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8220b8e780)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, ts@ent=
ry=3D0x0, =
q=3Dq@entry=3D1, obj=3Dobj@entry=3D0xfffffe8224a7b130, =
sobj=3Dsobj@entry=3D0xffffffff81482720 <rw_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80977cb3 in rw_vector_enter (rw=3D0xfffffe8224a7b130, op=3D=
RW_WRITER)
at /usr/src/sys/kern/kern_rwlock.c:395
#4 0xffffffff80a17d7e in genfs_lock (v=3D<optimized out>)
at /usr/src/sys/miscfs/genfs/genfs_vnops.c:343
#5 0xffffffff80a0ed35 in VOP_LOCK (vp=3Dvp@entry=3D0xfffffe8224a7b000,=
=
flags=3Dflags@entry=3D131074) at /usr/src/sys/kern/vnode_if.c:1284
#6 0xffffffff80a05846 in vn_lock (vp=3Dvp@entry=3D0xfffffe8224a7b000, =
flags=3Dflags@entry=3D131074) at /usr/src/sys/kern/vfs_vnops.c:1036=
#7 0xffffffff80a0601d in vn_write (fp=3D<optimized out>, =
offset=3D0xfffffe8225588b00, uio=3D0xffff80009ebf8df0, =
cred=3D0xfffffe822f724f00, flags=3D1) at /usr/src/sys/kern/vfs_vnop=
s.c:592
#8 0xffffffff809b4c1d in dofilewrite (fd=3Dfd@entry=3D2, fp=3D0xfffffe=
8225588b00, =
buf=3D0x764fb6902000, nbyte=3D90, offset=3D<optimized out>, flags=3D=
flags@entry=3D1, =
retval=3Dretval@entry=3D0xffff80009ebf8eb0)
at /usr/src/sys/kern/sys_generic.c:350
#9 0xffffffff809b4d22 in sys_write (l=3D<optimized out>, =
uap=3D0xffff80009ebf8f00, retval=3D0xffff80009ebf8eb0)
at /usr/src/sys/kern/sys_generic.c:318
#10 0xffffffff80248dbd in sy_call (rval=3D0xffff80009ebf8eb0, =
uap=3D0xffff80009ebf8f00, l=3D0xfffffe8220b8e780, =
sy=3D0xffffffff8147da40 <sysent+96>) at /usr/src/sys/sys/syscallvar=
.h:65
#11 sy_invoke (code=3D4, rval=3D0xffff80009ebf8eb0, uap=3D0xffff80009eb=
f8f00, =
l=3D0xfffffe8220b8e780, sy=3D0xffffffff8147da40 <sysent+96>)
at /usr/src/sys/sys/syscallvar.h:94
#12 syscall (frame=3D0xffff80009ebf8f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#13 0xffffffff8020768d in handle_syscall ()
#14 0x0000000000000002 in ?? ()
#15 0x0000764fb6902000 in ?? ()
#16 0x000000000000005a in ?? ()
#17 0x0000764fb690205a in ?? ()
#18 0x000000000000001e in ?? ()
#19 0x0000000000000001 in ?? ()
#20 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe8221e57600
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8221e57600)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8221e57600)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =
ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =
obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =
sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fef2d in _fstrans_start (wait=3D1, lock_type=3DFSTRANS=
_SHARED, =
mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
#5 fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
.c:414
#6 0xffffffff80a0e399 in vop_pre (vp=3D0xfffffe8221e36bd0, =
vp=3D0xfffffe8221e36bd0, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
, =
mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#7 VOP_POLL (vp=3D0xfffffe8221e36bd0, events=3D<optimized out>)
at /usr/src/sys/kern/vnode_if.c:677
#8 0xffffffff809bd7e8 in selscan (retval=3D0xffff80009eae8eb0, ni=3D20=
46820352, =
nfd=3D52, bits=3D0xffff80009eae8d48 "H") at /usr/src/sys/kern/sys_s=
elect.c:402
#9 sel_do_scan (op=3Dop@entry=3D1, fds=3Dfds@entry=3D0xffff80009eae8d4=
8, =
nf=3Dnf@entry=3D52, ni=3Dni@entry=3D8, ts=3Dts@entry=3D0xffff80009e=
ae8e70, =
mask=3Dmask@entry=3D0x0, retval=3Dretval@entry=3D0xffff80009eae8eb0=
)
at /usr/src/sys/kern/sys_select.c:264
#10 0xffffffff809bdb96 in selcommon (retval=3Dretval@entry=3D0xffff8000=
9eae8eb0, =
nd=3D52, u_in=3D0x7f7fff4c5a10, u_ou=3D0x7f7fff4c5a30, u_ex=3D0x0, =
ts=3D0xffff80009eae8e70, mask=3Dmask@entry=3D0x0)
at /usr/src/sys/kern/sys_select.c:362
#11 0xffffffff809bdeab in selcommon (mask=3D0x0, ts=3D<optimized out>, =
u_ex=3D<optimized out>, u_ou=3D<optimized out>, u_in=3D<optimized o=
ut>, =
nd=3D<optimized out>, retval=3D0xffff80009eae8eb0)
at /usr/src/sys/kern/sys_select.c:337
#12 sys___select50 (l=3D<optimized out>, uap=3D0xffff80009eae8f00, =
retval=3D0xffff80009eae8eb0) at /usr/src/sys/kern/sys_select.c:212
#13 0xffffffff80248dbd in sy_call (rval=3D0xffff80009eae8eb0, =
uap=3D0xffff80009eae8f00, l=3D0xfffffe8221e57600, =
sy=3D0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscall=
var.h:65
#14 sy_invoke (code=3D417, rval=3D0xffff80009eae8eb0, uap=3D0xffff80009=
eae8f00, =
l=3D0xfffffe8221e57600, sy=3D0xffffffff814800f8 <sysent+10008>)
at /usr/src/sys/sys/syscallvar.h:94
#15 syscall (frame=3D0xffff80009eae8f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#16 0xffffffff8020768d in handle_syscall ()
#17 0x0000000000000034 in ?? ()
#18 0x00007f7fff4c5a10 in ?? ()
#19 0x00007f7fff4c5a30 in ?? ()
#20 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe8222569900
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8222569900)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8222569900)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =
ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =
obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =
sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fef2d in _fstrans_start (wait=3D1, lock_type=3DFSTRANS=
_SHARED, =
mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
#5 fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
.c:414
#6 0xffffffff80a0e399 in vop_pre (vp=3D0xfffffe8223aa62a0, =
vp=3D0xfffffe8223aa62a0, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
, =
mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#7 VOP_POLL (vp=3D0xfffffe8223aa62a0, events=3D<optimized out>)
at /usr/src/sys/kern/vnode_if.c:677
#8 0xffffffff809bd7e8 in selscan (retval=3D0xffff80009e9cdeb0, ni=3D79=
6917760, =
nfd=3D53, bits=3D0xffff80009e9cdd48 "\b") at /usr/src/sys/kern/sys_=
select.c:402
#9 sel_do_scan (op=3Dop@entry=3D1, fds=3Dfds@entry=3D0xffff80009e9cdd4=
8, =
nf=3Dnf@entry=3D53, ni=3Dni@entry=3D8, ts=3Dts@entry=3D0xffff80009e=
9cde70, =
mask=3Dmask@entry=3D0x0, retval=3Dretval@entry=3D0xffff80009e9cdeb0=
)
at /usr/src/sys/kern/sys_select.c:264
#10 0xffffffff809bdb96 in selcommon (retval=3Dretval@entry=3D0xffff8000=
9e9cdeb0, =
nd=3D53, u_in=3D0x7f7fff4c5a10, u_ou=3D0x7f7fff4c5a30, u_ex=3D0x0, =
ts=3D0xffff80009e9cde70, mask=3Dmask@entry=3D0x0)
at /usr/src/sys/kern/sys_select.c:362
#11 0xffffffff809bdeab in selcommon (mask=3D0x0, ts=3D<optimized out>, =
u_ex=3D<optimized out>, u_ou=3D<optimized out>, u_in=3D<optimized o=
ut>, =
nd=3D<optimized out>, retval=3D0xffff80009e9cdeb0)
at /usr/src/sys/kern/sys_select.c:337
#12 sys___select50 (l=3D<optimized out>, uap=3D0xffff80009e9cdf00, =
retval=3D0xffff80009e9cdeb0) at /usr/src/sys/kern/sys_select.c:212
#13 0xffffffff80248dbd in sy_call (rval=3D0xffff80009e9cdeb0, =
uap=3D0xffff80009e9cdf00, l=3D0xfffffe8222569900, =
sy=3D0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscall=
var.h:65
#14 sy_invoke (code=3D417, rval=3D0xffff80009e9cdeb0, uap=3D0xffff80009=
e9cdf00, =
l=3D0xfffffe8222569900, sy=3D0xffffffff814800f8 <sysent+10008>)
at /usr/src/sys/sys/syscallvar.h:94
#15 syscall (frame=3D0xffff80009e9cdf00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#16 0xffffffff8020768d in handle_syscall ()
#17 0x0000000000000035 in ?? ()
#18 0x00007f7fff4c5a10 in ?? ()
#19 0x00007f7fff4c5a30 in ?? ()
#20 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe8225709840
0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8225709840)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp =3D cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8225709840)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =
catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
:264
#2 0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =
ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =
obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =
sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fef2d in _fstrans_start (wait=3D1, lock_type=3DFSTRANS=
_SHARED, =
mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
#5 fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
.c:414
#6 0xffffffff80a0e399 in vop_pre (vp=3D0xfffffe8222fe4bd8, =
vp=3D0xfffffe8222fe4bd8, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
, =
mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#7 VOP_POLL (vp=3D0xfffffe8222fe4bd8, events=3D<optimized out>)
at /usr/src/sys/kern/vnode_if.c:677
#8 0xffffffff809bd7e8 in selscan (retval=3D0xffff80009e904eb0, ni=3D26=
25634304, =
nfd=3D53, bits=3D0xffff80009e904d48 "\b") at /usr/src/sys/kern/sys_=
select.c:402
#9 sel_do_scan (op=3Dop@entry=3D1, fds=3Dfds@entry=3D0xffff80009e904d4=
8, =
nf=3Dnf@entry=3D53, ni=3Dni@entry=3D8, ts=3Dts@entry=3D0xffff80009e=
904e70, =
mask=3Dmask@entry=3D0x0, retval=3Dretval@entry=3D0xffff80009e904eb0=
)
at /usr/src/sys/kern/sys_select.c:264
#10 0xffffffff809bdb96 in selcommon (retval=3Dretval@entry=3D0xffff8000=
9e904eb0, =
nd=3D53, u_in=3D0x7f7fff4c5a10, u_ou=3D0x7f7fff4c5a30, u_ex=3D0x0, =
ts=3D0xffff80009e904e70, mask=3Dmask@entry=3D0x0)
at /usr/src/sys/kern/sys_select.c:362
#11 0xffffffff809bdeab in selcommon (mask=3D0x0, ts=3D<optimized out>, =
u_ex=3D<optimized out>, u_ou=3D<optimized out>, u_in=3D<optimized o=
ut>, =
nd=3D<optimized out>, retval=3D0xffff80009e904eb0)
at /usr/src/sys/kern/sys_select.c:337
#12 sys___select50 (l=3D<optimized out>, uap=3D0xffff80009e904f00, =
retval=3D0xffff80009e904eb0) at /usr/src/sys/kern/sys_select.c:212
#13 0xffffffff80248dbd in sy_call (rval=3D0xffff80009e904eb0, =
uap=3D0xffff80009e904f00, l=3D0xfffffe8225709840, =
sy=3D0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscall=
var.h:65
#14 sy_invoke (code=3D417, rval=3D0xffff80009e904eb0, uap=3D0xffff80009=
e904f00, =
l=3D0xfffffe8225709840, sy=3D0xffffffff814800f8 <sysent+10008>)
at /usr/src/sys/sys/syscallvar.h:94
#15 syscall (frame=3D0xffff80009e904f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#16 0xffffffff8020768d in handle_syscall ()
#17 0x0000000000000035 in ?? ()
#18 0x00007f7fff4c5a10 in ?? ()
#19 0x00007f7fff4c5a30 in ?? ()
#20 0x0000000000000000 in ?? ()
(gdb)
From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Wed, 16 May 2018 13:37:28 +0200 (CEST)
[[ Hmm, here's a non-quoted-printable version of the previous ]]
Hi,
we provoked another wedge, and captured a kernel core dump from
the wedging machine. The kernel core dump and NetBSD images are
available for looking at.
using gdb and crash, here's a brief summary of the "interesting"
processes in the crash dump, and below that I include backtraces
of all the waiting processes.
There's lots of contention for fstrans_lock.
Offhand I don't see a deadlock which might explain the observed
behaviour (goes totally "deaf" on the network, i.e. doesn't even
respond to ping).
It doesn't look like gdb can trace through interrupt frames (?),
looking at proc 788 all I get is:
(gdb) kvm proc 0xfffffe8220b8e360
0xffffffff8021cfe0 in softintr_ret ()
(gdb) where
#0 0xffffffff8021cfe0 in softintr_ret ()
#1 0x0000000000000000 in ?? ()
(gdb)
Crash manages to do this one, though, apparently (see below).
Furthermore, using the various gdb scripts in
/usr/src/sys/gdbscripts/ I can look at some of the locks.
It is quite possible that up'ing the interface in question causes
lots of activity for opening pty pairs, and that the root cause
of the issue is there rather than related to networking in itself(?)
Further hints?
Regards,
- Havard
------------------------------
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
7052 1 3 1 8020000 fffffe8220c58540 cron tstile
Wants fstrans_lock
9187 1 3 6 8020000 fffffe821faa60c0 expect xchicv
Holds fstrans_lock, in pserialize_perform, waits on condition variable
after doing xc_broadcast(XC_HIGHPRI, nullop)
Doing (roughly) pty_grant_slave -> genfs_revoke -> vfs_suspend ->
fstrans_setstate -> pserialize_perform -> xc_wait -> cv_wait
3794 1 3 7 8020000 fffffe82045da540 ssh tstile
Wants fstrans_lock
2853 2 3 6 8020000 fffffe8216d35140 expect tstile
Wants fstrans_lock
2853 1 3 7 8020000 fffffe82214bf2c0 expect fdclose
Doing cv_wait(&ff->ff_closing, &fdp->fd_lock);
8327 2 3 3 8020000 fffffe8225709000 expect tstile
Wants fstrans_lock
8327 1 3 1 8020000 fffffe820df34480 expect fdclose
Doing cv_wait(&ff->ff_closing, &fdp->fd_lock);
5687 2 3 3 8020000 fffffe8207fd0360 expect tstile
Wants fstrans_lock
5687 1 3 2 8020000 fffffe822346f080 expect fdclose
Doing cv_wait(&ff->ff_closing, &fdp->fd_lock);
9671 2 3 5 8020000 fffffe821d987a40 expect tstile
Wants fstrans_lock
9671 1 3 4 8020000 fffffe821e90c240 expect fdclose
Doing cv_wait(&ff->ff_closing, &fdp->fd_lock);
704 1 3 6 8020000 fffffe821d362980 conserver tstile
Wants rw_enter(&vip->vi_lock, op);, lock held by 788 apparently
564 1 3 1 8020000 fffffe8229b924c0 conserver tstile
Wants fstrans_lock
788 > 1 7 0 8020000 fffffe8220b8e360 conserver
Doing softintr_ret(), gdb doesn't trace further (interrupt frame?)
Crash, though, says:
crash> trace/a 0xfffffe8220b8e360
trace: pid 788 lid 1 at 0xffff80009ebfdc40
ufs_post_write_update.isra.0() at ufs_post_write_update.isra.0+0x95
ffs_write() at ffs_write+0x56f
VOP_WRITE() at VOP_WRITE+0x37
vn_write() at vn_write+0xec
dofilewrite() at dofilewrite+0x97
sys_write() at sys_write+0x5f
syscall() at syscall+0x1ed
--- syscall (number 4) ---
764fb503e02a:
crash>
275 1 3 5 8020000 fffffe8220b8e780 conserver tstile
Wants rw_enter(&vip->vi_lock, op);, lock held by 788 apparently
878 1 3 2 8020000 fffffe8221e57600 conserver tstile
Wants fstrans_lock
136 1 3 3 8020000 fffffe8222569900 conserver tstile
Wants fstrans_lock
773 1 3 1 8020000 fffffe8225709840 conserver tstile
Wants fstrans_lock
------------------------------
crash> ps
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
7052 1 3 1 8020000 fffffe8220c58540 cron tstile
8587 1 3 1 8020000 fffffe821d362560 cron wait
9187 1 3 6 8020000 fffffe821faa60c0 expect xchicv
5064 1 2 6 8020000 fffffe8219f93540 sh
7813 1 2 0 8020000 fffffe8220ecf860 perl
7354 1 2 0 8020000 fffffe8207aba240 sh
3794 1 3 7 8020000 fffffe82045da540 ssh tstile
2853 2 3 6 8020000 fffffe8216d35140 expect tstile
2853 1 3 7 8020000 fffffe82214bf2c0 expect fdclose
6097 1 2 0 8020000 fffffe8221165720 sh
5738 1 2 0 8020000 fffffe821e90ca80 perl
4461 1 2 0 8020000 fffffe8207a90aa0 sh
4310 1 2 3 8020000 fffffe8220c2c5a0 ssh
8327 2 3 3 8020000 fffffe8225709000 expect tstile
8327 1 3 1 8020000 fffffe820df34480 expect fdclose
6086 1 2 5 8020000 fffffe821d2d9a60 sh
3214 1 2 0 8020000 fffffe821f5d8360 perl
5921 1 2 0 8020000 fffffe8207a90260 sh
2890 1 2 2 8020000 fffffe8228bacae0 ssh
3772 1 2 0 8020000 fffffe821a27f420 sh
4425 1 2 0 8020000 fffffe821f5d8780 ssh
5793 1 2 0 8020000 fffffe821d342340 sh
4833 1 2 0 8020000 fffffe82207f5460 ssh
7238 1 2 0 8020000 fffffe822123b640 sh
6312 1 2 4 8020000 fffffe8222a9b500 ssh
7003 1 2 0 8020000 fffffe822160e700 sh
6900 1 2 0 8020000 fffffe8221d82140 ssh
4139 1 2 0 8020000 fffffe822346f8c0 sh
3765 1 2 0 8020000 fffffe821f94f060 ssh
5386 1 2 1 8020000 fffffe8222a7a8c0 sh
4633 1 2 0 8020000 fffffe821f8969c0 ssh
9691 1 2 0 8020000 fffffe821928d180 sh
6528 1 2 7 8020000 fffffe8207f4b460 ssh
5816 1 2 0 8020000 fffffe8207f4b880 sh
6285 1 2 0 8020000 fffffe821e1e28c0 ssh
8366 1 2 0 8020000 fffffe820d267520 sh
8041 1 2 0 8020000 fffffe821f94f8a0 ssh
3407 1 2 0 8020000 fffffe82045da960 sh
2815 1 2 0 8020000 fffffe8225709420 ssh
5687 2 3 3 8020000 fffffe8207fd0360 expect tstile
5687 1 3 2 8020000 fffffe822346f080 expect fdclose
2928 1 2 0 8020000 fffffe8228bac2a0 sh
5127 1 2 0 8020000 fffffe821b498860 perl
5658 1 2 0 8020000 fffffe820ea5eac0 sh
6746 1 2 0 8020000 fffffe821d443520 ssh
9671 2 3 5 8020000 fffffe821d987a40 expect tstile
9671 1 3 4 8020000 fffffe821e90c240 expect fdclose
9105 1 2 0 8020000 fffffe8226ea8b40 sh
3990 1 2 0 8020000 fffffe8227069700 perl
4504 1 2 0 8020000 fffffe8207a90680 sh
9019 1 2 0 8020000 fffffe821f4c78e0 ssh
2825 1 2 0 8020000 fffffe821f896180 sh
5779 1 2 0 8020000 fffffe821a27f840 ssh
2846 1 2 0 8020000 fffffe822af57440 sh
3544 1 2 0 8020000 fffffe82216b1a40 ssh
6513 1 2 0 8020000 fffffe822728b460 sh
5965 1 2 0 8020000 fffffe821faa6900 ssh
6327 1 2 0 8020000 fffffe8221837a80 sh
2739 1 2 0 8020000 fffffe822708e740 ssh
6361 1 2 0 8020000 fffffe82239772a0 sh
3534 1 2 0 8020000 fffffe821f94f480 ssh
5988 1 2 0 8020000 fffffe8222a7a080 sh
7283 1 2 0 8020000 fffffe8222b99980 ssh
6596 1 2 0 8020000 fffffe820ea5e6a0 sh
3343 1 2 0 8020000 fffffe8223049740 ssh
7708 1 2 0 8020000 fffffe822da31140 sh
6381 1 2 0 8020000 fffffe8221d82980 ssh
7056 1 2 0 8020000 fffffe821a542500 sh
4161 1 2 0 8020000 fffffe821c708100 ssh
7144 1 2 0 8020000 fffffe8223049b60 sh
6276 1 2 0 8020000 fffffe821d2d9220 ssh
4460 1 2 0 8020000 fffffe821f36a040 sh
6845 1 2 0 8020000 fffffe8222a9b0e0 ssh
4669 1 2 0 8020000 fffffe821e1e24a0 sh
3652 1 2 0 8020000 fffffe821c708520 ssh
5131 1 2 0 8020000 fffffe820df34060 sh
5704 1 2 0 8020000 fffffe8220ce6760 ssh
6031 1 2 0 8020000 fffffe821ecaea00 sh
5875 1 2 0 8020000 fffffe82240071a0 ssh
5464 1 2 0 8020000 fffffe8205110580 sh
2852 1 2 0 8020000 fffffe82207f5040 ssh
3432 1 2 0 8020000 fffffe820dd5e6e0 sh
4917 1 2 0 8020000 fffffe8221317ac0 ssh
2850 1 2 0 8020000 fffffe8207fd0780 sh
3399 1 2 0 8020000 fffffe822d92e9c0 ssh
3497 1 2 0 8020000 fffffe821d987200 sh
3880 1 2 0 8020000 fffffe821d342b80 ssh
4728 1 2 0 8020000 fffffe822188a5a0 sh
3904 1 2 0 8020000 fffffe820df348a0 ssh
4847 1 2 0 8020000 fffffe8204072900 sh
3465 1 2 0 8020000 fffffe8207fd0ba0 ssh
3881 1 2 0 8020000 fffffe821a27f000 sh
4000 1 2 0 8020000 fffffe821928d9c0 ssh
4242 1 2 0 8020000 fffffe8221165300 sh
4074 1 2 0 8020000 fffffe82207f5880 ssh
3902 1 2 0 8020000 fffffe8205110160 sh
3293 1 2 0 8020000 fffffe8224d918a0 ssh
3228 1 2 0 8020000 fffffe821d362140 sh
4064 1 2 0 8020000 fffffe8224b0c760 ssh
4762 1 2 0 8020000 fffffe822037f1a0 sh
2903 1 2 0 8020000 fffffe8221dab1a0 ssh
2218 1 2 0 8020000 fffffe821fdf9b00 sh
2497 1 2 0 8020000 fffffe822728b880 ssh
3080 1 2 0 8020000 fffffe8223563b40 sh
2573 1 2 0 8020000 fffffe8206fea440 ssh
2635 1 2 0 8020000 fffffe822af57020 sh
2114 1 2 0 8020000 fffffe82220aa520 ssh
4141 1 2 0 8020000 fffffe82240079e0 sh
4477 1 2 0 8020000 fffffe822160eb20 ssh
3978 1 2 0 8020000 fffffe821f6f9b60 sh
3243 1 2 0 8020000 fffffe821f1b0680 ssh
2652 1 2 0 8020000 fffffe821c49f5e0 sh
3804 1 2 0 8020000 fffffe821f37d9a0 ssh
4348 1 2 0 8020000 fffffe822c1e7660 sh
4334 1 2 0 8020000 fffffe8220c2c9c0 ssh
3781 1 2 0 8020000 fffffe82214bfb00 sh
3422 1 2 0 8020000 fffffe82216b1200 ssh
3277 1 2 0 8020000 fffffe8221165b40 sh
3997 1 2 0 8020000 fffffe8206fea020 ssh
3863 1 2 0 8020000 fffffe82239776c0 sh
2950 1 2 0 8020000 fffffe821f6f9320 par
2644 1 2 0 8020000 fffffe822220f9a0 ssh
2418 1 2 0 8020000 fffffe822253d1c0 sh
2616 1 2 0 8020000 fffffe821e1e2080 ssh
2724 1 2 0 8020000 fffffe821b0306a0 sh
1968 1 2 0 8020000 fffffe8224d91060 sh
3042 1 2 0 8020000 fffffe8221dab5c0 sh
2259 1 2 0 8020000 fffffe821c4bb4e0 sh
2733 1 2 0 8020000 fffffe8223c97a40 sh
2608 1 2 0 8020000 fffffe821b498020 cron
2402 1 2 0 8020000 fffffe822123ba60 ssh
2346 1 2 0 8020000 fffffe821f6f9740 sh
3141 1 2 0 8020000 fffffe8221837240 ssh
1970 1 2 0 8020000 fffffe821f4c70a0 sh
2649 1 2 0 8020000 fffffe821a5f28e0 ssh
2129 1 2 0 8020000 fffffe8227c3cb00 sh
2384 1 2 0 8020000 fffffe8221afe480 ssh
2971 1 2 0 8020000 fffffe82225690c0 sh
2070 1 2 0 8020000 fffffe82212562e0 ssh
2098 1 2 0 8020000 fffffe82201c92a0 sh
2764 1 2 0 8020000 fffffe821bf22680 ssh
3322 1 2 0 8020000 fffffe8221256b20 sh
2511 1 2 0 8020000 fffffe821f37d160 ssh
2661 1 2 0 8020000 fffffe821ecae1c0 sh
3491 1 2 0 8020000 fffffe822220f160 ssh
3195 1 2 0 8020000 fffffe822188a9c0 sh
3305 1 2 0 8020000 fffffe821b800600 ssh
2197 1 2 0 8020000 fffffe821d2d9640 sh
3011 1 2 0 8020000 fffffe82214bf6e0 ssh
3056 1 2 0 8020000 fffffe82288f2ac0 sh
2350 1 2 0 8020000 fffffe8219f93960 ssh
2297 1 2 0 8020000 fffffe8221e571e0 sh
2638 1 2 0 8020000 fffffe821a5f20a0 ssh
2169 1 2 0 8020000 fffffe821c4bb0c0 sh
2550 1 2 0 8020000 fffffe82213176a0 ssh
2225 1 2 0 8020000 fffffe82220aa100 sh
2677 1 2 0 8020000 fffffe821e90c660 ssh
3183 1 2 0 8020000 fffffe8229af4260 sh
2214 1 2 0 8020000 fffffe821f232000 ssh
3259 1 2 0 8020000 fffffe8223563720 sh
2092 1 2 0 8020000 fffffe822220f580 ssh
2331 1 2 0 8020000 fffffe8223563300 sh
2750 1 2 0 8020000 fffffe8226888000 ssh
1921 1 2 0 8020000 fffffe821bf22aa0 sh
2636 1 2 0 8020000 fffffe822d818160 ssh
3223 1 2 0 8020000 fffffe821d443100 sh
2773 1 2 0 8020000 fffffe8226888420 ssh
2150 1 2 0 8020000 fffffe8223049320 sh
2310 1 2 0 8020000 fffffe8222b99560 ssh
1587 1 2 0 8020000 fffffe8220d110e0 sh
2901 1 2 0 8020000 fffffe822188a180 ssh
2627 1 2 0 8020000 fffffe821c4bb900 sh
2060 1 2 0 8020000 fffffe821f36a460 ssh
2599 1 2 0 8020000 fffffe821bf22260 sh
2422 1 2 0 8020000 fffffe821f8965a0 ssh
2164 1 2 0 8020000 fffffe821d443940 sh
2022 1 2 0 8020000 fffffe821f556960 ssh
2000 1 2 0 8020000 fffffe821a542920 sh
1593 1 2 0 8020000 fffffe822d481a40 bash
1914 1 2 0 8020000 fffffe821f5d8ba0 sshd
2167 1 2 0 8020000 fffffe822037f5c0 sshd
1821 1 2 0 8020000 fffffe8220d11500 sshd
1590 1 2 0 8020000 fffffe822dcff120 sshd
1041 1 2 0 8020000 fffffe821c708940 bash
594 1 2 0 8020000 fffffe822037f9e0 bash
895 1 2 0 8020000 fffffe822708eb60 sshd
1533 1 2 0 8020000 fffffe8227069b20 sshd
704 1 3 6 8020000 fffffe821d362980 conserver tstile
564 1 3 1 8020000 fffffe8229b924c0 conserver tstile
789 1 2 0 8020000 fffffe821f232840 conserver
788 > 1 7 0 8020000 fffffe8220b8e360 conserver
275 1 3 5 8020000 fffffe8220b8e780 conserver tstile
772 1 2 0 8020000 fffffe8221256700 conserver
878 1 3 2 8020000 fffffe8221e57600 conserver tstile
136 1 3 3 8020000 fffffe8222569900 conserver tstile
773 1 3 1 8020000 fffffe8225709840 conserver tstile
643 1 2 0 8020000 fffffe82251ce360 conserver
162 1 2 0 8020000 fffffe822dd620e0 getty
861 1 2 0 8020000 fffffe82251ce780 conserver
575 1 2 0 8020000 fffffe82251ceba0 conserver
892 1 2 0 8020000 fffffe8224b0c340 cron
570 1 2 0 8020000 fffffe8224b0cb80 python2.7
107 1 2 0 8020000 fffffe822d7de5c0 inetd
740 1 2 0 8020000 fffffe822cf91a00 nginx
97 1 2 0 8020000 fffffe822d4cd600 nginx
677 1 2 0 8020000 fffffe822d481200 qmgr
479 1 2 0 8020000 fffffe822c312a60 pickup
760 1 2 0 8020000 fffffe822c1e7240 master
436 1 2 0 8020000 fffffe822d818580 sshd
474 1 2 0 8020000 fffffe822d92e5a0 tac_plus
428 1 2 0 8020000 fffffe822cf915e0 powerd
431 1 2 0 8020000 fffffe822d8189a0 ntpd
342 4 2 4 8020000 fffffe822c1e7a80 dhcpd
342 3 2 0 8020000 fffffe822c312220 dhcpd
342 2 2 1 8020000 fffffe822c312640 dhcpd
342 1 2 3 8020000 fffffe822d481620 dhcpd
83 > 1 7 6 8020000 fffffe822cf911c0 syslogd
1 1 2 0 8020000 fffffe810f4fa900 init
0 123 3 7 200 fffffe822d4cd1e0 npfgc-0 xchicv
0 122 3 7 200 fffffe822d4cda20 ipmi0 ipmi0
0 121 3 0 200 fffffe822dd10940 physiod physiod
0 120 3 3 200 fffffe822dcff960 aiodoned aiodoned
0 119 3 5 200 fffffe822dd10100 ioflush biowait
0 118 3 0 200 fffffe822dd10520 pgdaemon pgdaemon
0 115 3 0 280 fffffe822dd62500 spkr1 bellcv
0 114 3 3 280 fffffe822dd62920 audiomix play
0 113 3 3 280 fffffe810ed3b2a0 audiorec record
0 112 3 0 200 fffffe810f4fa0c0 atapibus0 sccomp
0 110 3 6 200 fffffe810f2e9b60 usb1 usbevt
0 109 3 7 200 fffffe810f2c3b20 usb3 usbevt
0 108 3 6 200 fffffe810f20d2c0 usb5 usbevt
0 107 3 0 200 fffffe810f2da300 usb0 usbevt
0 106 3 0 200 fffffe810f2c32e0 usb4 usbevt
0 105 3 0 200 fffffe810f4fa4e0 usb2 usbevt
0 104 3 0 200 fffffe810f4a30a0 rt_free rt_free
0 103 3 0 200 fffffe810f4a34c0 unpgc unpgc
0 102 3 7 200 fffffe810f4a38e0 key_timehandler key_timehandler
0 101 3 7 200 fffffe810f45c080 icmp6_wqinput/7 icmp6_wqinput
0 100 3 6 200 fffffe810f45c4a0 icmp6_wqinput/6 icmp6_wqinput
0 99 3 5 200 fffffe810f45c8c0 icmp6_wqinput/5 icmp6_wqinput
0 98 3 4 200 fffffe810f4b5060 icmp6_wqinput/4 icmp6_wqinput
0 97 3 3 200 fffffe810f4b5480 icmp6_wqinput/3 icmp6_wqinput
0 96 3 2 200 fffffe810f4b58a0 icmp6_wqinput/2 icmp6_wqinput
0 95 3 1 200 fffffe810f396040 icmp6_wqinput/1 icmp6_wqinput
0 94 3 0 200 fffffe810f396460 icmp6_wqinput/0 icmp6_wqinput
0 93 3 2 200 fffffe810f396880 nd6_timer nd6_timer
0 92 3 7 200 fffffe810f367020 icmp_wqinput/7 icmp_wqinput
0 91 3 6 200 fffffe810f367440 icmp_wqinput/6 icmp_wqinput
0 90 3 5 200 fffffe810f367860 icmp_wqinput/5 icmp_wqinput
0 89 3 4 200 fffffe810f378000 icmp_wqinput/4 icmp_wqinput
0 88 3 3 200 fffffe810f378420 icmp_wqinput/3 icmp_wqinput
0 87 3 2 200 fffffe810f378840 icmp_wqinput/2 icmp_wqinput
0 86 3 1 200 fffffe810f329360 icmp_wqinput/1 icmp_wqinput
0 85 3 0 200 fffffe810f329780 icmp_wqinput/0 icmp_wqinput
0 84 3 0 200 fffffe810f2da720 rt_timer rt_timer
0 83 3 7 200 fffffe810f329ba0 vmem_rehash vmem_rehash
0 82 3 6 200 fffffe810f2f8340 coretemp3 coretemp3
0 81 3 7 200 fffffe810f2f8760 coretemp2 coretemp2
0 80 3 2 200 fffffe810f2f8b80 coretemp1 coretemp1
0 79 3 3 200 fffffe810f2e9320 coretemp0 coretemp0
0 78 3 3 200 fffffe810f2e9740 mfi0 mfi0
0 69 3 5 200 fffffe810f20d6e0 atabus1 atath
0 68 3 4 200 fffffe810f20db00 atabus0 atath
0 66 3 0 200 fffffe810ed3b6c0 scsibus0 sccomp
0 65 3 0 200 fffffe810ed3bae0 usbtask-dr usbtsk
0 64 3 0 200 fffffe810ecaa280 usbtask-hc usbtsk
0 63 3 0 200 fffffe810ecaa6a0 bnx3 bnx3
0 62 3 0 200 fffffe810ecaaac0 bnx2 bnx2
0 61 3 0 200 fffffe810eb61260 bnx1 bnx1
0 60 3 7 200 fffffe810eb61680 bnx0 bnx0
0 > 59 7 2 200 fffffe810eb61aa0 ipmi
0 58 3 7 200 fffffe810ea7c240 xcall/7 xcall
0 57 1 7 200 fffffe810ea7c660 softser/7
0 56 1 7 200 fffffe810ea7ca80 softclk/7
0 55 1 7 200 fffffe810ea5d220 softbio/7
0 54 1 7 200 fffffe810ea5d640 softnet/7
0 > 53 7 7 201 fffffe810ea5da60 idle/7
0 52 3 6 200 fffffe810ea36200 xcall/6 xcall
0 51 1 6 200 fffffe810ea36620 softser/6
0 50 1 6 200 fffffe810ea36a40 softclk/6
0 49 1 6 200 fffffe810ea0f1e0 softbio/6
0 48 1 6 200 fffffe810ea0f600 softnet/6
0 47 1 6 201 fffffe810ea0fa20 idle/6
0 46 3 5 200 fffffe810e9e01c0 xcall/5 xcall
0 45 1 5 200 fffffe810e9e05e0 softser/5
0 44 1 5 200 fffffe810e9e0a00 softclk/5
0 43 1 5 200 fffffe810e9d11a0 softbio/5
0 42 1 5 200 fffffe810e9d15c0 softnet/5
0 > 41 7 5 201 fffffe810e9d19e0 idle/5
0 40 3 4 200 fffffe810e9a2180 xcall/4 xcall
0 39 1 4 200 fffffe810e9a25a0 softser/4
0 38 1 4 200 fffffe810e9a29c0 softclk/4
0 37 1 4 200 fffffe810e973160 softbio/4
0 36 1 4 200 fffffe810e973580 softnet/4
0 > 35 7 4 201 fffffe810e9739a0 idle/4
0 34 3 3 200 fffffe810e94c140 xcall/3 xcall
0 33 1 3 200 fffffe810e94c560 softser/3
0 32 1 3 200 fffffe810e94c980 softclk/3
0 31 1 3 200 fffffe810e93d120 softbio/3
0 30 1 3 200 fffffe810e93d540 softnet/3
0 > 29 7 3 201 fffffe810e93d960 idle/3
0 28 3 2 200 fffffe810e906100 xcall/2 xcall
0 27 1 2 200 fffffe810e906520 softser/2
0 26 1 2 200 fffffe810e906940 softclk/2
0 25 1 2 200 fffffe810e8ef0e0 softbio/2
0 24 1 2 200 fffffe810e8ef500 softnet/2
0 23 1 2 201 fffffe810e8ef920 idle/2
0 22 3 1 200 fffffe810e8c50c0 xcall/1 xcall
0 21 1 1 200 fffffe810e8c54e0 softser/1
0 20 1 1 200 fffffe810e8c5900 softclk/1
0 19 1 1 200 fffffe810e8b10a0 softbio/1
0 18 1 1 200 fffffe810e8b14c0 softnet/1
0 > 17 7 1 201 fffffe810e8b18e0 idle/1
0 16 3 0 200 fffffe822de92080 lnxsyswq lnxsyswq
0 15 3 0 200 fffffe822de924a0 sysmon smtaskq
0 14 3 2 200 fffffe822de928c0 pmfsuspend pmfsuspend
0 13 3 7 200 fffffe822e2b1060 pmfevent pmfevent
0 12 3 0 200 fffffe822e2b1480 sopendfree sopendfr
0 11 3 2 200 fffffe822e2b18a0 nfssilly nfssilly
0 10 3 2 200 fffffe822f6d0040 cachegc cachegc
0 9 3 3 200 fffffe822f6d0460 vdrain vdrain
0 8 3 0 200 fffffe822f6d0880 modunload mod_unld
0 7 3 0 200 fffffe822f6eb020 xcall/0 xcall
0 > 6 7 0 200 fffffe822f6eb440 softser/0
0 > 5 7 0 200 fffffe822f6eb860 softclk/0
0 4 1 0 200 fffffe822f707000 softbio/0
0 3 1 0 200 fffffe822f707420 softnet/0
0 2 1 0 201 fffffe822f707840 idle/0
0 1 3 4 200 ffffffff81481b20 swapper uvm
crash>
Backtraces for all the lwps which have non-zero "WAIT" strings:
crash> trace/a fffffe8220c58540
trace: pid 7052 lid 1 at 0xffff80009ea3d8f0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x460
VOP_LOCK() at VOP_LOCK+0x4d
vn_lock() at vn_lock+0x90
namei_tryemulroot() at namei_tryemulroot+0x193
namei() at namei+0x29
vn_open() at vn_open+0x92
do_open() at do_open+0x112
do_sys_openat() at do_sys_openat+0x68
sys_open() at sys_open+0x24
syscall() at syscall+0x1ed
--- syscall (number 5) ---
70efab63e2ca:
crash>
crash> trace/a fffffe821d362560
trace: pid 8587 lid 1 at 0xffff80009edb8d20
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
fork1() at fork1+0x774
sys___vfork14() at sys___vfork14+0x35
syscall() at syscall+0x1ed
--- syscall (number 282) ---
70efab68204c:
crash>
crash> trace/a fffffe821faa60c0
trace: pid 9187 lid 1 at 0xffff80009ed4e890
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_wait() at xc_wait+0x7c
pserialize_perform() at pserialize_perform+0xdf
fstrans_setstate() at fstrans_setstate+0x42
genfs_suspendctl() at genfs_suspendctl+0x57
VFS_SUSPENDCTL() at VFS_SUSPENDCTL+0x1f
vfs_suspend() at vfs_suspend+0x5b
vrevoke_suspend_next() at vrevoke_suspend_next+0x2a
vrevoke() at vrevoke+0x2b
genfs_revoke() at genfs_revoke+0x13
VOP_REVOKE() at VOP_REVOKE+0x54
pty_grant_slave() at pty_grant_slave+0xa8
ptmioctl() at ptmioctl+0x90
cdev_ioctl() at cdev_ioctl+0x88
VOP_IOCTL() at VOP_IOCTL+0x3b
vn_ioctl() at vn_ioctl+0xa6
sys_ioctl() at sys_ioctl+0x101
syscall() at syscall+0x1ed
--- syscall (number 54) ---
6f9ce08feeaa:
crash>
crash> trace/a fffffe82045da540
trace: pid 3794 lid 1 at 0xffff80009f6c6be0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x74d
VOP_LOCK() at VOP_LOCK+0x4d
vn_lock() at vn_lock+0x90
vn_write() at vn_write+0x9b
dofilewrite() at dofilewrite+0x97
sys_write() at sys_write+0x5f
syscall() at syscall+0x1ed
--- syscall (number 4) ---
73b7cfe3e02a:
crash>
crash> trace/a fffffe8216d35140
trace: pid 2853 lid 2 at 0xffff80009f6eaab0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x460
VOP_POLL() at VOP_POLL+0x37
sel_do_scan() at sel_do_scan+0x324
selcommon.part.2() at selcommon.part.2+0x18b
sys___select50() at sys___select50+0x6f
syscall() at syscall+0x1ed
--- syscall (number 417) ---
7b328823e10a:
crash>
crash> trace/a fffffe82214bf2c0
trace: pid 2853 lid 1 at 0xffff80009ef1ddb0
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
fd_close() at fd_close+0x3ba
sys_close() at sys_close+0x20
syscall() at syscall+0x1ed
--- syscall (number 6) ---
7b328823e4ea:
crash>
crash> trace/a fffffe8225709000
trace: pid 8327 lid 2 at 0xffff80009e90eab0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x74d
VOP_POLL() at VOP_POLL+0x37
sel_do_scan() at sel_do_scan+0x324
selcommon.part.2() at selcommon.part.2+0x18b
sys___select50() at sys___select50+0x6f
syscall() at syscall+0x1ed
--- syscall (number 417) ---
7aeb6723e10a:
crash>
crash> trace/a fffffe820df34480
trace: pid 8327 lid 1 at 0xffff80009f668db0
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
fd_close() at fd_close+0x3ba
sys_close() at sys_close+0x20
syscall() at syscall+0x1ed
--- syscall (number 6) ---
7aeb6723e4ea:
crash>
crash> trace/a fffffe8207fd0360
trace: pid 5687 lid 2 at 0xffff80009f5f9ab0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x74d
VOP_POLL() at VOP_POLL+0x37
sel_do_scan() at sel_do_scan+0x324
selcommon.part.2() at selcommon.part.2+0x18b
sys___select50() at sys___select50+0x6f
syscall() at syscall+0x1ed
--- syscall (number 417) ---
7d665343e10a:
crash>
crash> trace/a fffffe822346f080
trace: pid 5687 lid 1 at 0xffff80009efd4db0
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
fd_close() at fd_close+0x3ba
sys_close() at sys_close+0x20
syscall() at syscall+0x1ed
--- syscall (number 6) ---
7d665343e4ea:
crash>
crash> trace/a fffffe821d987a40
trace: pid 9671 lid 2 at 0xffff80009ee64ab0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x74d
VOP_POLL() at VOP_POLL+0x37
sel_do_scan() at sel_do_scan+0x324
selcommon.part.2() at selcommon.part.2+0x18b
sys___select50() at sys___select50+0x6f
syscall() at syscall+0x1ed
--- syscall (number 417) ---
7ea7fc43e10a:
crash>
crash> trace/a fffffe821e90c240
trace: pid 9671 lid 1 at 0xffff80009ee94db0
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
fd_close() at fd_close+0x3ba
sys_close() at sys_close+0x20
syscall() at syscall+0x1ed
--- syscall (number 6) ---
7ea7fc43e4ea:
crash>
crash> trace/a fffffe821d362980
trace: pid 704 lid 1 at 0xffff80009edaebc0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
rw_enter() at rw_enter+0x49d
genfs_lock() at genfs_lock+0x3c
VOP_LOCK() at VOP_LOCK+0x67
vn_lock() at vn_lock+0x90
vn_write() at vn_write+0x9b
dofilewrite() at dofilewrite+0x97
sys_write() at sys_write+0x5f
syscall() at syscall+0x1ed
--- syscall (number 4) ---
764fb503e02a:
crash>
crash> trace/a fffffe8229b924c0
trace: pid 564 lid 1 at 0xffff80009e9c3ab0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x74d
VOP_POLL() at VOP_POLL+0x37
sel_do_scan() at sel_do_scan+0x324
selcommon.part.2() at selcommon.part.2+0x18b
sys___select50() at sys___select50+0x6f
syscall() at syscall+0x1ed
--- syscall (number 417) ---
764fb503e10a:
crash>
crash> trace/a fffffe8220b8e780
trace: pid 275 lid 1 at 0xffff80009ebf8bc0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
rw_enter() at rw_enter+0x49d
genfs_lock() at genfs_lock+0x3c
VOP_LOCK() at VOP_LOCK+0x67
vn_lock() at vn_lock+0x90
vn_write() at vn_write+0x9b
dofilewrite() at dofilewrite+0x97
sys_write() at sys_write+0x5f
syscall() at syscall+0x1ed
--- syscall (number 4) ---
764fb503e02a:
crash>
crash> trace/a fffffe8221e57600
trace: pid 878 lid 1 at 0xffff80009eae8ab0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x74d
VOP_POLL() at VOP_POLL+0x37
sel_do_scan() at sel_do_scan+0x324
selcommon.part.2() at selcommon.part.2+0x18b
sys___select50() at sys___select50+0x6f
syscall() at syscall+0x1ed
--- syscall (number 417) ---
764fb503e10a:
crash>
crash> trace/a fffffe8222569900
trace: pid 136 lid 1 at 0xffff80009e9cdab0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x74d
VOP_POLL() at VOP_POLL+0x37
sel_do_scan() at sel_do_scan+0x324
selcommon.part.2() at selcommon.part.2+0x18b
sys___select50() at sys___select50+0x6f
syscall() at syscall+0x1ed
--- syscall (number 417) ---
764fb503e10a:
crash>
crash> trace/a fffffe8225709840
trace: pid 773 lid 1 at 0xffff80009e904ab0
sleepq_block() at sleepq_block+0x97
turnstile_block() at turnstile_block+0x3e5
mutex_enter() at mutex_enter+0x36d
fstrans_start() at fstrans_start+0x74d
VOP_POLL() at VOP_POLL+0x37
sel_do_scan() at sel_do_scan+0x324
selcommon.part.2() at selcommon.part.2+0x18b
sys___select50() at sys___select50+0x6f
syscall() at syscall+0x1ed
--- syscall (number 417) ---
764fb503e10a:
crash>
crash> trace/a fffffe822d4cd1e0
trace: pid 0 lid 123 at 0xffff80009dee1da0
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_broadcast() at xc_broadcast+0x17a
pserialize_perform() at pserialize_perform+0xd7
npf_conn_gc() at npf_conn_gc+0x1b5
npf_worker() at npf_worker+0x4b
crash>
crash> trace/a fffffe822d4cda20
trace: pid 0 lid 122 at 0xffff80009dec3e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe822dd10940
trace: pid 0 lid 121 at 0xffff80009ddc4e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe822dcff960
trace: pid 0 lid 120 at 0xffff80009dceee60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe822dd10100
trace: pid 0 lid 119 at 0xffff80009dce9bf0
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
biowait() at biowait+0x5d
wapbl_buffered_flush() at wapbl_buffered_flush+0xc6
wapbl_buffered_write() at wapbl_buffered_write+0x69
wapbl_circ_write() at wapbl_circ_write+0xa7
wapbl_flush() at wapbl_flush+0x354
ffs_sync() at ffs_sync+0x221
VFS_SYNC() at VFS_SYNC+0x1c
sched_sync() at sched_sync+0x91
crash>
crash> trace/a fffffe822dd10520
trace: pid 0 lid 118 at 0xffff80009dce4d80
sleepq_block() at sleepq_block+0x97
mtsleep() at mtsleep+0x102
uvm_pageout() at uvm_pageout+0x62e
crash>
crash> trace/a fffffe822dd62500
trace: pid 0 lid 115 at 0xffff80009dc8ee60
sleepq_block() at sleepq_block+0x97
cv_wait_sig() at cv_wait_sig+0x12b
bell_thread() at bell_thread+0x5f
crash>
crash> trace/a fffffe822dd62920
trace: pid 0 lid 114 at 0xffff80009db5fe80
sleepq_block() at sleepq_block+0x97
cv_wait_sig() at cv_wait_sig+0x12b
audio_play_thread() at audio_play_thread+0x55
crash>
crash> trace/a fffffe810ed3b2a0
trace: pid 0 lid 113 at 0xffff80009dc89e90
sleepq_block() at sleepq_block+0x97
cv_wait_sig() at cv_wait_sig+0x12b
audio_rec_thread() at audio_rec_thread+0x3a
crash>
crash> trace/a fffffe810f4fa0c0
trace: pid 0 lid 112 at 0xffff80009dc32e80
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
scsipi_completion_thread() at scsipi_completion_thread+0xf5
crash>
crash> trace/a fffffe810f2e9b60
trace: pid 0 lid 110 at 0xffff80009dc27e80
sleepq_block() at sleepq_block+0x106
cv_timedwait() at cv_timedwait+0x131
usb_event_thread() at usb_event_thread+0xa2
crash>
crash> trace/a fffffe810f2c3b20
trace: pid 0 lid 109 at 0xffff80009dc22e80
sleepq_block() at sleepq_block+0x106
cv_timedwait() at cv_timedwait+0x131
usb_event_thread() at usb_event_thread+0xa2
crash>
crash> trace/a fffffe810f20d2c0
trace: pid 0 lid 108 at 0xffff80009db64e80
sleepq_block() at sleepq_block+0x106
cv_timedwait() at cv_timedwait+0x131
usb_event_thread() at usb_event_thread+0xa2
crash>
crash> trace/a fffffe810f2da300
trace: pid 0 lid 107 at 0xffff80009dc1de80
sleepq_block() at sleepq_block+0x106
cv_timedwait() at cv_timedwait+0x131
usb_event_thread() at usb_event_thread+0xa2
crash>
crash> trace/a fffffe810f2c32e0
trace: pid 0 lid 106 at 0xffff80009dc18e80
sleepq_block() at sleepq_block+0x106
cv_timedwait() at cv_timedwait+0x131
usb_event_thread() at usb_event_thread+0xa2
crash>
crash> trace/a fffffe810f4fa4e0
trace: pid 0 lid 105 at 0xffff80009dc13e80
sleepq_block() at sleepq_block+0x106
cv_timedwait() at cv_timedwait+0x131
usb_event_thread() at usb_event_thread+0xa2
crash>
crash> trace/a fffffe810f4a30a0
trace: pid 0 lid 104 at 0xffff80009dbfce60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f4a34c0
trace: pid 0 lid 103 at 0xffff80009dbf7e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
unp_thread() at unp_thread+0x700
crash>
crash> trace/a fffffe810f4a38e0
trace: pid 0 lid 102 at 0xffff80009dbf2e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f45c080
trace: pid 0 lid 101 at 0xffff80009dbede60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f45c4a0
trace: pid 0 lid 100 at 0xffff80009dbe8e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f45c8c0
trace: pid 0 lid 99 at 0xffff80009dbe3e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f4b5060
trace: pid 0 lid 98 at 0xffff80009dbdee60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f4b5480
trace: pid 0 lid 97 at 0xffff80009dbd9e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f4b58a0
trace: pid 0 lid 96 at 0xffff80009dbd4e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f396040
trace: pid 0 lid 95 at 0xffff80009dbcfe60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f396460
trace: pid 0 lid 94 at 0xffff80009dbcae60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f396880
trace: pid 0 lid 93 at 0xffff80009dbc5e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f367020
trace: pid 0 lid 92 at 0xffff80009dbc0e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f367440
trace: pid 0 lid 91 at 0xffff80009dbbbe60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f367860
trace: pid 0 lid 90 at 0xffff80009dbb6e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f378000
trace: pid 0 lid 89 at 0xffff80009dbb1e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f378420
trace: pid 0 lid 88 at 0xffff80009dbace60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f378840
trace: pid 0 lid 87 at 0xffff80009dba7e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f329360
trace: pid 0 lid 86 at 0xffff80009dba2e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f329780
trace: pid 0 lid 85 at 0xffff80009db6ee60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f2da720
trace: pid 0 lid 84 at 0xffff80009db9de60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f329ba0
trace: pid 0 lid 83 at 0xffff80009db98e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f2f8340
trace: pid 0 lid 82 at 0xffff80009db91e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f2f8760
trace: pid 0 lid 81 at 0xffff80009db8ce60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f2f8b80
trace: pid 0 lid 80 at 0xffff80009db87e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f2e9320
trace: pid 0 lid 79 at 0xffff80009db82e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f2e9740
trace: pid 0 lid 78 at 0xffff80009db7de60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810f20d6e0
trace: pid 0 lid 69 at 0xffff80009d349eb0
sleepq_block() at sleepq_block+0x97
atabus_thread() at atabus_thread+0xcc
crash>
crash> trace/a fffffe810f20db00
trace: pid 0 lid 68 at 0xffff80009d344eb0
sleepq_block() at sleepq_block+0x97
atabus_thread() at atabus_thread+0xcc
crash>
crash> trace/a fffffe810ed3b6c0
trace: pid 0 lid 66 at 0xffff80009d0ffe80
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
scsipi_completion_thread() at scsipi_completion_thread+0xf5
crash>
crash> trace/a fffffe810ed3bae0
trace: pid 0 lid 65 at 0xffff800099072e70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
usb_task_thread() at usb_task_thread+0x8a
crash>
crash> trace/a fffffe810ecaa280
trace: pid 0 lid 64 at 0xffff80009906de70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
usb_task_thread() at usb_task_thread+0x8a
crash>
crash> trace/a fffffe810ecaa6a0
trace: pid 0 lid 63 at 0xffff800099056e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810ecaaac0
trace: pid 0 lid 62 at 0xffff800096a41e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810eb61260
trace: pid 0 lid 61 at 0xffff80009442ae60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810eb61680
trace: pid 0 lid 60 at 0xffff800091e15e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe810eb61aa0
trace: pid 0 lid 59 at 0xffff80008f7f8e30
sleepq_block() at sleepq_block+0x1c5
cv_timedwait() at cv_timedwait+0x131
ipmi_thread() at ipmi_thread+0x2f4
crash>
crash> trace/a fffffe810ea7c240
trace: pid 0 lid 58 at 0xffff80008f7ede70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_thread() at xc_thread+0x5a
crash>
crash> trace/a fffffe810ea36200
trace: pid 0 lid 52 at 0xffff80008f7c7e70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_thread() at xc_thread+0x5a
crash>
crash> trace/a fffffe810e9e01c0
trace: pid 0 lid 46 at 0xffff80008f7a1e70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_thread() at xc_thread+0x5a
crash>
crash> trace/a fffffe810e9a2180
trace: pid 0 lid 40 at 0xffff80008f77be70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_thread() at xc_thread+0x5a
crash>
crash> trace/a fffffe810e94c140
trace: pid 0 lid 34 at 0xffff80008f755e70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_thread() at xc_thread+0x5a
crash>
crash> trace/a fffffe810e906100
trace: pid 0 lid 28 at 0xffff80008f72fe70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_thread() at xc_thread+0x5a
crash>
crash> trace/a fffffe810e8c50c0
trace: pid 0 lid 22 at 0xffff80008f706e70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_thread() at xc_thread+0x5a
crash>
crash> trace/a fffffe822de92080
trace: pid 0 lid 16 at 0xffff80008f6cde60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe822de924a0
trace: pid 0 lid 15 at 0xffff80008f6c8e90
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
sysmon_task_queue_thread() at sysmon_task_queue_thread+0x81
crash>
crash> trace/a fffffe822de928c0
trace: pid 0 lid 14 at 0xffff80008f6c3e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe822e2b1060
trace: pid 0 lid 13 at 0xffff80008f6bee60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe822e2b1480
trace: pid 0 lid 12 at 0xffff80008f6b9e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
sopendfree_thread() at sopendfree_thread+0x1da
crash>
crash> trace/a fffffe822e2b18a0
trace: pid 0 lid 11 at 0xffff80008f6b4e60
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
workqueue_worker() at workqueue_worker+0x8f
crash>
crash> trace/a fffffe822f6d0040
trace: pid 0 lid 10 at 0xffff80008f6afe80
sleepq_block() at sleepq_block+0x106
kpause() at kpause+0xea
cache_thread() at cache_thread+0x30
crash>
crash> trace/a fffffe822f6d0460
trace: pid 0 lid 9 at 0xffff80008f6aae30
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
vdrain_thread() at vdrain_thread+0x3f1
crash>
crash> trace/a fffffe822f6d0880
trace: pid 0 lid 8 at 0xffff80008f6a5e80
sleepq_block() at sleepq_block+0x97
cv_timedwait() at cv_timedwait+0x131
module_thread() at module_thread+0xe9
crash>
crash> trace/a fffffe822f6eb020
trace: pid 0 lid 7 at 0xffff80008f6a0e70
sleepq_block() at sleepq_block+0x97
cv_wait() at cv_wait+0x127
xc_thread() at xc_thread+0x5a
crash>
crash> trace/a ffffffff81481b20
trace: pid 0 lid 1 at 0xffffffff817dde20
sleepq_block() at sleepq_block+0x106
kpause() at kpause+0xea
uvm_scheduler() at uvm_scheduler+0x69
sysctl_alloc() at sysctl_alloc
crash>
------------------------------
(gdb) kvm proc 0xfffffe8220c58540
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8220c58540)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8220c58540)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8098b761 in turnstile_block (ts=<optimized out>,
ts@entry=0xfffffe82123900c8, q=q@entry=1,
obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>,
sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fec40 in fstrans_alloc_lwp_info (mp=0xfffffe822dd7a008)
at /usr/src/sys/kern/vfs_trans.c:274
#5 fstrans_get_lwp_info (do_alloc=true, mp=0xfffffe822dd7a008)
at /usr/src/sys/kern/vfs_trans.c:331
#6 _fstrans_start (wait=1, lock_type=FSTRANS_SHARED, mp=<optimized out>)
at /usr/src/sys/kern/vfs_trans.c:375
#7 fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
#8 0xffffffff80a0ed1b in vop_pre (vp=0xfffffe810f2f2930,
vp=0xfffffe810f2f2930, op=FST_YES, mpsafe=<synthetic pointer>,
mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#9 VOP_LOCK (vp=vp@entry=0xfffffe810f2f2930, flags=flags@entry=131074)
at /usr/src/sys/kern/vnode_if.c:1281
#10 0xffffffff80a05846 in vn_lock (vp=vp@entry=0xfffffe810f2f2930,
flags=flags@entry=131074) at /usr/src/sys/kern/vfs_vnops.c:1036
#11 0xffffffff809ef7c3 in namei_start (startdir_ret=0xffff80009ea3db28,
isnfsd=0, state=0xffff80009ea3dbd0) at /usr/src/sys/kern/vfs_lookup.c:681
#12 namei_oneroot (isnfsd=0, inhibitmagic=0, neverfollow=0,
state=0xffff80009ea3dbd0) at /usr/src/sys/kern/vfs_lookup.c:1156
#13 namei_tryemulroot (state=state@entry=0xffff80009ea3dbd0,
neverfollow=neverfollow@entry=0, inhibitmagic=inhibitmagic@entry=0,
isnfsd=isnfsd@entry=0) at /usr/src/sys/kern/vfs_lookup.c:1510
#14 0xffffffff809f1905 in namei (ndp=ndp@entry=0xffff80009ea3dd58)
at /usr/src/sys/kern/vfs_lookup.c:1546
#15 0xffffffff80a061fd in vn_open (ndp=ndp@entry=0xffff80009ea3dd58,
fmode=fmode@entry=4194305, cmode=cmode@entry=3116)
at /usr/src/sys/kern/vfs_vnops.c:175
#16 0xffffffff809fb7c4 in do_open (l=l@entry=0xfffffe8220c58540, dvp=0x0,
pb=<optimized out>, open_flags=open_flags@entry=4194304,
open_mode=open_mode@entry=774778414, fd=fd@entry=0xffff80009ea3de7c)
at /usr/src/sys/kern/vfs_syscalls.c:1576
#17 0xffffffff809fb915 in do_sys_openat (l=0xfffffe8220c58540,
fdat=fdat@entry=-100, path=<optimized out>, flags=4194304, mode=774778414,
fd=fd@entry=0xffff80009ea3de7c) at /usr/src/sys/kern/vfs_syscalls.c:1656
#18 0xffffffff809fb9d3 in sys_open (l=<optimized out>, uap=<optimized out>,
retval=0xffff80009ea3deb0) at /usr/src/sys/kern/vfs_syscalls.c:1676
#19 0xffffffff80248dbd in sy_call (rval=0xffff80009ea3deb0,
uap=0xffff80009ea3df00, l=0xfffffe8220c58540,
sy=0xffffffff8147da58 <sysent+120>) at /usr/src/sys/sys/syscallvar.h:65
#20 sy_invoke (code=5, rval=0xffff80009ea3deb0, uap=0xffff80009ea3df00,
l=0xfffffe8220c58540, sy=0xffffffff8147da58 <sysent+120>)
at /usr/src/sys/sys/syscallvar.h:94
#21 syscall (frame=0xffff80009ea3df00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#22 0xffffffff8020768d in handle_syscall ()
#23 0x000070efab549000 in ?? ()
#24 0x0000000000400000 in ?? ()
#25 0x2e2e2e2e2e2e2e2e in ?? ()
#26 0x0000000000000000 in ?? ()
(gdb)
Looking at fstrans_lock:
(gdb) lock 0xffffffff815cb9b0
lock address : 0xffffffff815cb9b0 type : sleep/adaptive
initialized : 0xffffffff809fe6a3
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 10
cpu last held: 6
current lwp : 000000000000000000 last held: 0xfffffe821faa60c0
last locked : 0xffffffff809ffb6d unlocked : 0xffffffff809fecc4
(gdb)
What I don't understand is that "current lwp" is 0x0, even though
this one seems to be held at the moment, and there's lots of
contention for the lock judging by the "shares wanted, exclusive: 10".
"last held" appears to point to
9187 1 3 6 8020000 fffffe821faa60c0 expect xchicv
(gdb) kvm proc 0xfffffe821faa60c0
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821faa60c0)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821faa60c0)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8094afca in cv_wait (
cv=cv@entry=0xffffffff814c5c88 <xc_high_pri+8>,
mtx=mtx@entry=0xffffffff814c5c80 <xc_high_pri>)
at /usr/src/sys/kern/kern_condvar.c:217
#3 0xffffffff809b1293 in xc_wait (where=64992)
at /usr/src/sys/kern/subr_xcall.c:321
#4 0xffffffff809aa122 in pserialize_perform (psz=0xfffffe822fbd20e8)
at /usr/src/sys/kern/subr_pserialize.c:187
#5 0xffffffff809ffb7c in fstrans_setstate (mp=0xfffffe822c993008,
new_state=new_state@entry=FSTRANS_SUSPENDED)
at /usr/src/sys/kern/vfs_trans.c:530
#6 0xffffffff80a17729 in genfs_suspendctl (mp=0xfffffe822c993008,
cmd=<optimized out>) at /usr/src/sys/miscfs/genfs/genfs_vfsops.c:86
#7 0xffffffff809f781e in VFS_SUSPENDCTL (mp=mp@entry=0xfffffe822c993008,
a=a@entry=1) at /usr/src/sys/kern/vfs_subr.c:1478
#8 0xffffffff80a00530 in vfs_suspend (mp=<optimized out>,
mp@entry=0xfffffe822c993008, nowait=nowait@entry=0)
at /usr/src/sys/kern/vfs_trans.c:587
#9 0xffffffff80a01769 in vrevoke_suspend_next (lastmp=<optimized out>,
thismp=0xfffffe822c993008) at /usr/src/sys/kern/vfs_vnode.c:978
#10 0xffffffff80a030ed in vrevoke (vp=0xfffffe8221e36690)
at /usr/src/sys/kern/vfs_vnode.c:1002
#11 0xffffffff80a17caa in genfs_revoke (v=<optimized out>)
at /usr/src/sys/miscfs/genfs/genfs_vnops.c:273
#12 0xffffffff80a0e4f7 in VOP_REVOKE (vp=0xfffffe8221e36690,
flags=flags@entry=1) at /usr/src/sys/kern/vnode_if.c:744
#13 0xffffffff809cfb00 in pty_grant_slave (l=l@entry=0xfffffe821faa60c0,
dev=<optimized out>, mp=<optimized out>) at /usr/src/sys/kern/tty_ptm.c:245
#14 0xffffffff809cfd7b in ptmioctl (dev=<optimized out>, cmd=<optimized out>,
data=0xfffffe822d624008, flag=<optimized out>, l=0xfffffe821faa60c0)
at /usr/src/sys/kern/tty_ptm.c:397
#15 0xffffffff80996a52 in cdev_ioctl (dev=42241, cmd=1208513606,
data=0xfffffe822d624008, flag=3, l=0xfffffe821faa60c0)
at /usr/src/sys/kern/subr_devsw.c:938
#16 0xffffffff80a0e2b8 in VOP_IOCTL (vp=vp@entry=0xfffffe822caabd20,
command=command@entry=1208513606, data=data@entry=0xfffffe822d624008,
fflag=<optimized out>, cred=<optimized out>)
at /usr/src/sys/kern/vnode_if.c:610
#17 0xffffffff80a05e1e in vn_ioctl (fp=0xfffffe821d011a40, com=1208513606,
data=0xfffffe822d624008) at /usr/src/sys/kern/vfs_vnops.c:765
#18 0xffffffff809b4e81 in sys_ioctl (l=<optimized out>,
uap=0xffff80009ed4ef00, retval=<optimized out>)
at /usr/src/sys/kern/sys_generic.c:671
#19 0xffffffff80248dbd in sy_call (rval=0xffff80009ed4eeb0,
uap=0xffff80009ed4ef00, l=0xfffffe821faa60c0,
sy=0xffffffff8147def0 <sysent+1296>) at /usr/src/sys/sys/syscallvar.h:65
#20 sy_invoke (code=54, rval=0xffff80009ed4eeb0, uap=0xffff80009ed4ef00,
l=0xfffffe821faa60c0, sy=0xffffffff8147def0 <sysent+1296>)
at /usr/src/sys/sys/syscallvar.h:94
#21 syscall (frame=0xffff80009ed4ef00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#22 0xffffffff8020768d in handle_syscall ()
#23 0x0000000000000003 in ?? ()
#24 0x0000000048087446 in ?? ()
#25 0x00007f7fff318360 in ?? ()
#26 0x0000000000000000 in ?? ()
(gdb)
The xc_high_pri looks like this:
(gdb) lock 0xffffffff814c5c80
lock address : 0xffffffff814c5c80 type : sleep/adaptive
initialized : 0xffffffff809b1053
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 0
cpu last held: 7
current lwp : 000000000000000000 last held: 000000000000000000
last locked : 0xffffffff809b17d1 unlocked : 0xffffffff8094afb7
(gdb)
but that doesn't appear to lead anywhere.
------------------------------
Decode of the stack backtraces of the non-kernel lwps using gdb:
(gdb) kvm proc 0xfffffe8220c58540
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8220c58540)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8220c58540)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8098b761 in turnstile_block (ts=<optimized out>,
ts@entry=0xfffffe82123900c8, q=q@entry=1,
obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>,
sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fec40 in fstrans_alloc_lwp_info (mp=0xfffffe822dd7a008)
at /usr/src/sys/kern/vfs_trans.c:274
#5 fstrans_get_lwp_info (do_alloc=true, mp=0xfffffe822dd7a008)
at /usr/src/sys/kern/vfs_trans.c:331
#6 _fstrans_start (wait=1, lock_type=FSTRANS_SHARED, mp=<optimized out>)
at /usr/src/sys/kern/vfs_trans.c:375
#7 fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
#8 0xffffffff80a0ed1b in vop_pre (vp=0xfffffe810f2f2930,
vp=0xfffffe810f2f2930, op=FST_YES, mpsafe=<synthetic pointer>,
mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#9 VOP_LOCK (vp=vp@entry=0xfffffe810f2f2930, flags=flags@entry=131074)
at /usr/src/sys/kern/vnode_if.c:1281
#10 0xffffffff80a05846 in vn_lock (vp=vp@entry=0xfffffe810f2f2930,
flags=flags@entry=131074) at /usr/src/sys/kern/vfs_vnops.c:1036
#11 0xffffffff809ef7c3 in namei_start (startdir_ret=0xffff80009ea3db28,
isnfsd=0, state=0xffff80009ea3dbd0) at /usr/src/sys/kern/vfs_lookup.c:681
#12 namei_oneroot (isnfsd=0, inhibitmagic=0, neverfollow=0,
state=0xffff80009ea3dbd0) at /usr/src/sys/kern/vfs_lookup.c:1156
#13 namei_tryemulroot (state=state@entry=0xffff80009ea3dbd0,
neverfollow=neverfollow@entry=0, inhibitmagic=inhibitmagic@entry=0,
isnfsd=isnfsd@entry=0) at /usr/src/sys/kern/vfs_lookup.c:1510
#14 0xffffffff809f1905 in namei (ndp=ndp@entry=0xffff80009ea3dd58)
at /usr/src/sys/kern/vfs_lookup.c:1546
#15 0xffffffff80a061fd in vn_open (ndp=ndp@entry=0xffff80009ea3dd58,
fmode=fmode@entry=4194305, cmode=cmode@entry=3116)
at /usr/src/sys/kern/vfs_vnops.c:175
#16 0xffffffff809fb7c4 in do_open (l=l@entry=0xfffffe8220c58540, dvp=0x0,
pb=<optimized out>, open_flags=open_flags@entry=4194304,
open_mode=open_mode@entry=774778414, fd=fd@entry=0xffff80009ea3de7c)
at /usr/src/sys/kern/vfs_syscalls.c:1576
#17 0xffffffff809fb915 in do_sys_openat (l=0xfffffe8220c58540,
fdat=fdat@entry=-100, path=<optimized out>, flags=4194304, mode=774778414,
fd=fd@entry=0xffff80009ea3de7c) at /usr/src/sys/kern/vfs_syscalls.c:1656
#18 0xffffffff809fb9d3 in sys_open (l=<optimized out>, uap=<optimized out>,
retval=0xffff80009ea3deb0) at /usr/src/sys/kern/vfs_syscalls.c:1676
#19 0xffffffff80248dbd in sy_call (rval=0xffff80009ea3deb0,
uap=0xffff80009ea3df00, l=0xfffffe8220c58540,
sy=0xffffffff8147da58 <sysent+120>) at /usr/src/sys/sys/syscallvar.h:65
#20 sy_invoke (code=5, rval=0xffff80009ea3deb0, uap=0xffff80009ea3df00,
l=0xfffffe8220c58540, sy=0xffffffff8147da58 <sysent+120>)
at /usr/src/sys/sys/syscallvar.h:94
#21 syscall (frame=0xffff80009ea3df00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#22 0xffffffff8020768d in handle_syscall ()
#23 0x000070efab549000 in ?? ()
#24 0x0000000000400000 in ?? ()
#25 0x2e2e2e2e2e2e2e2e in ?? ()
#26 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe821d362560
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821d362560)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821d362560)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8094afca in cv_wait (cv=cv@entry=0xfffffe8210396788,
mtx=0xfffffe822fb97f00) at /usr/src/sys/kern/kern_condvar.c:217
#3 0xffffffff80959f24 in fork1 (l1=0xfffffe821d362560, flags=flags@entry=3,
exitsig=exitsig@entry=20, stack=stack@entry=0x0,
stacksize=stacksize@entry=0, func=func@entry=0x0, arg=arg@entry=0x0,
retval=0xffff80009edb8eb0, rnewprocp=rnewprocp@entry=0x0)
at /usr/src/sys/kern/kern_fork.c:592
#4 0xffffffff8095a424 in sys___vfork14 (l=<optimized out>, v=<optimized out>,
retval=<optimized out>) at /usr/src/sys/kern/kern_fork.c:143
#5 0xffffffff80248dbd in sy_call (rval=0xffff80009edb8eb0,
uap=0xffff80009edb8f00, l=0xfffffe821d362560,
sy=0xffffffff8147f450 <sysent+6768>) at /usr/src/sys/sys/syscallvar.h:65
#6 sy_invoke (code=282, rval=0xffff80009edb8eb0, uap=0xffff80009edb8f00,
l=0xfffffe821d362560, sy=0xffffffff8147f450 <sysent+6768>)
at /usr/src/sys/sys/syscallvar.h:94
#7 syscall (frame=0xffff80009edb8f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#8 0xffffffff8020768d in handle_syscall ()
#9 0x00007f7fff9fa3d8 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe821faa60c0
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821faa60c0)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821faa60c0)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8094afca in cv_wait (
cv=cv@entry=0xffffffff814c5c88 <xc_high_pri+8>,
mtx=mtx@entry=0xffffffff814c5c80 <xc_high_pri>)
at /usr/src/sys/kern/kern_condvar.c:217
#3 0xffffffff809b1293 in xc_wait (where=64992)
at /usr/src/sys/kern/subr_xcall.c:321
#4 0xffffffff809aa122 in pserialize_perform (psz=0xfffffe822fbd20e8)
at /usr/src/sys/kern/subr_pserialize.c:187
#5 0xffffffff809ffb7c in fstrans_setstate (mp=0xfffffe822c993008,
new_state=new_state@entry=FSTRANS_SUSPENDED)
at /usr/src/sys/kern/vfs_trans.c:530
#6 0xffffffff80a17729 in genfs_suspendctl (mp=0xfffffe822c993008,
cmd=<optimized out>) at /usr/src/sys/miscfs/genfs/genfs_vfsops.c:86
#7 0xffffffff809f781e in VFS_SUSPENDCTL (mp=mp@entry=0xfffffe822c993008,
a=a@entry=1) at /usr/src/sys/kern/vfs_subr.c:1478
#8 0xffffffff80a00530 in vfs_suspend (mp=<optimized out>,
mp@entry=0xfffffe822c993008, nowait=nowait@entry=0)
at /usr/src/sys/kern/vfs_trans.c:587
#9 0xffffffff80a01769 in vrevoke_suspend_next (lastmp=<optimized out>,
thismp=0xfffffe822c993008) at /usr/src/sys/kern/vfs_vnode.c:978
#10 0xffffffff80a030ed in vrevoke (vp=0xfffffe8221e36690)
at /usr/src/sys/kern/vfs_vnode.c:1002
#11 0xffffffff80a17caa in genfs_revoke (v=<optimized out>)
at /usr/src/sys/miscfs/genfs/genfs_vnops.c:273
#12 0xffffffff80a0e4f7 in VOP_REVOKE (vp=0xfffffe8221e36690,
flags=flags@entry=1) at /usr/src/sys/kern/vnode_if.c:744
#13 0xffffffff809cfb00 in pty_grant_slave (l=l@entry=0xfffffe821faa60c0,
dev=<optimized out>, mp=<optimized out>) at /usr/src/sys/kern/tty_ptm.c:245
#14 0xffffffff809cfd7b in ptmioctl (dev=<optimized out>, cmd=<optimized out>,
data=0xfffffe822d624008, flag=<optimized out>, l=0xfffffe821faa60c0)
at /usr/src/sys/kern/tty_ptm.c:397
#15 0xffffffff80996a52 in cdev_ioctl (dev=42241, cmd=1208513606,
data=0xfffffe822d624008, flag=3, l=0xfffffe821faa60c0)
at /usr/src/sys/kern/subr_devsw.c:938
#16 0xffffffff80a0e2b8 in VOP_IOCTL (vp=vp@entry=0xfffffe822caabd20,
command=command@entry=1208513606, data=data@entry=0xfffffe822d624008,
fflag=<optimized out>, cred=<optimized out>)
at /usr/src/sys/kern/vnode_if.c:610
#17 0xffffffff80a05e1e in vn_ioctl (fp=0xfffffe821d011a40, com=1208513606,
data=0xfffffe822d624008) at /usr/src/sys/kern/vfs_vnops.c:765
#18 0xffffffff809b4e81 in sys_ioctl (l=<optimized out>,
uap=0xffff80009ed4ef00, retval=<optimized out>)
at /usr/src/sys/kern/sys_generic.c:671
#19 0xffffffff80248dbd in sy_call (rval=0xffff80009ed4eeb0,
uap=0xffff80009ed4ef00, l=0xfffffe821faa60c0,
sy=0xffffffff8147def0 <sysent+1296>) at /usr/src/sys/sys/syscallvar.h:65
#20 sy_invoke (code=54, rval=0xffff80009ed4eeb0, uap=0xffff80009ed4ef00,
l=0xfffffe821faa60c0, sy=0xffffffff8147def0 <sysent+1296>)
at /usr/src/sys/sys/syscallvar.h:94
#21 syscall (frame=0xffff80009ed4ef00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#22 0xffffffff8020768d in handle_syscall ()
#23 0x0000000000000003 in ?? ()
#24 0x0000000048087446 in ?? ()
#25 0x00007f7fff318360 in ?? ()
#26 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe82045da540
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe82045da540)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe82045da540)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8098b761 in turnstile_block (ts=<optimized out>,
ts@entry=0xfffffe82123900c8, q=q@entry=1,
obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>,
sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fef2d in _fstrans_start (wait=1, lock_type=FSTRANS_SHARED,
mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
#5 fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
#6 0xffffffff80a0ed1b in vop_pre (vp=0xfffffe821a9bc7e0,
vp=0xfffffe821a9bc7e0, op=FST_YES, mpsafe=<synthetic pointer>,
mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#7 VOP_LOCK (vp=vp@entry=0xfffffe821a9bc7e0, flags=flags@entry=131074)
at /usr/src/sys/kern/vnode_if.c:1281
#8 0xffffffff80a05846 in vn_lock (vp=vp@entry=0xfffffe821a9bc7e0,
flags=flags@entry=131074) at /usr/src/sys/kern/vfs_vnops.c:1036
#9 0xffffffff80a0601d in vn_write (fp=<optimized out>,
offset=0xfffffe821ed94300, uio=0xffff80009f6c6df0,
cred=0xfffffe8220120e40, flags=1) at /usr/src/sys/kern/vfs_vnops.c:592
#10 0xffffffff809b4c1d in dofilewrite (fd=fd@entry=2, fp=0xfffffe821ed94300,
buf=0x7f7fff0d1230, nbyte=102, offset=<optimized out>,
flags=flags@entry=1, retval=retval@entry=0xffff80009f6c6eb0)
at /usr/src/sys/kern/sys_generic.c:350
#11 0xffffffff809b4d22 in sys_write (l=<optimized out>,
uap=0xffff80009f6c6f00, retval=0xffff80009f6c6eb0)
at /usr/src/sys/kern/sys_generic.c:318
#12 0xffffffff80248dbd in sy_call (rval=0xffff80009f6c6eb0,
uap=0xffff80009f6c6f00, l=0xfffffe82045da540,
sy=0xffffffff8147da40 <sysent+96>) at /usr/src/sys/sys/syscallvar.h:65
#13 sy_invoke (code=4, rval=0xffff80009f6c6eb0, uap=0xffff80009f6c6f00,
l=0xfffffe82045da540, sy=0xffffffff8147da40 <sysent+96>)
at /usr/src/sys/sys/syscallvar.h:94
#14 syscall (frame=0xffff80009f6c6f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#15 0xffffffff8020768d in handle_syscall ()
#16 0x0000000000000002 in ?? ()
#17 0x00007f7fff0d1230 in ?? ()
#18 0x0000000000000066 in ?? ()
#19 0x00007f7fff0d129a in ?? ()
#20 0x0101010101010101 in ?? ()
#21 0x00007f7fff0d128c in ?? ()
#22 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe8216d35140
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8216d35140)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8216d35140)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8098b761 in turnstile_block (ts=<optimized out>, ts@entry=0x0,
q=q@entry=1, obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>,
sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fec40 in fstrans_alloc_lwp_info (mp=0xfffffe822c993008)
at /usr/src/sys/kern/vfs_trans.c:274
#5 fstrans_get_lwp_info (do_alloc=true, mp=0xfffffe822c993008)
at /usr/src/sys/kern/vfs_trans.c:331
#6 _fstrans_start (wait=1, lock_type=FSTRANS_SHARED, mp=<optimized out>)
at /usr/src/sys/kern/vfs_trans.c:375
#7 fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
#8 0xffffffff80a0e399 in vop_pre (vp=0xfffffe82226253f0,
vp=0xfffffe82226253f0, op=FST_YES, mpsafe=<synthetic pointer>,
mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#9 VOP_POLL (vp=0xfffffe82226253f0, events=<optimized out>)
at /usr/src/sys/kern/vnode_if.c:677
#10 0xffffffff809bd7e8 in selscan (retval=0xffff80009f6eaeb0, ni=0, nfd=5,
bits=0xffff80009f6ead48 "\030") at /usr/src/sys/kern/sys_select.c:402
#11 sel_do_scan (op=op@entry=1, fds=fds@entry=0xffff80009f6ead48,
nf=nf@entry=5, ni=ni@entry=4, ts=ts@entry=0x0, mask=mask@entry=0x0,
retval=retval@entry=0xffff80009f6eaeb0)
at /usr/src/sys/kern/sys_select.c:264
#12 0xffffffff809bdb96 in selcommon (retval=retval@entry=0xffff80009f6eaeb0,
nd=5, u_in=0x7b32875fff40, u_ou=0x7b32875fff60, u_ex=0x7b32875fff80,
ts=0x0, mask=mask@entry=0x0) at /usr/src/sys/kern/sys_select.c:362
#13 0xffffffff809bdeab in selcommon (mask=0x0, ts=<optimized out>,
u_ex=<optimized out>, u_ou=<optimized out>, u_in=<optimized out>,
nd=<optimized out>, retval=0xffff80009f6eaeb0)
at /usr/src/sys/kern/sys_select.c:337
#14 sys___select50 (l=<optimized out>, uap=0xffff80009f6eaf00,
retval=0xffff80009f6eaeb0) at /usr/src/sys/kern/sys_select.c:212
#15 0xffffffff80248dbd in sy_call (rval=0xffff80009f6eaeb0,
uap=0xffff80009f6eaf00, l=0xfffffe8216d35140,
sy=0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscallvar.h:65
#16 sy_invoke (code=417, rval=0xffff80009f6eaeb0, uap=0xffff80009f6eaf00,
l=0xfffffe8216d35140, sy=0xffffffff814800f8 <sysent+10008>)
at /usr/src/sys/sys/syscallvar.h:94
#17 syscall (frame=0xffff80009f6eaf00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#18 0xffffffff8020768d in handle_syscall ()
#19 0x0000000000000005 in ?? ()
#20 0x00007b32875fff40 in ?? ()
#21 0x00007b32875fff60 in ?? ()
#22 0x00007b32875fff80 in ?? ()
#23 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe82214bf2c0
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe82214bf2c0)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe82214bf2c0)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8094afca in cv_wait (cv=cv@entry=0xfffffe8221620998,
mtx=mtx@entry=0xfffffe8221620a00) at /usr/src/sys/kern/kern_condvar.c:217
#3 0xffffffff8094e4e5 in fd_close (fd=<optimized out>)
at /usr/src/sys/kern/kern_descrip.c:684
#4 0xffffffff809b3c88 in sys_close (l=0xfffffe82214bf2c0,
uap=0xffff80009ef1df00, retval=<optimized out>)
at /usr/src/sys/kern/sys_descrip.c:488
#5 0xffffffff80248dbd in sy_call (rval=0xffff80009ef1deb0,
uap=0xffff80009ef1df00, l=0xfffffe82214bf2c0,
sy=0xffffffff8147da70 <sysent+144>) at /usr/src/sys/sys/syscallvar.h:65
#6 sy_invoke (code=6, rval=0xffff80009ef1deb0, uap=0xffff80009ef1df00,
l=0xfffffe82214bf2c0, sy=0xffffffff8147da70 <sysent+144>)
at /usr/src/sys/sys/syscallvar.h:94
#7 syscall (frame=0xffff80009ef1df00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#8 0xffffffff8020768d in handle_syscall ()
#9 0x0000000000000004 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe8225709000
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8225709000)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8225709000)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8098b761 in turnstile_block (ts=<optimized out>,
ts@entry=0xfffffe82123900c8, q=q@entry=1,
obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>,
sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fef2d in _fstrans_start (wait=1, lock_type=FSTRANS_SHARED,
mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
#5 fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
#6 0xffffffff80a0e399 in vop_pre (vp=0xfffffe82220a8690,
vp=0xfffffe82220a8690, op=FST_YES, mpsafe=<synthetic pointer>,
mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#7 VOP_POLL (vp=0xfffffe82220a8690, events=<optimized out>)
at /usr/src/sys/kern/vnode_if.c:677
#8 0xffffffff809bd7e8 in selscan (retval=0xffff80009e90eeb0, ni=0, nfd=5,
bits=0xffff80009e90ed48 "\030") at /usr/src/sys/kern/sys_select.c:402
#9 sel_do_scan (op=op@entry=1, fds=fds@entry=0xffff80009e90ed48,
nf=nf@entry=5, ni=ni@entry=4, ts=ts@entry=0x0, mask=mask@entry=0x0,
retval=retval@entry=0xffff80009e90eeb0)
at /usr/src/sys/kern/sys_select.c:264
#10 0xffffffff809bdb96 in selcommon (retval=retval@entry=0xffff80009e90eeb0,
nd=5, u_in=0x7aeb665fff40, u_ou=0x7aeb665fff60, u_ex=0x7aeb665fff80,
ts=0x0, mask=mask@entry=0x0) at /usr/src/sys/kern/sys_select.c:362
#11 0xffffffff809bdeab in selcommon (mask=0x0, ts=<optimized out>,
u_ex=<optimized out>, u_ou=<optimized out>, u_in=<optimized out>,
nd=<optimized out>, retval=0xffff80009e90eeb0)
at /usr/src/sys/kern/sys_select.c:337
#12 sys___select50 (l=<optimized out>, uap=0xffff80009e90ef00,
retval=0xffff80009e90eeb0) at /usr/src/sys/kern/sys_select.c:212
#13 0xffffffff80248dbd in sy_call (rval=0xffff80009e90eeb0,
uap=0xffff80009e90ef00, l=0xfffffe8225709000,
sy=0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscallvar.h:65
#14 sy_invoke (code=417, rval=0xffff80009e90eeb0, uap=0xffff80009e90ef00,
l=0xfffffe8225709000, sy=0xffffffff814800f8 <sysent+10008>)
at /usr/src/sys/sys/syscallvar.h:94
#15 syscall (frame=0xffff80009e90ef00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#16 0xffffffff8020768d in handle_syscall ()
#17 0x0000000000000005 in ?? ()
#18 0x00007aeb665fff40 in ?? ()
#19 0x00007aeb665fff60 in ?? ()
#20 0x00007aeb665fff80 in ?? ()
#21 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe820df34480
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe820df34480)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe820df34480)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8094afca in cv_wait (cv=cv@entry=0xfffffe8207461358,
mtx=mtx@entry=0xfffffe82074613c0) at /usr/src/sys/kern/kern_condvar.c:217
#3 0xffffffff8094e4e5 in fd_close (fd=<optimized out>)
at /usr/src/sys/kern/kern_descrip.c:684
#4 0xffffffff809b3c88 in sys_close (l=0xfffffe820df34480,
uap=0xffff80009f668f00, retval=<optimized out>)
at /usr/src/sys/kern/sys_descrip.c:488
#5 0xffffffff80248dbd in sy_call (rval=0xffff80009f668eb0,
uap=0xffff80009f668f00, l=0xfffffe820df34480,
sy=0xffffffff8147da70 <sysent+144>) at /usr/src/sys/sys/syscallvar.h:65
#6 sy_invoke (code=6, rval=0xffff80009f668eb0, uap=0xffff80009f668f00,
l=0xfffffe820df34480, sy=0xffffffff8147da70 <sysent+144>)
at /usr/src/sys/sys/syscallvar.h:94
#7 syscall (frame=0xffff80009f668f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#8 0xffffffff8020768d in handle_syscall ()
#9 0x0000000000000004 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe8207fd0360
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8207fd0360)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8207fd0360)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8098b761 in turnstile_block (ts=<optimized out>,
ts@entry=0xfffffe82123900c8, q=q@entry=1,
obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>,
sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fef2d in _fstrans_start (wait=1, lock_type=FSTRANS_SHARED,
mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
#5 fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
#6 0xffffffff80a0e399 in vop_pre (vp=0xfffffe822122fa88,
vp=0xfffffe822122fa88, op=FST_YES, mpsafe=<synthetic pointer>,
mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#7 VOP_POLL (vp=0xfffffe822122fa88, events=<optimized out>)
at /usr/src/sys/kern/vnode_if.c:677
#8 0xffffffff809bd7e8 in selscan (retval=0xffff80009f5f9eb0, ni=0, nfd=5,
bits=0xffff80009f5f9d48 "\030") at /usr/src/sys/kern/sys_select.c:402
#9 sel_do_scan (op=op@entry=1, fds=fds@entry=0xffff80009f5f9d48,
nf=nf@entry=5, ni=ni@entry=4, ts=ts@entry=0x0, mask=mask@entry=0x0,
retval=retval@entry=0xffff80009f5f9eb0)
at /usr/src/sys/kern/sys_select.c:264
#10 0xffffffff809bdb96 in selcommon (retval=retval@entry=0xffff80009f5f9eb0,
nd=5, u_in=0x7d66527fff40, u_ou=0x7d66527fff60, u_ex=0x7d66527fff80,
ts=0x0, mask=mask@entry=0x0) at /usr/src/sys/kern/sys_select.c:362
#11 0xffffffff809bdeab in selcommon (mask=0x0, ts=<optimized out>,
u_ex=<optimized out>, u_ou=<optimized out>, u_in=<optimized out>,
nd=<optimized out>, retval=0xffff80009f5f9eb0)
at /usr/src/sys/kern/sys_select.c:337
#12 sys___select50 (l=<optimized out>, uap=0xffff80009f5f9f00,
retval=0xffff80009f5f9eb0) at /usr/src/sys/kern/sys_select.c:212
#13 0xffffffff80248dbd in sy_call (rval=0xffff80009f5f9eb0,
uap=0xffff80009f5f9f00, l=0xfffffe8207fd0360,
sy=0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscallvar.h:65
#14 sy_invoke (code=417, rval=0xffff80009f5f9eb0, uap=0xffff80009f5f9f00,
l=0xfffffe8207fd0360, sy=0xffffffff814800f8 <sysent+10008>)
at /usr/src/sys/sys/syscallvar.h:94
#15 syscall (frame=0xffff80009f5f9f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#16 0xffffffff8020768d in handle_syscall ()
#17 0x0000000000000005 in ?? ()
#18 0x00007d66527fff40 in ?? ()
#19 0x00007d66527fff60 in ?? ()
#20 0x00007d66527fff80 in ?? ()
#21 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe822346f080
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe822346f080)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe822346f080)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8094afca in cv_wait (cv=cv@entry=0xfffffe82243bfc18,
mtx=mtx@entry=0xfffffe82243bfc80) at /usr/src/sys/kern/kern_condvar.c:217
#3 0xffffffff8094e4e5 in fd_close (fd=<optimized out>)
at /usr/src/sys/kern/kern_descrip.c:684
#4 0xffffffff809b3c88 in sys_close (l=0xfffffe822346f080,
uap=0xffff80009efd4f00, retval=<optimized out>)
at /usr/src/sys/kern/sys_descrip.c:488
#5 0xffffffff80248dbd in sy_call (rval=0xffff80009efd4eb0,
uap=0xffff80009efd4f00, l=0xfffffe822346f080,
sy=0xffffffff8147da70 <sysent+144>) at /usr/src/sys/sys/syscallvar.h:65
#6 sy_invoke (code=6, rval=0xffff80009efd4eb0, uap=0xffff80009efd4f00,
l=0xfffffe822346f080, sy=0xffffffff8147da70 <sysent+144>)
at /usr/src/sys/sys/syscallvar.h:94
#7 syscall (frame=0xffff80009efd4f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#8 0xffffffff8020768d in handle_syscall ()
#9 0x0000000000000004 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe821d987a40
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821d987a40)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821d987a40)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8098b761 in turnstile_block (ts=<optimized out>,
ts@entry=0xfffffe82123900c8, q=q@entry=1,
obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>,
sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fef2d in _fstrans_start (wait=1, lock_type=FSTRANS_SHARED,
mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
#5 fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
#6 0xffffffff80a0e399 in vop_pre (vp=0xfffffe820ecde930,
vp=0xfffffe820ecde930, op=FST_YES, mpsafe=<synthetic pointer>,
mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#7 VOP_POLL (vp=0xfffffe820ecde930, events=<optimized out>)
at /usr/src/sys/kern/vnode_if.c:677
#8 0xffffffff809bd7e8 in selscan (retval=0xffff80009ee64eb0, ni=0, nfd=5,
bits=0xffff80009ee64d48 "\030") at /usr/src/sys/kern/sys_select.c:402
#9 sel_do_scan (op=op@entry=1, fds=fds@entry=0xffff80009ee64d48,
nf=nf@entry=5, ni=ni@entry=4, ts=ts@entry=0x0, mask=mask@entry=0x0,
retval=retval@entry=0xffff80009ee64eb0)
at /usr/src/sys/kern/sys_select.c:264
#10 0xffffffff809bdb96 in selcommon (retval=retval@entry=0xffff80009ee64eb0,
nd=5, u_in=0x7ea7fb7fff40, u_ou=0x7ea7fb7fff60, u_ex=0x7ea7fb7fff80,
ts=0x0, mask=mask@entry=0x0) at /usr/src/sys/kern/sys_select.c:362
#11 0xffffffff809bdeab in selcommon (mask=0x0, ts=<optimized out>,
u_ex=<optimized out>, u_ou=<optimized out>, u_in=<optimized out>,
nd=<optimized out>, retval=0xffff80009ee64eb0)
at /usr/src/sys/kern/sys_select.c:337
#12 sys___select50 (l=<optimized out>, uap=0xffff80009ee64f00,
retval=0xffff80009ee64eb0) at /usr/src/sys/kern/sys_select.c:212
#13 0xffffffff80248dbd in sy_call (rval=0xffff80009ee64eb0,
uap=0xffff80009ee64f00, l=0xfffffe821d987a40,
sy=0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscallvar.h:65
#14 sy_invoke (code=417, rval=0xffff80009ee64eb0, uap=0xffff80009ee64f00,
l=0xfffffe821d987a40, sy=0xffffffff814800f8 <sysent+10008>)
at /usr/src/sys/sys/syscallvar.h:94
#15 syscall (frame=0xffff80009ee64f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#16 0xffffffff8020768d in handle_syscall ()
#17 0x0000000000000005 in ?? ()
#18 0x00007ea7fb7fff40 in ?? ()
#19 0x00007ea7fb7fff60 in ?? ()
#20 0x00007ea7fb7fff80 in ?? ()
#21 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe821e90c240
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821e90c240)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821e90c240)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8094afca in cv_wait (cv=cv@entry=0xfffffe822400e918,
mtx=mtx@entry=0xfffffe822400e980) at /usr/src/sys/kern/kern_condvar.c:217
#3 0xffffffff8094e4e5 in fd_close (fd=<optimized out>)
at /usr/src/sys/kern/kern_descrip.c:684
#4 0xffffffff809b3c88 in sys_close (l=0xfffffe821e90c240,
uap=0xffff80009ee94f00, retval=<optimized out>)
at /usr/src/sys/kern/sys_descrip.c:488
#5 0xffffffff80248dbd in sy_call (rval=0xffff80009ee94eb0,
uap=0xffff80009ee94f00, l=0xfffffe821e90c240,
sy=0xffffffff8147da70 <sysent+144>) at /usr/src/sys/sys/syscallvar.h:65
#6 sy_invoke (code=6, rval=0xffff80009ee94eb0, uap=0xffff80009ee94f00,
l=0xfffffe821e90c240, sy=0xffffffff8147da70 <sysent+144>)
at /usr/src/sys/sys/syscallvar.h:94
#7 syscall (frame=0xffff80009ee94f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#8 0xffffffff8020768d in handle_syscall ()
#9 0x0000000000000004 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe821d362980
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821d362980)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821d362980)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8098b761 in turnstile_block (ts=<optimized out>,
ts@entry=0xfffffe821f9db480, q=q@entry=1,
obj=obj@entry=0xfffffe8224a7b130,
sobj=sobj@entry=0xffffffff81482720 <rw_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80977cb3 in rw_vector_enter (rw=0xfffffe8224a7b130, op=RW_WRITER)
at /usr/src/sys/kern/kern_rwlock.c:395
#4 0xffffffff80a17d7e in genfs_lock (v=<optimized out>)
at /usr/src/sys/miscfs/genfs/genfs_vnops.c:343
#5 0xffffffff80a0ed35 in VOP_LOCK (vp=vp@entry=0xfffffe8224a7b000,
flags=flags@entry=131074) at /usr/src/sys/kern/vnode_if.c:1284
#6 0xffffffff80a05846 in vn_lock (vp=vp@entry=0xfffffe8224a7b000,
flags=flags@entry=131074) at /usr/src/sys/kern/vfs_vnops.c:1036
#7 0xffffffff80a0601d in vn_write (fp=<optimized out>,
offset=0xfffffe8225588b00, uio=0xffff80009edaedf0,
cred=0xfffffe822f724f00, flags=1) at /usr/src/sys/kern/vfs_vnops.c:592
#8 0xffffffff809b4c1d in dofilewrite (fd=fd@entry=2, fp=0xfffffe8225588b00,
buf=0x764fb6902000, nbyte=94, offset=<optimized out>, flags=flags@entry=1,
retval=retval@entry=0xffff80009edaeeb0)
at /usr/src/sys/kern/sys_generic.c:350
#9 0xffffffff809b4d22 in sys_write (l=<optimized out>,
uap=0xffff80009edaef00, retval=0xffff80009edaeeb0)
at /usr/src/sys/kern/sys_generic.c:318
#10 0xffffffff80248dbd in sy_call (rval=0xffff80009edaeeb0,
uap=0xffff80009edaef00, l=0xfffffe821d362980,
sy=0xffffffff8147da40 <sysent+96>) at /usr/src/sys/sys/syscallvar.h:65
#11 sy_invoke (code=4, rval=0xffff80009edaeeb0, uap=0xffff80009edaef00,
l=0xfffffe821d362980, sy=0xffffffff8147da40 <sysent+96>)
at /usr/src/sys/sys/syscallvar.h:94
#12 syscall (frame=0xffff80009edaef00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#13 0xffffffff8020768d in handle_syscall ()
#14 0x0000000000000002 in ?? ()
#15 0x0000764fb6902000 in ?? ()
#16 0x000000000000005e in ?? ()
#17 0x0000764fb690205e in ?? ()
#18 0x000000000000001f in ?? ()
#19 0x0000000000000001 in ?? ()
#20 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe8229b924c0
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8229b924c0)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8229b924c0)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8098b761 in turnstile_block (ts=<optimized out>,
ts@entry=0xfffffe82123900c8, q=q@entry=1,
obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>,
sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fef2d in _fstrans_start (wait=1, lock_type=FSTRANS_SHARED,
mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
#5 fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
#6 0xffffffff80a0e399 in vop_pre (vp=0xfffffe821d7d1690,
vp=0xfffffe821d7d1690, op=FST_YES, mpsafe=<synthetic pointer>,
mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#7 VOP_POLL (vp=0xfffffe821d7d1690, events=<optimized out>)
at /usr/src/sys/kern/vnode_if.c:677
#8 0xffffffff809bd7e8 in selscan (retval=0xffff80009e9c3eb0, ni=3639083008,
nfd=48, bits=0xffff80009e9c3d48 "\b@\350\330(")
at /usr/src/sys/kern/sys_select.c:402
#9 sel_do_scan (op=op@entry=1, fds=fds@entry=0xffff80009e9c3d48,
nf=nf@entry=48, ni=ni@entry=8, ts=ts@entry=0xffff80009e9c3e70,
mask=mask@entry=0x0, retval=retval@entry=0xffff80009e9c3eb0)
at /usr/src/sys/kern/sys_select.c:264
#10 0xffffffff809bdb96 in selcommon (retval=retval@entry=0xffff80009e9c3eb0,
nd=48, u_in=0x7f7fff4c5a10, u_ou=0x7f7fff4c5a30, u_ex=0x0,
ts=0xffff80009e9c3e70, mask=mask@entry=0x0)
at /usr/src/sys/kern/sys_select.c:362
#11 0xffffffff809bdeab in selcommon (mask=0x0, ts=<optimized out>,
u_ex=<optimized out>, u_ou=<optimized out>, u_in=<optimized out>,
nd=<optimized out>, retval=0xffff80009e9c3eb0)
at /usr/src/sys/kern/sys_select.c:337
#12 sys___select50 (l=<optimized out>, uap=0xffff80009e9c3f00,
retval=0xffff80009e9c3eb0) at /usr/src/sys/kern/sys_select.c:212
#13 0xffffffff80248dbd in sy_call (rval=0xffff80009e9c3eb0,
uap=0xffff80009e9c3f00, l=0xfffffe8229b924c0,
sy=0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscallvar.h:65
#14 sy_invoke (code=417, rval=0xffff80009e9c3eb0, uap=0xffff80009e9c3f00,
l=0xfffffe8229b924c0, sy=0xffffffff814800f8 <sysent+10008>)
at /usr/src/sys/sys/syscallvar.h:94
#15 syscall (frame=0xffff80009e9c3f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#16 0xffffffff8020768d in handle_syscall ()
#17 0x0000000000000030 in ?? ()
#18 0x00007f7fff4c5a10 in ?? ()
#19 0x00007f7fff4c5a30 in ?? ()
#20 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe8220b8e780
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8220b8e780)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8220b8e780)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8098b761 in turnstile_block (ts=<optimized out>, ts@entry=0x0,
q=q@entry=1, obj=obj@entry=0xfffffe8224a7b130,
sobj=sobj@entry=0xffffffff81482720 <rw_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80977cb3 in rw_vector_enter (rw=0xfffffe8224a7b130, op=RW_WRITER)
at /usr/src/sys/kern/kern_rwlock.c:395
#4 0xffffffff80a17d7e in genfs_lock (v=<optimized out>)
at /usr/src/sys/miscfs/genfs/genfs_vnops.c:343
#5 0xffffffff80a0ed35 in VOP_LOCK (vp=vp@entry=0xfffffe8224a7b000,
flags=flags@entry=131074) at /usr/src/sys/kern/vnode_if.c:1284
#6 0xffffffff80a05846 in vn_lock (vp=vp@entry=0xfffffe8224a7b000,
flags=flags@entry=131074) at /usr/src/sys/kern/vfs_vnops.c:1036
#7 0xffffffff80a0601d in vn_write (fp=<optimized out>,
offset=0xfffffe8225588b00, uio=0xffff80009ebf8df0,
cred=0xfffffe822f724f00, flags=1) at /usr/src/sys/kern/vfs_vnops.c:592
#8 0xffffffff809b4c1d in dofilewrite (fd=fd@entry=2, fp=0xfffffe8225588b00,
buf=0x764fb6902000, nbyte=90, offset=<optimized out>, flags=flags@entry=1,
retval=retval@entry=0xffff80009ebf8eb0)
at /usr/src/sys/kern/sys_generic.c:350
#9 0xffffffff809b4d22 in sys_write (l=<optimized out>,
uap=0xffff80009ebf8f00, retval=0xffff80009ebf8eb0)
at /usr/src/sys/kern/sys_generic.c:318
#10 0xffffffff80248dbd in sy_call (rval=0xffff80009ebf8eb0,
uap=0xffff80009ebf8f00, l=0xfffffe8220b8e780,
sy=0xffffffff8147da40 <sysent+96>) at /usr/src/sys/sys/syscallvar.h:65
#11 sy_invoke (code=4, rval=0xffff80009ebf8eb0, uap=0xffff80009ebf8f00,
l=0xfffffe8220b8e780, sy=0xffffffff8147da40 <sysent+96>)
at /usr/src/sys/sys/syscallvar.h:94
#12 syscall (frame=0xffff80009ebf8f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#13 0xffffffff8020768d in handle_syscall ()
#14 0x0000000000000002 in ?? ()
#15 0x0000764fb6902000 in ?? ()
#16 0x000000000000005a in ?? ()
#17 0x0000764fb690205a in ?? ()
#18 0x000000000000001e in ?? ()
#19 0x0000000000000001 in ?? ()
#20 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe8221e57600
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8221e57600)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8221e57600)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8098b761 in turnstile_block (ts=<optimized out>,
ts@entry=0xfffffe82123900c8, q=q@entry=1,
obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>,
sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fef2d in _fstrans_start (wait=1, lock_type=FSTRANS_SHARED,
mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
#5 fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
#6 0xffffffff80a0e399 in vop_pre (vp=0xfffffe8221e36bd0,
vp=0xfffffe8221e36bd0, op=FST_YES, mpsafe=<synthetic pointer>,
mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#7 VOP_POLL (vp=0xfffffe8221e36bd0, events=<optimized out>)
at /usr/src/sys/kern/vnode_if.c:677
#8 0xffffffff809bd7e8 in selscan (retval=0xffff80009eae8eb0, ni=2046820352,
nfd=52, bits=0xffff80009eae8d48 "H") at /usr/src/sys/kern/sys_select.c:402
#9 sel_do_scan (op=op@entry=1, fds=fds@entry=0xffff80009eae8d48,
nf=nf@entry=52, ni=ni@entry=8, ts=ts@entry=0xffff80009eae8e70,
mask=mask@entry=0x0, retval=retval@entry=0xffff80009eae8eb0)
at /usr/src/sys/kern/sys_select.c:264
#10 0xffffffff809bdb96 in selcommon (retval=retval@entry=0xffff80009eae8eb0,
nd=52, u_in=0x7f7fff4c5a10, u_ou=0x7f7fff4c5a30, u_ex=0x0,
ts=0xffff80009eae8e70, mask=mask@entry=0x0)
at /usr/src/sys/kern/sys_select.c:362
#11 0xffffffff809bdeab in selcommon (mask=0x0, ts=<optimized out>,
u_ex=<optimized out>, u_ou=<optimized out>, u_in=<optimized out>,
nd=<optimized out>, retval=0xffff80009eae8eb0)
at /usr/src/sys/kern/sys_select.c:337
#12 sys___select50 (l=<optimized out>, uap=0xffff80009eae8f00,
retval=0xffff80009eae8eb0) at /usr/src/sys/kern/sys_select.c:212
#13 0xffffffff80248dbd in sy_call (rval=0xffff80009eae8eb0,
uap=0xffff80009eae8f00, l=0xfffffe8221e57600,
sy=0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscallvar.h:65
#14 sy_invoke (code=417, rval=0xffff80009eae8eb0, uap=0xffff80009eae8f00,
l=0xfffffe8221e57600, sy=0xffffffff814800f8 <sysent+10008>)
at /usr/src/sys/sys/syscallvar.h:94
#15 syscall (frame=0xffff80009eae8f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#16 0xffffffff8020768d in handle_syscall ()
#17 0x0000000000000034 in ?? ()
#18 0x00007f7fff4c5a10 in ?? ()
#19 0x00007f7fff4c5a30 in ?? ()
#20 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe8222569900
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8222569900)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8222569900)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8098b761 in turnstile_block (ts=<optimized out>,
ts@entry=0xfffffe82123900c8, q=q@entry=1,
obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>,
sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fef2d in _fstrans_start (wait=1, lock_type=FSTRANS_SHARED,
mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
#5 fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
#6 0xffffffff80a0e399 in vop_pre (vp=0xfffffe8223aa62a0,
vp=0xfffffe8223aa62a0, op=FST_YES, mpsafe=<synthetic pointer>,
mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#7 VOP_POLL (vp=0xfffffe8223aa62a0, events=<optimized out>)
at /usr/src/sys/kern/vnode_if.c:677
#8 0xffffffff809bd7e8 in selscan (retval=0xffff80009e9cdeb0, ni=796917760,
nfd=53, bits=0xffff80009e9cdd48 "\b") at /usr/src/sys/kern/sys_select.c:402
#9 sel_do_scan (op=op@entry=1, fds=fds@entry=0xffff80009e9cdd48,
nf=nf@entry=53, ni=ni@entry=8, ts=ts@entry=0xffff80009e9cde70,
mask=mask@entry=0x0, retval=retval@entry=0xffff80009e9cdeb0)
at /usr/src/sys/kern/sys_select.c:264
#10 0xffffffff809bdb96 in selcommon (retval=retval@entry=0xffff80009e9cdeb0,
nd=53, u_in=0x7f7fff4c5a10, u_ou=0x7f7fff4c5a30, u_ex=0x0,
ts=0xffff80009e9cde70, mask=mask@entry=0x0)
at /usr/src/sys/kern/sys_select.c:362
#11 0xffffffff809bdeab in selcommon (mask=0x0, ts=<optimized out>,
u_ex=<optimized out>, u_ou=<optimized out>, u_in=<optimized out>,
nd=<optimized out>, retval=0xffff80009e9cdeb0)
at /usr/src/sys/kern/sys_select.c:337
#12 sys___select50 (l=<optimized out>, uap=0xffff80009e9cdf00,
retval=0xffff80009e9cdeb0) at /usr/src/sys/kern/sys_select.c:212
#13 0xffffffff80248dbd in sy_call (rval=0xffff80009e9cdeb0,
uap=0xffff80009e9cdf00, l=0xfffffe8222569900,
sy=0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscallvar.h:65
#14 sy_invoke (code=417, rval=0xffff80009e9cdeb0, uap=0xffff80009e9cdf00,
l=0xfffffe8222569900, sy=0xffffffff814800f8 <sysent+10008>)
at /usr/src/sys/sys/syscallvar.h:94
#15 syscall (frame=0xffff80009e9cdf00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#16 0xffffffff8020768d in handle_syscall ()
#17 0x0000000000000035 in ?? ()
#18 0x00007f7fff4c5a10 in ?? ()
#19 0x00007f7fff4c5a30 in ?? ()
#20 0x0000000000000000 in ?? ()
(gdb)
(gdb) kvm proc 0xfffffe8225709840
0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8225709840)
at /usr/src/sys/kern/kern_synch.c:720
720 prevlwp = cpu_switchto(l, newl, returning);
(gdb) where
#0 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8225709840)
at /usr/src/sys/kern/kern_synch.c:720
#1 0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0,
catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
#2 0xffffffff8098b761 in turnstile_block (ts=<optimized out>,
ts@entry=0xfffffe82123900c8, q=q@entry=1,
obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>,
sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
at /usr/src/sys/kern/kern_turnstile.c:430
#3 0xffffffff80965a59 in mutex_vector_enter (
mtx=0xffffffff815cb9b0 <fstrans_lock>)
at /usr/src/sys/kern/kern_mutex.c:693
#4 0xffffffff809fef2d in _fstrans_start (wait=1, lock_type=FSTRANS_SHARED,
mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
#5 fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
#6 0xffffffff80a0e399 in vop_pre (vp=0xfffffe8222fe4bd8,
vp=0xfffffe8222fe4bd8, op=FST_YES, mpsafe=<synthetic pointer>,
mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
#7 VOP_POLL (vp=0xfffffe8222fe4bd8, events=<optimized out>)
at /usr/src/sys/kern/vnode_if.c:677
#8 0xffffffff809bd7e8 in selscan (retval=0xffff80009e904eb0, ni=2625634304,
nfd=53, bits=0xffff80009e904d48 "\b") at /usr/src/sys/kern/sys_select.c:402
#9 sel_do_scan (op=op@entry=1, fds=fds@entry=0xffff80009e904d48,
nf=nf@entry=53, ni=ni@entry=8, ts=ts@entry=0xffff80009e904e70,
mask=mask@entry=0x0, retval=retval@entry=0xffff80009e904eb0)
at /usr/src/sys/kern/sys_select.c:264
#10 0xffffffff809bdb96 in selcommon (retval=retval@entry=0xffff80009e904eb0,
nd=53, u_in=0x7f7fff4c5a10, u_ou=0x7f7fff4c5a30, u_ex=0x0,
ts=0xffff80009e904e70, mask=mask@entry=0x0)
at /usr/src/sys/kern/sys_select.c:362
#11 0xffffffff809bdeab in selcommon (mask=0x0, ts=<optimized out>,
u_ex=<optimized out>, u_ou=<optimized out>, u_in=<optimized out>,
nd=<optimized out>, retval=0xffff80009e904eb0)
at /usr/src/sys/kern/sys_select.c:337
#12 sys___select50 (l=<optimized out>, uap=0xffff80009e904f00,
retval=0xffff80009e904eb0) at /usr/src/sys/kern/sys_select.c:212
#13 0xffffffff80248dbd in sy_call (rval=0xffff80009e904eb0,
uap=0xffff80009e904f00, l=0xfffffe8225709840,
sy=0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscallvar.h:65
#14 sy_invoke (code=417, rval=0xffff80009e904eb0, uap=0xffff80009e904f00,
l=0xfffffe8225709840, sy=0xffffffff814800f8 <sysent+10008>)
at /usr/src/sys/sys/syscallvar.h:94
#15 syscall (frame=0xffff80009e904f00)
at /usr/src/sys/arch/x86/x86/syscall.c:144
#16 0xffffffff8020768d in handle_syscall ()
#17 0x0000000000000035 in ?? ()
#18 0x00007f7fff4c5a10 in ?? ()
#19 0x00007f7fff4c5a30 in ?? ()
#20 0x0000000000000000 in ?? ()
(gdb)
From: Ryota Ozaki <ozaki-r@netbsd.org>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc: port-amd64-maintainer@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, Havard Eidnes <he@netbsd.org>
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network
interfaces in use
Date: Thu, 17 May 2018 10:50:59 +0900
On Wed, May 16, 2018 at 8:41 PM Havard Eidnes <he@netbsd.org> wrote:
> The following reply was made to PR port-amd64/53155; it has been noted by
GNATS.
> From: Havard Eidnes <he@NetBSD.org>
> To: gnats-bugs@NetBSD.org
> Cc:
> Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
> network interfaces in use
> Date: Wed, 16 May 2018 13:37:28 +0200 (CEST)
> [[ Hmm, here's a non-quoted-printable version of the previous ]]
> Hi,
> we provoked another wedge, and captured a kernel core dump from
> the wedging machine. The kernel core dump and NetBSD images are
> available for looking at.
> using gdb and crash, here's a brief summary of the "interesting"
> processes in the crash dump, and below that I include backtraces
> of all the waiting processes.
> There's lots of contention for fstrans_lock.
> Offhand I don't see a deadlock which might explain the observed
> behaviour (goes totally "deaf" on the network, i.e. doesn't even
> respond to ping).
> It doesn't look like gdb can trace through interrupt frames (?),
> looking at proc 788 all I get is:
> (gdb) kvm proc 0xfffffe8220b8e360
> 0xffffffff8021cfe0 in softintr_ret ()
> (gdb) where
> #0 0xffffffff8021cfe0 in softintr_ret ()
> #1 0x0000000000000000 in ?? ()
> (gdb)
> Crash manages to do this one, though, apparently (see below).
> Furthermore, using the various gdb scripts in
> /usr/src/sys/gdbscripts/ I can look at some of the locks.
> It is quite possible that up'ing the interface in question causes
> lots of activity for opening pty pairs, and that the root cause
> of the issue is there rather than related to networking in itself(?)
> Further hints?
> Regards,
> - Havard
> ------------------------------
> PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
> 7052 1 3 1 8020000 fffffe8220c58540 cron tstile
> Wants fstrans_lock
> 9187 1 3 6 8020000 fffffe821faa60c0 expect xchicv
> Holds fstrans_lock, in pserialize_perform, waits on condition variable
> after doing xc_broadcast(XC_HIGHPRI, nullop)
> Doing (roughly) pty_grant_slave -> genfs_revoke -> vfs_suspend ->
> fstrans_setstate -> pserialize_perform -> xc_wait -> cv_wait
This xcall requires that the softint of SOFTINT_SERIAL (softser/N)
on all CPUs processes a callback of the xcall. If any of the softints
get stuck for some reason, the xcall never finish.
Could you show the stack trace of each softser/N? In particular softser/0
looks running and is a suspect.
Thanks,
ozaki-r
From: Havard Eidnes <he@NetBSD.org>
To: ozaki-r@netbsd.org
Cc: gnats-bugs@netbsd.org, port-amd64-maintainer@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Thu, 17 May 2018 12:34:28 +0200 (CEST)
>> PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
>> 7052 1 3 1 8020000 fffffe8220c58540 cron tstile
>> Wants fstrans_lock
>>
>> 9187 1 3 6 8020000 fffffe821faa60c0 expect xchicv
>> Holds fstrans_lock, in pserialize_perform, waits on condition variable
>> after doing xc_broadcast(XC_HIGHPRI, nullop)
>> Doing (roughly) pty_grant_slave -> genfs_revoke -> vfs_suspend ->
>> fstrans_setstate -> pserialize_perform -> xc_wait -> cv_wait
>
> This xcall requires that the softint of SOFTINT_SERIAL (softser/N)
> on all CPUs processes a callback of the xcall. If any of the softints
> get stuck for some reason, the xcall never finish.
>
> Could you show the stack trace of each softser/N? In particular softser/0
> looks running and is a suspect.
Getting the tracebacks of these LWPs appears to be somewhat
problematic; I don't get much useful info from crash:
0 78 3 3 200 fffffe810f2e9740 mfi0 mfi0
0 69 3 5 200 fffffe810f20d6e0 atabus1 atath
0 68 3 4 200 fffffe810f20db00 atabus0 atath
0 66 3 0 200 fffffe810ed3b6c0 scsibus0 sccomp
0 65 3 0 200 fffffe810ed3bae0 usbtask-dr usbtsk
0 64 3 0 200 fffffe810ecaa280 usbtask-hc usbtsk
0 63 3 0 200 fffffe810ecaa6a0 bnx3 bnx3
0 62 3 0 200 fffffe810ecaaac0 bnx2 bnx2
0 61 3 0 200 fffffe810eb61260 bnx1 bnx1
0 60 3 7 200 fffffe810eb61680 bnx0 bnx0
0 > 59 7 2 200 fffffe810eb61aa0 ipmi
0 58 3 7 200 fffffe810ea7c240 xcall/7 xcall
0 57 1 7 200 fffffe810ea7c660 softser/7
0 56 1 7 200 fffffe810ea7ca80 softclk/7
0 55 1 7 200 fffffe810ea5d220 softbio/7
0 54 1 7 200 fffffe810ea5d640 softnet/7
0 > 53 7 7 201 fffffe810ea5da60 idle/7
0 52 3 6 200 fffffe810ea36200 xcall/6 xcall
0 51 1 6 200 fffffe810ea36620 softser/6
0 50 1 6 200 fffffe810ea36a40 softclk/6
0 49 1 6 200 fffffe810ea0f1e0 softbio/6
0 48 1 6 200 fffffe810ea0f600 softnet/6
0 47 1 6 201 fffffe810ea0fa20 idle/6
0 46 3 5 200 fffffe810e9e01c0 xcall/5 xcall
0 45 1 5 200 fffffe810e9e05e0 softser/5
0 44 1 5 200 fffffe810e9e0a00 softclk/5
0 43 1 5 200 fffffe810e9d11a0 softbio/5
0 42 1 5 200 fffffe810e9d15c0 softnet/5
0 > 41 7 5 201 fffffe810e9d19e0 idle/5
0 40 3 4 200 fffffe810e9a2180 xcall/4 xcall
0 39 1 4 200 fffffe810e9a25a0 softser/4
0 38 1 4 200 fffffe810e9a29c0 softclk/4
0 37 1 4 200 fffffe810e973160 softbio/4
0 36 1 4 200 fffffe810e973580 softnet/4
0 > 35 7 4 201 fffffe810e9739a0 idle/4
0 34 3 3 200 fffffe810e94c140 xcall/3 xcall
0 33 1 3 200 fffffe810e94c560 softser/3
0 32 1 3 200 fffffe810e94c980 softclk/3
0 31 1 3 200 fffffe810e93d120 softbio/3
0 30 1 3 200 fffffe810e93d540 softnet/3
0 > 29 7 3 201 fffffe810e93d960 idle/3
0 28 3 2 200 fffffe810e906100 xcall/2 xcall
0 27 1 2 200 fffffe810e906520 softser/2
0 26 1 2 200 fffffe810e906940 softclk/2
0 25 1 2 200 fffffe810e8ef0e0 softbio/2
0 24 1 2 200 fffffe810e8ef500 softnet/2
0 23 1 2 201 fffffe810e8ef920 idle/2
0 22 3 1 200 fffffe810e8c50c0 xcall/1 xcall
0 21 1 1 200 fffffe810e8c54e0 softser/1
0 20 1 1 200 fffffe810e8c5900 softclk/1
0 19 1 1 200 fffffe810e8b10a0 softbio/1
0 18 1 1 200 fffffe810e8b14c0 softnet/1
0 > 17 7 1 201 fffffe810e8b18e0 idle/1
0 16 3 0 200 fffffe822de92080 lnxsyswq lnxsyswq
0 15 3 0 200 fffffe822de924a0 sysmon smtaskq
0 14 3 2 200 fffffe822de928c0 pmfsuspend pmfsuspend
0 13 3 7 200 fffffe822e2b1060 pmfevent pmfevent
0 12 3 0 200 fffffe822e2b1480 sopendfree sopendfr
0 11 3 2 200 fffffe822e2b18a0 nfssilly nfssilly
0 10 3 2 200 fffffe822f6d0040 cachegc cachegc
0 9 3 3 200 fffffe822f6d0460 vdrain vdrain
0 8 3 0 200 fffffe822f6d0880 modunload mod_unld
0 7 3 0 200 fffffe822f6eb020 xcall/0 xcall
0 > 6 7 0 200 fffffe822f6eb440 softser/0
0 > 5 7 0 200 fffffe822f6eb860 softclk/0
0 4 1 0 200 fffffe822f707000 softbio/0
0 3 1 0 200 fffffe822f707420 softnet/0
0 2 1 0 201 fffffe822f707840 idle/0
0 1 3 4 200 ffffffff81481b20 swapper uvm
crash> trace/a fffffe810ea7c660
trace: pid 0 lid 57 at 0xffff80008f7e8e80
lockdebug_wantlock() at lockdebug_wantlock+0xf9
_KERNEL_OPT_WSDISPLAY_SCROLLBACK_LINES() at _KERNEL_OPT_WSDISPLAY_SCROLLBACK_LIN
ES+0x173
crash> trace/a fffffe810ea36620
trace: pid 0 lid 51 at 0x0
crash> trace/a fffffe810e9e05e0
trace: pid 0 lid 45 at 0x0
0:
crash> trace/a fffffe810e9a25a0
trace: pid 0 lid 39 at 0x0
0:
crash> trace/a fffffe810e94c560
trace: pid 0 lid 33 at 0x0
0:
crash> trace/a fffffe810e906520
trace: pid 0 lid 27 at 0xffff80008f72ae80
lockdebug_wantlock() at lockdebug_wantlock+0xf9
_KERNEL_OPT_WSDISPLAY_SCROLLBACK_LINES() at _KERNEL_OPT_WSDISPLAY_SCROLLBACK_LIN
ES+0x173
crash> trace/a fffffe810e8c54e0
trace: pid 0 lid 21 at 0x0
crash> trace/a fffffe822f6eb440
trace: pid 0 lid 6 at 0x0
crash>
GDB doesn't look a lot better either:
(gdb) kvm proc 0xfffffe822f6eb440
0x0000000000000005 in ?? ()
(gdb) where
#0 0x0000000000000005 in ?? ()
#1 0xfffffe810ed5be08 in ?? ()
#2 0xffffffff809a2791 in percpu_getref (pc=0xffff80008f69bfe0)
at /usr/src/sys/kern/subr_percpu.c:293
#3 0x0000000000000000 in ?? ()
(gdb)
(gdb) x/i 0xfffffe810ed5be08
0xfffffe810ed5be08: (bad)
(gdb)
(gdb) up
#1 0xfffffe810ed5be08 in ?? ()
(gdb) up
#2 0xffffffff809a2791 in percpu_getref (pc=0xffff80008f69bfe0)
at /usr/src/sys/kern/subr_percpu.c:293
293 kpreempt_disable();
(gdb)
(gdb) list
288
289 void *
290 percpu_getref(percpu_t *pc)
291 {
292
293 kpreempt_disable();
294 return percpu_getptr_remote(pc, curcpu());
295 }
296
297 /*
(gdb)
Regards,
- Havard
From: "J. Hannken-Illjes" <hannken@eis.cs.tu-bs.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network
interfaces in use
Date: Thu, 17 May 2018 12:58:45 +0200
>> Could you show the stack trace of each softser/N? In particular softser/0
>> looks running and is a suspect.
>
> Getting the tracebacks of these LWPs appears to be somewhat
> problematic; I don't get much useful info from crash:
<snip>
>
> 0 > 6 7 0 200 fffffe822f6eb440 softser/0
> 0 > 5 7 0 200 fffffe822f6eb860 softclk/0
Crash reports two threads on cpu 0.
If you get this type of hang again please try to get the corresponding
stack traces from DDB: if softXXX/N is on cpu (>) use "mach cpu N; bt"
--
J. Hannken-Illjes - hannken@eis.cs.tu-bs.de - TU Braunschweig (Germany)
From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org, hannken@eis.cs.tu-bs.de
Cc: port-amd64-maintainer@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Thu, 17 May 2018 15:54:45 +0200 (CEST)
> >> Could you show the stack trace of each softser/N? In particular softser/0
> >> looks running and is a suspect.
> >
> > Getting the tracebacks of these LWPs appears to be somewhat
> > problematic; I don't get much useful info from crash:
> <snip>
> >
> > 0 > 6 7 0 200 fffffe822f6eb440 softser/0
> > 0 > 5 7 0 200 fffffe822f6eb860 softclk/0
>
> Crash reports two threads on cpu 0.
>
> If you get this type of hang again please try to get the corresponding
> stack traces from DDB: if softXXX/N is on cpu (>) use "mach cpu N; bt"
Hmm, looking back at earlier submitted information where I dupmd
various information from DDB, I find this identical situation:
0 > 6 7 0 200 fffffe822f6eb440 softser/0
0 > 5 7 0 200 fffffe822f6eb860 softclk/0
and the backtraces for these were:
db{0}> trace/a fffffe822f6eb440
trace: pid 0 lid 6 at 0xffff80008f69bc30
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
uhci_intr() at netbsd:uhci_intr+0x19
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}>
db{0}> trace/a fffffe822f6eb860
trace: pid 0 lid 5 at 0xffff80009edadd60
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
ffff80009edadf00:
db{0}>
Not sure this provides any more useful information, though...
Regards,
- Havard
From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org, hannken@eis.cs.tu-bs.de
Cc: port-amd64-maintainer@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Thu, 17 May 2018 16:28:56 +0200 (CEST)
And here's similar of a live kernel (I've updated to 8.0_RC1 with
this kernel), where there's apparently no less than 3 lwps active
on CPU 0(!). It takes 5-10 seconds to provoke the hang after I
"up" the third network interface...
fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip 0xffffffff8021ff65 cs 0x8 rflags 0x202 cr2 0x7f7fff36db08 ilevel 0x8 rsp 0xffff8000ae6a2d78
curlwp 0xfffffe822f5eb440 pid 0.6 lowest kstack 0xffff8000ae69f2c0
Stopped in pid 0.6 (system) at netbsd:breakpoint+0x5: leave
db{0}> ps
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
16020 1 3 4 0 fffffe820b650b00 w tstile
12 > 1 7 2 0 fffffe820a0d92a0 pickup
21344 1 3 0 80 fffffe820ec515a0 sshd select
9214 1 3 2 80 fffffe821780f9c0 sshd select
5857 1 3 2 80 fffffe81c7c8d020 sshd select
17622 1 3 2 80 fffffe8220a260a0 sshd select
6326 1 3 4 80 fffffe82181fb8e0 bash select
5489 1 3 4 80 fffffe82187b8340 sshd select
11299 1 3 0 80 fffffe80b2611580 sshd select
25883 1 3 6 80 fffffe820aa4da40 sshd select
2169 1 3 4 80 fffffe821aa8d700 sshd select
11556 1 3 6 80 fffffe82193e5900 bash select
29717 1 3 4 80 fffffe81d20356a0 sshd select
27130 1 3 7 80 fffffe822297f2c0 sshd select
2956 1 3 0 80 fffffe8212a8a760 tcsh pause
20252 1 3 6 80 fffffe821a113280 tcsh pause
24846 1 3 3 80 fffffe82060d0580 xterm select
24052 1 3 1 80 fffffe81bc2e02c0 tcsh pause
18703 1 3 2 80 fffffe8211a67120 sshd select
29547 1 3 1 80 fffffe820de4f780 sshd select
585 1 3 0 80 fffffe822d7b20c0 tcsh ttyraw
11254 1 3 0 80 fffffe820ad30460 xterm select
21566 1 3 0 80 fffffe820cd798c0 tcsh pause
29317 1 3 5 80 fffffe820fa32900 sshd select
19621 1 3 0 80 fffffe8220a264c0 sshd select
23570 1 3 2 80 fffffe821b5bd1a0 ssh select
14266 1 3 2 80 fffffe821971d640 sh wait
20244 1 3 5 80 fffffe82190d4aa0 ssh select
20184 1 3 5 80 fffffe82213f5b80 sh wait
14347 1 3 2 80 fffffe8206800100 ssh select
22513 1 3 2 80 fffffe820fa320c0 sh wait
25132 1 3 2 80 fffffe8211a67960 ssh select
24207 1 3 2 80 fffffe821a8e19c0 sh wait
18230 1 3 6 80 fffffe822a6789e0 ssh select
15149 1 3 6 80 fffffe821f3e3860 sh wait
20547 1 3 2 80 fffffe820f5bb140 ssh select
28126 1 3 2 80 fffffe8219d9fb60 sh wait
21642 1 3 7 80 fffffe8219cc1aa0 ssh select
22234 1 3 7 80 fffffe82187b8760 sh wait
23629 1 3 7 80 fffffe8219973860 ssh select
17367 1 3 7 80 fffffe8208a40600 sh wait
20259 1 3 2 80 fffffe8212a8ab80 ssh select
26419 1 3 2 80 fffffe8211b7b420 sh wait
20593 1 3 7 80 fffffe822a314a40 ssh select
22687 1 3 7 80 fffffe81bc2e06e0 sh wait
29573 1 3 7 80 fffffe820ec519c0 ssh select
22337 1 3 7 80 fffffe820b8ff2e0 sh wait
29331 1 3 2 80 fffffe82061ab120 ssh select
20688 1 3 2 80 fffffe8217196680 sh wait
24552 1 3 7 80 fffffe82179df580 ssh select
16014 1 3 7 80 fffffe8209b005e0 sh wait
14412 1 3 2 80 fffffe8218d7f8a0 ssh select
16926 1 3 2 80 fffffe821a716720 sh wait
15537 1 3 7 80 fffffe81c5ad6880 ssh select
18282 1 3 7 80 fffffe8206800520 sh wait
14420 1 3 2 80 fffffe82060d0160 ssh select
17970 1 3 2 80 fffffe82193e54e0 sh wait
21443 1 3 7 80 fffffe8214d12ac0 ssh select
20009 1 3 7 80 fffffe82124940e0 sh wait
18174 1 3 0 80 fffffe820b6502c0 ssh select
25056 1 3 2 80 fffffe82215f3b60 sh wait
22994 1 3 2 80 fffffe821971da60 ssh select
25609 1 3 2 80 fffffe81c4e9f360 sh wait
22696 1 3 2 80 fffffe8222716b40 ssh select
17593 1 3 2 80 fffffe82251301e0 sh wait
16480 1 3 7 80 fffffe82215f3320 ssh select
20589 1 3 7 80 fffffe8216624ae0 sh wait
20596 1 3 7 80 fffffe822125a700 ssh select
10969 1 3 7 80 fffffe81bad31340 sh wait
14044 1 3 7 80 fffffe8218f87480 ssh select
17878 1 3 7 80 fffffe8217de6a80 sh wait
15657 1 3 0 80 fffffe8208a401e0 ssh select
21421 1 3 0 80 fffffe81c53a00c0 sh wait
14576 1 3 5 80 fffffe822125a2e0 ssh select
19912 1 3 2 80 fffffe82220202a0 sh wait
17455 1 3 2 80 fffffe822d8cd140 ssh select
16887 1 3 2 80 fffffe821f3e3440 sh wait
18158 1 3 2 80 fffffe8215e19200 ssh select
17985 1 3 2 80 fffffe820b8e8200 sh wait
16716 1 3 0 80 fffffe822d9a7120 ssh select
19884 1 3 5 80 fffffe820b8ffb20 sh wait
25660 1 3 0 80 fffffe8217da1320 ssh select
22067 1 3 0 80 fffffe820c5741e0 sh wait
17620 1 3 5 80 fffffe81c69ad4c0 ssh select
18165 1 3 5 80 fffffe82183b70e0 sh wait
22113 1 3 5 80 fffffe821b76b300 ssh select
18292 1 3 6 80 fffffe82183e59e0 ssh select
18510 1 3 5 80 fffffe8217b226a0 sh wait
20654 1 3 0 80 fffffe820de4f360 sh wait
21304 1 3 0 80 fffffe821995d480 ssh select
19551 1 3 0 80 fffffe820e4f9740 sh wait
27093 1 3 0 80 fffffe8208a7da00 ssh select
25978 1 3 0 80 fffffe821a716300 sh wait
17223 1 3 5 80 fffffe82183e51a0 ssh select
24635 1 3 0 80 fffffe820de4fba0 sh wait
22126 1 3 5 80 fffffe8219d9f320 ssh select
18244 1 3 0 80 fffffe81cffaeaa0 sh wait
22357 1 3 5 80 fffffe8225130a20 ssh select
17566 1 3 0 80 fffffe82179df160 sh wait
17650 1 3 5 80 fffffe81c5e1e8c0 ssh select
17943 1 3 0 80 fffffe821780f180 sh wait
21570 1 3 0 80 fffffe8212f7e4c0 ssh select
20536 1 3 0 80 fffffe820a0d9ae0 sh wait
18404 1 3 5 80 fffffe822d7055e0 ssh select
16408 1 3 0 80 fffffe8207a97180 sh wait
17272 1 3 0 80 fffffe8204fc9980 ssh select
9560 1 3 0 80 fffffe822d7051c0 sh wait
16215 1 3 0 80 fffffe820ad30040 ssh select
23670 1 3 0 80 fffffe821ac5db00 sh wait
16842 1 3 0 80 fffffe821d40c620 ssh select
12548 1 3 0 80 fffffe81c71ce700 sh wait
23637 1 3 5 80 fffffe81c69ad8e0 ssh select
17966 1 3 0 80 fffffe8207a979c0 sh wait
19110 1 3 0 80 fffffe821bd5c0e0 ssh select
17002 1 3 0 80 fffffe8214c6e360 sh wait
23925 1 3 0 80 fffffe8219e3b6c0 ssh select
19616 1 3 0 80 fffffe8218434560 sh wait
21852 1 3 5 80 fffffe82179cb100 ssh select
16322 1 3 0 80 fffffe82190bc240 sh wait
14599 1 3 1 80 fffffe8219b33540 ssh select
11614 1 3 1 80 fffffe8229aff220 sh wait
10527 1 3 0 80 fffffe82220206c0 ssh select
14101 1 3 1 80 fffffe8219cc1680 sh wait
13441 1 3 0 80 fffffe8219d9f740 ssh select
17221 1 3 1 80 fffffe8209b001c0 sh wait
14873 1 3 0 80 fffffe821ae039a0 ssh select
15011 1 3 0 80 fffffe822d7b2900 sh wait
14055 1 3 0 80 fffffe8212a8a340 ssh select
16749 1 3 1 80 fffffe82207c9780 sh wait
18012 1 3 1 80 fffffe8212787080 ssh select
10499 1 3 1 80 fffffe821a1fa240 sh wait
13041 1 3 0 80 fffffe821981c460 ssh select
15374 1 3 0 80 fffffe821bf6d4a0 sh wait
9803 1 3 5 80 fffffe820f3865c0 ssh select
12534 1 3 0 80 fffffe820f3861a0 sh wait
11674 1 3 2 80 fffffe82193e50c0 ssh select
17698 1 3 0 80 fffffe81c4e9f780 sh wait
13811 1 3 0 80 fffffe820cc7d440 ssh select
11034 1 3 0 80 fffffe821aa2b220 sh wait
6294 1 3 0 80 fffffe8218d7f480 ssh select
12577 1 3 0 80 fffffe82183e55c0 sh wait
10701 1 3 5 80 fffffe8217196aa0 ssh select
6593 1 3 0 80 fffffe8222020ae0 sh wait
12702 1 3 1 80 fffffe82188ea360 ssh select
10806 1 3 0 80 fffffe8208a7d5e0 sh wait
6638 1 3 4 80 fffffe820cc7d860 ssh select
14569 1 3 0 80 fffffe8215e19620 sh wait
5202 1 3 3 80 fffffe820c2a6900 ssh select
10543 1 3 0 80 fffffe82166242a0 sh wait
17177 1 3 5 80 fffffe820c574600 ssh select
11624 1 3 0 80 fffffe822d7b24e0 sh wait
16329 1 3 0 80 fffffe81cfe7f660 ssh select
9943 1 3 0 80 fffffe8222123b20 sh wait
16096 1 3 5 80 fffffe82183b7920 ssh select
11557 1 3 0 80 fffffe81c4e9fba0 sh wait
14572 1 3 0 80 fffffe8220f0a5e0 ssh select
13731 1 3 0 80 fffffe81c84acb60 sh wait
10907 1 3 5 80 fffffe822d6c45a0 ssh select
10785 1 3 0 80 fffffe81c53a04e0 sh wait
10925 1 3 5 80 fffffe8211b7b840 ssh select
10573 1 3 0 80 fffffe8219973020 sh wait
7697 1 3 5 80 fffffe821a084340 ssh select
16269 1 3 0 80 fffffe8217da1b60 sh wait
13970 1 3 0 80 fffffe82190d4680 ssh select
7270 1 3 0 80 fffffe820b7e4000 sh wait
16358 1 3 0 80 fffffe820e4f9320 ssh select
13381 1 3 0 80 fffffe821c93d040 sh wait
4222 1 3 0 80 fffffe820b6506e0 less ttyraw
2320 1 3 6 80 fffffe821bf6d8c0 dumpfs pipe_wr
7456 1 3 6 80 fffffe821b5bd5c0 tcsh pause
1250 1 3 0 80 fffffe821b76b720 tcsh pause
1607 1 3 4 80 fffffe8214c6eba0 xterm select
330 1 3 0 80 fffffe82169cdb00 tcsh pause
1457 1 3 5 80 fffffe8214d12280 sshd select
787 1 3 0 80 fffffe8214d126a0 sshd select
1251 1 3 0 80 fffffe82166246c0 tcsh ttyraw
746 1 3 1 80 fffffe8215cf9220 tcsh pause
1503 1 3 5 80 fffffe8215cf9640 xterm select
603 1 3 6 80 fffffe8215cf9a60 tcsh pause
1112 1 3 2 80 fffffe8215e19a40 sshd select
338 1 3 0 80 fffffe8215dea1e0 sshd select
1105 1 3 3 0 fffffe8215dea600 conserver tstile
1210 1 3 4 0 fffffe82179cb520 conserver tstile
921 1 3 7 0 fffffe82232b56a0 conserver tstile
918 1 3 7 0 fffffe8219973440 conserver tstile
527 1 3 3 0 fffffe8218bbb420 conserver tstile
264 1 3 3 0 fffffe82188ea780 conserver tstile
869 1 3 2 0 fffffe821a1fa660 conserver tstile
708 1 3 6 0 fffffe821b925560 conserver tstile
959 1 3 2 0 fffffe822297fb00 conserver tstile
1052 1 3 3 0 fffffe822a314200 conserver tstile
763 1 3 0 80 fffffe822dc72500 getty ttyraw
1149 1 3 4 0 fffffe822d8cd560 conserver tstile
435 1 3 6 80 fffffe82207c9ba0 conserver select
361 1 3 0 80 fffffe821d42a4c0 cron nanoslp
370 1 3 6 80 fffffe821bc42060 python2.7 select
646 1 3 5 80 fffffe821c93d460 tcsh ttyraw
475 1 3 0 80 fffffe821c93d880 tcsh pause
692 1 3 0 80 fffffe821cdfe020 xterm select
557 1 3 2 80 fffffe821cdfe440 tcsh pause
495 1 3 7 80 fffffe821cdfe860 sshd select
170 1 3 0 80 fffffe822026b000 sshd select
112 1 3 6 80 fffffe822026b420 inetd kqueue
709 1 3 6 80 fffffe822026b840 nginx kqueue
516 1 3 1 80 fffffe8222716720 nginx pause
643 1 3 1 80 fffffe8222716300 qmgr kqueue
785 > 1 7 6 0 fffffe8222123700 master
438 1 3 0 80 fffffe82232b5ac0 sshd select
335 1 3 6 0 fffffe822d6c49c0 tac_plus tstile
439 1 3 6 80 fffffe822555c260 powerd kqueue
446 1 3 3 80 fffffe822555c680 ntpd pause
398 4 3 2 80 fffffe822579d660 dhcpd parked
398 3 3 2 80 fffffe822579da80 dhcpd kqueue
398 2 3 0 80 fffffe822a8d3600 dhcpd parked
398 1 3 1 80 fffffe822a6785c0 dhcpd parked
83 > 1 7 7 0 fffffe810f34a300 syslogd
1 1 3 2 80 fffffe810f57a900 init wait
0 123 3 7 200 fffffe822a8d3a20 npfgc-0 xchicv
0 122 3 2 200 fffffe822d705a00 ipmi0 ipmi0
0 121 3 0 200 fffffe822dc720e0 physiod physiod
0 120 3 0 200 fffffe822dc08100 aiodoned aiodoned
0 > 119 7 5 200 fffffe822dc08520 ioflush
0 118 3 0 200 fffffe822dc08940 pgdaemon pgdaemon
0 115 3 0 280 fffffe822dc72920 spkr1 bellcv
0 114 3 5 280 fffffe810f57a0c0 audiomix play
0 113 3 5 280 fffffe810edcb2a0 audiorec record
0 112 3 0 200 fffffe810f23d2c0 atapibus0 sccomp
0 110 3 2 200 fffffe810f319b60 usb4 usbevt
0 109 3 3 200 fffffe810f34ab40 usb3 usbevt
0 108 3 6 200 fffffe810f33b700 usb1 usbevt
0 107 3 6 200 fffffe810f33b2e0 usb2 usbevt
0 106 3 2 200 fffffe810f34a720 usb0 usbevt
0 105 3 2 200 fffffe810f57a4e0 usb5 usbevt
0 104 3 0 200 fffffe810f4f30a0 rt_free rt_free
0 103 3 0 200 fffffe810f4f34c0 unpgc unpgc
0 102 3 0 200 fffffe810f4f38e0 key_timehandler key_timehandler
0 101 3 7 200 fffffe810f494080 icmp6_wqinput/7 icmp6_wqinput
0 100 3 6 200 fffffe810f4944a0 icmp6_wqinput/6 icmp6_wqinput
0 99 3 5 200 fffffe810f4948c0 icmp6_wqinput/5 icmp6_wqinput
0 98 3 4 200 fffffe810f4f5060 icmp6_wqinput/4 icmp6_wqinput
0 97 3 3 200 fffffe810f4f5480 icmp6_wqinput/3 icmp6_wqinput
0 96 3 2 200 fffffe810f4f58a0 icmp6_wqinput/2 icmp6_wqinput
0 95 3 1 200 fffffe810f3ce040 icmp6_wqinput/1 icmp6_wqinput
0 94 3 0 200 fffffe810f3ce460 icmp6_wqinput/0 icmp6_wqinput
0 93 3 6 200 fffffe810f3ce880 nd6_timer tstile
0 92 3 7 200 fffffe810f38f020 icmp_wqinput/7 icmp_wqinput
0 91 3 6 200 fffffe810f38f440 icmp_wqinput/6 icmp_wqinput
0 90 3 5 200 fffffe810f38f860 icmp_wqinput/5 icmp_wqinput
0 89 3 4 200 fffffe810f3a0000 icmp_wqinput/4 icmp_wqinput
0 88 3 3 200 fffffe810f3a0420 icmp_wqinput/3 icmp_wqinput
0 87 3 2 200 fffffe810f3a0840 icmp_wqinput/2 icmp_wqinput
0 86 3 1 200 fffffe810f361360 icmp_wqinput/1 icmp_wqinput
0 85 3 0 200 fffffe810f361780 icmp_wqinput/0 icmp_wqinput
0 84 3 4 200 fffffe810f33bb20 rt_timer rt_timer
0 83 3 0 200 fffffe810f361ba0 vmem_rehash vmem_rehash
0 82 3 0 200 fffffe810f320340 coretemp3 coretemp3
0 81 3 3 200 fffffe810f320760 coretemp2 coretemp2
0 80 3 4 200 fffffe810f320b80 coretemp1 coretemp1
0 79 3 5 200 fffffe810f319320 coretemp0 coretemp0
0 78 3 5 200 fffffe810f319740 mfi0 mfi0
0 69 3 5 200 fffffe810f23d6e0 atabus1 atath
0 68 3 1 200 fffffe810f23db00 atabus0 atath
0 66 3 0 200 fffffe810edcb6c0 scsibus0 sccomp
0 65 3 6 200 fffffe810edcbae0 usbtask-dr usbtsk
0 64 3 0 200 fffffe810ed12280 usbtask-hc usbtsk
0 63 3 7 200 fffffe810ed126a0 bnx3 bnx3
0 62 3 5 200 fffffe810ed12ac0 bnx2 bnx2
0 61 3 4 200 fffffe810eb81260 bnx1 bnx1
0 60 3 6 200 fffffe810eb81680 bnx0 bnx0
0 > 59 7 1 200 fffffe810eb81aa0 ipmi
0 58 3 7 200 fffffe810ead2240 xcall/7 xcall
0 57 1 7 200 fffffe810ead2660 softser/7
0 56 1 7 200 fffffe810ead2a80 softclk/7
0 55 1 7 200 fffffe810eacb220 softbio/7
0 54 1 7 200 fffffe810eacb640 softnet/7
0 53 1 7 201 fffffe810eacba60 idle/7
0 52 3 6 200 fffffe810ea7a200 xcall/6 xcall
0 51 1 6 200 fffffe810ea7a620 softser/6
0 50 1 6 200 fffffe810ea7aa40 softclk/6
0 49 1 6 200 fffffe810ea731e0 softbio/6
0 48 1 6 200 fffffe810ea73600 softnet/6
0 47 1 6 201 fffffe810ea73a20 idle/6
0 46 3 5 200 fffffe810ea2a1c0 xcall/5 xcall
0 45 1 5 200 fffffe810ea2a5e0 softser/5
0 44 1 5 200 fffffe810ea2aa00 softclk/5
0 43 1 5 200 fffffe810ea131a0 softbio/5
0 42 1 5 200 fffffe810ea135c0 softnet/5
0 41 1 5 201 fffffe810ea139e0 idle/5
0 40 3 4 200 fffffe810e9d2180 xcall/4 xcall
0 39 1 4 200 fffffe810e9d25a0 softser/4
0 38 1 4 200 fffffe810e9d29c0 softclk/4
0 37 1 4 200 fffffe810e9bb160 softbio/4
0 36 1 4 200 fffffe810e9bb580 softnet/4
0 > 35 7 4 201 fffffe810e9bb9a0 idle/4
0 34 3 3 200 fffffe810e97a140 xcall/3 xcall
0 33 1 3 200 fffffe810e97a560 softser/3
0 32 1 3 200 fffffe810e97a980 softclk/3
0 31 1 3 200 fffffe810e963120 softbio/3
0 30 1 3 200 fffffe810e963540 softnet/3
0 > 29 7 3 201 fffffe810e963960 idle/3
0 28 3 2 200 fffffe810e92a100 xcall/2 xcall
0 27 1 2 200 fffffe810e92a520 softser/2
0 26 1 2 200 fffffe810e92a940 softclk/2
0 25 1 2 200 fffffe810e90b0e0 softbio/2
0 24 1 2 200 fffffe810e90b500 softnet/2
0 23 1 2 201 fffffe810e90b920 idle/2
0 22 3 1 200 fffffe810e8d20c0 xcall/1 xcall
0 21 1 1 200 fffffe810e8d24e0 softser/1
0 20 1 1 200 fffffe810e8d2900 softclk/1
0 19 1 1 200 fffffe810e8ab0a0 softbio/1
0 18 1 1 200 fffffe810e8ab4c0 softnet/1
0 17 1 1 201 fffffe810e8ab8e0 idle/1
0 16 3 7 200 fffffe822dd92080 lnxsyswq lnxsyswq
0 15 3 0 200 fffffe822dd924a0 sysmon smtaskq
0 14 3 0 200 fffffe822dd928c0 pmfsuspend pmfsuspend
0 13 3 7 200 fffffe822e1b1060 pmfevent pmfevent
0 12 3 0 200 fffffe822e1b1480 sopendfree sopendfr
0 11 3 6 200 fffffe822e1b18a0 nfssilly nfssilly
0 10 3 3 200 fffffe822f5d0040 cachegc cachegc
0 9 3 7 200 fffffe822f5d0460 vdrain vdrain
0 8 3 0 200 fffffe822f5d0880 modunload mod_unld
0 7 3 0 200 fffffe822f5eb020 xcall/0 xcall
0 > 6 7 0 200 fffffe822f5eb440 softser/0
0 5 1 0 200 fffffe822f5eb860 softclk/0
0 4 1 0 200 fffffe822f607000 softbio/0
0 > 3 7 0 200 fffffe822f607420 softnet/0
0 > 2 7 0 201 fffffe822f607840 idle/0
0 1 3 7 200 ffffffff81481b80 swapper uvm
db{0}> trace/a fffffe822f5eb440
trace: pid 0 lid 6 at 0xffff8000ae6a2d78
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
bus_space_read_4() at netbsd:bus_space_read_4+0xa
mfi_intr() at netbsd:mfi_intr+0x37
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x36
Xhandle_ioapic_level8() at netbsd:Xhandle_ioapic_level8+0x6a
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff8000ae6a30f0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}> trace/a fffffe822f607420
trace: pid 0 lid 3 at 0xffff8000ae693e20
sowakeup() at netbsd:sowakeup+0x58
udp4_sendup() at netbsd:udp4_sendup+0x15d
udp_input() at netbsd:udp_input+0x4ed
ipintr() at netbsd:ipintr+0x9e1
softint_dispatch() at netbsd:softint_dispatch+0xeb
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff8000ae6940f0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}> trace/a fffffe822f607840
trace: pid 0 lid 2 at 0xffff8000ae645f90
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x1b5
db{0}>
and all the locks:
db{0}> show all locks
[Locks tracked through LWPs]
Locks held by an LWP (pickup):
Lock 0 (initialized at filedesc_ctor)
lock address : 0xfffffe821a4f6c40 type : sleep/adaptive
initialized : 0xffffffff8098863f
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 2
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe820a0d92a0
last locked* : 0xffffffff8098ec35 unlocked : 0xffffffff8098ecde
owner field : 0xfffffe820a0d92a0 wait/spin: 0/0
Turnstile chain at 0xffffffff814c4f40.
=> No active turnstile for this lock.
Locks held by an LWP (master):
Lock 0 (initialized at filedesc_ctor)
lock address : 0xfffffe822a3a4980 type : sleep/adaptive
initialized : 0xffffffff8098863f
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 6
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe8222123700
last locked* : 0xffffffff8098d870 unlocked : 0xffffffff8098a1d6
owner field : 0xfffffe8222123700 wait/spin: 0/0
Turnstile chain at 0xffffffff814c51c0.
=> No active turnstile for this lock.
Locks held by an LWP (syslogd):
Lock 0 (initialized at filedesc_ctor)
lock address : 0xfffffe822a3a4640 type : sleep/adaptive
initialized : 0xffffffff8098863f
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 7
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe810f34a300
last locked* : 0xffffffff8098ec35 unlocked : 0xffffffff8098ecde
owner field : 0xfffffe810f34a300 wait/spin: 0/0
Turnstile chain at 0xffffffff814c4f40.
=> No active turnstile for this lock.
Locks held by an LWP (npfgc-0):
Lock 0 (initialized at npf_config_init)
lock address : 0xfffffe822dc5f608 type : sleep/adaptive
initialized : 0xffffffff8186e8cf
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 7
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe822a8d3a20
last locked* : 0xffffffff81873df3 unlocked : 0xffffffff81873e03
owner field : 0xfffffe822a8d3a20 wait/spin: 0/0
Turnstile chain at 0xffffffff814c4ed0.
=> No active turnstile for this lock.
Locks held by an LWP (ioflush):
Lock 0 (initialized at vcache_alloc)
lock address : 0xfffffe82278a2130 type : sleep/adaptive
initialized : 0xffffffff80a402a2
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 5
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe822dc08520
last locked* : 0xffffffff80a57979 unlocked : 0xffffffff80a579c4
owner/count : 0xfffffe822dc08520 flags : 0x0000000000000004
Turnstile chain at 0xffffffff814c5120.
=> No active turnstile for this lock.
Locks held by an LWP (softnet/0):
Lock 0 (initialized at soinit)
lock address : 0xfffffe822fa90f40 type : sleep/adaptive
initialized : 0xffffffff80a190ce
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 14
current cpu : 0 last held: 0
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe822f607420
last locked* : 0xffffffff806ce6bc unlocked : 0xffffffff806ceb1f
owner field : 0xfffffe822f607420 wait/spin: 1/0
Turnstile chain at 0xffffffff814c5140.
=> Turnstile at 0xfffffe8208bfe7e8 (wrq=0xfffffe8208bfe808, rdq=0xfffffe8208bfe8
18).
=> 0 waiting readers:
=> 14 waiting writers: 0xfffffe810f3ce880 0xfffffe820b650b00 0xfffffe821a1fa660 0xfffffe821b925560 0xfffffe822297fb00 0xfffffe822a314200 0xfffffe82188ea780 0xfffffe8215dea600 0xfffffe82179cb520 0xfffffe8218bbb420 0xfffffe82232b56a0 0xfffffe822d6c49c0 0xfffffe8219973440 0xfffffe822d8cd560
[Locks tracked through CPUs]
Locks held on CPU 0:
Lock 0 (initialized at main)
lock address : 0xffffffff814c3940 type : spin
initialized : 0xffffffff80d77491
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 5
current cpu : 0 last held: 0
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe822f607420
last locked* : 0xffffffff806ce6c6 unlocked : 0xffffffff809ba51e
curcpu holds : 2 wanted by: 000000000000000000
Lock 1 (initialized at com_attach_subr)
lock address : 0xfffffe810f27b1e0 type : spin
initialized : 0xffffffff805f4924
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 0
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe822f5eb440
last locked* : 0xffffffff805f3e84 unlocked : 0xffffffff805f4063
owner field : 0x0000000000010800 wait/spin: 0/1
db{0}>
From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, port-amd64-maintainer@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, he@NetBSD.org
Cc:
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network interfaces in use
Date: Thu, 17 May 2018 14:28:54 -0400
On May 17, 2:30pm, he@NetBSD.org (Havard Eidnes) wrote:
-- Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx net
| The following reply was made to PR port-amd64/53155; it has been noted by GNATS.
|
| From: Havard Eidnes <he@NetBSD.org>
| To: gnats-bugs@NetBSD.org, hannken@eis.cs.tu-bs.de
| Cc: port-amd64-maintainer@netbsd.org
| Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
| network interfaces in use
| Date: Thu, 17 May 2018 16:28:56 +0200 (CEST)
Try this?
christos
Index: if_bnx.c
===================================================================
RCS file: /cvsroot/src/sys/dev/pci/if_bnx.c,v
retrieving revision 1.64
diff -u -u -r1.64 if_bnx.c
--- if_bnx.c 8 May 2018 04:11:09 -0000 1.64
+++ if_bnx.c 17 May 2018 18:28:21 -0000
@@ -3942,6 +3942,7 @@
struct ifnet *ifp = &sc->bnx_ec.ec_if;
struct bnx_pkt *pkt;
int i, s;
+ bool got = false;
for (i = 0; i < 4; i++) { /* magic! */
pkt = pool_get(bnx_tx_pool, PR_WAITOK);
@@ -3961,17 +3962,20 @@
TAILQ_INSERT_TAIL(&sc->tx_free_pkts, pkt, pkt_entry);
sc->tx_pkt_count++;
mutex_exit(&sc->tx_pkt_mtx);
+ got = true;
}
mutex_enter(&sc->tx_pkt_mtx);
CLR(sc->bnx_flags, BNX_ALLOC_PKTS_FLAG);
mutex_exit(&sc->tx_pkt_mtx);
- /* fire-up TX now that allocations have been done */
- s = splnet();
- if (!IFQ_IS_EMPTY(&ifp->if_snd))
- bnx_start(ifp);
- splx(s);
+ if (got) {
+ /* fire-up TX now that allocations have been done */
+ s = splnet();
+ if (!IFQ_IS_EMPTY(&ifp->if_snd))
+ bnx_start(ifp);
+ splx(s);
+ }
return;
@@ -3979,6 +3983,10 @@
bus_dmamap_destroy(sc->bnx_dmatag, pkt->pkt_dmamap);
put:
pool_put(bnx_tx_pool, pkt);
+ mutex_enter(&sc->tx_pkt_mtx);
+ CLR(sc->bnx_flags, BNX_ALLOC_PKTS_FLAG);
+ mutex_exit(&sc->tx_pkt_mtx);
+
return;
}
@@ -4949,8 +4957,8 @@
if (sc->tx_pkt_count <= TOTAL_TX_BD &&
!ISSET(sc->bnx_flags, BNX_ALLOC_PKTS_FLAG)) {
- workqueue_enqueue(sc->bnx_wq, &bnx_wk, NULL);
SET(sc->bnx_flags, BNX_ALLOC_PKTS_FLAG);
+ workqueue_enqueue(sc->bnx_wq, &bnx_wk, NULL);
}
mutex_exit(&sc->tx_pkt_mtx);
From: Havard Eidnes <he@NetBSD.org>
To: christos@zoulas.com
Cc: gnats-bugs@NetBSD.org, port-amd64-maintainer@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Thu, 17 May 2018 21:26:02 +0200 (CEST)
> Try this?
Thanks, did that, but it didn't fix the problem.
Additionally, when I tried breaking into DDB , all I got was
"fata" (first part of the text I usually get) and it hung,
requiring a hard reset. Luckily we have remote management of the
host so it'll be ready for another try shortly.
I did a second try, and this time I got into DDB; here's the
console log:
fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip 0xffffffff8021ff65 cs 0x8 rflags 0x202 cr2 0x7ece618de000
curlwp 0xfffffe822f5eb440 pid 0.6 lowest kstack 0xffff8000ae6972c0
Stopped in pid 0.6 (system) at netbsd:breakpoint+0x5: leave
db{0}> ps
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
1212 > 1 7 0 0 fffffe8220810940 ssh
829 1 3 1 80 fffffe821cd5dac0 sh wait
490 1 3 5 80 fffffe821e5d40e0 tcsh ttyraw
745 1 3 6 80 fffffe821c260680 systat ttyraw
1251 1 3 5 80 fffffe821c260260 tcsh pause
861 1 3 7 80 fffffe821c260aa0 tcsh pause
860 1 3 6 80 fffffe821dad51c0 ssh select
1243 1 3 6 80 fffffe821dd75240 sh wait
602 1 3 6 80 fffffe821dc2d9e0 ssh select
985 1 3 6 80 fffffe821dd75660 sh wait
1240 1 3 0 80 fffffe821cb1b160 ssh select
1239 1 3 6 80 fffffe821dd75a80 sh wait
854 1 3 5 80 fffffe821e10d140 ssh select
1237 1 3 6 80 fffffe821dce7220 sh wait
340 1 3 0 80 fffffe821e26d540 ssh select
979 1 3 6 80 fffffe821dce7640 sh wait
338 1 3 0 80 fffffe82223d3920 ssh select
721 1 3 0 80 fffffe821fd2f2a0 sh wait
336 1 3 2 80 fffffe821e8f64e0 ssh select
591 1 3 2 80 fffffe821dce7a60 sh wait
1230 1 3 0 80 fffffe821d394200 conserver select
1229 1 3 4 80 fffffe821d394620 ssh select
844 1 3 6 80 fffffe821d394a40 ssh select
459 1 3 3 80 fffffe821d87d1e0 sh wait
457 1 3 1 80 fffffe821d87da20 sh wait
1223 1 3 0 80 fffffe821dad55e0 ssh select
838 1 3 2 80 fffffe821dad5a00 sh wait
580 1 3 7 80 fffffe821dc2d5c0 ssh select
322 1 3 1 80 fffffe821c99b180 sh wait
320 1 3 0 80 fffffe821c99b9c0 ssh select
830 1 3 5 80 fffffe821cb1b580 sh wait
1211 1 3 1 80 fffffe821e10d560 ssh select
954 1 3 0 80 fffffe821e10d980 sh wait
1207 1 3 7 80 fffffe821e26d960 conserver select
950 1 3 2 80 fffffe821d3f3100 ssh select
1077 1 3 2 80 fffffe821d3f3520 sh wait
692 1 3 0 80 fffffe821d3f3940 ssh select
306 1 3 5 80 fffffe821e5d4500 sh wait
945 1 3 4 80 fffffe821e5d4920 ssh select
304 1 3 0 80 fffffe821e8f60c0 ssh select
942 1 3 4 80 fffffe821e8f6900 sh wait
813 1 3 4 80 fffffe82214745c0 ssh select
556 1 3 0 80 fffffe821d97b0a0 sh wait
299 1 3 7 80 fffffe821d97b4c0 sh wait
298 1 3 7 80 fffffe821d97b8e0 ssh select
297 1 3 6 80 fffffe821ed7d080 sh wait
680 1 3 0 80 fffffe821ed7d4a0 ssh select
1063 1 3 6 80 fffffe821ed7d8c0 sh wait
550 1 3 5 80 fffffe821dd63060 ssh select
933 1 3 6 80 fffffe821dd63480 sh wait
804 1 3 3 80 fffffe821dd638a0 ssh select
803 1 3 6 80 fffffe821e8b7040 sh wait
674 1 3 7 80 fffffe821e8b7460 ssh select
289 1 3 0 80 fffffe821e8b7880 sh wait
288 1 3 0 80 fffffe821e163020 ssh select
1183 1 3 5 80 fffffe821e163440 ssh select
1182 1 3 4 80 fffffe821e163860 sh wait
925 1 3 5 80 fffffe822023e6e0 sh wait
1180 1 3 2 80 fffffe8220116000 conserver select
1051 1 3 1 80 fffffe82216345a0 ssh select
922 1 3 6 80 fffffe8220116420 sh wait
793 1 3 4 80 fffffe8220116840 conserver select
918 1 3 7 80 fffffe821f6f5360 conserver select
1045 1 3 0 80 fffffe82218ba4e0 ssh select
660 1 3 3 80 fffffe821f6f5780 sh wait
531 1 3 0 80 fffffe821f6f5ba0 conserver select
402 1 3 7 80 fffffe821fa76340 ssh select
529 1 3 2 80 fffffe821fa76760 ssh select
400 1 3 7 80 fffffe821fa76b80 sh wait
911 1 3 0 80 fffffe821f4bf320 ssh select
782 1 3 4 80 fffffe821f4bf740 sh wait
1037 1 3 0 80 fffffe821f4bfb60 sh wait
908 1 3 1 80 fffffe821eb92300 ssh select
267 1 3 4 80 fffffe821eb92720 ssh select
266 1 3 3 80 fffffe821eb92b40 sh wait
265 1 3 0 80 fffffe82200462e0 sh wait
264 1 3 6 80 fffffe8220046700 ssh select
775 1 3 1 80 fffffe8220046b20 sh wait
902 1 3 3 80 fffffe822023e2c0 ssh select
900 1 3 6 80 fffffe822023eb00 sh wait
386 1 3 4 80 fffffe821fd2f6c0 ssh select
385 1 3 4 80 fffffe821fd2fae0 sh wait
384 1 3 2 80 fffffe822057e280 ssh select
762 1 3 2 80 fffffe822057e6a0 sh wait
827 1 3 7 80 fffffe822057eac0 ssh select
889 1 3 1 80 fffffe82207e6260 sh wait
824 1 3 0 80 fffffe82207e6680 ssh select
823 1 3 2 80 fffffe82207e6aa0 sh wait
566 1 3 5 80 fffffe82208fe240 ssh select
949 1 3 0 80 fffffe82208fe660 ssh select
500 1 3 5 80 fffffe82208fea80 sh wait
755 1 3 7 80 fffffe822099d220 sh wait
370 1 3 5 80 fffffe822099d640 ssh select
753 1 3 0 80 fffffe822099da60 sh wait
368 1 3 1 80 fffffe821ff42200 conserver select
751 1 3 5 80 fffffe821ff42620 ssh select
558 1 3 0 80 fffffe821ff42a40 ssh select
877 1 3 1 80 fffffe82211651e0 sh wait
620 1 3 6 80 fffffe8221165600 ssh select
170 1 3 0 80 fffffe822d985980 sh wait
171 1 3 2 80 fffffe8221165a20 ssh select
361 1 3 7 80 fffffe822d65e5c0 sh wait
296 1 3 5 80 fffffe822079b1c0 ssh select
935 1 3 2 80 fffffe822c1fc200 sh wait
614 1 3 1 80 fffffe822079b5e0 ssh select
741 1 3 3 80 fffffe822c3a6660 sh wait
868 1 3 4 80 fffffe822079ba00 ssh select
867 1 3 4 80 fffffe82214741a0 sh wait
161 1 3 2 80 fffffe82214749e0 sh wait
160 1 3 7 80 fffffe8221634180 ssh select
798 1 3 3 80 fffffe82216349c0 ssh select
733 1 3 6 80 fffffe8221afd160 ssh select
796 1 3 3 80 fffffe8221afd580 sh wait
923 1 3 4 80 fffffe8221afd9a0 sh wait
538 1 3 7 80 fffffe822c8105e0 ssh select
665 1 3 6 80 fffffe82219fc140 sh wait
920 1 3 1 80 fffffe82288c56a0 ssh select
855 1 3 7 80 fffffe82219fc560 sh wait
726 1 3 1 80 fffffe82219fc980 ssh select
917 1 3 1 80 fffffe82211da120 sh wait
276 1 3 2 80 fffffe82211da540 sh wait
595 1 3 0 80 fffffe82211da960 ssh select
210 1 3 5 80 fffffe8220810100 sh wait
527 1 3 1 80 fffffe82223d30e0 conserver select
846 1 3 1 80 fffffe82223d3500 ssh select
909 1 3 3 80 fffffe82243a3780 sh wait
716 1 3 0 80 fffffe822d9659c0 ssh select
138 1 3 3 80 fffffe82218ba0c0 sh wait
136 1 3 3 80 fffffe82218ba900 ssh select
839 1 3 0 80 fffffe8222b380a0 sh wait
710 1 3 0 80 fffffe8222b384c0 ssh select
581 1 3 6 80 fffffe8222b388e0 sh wait
516 1 3 4 80 fffffe822d8e6160 ssh select
643 1 3 7 80 fffffe822594f340 ssh select
194 1 3 3 80 fffffe822646c080 sh wait
193 1 3 1 80 fffffe822646c4a0 ssh select
192 1 3 0 80 fffffe822646c8c0 sh wait
856 1 3 4 80 fffffe8227d1b060 sh wait
73 1 3 0 80 fffffe822d8e69a0 ssh select
72 1 3 1 80 fffffe8227d1b480 sh wait
75 1 3 5 80 fffffe8227d1b8a0 ssh select
74 1 3 5 80 fffffe822d763040 sh wait
579 1 3 0 80 fffffe822d763460 ssh select
561 1 3 6 80 fffffe82269e6b60 sh wait
845 1 3 1 80 fffffe810f2352c0 ssh select
654 1 3 7 80 fffffe8223d7b840 sh wait
871 1 3 6 80 fffffe822d985140 ssh select
232 1 3 7 80 fffffe822d985560 sh wait
178 1 3 6 80 fffffe822dbc9540 ssh select
563 1 3 0 80 fffffe822dbc9120 sh wait
676 1 3 4 80 fffffe822d763880 ssh select
226 1 3 2 80 fffffe822bcfb020 sh wait
675 1 3 4 80 fffffe822c1fc620 ssh select
534 1 3 5 80 fffffe822bcfb440 sh wait
541 1 3 3 80 fffffe822bcfb860 conserver select
859 1 3 1 80 fffffe8223d7b000 conserver select
732 1 3 2 80 fffffe822dc740e0 getty ttyraw
759 1 3 5 80 fffffe8223d7b420 conserver select
176 1 3 1 80 fffffe822c8101c0 conserver select
524 1 3 7 80 fffffe82243a3360 cron nanoslp
169 1 3 4 80 fffffe82243a3ba0 python2.7 select
549 1 3 0 80 fffffe822594f760 xterm select
853 1 3 0 80 fffffe82288c5280 tcsh pause
663 1 3 1 80 fffffe82288c5ac0 sshd select
895 1 3 5 80 fffffe822d65e1a0 sshd select
734 1 3 2 80 fffffe822594fb80 xterm select
570 1 3 0 80 fffffe822d9655a0 tcsh pause
697 1 3 5 80 fffffe822cdf5a60 sshd select
502 1 3 0 80 fffffe822c3a6a80 sshd select
107 1 3 0 80 fffffe82273e8720 inetd kqueue
97 1 3 6 80 fffffe82271af2e0 nginx kqueue
96 1 3 0 80 fffffe82271af700 nginx pause
647 1 3 0 80 fffffe8226ea06c0 trivial-rewrite kqueue
601 1 3 0 80 fffffe8228feb680 qmgr kqueue
702 1 3 6 80 fffffe822d65e9e0 pickup kqueue
763 1 3 2 80 fffffe8226ea0ae0 master kqueue
474 1 3 7 80 fffffe8228febaa0 sshd select
463 1 3 4 80 fffffe822c3a6240 tac_plus select
428 1 3 6 80 fffffe822cdf5220 powerd kqueue
422 1 3 7 80 fffffe822cdf5640 ntpd pause
337 4 3 7 80 fffffe822c1fca40 dhcpd parked
337 3 3 7 80 fffffe822c810a00 dhcpd kqueue
337 2 3 7 0 fffffe822d8e6580 dhcpd biowait
337 1 3 0 80 fffffe822c7601e0 dhcpd parked
212 1 2 7 0 fffffe810f33a300 syslogd
1 1 3 0 80 fffffe810f572900 init wait
0 123 3 3 200 fffffe822c760a20 npfgc-0 xchicv
0 122 3 3 200 fffffe822d965180 ipmi0 ipmi0
0 121 3 7 200 fffffe822dbf2940 physiod physiod
0 120 3 4 200 fffffe822dbc9960 aiodoned aiodoned
0 119 3 5 200 fffffe822dbf2100 ioflush tstile
0 118 3 0 200 fffffe822dbf2520 pgdaemon pgdaemon
0 115 3 0 280 fffffe822dc74500 spkr1 bellcv
0 114 3 5 280 fffffe822dc74920 audiomix play
0 113 3 5 280 fffffe810edc32a0 audiorec record
0 112 3 5 200 fffffe810f5720c0 atapibus0 sccomp
0 110 3 0 200 fffffe810f33ab40 usb1 usbevt
0 109 3 6 200 fffffe810f3332e0 usb3 usbevt
0 108 3 5 200 fffffe810f333700 usb2 usbevt
0 107 3 0 200 fffffe810f33a720 usb0 usbevt
0 106 3 5 200 fffffe810f333b20 usb5 usbevt
0 105 3 5 200 fffffe810f5724e0 usb4 usbevt
0 104 3 0 200 fffffe810f50b0a0 rt_free rt_free
0 103 3 0 200 fffffe810f50b4c0 unpgc unpgc
0 102 3 5 200 fffffe810f50b8e0 key_timehandler key_timehandler
0 101 3 7 200 fffffe810f494080 icmp6_wqinput/7 icmp6_wqinput
0 100 3 6 200 fffffe810f4944a0 icmp6_wqinput/6 icmp6_wqinput
0 99 3 5 200 fffffe810f4948c0 icmp6_wqinput/5 icmp6_wqinput
0 98 3 4 200 fffffe810f4d5060 icmp6_wqinput/4 icmp6_wqinput
0 97 3 3 200 fffffe810f4d5480 icmp6_wqinput/3 icmp6_wqinput
0 96 3 2 200 fffffe810f4d58a0 icmp6_wqinput/2 icmp6_wqinput
0 95 3 1 200 fffffe810f3c4040 icmp6_wqinput/1 icmp6_wqinput
0 94 3 0 200 fffffe810f3c4460 icmp6_wqinput/0 icmp6_wqinput
0 93 3 7 200 fffffe810f3c4880 nd6_timer nd6_timer
0 92 3 7 200 fffffe810f38c020 icmp_wqinput/7 icmp_wqinput
0 91 3 6 200 fffffe810f38c440 icmp_wqinput/6 icmp_wqinput
0 90 3 5 200 fffffe810f38c860 icmp_wqinput/5 icmp_wqinput
0 89 3 4 200 fffffe810f385000 icmp_wqinput/4 icmp_wqinput
0 88 3 3 200 fffffe810f385420 icmp_wqinput/3 icmp_wqinput
0 87 3 2 200 fffffe810f385840 icmp_wqinput/2 icmp_wqinput
0 86 3 1 200 fffffe810f36e360 icmp_wqinput/1 icmp_wqinput
0 85 3 0 200 fffffe810f36e780 icmp_wqinput/0 icmp_wqinput
0 84 3 5 200 fffffe810f36eba0 rt_timer rt_timer
0 83 3 0 200 fffffe810f311b60 vmem_rehash vmem_rehash
0 82 3 5 200 fffffe810f318340 coretemp3 coretemp3
0 81 3 4 200 fffffe810f318760 coretemp2 coretemp2
0 80 3 5 200 fffffe810f318b80 coretemp1 coretemp1
0 79 3 7 200 fffffe810f311320 coretemp0 coretemp0
0 78 3 7 200 fffffe810f311740 mfi0 mfi0
0 69 3 5 200 fffffe810f2356e0 atabus1 atath
0 68 3 3 200 fffffe810f235b00 atabus0 atath
0 66 3 0 200 fffffe810edc36c0 scsibus0 sccomp
0 65 3 0 200 fffffe810edc3ae0 usbtask-dr usbtsk
0 64 3 0 200 fffffe810ed02280 usbtask-hc usbtsk
0 63 3 7 200 fffffe810ed026a0 bnx3 bnx3
0 62 3 6 200 fffffe810ed02ac0 bnx2 bnx2
0 61 3 4 200 fffffe810eb79260 bnx1 bnx1
0 60 3 1 200 fffffe810eb79680 bnx0 bnx0
0 59 3 5 200 fffffe810eb79aa0 ipmi ipmipoll
0 58 3 7 200 fffffe810eaca240 xcall/7 xcall
0 57 1 7 200 fffffe810eaca660 softser/7
0 56 1 7 200 fffffe810eacaa80 softclk/7
0 55 1 7 200 fffffe810eac3220 softbio/7
0 54 1 7 200 fffffe810eac3640 softnet/7
0 > 53 7 7 201 fffffe810eac3a60 idle/7
0 52 3 6 200 fffffe810ea72200 xcall/6 xcall
0 51 1 6 200 fffffe810ea72620 softser/6
0 50 1 6 200 fffffe810ea72a40 softclk/6
0 49 1 6 200 fffffe810ea6b1e0 softbio/6
0 48 1 6 200 fffffe810ea6b600 softnet/6
0 > 47 7 6 201 fffffe810ea6ba20 idle/6
0 46 3 5 200 fffffe810ea221c0 xcall/5 xcall
0 45 1 5 200 fffffe810ea225e0 softser/5
0 44 1 5 200 fffffe810ea22a00 softclk/5
0 43 1 5 200 fffffe810ea0b1a0 softbio/5
0 42 1 5 200 fffffe810ea0b5c0 softnet/5
0 > 41 7 5 201 fffffe810ea0b9e0 idle/5
0 40 3 4 200 fffffe810e9ca180 xcall/4 xcall
0 39 1 4 200 fffffe810e9ca5a0 softser/4
0 38 1 4 200 fffffe810e9ca9c0 softclk/4
0 37 1 4 200 fffffe810e9b3160 softbio/4
0 36 1 4 200 fffffe810e9b3580 softnet/4
0 > 35 7 4 201 fffffe810e9b39a0 idle/4
0 34 3 3 200 fffffe810e972140 xcall/3 xcall
0 33 1 3 200 fffffe810e972560 softser/3
0 32 1 3 200 fffffe810e972980 softclk/3
0 31 1 3 200 fffffe810e95b120 softbio/3
0 30 1 3 200 fffffe810e95b540 softnet/3
0 > 29 7 3 201 fffffe810e95b960 idle/3
0 28 3 2 200 fffffe810e922100 xcall/2 xcall
0 27 1 2 200 fffffe810e922520 softser/2
0 26 1 2 200 fffffe810e922940 softclk/2
0 25 1 2 200 fffffe810e9030e0 softbio/2
0 24 1 2 200 fffffe810e903500 softnet/2
0 > 23 7 2 201 fffffe810e903920 idle/2
0 22 3 1 200 fffffe810e8ca0c0 xcall/1 xcall
0 21 1 1 200 fffffe810e8ca4e0 softser/1
0 20 1 1 200 fffffe810e8ca900 softclk/1
0 19 1 1 200 fffffe810e8a30a0 softbio/1
0 18 1 1 200 fffffe810e8a34c0 softnet/1
0 > 17 7 1 201 fffffe810e8a38e0 idle/1
0 16 3 1 200 fffffe822dd92080 lnxsyswq lnxsyswq
0 15 3 0 200 fffffe822dd924a0 sysmon smtaskq
0 14 3 7 200 fffffe822dd928c0 pmfsuspend pmfsuspend
0 13 3 2 200 fffffe822e1b1060 pmfevent pmfevent
0 12 3 0 200 fffffe822e1b1480 sopendfree sopendfr
0 11 3 0 200 fffffe822e1b18a0 nfssilly nfssilly
0 10 3 1 200 fffffe822f5d0040 cachegc cachegc
0 9 3 0 200 fffffe822f5d0460 vdrain vdrain
0 8 3 6 200 fffffe822f5d0880 modunload mod_unld
0 7 3 0 200 fffffe822f5eb020 xcall/0 xcall
0 > 6 7 0 200 fffffe822f5eb440 softser/0
0 > 5 7 0 200 fffffe822f5eb860 softclk/0
0 4 1 0 200 fffffe822f607000 softbio/0
0 > 3 7 0 200 fffffe822f607420 softnet/0
0 2 1 0 201 fffffe822f607840 idle/0
0 1 3 3 200 ffffffff81481b80 swapper uvm
db{0}> trace/a fffffe822f5eb440
trace: pid 0 lid 6 at 0xffff8000ae69adf8
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
DDB lost frame for netbsd:Xdoreti, trying 0xffff8000ae69ae70
Xdoreti() at netbsd:Xdoreti
--- interrupt ---
Bad frame pointer: 0xffff800000000000
206:
db{0}> trace/a fffffe822f5eb860
trace: pid 0 lid 5 at 0xffff8000bdab8ba0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
7f8000000000:
db{0}> trace/a fffffe822f607420
trace: pid 0 lid 3 at 0xffff8000ae6960e0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}> show all locks
[Locks tracked through LWPs]
Locks held by an LWP (ssh):
Lock 0 (initialized at uvm_obj_init)
lock address : 0xfffffe822cf2df00 type : sleep/adaptive
initialized : 0xffffffff8096aed9
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 1
current cpu : 0 last held: 0
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe8220810940
last locked* : 0xffffffff8095cf70 unlocked : 0xffffffff80a41ef5
owner field : 0xfffffe8220810940 wait/spin: 1/0
Turnstile chain at 0xffffffff814c50c0.
=> Turnstile at 0xfffffe810f3865a0 (wrq=0xfffffe810f3865c0, rdq=0xfffffe810f3865d0).
=> 0 waiting readers:
=> 1 waiting writers: 0xfffffe822dbf2100
Lock 1 (initialized at pmap_create)
lock address : 0xfffffe8226aa4760 type : sleep/adaptive
initialized : 0xffffffff802463c2
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 0
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe8220810940
last locked* : 0xffffffff80247877 unlocked : 0xffffffff80249014
owner field : 0xfffffe8220810940 wait/spin: 0/0
Turnstile chain at 0xffffffff814c5180.
=> No active turnstile for this lock.
Locks held by an LWP (dhcpd):
Lock 0 (initialized at vcache_alloc)
lock address : 0xfffffe822ce13e58 type : sleep/adaptive
initialized : 0xffffffff80a402e2
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 7
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe822d8e6580
last locked* : 0xffffffff80a579ce unlocked : 0xffffffff80a57a04
owner/count : 0xfffffe822d8e6580 flags : 0x0000000000000004
Turnstile chain at 0xffffffff814c4f70.
=> No active turnstile for this lock.
Locks held by an LWP (npfgc-0):
Lock 0 (initialized at npf_config_init)
lock address : 0xfffffe822ca22f08 type : sleep/adaptive
initialized : 0xffffffff8196d8cf
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 3
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe822c760a20
last locked* : 0xffffffff81972df3 unlocked : 0xffffffff81972e03
owner field : 0xfffffe822c760a20 wait/spin: 0/0
Turnstile chain at 0xffffffff814c50d0.
=> No active turnstile for this lock.
Locks held by an LWP (ioflush):
Lock 0 (initialized at vfs_mount_sysinit)
lock address : 0xffffffff815cc8d8 type : sleep/adaptive
initialized : 0xffffffff80a30a85
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 5
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe822dbf2100
last locked* : 0xffffffff80a30864 unlocked : 0xffffffff80a30966
owner field : 0xfffffe822dbf2100 wait/spin: 0/0
Turnstile chain at 0xffffffff814c5070.
=> No active turnstile for this lock.
[Locks tracked through CPUs]
Locks held on CPU 0:
Lock 0 (initialized at com_attach_subr)
lock address : 0xfffffe810f2731e0 type : spin
initialized : 0xffffffff805f4964
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 0
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe822f5eb440
last locked* : 0xffffffff805f3ec4 unlocked : 0xffffffff805f40a3
owner field : 0x0000000000010800 wait/spin: 0/1
db{0}>
Regards,
- Havard
From: Ryota Ozaki <ozaki-r@netbsd.org>
To: Havard Eidnes <he@netbsd.org>
Cc: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>, port-amd64-maintainer@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network
interfaces in use
Date: Fri, 18 May 2018 18:07:29 +0900
On Fri, May 18, 2018 at 4:26 AM Havard Eidnes <he@netbsd.org> wrote:
> > Try this?
> Thanks, did that, but it didn't fix the problem.
> Additionally, when I tried breaking into DDB , all I got was
> "fata" (first part of the text I usually get) and it hung,
> requiring a hard reset. Luckily we have remote management of the
> host so it'll be ready for another try shortly.
> I did a second try, and this time I got into DDB; here's the
> console log:
Please add '/t' option to show all locks next time. It provides stack traces
of related LWPs in addition.
BTW softser/0 I suspected looks just a trace of DDB, so it's not a stopper
of pserialize_perform...
Thanks,
ozaki-r
From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: port-amd64-maintainer@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Sun, 27 May 2018 23:22:16 +0200 (CEST)
Hi,
another wedge. This time with "show all locks/t".
This time I did not assign an IP address to the bnx3 interface
which I took up right before it wedged.
I have the matching kernel core dump.
Regards,
- Havard
------------------------------
db{0}> show all locks/t
[Locks tracked through LWPs]
Locks held by an LWP (npfgc-0):
Lock 0 (initialized at npf_config_init)
lock address : 0xfffffe822d7e54c8 type : sleep/adaptive
initialized : 0xffffffff8196d8cf
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 5
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe822cf3d1c0
last locked* : 0xffffffff81972df3 unlocked : 0xffffffff81972e03
owner field : 0xfffffe822cf3d1c0 wait/spin: 0/0
Turnstile chain at 0xffffffff814c5050.
=> No active turnstile for this lock.
trace: pid 0 lid 123 at 0xffff8000bcec6eb0
sleepq_block() at netbsd:sleepq_block+0x97
cv_wait() at netbsd:cv_wait+0x127
xc_wait() at netbsd:xc_wait+0x7c
pserialize_perform() at netbsd:pserialize_perform+0x102
npf_conn_gc() at npf:npf_conn_gc+0x1b5
npf_worker() at npf:npf_worker+0x4b
Locks held by an LWP (ioflush):
Lock 0 (initialized at wapbl_start)
lock address : 0xfffffe810f4fd480 type : sleep/adaptive
initialized : 0xffffffff80a4a13c
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 2
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe822dc08940
last locked* : 0xffffffff80a480ce unlocked : 0xffffffff8091396b
owner/count : 0xfffffe822dc08940 flags : 0x0000000000000004
Turnstile chain at 0xffffffff814c4fc0.
=> No active turnstile for this lock.
trace: pid 0 lid 119 at 0xffff8000bcceecf0
sleepq_block() at netbsd:sleepq_block+0x97
cv_wait() at netbsd:cv_wait+0x127
biowait() at netbsd:biowait+0x61
wapbl_buffered_flush() at netbsd:wapbl_buffered_flush+0xc6
wapbl_buffered_write() at netbsd:wapbl_buffered_write+0x69
wapbl_circ_write() at netbsd:wapbl_circ_write+0xa7
wapbl_flush() at netbsd:wapbl_flush+0x386
ffs_sync() at netbsd:ffs_sync+0x220
VFS_SYNC() at netbsd:VFS_SYNC+0x35
sched_sync() at netbsd:sched_sync+0x91
[Locks tracked through CPUs]
Locks held on CPU 0:
Lock 0 (initialized at com_attach_subr)
lock address : 0xfffffe810f2731e0 type : spin
initialized : 0xffffffff805f4964
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 0
current lwp : 0xfffffe822f5eb440 last held: 0xfffffe822f5eb440
last locked* : 0xffffffff805f3ec4 unlocked : 0xffffffff805f40a3
owner field : 0x0000000000010800 wait/spin: 0/1
trace: pid 0 lid 6 at 0xffff8000ae69ad18
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
splraise() at netbsd:splraise
_kernel_lock() at netbsd:_kernel_lock+0x129
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x16
Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff8000ae69b0f0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}>
db{0}> ps
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
28127 1 3 0 80 fffffe821b45c660 sshd select
3765 1 3 0 80 fffffe821c860980 sshd select
10510 1 3 0 80 fffffe820c1a95a0 sshd select
12095 1 3 0 80 fffffe8201ea4880 sshd select
24917 1 3 2 80 fffffe82143e10c0 sshd select
14706 1 3 2 80 fffffe81f95ab000 sshd select
24718 1 3 4 80 fffffe822d981420 tcsh ttyraw
14768 1 3 6 80 fffffe8127c869c0 pickup kqueue
17173 1 3 7 80 fffffe821b7d1300 tcsh ttyraw
22683 1 3 0 80 fffffe821e368720 tcsh pause
6496 1 3 2 80 fffffe81fdf25980 xterm select
8713 1 3 4 80 fffffe8223f352e0 tcsh pause
18715 1 3 1 80 fffffe8203885760 sshd select
27311 1 3 0 80 fffffe821f2322c0 sshd select
12100 1 3 1 80 fffffe82208de4a0 tcsh ttyraw
19259 1 3 7 80 fffffe820b1e85c0 xterm select
27176 1 3 1 80 fffffe81bf43b260 tcsh pause
17081 1 3 0 80 fffffe82261ef260 sshd select
20646 1 3 0 80 fffffe821e413100 sshd select
12410 1 3 0 80 fffffe821c761640 tcsh ttyraw
17135 1 3 7 80 fffffe821e413520 xterm select
5138 1 3 1 80 fffffe810f791ba0 tcsh pause
4590 1 3 5 80 fffffe82261efaa0 sshd select
14049 1 3 7 80 fffffe81f72641a0 sshd select
13260 1 3 2 80 fffffe8218a44280 ssh select
12084 1 3 5 80 fffffe8126a67a60 tcsh pause
20039 1 3 0 80 fffffe8214096100 xterm select
12456 1 3 5 80 fffffe820b1e81a0 tcsh pause
16421 1 3 3 80 fffffe81afe42600 sshd select
2132 1 3 5 80 fffffe8200185120 sshd select
28308 1 3 6 80 fffffe81fdf25140 ssh select
153 1 3 6 80 fffffe81cfc24240 sh wait
29602 1 3 6 80 fffffe81f77cd640 ssh select
26803 1 3 6 80 fffffe820c1a99c0 sh wait
25619 1 3 5 80 fffffe822d908120 ssh select
24644 1 3 6 80 fffffe821acfb480 sh wait
28650 1 3 6 80 fffffe82208de8c0 ssh select
28765 1 3 6 80 fffffe822d7f8140 sh wait
26866 1 3 0 80 fffffe8218a446a0 ssh select
28190 1 3 6 80 fffffe821a21e600 sh wait
23270 1 3 0 80 fffffe820f1bc640 ssh select
29000 1 3 0 80 fffffe820bb76b60 sh wait
556 1 3 0 80 fffffe822478cb00 ssh select
27618 1 3 0 80 fffffe81ffea40e0 sh wait
25866 1 3 0 80 fffffe821c341260 ssh select
29240 1 3 0 80 fffffe82222d0a00 sh wait
27689 1 3 0 80 fffffe82206a4340 ssh select
2714 1 3 0 80 fffffe81f95db780 sh wait
1755 1 3 5 80 fffffe821d1b00e0 ssh select
27303 1 3 6 80 fffffe821fb47060 sh wait
25810 1 3 6 80 fffffe821f3a70c0 ssh select
28383 1 3 6 80 fffffe81f77cd220 sh wait
21350 1 3 6 80 fffffe8219cd4740 ssh select
24176 1 3 6 80 fffffe8201ea4040 sh wait
25377 1 3 0 80 fffffe82251f3280 ssh select
28753 1 3 6 80 fffffe8208905280 sh wait
28152 1 3 0 80 fffffe820ad5b080 ssh select
22480 1 3 6 80 fffffe820bb76320 sh wait
397 1 3 6 80 fffffe8208905ac0 ssh select
27414 1 3 6 80 fffffe82134910a0 sh wait
25934 1 3 1 80 fffffe82102b8500 ssh select
1603 1 3 0 80 fffffe81ad7c26a0 sh wait
28109 1 3 0 80 fffffe822d981000 ssh select
29687 1 3 0 80 fffffe8214096940 sh wait
22723 1 3 7 80 fffffe821630a060 ssh select
936 1 3 7 80 fffffe81f9e538a0 sh wait
28299 1 3 7 80 fffffe822c2c2860 ssh select
25799 1 3 7 80 fffffe821a3b8b20 sh wait
25713 1 3 6 80 fffffe81be3c16c0 ssh select
28263 1 3 6 80 fffffe82207b6ba0 sh wait
28024 1 3 6 80 fffffe821a21e1e0 ssh select
27324 1 3 6 80 fffffe81f95dbba0 sh wait
23843 1 3 6 80 fffffe821e368b40 ssh select
25187 1 3 6 80 fffffe8220b67760 sh wait
26257 1 3 6 80 fffffe822d60d9c0 ssh select
28946 1 3 6 80 fffffe822d887580 sh wait
29668 1 3 6 80 fffffe821e33b860 ssh select
28070 1 3 6 80 fffffe821f2326e0 sh wait
8919 1 3 5 80 fffffe821a852020 tcsh ttyraw
792 1 3 0 80 fffffe8218a44ac0 tcsh pause
1018 1 3 0 80 fffffe82174cb680 tcsh pause
1284 1 3 0 80 fffffe82174cbaa0 ssh select
1017 1 3 5 80 fffffe821b45c240 sh wait
1143 1 3 4 80 fffffe821b45ca80 xterm select
563 1 3 5 80 fffffe8219b27220 tcsh pause
1076 1 3 1 80 fffffe8219b27640 sshd select
626 1 3 5 80 fffffe8219b27a60 sshd select
272 1 3 0 80 fffffe8218a621e0 xterm select
1086 1 3 6 80 fffffe821af99b80 tcsh pause
893 1 3 3 80 fffffe82184f89a0 sshd select
639 1 3 0 80 fffffe821c0181a0 sshd select
884 1 3 4 80 fffffe8218bb2a00 ssh select
371 1 3 4 80 fffffe8218bb25e0 sh wait
878 1 3 1 80 fffffe821d22e280 ssh select
877 1 3 1 80 fffffe821a2e9200 sh wait
1260 1 3 7 80 fffffe821a2e9620 ssh select
747 1 3 2 80 fffffe821d7af100 sh wait
354 1 3 3 80 fffffe8218ff04c0 ssh select
481 1 3 0 80 fffffe821a2e9a40 sh wait
1118 1 3 1 80 fffffe8218a62600 ssh select
989 1 3 4 80 fffffe8218a62a20 conserver select
988 1 3 1 80 fffffe8218bb21c0 sh wait
601 1 3 3 80 fffffe821c0185c0 ssh select
984 1 3 0 80 fffffe82196e11a0 ssh select
855 1 3 3 80 fffffe82196e15c0 sh wait
598 1 3 6 80 fffffe82196e19e0 sh wait
597 1 3 7 80 fffffe821af99340 ssh select
1236 1 3 3 80 fffffe821a3a7180 ssh select
467 1 3 5 80 fffffe821a3a75a0 sh wait
338 1 3 7 80 fffffe821a3a79c0 ssh select
1233 1 3 1 80 fffffe82184f8160 sh wait
336 1 3 4 80 fffffe82184f8580 sh wait
1230 1 3 6 80 fffffe821be405e0 ssh select
844 1 3 5 80 fffffe821c064140 sh wait
459 1 3 4 80 fffffe821c064560 ssh select
458 1 3 2 80 fffffe821c064980 sh wait
457 1 3 3 80 fffffe8219be1120 ssh select
456 1 3 7 80 fffffe8219be1540 ssh select
1222 1 3 6 80 fffffe8219be1960 sh wait
1220 1 3 0 80 fffffe821d7af940 sh wait
321 1 3 0 80 fffffe821af99760 ssh select
320 1 3 0 80 fffffe821d7af520 sh wait
957 1 3 4 80 fffffe821a0110e0 conserver select
954 1 3 0 80 fffffe821c8b5320 ssh select
953 1 3 0 80 fffffe821a011500 sh wait
952 1 3 5 80 fffffe821a011920 ssh select
566 1 3 7 80 fffffe821a1424e0 ssh select
949 1 3 5 80 fffffe821a142900 sh wait
1204 1 3 7 80 fffffe8218ff00a0 sh wait
306 1 3 4 80 fffffe8218ff08e0 ssh select
1201 1 3 4 80 fffffe821ab0b080 sh wait
304 1 3 6 80 fffffe821ab0b4a0 ssh select
943 1 3 1 80 fffffe821a9b8b00 ssh select
1198 1 3 6 80 fffffe821ab0b8c0 sh wait
1197 1 3 1 80 fffffe821acfb060 sh wait
1182 1 3 4 80 fffffe821b9b9360 conserver select
797 1 3 5 80 fffffe821b7d1b40 ssh select
540 1 3 0 80 fffffe821b9b9780 sh wait
539 1 3 0 80 fffffe821b9b9ba0 conserver select
918 1 3 0 80 fffffe821d22e6a0 ssh select
917 1 3 0 80 fffffe821c341aa0 sh wait
1044 1 3 4 80 fffffe821c341680 conserver select
1041 1 3 0 80 fffffe821c761220 ssh select
400 1 3 0 80 fffffe821d107a80 sh wait
911 1 3 4 80 fffffe821c8b1200 ssh select
1038 1 3 0 80 fffffe821c8b5740 sh wait
781 1 3 4 80 fffffe821c8b5b60 conserver select
267 1 3 4 80 fffffe821b7d1720 ssh select
264 1 3 4 80 fffffe821cc7e2e0 sh wait
265 1 3 6 80 fffffe821cc7e700 ssh select
1415 1 3 6 80 fffffe821cc7eb20 sh wait
774 1 3 7 80 fffffe821a9b82c0 ssh select
901 1 3 0 80 fffffe821a9b86e0 sh wait
386 1 3 2 80 fffffe821bd2a6c0 ssh select
385 1 3 2 80 fffffe821bd2aae0 sh wait
630 1 3 4 80 fffffe821d107240 ssh select
757 1 3 4 80 fffffe821d107660 sh wait
1009 1 3 6 80 fffffe821c761a60 ssh select
559 1 3 6 80 fffffe821c8b1620 ssh select
814 1 3 0 80 fffffe821c8b1a40 sh wait
813 1 3 6 80 fffffe821ce4b1e0 sh wait
876 1 3 7 80 fffffe821ce4b600 ssh select
363 1 3 7 80 fffffe821ce4ba20 sh wait
362 1 3 2 80 fffffe821be401c0 conserver select
296 1 3 0 80 fffffe821be40a00 ssh select
613 1 3 1 80 fffffe821c0189e0 ssh select
995 1 3 0 80 fffffe821d462180 sh wait
290 1 3 0 80 fffffe821d4625a0 ssh select
161 1 3 1 80 fffffe821d4629c0 sh wait
160 1 3 4 80 fffffe82251f3ac0 sh wait
671 1 3 0 80 fffffe821e38c160 ssh select
990 1 3 6 80 fffffe821e38c580 ssh select
669 1 3 2 80 fffffe821e38c9a0 sh wait
604 1 3 6 80 fffffe821c860140 sh wait
603 1 3 0 80 fffffe821c860560 ssh select
537 1 3 3 80 fffffe821dba9120 sh wait
137 1 3 3 80 fffffe821f3a74e0 conserver select
136 1 3 0 80 fffffe82209a0320 ssh select
1031 1 3 7 80 fffffe822cf3da00 sh wait
838 1 3 2 80 fffffe821f3a7900 ssh select
517 1 3 4 80 fffffe821ed310a0 ssh select
900 1 3 2 80 fffffe822d887160 sh wait
899 1 3 4 80 fffffe82285b7640 sh wait
194 1 3 6 80 fffffe8224659ae0 ssh select
193 1 3 3 80 fffffe821ed314c0 sh wait
192 1 3 5 80 fffffe821ed318e0 ssh select
473 1 3 0 80 fffffe821e330080 sh wait
73 1 3 1 80 fffffe822d908540 ssh select
569 1 3 7 80 fffffe821e3304a0 sh wait
75 1 3 3 80 fffffe822d7f8980 ssh select
232 1 3 6 80 fffffe821fb47480 sh wait
749 1 3 5 80 fffffe822dc08100 conserver select
170 1 3 6 80 fffffe82285b7a60 conserver select
171 1 3 0 80 fffffe822dc01920 getty ttyraw
169 1 3 4 80 fffffe822d908960 conserver select
646 1 3 7 80 fffffe82207b6780 conserver select
766 1 3 0 80 fffffe82209a0740 cron nanoslp
716 1 3 1 80 fffffe82234a5b40 inetd kqueue
97 1 3 4 80 fffffe822cff7200 nginx kqueue
96 1 3 4 80 fffffe8223f35700 nginx pause
589 1 3 4 80 fffffe822478c6e0 qmgr kqueue
756 1 3 6 80 fffffe82246592a0 master kqueue
408 1 3 0 80 fffffe822d34d5c0 sshd select
469 1 3 0 80 fffffe822705ea80 tac_plus select
438 1 3 0 80 fffffe82285b7220 powerd kqueue
421 1 3 3 80 fffffe822d60d5a0 ntpd pause
380 4 3 4 80 fffffe822cfce600 dhcpd parked
380 3 3 0 80 fffffe822d34d9e0 dhcpd kqueue
380 2 3 7 80 fffffe822d7f8560 dhcpd parked
380 1 3 4 80 fffffe822cff7620 dhcpd parked
83 1 2 6 0 fffffe822cfce1e0 syslogd
1 1 3 7 80 fffffe810f56a900 init wait
0 124 5 0 200 fffffe81ffea4920 (zombie)
0 123 3 5 200 fffffe822cf3d1c0 npfgc-0 xchicv
0 122 3 7 200 fffffe822cfcea20 ipmi0 ipmi0
0 121 3 3 200 fffffe822dc01500 physiod physiod
0 120 3 0 200 fffffe822dc08520 aiodoned aiodoned
0 119 3 2 200 fffffe822dc08940 ioflush biowait
0 118 3 0 200 fffffe822dc010e0 pgdaemon pgdaemon
0 115 3 0 280 fffffe810f56a0c0 spkr1 bellcv
0 114 3 5 280 fffffe810f56a4e0 audiomix play
0 113 3 5 280 fffffe810edc32a0 audiorec record
0 112 3 0 200 fffffe810f33a300 atapibus0 sccomp
0 110 3 1 200 fffffe810f3332e0 usb2 usbevt
0 109 3 3 200 fffffe810f333b20 usb5 usbevt
0 108 3 3 200 fffffe810f333700 usb0 usbevt
0 107 3 1 200 fffffe810f33ab40 usb3 usbevt
0 106 3 1 200 fffffe810f2352c0 usb1 usbevt
0 105 3 3 200 fffffe810f311b60 usb4 usbevt
0 104 3 0 200 fffffe810f99b0a0 rt_free rt_free
0 103 3 0 200 fffffe810f99b4c0 unpgc unpgc
0 102 3 0 200 fffffe810f99b8e0 key_timehandler key_timehandler
0 101 3 7 200 fffffe810f484080 icmp6_wqinput/7 icmp6_wqinput
0 100 3 6 200 fffffe810f4844a0 icmp6_wqinput/6 icmp6_wqinput
0 99 3 5 200 fffffe810f4848c0 icmp6_wqinput/5 icmp6_wqinput
0 98 3 4 200 fffffe810f495060 icmp6_wqinput/4 icmp6_wqinput
0 97 3 3 200 fffffe810f495480 icmp6_wqinput/3 icmp6_wqinput
0 96 3 2 200 fffffe810f4958a0 icmp6_wqinput/2 icmp6_wqinput
0 95 3 1 200 fffffe810f3be040 icmp6_wqinput/1 icmp6_wqinput
0 94 3 0 200 fffffe810f3be460 icmp6_wqinput/0 icmp6_wqinput
0 93 3 0 200 fffffe810f3be880 nd6_timer nd6_timer
0 92 3 7 200 fffffe810f387020 icmp_wqinput/7 icmp_wqinput
0 91 3 6 200 fffffe810f387440 icmp_wqinput/6 icmp_wqinput
0 90 3 5 200 fffffe810f387860 icmp_wqinput/5 icmp_wqinput
0 89 3 4 200 fffffe810f3a8000 icmp_wqinput/4 icmp_wqinput
0 88 3 3 200 fffffe810f3a8420 icmp_wqinput/3 icmp_wqinput
0 87 3 2 200 fffffe810f3a8840 icmp_wqinput/2 icmp_wqinput
0 86 3 1 200 fffffe810f36e360 icmp_wqinput/1 icmp_wqinput
0 85 3 0 200 fffffe810f36e780 icmp_wqinput/0 icmp_wqinput
0 84 3 0 200 fffffe810f36eba0 rt_timer rt_timer
0 83 3 0 200 fffffe810f33a720 vmem_rehash vmem_rehash
0 82 3 7 200 fffffe810f318340 coretemp3 coretemp3
0 81 3 7 200 fffffe810f318760 coretemp2 coretemp2
0 80 3 5 200 fffffe810f318b80 coretemp1 coretemp1
0 79 3 1 200 fffffe810f311320 coretemp0 coretemp0
0 78 3 1 200 fffffe810f311740 mfi0 mfi0
0 69 3 5 200 fffffe810f2356e0 atabus1 atath
0 68 3 5 200 fffffe810f235b00 atabus0 atath
0 66 3 0 200 fffffe810edc36c0 scsibus0 sccomp
0 65 3 3 200 fffffe810edc3ae0 usbtask-dr usbtsk
0 64 3 0 200 fffffe810ed02280 usbtask-hc usbtsk
0 63 3 1 200 fffffe810ed026a0 bnx3 bnx3
0 62 3 7 200 fffffe810ed02ac0 bnx2 bnx2
0 61 3 2 200 fffffe810eb79260 bnx1 bnx1
0 60 3 7 200 fffffe810eb79680 bnx0 bnx0
0 59 3 5 200 fffffe810eb79aa0 ipmi ipmipoll
0 58 3 7 200 fffffe810eaca240 xcall/7 xcall
0 57 1 7 200 fffffe810eaca660 softser/7
0 56 1 7 200 fffffe810eacaa80 softclk/7
0 55 1 7 200 fffffe810eac3220 softbio/7
0 54 1 7 200 fffffe810eac3640 softnet/7
0 > 53 7 7 201 fffffe810eac3a60 idle/7
0 52 3 6 200 fffffe810ea72200 xcall/6 xcall
0 51 1 6 200 fffffe810ea72620 softser/6
0 50 1 6 200 fffffe810ea72a40 softclk/6
0 49 1 6 200 fffffe810ea6b1e0 softbio/6
0 48 1 6 200 fffffe810ea6b600 softnet/6
0 > 47 7 6 201 fffffe810ea6ba20 idle/6
0 46 3 5 200 fffffe810ea221c0 xcall/5 xcall
0 45 1 5 200 fffffe810ea225e0 softser/5
0 44 1 5 200 fffffe810ea22a00 softclk/5
0 43 1 5 200 fffffe810ea0b1a0 softbio/5
0 42 1 5 200 fffffe810ea0b5c0 softnet/5
0 > 41 7 5 201 fffffe810ea0b9e0 idle/5
0 40 3 4 200 fffffe810e9ca180 xcall/4 xcall
0 39 1 4 200 fffffe810e9ca5a0 softser/4
0 38 1 4 200 fffffe810e9ca9c0 softclk/4
0 37 1 4 200 fffffe810e9b3160 softbio/4
0 36 1 4 200 fffffe810e9b3580 softnet/4
0 > 35 7 4 201 fffffe810e9b39a0 idle/4
0 34 3 3 200 fffffe810e972140 xcall/3 xcall
0 33 1 3 200 fffffe810e972560 softser/3
0 32 1 3 200 fffffe810e972980 softclk/3
0 31 1 3 200 fffffe810e95b120 softbio/3
0 30 1 3 200 fffffe810e95b540 softnet/3
0 > 29 7 3 201 fffffe810e95b960 idle/3
0 28 3 2 200 fffffe810e922100 xcall/2 xcall
0 27 1 2 200 fffffe810e922520 softser/2
0 26 1 2 200 fffffe810e922940 softclk/2
0 25 1 2 200 fffffe810e9030e0 softbio/2
0 24 1 2 200 fffffe810e903500 softnet/2
0 > 23 7 2 201 fffffe810e903920 idle/2
0 22 3 1 200 fffffe810e8ca0c0 xcall/1 xcall
0 21 1 1 200 fffffe810e8ca4e0 softser/1
0 20 1 1 200 fffffe810e8ca900 softclk/1
0 19 1 1 200 fffffe810e8a30a0 softbio/1
0 18 1 1 200 fffffe810e8a34c0 softnet/1
0 > 17 7 1 201 fffffe810e8a38e0 idle/1
0 16 3 0 200 fffffe822dd92080 lnxsyswq lnxsyswq
0 15 3 2 200 fffffe822dd924a0 sysmon smtaskq
0 14 3 6 200 fffffe822dd928c0 pmfsuspend pmfsuspend
0 13 3 5 200 fffffe822e1b1060 pmfevent pmfevent
0 12 3 0 200 fffffe822e1b1480 sopendfree sopendfr
0 11 3 7 200 fffffe822e1b18a0 nfssilly nfssilly
0 10 3 0 200 fffffe822f5d0040 cachegc cachegc
0 9 3 0 200 fffffe822f5d0460 vdrain vdrain
0 8 3 0 200 fffffe822f5d0880 modunload mod_unld
0 7 3 0 200 fffffe822f5eb020 xcall/0 xcall
0 > 6 7 0 200 fffffe822f5eb440 softser/0
0 > 5 7 0 200 fffffe822f5eb860 softclk/0
0 4 1 0 200 fffffe822f607000 softbio/0
0 > 3 7 0 200 fffffe822f607420 softnet/0
0 > 2 7 0 201 fffffe822f607840 idle/0
0 1 3 0 200 ffffffff81481b80 swapper uvm
db{0}>
db{0}> trace/a fffffe822f607420
trace: pid 0 lid 3 at 0xffff8000ae6960e0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}> trace/a fffffe822f5eb860
trace: pid 0 lid 5 at 0xffff8000ae63df50
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
?() at 0
db{0}> trace/a fffffe822f5eb440
trace: pid 0 lid 6 at 0xffff8000ae69ad18
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
splraise() at netbsd:splraise
_kernel_lock() at netbsd:_kernel_lock+0x129
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x16
Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff8000ae69b0f0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}>
db{0}> show all callout
hardclock_ticks now: 86821384
ticks wheel arg func
-863 -1/-256 0 pffasttimo
-861 -1/-256 0 pfslowtimo
-861 -1/-256 0 tcp_slowtimo
-851 -1/-256 0 sched_balance
-849 -1/-256 fffffe812d614ad0 tcp_delack
-848 -1/-256 fffffe810f382e10 uhci_poll_hub
-833 -1/-256 0 key_timehandler
-828 -1/-256 ffffffff81481b80 sleepq_timeout
-822 -1/-256 0 nd6_timer
-819 -1/-256 0 rt_timer_timer
-811 -1/-256 ffff80000e836000 bnx_tick
-811 -1/-256 ffff80000e830008 if_slowtimo
-811 -1/-256 ffff80000e833008 if_slowtimo
-811 -1/-256 ffff80000e836008 if_slowtimo
-811 -1/-256 ffff80000e839008 if_slowtimo
-811 -1/-256 ffff80000e830000 bnx_tick
-798 -1/-256 ffff80000e839000 bnx_tick
-784 -1/-256 fffffe822f5d0040 sleepq_timeout
-561 -1/-256 0 vmem_rehash_all_kick
-430 -1/-256 fffffe81f8e8f5d8 tcp_timer_rexmt
13 0/107 fffffe810f372e10 uhci_poll_hub
14 0/129 fffffe810f330e10 uhci_poll_hub
43 0/163 fffffe810eb79aa0 sleepq_timeout
13 0/182 fffffe810f351e10 uhci_poll_hub
50 0/220 fffffe822c048e70 realtimerexpire
30475 1/321 fffffe822c048d68 realtimerexpire
36544 1/344 fffffe8222de5218 realtimerexpire
58821 1/431 fffffe8222de5950 realtimerexpire
511 1/460 fffffe822705ea80 sleepq_timeout
606 1/460 fffffe821f08ba00 tcp_timer_rexmt
608 1/460 fffffe812d614790 tcp_timer_rexmt
969 1/462 fffffe822478c6e0 sleepq_timeout
975 1/462 fffffe82246592a0 sleepq_timeout
1713 1/464 fffffe810ed76188 sme_events_check
1713 1/464 fffffe810f24e708 sme_events_check
1713 1/464 fffffe810f2afec8 sme_events_check
1713 1/464 fffffe810f2af748 sme_events_check
1713 1/464 fffffe810f2f2dc8 sme_events_check
1773 1/465 fffffe810f5711c8 sme_events_check
1821 1/465 fffffe821f3a74e0 sleepq_timeout
1943 1/466 fffffe821be401c0 sleepq_timeout
1976 1/466 fffffe821c341680 sleepq_timeout
2134 1/466 fffffe822cfce1e0 sleepq_timeout
2407 1/467 fffffe82209a0740 sleepq_timeout
2891 1/469 fffffe82285b7a60 sleepq_timeout
3302 1/470 fffffe821b565120 tcp_timer_rexmt
3075 1/470 fffffe8218a62a20 sleepq_timeout
2982 1/470 fffffe821b9b9360 sleepq_timeout
3251 1/471 fffffe822d908960 sleepq_timeout
3409 1/471 fffffe822dc08100 sleepq_timeout
3553 1/472 fffffe821c8b5b60 sleepq_timeout
3685 1/472 fffffe821a0110e0 sleepq_timeout
4352 1/475 fffffe821b9b9ba0 sleepq_timeout
4269 1/475 fffffe822c048c60 realtimerexpire
4641 1/476 fffffe810f2352c0 sleepq_timeout
4652 1/476 fffffe810f333700 sleepq_timeout
4685 1/476 fffffe810f333b20 sleepq_timeout
4906 1/477 fffffe821b466498 tcp_timer_rexmt
4852 1/477 fffffe810f3332e0 sleepq_timeout
4974 1/477 fffffe810f33ab40 sleepq_timeout
4718 1/477 fffffe810f311b60 sleepq_timeout
5146 1/478 fffffe81c2dc3b10 tcp_timer_rexmt
5434 1/479 fffffe8222de5008 realtimerexpire
6033 1/482 fffffe82068d5b60 tcp_timer_rexmt
6368 1/483 fffffe821f08b380 tcp_timer_rexmt
7175 1/486 fffffe8127c869c0 sleepq_timeout
34620 2/557 fffffe821a7580d0 tcp_timer_keep
34639 2/557 fffffe8210a5d1f0 tcp_timer_keep
34665 2/557 fffffe81c2dc37d0 tcp_timer_keep
34686 2/557 fffffe82068d54e0 tcp_timer_keep
18442 2/557 fffffe81de573330 arptimer
30184 2/557 fffffe822cdcc990 arptimer
73369 2/557 fffffe81de573220 arptimer
34607 2/557 fffffe81d8010030 tcp_timer_keep
34626 2/557 fffffe821b96a778 tcp_timer_keep
39854 2/557 fffffe821ada5480 tcp_timer_keep
41498 2/557 fffffe821cb3a3c8 tcp_timer_keep
63262 2/557 fffffe82173407f0 tcp_timer_keep
83173 2/558 fffffe81dbf21228 arptimer
99760 2/558 fffffe822cdcc110 arptimer
102304 2/558 fffffe822cdcc880 arptimer
110475 2/558 fffffe822cdccee0 arptimer
113194 2/558 fffffe822cdcc550 arptimer
119123 2/558 fffffe822cdcc440 arptimer
83485 2/558 fffffe822cfce600 sleepq_timeout
170902 2/559 fffffe821ed4a6c8 tcp_timer_keep
178082 2/559 fffffe8210a5d530 tcp_timer_keep
178095 2/559 fffffe821b4667d8 tcp_timer_keep
178100 2/559 fffffe820cd5cb98 tcp_timer_keep
178167 2/559 fffffe821f039038 tcp_timer_keep
180463 2/559 fffffe821b937760 tcp_timer_keep
180673 2/559 fffffe8217340b30 tcp_timer_keep
178141 2/559 fffffe81f4ad9208 tcp_timer_keep
178182 2/559 fffffe82068d51a0 tcp_timer_keep
178181 2/559 fffffe821f08b040 tcp_timer_keep
183651 2/559 fffffe820cd5c1d8 tcp_timer_keep
185007 2/559 fffffe82173404b0 tcp_timer_keep
186205 2/559 fffffe81f4ad9888 tcp_timer_keep
186244 2/559 fffffe81f4ad9bc8 tcp_timer_keep
180493 2/559 fffffe81f8e8f918 tcp_timer_keep
180504 2/559 fffffe82173384a8 tcp_timer_keep
180556 2/559 fffffe81f8e8fc58 tcp_timer_keep
186851 2/559 fffffe8210a5dbb0 tcp_timer_keep
188512 2/559 fffffe821c046428 tcp_timer_keep
188800 2/559 fffffe8219e307a8 tcp_timer_keep
189621 2/559 fffffe821b565460 tcp_timer_keep
189932 2/559 fffffe821ada5b00 tcp_timer_keep
190283 2/559 fffffe81f30d3578 tcp_timer_keep
190659 2/559 fffffe821ae20430 tcp_timer_keep
266181 2/560 fffffe821ed4aa08 tcp_timer_keep
266494 2/560 fffffe821ed4a048 tcp_timer_keep
267094 2/560 fffffe821b4d2a68 tcp_timer_keep
267288 2/560 fffffe821ae20770 tcp_timer_keep
267310 2/560 fffffe821ae200f0 tcp_timer_keep
267318 2/560 fffffe821b96a438 tcp_timer_keep
267334 2/560 fffffe821ab59ac0 tcp_timer_keep
267334 2/560 fffffe821ab59100 tcp_timer_keep
267406 2/560 fffffe8210a5d870 tcp_timer_keep
275093 2/560 fffffe8219e30468 tcp_timer_keep
267330 2/560 fffffe821cc7d798 tcp_timer_keep
267349 2/560 fffffe8219e30ae8 tcp_timer_keep
267064 2/560 fffffe821a758750 tcp_timer_keep
267068 2/560 fffffe821ba920a0 tcp_timer_keep
267070 2/560 fffffe821c0c2740 tcp_timer_keep
267070 2/560 fffffe821c0c2400 tcp_timer_keep
267068 2/560 fffffe821b9c20c8 tcp_timer_keep
267071 2/560 fffffe821b9c2748 tcp_timer_keep
267071 2/560 fffffe821c4f9418 tcp_timer_keep
267072 2/560 fffffe821c4f9a98 tcp_timer_keep
267124 2/560 fffffe821b937420 tcp_timer_keep
267263 2/560 fffffe821c0460e8 tcp_timer_keep
268769 2/560 fffffe821ebc26d0 tcp_timer_keep
266964 2/560 fffffe821b9c2408 tcp_timer_keep
267662 2/560 fffffe81f30d3bf8 tcp_timer_keep
268000 2/560 fffffe81f8e8f298 tcp_timer_keep
268341 2/560 fffffe8219a92b20 tcp_timer_keep
266820 2/560 fffffe821caaa090 tcp_timer_keep
267068 2/560 fffffe821b339a78 tcp_timer_keep
267068 2/560 fffffe821ba92a60 tcp_timer_keep
267068 2/560 fffffe821b3393f8 tcp_timer_keep
267070 2/560 fffffe821ba923e0 tcp_timer_keep
277109 2/561 fffffe8219e30128 tcp_timer_keep
278611 2/561 fffffe820cd5c858 tcp_timer_keep
278987 2/561 fffffe821b2023f0 tcp_timer_keep
280166 2/561 fffffe821b339738 tcp_timer_keep
280644 2/561 fffffe821b9c2a88 tcp_timer_keep
281529 2/561 fffffe821cb3a088 tcp_timer_keep
281820 2/561 fffffe821b0c8130 tcp_timer_keep
284392 2/561 fffffe821b0c8af0 tcp_timer_keep
298629 2/561 0 nd6_slowtimo
278048 2/561 fffffe821b565ae0 tcp_timer_keep
278713 2/561 fffffe821ebc2390 tcp_timer_keep
281860 2/561 fffffe821cb3a708 tcp_timer_keep
282331 2/561 fffffe821ab59440 tcp_timer_keep
283391 2/561 fffffe821cc7dad8 tcp_timer_keep
278133 2/561 fffffe821b4d2728 tcp_timer_keep
279419 2/561 fffffe821e59a058 tcp_timer_keep
279801 2/561 fffffe821aa0d148 tcp_timer_keep
281290 2/561 fffffe821cce2080 tcp_timer_keep
276329 2/561 fffffe821ada57c0 tcp_timer_keep
277660 2/561 fffffe821ebc2a10 tcp_timer_keep
279103 2/561 fffffe8217920150 tcp_timer_keep
279484 2/561 fffffe821b466158 tcp_timer_keep
308870 2/561 fffffe821c0c20c0 tcp_timer_keep
277014 2/561 fffffe821cc7d458 tcp_timer_keep
278388 2/561 fffffe821ada5140 tcp_timer_keep
279047 2/561 fffffe821e59a398 tcp_timer_keep
279340 2/561 fffffe821b202730 tcp_timer_keep
279751 2/561 fffffe821c192098 tcp_timer_keep
282696 2/561 fffffe821b466b18 tcp_timer_keep
276424 2/561 fffffe821cce23c0 tcp_timer_keep
277575 2/561 fffffe821b5657a0 tcp_timer_keep
278335 2/561 fffffe821e59aa18 tcp_timer_keep
281000 2/561 fffffe821b96aab8 tcp_timer_keep
281394 2/561 fffffe821b0c87b0 tcp_timer_keep
281520 2/561 fffffe8217920490 tcp_timer_keep
276779 2/561 fffffe821c1923d8 tcp_timer_keep
277337 2/561 fffffe821c192718 tcp_timer_keep
277983 2/561 fffffe821e59a6d8 tcp_timer_keep
278977 2/561 fffffe821b202a70 tcp_timer_keep
279866 2/561 fffffe81f4ad9548 tcp_timer_keep
280139 2/561 fffffe821cce2700 tcp_timer_keep
280193 2/561 fffffe821aa0d488 tcp_timer_keep
280586 2/561 fffffe81f30d38b8 tcp_timer_keep
280886 2/561 fffffe821caaaa50 tcp_timer_keep
280948 2/561 fffffe821c0c2a80 tcp_timer_keep
276057 2/561 fffffe821c192a58 tcp_timer_keep
278567 2/561 fffffe821b4d20a8 tcp_timer_keep
279767 2/561 fffffe821e51aa20 tcp_timer_keep
280560 2/561 fffffe821cc7d118 tcp_timer_keep
281960 2/561 fffffe82179207d0 tcp_timer_keep
282179 2/561 fffffe8219a927e0 tcp_timer_keep
282625 2/561 fffffe821cb3aa48 tcp_timer_keep
283796 2/561 fffffe821aa0d7c8 tcp_timer_keep
352289 2/562 fffffe821c046768 tcp_timer_keep
345110 2/562 fffffe81f30d3238 tcp_timer_keep
717815 2/563 fffffe8222cba9d0 tcp_timer_keep
717812 2/563 fffffe8222cba690 tcp_timer_keep
714678 2/563 fffffe8222cba350 tcp_timer_keep
717813 2/563 fffffe8222cba010 tcp_timer_keep
717814 2/563 fffffe821e1189d8 tcp_timer_keep
714584 2/563 fffffe821e118358 tcp_timer_keep
714678 2/563 fffffe821e118018 tcp_timer_keep
714678 2/563 fffffe821f87c9e0 tcp_timer_keep
717813 2/563 fffffe821f87c6a0 tcp_timer_keep
719074 2/563 fffffe821f87c360 tcp_timer_keep
714966 2/563 fffffe821f87c020 tcp_timer_keep
714678 2/563 fffffe821f8549e8 tcp_timer_keep
714967 2/563 fffffe821f8546a8 tcp_timer_keep
714678 2/563 fffffe821f854368 tcp_timer_keep
714678 2/563 fffffe821f854028 tcp_timer_keep
714967 2/563 fffffe821fc8e9f0 tcp_timer_keep
714678 2/563 fffffe821fc8e6b0 tcp_timer_keep
714147 2/563 fffffe821fc8e370 tcp_timer_keep
717812 2/563 fffffe821f0399f8 tcp_timer_keep
717815 2/563 fffffe821f039378 tcp_timer_keep
714678 2/563 fffffe821ebc2050 tcp_timer_keep
717812 2/563 fffffe821e51a3a0 tcp_timer_keep
714966 2/563 fffffe821e51a060 tcp_timer_keep
714966 2/563 fffffe821d280a28 tcp_timer_keep
714147 2/563 fffffe821d2806e8 tcp_timer_keep
714678 2/563 fffffe821d280068 tcp_timer_keep
714678 2/563 fffffe821cce2a40 tcp_timer_keep
717791 2/563 fffffe821f0396b8 tcp_timer_keep
714654 2/563 fffffe821ed4a388 tcp_timer_keep
717789 2/563 fffffe821e51a6e0 tcp_timer_keep
714943 2/563 fffffe821caaa710 tcp_timer_keep
717791 2/563 fffffe82255b6008 tcp_timer_keep
474660 2/564 fffffe821c046aa8 tcp_timer_keep
719129 2/565 fffffe812d614790 tcp_timer_keep
716178 2/565 fffffe821b0c8470 tcp_timer_keep
575467 2/565 fffffe81ba3c3918 tcp_timer_keep
716183 2/565 fffffe821c4f9758 tcp_timer_keep
716203 2/565 fffffe821aa0db08 tcp_timer_keep
716203 2/565 fffffe82173387e8 tcp_timer_keep
716180 2/565 fffffe8219a924a0 tcp_timer_keep
719131 2/565 fffffe821f08ba00 tcp_timer_keep
575487 2/565 fffffe81966cb070 tcp_timer_keep
582001 2/565 fffffe821ba92720 tcp_timer_keep
719107 2/565 fffffe812d614ad0 tcp_timer_keep
716173 2/565 fffffe821b937aa0 tcp_timer_keep
575462 2/565 fffffe81ba3c3c58 tcp_timer_keep
597175 2/565 fffffe8222de5e78 realtimerexpire
620866 2/566 fffffe821b466498 tcp_timer_keep
657207 2/566 fffffe82068d5b60 tcp_timer_keep
666161 2/566 fffffe820cd5c518 tcp_timer_keep
697102 2/567 fffffe821b565120 tcp_timer_keep
718814 2/567 fffffe81f8e8f5d8 tcp_timer_keep
680446 2/567 fffffe81c2dc3b10 tcp_timer_keep
710518 2/567 fffffe821f08b380 tcp_timer_keep
696427 2/567 fffffe81966cb6f0 tcp_timer_keep
938912 2/571 fffffe81de573770 nd6_llinfo_timer
3549262 2/610 fffffe81de573dd0 nd6_llinfo_timer
8163629 2/681 0 in6_tmpaddrtimer
8269307 2/682 fffffe81de573000 nd6_llinfo_timer
8271207 2/682 fffffe81de573aa0 nd6_llinfo_timer
8271207 2/682 fffffe81de573110 nd6_llinfo_timer
db{0}>
db{0}> trace/a fffffe822f607840
trace: pid 0 lid 2 at 0xffff8000ae63df90
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x1b5
db{0}>
db{0}> trace/a fffffe810e8a38e0
trace: pid 0 lid 17 at 0xffff8000ae4f1f30
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x1b5
cpu_hatch() at netbsd:cpu_hatch+0x17a
md_root_setconf() at netbsd:md_root_setconf
db{0}>
From: "J. Hannken-Illjes" <hannken@eis.cs.tu-bs.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network
interfaces in use
Date: Fri, 1 Jun 2018 12:40:37 +0200
> db{0}>
>
> db{0}> ps
> PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
<snip>
> 0 > 6 7 0 200 fffffe822f5eb440 softser/0
> 0 > 5 7 0 200 fffffe822f5eb860 softclk/0
> 0 4 1 0 200 fffffe822f607000 softbio/0
> 0 > 3 7 0 200 fffffe822f607420 softnet/0
> 0 > 2 7 0 201 fffffe822f607840 idle/0
> 0 1 3 0 200 ffffffff81481b80 swapper uvm
<snip>
> db{0}> show all callout
> hardclock_ticks now: 86821384
> ticks wheel arg func
> -863 -1/-256 0 pffasttimo
> -861 -1/-256 0 pfslowtimo
> -861 -1/-256 0 tcp_slowtimo
> -851 -1/-256 0 sched_balance
> -849 -1/-256 fffffe812d614ad0 tcp_delack
> -848 -1/-256 fffffe810f382e10 uhci_poll_hub
> -833 -1/-256 0 key_timehandler
> -828 -1/-256 ffffffff81481b80 sleepq_timeout
> -822 -1/-256 0 nd6_timer
> -819 -1/-256 0 rt_timer_timer
> -811 -1/-256 ffff80000e836000 bnx_tick
> -811 -1/-256 ffff80000e830008 if_slowtimo
> -811 -1/-256 ffff80000e833008 if_slowtimo
> -811 -1/-256 ffff80000e836008 if_slowtimo
> -811 -1/-256 ffff80000e839008 if_slowtimo
> -811 -1/-256 ffff80000e830000 bnx_tick
> -798 -1/-256 ffff80000e839000 bnx_tick
> -784 -1/-256 fffffe822f5d0040 sleepq_timeout
> -561 -1/-256 0 vmem_rehash_all_kick
> -430 -1/-256 fffffe81f8e8f5d8 tcp_timer_rexmt
> 13 0/107 fffffe810f372e10 uhci_poll_hub
> 14 0/129 fffffe810f330e10 uhci_poll_hub
> 43 0/163 fffffe810eb79aa0 sleepq_timeout
> 13 0/182 fffffe810f351e10 uhci_poll_hub
Suppose negative ticks means callout processing has stopped.
Examined all softhandler on cpu0 and got:
index func flags func ACTIVE,PENDING LEVEL
108 0xffffffff809c5cf0 0x3101 callout_softclock AP CLOCK
113 0xffffffff809ee1ce 0x2102 xc__highpri_intr A SERIAL
148 0xffffffff80a25ec8 0x1100 biointr P BIO
268 0xffffffff805f39e9 0x1002 comsoft P SERIAL
303 0xffffffff80705118 0x3303 ip6intr AP NET
If I get it right we have callout_softclock and ip6intr busy and with
more work and we have xc__highpri_intr busy (there is a
pserialize_perform running).
Is it right the running ip6intr will block all other handlers as NET
is the highest priority here?
Any idea how to debug the current state of "softnet/0"
--
J. Hannken-Illjes - hannken@eis.cs.tu-bs.de - TU Braunschweig (Germany)
From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: port-amd64-maintainer@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
network interfaces in use
Date: Sun, 17 Jun 2018 13:18:11 +0200 (CEST)
Hi,
two additional observations on this case:
1) When provoking the wedge, it is sufficient to only "up" the
third interface on the box. It's not required to configure
any IP addresses on the third interface to cause the wedge.
2) We have also provoked the wedge by using a quad wm(4) card,
and trying to use three interfaces on that card will also
provoke the wedge, so whatever is causing this issue is
unlikely to be a device driver bug.
Best regards,
- H=E5vard
State-Changed-From-To: open->feedback
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Sat, 24 Jul 2021 20:38:17 +0000
State-Changed-Why:
Is this the same as 47229?
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.46 2020/01/03 16:35:01 leot Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2020
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home