NetBSD Problem Report #53155

From he@smistad.uninett.no  Wed Apr  4 13:36:34 2018
Return-Path: <he@smistad.uninett.no>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 2F4A97A172
	for <gnats-bugs@gnats.NetBSD.org>; Wed,  4 Apr 2018 13:36:34 +0000 (UTC)
Message-Id: <20180404133555.42D0043F11C@smistad.uninett.no>
Date: Wed,  4 Apr 2018 15:35:55 +0200 (CEST)
From: he@NetBSD.org
Reply-To: he@NetBSD.org
To: gnats-bugs@NetBSD.org
Subject: OS wedges after <12h uptime when >2 bnx network interfaces in use
X-Send-Pr-Version: 3.95

>Number:         53155
>Category:       port-amd64
>Synopsis:       Wedge after <12h uptime when >2 bnx network interfaces in use
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    port-amd64-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Apr 04 13:40:00 +0000 2018
>Last-Modified:  Sun Jun 17 11:20:00 +0000 2018
>Originator:     Havard Eidnes
>Release:        NetBSD 8.0_BETA
>Organization:
	I try...
>Environment:
System: NetBSD xx.xx.xxx 8.0_BETA NetBSD 8.0_BETA (GENERIC.201803261630Z) amd64
Architecture: x86_64
Machine: amd64
>Description:
	This particular system appears to wedge at unpredictable times
	with less than 12 hours uptime when more than two network
	interfaces are configured and in use.  We have tried with both
	bnx and wm interfaces, and the mix doesn't appear to make any
	difference.

	The symptoms when it wedges are that the console (VGA with USB
	keyboard, or physical serial port) is unresponsive, and the
	host no longer responds to network traffic on any of its
	network interfaces.

	I should perhaps mention that this host has been tuned to
	allow up to 1MB socket buffers, both for ipv4 and ipv6, and is
	using the "cubic" congestion control algorithm, via the
	following /etc/sysctl.conf entries:

kern.sbmax=1048576
net.inet6.tcp6.sendbuf_max=1048576
net.inet6.tcp6.recvbuf_max=1048576
net.inet.tcp.sendbuf_max=1048576
net.inet.tcp.recvbuf_max=1048576
net.inet.tcp.congctl.selected=cubic

	and in normal use it has at least ~170 active TCP sessions,
	most if not all all related to running conserver, so normally
	carries very little traffic.  The amount of memory allocated
	to mbuf cluseters is (defaulted to):

kern.mbuf.nmbclusters = 261730

	On the host I've rigged it to run "vmstat -m" every 100
	seconds or so, but the last it logged didn't look strange, it
	ended in

In use 92189K, total allocated 96140K; utilization 95.9%

	and the mbpl and mclpl usage isn't anywhere near their
	high-water marks if I read this correctly:

Memory resource pool statistics
Name        Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
...
mbpl         512     2176    0        0   274     0   274   274     2   inf    2
mclpl       2048     1953    0        0   981     0   981   981     4 130865   4
...

	After I got rigged up a serial console, I managed to break
	into DDB, and below you'll find an edited part of the DDB
	session.  What I observe is:

	 - Only one of the 8 CPUs is doing anything (running expect in
           user-mode), but is apparently busy doing interrupt
           processing.  However, DDB apparently isn't able to decipher
           the stack through the interrupt frames (any hints for
           fixing that?)
	 - There is plenty of free memory available
	 - A number of processes have status "tstile", possibly
           indicating a locking problem or contention of some sort,
           but my ability to debug that sort of stuff isn't there, and
           we're not good at documenting what info to collect (hint,
           hint), and "tstile" doesn't really point towards the lock
           which is contested, only indicates "some lock somewhere is
           contested".
	 - My gut says this may be an interrupt storm, but none of the
           event counters seems to back that up

	I beleive this problem is the same one which was reported
	against NetBSD/amd64 6.0 in PR#47229, in fact it probably is
	the exact same host system.

	Here's the edited DDB session (with "greater-than" replaced
	with ":" in order to not confuse gnats):

:DDB break sequence:
fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0xffff80009db2
3000 ilevel 0x8 rsp 0xffff80008f69bcf8
curlwp 0xfffffe822f6fd440 pid 0.6 lowest kstack 0xffff80008f6972c0
Stopped in pid 0.6 (system) at  netbsd:breakpoint+0x5:  leave
db{0}: trace
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
DDB lost frame for netbsd:Xdoreti, trying 0xffff80008f69bd70
Xdoreti() at netbsd:Xdoreti
--- interrupt ---
Bad frame pointer: 0xffff800000000000
206:
db{0}: show uvmexp
Current UVM status:
  pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12, ncolors=8
  2028595 VM pages: 1037330 active, 0 inactive, 1805 wired, 923564 free
  pages  54144 anon, 979303 file, 5688 exec
  freemin=2048, free-target=2730, wired-max=676198
  resv-pg=1, resv-kernel=40, zeropages=820585
  cpu0:
    faults=5107500, traps=2516760, intrs=5678897, ctxswitch=3206198
    softint=6171203, syscalls=10897655
  cpu1:
    faults=601410, traps=562379, intrs=0, ctxswitch=407170
    softint=64702, syscalls=2203289
  cpu2:
    faults=607483, traps=592645, intrs=0, ctxswitch=357226
    softint=127481, syscalls=1962203
  cpu3:
    faults=585411, traps=548343, intrs=0, ctxswitch=348991
    softint=88312, syscalls=2002413
  cpu4:
    faults=843615, traps=531147, intrs=0, ctxswitch=538383
    softint=104299, syscalls=1896227
  cpu5:
    faults=909578, traps=884707, intrs=0, ctxswitch=2440981
    softint=103484, syscalls=8171151
  cpu6:
    faults=700436, traps=693435, intrs=0, ctxswitch=1017896
    softint=139013, syscalls=4260226
  cpu7:
    faults=804685, traps=791518, intrs=0, ctxswitch=877603
    softint=156355, syscalls=3733242
  fault counts:
    noram=0, noanon=0, pgwait=0, pgrele=0
    ok relocks(total)=21248(21250), anget(retrys)=4091394(0), amapcopy=457812
    neighbor anon/obj pg=406830/8542654, gets(lock/unlock)=3654322/21250
    cases: anon=3893186, anoncow=198224, obj=2917328, prcopy=737081, przero=1448638
  daemon and swap counts:
    woke=0, revs=0, scans=0, obscans=0, anscans=0
    busy=0, freed=0, reactivate=0, deactivate=0
    pageouts=0, pending=0, nswget=0
    nswapdev=1, swpgavail=32767
    swpages=32767, swpginuse=0, swpgonly=0, paging=0
db{0}: machine cpu 0
using CPU 0
db{0}: trace
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x562
Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
--- interrupt ---
DDB lost frame for netbsd:Xdoreti, trying 0xffff80008f69bd70
Xdoreti() at netbsd:Xdoreti
--- interrupt ---
Bad frame pointer: 0xffff800000000000
206:
db{0}: machine cpu 1
using CPU 1
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: machine cpu 2
using CPU 2
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: machine cpu 3
using CPU 3
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: machine cpu 4
using CPU 4
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: machine cpu 5
using CPU 5
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: machine cpu 6
using CPU 6
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: machine cpu 7
using CPU 7
db{0}: trace
x86_stihlt() at netbsd:x86_stihlt+0x6
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
idle_loop() at netbsd:idle_loop+0x18c
cpu_hatch() at netbsd:cpu_hatch+0x17d
md_root_setconf() at netbsd:md_root_setconf
db{0}: help show
db{0}: show event
evcnt type 0: bus_dma bounces = 8814235
evcnt type 0: bus_dma loads = 9314511
evcnt type 0: bus_dma nbouncebufs = 3211
evcnt type 0: vmcmd kills = 12944
evcnt type 0: vmcmd extends = 12219
evcnt type 0: vmcmd calls = 138643
evcnt type 0: vmem static_bt_inuse = 200
evcnt type 0: vmem static_bt_count = 200
evcnt type 0: rndpseudo open soft = 4559
evcnt type 1: TLB shootdown = 2192282
evcnt type 0: pserialize exclusive access = 18952
evcnt type 0: cpu0 runqueue pull = 34799
evcnt type 0: cpu0 runqueue push = 1042
evcnt type 0: cpu0 runqueue stay = 1725421
evcnt type 0: cpu0 runqueue localize = 8211
evcnt type 0: softint net/0 = 5191013
evcnt type 0: softint net block/0 = 1497
evcnt type 0: softint bio/0 = 487407
evcnt type 0: softint bio block/0 = 2270
evcnt type 0: softint clk/0 = 427502
evcnt type 0: softint clk block/0 = 12
evcnt type 0: softint ser/0 = 65281
evcnt type 0: callout late/0 = 13
evcnt type 0: callout wait/0 = 1
evcnt type 0: crosscall unicast = 1560
evcnt type 0: crosscall broadcast = 17
evcnt type 0: namecache entries collected = 10484
evcnt type 0: namecache under scan target = 12858
evcnt type 1: cpu0 timer = 1285733
evcnt type 0: cpu0 generic IPI = 437
evcnt type 0: cpu0 FPU synch IPI = 364
evcnt type 0: cpu0 xcall IPI = 26520
evcnt type 0: cpu0 kpreempt IPI = 61
evcnt type 0: cpu1 runqueue pull = 4176
evcnt type 0: cpu1 runqueue push = 1154
evcnt type 0: cpu1 runqueue stay = 217239
evcnt type 0: cpu1 runqueue localize = 188
evcnt type 0: softint net/1 = 2
evcnt type 0: softint net block/1 = 1
evcnt type 0: softint clk/1 = 27146
evcnt type 0: softint ser/1 = 37554
evcnt type 0: softint ser block/1 = 1
evcnt type 0: cpu1 timer = 1285906
evcnt type 0: cpu1 FPU synch IPI = 325
evcnt type 0: cpu1 xcall IPI = 34579
evcnt type 0: cpu2 runqueue pull = 3811
evcnt type 0: cpu2 runqueue push = 968
evcnt type 0: cpu2 runqueue stay = 193225
evcnt type 0: cpu2 runqueue localize = 300
evcnt type 0: softint clk/2 = 90104
evcnt type 0: softint ser/2 = 37377
evcnt type 0: cpu2 timer = 1285894
evcnt type 0: cpu2 FPU synch IPI = 291
evcnt type 0: cpu2 xcall IPI = 33748
evcnt type 0: cpu3 runqueue pull = 4132
evcnt type 0: cpu3 runqueue push = 1000
evcnt type 0: cpu3 runqueue stay = 188850
evcnt type 0: cpu3 runqueue localize = 487
evcnt type 0: softint net/3 = 1
evcnt type 0: softint clk/3 = 50860
evcnt type 0: softint ser/3 = 37451
evcnt type 0: cpu3 timer = 1285909
evcnt type 0: cpu3 FPU synch IPI = 276
evcnt type 0: cpu3 xcall IPI = 34139
evcnt type 0: cpu4 runqueue pull = 22245
evcnt type 0: cpu4 runqueue push = 1481
evcnt type 0: cpu4 runqueue stay = 281034
evcnt type 0: cpu4 runqueue localize = 298
evcnt type 0: softint clk/4 = 66905
evcnt type 0: softint ser/4 = 37394
evcnt type 0: cpu4 timer = 1285902
evcnt type 0: cpu4 FPU synch IPI = 290
evcnt type 0: cpu4 xcall IPI = 34811
evcnt type 0: cpu4 kpreempt IPI = 1
evcnt type 0: cpu5 runqueue pull = 7977
evcnt type 0: cpu5 runqueue push = 1309
evcnt type 0: cpu5 runqueue stay = 1249669
evcnt type 0: cpu5 runqueue localize = 1286
evcnt type 0: softint net/5 = 1
evcnt type 0: softint clk/5 = 65774
evcnt type 0: softint clk block/5 = 1
evcnt type 0: softint ser/5 = 37709
evcnt type 0: softint ser block/5 = 1
evcnt type 0: cpu5 timer = 1285895
evcnt type 0: cpu5 FPU synch IPI = 485
evcnt type 0: cpu5 xcall IPI = 31308
evcnt type 0: cpu6 runqueue pull = 5535
evcnt type 0: cpu6 runqueue push = 1795
evcnt type 0: cpu6 runqueue stay = 536359
evcnt type 0: cpu6 runqueue localize = 575
evcnt type 0: softint clk/6 = 101534
evcnt type 0: softint ser/6 = 37479
evcnt type 0: cpu6 timer = 1285900
evcnt type 0: cpu6 FPU synch IPI = 357
evcnt type 0: cpu6 xcall IPI = 34247
evcnt type 0: cpu7 runqueue pull = 7441
evcnt type 0: cpu7 runqueue push = 3442
evcnt type 0: cpu7 runqueue stay = 469377
evcnt type 0: cpu7 runqueue localize = 727
evcnt type 0: softint clk/7 = 118815
evcnt type 0: softint clk block/7 = 3
evcnt type 0: softint ser/7 = 37540
evcnt type 0: callout late/7 = 2
evcnt type 0: cpu7 timer = 1285900
evcnt type 0: cpu7 FPU synch IPI = 436
evcnt type 0: cpu7 xcall IPI = 31790
evcnt type 1: ioapic1 pin 4 = 5153711
evcnt type 1: ioapic1 pin 0 = 29577
evcnt type 1: ioapic1 pin 10 = 7447
evcnt type 1: ioapic0 pin 17 = 20
evcnt type 1: ioapic0 pin 19 = 21
evcnt type 1: ioapic0 pin 16 = 487731
evcnt type 1: ioapic0 pin 21 = 29
evcnt type 1: ioapic0 pin 23 = 12
evcnt type 1: ioapic0 pin 4 = 349
evcnt type 0: acpicpu0 C1 (HALT) = 6993829
evcnt type 0: acpicpu1 C1 (HALT) = 3158420
evcnt type 0: acpicpu2 C1 (HALT) = 3155060
evcnt type 0: acpicpu3 C1 (HALT) = 3049008
evcnt type 0: acpicpu4 C1 (HALT) = 3255477
evcnt type 0: acpicpu5 C1 (HALT) = 3851591
evcnt type 0: acpicpu6 C1 (HALT) = 3459820
evcnt type 0: acpicpu7 C1 (HALT) = 3405818
evcnt type 0: kpreempt defer: critical section = 48
evcnt type 0: kpreempt defer: kernel_lock = 60
evcnt type 0: kpreempt immediate = 3524
db{0}: ps
PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
20105    1 3   1         0   fffffe8224d7d780               cron tstile
26131    1 3   1         0   fffffe822281d620               cron wait
23319    1 3   1         0   fffffe8224850b60               cron tstile
25811    1 3   7         0   fffffe82036d0740               cron tstile
22533    1 3   7         0   fffffe820a1115c0               cron wait
24189    1 3   1         0   fffffe8221441880               cron wait
26325    1 3   5         0   fffffe8226e1e900               cron tstile
18295    1 3   1         0   fffffe8229822360               cron tstile
24699    1 3   6         0   fffffe822571b100               cron tstile
20160    1 3   7         0   fffffe821430ea20               cron tstile
24819    1 3   5         0   fffffe821e31f2a0               cron wait
18159    1 3   6         0   fffffe81fdf214a0               cron wait
20826    1 3   7         0   fffffe8215beeaa0               cron wait
19785    1 3   1         0   fffffe822a5968c0               cron wait
19192    1 3   1         0   fffffe822da91540               cron tstile
24053    1 3   7         0   fffffe822d9c9140               cron tstile
22204    1 3   7         0   fffffe8224bf4000               cron wait
19906    1 3   1         0   fffffe82260d4760               cron wait
25501    1 3   4         0   fffffe821e715260               bash tstile
16391    1 3   1         0   fffffe822b3fb8a0               cron tstile
22719    1 3   1         0   fffffe8215236740               cron wait
27003    1 3   2        80   fffffe820a90e6e0             pickup kqueue
25066    1 3   5         0   fffffe8212edf620                ssh xchicv
21812    2 3   5        80   fffffe822a960740             expect select
21812    1 3   4        80   fffffe8226255060             expect wait
22983    1 3   5        80   fffffe8211b16780                 sh wait
24598    1 3   5        80   fffffe821ed7e200               perl wait
23813    1 3   5        80   fffffe8209d819a0                 sh wait
20606    1 3   0        80   fffffe821430e600                ssh select
14822    1 3   6        80   fffffe821f1f05c0                 sh wait
29431    1 3   0        80   fffffe815c71a4c0                ssh select
25005    1 3   4        80   fffffe82140376c0                 sh wait
23260    1 3   3         0   fffffe8212bd1520                ssh tstile
22933    1 3   4        80   fffffe82142275a0                 sh wait
23169    1 3   0        80   fffffe821431d120                ssh select
20146    2 3   6         0   fffffe820f2f5b20             expect tstile
20146    1 3   6         0   fffffe822b7e3ae0             expect fdclose
27429    1 3   7        80   fffffe82200e3860                 sh wait
23732    1 3   7        80   fffffe82257e8120               perl pipe_rd
19502    1 3   7        80   fffffe82107e9020                 sh wait
23886    1 3   5        80   fffffe820f982220                ssh select
20022    2 3   4         0   fffffe821e467a00             expect tstile
20022    1 3   5         0   fffffe8210e401e0             expect fdclose
20799    1 3   0        80   fffffe82114f39c0                 sh wait
21173    1 3   5        80   fffffe82036d0b60               perl wait
21675    1 3   6        80   fffffe82284e8260                 sh wait
18171    1 3   0        80   fffffe8209d81160                ssh select
19295    2 3   4        80   fffffe82107e9860             expect select
19295:   1 7   0         0   fffffe82288daa80             expect
21354    1 3   5        80   fffffe8206f1a260                 sh wait
26421    1 3   5        80   fffffe820e5634c0               perl pipe_rd
24533    1 3   5        80   fffffe82273078e0                 sh wait
22388    1 3   0        80   fffffe8225d0d760                ssh select
20900    2 3   7         0   fffffe822a596080             expect tstile
20900    1 3   7         0   fffffe8212a4d080             expect fdclose
20540    1 3   0        80   fffffe822a3a9500                 sh wait
22103    1 3   7        80   fffffe822571b940               perl pipe_rd
24457    1 3   5        80   fffffe8222326aa0                 sh wait
23314    1 3   0        80   fffffe8215f8d840                ssh select
23095    1 3   0        80   fffffe82200e3020                 sh wait
18806    1 3   0        80   fffffe821f97e980                ssh select
17372    1 3   4        80   fffffe820e5638e0                 sh wait
23989    1 3   0        80   fffffe82201ad520                ssh select
23534    1 3   3        80   fffffe820d6e9440                 sh wait
19407    1 3   2        80   fffffe821eb7e0e0                ssh select
25841    1 3   6        80   fffffe8222107a80                 sh wait
26243    1 3   0        80   fffffe820fe60720                ssh select
17183    1 3   6        80   fffffe820acaa0e0                 sh wait
24185    1 3   5        80   fffffe822a3a90e0                ssh select
19244    1 3   4        80   fffffe821dc670c0                 sh wait
18735    1 3   0        80   fffffe822acb2460                ssh select
21395    1 3   6        80   fffffe8210521340                 sh wait
26341    1 3   0        80   fffffe822604b700                ssh select
20515    1 3   0        80   fffffe822a5964a0                 sh wait
24911    1 3   0        80   fffffe820de6a240                ssh select
19983    1 3   1        80   fffffe822da9c580                 sh wait
18877    1 3   0        80   fffffe822281d200                ssh select
23537    1 3   7        80   fffffe82273074c0                 sh wait
21663    1 3   0        80   fffffe821fad9420                ssh select
25421    1 3   1        80   fffffe82218a3ac0                 sh wait
24440    1 3   0        80   fffffe8215f8d420                ssh select
26449    1 3   0        80   fffffe82253a8b40                 sh wait
22382    1 3   0        80   fffffe820b0600c0                ssh select
21078    1 3   1        80   fffffe8166096900                 sh wait
23069    1 3   5        80   fffffe821c068100                ssh select
20686    1 3   5        80   fffffe8220479080                 sh wait
19346    1 3   0        80   fffffe820f2f5700                ssh select
21254    1 3   6        80   fffffe8212a4d8c0                 sh wait
23663    1 3   0        80   fffffe81fdf21080                ssh select
22711    1 3   1        80   fffffe816482f960                 sh wait
22549    1 3   0        80   fffffe821eb7e500                ssh select
12635    1 3   7        80   fffffe821de42120                 sh wait
22248    1 3   0        80   fffffe822a3a9920                ssh select
16468    1 3   4        80   fffffe8228cb76e0                 sh wait
17532    1 3   0        80   fffffe82111a0280                ssh select
24374    1 3   0        80   fffffe821f05d240                 sh wait
23230    1 3   0        80   fffffe822b7e36c0                ssh select
18702    1 3   0        80   fffffe820a19ba80                 sh wait
23591    1 3   0        80   fffffe822b6b6200                ssh select
24665    1 3   7        80   fffffe820de6a660                 sh wait
20189    1 3   0        80   fffffe822a025a60                ssh select
20252    1 3   0        80   fffffe822d9c9980                ssh select
23080    1 3   0        80   fffffe82172c4b80                 sh wait
20740    1 3   7        80   fffffe820a19b660                 sh wait
24958    1 3   0        80   fffffe8210e40600                ssh select
22951    1 3   5        80   fffffe82204794a0                 sh wait
21698    1 3   0        80   fffffe82143bbba0                ssh select
17032    1 3   2        80   fffffe821fad9840                 sh wait
24314    1 3   0        80   fffffe821eb7e920                ssh select
20886    1 3   7        80   fffffe820e5630a0                 sh wait
22406    1 3   0        80   fffffe821d814140                ssh select
23192    1 3   2        80   fffffe8215475420                 sh wait
17303    1 3   0        80   fffffe822a960320                ssh select
24518    1 3   7        80   fffffe820e350b40                 sh wait
21932    1 3   0        80   fffffe821227e040                ssh select
22174    1 3   6        80   fffffe822b7e32a0                 sh wait
20504    1 3   0        80   fffffe821d855720                ssh select
19436    1 3   7        80   fffffe820ef105e0                 sh wait
23481    1 3   0        80   fffffe820d6e9860                ssh select
22354    1 3   0        80   fffffe821f05d660                ssh select
21891    1 3   3        80   fffffe8159f125a0                 sh wait
17000    1 3   6        80   fffffe8229822780                 sh wait
21472    1 3   0        80   fffffe8221af7ac0                ssh select
24341    1 3   7        80   fffffe8212536220                 sh wait
24131    1 3   0        80   fffffe821ed7e620                ssh select
22981    1 3   0        80   fffffe816482f540                ssh select
15043    1 3   4        80   fffffe820f982640                 sh wait
21177    1 3   0        80   fffffe815dcc1160                 sh wait
21503    1 3   0        80   fffffe820c092060                ssh select
25824    1 3   1        80   fffffe821e40d880                 sh wait
17304    1 3   0        80   fffffe8225b3a540                ssh select
19360    1 3   6        80   fffffe820f2f52e0                 sh wait
16456    1 3   5        80   fffffe821548d2e0                par pause
16577    1 3   0        80   fffffe8166b46500                 sh wait
21790    1 3   0        80   fffffe82200e3440                 sh wait
18068    1 3   1        80   fffffe821431d540                 sh wait
22385    1 3   1        80   fffffe821d855300                 sh wait
17963    1 3   1        80   fffffe81660964e0               cron pipe_rd
5078     1 3   0        80   fffffe820c092480                ssh select
7768     1 3   3        80   fffffe8164637140                 sh wait
7239     1 3   0        80   fffffe8212edf200                ssh select
10982    1 3   5        80   fffffe8210521b80                 sh wait
5126     1 3   0        80   fffffe820f982a60                ssh select
8692     1 3   2        80   fffffe821c1f80a0                 sh wait
9318     1 3   0        80   fffffe82098348a0                ssh select
9227     1 3   2        80   fffffe821f97e140                 sh wait
10068    1 3   0        80   fffffe821dfe99a0                ssh select
8697     1 3   0        80   fffffe821e69f5c0                 sh wait
10511    1 3   0        80   fffffe82036d0320                ssh select
11284    1 3   2        80   fffffe82201ad100                 sh wait
9755     1 3   0        80   fffffe82289472a0                ssh select
9929     1 3   5        80   fffffe821fad9000                 sh wait
4650     1 3   0        80   fffffe820de6aa80                ssh select
9232     1 3   1        80   fffffe8215475000                 sh wait
11976    1 3   0        80   fffffe820acaa920                ssh select
8477     1 3   4        80   fffffe815de49520                 sh wait
5558     1 3   0        80   fffffe820d891420                ssh select
3911     1 3   3        80   fffffe82111a0ac0                 sh wait
9329     1 3   0        80   fffffe815de49100                ssh select
9759     1 3   6        80   fffffe8221af76a0                 sh wait
11080    1 3   0        80   fffffe82172c4760                ssh select
12525    1 3   5        80   fffffe820b060900                 sh wait
10080    1 3   0        80   fffffe8209834480                ssh select
12878    1 3   7        80   fffffe822b3fb480                 sh wait
14096    1 3   0        80   fffffe82260d4340                ssh select
9034     1 3   7        80   fffffe8221e7ba60                 sh wait
8694     1 3   0        80   fffffe8164637560                ssh select
4956     1 3   7        80   fffffe8226e1e0c0                 sh wait
8298     1 3   0        80   fffffe8215bee680                ssh select
8048     1 3   7        80   fffffe820e8582c0                 sh wait
6948     1 3   0        80   fffffe8214227180                ssh select
10692    1 3   7        80   fffffe821df299c0                 sh wait
3628     1 3   0        80   fffffe821227e460                ssh select
4171     1 3   6        80   fffffe822da91960                 sh wait
11767    1 3   0        80   fffffe8215236b60                ssh select
11095    1 3   0        80   fffffe82172c4340                ssh select
7282     1 3   5        80   fffffe822995e1e0                 sh wait
9931     1 3   6        80   fffffe82204798c0                 sh wait
11290    1 3   0        80   fffffe821b623ac0                ssh select
13455    1 3   0        80   fffffe8215236320                ssh select
12282    1 3   6        80   fffffe822acb2880                 sh wait
9284     1 3   7        80   fffffe821ba38640                 sh wait
7897     1 3   0        80   fffffe821e31fae0                ssh select
10861    1 3   0        80   fffffe8206f1aaa0                ssh select
9946     1 3   6        80   fffffe8224850320                 sh wait
4820     1 3   7        80   fffffe821e9dc880                 sh wait
8676     1 3   0        80   fffffe8228cb7b00                ssh select
11199    1 3   6        80   fffffe821e40d040                 sh wait
10988    1 3   0        80   fffffe8221af7280                ssh select
10711    1 3   0        80   fffffe8211d1f5e0                 sh wait
9326     1 3   0        80   fffffe820d891840                ssh select
6637     1 3   2        80   fffffe820fe60b40                 sh wait
5607     1 3   0        80   fffffe82143bb780                ssh select
6223     1 3   7        80   fffffe8218c8f620                 sh wait
6174     1 3   0        80   fffffe82273070a0                ssh select
3660     1 3   0        80   fffffe820a90e2c0                 sh wait
3691     1 3   0        80   fffffe821e9dc040                ssh select
13059    1 3   0        80   fffffe8211b16ba0                 sh wait
8242     1 3   0        80   fffffe822c00eb20                ssh select
10272    1 3   3        80   fffffe821dc674e0                 sh wait
9712     1 3   7        80   fffffe816482f120               bash select
8589     1 3   0        80   fffffe8229aeda00               sshd select
10949    1 3   1        80   fffffe8224d7d360          conserver select
7765     1 3   4        80   fffffe821f05da80               sshd select
11101    1 3   7         0   fffffe820e858b00          conserver tstile
11558    1 3   6         0   fffffe820b0604e0          conserver tstile
8956     1 3   5        80   fffffe8212bd1940          conserver select
10634    1 3   7        80   fffffe820e350300          conserver select
9547     1 3   0        80   fffffe820d6e9020          conserver select
7586     1 3   4         0   fffffe8212536a60          conserver tstile
8569     1 3   6         0   fffffe8218c8f200          conserver tstile
11721    1 3   6         0   fffffe8225b3a120          conserver tstile
9474     1 3   5        80   fffffe821c068940          conserver select
9068     1 3   0        80   fffffe8212a4d4a0          conserver select
8864     1 3   0        80   fffffe82101bc860          conserver select
6021     1 3   0        80   fffffe82238ef020              getty ttyraw
747      1 3   4        80   fffffe821e715680               bash wait
1233     1 3   5        80   fffffe82285c76a0                ssh select
336      1 3   0        80   fffffe821ba38a60                 sh wait
1101     1 3   4        80   fffffe821ed7ea40                ssh select
1228     1 3   0        80   fffffe821bf101e0                 sh wait
459      1 3   3        80   fffffe821bf10600                ssh select
458      1 3   0        80   fffffe821bf10a20                 sh wait
836      1 3   5        80   fffffe82285c7ac0                ssh select
322      1 3   5        80   fffffe821e69f9e0                 sh wait
321      1 3   0        80   fffffe821df29180                ssh select
320      1 3   0        80   fffffe821df295a0                 sh wait
1470     1 3   7        80   fffffe8222107660                ssh select
829      1 3   7        80   fffffe821e4a1160                 sh wait
956      1 3   4        80   fffffe821e4a1580                ssh select
1211     1 3   0        80   fffffe821e4a19a0                 sh wait
1203     1 3   6        80   fffffe821de42540                ssh select
306      1 3   0        80   fffffe821de42960                 sh wait
1455     1 3   6        80   fffffe82201ad940                ssh select
1326     1 3   0        80   fffffe821f1730e0                 sh wait
813      1 3   2        80   fffffe821f173500                ssh select
1196     1 3   0        80   fffffe821f173920                 sh wait
299      1 3   2        80   fffffe82201860c0                ssh select
298      1 3   5        80   fffffe82201864e0                 sh wait
297      1 3   5        80   fffffe8220186900                ssh select
296      1 3   5        80   fffffe821eca10a0                 sh wait
1191     1 3   7        80   fffffe821eca14c0                ssh select
1062     1 3   0        80   fffffe821eca18e0                 sh wait
1445     1 3   6        80   fffffe8220596080                ssh select
1188     1 3   0        80   fffffe82205964a0                 sh wait
1443     1 3   5        80   fffffe82205968c0                ssh select
290      1 3   0        80   fffffe821f131060                 sh wait
289      1 3   4        80   fffffe821f131480                ssh select
288      1 3   0        80   fffffe821f1318a0                 sh wait
1439     1 3   4        80   fffffe8221441040                ssh select
1438     1 3   0        80   fffffe8221441460                 sh wait
1181     1 3   7        80   fffffe8222326260                ssh select
540      1 3   0        80   fffffe82209cb780                 sh wait
1422     1 3   2        80   fffffe822d8b55a0                ssh select
1165     1 3   5        80   fffffe82209cb360                 sh wait
1164     1 3   6        80   fffffe822c5039e0                ssh select
267      1 3   4        80   fffffe822da9c160                 sh wait
264      1 3   5        80   fffffe82209cbba0                ssh select
775      1 3   3        80   fffffe8222326680                 sh wait
1156     1 3   7        80   fffffe8222107240                ssh select
386      1 3   7        80   fffffe8220131340                 sh wait
385      1 3   5        80   fffffe8220131760                ssh select
384      1 3   0        80   fffffe8220131b80                 sh wait
780      1 3   6        80   fffffe8221a55320                ssh select
144      1 3   0        80   fffffe8221a55740                 sh wait
146      1 3   5        80   fffffe82253a8300                ssh select
1279     1 3   7        80   fffffe8221a55b60                 sh wait
1278     1 3   4        80   fffffe822105b300                ssh select
893      1 3   0        80   fffffe822105b720                 sh wait
1019     1 3   3        80   fffffe822105bb40                ssh select
1146     1 3   0        80   fffffe8221dc52e0                 sh wait
1080     1 3   6        80   fffffe8221dc5700                ssh select
694      1 3   0        80   fffffe8221dc5b20                 sh wait
370      1 3   6        80   fffffe8220ab16e0                ssh select
1073     1 3   6        80   fffffe8220ab1b00                 sh wait
176      1 3   3        80   fffffe8220d512a0                ssh select
1071     1 3   0        80   fffffe8220d516c0                 sh wait
1198     1 3   0        80   fffffe8220d51ae0                ssh select
877      1 3   0        80   fffffe82218a3280                 sh wait
795      1 3   3        80   fffffe822227b1e0                ssh select
986      1 3   0        80   fffffe822227b600                 sh wait
985      1 3   4        80   fffffe822227ba20                ssh select
1240     1 3   0        80   fffffe82223491c0                 sh wait
727      1 3   0        80   fffffe82223495e0                ssh select
790      1 3   0        80   fffffe8222349a00                 sh wait
1173     1 3   1        80   fffffe82232021a0                ssh select
980      1 3   1        80   fffffe82232025c0                 sh wait
659      1 3   1        80   fffffe82232029e0                ssh select
210      1 3   1        80   fffffe822a57f180                 sh wait
913      1 3   5        80   fffffe822a57f5a0                ssh select
208      1 3   1        80   fffffe822a57f9c0                 sh wait
1231     1 3   7        80   fffffe822bd73160                ssh select
1038     1 3   1        80   fffffe822bd73580                 sh wait
973      1 3   2        80   fffffe822bd739a0                ssh select
844      1 3   1        80   fffffe8223fb7140                 sh wait
139      1 3   1        80   fffffe8223fb7560                ssh select
138      1 3   1        80   fffffe8223fb7980                 sh wait
508      1 3   1        80   fffffe82238ef440               cron nanoslp
827      1 3   1        80   fffffe82238ef860          python2.7 select
112      1 3   0        80   fffffe822d8b5180              inetd kqueue
743      1 3   0        80   fffffe8224bf4420              nginx kqueue
870      1 3   0        80   fffffe8224bf4840              nginx pause
974      1 3   1        80   fffffe822604bb20               qmgr kqueue
991      1 3   2        80   fffffe8224850740             master kqueue
647      1 3   4        80   fffffe822da9c9a0               sshd select
701      1 3   2        80   fffffe822995ea20           tac_plus select
685      1 3   1        80   fffffe822d9c9560             powerd kqueue
630      1 3   0        80   fffffe82289476c0               ntpd pause
595      4 3   6        80   fffffe8228947ae0              dhcpd parked
595      3 3   0        80   fffffe82284e8680              dhcpd kqueue
595      2 3   6        80   fffffe82284e8aa0              dhcpd parked
595      1 3   0        80   fffffe82288da660              dhcpd parked
439      1 3   7         0   fffffe82288da240            syslogd tstile
1        1 3   2        80   fffffe810f9b20a0               init wait
0      123 3   5       200   fffffe8229aed1c0            npfgc-0 xchicv
0      122 3   0       200   fffffe822d8b59c0              ipmi0 ipmi0
0      121 3   0       200   fffffe822ddb5500            physiod physiod
0      120 3   0       200   fffffe822dd44520           aiodoned aiodoned
0      119 3   6       200   fffffe822dd44940            ioflush biolock
0      118 3   0       200   fffffe822ddb50e0           pgdaemon pgdaemon
0      115 3   0       280   fffffe810f41e0c0              spkr1 bellcv
0      114 3   5       280   fffffe810f41e4e0           audiomix play
0      113 3   5       280   fffffe810ed322a0           audiorec record
0      112 3   5       200   fffffe810f2c9300          atapibus0 sccomp
0      110 3   3       200   fffffe810f2e0b60               usb5 usbevt
0      109 3   5       200   fffffe810f2c9b40               usb4 usbevt
0      108 3   0       200   fffffe810f1fc2c0               usb1 usbevt
0      107 3   0       200   fffffe810f2c9720               usb0 usbevt
0      106 3   5       200   fffffe810f2ba2e0               usb3 usbevt
0      105 3   5       200   fffffe810f41e900               usb2 usbevt
0      104 3   0       200   fffffe810f9b24c0            rt_free rt_free
0      103 3   0       200   fffffe810f9b28e0              unpgc unpgc
0      102 3   2       200   fffffe810f47b080    key_timehandler key_timehandler
0      101 3   7       200   fffffe810f47b4a0    icmp6_wqinput/7 icmp6_wqinput
0      100 3   6       200   fffffe810f47b8c0    icmp6_wqinput/6 icmp6_wqinput
0       99 3   5       200   fffffe810f3b4060    icmp6_wqinput/5 icmp6_wqinput
0       98 3   4       200   fffffe810f3b4480    icmp6_wqinput/4 icmp6_wqinput
0       97 3   3       200   fffffe810f3b48a0    icmp6_wqinput/3 icmp6_wqinput
0       96 3   2       200   fffffe810f435040    icmp6_wqinput/2 icmp6_wqinput
0       95 3   1       200   fffffe810f435460    icmp6_wqinput/1 icmp6_wqinput
0       94 3   0       200   fffffe810f435880    icmp6_wqinput/0 icmp6_wqinput
0       93 3   0       200   fffffe810f34e020          nd6_timer nd6_timer
0       92 3   7       200   fffffe810f34e440     icmp_wqinput/7 icmp_wqinput
0       91 3   6       200   fffffe810f34e860     icmp_wqinput/6 icmp_wqinput
0       90 3   5       200   fffffe810f337000     icmp_wqinput/5 icmp_wqinput
0       89 3   4       200   fffffe810f337420     icmp_wqinput/4 icmp_wqinput
0       88 3   3       200   fffffe810f337840     icmp_wqinput/3 icmp_wqinput
0       87 3   2       200   fffffe810f360360     icmp_wqinput/2 icmp_wqinput
0       86 3   1       200   fffffe810f360780     icmp_wqinput/1 icmp_wqinput
0       85 3   0       200   fffffe810f360ba0     icmp_wqinput/0 icmp_wqinput
0       84 3   5       200   fffffe810f2ba700           rt_timer rt_timer
0       83 3   5       200   fffffe810f2bab20        vmem_rehash vmem_rehash
0       82 3   3       200   fffffe810f2d7340          coretemp3 coretemp3
0       81 3   6       200   fffffe810f2d7760          coretemp2 coretemp2
0       80 3   1       200   fffffe810f2d7b80          coretemp1 coretemp1
0       79 3   7       200   fffffe810f2e0320          coretemp0 coretemp0
0       78 3   6       200   fffffe810f2e0740               mfi0 mfi0
0       69 3   0       200   fffffe810f1fc6e0            atabus1 atath
0       68 3   0       200   fffffe810f1fcb00            atabus0 atath
0       66 3   0       200   fffffe810ed326c0           scsibus0 sccomp
0       65 3   5       200   fffffe810ed32ae0         usbtask-dr usbtsk
0       64 3   0       200   fffffe810eca9280         usbtask-hc usbtsk
0       63 3   7       200   fffffe810eca96a0               bnx3 bnx3
0       62 3   4       200   fffffe810eca9ac0               bnx2 bnx2
0       61 3   0       200   fffffe810eb60260               bnx1 bnx1
0       60 3   7       200   fffffe810eb60680               bnx0 bnx0
0       59 3   5       200   fffffe810eb60aa0               ipmi ipmipoll
0       58 3   7       200   fffffe810ea73240            xcall/7 xcall
0       57 1   7       200   fffffe810ea73660          softser/7
0       56 1   7       200   fffffe810ea73a80          softclk/7
0       55 1   7       200   fffffe810ea54220          softbio/7
0       54 1   7       200   fffffe810ea54640          softnet/7
0    :  53 7   7       201   fffffe810ea54a60             idle/7
0       52 3   6       200   fffffe810ea3d200            xcall/6 xcall
0       51 1   6       200   fffffe810ea3d620          softser/6
0       50 1   6       200   fffffe810ea3da40          softclk/6
0       49 1   6       200   fffffe810ea161e0          softbio/6
0       48 1   6       200   fffffe810ea16600          softnet/6
0    :  47 7   6       201   fffffe810ea16a20             idle/6
0       46 3   5       200   fffffe810e9d71c0            xcall/5 xcall
0       45 1   5       200   fffffe810e9d75e0          softser/5
0       44 1   5       200   fffffe810e9d7a00          softclk/5
0       43 1   5       200   fffffe810e9d01a0          softbio/5
0       42 1   5       200   fffffe810e9d05c0          softnet/5
0    :  41 7   5       201   fffffe810e9d09e0             idle/5
0       40 3   4       200   fffffe810e9a1180            xcall/4 xcall
0       39 1   4       200   fffffe810e9a15a0          softser/4
0       38 1   4       200   fffffe810e9a19c0          softclk/4
0       37 1   4       200   fffffe810e97a160          softbio/4
0       36 1   4       200   fffffe810e97a580          softnet/4
0    :  35 7   4       201   fffffe810e97a9a0             idle/4
0       34 3   3       200   fffffe810e943140            xcall/3 xcall
0       33 1   3       200   fffffe810e943560          softser/3
0       32 1   3       200   fffffe810e943980          softclk/3
0       31 1   3       200   fffffe810e934120          softbio/3
0       30 1   3       200   fffffe810e934540          softnet/3
0    :  29 7   3       201   fffffe810e934960             idle/3
0       28 3   2       200   fffffe810e90d100            xcall/2 xcall
0       27 1   2       200   fffffe810e90d520          softser/2
0       26 1   2       200   fffffe810e90d940          softclk/2
0       25 1   2       200   fffffe810e8f60e0          softbio/2
0       24 1   2       200   fffffe810e8f6500          softnet/2
0    :  23 7   2       201   fffffe810e8f6920             idle/2
0       22 3   1       200   fffffe810e8c50c0            xcall/1 xcall
0       21 1   1       200   fffffe810e8c54e0          softser/1
0       20 1   1       200   fffffe810e8c5900          softclk/1
0       19 1   1       200   fffffe810e8a90a0          softbio/1
0       18 1   1       200   fffffe810e8a94c0          softnet/1
0    :  17 7   1       201   fffffe810e8a98e0             idle/1
0       16 3   0       200   fffffe822de9e080           lnxsyswq lnxsyswq
0       15 3   5       200   fffffe822de9e4a0             sysmon smtaskq
0       14 3   7       200   fffffe822de9e8c0         pmfsuspend pmfsuspend
0       13 3   4       200   fffffe822e2c5060           pmfevent pmfevent
0       12 3   0       200   fffffe822e2c5480         sopendfree sopendfr
0       11 3   2       200   fffffe822e2c58a0           nfssilly nfssilly
0       10 3   6       200   fffffe822f6dc040            cachegc cachegc
0        9 3   6       200   fffffe822f6dc460             vdrain vdrain
0        8 3   1       200   fffffe822f6dc880          modunload mod_unld
0        7 3   0       200   fffffe822f6fd020            xcall/0 xcall
0    :   6 7   0       200   fffffe822f6fd440          softser/0
0    :   5 7   0       200   fffffe822f6fd860          softclk/0
0        4 1   0       200   fffffe822f719000          softbio/0
0        3 1   0       200   fffffe822f719420          softnet/0
0        2 1   0       201   fffffe822f719840             idle/0
0        1 3   3       200   ffffffff81481b20            swapper uvm
db{0}: 

	and here's the dmesg output for the host:

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 8.0_BETA (GENERIC.201803261630Z)
total memory = 8179 MB
avail memory = 7920 MB
timecounter: Timecounters tick every 10.000 msec
Kernelized RAIDframe activated
running cgd selftest aes-xts-256 aes-xts-512 done
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
Dell Inc. PowerEdge R610
mainbus0 (root)
ACPI: RSDP 0x00000000000F1150 000024 (v02 DELL  )
ACPI: XSDT 0x00000000000F1254 000094 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: FACP 0x00000000CF3B3F9C 0000F4 (v03 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: DSDT 0x00000000CF38F000 003DD0 (v01 DELL   PE_SC3   00000001 INTL 20050624)
ACPI: FACS 0x00000000CF3B6000 000040
ACPI: FACS 0x00000000CF3B6000 000040
ACPI: APIC 0x00000000CF3B3478 00015E (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: SPCR 0x00000000CF3B35D8 000050 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: HPET 0x00000000CF3B362C 000038 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: DM__ 0x00000000CF3B3668 0001A8 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: MCFG 0x00000000CF3B38C4 00003C (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: WD__ 0x00000000CF3B3904 000134 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: SLIC 0x00000000CF3B3A3C 000176 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: ERST 0x00000000CF392F70 000270 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: HEST 0x00000000CF3931E0 0003A8 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: BERT 0x00000000CF392DD0 000030 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: EINJ 0x00000000CF392E00 000170 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: SRAT 0x00000000CF3B3BC0 000370 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: TCPA 0x00000000CF3B3F34 000064 (v02 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: 1 ACPI AML tables successfully acquired and loaded
ioapic0 at mainbus0 apid 0: pa 0xfec00000, version 0x20, 24 pins
ioapic1 at mainbus0 apid 1: pa 0xfec80000, version 0x20, 24 pins
cpu0 at mainbus0 apid 32
cpu0: Intel(R) Xeon(R) CPU           E5640  @ 2.67GHz, id 0x206c2
cpu0: package 1, core 0, smt 0
cpu1 at mainbus0 apid 34
cpu1: Intel(R) Xeon(R) CPU           E5640  @ 2.67GHz, id 0x206c2
cpu1: package 1, core 1, smt 0
cpu2 at mainbus0 apid 50
cpu2: Intel(R) Xeon(R) CPU           E5640  @ 2.67GHz, id 0x206c2
cpu2: package 1, core 9, smt 0
cpu3 at mainbus0 apid 52
cpu3: Intel(R) Xeon(R) CPU           E5640  @ 2.67GHz, id 0x206c2
cpu3: package 1, core 10, smt 0
cpu4 at mainbus0 apid 33
cpu4: Intel(R) Xeon(R) CPU           E5640  @ 2.67GHz, id 0x206c2
cpu4: package 1, core 0, smt 1
cpu5 at mainbus0 apid 35
cpu5: Intel(R) Xeon(R) CPU           E5640  @ 2.67GHz, id 0x206c2
cpu5: package 1, core 1, smt 1
cpu6 at mainbus0 apid 51
cpu6: Intel(R) Xeon(R) CPU           E5640  @ 2.67GHz, id 0x206c2
cpu6: package 1, core 9, smt 1
cpu7 at mainbus0 apid 53
cpu7: Intel(R) Xeon(R) CPU           E5640  @ 2.67GHz, id 0x206c2
cpu7: package 1, core 10, smt 1
acpi0 at mainbus0: Intel ACPICA 20170303
acpi0: X/RSDT: OemId <DELL  ,PE_SC3  ,00000001>, AslId <DELL,00000001>
acpi0: MCFG: segment 0, bus 0-255, address 0x00000000e0000000
acpi0: SCI interrupting at int 9
timecounter: Timecounter "ACPI-Fast" frequency 3579545 Hz quality 1000
hpet0 at acpi0: high precision event timer (mem 0xfed00000-0xfed00400)
timecounter: Timecounter "hpet0" frequency 14318180 Hz quality 2000
WHEA (PNP0C33) at acpi0 not configured
PMI0 (ACPI000D) at acpi0 not configured
SPK (PNP0C01) at acpi0 not configured
attimer1 at acpi0 (TMR, PNP0100): io 0x40-0x5f irq 0
COMA (PNP0501) at acpi0 not configured
COMB (PNP0501) at acpi0 not configured
MBIO (PNP0C01) at acpi0 not configured
NIPM (IPI0001) at acpi0 not configured
PEHB (PNP0C02) at acpi0 not configured
VTD (PNP0C02) at acpi0 not configured
acpiwmi0 at acpi0 (SRIO, PNP0C14-SRIO): ACPI WMI Interface
acpiwmibus at acpiwmi0 not configured
ACPI: Enabled 1 GPEs in block 00 to 3F
ipmi0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0: vendor 8086 product 3403 (rev. 0x13)
ppb0 at pci0 dev 1 function 0: vendor 8086 product 3408 (rev. 0x13)
ppb0: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x4 @ 5.0GT/s
ppb0: link is x4 @ 2.5GT/s
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
bnx0 at pci1 dev 0 function 0: Broadcom NetXtreme II BCM5709 1000Base-T
bnx0: Ethernet address f0:4d:a2:03:7b:f3
bnx0: interrupting at ioapic1 pin 4
bnx0: ASIC BCM5709 C0 (0x57092003)
bnx0: PCIe x4 2.5Gbps
bnx0: Coal (RX:6,6,18,18; TX:20,20,80,80)
brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bnx1 at pci1 dev 0 function 1: Broadcom NetXtreme II BCM5709 1000Base-T
bnx1: Ethernet address f0:4d:a2:03:7b:f5
bnx1: interrupting at ioapic1 pin 16
bnx1: ASIC BCM5709 C0 (0x57092003)
bnx1: PCIe x4 2.5Gbps
bnx1: Coal (RX:6,6,18,18; TX:20,20,80,80)
brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
ppb1 at pci0 dev 3 function 0: vendor 8086 product 340a (rev. 0x13)
ppb1: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x4 @ 5.0GT/s
ppb1: link is x4 @ 2.5GT/s
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
bnx2 at pci2 dev 0 function 0: Broadcom NetXtreme II BCM5709 1000Base-T
bnx2: Ethernet address f0:4d:a2:03:7b:f7
bnx2: interrupting at ioapic1 pin 0
bnx2: ASIC BCM5709 C0 (0x57092003)
bnx2: PCIe x4 2.5Gbps
bnx2: Coal (RX:6,6,18,18; TX:20,20,80,80)
brgphy2 at bnx2 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
brgphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bnx3 at pci2 dev 0 function 1: Broadcom NetXtreme II BCM5709 1000Base-T
bnx3: Ethernet address f0:4d:a2:03:7b:f9
bnx3: interrupting at ioapic1 pin 10
bnx3: ASIC BCM5709 C0 (0x57092003)
bnx3: PCIe x4 2.5Gbps
bnx3: Coal (RX:6,6,18,18; TX:20,20,80,80)
brgphy3 at bnx3 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
brgphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
ppb2 at pci0 dev 7 function 0: vendor 8086 product 340e (rev. 0x13)
ppb2: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x8 @ 5.0GT/s
pci3 at ppb2 bus 4
pci3: i/o space, memory space enabled, rd/line, wr/inv ok
ppb3 at pci0 dev 9 function 0: vendor 8086 product 3410 (rev. 0x13)
ppb3: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x8 @ 5.0GT/s
pci4 at ppb3 bus 5
pci4: i/o space, memory space enabled, rd/line, wr/inv ok
vendor 8086 product 342e (interrupt system, generic 8259, revision 0x13) at pci0 dev 20 function 0 not configured
vendor 8086 product 3422 (interrupt system, generic 8259, revision 0x13) at pci0 dev 20 function 1 not configured
vendor 8086 product 3423 (interrupt system, generic 8259, revision 0x13) at pci0 dev 20 function 2 not configured
uhci0 at pci0 dev 26 function 0: vendor 8086 product 2937 (rev. 0x02)
uhci0: interrupting at ioapic0 pin 17
usb0 at uhci0: USB revision 1.0
uhci1 at pci0 dev 26 function 1: vendor 8086 product 2938 (rev. 0x02)
uhci1: interrupting at ioapic0 pin 18
usb1 at uhci1: USB revision 1.0
ehci0 at pci0 dev 26 function 7: vendor 8086 product 293c (rev. 0x02)
ehci0: interrupting at ioapic0 pin 19
ehci0: BIOS has given up ownership
ehci0: EHCI version 1.0
ehci0: 2 companion controllers, 2 ports each: uhci0 uhci1
usb2 at ehci0: USB revision 2.0
ppb4 at pci0 dev 28 function 0: vendor 8086 product 2940 (rev. 0x02)
ppb4: PCI Express capability version 1 <Root Port of PCI-E Root Complex> x4 @ 2.5GT/s
pci5 at ppb4 bus 3
pci5: i/o space, memory space enabled, rd/line, wr/inv ok
mfi0 at pci5 dev 0 function 0: Dell PERC H700 Integrated
mfi0: interrupting at ioapic0 pin 16
mfi0: PERC H700 Integrated version 12.10.4-0001
mfi0: logical drives 2, 1024MB RAM, BBU type BBU, status good
scsibus0 at mfi0: 64 targets, 8 luns per target
uhci2 at pci0 dev 29 function 0: vendor 8086 product 2934 (rev. 0x02)
uhci2: interrupting at ioapic0 pin 21
usb3 at uhci2: USB revision 1.0
uhci3 at pci0 dev 29 function 1: vendor 8086 product 2935 (rev. 0x02)
uhci3: interrupting at ioapic0 pin 20
usb4 at uhci3: USB revision 1.0
ehci1 at pci0 dev 29 function 7: vendor 8086 product 293a (rev. 0x02)
ehci1: interrupting at ioapic0 pin 21
ehci1: EHCI version 1.0
ehci1: 2 companion controllers, 2 ports each: uhci2 uhci3
usb5 at ehci1: USB revision 2.0
ppb5 at pci0 dev 30 function 0: vendor 8086 product 244e (rev. 0x92)
pci6 at ppb5 bus 6
pci6: i/o space, memory space enabled
vga0 at pci6 dev 3 function 0: vendor 102b product 0532 (rev. 0x0a)
wsdisplay0 at vga0 kbdmux 1
wsmux1: connecting to wsdisplay0
drm at vga0 not configured
ichlpcib0 at pci0 dev 31 function 0: vendor 8086 product 2918 (rev. 0x02)
timecounter: Timecounter "ichlpcib0" frequency 3579545 Hz quality 1000
ichlpcib0: 24-bit timer
tco0 at ichlpcib0: TCO (watchdog) timer configured.
tco0: Min/Max interval 1/367 seconds
piixide0 at pci0 dev 31 function 2: Intel 82801I Serial ATA Controller (ICH9) (rev. 0x02)
piixide0: bus-master DMA support present
piixide0: primary channel configured to native-PCI mode
piixide0: using ioapic0 pin 23 for native-PCI interrupt
atabus0 at piixide0 channel 0
piixide0: secondary channel configured to native-PCI mode
atabus1 at piixide0 channel 1
isa0 at ichlpcib0
tpm0 at isa0 iomem 0xfed40000-0xfed44fff irq 7: device 0x0000104a rev 0x4e
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com0: console
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pcppi0 at isa0 port 0x61
spkr0 at pcppi0: PC Speaker
midi0 at pcppi0: PC speaker
sysbeep0 at pcppi0
attimer1: attached to pcppi0
acpicpu0 at cpu0: ACPI CPU
acpicpu0: C1: HLT, lat   0 us, pow     0 mW
coretemp0 at cpu0: thermal sensor, 1 C resolution, Tjmax=95
acpicpu1 at cpu1: ACPI CPU
coretemp1 at cpu1: thermal sensor, 1 C resolution, Tjmax=95
acpicpu2 at cpu2: ACPI CPU
coretemp2 at cpu2: thermal sensor, 1 C resolution, Tjmax=95
acpicpu3 at cpu3: ACPI CPU
coretemp3 at cpu3: thermal sensor, 1 C resolution, Tjmax=95
acpicpu4 at cpu4: ACPI CPU
acpicpu5 at cpu5: ACPI CPU
acpicpu6 at cpu6: ACPI CPU
acpicpu7 at cpu7: ACPI CPU
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
timecounter: Timecounter "TSC" frequency 2660187410 Hz quality 3000
scsibus0: waiting 2 seconds for devices to settle...
IPsec: Initialized Security Association Processing.
uhub0 at usb4: vendor 8086 (0x8086) UHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhub1 at usb1: vendor 8086 (0x8086) UHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhub2 at usb3: vendor 8086 (0x8086) UHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhub3 at usb2: vendor 8086 (0x8086) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub3: 4 ports with 4 removable, self powered
uhub4 at usb0: vendor 8086 (0x8086) UHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub4: 2 ports with 2 removable, self powered
uhub5 at usb5: vendor 8086 (0x8086) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub5: 4 ports with 4 removable, self powered
ehci0: handing over low speed device on port 1 to uhci0
ehci1: handing over full speed device on port 2 to uhci2
sd0 at scsibus0 target 0 lun 0: <DELL, PERC H700, 2.10> disk fixed
sd0: fabricating a geometry
sd0: 69376 MB, 69376 cyl, 64 head, 32 sec, 512 bytes/sect x 142082048 sectors
sd0: fabricating a geometry
sd0: tagged queueing
sd1 at scsibus0 target 1 lun 0: <DELL, PERC H700, 2.10> disk fixed
sd1: fabricating a geometry
sd1: 558 GB, 571776 cyl, 64 head, 32 sec, 512 bytes/sect x 1170997248 sectors
sd1: fabricating a geometry
sd1: tagged queueing
uhub6 at uhub3 port 3: vendor 0424 (0x424) product 2514 (0x2514), class 9/0, rev 2.00/0.00, addr 2
uhub6: multiple transaction translators
uhub6: 3 ports with 3 removable, self powered
uhidev0 at uhub4 port 1 configuration 1 interface 0
uhidev0: LITE-ON Technology (0x413c) Dell USB/PS2 Combo Rackmount Keyboard (0x2103), rev 1.10/1.00, addr 2, iclass 3/1
ukbd0 at uhidev0: 8 Variable keys, 6 Array codes
uhidev1 at uhub2 port 2 configuration 1 interface 0
uhidev1: Avocent (0x624) USB Composite Device-0 (0x248), rev 1.10/0.00, addr 2, iclass 3/1
ukbd1 at uhidev1: 8 Variable keys, 6 Array codes
wskbd0 at ukbd0 mux 1
wskbd0: connecting to wsdisplay0
uhidev2 at uhub4 port 1 configuration 1 interface 1
uhidev2: LITE-ON Technology (0x413c) Dell USB/PS2 Combo Rackmount Keyboard (0x2103), rev 1.10/1.00, addr 2, iclass 3/1
ums0 at uhidev2: 2 buttons
wsmouse0 at ums0 mux 0
wskbd1 at ukbd1 mux 1
wskbd1: connecting to wsdisplay0
uhidev3 at uhub2 port 2 configuration 1 interface 1
uhidev3: Avocent (0x624) USB Composite Device-0 (0x248), rev 1.10/0.00, addr 2, iclass 3/1
ums1 at uhidev3: 3 buttons and Z dir
wsmouse1 at ums1 mux 0
atapibus0 at atabus0: 2 targets
cd0 at atapibus0 drive 0: <TEAC DVD-ROM DV-28SW, 10092412144820, R.2A> cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
cd0(piixide0:0:0): using PIO mode 4, Ultra-DMA mode 5 (Ultra/100) (using DMA)
pad0: outputs: 44100Hz, 16-bit, stereo
audio0 at pad0: half duplex, playback, capture, mmap
pad0: Virtual format configured - Format SLINEAR, precision 16, channels 2, frequency 44100
pad0: Latency: 139 milliseconds
spkr1 at audio0: PC Speaker (synthesized)
boot device: sd0
root on sd0a dumps on sd0b
root file system type: ffs
kern.module.path=/stand/amd64/8.0/modules
mfi0: normal state on 'mfi0:0' (online)
mfi0: normal state on 'mfi0:1' (online)
ipmi0: version 2.0 interface KCS iobase 0xca8/8 spacing 4
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)


>How-To-Repeat:
	Configure a host with 3 active interfaces, use it moderately,
	watch it wedge after less than a day.
	(We have other identical systems which only use one network
	interface and which don't wedge.)

>Fix:
	Sorry, don't know.

	I'm asking for more information about what to do next to zero
	in on the root cause of this problem.
	A kernel with DEBUG? DIAGNOSTIC? LOCKDEBUG?  All of them?  Or
	only two?
	What other information needs to be collected next time it
	wedges to get closer to the root cause of the problem?

>Release-Note:

>Audit-Trail:
From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Wed, 04 Apr 2018 16:24:58 +0200 (CEST)

 Hi,

 a follow-up with a new hang with similar symptoms.

 This time fewer processes in "tstile" and a few in biowait.
 Doing "c" once and waiting a few seconds and collecting new info
 appears to indicate it's not making any progress.

 Again, edited DDB console log follows.

 fatal breakpoint trap in supervisor mode
 trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0x767a01b11cb0
  ilevel 0x8 rsp 0xffff80008f69bc88
 curlwp 0xfffffe822f6fd440 pid 0.6 lowest kstack 0xffff80008f6972c0
 Stopped in pid 0.6 (system) at  netbsd:breakpoint+0x5:  leave
 db{0}: trace
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 bus_space_read_2() at netbsd:bus_space_read_2+0x10
 uhci_intr() at netbsd:uhci_intr+0x8f
 intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
 Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}: c
 fatal breakpoint trap in supervisor mode
 trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0x767a01b11cb0
  ilevel 0x8 rsp 0xffff80008f69bc88
 curlwp 0xfffffe822f6fd440 pid 0.6 lowest kstack 0xffff80008f6972c0
 Stopped in pid 0.6 (system) at  netbsd:breakpoint+0x5:  leave
 db{0}: tra
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 bus_space_read_2() at netbsd:bus_space_read_2+0x10
 uhci_intr() at netbsd:uhci_intr+0x8f
 intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
 Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}: ps
 PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 5868     1 3   5         0   fffffe8215553960           postdrop biowait
 11951    1 3   5        80   fffffe81da398080           sendmail netio
 6001     1 3   7         0   fffffe82180044a0             vmstat biowait
 9372     1 3   2         0   fffffe81f0125420               tail tstile
 8217     1 3   6         0   fffffe81efc4d320               tail tstile
 9862     1 3   3        80   fffffe81d8509060               cron wait
 4632     1 3   0        80   fffffe822b3bb080                ssh select
 4405     2 3   4        80   fffffe8224c45b40             expect select
 4405     1 3   0        80   fffffe821f14e540             expect parked
 4495     1 3   0        80   fffffe8226a1e8a0                 sh wait
 9356     1 3   5        80   fffffe821b5d0200               perl pipe_rd
 6548     1 3   5        80   fffffe8221863a60                 sh wait
 10228:   2 7   0         0   fffffe81d9f12440             expect
 10228    1 3   7        80   fffffe820e8f9160             expect parked
 7679     1 3   2        80   fffffe822c143460                 sh wait
 12514    1 3   2        80   fffffe821daaa100               perl pipe_rd
 10055    1 3   2        80   fffffe82140c9140                 sh wait
 11571    1 3   0        80   fffffe8221215460                ssh select
 7806     1 3   4        80   fffffe82205b9280                 sh wait
 9266     1 3   0        80   fffffe8221215880                ssh select
 10356    1 3   5        80   fffffe8229b8e760                 sh wait
 7934     1 3   0        80   fffffe8225e1f6e0                ssh select
 10592    1 3   1        80   fffffe82170dc500                 sh wait
 9681     1 3   2        80   fffffe820e8f9580               perl wait
 9305     1 3   1        80   fffffe8219f67b60                 sh wait
 9908     1 3   0        80   fffffe8225fad840                ssh select
 8779     1 3   0        80   fffffe82205b96a0                 sh wait
 6856     1 3   0        80   fffffe8215553120                ssh select
 8803     1 3   1        80   fffffe81eb91a620                 sh wait
 8994     1 3   0        80   fffffe81d9f12860                ssh select
 10470    1 3   6        80   fffffe8225fad420                 sh wait
 9662     1 3   6        80   fffffe821a15d8e0               perl wait
 5076     1 3   5        80   fffffe82140c9980                 sh wait
 7123     1 3   7         0   fffffe81f997ca80               perl biowait
 8956     1 3   5        80   fffffe8225f1c920                 sh wait
 9975     1 3   0        80   fffffe81f0b356e0                ssh select
 5270     1 3   2        80   fffffe822b3bb4a0                 sh wait
 8794     1 3   0        80   fffffe822b9d6760                ssh select
 5168     1 3   2        80   fffffe8215553540                 sh wait
 8565     1 3   0        80   fffffe821951e020                ssh select
 8522     1 3   0        80   fffffe822a8f4100                 sh wait
 8946     1 3   0        80   fffffe820dddda20                ssh select
 8677     1 3   5        80   fffffe822d94a560                 sh wait
 6875     1 3   0        80   fffffe821951e440                ssh select
 8644     1 3   7        80   fffffe821d24b660                 sh wait
 7694     1 3   0        80   fffffe82251dfb20                ssh select
 7939     1 3   3        80   fffffe820354a5a0                 sh wait
 5953     1 3   5        80   fffffe81f997c240                ssh select
 5952     1 3   6        80   fffffe8224c45720                 sh wait
 7773     1 3   0        80   fffffe822140b680                ssh select
 6643     1 3   7        80   fffffe822c5ab680                 sh wait
 6757     1 3   0        80   fffffe81f0125840                ssh select
 8943     1 3   4        80   fffffe81fc3e5640                 sh wait
 7329     1 3   0        80   fffffe822d946780                ssh select
 6222     1 3   0        80   fffffe820af7d1a0                 sh wait
 6930     1 3   0        80   fffffe821a2012a0                ssh select
 6708     1 3   6        80   fffffe821a15d0a0                 sh wait
 5857     1 3   6        80   fffffe8223ee1320                ssh select
 7463     1 3   2        80   fffffe821d4af220                 sh wait
 5817     1 3   0        80   fffffe8222cfd540                ssh select
 6855     1 3   5        80   fffffe8225a53780                 sh wait
 7144     1 3   0        80   fffffe821b5d0620                ssh select
 7222     1 3   5        80   fffffe8221863640                 sh wait
 6589     1 3   0        80   fffffe820af7d5c0                ssh select
 7157     1 3   3        80   fffffe822d94a140                 sh wait
 6877     1 3   2        80   fffffe82039c65e0                ssh select
 6057     1 3   7        80   fffffe821d4af640                 sh wait
 6392     1 3   5        80   fffffe822d71a6e0                ssh select
 4924     1 3   4        80   fffffe8222a64100                 sh wait
 5490     1 3   0        80   fffffe8218a57000                ssh select
 5061     1 3   6        80   fffffe81fbd2daa0                 sh wait
 6508     1 3   0        80   fffffe82251df2e0                ssh select
 4832     1 3   6        80   fffffe81fbd2d680                 sh wait
 5990     1 3   0        80   fffffe8225e1f2c0                ssh select
 5243     1 3   2        80   fffffe820dddd600                 sh wait
 3871     1 3   5        80   fffffe8221476b20                par pause
 2696     1 3   5        80   fffffe822aad3900                 sh wait
 4142     1 3   0        80   fffffe821a62c060                 sh wait
 4763     1 3   0        80   fffffe82299ed1a0                 sh wait
 5220     1 3   0        80   fffffe8222a64520                 sh wait
 4858     1 3   3        80   fffffe821daba920               cron pipe_rd
 3551     1 3   0        80   fffffe821de8b9e0                ssh select
 4159     1 3   5        80   fffffe8226fc48e0                 sh wait
 5091     1 3   0        80   fffffe8221863220                ssh select
 3848     1 3   0        80   fffffe822c143880                 sh wait
 4429     1 3   0        80   fffffe82140c9560                ssh select
 3178     1 3   0        80   fffffe81eb91aa40                 sh wait
 4329     1 3   0        80   fffffe820354a180                ssh select
 4193     1 3   1        80   fffffe820dddd1e0                 sh wait
 3408     1 3   0        80   fffffe82251df700                ssh select
 4951     1 3   0        80   fffffe8221ea5660               bash wait
 4422     1 3   5        80   fffffe820af7d9e0               sshd select
 3919     1 3   0        80   fffffe82299aa6a0               sshd select
 3312     1 3   0        80   fffffe8229b8eb80                ssh select
 3607     1 3   7        80   fffffe8225a53ba0               bash wait
 4021     1 3   0        80   fffffe821e8b72e0               sshd select
 3876     1 3   0        80   fffffe821daba0e0               sshd select
 3633     1 3   1        80   fffffe822aad34e0                ssh select
 3679     1 3   7        80   fffffe822a8f4520               bash wait
 3250     1 3   0        80   fffffe8224c45300               sshd select
 3676     1 3   0        80   fffffe821ffb8200               sshd select
 3306     1 3   0        80   fffffe82253fc020                ssh select
 3556     1 3   2        80   fffffe821951e860                 sh wait
 2753     1 3   0        80   fffffe8221215040                ssh select
 2752     1 3   0        80   fffffe821a201ae0               bash wait
 3789     1 3   0        80   fffffe8228146640               sshd select
 2376     1 3   5        80   fffffe820e8f99a0               sshd select
 1466     1 3   7        80   fffffe8221476700               bash wait
 1782     1 3   0        80   fffffe821daba500               tcsh ttyraw
 2030     1 3   2        80   fffffe821ffb8620               sshd select
 1515     1 3   5        80   fffffe822c5abaa0               sshd select
 1238     1 3   0        80   fffffe821d4afa60                ssh select
 853      1 3   3        80   fffffe822d772580                 sh wait
 1740     1 3   0        80   fffffe821e6ec1e0                ssh select
 459      1 3   7        80   fffffe821e6ec600                 sh wait
 458      1 3   0        80   fffffe821daaa940                ssh select
 457      1 3   1        80   fffffe821e6eca20                 sh wait
 1222     1 3   0        80   fffffe821dd2b1c0                ssh select
 1093     1 3   0        80   fffffe821dd2b5e0                 sh wait
 1092     1 3   0        80   fffffe821dd2ba00                ssh select
 835      1 3   0        80   fffffe821de8b1a0                 sh wait
 1212     1 3   0        80   fffffe821d19a9c0                ssh select
 1467     1 3   0        80   fffffe821e0db160          conserver nanoslp
 1082     1 3   5        80   fffffe821e0db580                 sh wait
 1081     1 3   0        80   fffffe821e0db9a0                ssh select
 824      1 3   5        80   fffffe821ef36140                 sh wait
 1079     1 3   0        80   fffffe821ef36560                ssh select
 1206     1 3   5        80   fffffe821ef36980                 sh wait
 1203     1 3   6        80   fffffe821f14e960          conserver select
 1324     1 3   0        80   fffffe821df920c0                ssh select
 299      1 3   0        80   fffffe821df924e0                 sh wait
 298      1 3   0        80   fffffe821df92900                ssh select
 297      1 3   0        80   fffffe821e2910a0                 sh wait
 296      1 3   0        80   fffffe821e2914c0                ssh select
 1703     1 3   0        80   fffffe821e2918e0                 sh wait
 1190     1 3   0        80   fffffe821e442080                ssh select
 1061     1 3   0        80   fffffe821e4424a0                 sh wait
 1188     1 3   0        80   fffffe821e4428c0                ssh select
 1187     1 3   0        80   fffffe821e6e9060                 sh wait
 674      1 3   0        80   fffffe821e6e9480                ssh select
 289      1 3   0        80   fffffe821e6e98a0                 sh wait
 288      1 3   0        80   fffffe821ff14040                ssh select
 927      1 3   0        80   fffffe821ff14460                 sh wait
 798      1 3   0        80   fffffe821ff14880                ssh select
 1181     1 3   0        80   fffffe822129d020                 sh wait
 1052     1 3   0        80   fffffe822129d440                ssh select
 1051     1 3   0        80   fffffe822dafd120                 sh wait
 1178     1 3   0        80   fffffe821f670280                ssh select
 1177     1 3   0        80   fffffe82283bd680                 sh wait
 1176     1 3   0        80   fffffe8225e1fb00                ssh select
 1431     1 3   7        80   fffffe822129d860                 sh wait
 790      1 3   0        80   fffffe821edea000          conserver select
 1173     1 3   0        80   fffffe821edea420          conserver nanoslp
 1044     1 3   0        80   fffffe82299aaac0                ssh select
 402      1 3   7        80   fffffe822140b260                 sh wait
 400      1 3   0        80   fffffe82214d7360          conserver select
 1293     1 3   0        80   fffffe822d1cc5a0                ssh select
 1292     1 3   0        80   fffffe821f670ac0                 sh wait
 267      1 3   0        80   fffffe82214d7780                ssh select
 266      1 3   4        80   fffffe82214d7ba0                 sh wait
 1415     1 3   5        80   fffffe821f5f2760          conserver select
 1030     1 3   0        80   fffffe821f5f2b80                ssh select
 1157     1 3   3        80   fffffe82299ed9e0                 sh wait
 1156     1 3   0        80   fffffe82299ed5c0                ssh select
 1155     1 3   0        80   fffffe8229e40ae0                 sh wait
 386      1 3   0        80   fffffe821f9a1320                ssh select
 385      1 3   7        80   fffffe821f9a1740                 sh wait
 384      1 3   0        80   fffffe821f9a1b60                ssh select
 447      1 3   7        80   fffffe82211cb300                 sh wait
 830      1 3   0        80   fffffe82211cb720                ssh select
 1021     1 3   7        80   fffffe82299aa280                 sh wait
 888      1 3   0        80   fffffe82211cbb40                ssh select
 1143     1 3   1        80   fffffe82214762e0                 sh wait
 176      1 3   0        80   fffffe8220dab2c0                ssh select
 815      1 3   0        80   fffffe8220dab6e0                 sh wait
 878      1 3   0        80   fffffe8220dabb00                ssh select
 1069     1 3   0        80   fffffe8220f432a0                 sh wait
 1196     1 3   7        80   fffffe8220f436c0                ssh select
 363      1 3   5        80   fffffe8220f43ae0                 sh wait
 361      1 3   0        80   fffffe821f6706a0          conserver select
 1115     1 3   0        80   fffffe8221cb31e0                ssh select
 986      1 3   5        80   fffffe8221cb3600                 sh wait
 985      1 3   0        80   fffffe8221cb3a20                ssh select
 984      1 3   5        80   fffffe822261e1c0                 sh wait
 1239     1 3   0        80   fffffe822261e5e0                ssh select
 854      1 3   5        80   fffffe822261ea00                 sh wait
 789      1 3   0        80   fffffe822a8ba1a0                ssh select
 1108     1 3   5        80   fffffe822a8ba5c0                 sh wait
 851      1 3   0        80   fffffe822a8ba9e0                ssh select
 210      1 3   5        80   fffffe8220918180                 sh wait
 913      1 3   0        80   fffffe82209185a0                ssh select
 208      1 3   5        80   fffffe82209189c0                 sh wait
 783      1 3   0        80   fffffe8222d0a160                ssh select
 974      1 3   0        80   fffffe8222d0a580                 sh wait
 1165     1 3   0        80   fffffe8222d0a9a0                ssh select
 1356     1 3   0        80   fffffe822287f140                 sh wait
 139      1 3   0        80   fffffe822287f560                ssh select
 138      1 3   0        80   fffffe822287f980                 sh wait
 137      1 3   5        80   fffffe8222cfd120          conserver select
 168      1 3   1        80   fffffe822dafd960          conserver select
 1127     1 3   5        80   fffffe8228f78620          conserver select
 933      1 3   5        80   fffffe822dda2920              getty ttyraw
 1031     1 3   0        80   fffffe822dd49100          conserver select
 797      1 3   5        80   fffffe8225a53360          conserver select
 829      1 3   3        80   fffffe82283bd260               cron nanoslp
 1019     1 3   3         0   fffffe822bcd0a00          python2.7 biowait
 112      1 3   0        80   fffffe8229b8e340              inetd kqueue
 772      1 3   0        80   fffffe8223ee1740              nginx kqueue
 803      1 3   0        80   fffffe8223ee1b60              nginx pause
 773      1 3   5        80   fffffe8229e406c0               qmgr kqueue
 951      1 3   5        80   fffffe8228146a60             pickup kqueue
 794      1 3   5        80   fffffe8229e402a0             master kqueue
 696      1 3   0        80   fffffe822d1cc180               sshd select
 725      1 3   3        80   fffffe822d772160           tac_plus select
 686      1 3   6        80   fffffe822806d240             powerd kqueue
 659      1 3   0        80   fffffe822806d660               ntpd pause
 606      4 3   6        80   fffffe822806da80              dhcpd parked
 606      3 3   6        80   fffffe8228146220              dhcpd kqueue
 606      2 3   6        80   fffffe8228f78200              dhcpd parked
 606      1 3   0        80   fffffe822986b1e0              dhcpd parked
 383  :   1 7   7         0   fffffe8228f78a40            syslogd
 1        1 3   3        80   fffffe810f45a0a0               init wait
 0      123 3   3       200   fffffe822986ba20            npfgc-0 xchicv
 0      122 3   5       200   fffffe822d1cc9c0              ipmi0 ipmi0
 0      121 3   0       200   fffffe822dda2500            physiod physiod
 0      120 3   0       200   fffffe822dd49520           aiodoned aiodoned
 0      119 3   0       200   fffffe822dd49940            ioflush syncer
 0      118 3   0       200   fffffe822dda20e0           pgdaemon pgdaemon
 0      115 3   0       280   fffffe810f4260c0              spkr1 bellcv
 0      114 3   4       280   fffffe810f4264e0           audiomix play
 0      113 3   4       280   fffffe810ed322a0           audiorec record
 0      112 3   5       200   fffffe810f2c9720          atapibus0 sccomp
 0      110 3   0       200   fffffe810f2ba700               usb2 usbevt
 0      109 3   0       200   fffffe810f2ba2e0               usb1 usbevt
 0      108 3   0       200   fffffe810f2bab20               usb3 usbevt
 0      107 3   0       200   fffffe810f2c9300               usb0 usbevt
 0      106 3   5       200   fffffe810f2e0b60               usb4 usbevt
 0      105 3   5       200   fffffe810f426900               usb5 usbevt
 0      104 3   0       200   fffffe810f45a4c0            rt_free rt_free
 0      103 3   0       200   fffffe810f45a8e0              unpgc unpgc
 0      102 3   3       200   fffffe810f463080    key_timehandler key_timehandler

 0      101 3   7       200   fffffe810f4634a0    icmp6_wqinput/7 icmp6_wqinput
 0      100 3   6       200   fffffe810f4638c0    icmp6_wqinput/6 icmp6_wqinput
 0       99 3   5       200   fffffe810f3ec060    icmp6_wqinput/5 icmp6_wqinput
 0       98 3   4       200   fffffe810f3ec480    icmp6_wqinput/4 icmp6_wqinput
 0       97 3   3       200   fffffe810f3ec8a0    icmp6_wqinput/3 icmp6_wqinput
 0       96 3   2       200   fffffe810f38d040    icmp6_wqinput/2 icmp6_wqinput
 0       95 3   1       200   fffffe810f38d460    icmp6_wqinput/1 icmp6_wqinput
 0       94 3   0       200   fffffe810f2c9b40    icmp6_wqinput/0 icmp6_wqinput
 0       93 3   0       200   fffffe810f38d880          nd6_timer nd6_timer
 0       92 3   7       200   fffffe810f34e020     icmp_wqinput/7 icmp_wqinput
 0       91 3   6       200   fffffe810f34e440     icmp_wqinput/6 icmp_wqinput
 0       90 3   5       200   fffffe810f34e860     icmp_wqinput/5 icmp_wqinput
 0       89 3   4       200   fffffe810f337000     icmp_wqinput/4 icmp_wqinput
 0       88 3   3       200   fffffe810f337420     icmp_wqinput/3 icmp_wqinput
 0       87 3   2       200   fffffe810f337840     icmp_wqinput/2 icmp_wqinput
 0       86 3   1       200   fffffe810f33d360     icmp_wqinput/1 icmp_wqinput
 0       85 3   0       200   fffffe810f33d780     icmp_wqinput/0 icmp_wqinput
 0       84 3   6       200   fffffe810f33dba0           rt_timer rt_timer
 0       83 3   7       200   fffffe810f1fc2c0        vmem_rehash vmem_rehash
 0       82 3   3       200   fffffe810f2d7340          coretemp3 coretemp3
 0       81 3   7       200   fffffe810f2d7760          coretemp2 coretemp2
 0       80 3   1       200   fffffe810f2d7b80          coretemp1 coretemp1
 0       79 3   6       200   fffffe810f2e0320          coretemp0 coretemp0
 0       78 3   4       200   fffffe810f2e0740               mfi0 mfi0
 0       69 3   0       200   fffffe810f1fc6e0            atabus1 atath
 0       68 3   0       200   fffffe810f1fcb00            atabus0 atath
 0       66 3   0       200   fffffe810ed326c0           scsibus0 sccomp
 0       65 3   5       200   fffffe810ed32ae0         usbtask-dr usbtsk
 0       64 3   0       200   fffffe810eca9280         usbtask-hc usbtsk
 0       63 3   7       200   fffffe810eca96a0               bnx3 bnx3
 0       62 3   1       200   fffffe810eca9ac0               bnx2 bnx2
 0       61 3   5       200   fffffe810eb60260               bnx1 bnx1
 0       60 3   4       200   fffffe810eb60680               bnx0 bnx0
 0       59 3   5       200   fffffe810eb60aa0               ipmi ipmipoll
 0       58 3   7       200   fffffe810ea73240            xcall/7 xcall
 0       57 1   7       200   fffffe810ea73660          softser/7
 0       56 1   7       200   fffffe810ea73a80          softclk/7
 0       55 1   7       200   fffffe810ea54220          softbio/7
 0       54 1   7       200   fffffe810ea54640          softnet/7
 0       53 1   7       201   fffffe810ea54a60             idle/7
 0       52 3   6       200   fffffe810ea3d200            xcall/6 xcall
 0       51 1   6       200   fffffe810ea3d620          softser/6
 0       50 1   6       200   fffffe810ea3da40          softclk/6
 0       49 1   6       200   fffffe810ea161e0          softbio/6
 0       48 1   6       200   fffffe810ea16600          softnet/6
 0    :  47 7   6       201   fffffe810ea16a20             idle/6
 0       46 3   5       200   fffffe810e9d71c0            xcall/5 xcall
 0       45 1   5       200   fffffe810e9d75e0          softser/5
 0       44 1   5       200   fffffe810e9d7a00          softclk/5
 0       43 1   5       200   fffffe810e9d01a0          softbio/5
 0       42 1   5       200   fffffe810e9d05c0          softnet/5
 0    :  41 7   5       201   fffffe810e9d09e0             idle/5
 0       40 3   4       200   fffffe810e9a1180            xcall/4 xcall
 0       39 1   4       200   fffffe810e9a15a0          softser/4
 0       38 1   4       200   fffffe810e9a19c0          softclk/4
 0       37 1   4       200   fffffe810e97a160          softbio/4
 0       36 1   4       200   fffffe810e97a580          softnet/4
 0    :  35 7   4       201   fffffe810e97a9a0             idle/4
 0       34 3   3       200   fffffe810e943140            xcall/3 xcall
 0       33 1   3       200   fffffe810e943560          softser/3
 0       32 1   3       200   fffffe810e943980          softclk/3
 0       31 1   3       200   fffffe810e934120          softbio/3
 0       30 1   3       200   fffffe810e934540          softnet/3
 0    :  29 7   3       201   fffffe810e934960             idle/3
 0       28 3   2       200   fffffe810e90d100            xcall/2 xcall
 0       27 1   2       200   fffffe810e90d520          softser/2
 0       26 1   2       200   fffffe810e90d940          softclk/2
 0       25 1   2       200   fffffe810e8f60e0          softbio/2
 0       24 1   2       200   fffffe810e8f6500          softnet/2
 0    :  23 7   2       201   fffffe810e8f6920             idle/2
 0       22 3   1       200   fffffe810e8c50c0            xcall/1 xcall
 0       21 1   1       200   fffffe810e8c54e0          softser/1
 0       20 1   1       200   fffffe810e8c5900          softclk/1
 0       19 1   1       200   fffffe810e8a90a0          softbio/1
 0       18 1   1       200   fffffe810e8a94c0          softnet/1
 0    :  17 7   1       201   fffffe810e8a98e0             idle/1
 0       16 3   5       200   fffffe822de9e080           lnxsyswq lnxsyswq
 0       15 3   5       200   fffffe822de9e4a0             sysmon smtaskq
 0       14 3   2       200   fffffe822de9e8c0         pmfsuspend pmfsuspend
 0       13 3   4       200   fffffe822e2c5060           pmfevent pmfevent
 0       12 3   0       200   fffffe822e2c5480         sopendfree sopendfr
 0       11 3   6       200   fffffe822e2c58a0           nfssilly nfssilly
 0       10 3   3       200   fffffe822f6dc040            cachegc cachegc
 0        9 3   3       200   fffffe822f6dc460             vdrain vdrain
 0        8 3   3       200   fffffe822f6dc880          modunload mod_unld
 0        7 3   0       200   fffffe822f6fd020            xcall/0 xcall
 0    :   6 7   0       200   fffffe822f6fd440          softser/0
 0    :   5 7   0       200   fffffe822f6fd860          softclk/0
 0        4 1   0       200   fffffe822f719000          softbio/0
 0        3 1   0       200   fffffe822f719420          softnet/0
 0        2 1   0       201   fffffe822f719840             idle/0
 0        1 3   2       200   ffffffff81481b20            swapper uvm
 db{0}: trace/t 0t9372
 trace: pid 9372 lid 1 at 0xffff80009f6f78e0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 rw_vector_enter() at netbsd:rw_vector_enter+0x3e8
 genfs_lock() at netbsd:genfs_lock+0x3c
 VOP_LOCK() at netbsd:VOP_LOCK+0x67
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 781169a3e2ca:
 db{0}: trace/t 0t8217
 trace: pid 8217 lid 1 at 0xffff80009f6be8e0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 rw_vector_enter() at netbsd:rw_vector_enter+0x3e8
 genfs_lock() at netbsd:genfs_lock+0x3c
 VOP_LOCK() at netbsd:VOP_LOCK+0x67
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7ac9b2a3e2ca:
 db{0}: trace/t 0t6001
 trace: pid 6001 lid 1 at 0xffff80009eff7950
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 ffs_balloc() at netbsd:ffs_balloc+0x1b06
 ufs_gop_alloc() at netbsd:ufs_gop_alloc+0xa1
 ufs_balloc_range() at netbsd:ufs_balloc_range+0x136
 ffs_write() at netbsd:ffs_write+0x4d7
 VOP_WRITE() at netbsd:VOP_WRITE+0x37
 vn_write() at netbsd:vn_write+0xec
 dofilewrite() at netbsd:dofilewrite+0x97
 sys_write() at netbsd:sys_write+0x5f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 4) ---
 79de3d23e02a:
 db{0}: trace/t 0t5868
 trace: pid 5868 lid 1 at 0xffff80009f0c1a50
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 ffs_update.part.3() at netbsd:ffs_update.part.3+0x292
 ufs_makeinode() at netbsd:ufs_makeinode+0x134
 ufs_create() at netbsd:ufs_create+0x31
 VOP_CREATE() at netbsd:VOP_CREATE+0x3d
 vn_open() at netbsd:vn_open+0x351
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7d9dfe63e2ca:
 db{0}: trace/t 0t11951
 trace: pid 11951 lid 1 at 0xffff80009f7a1c80
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_timedwait_sig() at netbsd:cv_timedwait_sig+0x107
 sbwait() at netbsd:sbwait+0x69
 soreceive() at netbsd:soreceive+0x1233
 dofileread() at netbsd:dofileread+0x8f
 sys_read() at netbsd:sys_read+0x5f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 3) ---
 6fd303c3e22a:
 db{0}: trace/t 0t9862
 trace: pid 9862 lid 1 at 0xffff80009f772b60
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait_sig() at netbsd:cv_wait_sig+0xfe
 do_sys_waitid() at netbsd:do_sys_waitid+0x44e
 do_sys_wait() at netbsd:do_sys_wait+0x8f
 sys___wait450() at netbsd:sys___wait450+0x42
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 449) ---
 7cc8fa83e06a:
 db{0}: trace/t 0t10228
 trace: pid 10228 lid 2 at 0xffff80009f71de40
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 417) ---
 767a01c3e10a:
 db{0}: trace/t 0t7123
 trace: pid 7123 lid 1 at 0xffff80009f5d9c10
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 VOP_BWRITE() at netbsd:VOP_BWRITE+0x51
 ufs_dirremove() at netbsd:ufs_dirremove+0xd8
 ufs_remove() at netbsd:ufs_remove+0x142
 VOP_REMOVE() at netbsd:VOP_REMOVE+0x37
 do_sys_unlinkat() at netbsd:do_sys_unlinkat+0x237
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 10) ---
 7973de4fb2aa:
 db{0}: trace/t 0t9662
 trace: pid 9662 lid 1 at 0xffff80009f074b60
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait_sig() at netbsd:cv_wait_sig+0xfe
 do_sys_waitid() at netbsd:do_sys_waitid+0x44e
 do_sys_wait() at netbsd:do_sys_wait+0x8f
 sys___wait450() at netbsd:sys___wait450+0x42
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 449) ---
 72c419a3e06a:
 db{0}: trace/t 0t9681
 trace: pid 9681 lid 1 at 0xffff80009f0fdb60
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait_sig() at netbsd:cv_wait_sig+0xfe
 do_sys_waitid() at netbsd:do_sys_waitid+0x44e
 do_sys_wait() at netbsd:do_sys_wait+0x8f
 sys___wait450() at netbsd:sys___wait450+0x42
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 449) ---
 74946003e06a:
 db{0}: trace/t 0t1019
 trace: pid 1019 lid 1 at 0xffff80009de6e8a0
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 ffs_update.part.3() at netbsd:ffs_update.part.3+0x292
 ffs_truncate() at netbsd:ffs_truncate+0x1046
 ufs_truncate_retry() at netbsd:ufs_truncate_retry+0x79
 ufs_setattr() at netbsd:ufs_setattr+0x44c
 VOP_SETATTR() at netbsd:VOP_SETATTR+0x34
 vn_open() at netbsd:vn_open+0x26e
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 79538a23e2ca:
 db{0}: trace/t 0t383
 trace: pid 383 lid 1 at 0x1
 sleepq_block() at netbsd:sleepq_block+0x110
 fatal page fault in supervisor mode
 trap type 6 code 0 rip 0xffffffff80238eaa cs 0x8 rflags 0x10246 cr2 0x9 ilevel 0x8 rsp 0xffff80008f69afe0
 curlwp 0xfffffe822f6fd440 pid 0.6 lowest kstack 0xffff80008f6972c0
 kernel: page fault trap, code=0
 Faulted in DDB; continuing...
 db{0}: c
 fatal breakpoint trap in supervisor mode
 trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0x9 ilevel 0x8 rsp 0xffff80008f69bcf8
 curlwp 0xfffffe822f6fd440 pid 0.6 lowest kstack 0xffff80008f6972c0
 Stopped in pid 0.6 (system) at  netbsd:breakpoint+0x5:  leave
 db{0}: ps
 PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 5868     1 3   5         0   fffffe8215553960           postdrop biowait
 11951    1 3   5        80   fffffe81da398080           sendmail netio
 6001     1 3   7         0   fffffe82180044a0             vmstat biowait
 9372     1 3   2         0   fffffe81f0125420               tail tstile
 8217     1 3   6         0   fffffe81efc4d320               tail tstile
 9862     1 3   3        80   fffffe81d8509060               cron wait
 4632     1 3   0        80   fffffe822b3bb080                ssh select
 4405     2 3   4        80   fffffe8224c45b40             expect select
 4405     1 3   0        80   fffffe821f14e540             expect parked
 4495     1 3   0        80   fffffe8226a1e8a0                 sh wait
 9356     1 3   5        80   fffffe821b5d0200               perl pipe_rd
 6548     1 3   5        80   fffffe8221863a60                 sh wait
 10228:   2 7   0         0   fffffe81d9f12440             expect
 10228    1 3   7        80   fffffe820e8f9160             expect parked
 7679     1 3   2        80   fffffe822c143460                 sh wait
 12514    1 3   2        80   fffffe821daaa100               perl pipe_rd
 10055    1 3   2        80   fffffe82140c9140                 sh wait
 11571    1 3   0        80   fffffe8221215460                ssh select
 7806     1 3   4        80   fffffe82205b9280                 sh wait
 9266     1 3   0        80   fffffe8221215880                ssh select
 10356    1 3   5        80   fffffe8229b8e760                 sh wait
 7934     1 3   0        80   fffffe8225e1f6e0                ssh select
 10592    1 3   1        80   fffffe82170dc500                 sh wait
 9681     1 3   2        80   fffffe820e8f9580               perl wait
 9305     1 3   1        80   fffffe8219f67b60                 sh wait
 9908     1 3   0        80   fffffe8225fad840                ssh select
 8779     1 3   0        80   fffffe82205b96a0                 sh wait
 6856     1 3   0        80   fffffe8215553120                ssh select
 8803     1 3   1        80   fffffe81eb91a620                 sh wait
 8994     1 3   0        80   fffffe81d9f12860                ssh select
 10470    1 3   6        80   fffffe8225fad420                 sh wait
 9662     1 3   6        80   fffffe821a15d8e0               perl wait
 5076     1 3   5        80   fffffe82140c9980                 sh wait
 7123     1 3   7         0   fffffe81f997ca80               perl biowait
 8956     1 3   5        80   fffffe8225f1c920                 sh wait
 9975     1 3   0        80   fffffe81f0b356e0                ssh select
 5270     1 3   2        80   fffffe822b3bb4a0                 sh wait
 8794     1 3   0        80   fffffe822b9d6760                ssh select
 5168     1 3   2        80   fffffe8215553540                 sh wait
 8565     1 3   0        80   fffffe821951e020                ssh select
 8522     1 3   0        80   fffffe822a8f4100                 sh wait
 8946     1 3   0        80   fffffe820dddda20                ssh select
 8677     1 3   5        80   fffffe822d94a560                 sh wait
 6875     1 3   0        80   fffffe821951e440                ssh select
 8644     1 3   7        80   fffffe821d24b660                 sh wait
 7694     1 3   0        80   fffffe82251dfb20                ssh select
 7939     1 3   3        80   fffffe820354a5a0                 sh wait
 5953     1 3   5        80   fffffe81f997c240                ssh select
 5952     1 3   6        80   fffffe8224c45720                 sh wait
 7773     1 3   0        80   fffffe822140b680                ssh select
 6643     1 3   7        80   fffffe822c5ab680                 sh wait
 6757     1 3   0        80   fffffe81f0125840                ssh select
 8943     1 3   4        80   fffffe81fc3e5640                 sh wait
 7329     1 3   0        80   fffffe822d946780                ssh select
 6222     1 3   0        80   fffffe820af7d1a0                 sh wait
 6930     1 3   0        80   fffffe821a2012a0                ssh select
 6708     1 3   6        80   fffffe821a15d0a0                 sh wait
 5857     1 3   6        80   fffffe8223ee1320                ssh select
 7463     1 3   2        80   fffffe821d4af220                 sh wait
 5817     1 3   0        80   fffffe8222cfd540                ssh select
 6855     1 3   5        80   fffffe8225a53780                 sh wait
 7144     1 3   0        80   fffffe821b5d0620                ssh select
 7222     1 3   5        80   fffffe8221863640                 sh wait
 6589     1 3   0        80   fffffe820af7d5c0                ssh select
 7157     1 3   3        80   fffffe822d94a140                 sh wait
 6877     1 3   2        80   fffffe82039c65e0                ssh select
 6057     1 3   7        80   fffffe821d4af640                 sh wait
 6392     1 3   5        80   fffffe822d71a6e0                ssh select
 4924     1 3   4        80   fffffe8222a64100                 sh wait
 5490     1 3   0        80   fffffe8218a57000                ssh select
 5061     1 3   6        80   fffffe81fbd2daa0                 sh wait
 6508     1 3   0        80   fffffe82251df2e0                ssh select
 4832     1 3   6        80   fffffe81fbd2d680                 sh wait
 5990     1 3   0        80   fffffe8225e1f2c0                ssh select
 5243     1 3   2        80   fffffe820dddd600                 sh wait
 3871     1 3   5        80   fffffe8221476b20                par pause
 2696     1 3   5        80   fffffe822aad3900                 sh wait
 4142     1 3   0        80   fffffe821a62c060                 sh wait
 4763     1 3   0        80   fffffe82299ed1a0                 sh wait
 5220     1 3   0        80   fffffe8222a64520                 sh wait
 4858     1 3   3        80   fffffe821daba920               cron pipe_rd
 3551     1 3   0        80   fffffe821de8b9e0                ssh select
 4159     1 3   5        80   fffffe8226fc48e0                 sh wait
 5091     1 3   0        80   fffffe8221863220                ssh select
 3848     1 3   0        80   fffffe822c143880                 sh wait
 4429     1 3   0        80   fffffe82140c9560                ssh select
 3178     1 3   0        80   fffffe81eb91aa40                 sh wait
 4329     1 3   0        80   fffffe820354a180                ssh select
 4193     1 3   1        80   fffffe820dddd1e0                 sh wait
 3408     1 3   0        80   fffffe82251df700                ssh select
 4951     1 3   0        80   fffffe8221ea5660               bash wait
 4422     1 3   5        80   fffffe820af7d9e0               sshd select
 3919     1 3   0        80   fffffe82299aa6a0               sshd select
 3312     1 3   0        80   fffffe8229b8eb80                ssh select
 3607     1 3   7        80   fffffe8225a53ba0               bash wait
 4021     1 3   0        80   fffffe821e8b72e0               sshd select
 3876     1 3   0        80   fffffe821daba0e0               sshd select
 3633     1 3   1        80   fffffe822aad34e0                ssh select
 3679     1 3   7        80   fffffe822a8f4520               bash wait
 3250     1 3   0        80   fffffe8224c45300               sshd select
 3676     1 3   0        80   fffffe821ffb8200               sshd select
 3306     1 3   0        80   fffffe82253fc020                ssh select
 3556     1 3   2        80   fffffe821951e860                 sh wait
 2753     1 3   0        80   fffffe8221215040                ssh select
 2752     1 3   0        80   fffffe821a201ae0               bash wait
 3789     1 3   0        80   fffffe8228146640               sshd select
 2376     1 3   5        80   fffffe820e8f99a0               sshd select
 1466     1 3   7        80   fffffe8221476700               bash wait
 1782     1 3   0        80   fffffe821daba500               tcsh ttyraw
 2030     1 3   2        80   fffffe821ffb8620               sshd select
 1515     1 3   5        80   fffffe822c5abaa0               sshd select
 1238     1 3   0        80   fffffe821d4afa60                ssh select
 853      1 3   3        80   fffffe822d772580                 sh wait
 1740     1 3   0        80   fffffe821e6ec1e0                ssh select
 459      1 3   7        80   fffffe821e6ec600                 sh wait
 458      1 3   0        80   fffffe821daaa940                ssh select
 457      1 3   1        80   fffffe821e6eca20                 sh wait
 1222     1 3   0        80   fffffe821dd2b1c0                ssh select
 1093     1 3   0        80   fffffe821dd2b5e0                 sh wait
 1092     1 3   0        80   fffffe821dd2ba00                ssh select
 835      1 3   0        80   fffffe821de8b1a0                 sh wait
 1212     1 3   0        80   fffffe821d19a9c0                ssh select
 1467     1 3   0        80   fffffe821e0db160          conserver nanoslp
 1082     1 3   5        80   fffffe821e0db580                 sh wait
 1081     1 3   0        80   fffffe821e0db9a0                ssh select
 824      1 3   5        80   fffffe821ef36140                 sh wait
 1079     1 3   0        80   fffffe821ef36560                ssh select
 1206     1 3   5        80   fffffe821ef36980                 sh wait
 1203     1 3   6        80   fffffe821f14e960          conserver select
 1324     1 3   0        80   fffffe821df920c0                ssh select
 299      1 3   0        80   fffffe821df924e0                 sh wait
 298      1 3   0        80   fffffe821df92900                ssh select
 297      1 3   0        80   fffffe821e2910a0                 sh wait
 296      1 3   0        80   fffffe821e2914c0                ssh select
 1703     1 3   0        80   fffffe821e2918e0                 sh wait
 1190     1 3   0        80   fffffe821e442080                ssh select
 1061     1 3   0        80   fffffe821e4424a0                 sh wait
 1188     1 3   0        80   fffffe821e4428c0                ssh select
 1187     1 3   0        80   fffffe821e6e9060                 sh wait
 674      1 3   0        80   fffffe821e6e9480                ssh select
 289      1 3   0        80   fffffe821e6e98a0                 sh wait
 288      1 3   0        80   fffffe821ff14040                ssh select
 927      1 3   0        80   fffffe821ff14460                 sh wait
 798      1 3   0        80   fffffe821ff14880                ssh select
 1181     1 3   0        80   fffffe822129d020                 sh wait
 1052     1 3   0        80   fffffe822129d440                ssh select
 1051     1 3   0        80   fffffe822dafd120                 sh wait
 1178     1 3   0        80   fffffe821f670280                ssh select
 1177     1 3   0        80   fffffe82283bd680                 sh wait
 1176     1 3   0        80   fffffe8225e1fb00                ssh select
 1431     1 3   7        80   fffffe822129d860                 sh wait
 790      1 3   0        80   fffffe821edea000          conserver select
 1173     1 3   0        80   fffffe821edea420          conserver nanoslp
 1044     1 3   0        80   fffffe82299aaac0                ssh select
 402      1 3   7        80   fffffe822140b260                 sh wait
 400      1 3   0        80   fffffe82214d7360          conserver select
 1293     1 3   0        80   fffffe822d1cc5a0                ssh select
 1292     1 3   0        80   fffffe821f670ac0                 sh wait
 267      1 3   0        80   fffffe82214d7780                ssh select
 266      1 3   4        80   fffffe82214d7ba0                 sh wait
 1415     1 3   5        80   fffffe821f5f2760          conserver select
 1030     1 3   0        80   fffffe821f5f2b80                ssh select
 1157     1 3   3        80   fffffe82299ed9e0                 sh wait
 1156     1 3   0        80   fffffe82299ed5c0                ssh select
 1155     1 3   0        80   fffffe8229e40ae0                 sh wait
 386      1 3   0        80   fffffe821f9a1320                ssh select
 385      1 3   7        80   fffffe821f9a1740                 sh wait
 384      1 3   0        80   fffffe821f9a1b60                ssh select
 447      1 3   7        80   fffffe82211cb300                 sh wait
 830      1 3   0        80   fffffe82211cb720                ssh select
 1021     1 3   7        80   fffffe82299aa280                 sh wait
 888      1 3   0        80   fffffe82211cbb40                ssh select
 1143     1 3   1        80   fffffe82214762e0                 sh wait
 176      1 3   0        80   fffffe8220dab2c0                ssh select
 815      1 3   0        80   fffffe8220dab6e0                 sh wait
 878      1 3   0        80   fffffe8220dabb00                ssh select
 1069     1 3   0        80   fffffe8220f432a0                 sh wait
 1196     1 3   7        80   fffffe8220f436c0                ssh select
 363      1 3   5        80   fffffe8220f43ae0                 sh wait
 361      1 3   0        80   fffffe821f6706a0          conserver select
 1115     1 3   0        80   fffffe8221cb31e0                ssh select
 986      1 3   5        80   fffffe8221cb3600                 sh wait
 985      1 3   0        80   fffffe8221cb3a20                ssh select
 984      1 3   5        80   fffffe822261e1c0                 sh wait
 1239     1 3   0        80   fffffe822261e5e0                ssh select
 854      1 3   5        80   fffffe822261ea00                 sh wait
 789      1 3   0        80   fffffe822a8ba1a0                ssh select
 1108     1 3   5        80   fffffe822a8ba5c0                 sh wait
 851      1 3   0        80   fffffe822a8ba9e0                ssh select
 210      1 3   5        80   fffffe8220918180                 sh wait
 913      1 3   0        80   fffffe82209185a0                ssh select
 208      1 3   5        80   fffffe82209189c0                 sh wait
 783      1 3   0        80   fffffe8222d0a160                ssh select
 974      1 3   0        80   fffffe8222d0a580                 sh wait
 1165     1 3   0        80   fffffe8222d0a9a0                ssh select
 1356     1 3   0        80   fffffe822287f140                 sh wait
 139      1 3   0        80   fffffe822287f560                ssh select
 138      1 3   0        80   fffffe822287f980                 sh wait
 137      1 3   5        80   fffffe8222cfd120          conserver select
 168      1 3   1        80   fffffe822dafd960          conserver select
 1127     1 3   5        80   fffffe8228f78620          conserver select
 933      1 3   5        80   fffffe822dda2920              getty ttyraw
 1031     1 3   0        80   fffffe822dd49100          conserver select
 797      1 3   5        80   fffffe8225a53360          conserver select
 829      1 3   3        80   fffffe82283bd260               cron nanoslp
 1019     1 3   3         0   fffffe822bcd0a00          python2.7 biowait
 112      1 3   0        80   fffffe8229b8e340              inetd kqueue
 772      1 3   0        80   fffffe8223ee1740              nginx kqueue
 803      1 3   0        80   fffffe8223ee1b60              nginx pause
 773      1 3   5        80   fffffe8229e406c0               qmgr kqueue
 951      1 3   5        80   fffffe8228146a60             pickup kqueue
 794      1 3   5        80   fffffe8229e402a0             master kqueue
 696      1 3   0        80   fffffe822d1cc180               sshd select
 725      1 3   3        80   fffffe822d772160           tac_plus select
 686      1 3   6        80   fffffe822806d240             powerd kqueue
 659      1 3   0        80   fffffe822806d660               ntpd pause
 606      4 3   6        80   fffffe822806da80              dhcpd parked
 606      3 3   6        80   fffffe8228146220              dhcpd kqueue
 606      2 3   6        80   fffffe8228f78200              dhcpd parked
 606      1 3   0        80   fffffe822986b1e0              dhcpd parked
 383      1 2   7         0   fffffe8228f78a40            syslogd
 1        1 3   3        80   fffffe810f45a0a0               init wait
 0      123 3   3       200   fffffe822986ba20            npfgc-0 xchicv
 0      122 3   5       200   fffffe822d1cc9c0              ipmi0 ipmi0
 0      121 3   0       200   fffffe822dda2500            physiod physiod
 0      120 3   0       200   fffffe822dd49520           aiodoned aiodoned
 0      119 3   0       200   fffffe822dd49940            ioflush syncer
 0      118 3   0       200   fffffe822dda20e0           pgdaemon pgdaemon
 0      115 3   0       280   fffffe810f4260c0              spkr1 bellcv
 0      114 3   4       280   fffffe810f4264e0           audiomix play
 0      113 3   4       280   fffffe810ed322a0           audiorec record
 0      112 3   5       200   fffffe810f2c9720          atapibus0 sccomp
 0      110 3   0       200   fffffe810f2ba700               usb2 usbevt
 0      109 3   0       200   fffffe810f2ba2e0               usb1 usbevt
 0      108 3   0       200   fffffe810f2bab20               usb3 usbevt
 0      107 3   0       200   fffffe810f2c9300               usb0 usbevt
 0      106 3   5       200   fffffe810f2e0b60               usb4 usbevt
 0      105 3   5       200   fffffe810f426900               usb5 usbevt
 0      104 3   0       200   fffffe810f45a4c0            rt_free rt_free
 0      103 3   0       200   fffffe810f45a8e0              unpgc unpgc
 0      102 3   3       200   fffffe810f463080    key_timehandler key_timehandler

 0      101 3   7       200   fffffe810f4634a0    icmp6_wqinput/7 icmp6_wqinput
 0      100 3   6       200   fffffe810f4638c0    icmp6_wqinput/6 icmp6_wqinput
 0       99 3   5       200   fffffe810f3ec060    icmp6_wqinput/5 icmp6_wqinput
 0       98 3   4       200   fffffe810f3ec480    icmp6_wqinput/4 icmp6_wqinput
 0       97 3   3       200   fffffe810f3ec8a0    icmp6_wqinput/3 icmp6_wqinput
 0       96 3   2       200   fffffe810f38d040    icmp6_wqinput/2 icmp6_wqinput
 0       95 3   1       200   fffffe810f38d460    icmp6_wqinput/1 icmp6_wqinput
 0       94 3   0       200   fffffe810f2c9b40    icmp6_wqinput/0 icmp6_wqinput
 0       93 3   0       200   fffffe810f38d880          nd6_timer nd6_timer
 0       92 3   7       200   fffffe810f34e020     icmp_wqinput/7 icmp_wqinput
 0       91 3   6       200   fffffe810f34e440     icmp_wqinput/6 icmp_wqinput
 0       90 3   5       200   fffffe810f34e860     icmp_wqinput/5 icmp_wqinput
 0       89 3   4       200   fffffe810f337000     icmp_wqinput/4 icmp_wqinput
 0       88 3   3       200   fffffe810f337420     icmp_wqinput/3 icmp_wqinput
 0       87 3   2       200   fffffe810f337840     icmp_wqinput/2 icmp_wqinput
 0       86 3   1       200   fffffe810f33d360     icmp_wqinput/1 icmp_wqinput
 0       85 3   0       200   fffffe810f33d780     icmp_wqinput/0 icmp_wqinput
 0       84 3   6       200   fffffe810f33dba0           rt_timer rt_timer
 0       83 3   7       200   fffffe810f1fc2c0        vmem_rehash vmem_rehash
 0       82 3   3       200   fffffe810f2d7340          coretemp3 coretemp3
 0       81 3   7       200   fffffe810f2d7760          coretemp2 coretemp2
 0       80 3   1       200   fffffe810f2d7b80          coretemp1 coretemp1
 0       79 3   6       200   fffffe810f2e0320          coretemp0 coretemp0
 0       78 3   4       200   fffffe810f2e0740               mfi0 mfi0
 0       69 3   0       200   fffffe810f1fc6e0            atabus1 atath
 0       68 3   0       200   fffffe810f1fcb00            atabus0 atath
 0       66 3   0       200   fffffe810ed326c0           scsibus0 sccomp
 0       65 3   5       200   fffffe810ed32ae0         usbtask-dr usbtsk
 0       64 3   0       200   fffffe810eca9280         usbtask-hc usbtsk
 0       63 3   7       200   fffffe810eca96a0               bnx3 bnx3
 0       62 3   1       200   fffffe810eca9ac0               bnx2 bnx2
 0       61 3   5       200   fffffe810eb60260               bnx1 bnx1
 0       60 3   4       200   fffffe810eb60680               bnx0 bnx0
 0       59 3   1       200   fffffe810eb60aa0               ipmi ipmipoll
 0       58 3   7       200   fffffe810ea73240            xcall/7 xcall
 0       57 1   7       200   fffffe810ea73660          softser/7
 0       56 1   7       200   fffffe810ea73a80          softclk/7
 0       55 1   7       200   fffffe810ea54220          softbio/7
 0       54 1   7       200   fffffe810ea54640          softnet/7
 0    :  53 7   7       201   fffffe810ea54a60             idle/7
 0       52 3   6       200   fffffe810ea3d200            xcall/6 xcall
 0       51 1   6       200   fffffe810ea3d620          softser/6
 0       50 1   6       200   fffffe810ea3da40          softclk/6
 0       49 1   6       200   fffffe810ea161e0          softbio/6
 0       48 1   6       200   fffffe810ea16600          softnet/6
 0    :  47 7   6       201   fffffe810ea16a20             idle/6
 0       46 3   5       200   fffffe810e9d71c0            xcall/5 xcall
 0       45 1   5       200   fffffe810e9d75e0          softser/5
 0       44 1   5       200   fffffe810e9d7a00          softclk/5
 0       43 1   5       200   fffffe810e9d01a0          softbio/5
 0       42 1   5       200   fffffe810e9d05c0          softnet/5
 0    :  41 7   5       201   fffffe810e9d09e0             idle/5
 0       40 3   4       200   fffffe810e9a1180            xcall/4 xcall
 0       39 1   4       200   fffffe810e9a15a0          softser/4
 0       38 1   4       200   fffffe810e9a19c0          softclk/4
 0       37 1   4       200   fffffe810e97a160          softbio/4
 0       36 1   4       200   fffffe810e97a580          softnet/4
 0    :  35 7   4       201   fffffe810e97a9a0             idle/4
 0       34 3   3       200   fffffe810e943140            xcall/3 xcall
 0       33 1   3       200   fffffe810e943560          softser/3
 0       32 1   3       200   fffffe810e943980          softclk/3
 0       31 1   3       200   fffffe810e934120          softbio/3
 0       30 1   3       200   fffffe810e934540          softnet/3
 0    :  29 7   3       201   fffffe810e934960             idle/3
 0       28 3   2       200   fffffe810e90d100            xcall/2 xcall
 0       27 1   2       200   fffffe810e90d520          softser/2
 0       26 1   2       200   fffffe810e90d940          softclk/2
 0       25 1   2       200   fffffe810e8f60e0          softbio/2
 0       24 1   2       200   fffffe810e8f6500          softnet/2
 0    :  23 7   2       201   fffffe810e8f6920             idle/2
 0       22 3   1       200   fffffe810e8c50c0            xcall/1 xcall
 0       21 1   1       200   fffffe810e8c54e0          softser/1
 0       20 1   1       200   fffffe810e8c5900          softclk/1
 0       19 1   1       200   fffffe810e8a90a0          softbio/1
 0       18 1   1       200   fffffe810e8a94c0          softnet/1
 0    :  17 7   1       201   fffffe810e8a98e0             idle/1
 0       16 3   5       200   fffffe822de9e080           lnxsyswq lnxsyswq
 0       15 3   5       200   fffffe822de9e4a0             sysmon smtaskq
 0       14 3   2       200   fffffe822de9e8c0         pmfsuspend pmfsuspend
 0       13 3   4       200   fffffe822e2c5060           pmfevent pmfevent
 0       12 3   0       200   fffffe822e2c5480         sopendfree sopendfr
 0       11 3   6       200   fffffe822e2c58a0           nfssilly nfssilly
 0       10 3   3       200   fffffe822f6dc040            cachegc cachegc
 0        9 3   3       200   fffffe822f6dc460             vdrain vdrain
 0        8 3   3       200   fffffe822f6dc880          modunload mod_unld
 0        7 3   0       200   fffffe822f6fd020            xcall/0 xcall
 0    :   6 7   0       200   fffffe822f6fd440          softser/0
 0    :   5 7   0       200   fffffe822f6fd860          softclk/0
 0        4 1   0       200   fffffe822f719000          softbio/0
 0        3 1   0       200   fffffe822f719420          softnet/0
 0        2 1   0       201   fffffe822f719840             idle/0
 0        1 3   2       200   ffffffff81481b20            swapper uvm
 db{0}: trace/t 0t5868
 trace: pid 5868 lid 1 at 0xffff80009f0c1a50
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 ffs_update.part.3() at netbsd:ffs_update.part.3+0x292
 ufs_makeinode() at netbsd:ufs_makeinode+0x134
 ufs_create() at netbsd:ufs_create+0x31
 VOP_CREATE() at netbsd:VOP_CREATE+0x3d
 vn_open() at netbsd:vn_open+0x351
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7d9dfe63e2ca:
 db{0}: trace/t 0t6001
 trace: pid 6001 lid 1 at 0xffff80009eff7950
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 ffs_balloc() at netbsd:ffs_balloc+0x1b06
 ufs_gop_alloc() at netbsd:ufs_gop_alloc+0xa1
 ufs_balloc_range() at netbsd:ufs_balloc_range+0x136
 ffs_write() at netbsd:ffs_write+0x4d7
 VOP_WRITE() at netbsd:VOP_WRITE+0x37
 vn_write() at netbsd:vn_write+0xec
 dofilewrite() at netbsd:dofilewrite+0x97
 sys_write() at netbsd:sys_write+0x5f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 4) ---
 79de3d23e02a:
 db{0}: trace/t 0t9372
 trace: pid 9372 lid 1 at 0xffff80009f6f78e0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 rw_vector_enter() at netbsd:rw_vector_enter+0x3e8
 genfs_lock() at netbsd:genfs_lock+0x3c
 VOP_LOCK() at netbsd:VOP_LOCK+0x67
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 781169a3e2ca:
 db{0}: trace/t 0t8217
 trace: pid 8217 lid 1 at 0xffff80009f6be8e0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 rw_vector_enter() at netbsd:rw_vector_enter+0x3e8
 genfs_lock() at netbsd:genfs_lock+0x3c
 VOP_LOCK() at netbsd:VOP_LOCK+0x67
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7ac9b2a3e2ca:
 db{0}: trace/t 0t10228
 trace: pid 10228 lid 2 at 0xffff80009f71de40
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 417) ---
 767a01c3e10a:
 db{0}: trace/t 0t7123
 trace: pid 7123 lid 1 at 0xffff80009f5d9c10
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 VOP_BWRITE() at netbsd:VOP_BWRITE+0x51
 ufs_dirremove() at netbsd:ufs_dirremove+0xd8
 ufs_remove() at netbsd:ufs_remove+0x142
 VOP_REMOVE() at netbsd:VOP_REMOVE+0x37
 do_sys_unlinkat() at netbsd:do_sys_unlinkat+0x237
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 10) ---
 7973de4fb2aa:
 db{0}: trace/t 0t1019
 trace: pid 1019 lid 1 at 0xffff80009de6e8a0
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 ffs_update.part.3() at netbsd:ffs_update.part.3+0x292
 ffs_truncate() at netbsd:ffs_truncate+0x1046
 ufs_truncate_retry() at netbsd:ufs_truncate_retry+0x79
 ufs_setattr() at netbsd:ufs_setattr+0x44c
 VOP_SETATTR() at netbsd:VOP_SETATTR+0x34
 vn_open() at netbsd:vn_open+0x26e
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 79538a23e2ca:
 db{0}: call cpu_reset

From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Wed, 04 Apr 2018 19:00:49 +0200 (CEST)

 a new follow-up with a new hang with similar symptoms.

 This time quite a number of processes in "tstile" and a few in
 biowait.

 Edited DDB console log follows.

 login: [-- MARK -- Wed Apr  4 16:00:00 2018]
 fatal breakpoint trap in supervisor mode
 trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0x78a14a33a1e8
  ilevel 0x8 rsp 0xffff80008f69bc78
 curlwp 0xfffffe822f6fd440 pid 0.6 lowest kstack 0xffff80008f6972c0
 Stopped in pid 0.6 (system) at  netbsd:breakpoint+0x5:  leave
 db{0}: trace
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 bus_space_read_4() at netbsd:bus_space_read_4+0xa
 mfi_intr() at netbsd:mfi_intr+0x1e
 intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
 Xhandle_ioapic_level8() at netbsd:Xhandle_ioapic_level8+0x6a
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}: machine cpu 1
 using CPU 1
 db{0}: trace
 x86_pause() at netbsd:x86_pause+0x2
 kevent1() at netbsd:kevent1+0x648
 sys___kevent50() at netbsd:sys___kevent50+0x33
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 435) ---
 7ae67143e40a:
 db{0}: machine cpu 2
 using CPU 2
 db{0}: trace
 x86_stihlt() at netbsd:x86_stihlt+0x6
 acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
 acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
 idle_loop() at netbsd:idle_loop+0x18c
 cpu_hatch() at netbsd:cpu_hatch+0x17d
 md_root_setconf() at netbsd:md_root_setconf
 db{0}: machine cpu 3
 using CPU 3
 db{0}: trace
 x86_stihlt() at netbsd:x86_stihlt+0x6
 acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
 acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
 idle_loop() at netbsd:idle_loop+0x18c
 cpu_hatch() at netbsd:cpu_hatch+0x17d
 md_root_setconf() at netbsd:md_root_setconf
 db{0}: machine cpu 4
 using CPU 4
 db{0}: trace
 x86_stihlt() at netbsd:x86_stihlt+0x6
 acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
 acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
 idle_loop() at netbsd:idle_loop+0x18c
 cpu_hatch() at netbsd:cpu_hatch+0x17d
 md_root_setconf() at netbsd:md_root_setconf
 db{0}: machine cpu 5
 using CPU 5
 db{0}: trace
 _kernel_lock() at netbsd:_kernel_lock+0x96
 nd6_timer_work() at netbsd:nd6_timer_work+0x49
 workqueue_worker() at netbsd:workqueue_worker+0xcd
 db{0}: machine cpu 6
 using CPU 6
 db{0}: trace
 x86_stihlt() at netbsd:x86_stihlt+0x6
 acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
 acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
 idle_loop() at netbsd:idle_loop+0x18c
 cpu_hatch() at netbsd:cpu_hatch+0x17d
 md_root_setconf() at netbsd:md_root_setconf
 db{0}: machine cpu 7
 using CPU 7
 db{0}: trace
 x86_stihlt() at netbsd:x86_stihlt+0x6
 acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
 acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
 idle_loop() at netbsd:idle_loop+0x18c
 cpu_hatch() at netbsd:cpu_hatch+0x17d
 md_root_setconf() at netbsd:md_root_setconf
 db{0}: ps
 PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 5640     1 3   3         0   fffffe817b781140               cron tstile
 13063    1 3   2         0   fffffe82223452e0               cron tstile
 8710     1 3   2         0   fffffe817b781560               cron wait
 8453     1 3   3         0   fffffe817b781980               cron wait
 7684     1 3   3         0   fffffe8221017120               cron tstile
 1795     1 3   3         0   fffffe8221017540               cron wait
 770      1 3   5         0   fffffe821f18a900               cron tstile
 769      1 3   3         0   fffffe8221017960               cron tstile
 768      1 3   7         0   fffffe820f1fd580               cron tstile
 13308    1 3   1         0   fffffe821f786620               cron tstile
 14541    1 3   2         0   fffffe821e8b42a0               cron tstile
 14861    1 3   7         0   fffffe8180a91100               cron wait
 13152    1 3   5         0   fffffe8180a91520               cron wait
 12852    1 3   1         0   fffffe8180a91940               cron wait
 11020    1 3   2         0   fffffe8188fa90e0               cron wait
 17742    1 3   3         0   fffffe8188fa9500               cron wait
 15140    1 3   3         0   fffffe8188fa9920               cron tstile
 15067    1 3   3         0   fffffe81f7d541c0               cron wait
 11390    1 3   3         0   fffffe821fed45a0               cron tstile
 14640    1 3   2         0   fffffe822002c160               cron tstile
 17396    1 3   2         0   fffffe821cf5aba0               cron wait
 12021    1 3   3         0   fffffe82275b0280               cron wait
 13774    1 3   5         0   fffffe8229b76a00               cron tstile
 18260    1 3   3         0   fffffe81c991a4c0               cron tstile
 16295    1 3   1         0   fffffe8226aecae0               cron tstile
 18585    1 3   2         0   fffffe81e492e420               cron tstile
 17773    1 3   5         0   fffffe822455d900               cron wait
 15341    1 3   1         0   fffffe8210253120               cron wait
 17117    1 3   2         0   fffffe8221604860               cron wait
 14560    1 3   3         0   fffffe822178c000               cron wait
 15183    1 3   3         0   fffffe821d68b6a0               cron tstile
 17051    1 3   2         0   fffffe821e07b260               cron tstile
 15464    1 3   2         0   fffffe81dda0ba60               cron wait
 17435    1 3   3         0   fffffe822c6926c0               cron wait
 12406    1 3   5         0   fffffe82260f6b40             master tstile
 14315    1 3   3         0   fffffe822dba2640               cron tstile
 15832    1 3   3         0   fffffe8225d154c0               cron wait
 12474    1 3   1         0   fffffe81e492e000             expect xchicv
 12734    1 3   1        80   fffffe8221604440                 sh wait
 14618    1 3   1        80   fffffe822984c1e0               perl pipe_rd
 13905    1 3   1        80   fffffe81d2bdb8c0                 sh wait
 12785    1 3   1        80   fffffe82268cf560                ssh select
 14284    2 3   0        80   fffffe82262548c0             expect select
 14284    1 3   0        80   fffffe821f700760             expect parked
 14946    1 3   6        80   fffffe821a5a60a0                 sh wait
 12207    1 3   6        80   fffffe82233941e0               perl pipe_rd
 13422    1 3   6        80   fffffe81e289f480                 sh wait
 12051    1 3   0        80   fffffe8220655540                ssh select
 9942     2 3   6        80   fffffe81e289f8a0             expect select
 9942     1 3   0        80   fffffe81d4261ba0             expect parked
 10880    1 3   0        80   fffffe81e9aad860                 sh wait
 11122    1 3   0        80   fffffe822dba2220               perl pipe_rd
 10951    1 3   5        80   fffffe821e07b680                 sh wait
 17641    1 3   0        80   fffffe820d9cc5a0                ssh select
 15330    1 3   4        80   fffffe822ab49b80                 sh wait
 16497    1 3   0        80   fffffe822da7a980                ssh select
 12422    1 3   0        80   fffffe8220e1b780                 sh wait
 12129    1 3   0        80   fffffe81d06814e0                ssh select
 13232    1 3   0        80   fffffe8204535740                 sh wait
 13244    1 3   0        80   fffffe82221b3240                ssh select
 13911    2 3   7         0   fffffe821df300e0             expect tstile
 13911    1 3   7         0   fffffe8223394600             expect fdclose
 15831    1 3   7        80   fffffe822178c420                 sh wait
 14617    1 3   7        80   fffffe8222bc08c0               perl pipe_rd
 10185    1 3   7        80   fffffe81d2bdb080                 sh wait
 19345    1 3   0        80   fffffe82278df940                ssh select
 15377    2 3   5        80   fffffe82221b3660             expect select
 15377    1 3   0        80   fffffe8222bc0080             expect parked
 15019    1 3   3        80   fffffe82235045e0                 sh wait
 12888    1 3   5        80   fffffe82278df520               perl pipe_rd
 14017    1 3   3        80   fffffe81d2bdb4a0                 sh wait
 13129    1 3   0        80   fffffe822366d4e0                ssh select
 15179    1 3   6        80   fffffe81e9aad020                 sh wait
 10490    1 3   0        80   fffffe821d9fa600                ssh select
 11328    1 3   6        80   fffffe822dba2a60                 sh wait
 13321    1 3   0        80   fffffe81d9ca9680                ssh select
 13102    1 3   6        80   fffffe81f7d54a00                 sh wait
 12611    1 3   0        80   fffffe81fb83e300                ssh select
 16274    1 3   0        80   fffffe8225d158e0                 sh wait
 12373    1 3   0        80   fffffe81fecc5240                ssh select
 15573    1 3   6        80   fffffe8223a0b480                 sh wait
 11298    1 3   0        80   fffffe81fa035a20                ssh select
 16055    1 3   5        80   fffffe8221e59040                 sh wait
 15194    1 3   0        80   fffffe8221ffeb40                ssh select
 15688    1 3   5        80   fffffe810f2bab20                 sh wait
 11884    1 3   0        80   fffffe8225a112c0                ssh select
 13930    1 3   6        80   fffffe8223efd5c0                 sh wait
 12554    1 3   0        80   fffffe8210253960                ssh select
 13250    1 3   3        80   fffffe822c6922a0                 sh wait
 9817     1 3   0        80   fffffe8225a11b00                ssh select
 16604    1 3   4        80   fffffe822143e460                 sh wait
 12461    1 3   0        80   fffffe81f2f55a40                ssh select
 16683    1 3   0        80   fffffe8223209500                 sh wait
 10467    1 3   0        80   fffffe81fa0351e0                ssh select
 13535    1 3   4        80   fffffe821e2fdb40                 sh wait
 11893    1 3   0        80   fffffe81ef0c0ac0                ssh select
 13544    1 3   0        80   fffffe822455d4e0                 sh wait
 10768    1 3   0        80   fffffe822984c600                ssh select
 11064    1 3   5        80   fffffe822366d900                 sh wait
 13043    1 3   0        80   fffffe8226254080                ssh select
 13228    1 3   3        80   fffffe8222345b20                 sh wait
 11709    1 3   0        80   fffffe822cfa6360                ssh select
 13659    1 3   5        80   fffffe822cc325a0                 sh wait
 13760    1 3   0        80   fffffe82268cf980                ssh select
 13053    1 3   4        80   fffffe82267ce4c0                 sh wait
 17050    1 3   5        80   fffffe821fcf51c0                ssh select
 12595    1 3   0        80   fffffe8223373960                ssh select
 9355     1 3   4        80   fffffe822cdda320                 sh wait
 12295    1 3   1        80   fffffe8222866280                 sh wait
 11395    1 3   0        80   fffffe822002c9a0                ssh select
 13163    1 3   6        80   fffffe82267ce0a0                 sh wait
 8207     1 3   1        80   fffffe821c608240                par pause
 10163    1 3   0        80   fffffe821fcf55e0                 sh wait
 9328     1 3   0        80   fffffe821d68b280                 sh wait
 8257     1 3   3        80   fffffe822366d0c0                 sh wait
 8479     1 3   3        80   fffffe821d74a220                 sh wait
 10636    1 3   3        80   fffffe8220eaab80               cron pipe_rd
 10908    1 3   7        80   fffffe821fed4180                mtr pipe_rd
 9031     1 3   0        80   fffffe8209c5b140         mtr-packet select
 9787     1 3   7         0   fffffe820c3bb100                mtr tstile
 10625    1 3   7        80   fffffe821cf5a360               bash wait
 10526    1 3   5        80   fffffe82278df100               sshd select
 9327     1 3   0        80   fffffe82214b92e0               sshd select
 7813     1 3   0        80   fffffe81f7ac7b20                ssh select
 9090     1 3   3        80   fffffe821f18a0c0                 sh wait
 8423     1 3   0        80   fffffe82228666a0                ssh select
 11985    1 3   6        80   fffffe8221fbb260                 sh wait
 8121     1 3   0        80   fffffe81fb83eb40                ssh select
 6432     1 3   2        80   fffffe8220244560                 sh wait
 8170     1 3   0        80   fffffe8219548060                ssh select
 8983     1 3   7        80   fffffe81dda0b220                 sh wait
 7376     1 3   0        80   fffffe82221b3a80                ssh select
 7594     1 3   7        80   fffffe8221fbb680                 sh wait
 8867     1 3   0        80   fffffe822d95d580                ssh select
 6776     1 3   7        80   fffffe8223504a00                 sh wait
 8541     1 3   0        80   fffffe8205fa79e0                ssh select
 7823     1 3   7        80   fffffe81e6eee2c0                 sh wait
 10488    1 3   0        80   fffffe822178c840                ssh select
 6625     1 3   7        80   fffffe822cfa6ba0                 sh wait
 13028    1 3   0        80   fffffe8226947b20                ssh select
 13357    1 3   0        80   fffffe822ab49760                 sh wait
 7685     1 3   0        80   fffffe821fcf5a00                ssh select
 8454     1 3   2        80   fffffe81d2c38460                 sh wait
 8547     1 3   0        80   fffffe81fb83e720                ssh select
 7077     1 3   2        80   fffffe821c608a80                 sh wait
 7593     1 3   0        80   fffffe822143e040                ssh select
 9009     1 3   4        80   fffffe81dda0b640                 sh wait
 9463     1 3   0        80   fffffe822cbca040                ssh select
 8402     1 3   4        80   fffffe822c3905c0                 sh wait
 8739     1 3   0        80   fffffe8222345700                ssh select
 7028     1 3   4        80   fffffe821e07baa0                 sh wait
 8548     1 3   0        80   fffffe8220ec9b00                ssh select
 8630     1 3   7        80   fffffe82214b9700                 sh wait
 8138     1 3   0        80   fffffe822617c9a0                ssh select
 8911     1 3   7        80   fffffe821f18a4e0                 sh wait
 6100     1 3   0        80   fffffe81d81a3b80                ssh select
 8531     1 3   7        80   fffffe8218492440                 sh wait
 9062     1 3   0        80   fffffe82260f6300                ssh select
 6442     1 3   1        80   fffffe8223a0b060                 sh wait
 7332     1 3   0        80   fffffe8210253540                ssh select
 6557     1 3   1        80   fffffe8222c04500                 sh wait
 9950     1 3   0        80   fffffe822c692ae0                ssh select
 11071    1 3   5        80   fffffe81f2f55620                 sh wait
 8468     1 3   0        80   fffffe821c34b740                ssh select
 9075     1 3   7        80   fffffe821a5a64c0                 sh wait
 5832     1 3   0        80   fffffe8220553480                ssh select
 9604     1 3   7        80   fffffe822dd1a100                 sh wait
 8606     1 3   0        80   fffffe81d82e86c0                ssh select
 6015     1 3   2        80   fffffe820d9cc180                 sh wait
 4455     1 3   0        80   fffffe81d4261780                ssh select
 7799     1 3   6        80   fffffe822b379420                 sh wait
 6867     1 3   0        80   fffffe81fecc5a80                ssh select
 4749     1 3   6        80   fffffe8220244140                 sh wait
 5686     1 3   0        80   fffffe8221e59880                ssh select
 9353     1 3   6        80   fffffe8221604020                 sh wait
 5470     1 3   0        80   fffffe82297c7a60                ssh select
 6083     1 3   2        80   fffffe8220eaa340                 sh wait
 5782     1 3   0        80   fffffe8227899aa0                ssh select
 7547     1 3   3        80   fffffe821fcbe5c0                 sh wait
 5968     1 3   0        80   fffffe8220655120                ssh select
 8516     1 3   3        80   fffffe821fed49c0                 sh wait
 7154     1 3   0        80   fffffe822086d520                ssh select
 6185     1 3   6        80   fffffe821c34bb60                 sh wait
 10365    1 3   0        80   fffffe822617c160                ssh select
 4986     1 3   6        80   fffffe8223373540                 sh wait
 6230     1 3   0        80   fffffe820f1fd9a0                ssh select
 4380     1 3   4        80   fffffe822da7a560                 sh wait
 10536    1 3   0        80   fffffe81f2f55200                ssh select
 7189     1 3   6        80   fffffe8204535320                 sh wait
 9292     1 3   0        80   fffffe8222bc04a0                ssh select
 4786     1 3   4        80   fffffe822086d100                 sh wait
 6470     1 3   0        80   fffffe8209c5b560                ssh select
 6162     1 3   6        80   fffffe8204535b60                 sh wait
 6032     1 3   0        80   fffffe8227899260                ssh select
 6221     1 3   6        80   fffffe822455d0c0                 sh wait
 7423     1 3   0        80   fffffe821d68bac0                ssh select
 5297     1 3   4        80   fffffe821e8b46c0                 sh wait
 6168     1 3   0        80   fffffe821c34b320                ssh select
 8225     1 3   6        80   fffffe8220eaa760                 sh wait
 6670     1 3   0        80   fffffe8229b765e0                ssh select
 6562     1 3   4        80   fffffe821f700340                 sh wait
 4001     1 3   0        80   fffffe822daed540                ssh select
 7269     1 3   6        80   fffffe82205538a0                 sh wait
 5690     1 3   0        80   fffffe822c3909e0                ssh select
 6859     1 3   1        80   fffffe82249b2740                 sh wait
 6538     1 3   0        80   fffffe8219548480                ssh select
 7428     1 3   6        80   fffffe822617c580                 sh wait
 5384     1 3   0        80   fffffe8223dd0420                ssh select
 5593     1 3   0        80   fffffe822daed120                 sh wait
 4848     1 3   0        80   fffffe81d82e82a0                ssh select
 5676     1 3   2        80   fffffe82231fc620                 sh wait
 7822     1 3   0        80   fffffe81e6eeeb00                ssh select
 10050    1 3   4        80   fffffe81d2c38880                 sh wait
 5573     1 3   0        80   fffffe8223373120                ssh select
 7342     1 3   4        80   fffffe821e8b4ae0                 sh wait
 4518     1 3   0        80   fffffe8223efd9e0                ssh select
 4760     1 3   0        80   fffffe81d9ca9aa0                 sh wait
 2041     1 3   0        80   fffffe820c3bb940               tcsh ttyraw
 2022     1 3   5        80   fffffe82262544a0               tcsh pause
 1821     1 3   6        80   fffffe8223394a20               sshd select
 2038     1 3   0        80   fffffe821d9faa20               sshd select
 1081     1 3   5        80   fffffe822002c580          conserver select
 306      1 3   3         0   fffffe821fcbe9e0          conserver tstile
 1431     1 3   6         0   fffffe82267ce8e0          conserver tstile
 790      1 3   5        80   fffffe82210a4080          conserver select
 1173     1 3   0        80   fffffe822143e880                ssh select
 1172     1 3   7        80   fffffe8226947700                 sh wait
 1171     1 3   5         0   fffffe82210a44a0          conserver uvn_fp2
 267      1 3   6        80   fffffe8220553060          conserver select
 361      1 3   4         0   fffffe822247eb00          conserver tstile
 136      1 3   5         0   fffffe8224f80180          conserver tstile
 168      1 3   3         0   fffffe822cbca460          conserver tstile
 551      1 3   5        80   fffffe82297c7640          conserver select
 1027     1 3   0        80   fffffe8228d9ea80              getty ttyraw
 828      1 3   0        80   fffffe8229796a40          conserver select
 1022     1 3   5        80   fffffe822cbca880          conserver select
 1019     1 3   3        80   fffffe82249b2b60               cron nanoslp
 889      1 3   7         0   fffffe8226aec2a0          python2.7 biowait
 751      1 3   5        80   fffffe822a4fa020              inetd kqueue
 806      1 3   0        80   fffffe822a4fa440              nginx kqueue
 837      1 3   0        80   fffffe822a4fa860              nginx pause
 805      1 3   6        80   fffffe82269472e0               qmgr kqueue
 1006     1 3   5        80   fffffe82260f6720             master kqueue
 696      1 3   7        80   fffffe82275b06a0               sshd select
 710      1 3   7        80   fffffe8229b761c0           tac_plus select
 474      1 3   6        80   fffffe8227899680             powerd kqueue
 693      1 3   0        80   fffffe8228d9e660               ntpd pause
 607      4 3   3        80   fffffe8228d9e240              dhcpd parked
 607      3 3   6        80   fffffe822d95d160              dhcpd kqueue
 607      2 3   3         0   fffffe822984ca20              dhcpd biowait
 607      1 3   7        80   fffffe8229796620              dhcpd parked
 391  :   1 7   1         0   fffffe8229796200            syslogd
 1        1 3   7        80   fffffe810f599900               init wait
 0      123 3   5       200   fffffe822cc32180            npfgc-0 xchicv
 0      122 3   7       200   fffffe822cc329c0              ipmi0 ipmi0
 0      121 3   0       200   fffffe810f2e0b60            physiod physiod
 0      120 3   0       200   fffffe822dd1a520           aiodoned aiodoned
 0      119 3   7       200   fffffe822dd1a940            ioflush biolock
 0      118 3   0       200   fffffe822dd800e0           pgdaemon pgdaemon
 0      115 3   5       280   fffffe822dd80500              spkr1 bellcv
 0      114 3   0       280   fffffe822dd80920           audiomix play
 0      113 3   0       280   fffffe810ed322a0           audiorec record
 0      112 3   7       200   fffffe810f5990c0          atapibus0 sccomp
 0      110 3   7       200   fffffe810f2c9300               usb4 usbevt
 0      109 3   0       200   fffffe810f2c9720               usb2 usbevt
 0      108 3   0       200   fffffe810f2c9b40               usb3 usbevt
 0      107 3   0       200   fffffe810f2ba700               usb0 usbevt
 0      106 3   7       200   fffffe810f2ba2e0               usb5 usbevt
 0      105 3   7       200   fffffe810f5994e0               usb1 usbevt
 0      104 3   7       200   fffffe810f3a20a0            rt_free rt_free
 0      103 3   7       200   fffffe810f3a24c0              unpgc unpgc
 0      102 3   0       200   fffffe810f3a28e0    key_timehandler key_timehandler

 0      101 3   7       200   fffffe810f3ab080    icmp6_wqinput/7 icmp6_wqinput
 0      100 3   6       200   fffffe810f3ab4a0    icmp6_wqinput/6 icmp6_wqinput
 0       99 3   5       200   fffffe810f3ab8c0    icmp6_wqinput/5 icmp6_wqinput
 0       98 3   4       200   fffffe810f3c4060    icmp6_wqinput/4 icmp6_wqinput
 0       97 3   3       200   fffffe810f3c4480    icmp6_wqinput/3 icmp6_wqinput
 0       96 3   2       200   fffffe810f3c48a0    icmp6_wqinput/2 icmp6_wqinput
 0       95 3   1       200   fffffe810f395040    icmp6_wqinput/1 icmp6_wqinput
 0       94 3   0       200   fffffe810f395460    icmp6_wqinput/0 icmp6_wqinput
 0    :  93 7   5       200   fffffe810f395880          nd6_timer
 0       92 3   7       200   fffffe810f34e020     icmp_wqinput/7 icmp_wqinput
 0       91 3   6       200   fffffe810f34e440     icmp_wqinput/6 icmp_wqinput
 0       90 3   5       200   fffffe810f34e860     icmp_wqinput/5 icmp_wqinput
 0       89 3   4       200   fffffe810f337000     icmp_wqinput/4 icmp_wqinput
 0       88 3   3       200   fffffe810f337420     icmp_wqinput/3 icmp_wqinput
 0       87 3   2       200   fffffe810f337840     icmp_wqinput/2 icmp_wqinput
 0       86 3   1       200   fffffe810f345360     icmp_wqinput/1 icmp_wqinput
 0       85 3   0       200   fffffe810f345780     icmp_wqinput/0 icmp_wqinput
 0       84 3   0       200   fffffe810f345ba0           rt_timer rt_timer
 0       83 3   5       200   fffffe810f1fc2c0        vmem_rehash vmem_rehash
 0       82 3   3       200   fffffe810f2d7340          coretemp3 coretemp3
 0       81 3   4       200   fffffe810f2d7760          coretemp2 coretemp2
 0       80 3   5       200   fffffe810f2d7b80          coretemp1 coretemp1
 0       79 3   2       200   fffffe810f2e0320          coretemp0 coretemp0
 0       78 3   6       200   fffffe810f2e0740               mfi0 mfi0
 0       69 3   5       200   fffffe810f1fc6e0            atabus1 atath
 0       68 3   0       200   fffffe810f1fcb00            atabus0 atath
 0       66 3   0       200   fffffe810ed326c0           scsibus0 sccomp
 0       65 3   1       200   fffffe810ed32ae0         usbtask-dr usbtsk
 0       64 3   0       200   fffffe810eca9280         usbtask-hc usbtsk
 0       63 3   7       200   fffffe810eca96a0               bnx3 bnx3
 0       62 3   5       200   fffffe810eca9ac0               bnx2 bnx2
 0       61 3   6       200   fffffe810eb60260               bnx1 bnx1
 0       60 3   1       200   fffffe810eb60680               bnx0 bnx0
 0       59 3   0       200   fffffe810eb60aa0               ipmi ipmipoll
 0       58 3   7       200   fffffe810ea73240            xcall/7 xcall
 0       57 1   7       200   fffffe810ea73660          softser/7
 0       56 3   7       200   fffffe810ea73a80          softclk/7 tstile
 0       55 1   7       200   fffffe810ea54220          softbio/7
 0       54 1   7       200   fffffe810ea54640          softnet/7
 0    :  53 7   7       201   fffffe810ea54a60             idle/7
 0       52 3   6       200   fffffe810ea3d200            xcall/6 xcall
 0       51 1   6       200   fffffe810ea3d620          softser/6
 0       50 1   6       200   fffffe810ea3da40          softclk/6
 0       49 1   6       200   fffffe810ea161e0          softbio/6
 0       48 1   6       200   fffffe810ea16600          softnet/6
 0    :  47 7   6       201   fffffe810ea16a20             idle/6
 0       46 3   5       200   fffffe810e9d71c0            xcall/5 xcall
 0       45 1   5       200   fffffe810e9d75e0          softser/5
 0       44 1   5       200   fffffe810e9d7a00          softclk/5
 0       43 1   5       200   fffffe810e9d01a0          softbio/5
 0       42 1   5       200   fffffe810e9d05c0          softnet/5
 0       41 1   5       201   fffffe810e9d09e0             idle/5
 0       40 3   4       200   fffffe810e9a1180            xcall/4 xcall
 0       39 1   4       200   fffffe810e9a15a0          softser/4
 0       38 1   4       200   fffffe810e9a19c0          softclk/4
 0       37 1   4       200   fffffe810e97a160          softbio/4
 0       36 1   4       200   fffffe810e97a580          softnet/4
 0    :  35 7   4       201   fffffe810e97a9a0             idle/4
 0       34 3   3       200   fffffe810e943140            xcall/3 xcall
 0       33 1   3       200   fffffe810e943560          softser/3
 0       32 1   3       200   fffffe810e943980          softclk/3
 0       31 1   3       200   fffffe810e934120          softbio/3
 0       30 1   3       200   fffffe810e934540          softnet/3
 0    :  29 7   3       201   fffffe810e934960             idle/3
 0       28 3   2       200   fffffe810e90d100            xcall/2 xcall
 0       27 1   2       200   fffffe810e90d520          softser/2
 0       26 1   2       200   fffffe810e90d940          softclk/2
 0       25 1   2       200   fffffe810e8f60e0          softbio/2
 0       24 1   2       200   fffffe810e8f6500          softnet/2
 0    :  23 7   2       201   fffffe810e8f6920             idle/2
 0       22 3   1       200   fffffe810e8c50c0            xcall/1 xcall
 0       21 1   1       200   fffffe810e8c54e0          softser/1
 0       20 1   1       200   fffffe810e8c5900          softclk/1
 0       19 1   1       200   fffffe810e8a90a0          softbio/1
 0       18 1   1       200   fffffe810e8a94c0          softnet/1
 0       17 1   1       201   fffffe810e8a98e0             idle/1
 0       16 3   1       200   fffffe822de9e080           lnxsyswq lnxsyswq
 0       15 3   0       200   fffffe822de9e4a0             sysmon smtaskq
 0       14 3   4       200   fffffe822de9e8c0         pmfsuspend pmfsuspend
 0       13 3   7       200   fffffe822e2c5060           pmfevent pmfevent
 0       12 3   0       200   fffffe822e2c5480         sopendfree sopendfr
 0       11 3   2       200   fffffe822e2c58a0           nfssilly nfssilly
 0       10 3   0       200   fffffe822f6dc040            cachegc cachegc
 0        9 3   6       200   fffffe822f6dc460             vdrain vdrain
 0        8 3   6       200   fffffe822f6dc880          modunload mod_unld
 0        7 3   0       200   fffffe822f6fd020            xcall/0 xcall
 0    :   6 7   0       200   fffffe822f6fd440          softser/0
 0        5 1   0       200   fffffe822f6fd860          softclk/0
 0        4 1   0       200   fffffe822f719000          softbio/0
 0        3 1   0       200   fffffe822f719420          softnet/0
 0    :   2 7   0       201   fffffe822f719840             idle/0
 0        1 3   4       200   ffffffff81481b20            swapper uvm
 db{0}: trace/t 0t5640
 trace: pid 5640 lid 1 at 0xffff80009f9cf8f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t13063
 trace: pid 13063 lid 1 at 0xffff80009eb148f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t7684
 trace: pid 7684 lid 1 at 0xffff80009f9a88f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t770
 trace: pid 770 lid 1 at 0xffff80009ec898f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t769
 trace: pid 769 lid 1 at 0xffff80009f9968f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t768
 trace: pid 768 lid 1 at 0xffff80009f1098f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t13308
 trace: pid 13308 lid 1 at 0xffff80009edae8f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t14541
 trace: pid 14541 lid 1 at 0xffff80009ee768f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t15140
 trace: pid 15140 lid 1 at 0xffff80009f9688f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t11390
 trace: pid 11390 lid 1 at 0xffff80009ed2e8f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t14640
 trace: pid 14640 lid 1 at 0xffff80009ed248f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t13774
 trace: pid 13774 lid 1 at 0xffff80009de6d8f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t18260
 trace: pid 18260 lid 1 at 0xffff80009f6bd8f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t16295
 trace: pid 16295 lid 1 at 0xffff80009e6568f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t18585
 trace: pid 18585 lid 1 at 0xffff80009f66a8f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t15183
 trace: pid 15183 lid 1 at 0xffff80009ee448f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t17051
 trace: pid 17051 lid 1 at 0xffff80009ee048f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t12406
 trace: pid 12406 lid 1 at 0xffff80009e6b97a0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 check_exec() at netbsd:check_exec+0x4f
 execve_loadvm() at netbsd:execve_loadvm+0x247
 execve1() at netbsd:execve1+0x28
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 59) ---
 71a7b203e4aa:
 db{0}: trace/t 0t14315
 trace: pid 14315 lid 1 at 0xffff80009ea7d8f0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x460
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7344c183e2ca:
 db{0}: trace/t 0t13911
 trace: pid 13911 lid 2 at 0xffff80009ecc4ab0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x74d
 VOP_POLL() at netbsd:VOP_POLL+0x37
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 417) ---
 713dd9e3e10a:
 db{0}: trace/t 0t9787
 trace: pid 9787 lid 1 at 0xffff80009ef7cbe0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x74d
 VOP_LOCK() at netbsd:VOP_LOCK+0x4d
 vn_lock() at netbsd:vn_lock+0x90
 vn_write() at netbsd:vn_write+0x9b
 dofilewrite() at netbsd:dofilewrite+0x97
 sys_write() at netbsd:sys_write+0x5f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 4) ---
 6fa9c5e3e02a:
 db{0}: trace/t 0t306
 trace: pid 306 lid 1 at 0xffff80009ed38ab0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x74d
 VOP_POLL() at netbsd:VOP_POLL+0x37
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 417) ---
 7b059de3e10a:
 db{0}: trace/t 0t1431
 trace: pid 1431 lid 1 at 0xffff80009ec29ab0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x74d
 VOP_POLL() at netbsd:VOP_POLL+0x37
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 417) ---
 7b059de3e10a:
 db{0}: trace/t 0t1171
 trace: pid 1171 lid 1 at 0xffff80009ebce8c0
 sleepq_block() at netbsd:sleepq_block+0x97
 mtsleep() at netbsd:mtsleep+0x102
 uvn_findpage() at netbsd:uvn_findpage+0xaf
 uvn_findpages() at netbsd:uvn_findpages+0x78
 genfs_getpages() at netbsd:genfs_getpages+0x5f9
 VOP_GETPAGES() at netbsd:VOP_GETPAGES+0x4f
 ufs_balloc_range() at netbsd:ufs_balloc_range+0xf8
 ffs_write() at netbsd:ffs_write+0x4d7
 VOP_WRITE() at netbsd:VOP_WRITE+0x37
 vn_write() at netbsd:vn_write+0xec
 dofilewrite() at netbsd:dofilewrite+0x97
 sys_write() at netbsd:sys_write+0x5f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 4) ---
 7b059de3e02a:
 db{0}: trace/t 0t361
 trace: pid 361 lid 1 at 0xffff80009eae9ab0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x74d
 VOP_POLL() at netbsd:VOP_POLL+0x37
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 417) ---
 7b059de3e10a:
 db{0}: trace/t 0t136
 trace: pid 136 lid 1 at 0xffff80009e9f3ab0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x74d
 VOP_POLL() at netbsd:VOP_POLL+0x37
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 417) ---
 7b059de3e10a:
 db{0}: trace/t 0t168
 trace: pid 168 lid 1 at 0xffff80009e86dab0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 fstrans_start() at netbsd:fstrans_start+0x74d
 VOP_POLL() at netbsd:VOP_POLL+0x37
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 417) ---
 7b059de3e10a:
 db{0}: trace/t 0t607
 trace: pid 607 lid 4 at 0xffff80009e0d9e10
 sleepq_block() at netbsd:sleepq_block+0x106
 lwp_park() at netbsd:lwp_park+0x107
 sys____lwp_park60() at netbsd:sys____lwp_park60+0x52
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 478) ---
 74d67e09998a:
 db{0}: trace fffffe810f2e0740
 fffffe810ea0cd18:
 db{0}: trace/a fffffe810f2e0740
 trace: pid 0 lid 78 at 0xffff80009db7ce60
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 workqueue_worker() at netbsd:workqueue_worker+0x8f
 db{0}: trace/t 0t889
 trace: pid 889 lid 1 at 0xffff80009e6788a0
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 ffs_update.part.3() at netbsd:ffs_update.part.3+0x292
 ffs_truncate() at netbsd:ffs_truncate+0x1046
 ufs_truncate_retry() at netbsd:ufs_truncate_retry+0x79
 ufs_setattr() at netbsd:ufs_setattr+0x44c
 VOP_SETATTR() at netbsd:VOP_SETATTR+0x34
 vn_open() at netbsd:vn_open+0x26e
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7444cbe3e2ca:
 db{0}: trace/a fffffe810ea73a80
 trace: pid 0 lid 56 at 0xffff80008f7e2e20
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 mutex_vector_enter() at netbsd:mutex_vector_enter+0x29e
 ip_slowtimo() at netbsd:ip_slowtimo+0x10
 pfslowtimo() at netbsd:pfslowtimo+0x31
 callout_softclock() at netbsd:callout_softclock+0x201
 softint_dispatch() at netbsd:softint_dispatch+0xd3
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f7e2ff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}: 
 db{0}: call cpu_reset

From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Thu, 05 Apr 2018 09:20:14 +0200 (CEST)

 Another morning, another wedge:

 fatal breakpoint trap in supervisor mode
 trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0x7641663b8000 ilevel 0x8 rsp 0xffff80008f69bcf8
 curlwp 0xfffffe822f6eb440 pid 0.6 lowest kstack 0xffff80008f6972c0
 Stopped in pid 0.6 (system) at  netbsd:breakpoint+0x5:  leave
 db{0}: trace
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 DDB lost frame for netbsd:Xdoreti, trying 0xffff80008f69bd70
 Xdoreti() at netbsd:Xdoreti
 --- interrupt ---
 Bad frame pointer: 0xffff800000000000
 202:
 db{0}: machine cpu 1
 using CPU 1
 db{0}: tra
 x86_pause() at netbsd:x86_pause+0x2
 cdev_open() at netbsd:cdev_open+0xaf
 spec_open() at netbsd:spec_open+0x260
 VOP_OPEN() at netbsd:VOP_OPEN+0x33
 vn_open() at netbsd:vn_open+0x28b
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 5) ---
 799d4323e2ca:
 db{0}: machine cpu 2
 using CPU 2
 db{0}: tra
 x86_pause() at netbsd:x86_pause+0x2
 nd6_timer_work() at netbsd:nd6_timer_work+0x49
 workqueue_worker() at netbsd:workqueue_worker+0xcd
 db{0}: machine cpu 3
 using CPU 3
 db{0}: tra
 x86_pause() at netbsd:x86_pause
 _kernel_lock() at netbsd:_kernel_lock+0x183
 kevent1() at netbsd:kevent1+0x648
 sys___kevent50() at netbsd:sys___kevent50+0x33
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 435) ---
 79a32843e40a:
 db{0}: machine cpu 4
 using CPU 4
 db{0}: tra
 _kernel_lock() at netbsd:_kernel_lock+0x1a1
 kqueue_register() at netbsd:kqueue_register+0x431
 kevent1() at netbsd:kevent1+0x150
 sys___kevent50() at netbsd:sys___kevent50+0x33
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 435) ---
 7aec56c3e40a:
 db{0}: machine cpu 5
 using CPU 5
 db{0}: tra
 x86_pause() at netbsd:x86_pause+0x2
 sleepq_block() at netbsd:sleepq_block+0x1c5
 cv_timedwait() at netbsd:cv_timedwait+0x131
 ipmi_thread() at netbsd:ipmi_thread+0x2f4
 db{0}: machine cpu 6
 using CPU 6
 db{0}: tra
 x86_pause() at netbsd:x86_pause
 _kernel_lock() at netbsd:_kernel_lock+0x183
 VOP_POLL() at netbsd:VOP_POLL+0x7a
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 776367e3e10a:
 db{0}: machine cpu 7
 using CPU 7
 db{0}: tra
 x86_pause() at netbsd:x86_pause
 _kernel_lock() at netbsd:_kernel_lock+0x183
 VFS_ROOT() at netbsd:VFS_ROOT+0x49
 lookup_once() at netbsd:lookup_once+0x36d
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x413
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 5) ---
 74ec32c3e2ca:
 db{0}: show uvmexp
 Current UVM status:
   pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12, ncolors=8
   2028615 VM pages: 413947 active, 1908 inactive, 1814 wired, 1282873 free
   pages  43514 anon, 367957 file, 6198 exec
   freemin=2048, free-target=2730, wired-max=676205
   resv-pg=1, resv-kernel=40, zeropages=925073
   cpu0:
     faults=3009472, traps=3008330, intrs=6915999, ctxswitch=2672825
     softint=8300569, syscalls=14016898
   cpu1:
     faults=1117777, traps=1117078, intrs=0, ctxswitch=1020296
     softint=340527, syscalls=5307734
   cpu2:
     faults=1518940, traps=1517481, intrs=0, ctxswitch=944964
     softint=347368, syscalls=4438330
   cpu3:
     faults=975558, traps=974861, intrs=0, ctxswitch=894789
     softint=135439, syscalls=4914018
   cpu4:
     faults=1339014, traps=1337920, intrs=0, ctxswitch=989824
     softint=301182, syscalls=5570074
   cpu5:
     faults=2229358, traps=2229534, intrs=0, ctxswitch=2649435
     softint=222674, syscalls=12788873
   cpu6:
     faults=1292311, traps=1291923, intrs=0, ctxswitch=1382918
     softint=408535, syscalls=6133777
   cpu7:
     faults=1189005, traps=1189014, intrs=0, ctxswitch=1574424
     softint=308728, syscalls=6615055
   fault counts:
     noram=0, noanon=0, pgwait=0, pgrele=0
     ok relocks(total)=12242(12242), anget(retrys)=3183917(0), amapcopy=992662
     neighbor anon/obj pg=1091862/15499112, gets(lock/unlock)=5789978/12242
     cases: anon=2567450, anoncow=616481, obj=4466484, prcopy=1323525, przero=310
 4330
   daemon and swap counts:
     woke=0, revs=0, scans=0, obscans=0, anscans=0
     busy=0, freed=0, reactivate=0, deactivate=0
     pageouts=0, pending=0, nswget=0
     nswapdev=1, swpgavail=32767
     swpages=32767, swpginuse=0, swpgonly=0, paging=0
 db{0}: ps
 PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 6864 :   1 7   1         0   fffffe821e9b4a20             expect
 5145     1 3   1        80   fffffe821e6ae220                 sh wait
 23761:   1 7   7         0   fffffe8220dabac0                 sh
 11818    1 3   7         0   fffffe820e438340                 sh wait
 23101    1 3   7        80   fffffe8224f626e0             expect wait
 18472    1 3   7        80   fffffe822be434a0             expect pipe_rd
 21360    1 3   1        80   fffffe821654d260               perl pipe_rd
 9311     1 3   0        80   fffffe8220696420                 sh wait
 27159    1 3   5        80   fffffe821b40a540                 sh wait
 21823    1 3   0        80   fffffe821ef7d1a0               perl pipe_rd
 12285    1 3   5        80   fffffe821f2c3060                 sh wait
 21866    1 3   4        80   fffffe820d167520                ssh select
 3553     2 3   4        80   fffffe821b54a680             expect select
 3553     1 2   0         0   fffffe822dce5540             expect
 22723    1 3   5        80   fffffe8205c49600                 sh wait
 1399     1 3   0        80   fffffe82197b9660               perl pipe_rd
 16437    1 3   5        80   fffffe8220ff2200                 sh wait
 20095    1 3   5        80   fffffe821cfbe180                ssh select
 24470    2 3   5        80   fffffe821eb6b900             expect select
 24470    1 2   0         0   fffffe822be438c0             expect
 26012    1 3   5        80   fffffe821cfbe9c0                 sh wait
 21306    1 3   3        80   fffffe821ce608c0               perl pipe_rd
 637      1 3   5        80   fffffe8222a46140                 sh wait
 11788:   1 7   0         0   fffffe8210b32640                ssh
 13950:   2 7   6         0   fffffe821cf46780             expect
 13950    1 2   7         0   fffffe8221bfc660             expect
 29647    1 3   1        80   fffffe8227944720                 sh wait
 5400     1 3   3        80   fffffe820d9fa020               perl pipe_rd
 9552     1 3   5        80   fffffe820e2f4580                 sh wait
 28864    1 3   5        80   fffffe8221d871e0                par pause
 15339    1 3   6        80   fffffe820a9eb9c0                 sh wait
 11851    1 3   2        80   fffffe820a3ca060                 sh wait
 26008    1 3   2        80   fffffe820e8d7680                 sh wait
 12131    1 3   2        80   fffffe821cfd7100                 sh wait
 24961    1 3   2        80   fffffe821cfbe5a0               cron pipe_rd
 13841    1 3   4        80   fffffe8206c524c0               bash select
 12019    1 3   0        80   fffffe8222d370a0               sshd select
 9136     1 3   5        80   fffffe8210b32a60               sshd select
 6190     1 3   4        80   fffffe822a6372a0               bash select
 22250    1 3   5        80   fffffe821f50a1c0               sshd select
 14356    1 3   5        80   fffffe82214a21c0               sshd select
 18494    1 3   5        80   fffffe820d717540                ssh select
 13452    1 3   4        80   fffffe821e843940                ssh select
 5613     1 3   4        80   fffffe82197b9a80               bash wait
 4436     1 3   5        80   fffffe82229e34a0               sshd select
 12189    1 3   5        80   fffffe8220ff2a40               sshd select
 16044    1 3   7        80   fffffe820b365660               bash wait
 16554    1 3   3        80   fffffe81ecddf2a0               sshd select
 9867     1 3   1        80   fffffe821eb6b0c0               sshd select
 13669    1 3   2        80   fffffe8229b70260             pickup kqueue
 27835    1 3   5        80   fffffe821eb6b4e0                ssh select
 23377    1 3   5        80   fffffe821ce60080                 sh wait
 14548    1 3   0        80   fffffe822042c860                ssh select
 12025    1 3   6        80   fffffe820d717960                 sh wait
 13955    1 3   2        80   fffffe821a2e3600                ssh select
 10145    1 3   5        80   fffffe8207ee9080                ssh select
 10991    1 3   6        80   fffffe82246f4420                 sh wait
 10919    1 3   5        80   fffffe8220ec90c0                 sh wait
 8354     1 3   5        80   fffffe821b54aaa0                ssh select
 12457    1 3   0        80   fffffe822d0b3a40                 sh wait
 12262    1 3   4        80   fffffe821b67b2e0                ssh select
 8647     1 3   1        80   fffffe821cc3a020                 sh wait
 12615    1 3   6        80   fffffe82209b1a60                ssh select
 6504     1 3   6        80   fffffe821e835200                 sh wait
 10077    1 3   5        80   fffffe821f50a5e0                ssh select
 9097     1 3   7        80   fffffe8210032000                 sh wait
 12776    1 3   3        80   fffffe821e835a40                ssh select
 8975     1 3   0        80   fffffe821d71a160                 sh wait
 15272    1 3   4        80   fffffe820a3ca8a0                ssh select
 9186     1 3   1        80   fffffe821ce604a0                 sh wait
 11697    1 3   0        80   fffffe821b67bb20                ssh select
 11201    1 3   6        80   fffffe8227f4c2e0                 sh wait
 11492    1 3   1        80   fffffe82205f59e0                ssh select
 7480     1 3   1        80   fffffe8227944b40                 sh wait
 10626    1 3   0        80   fffffe8213be7580                ssh select
 9457     1 3   6        80   fffffe822776db00                 sh wait
 10478    1 3   1        80   fffffe820d167100                ssh select
 11749    1 3   1        80   fffffe821f762500                 sh wait
 10715    1 3   0        80   fffffe822373c020                ssh select
 11482    1 3   0        80   fffffe8224f622c0                 sh wait
 10165    1 3   0        80   fffffe820675a0c0                ssh select
 8333     1 3   1        80   fffffe8210032840                 sh wait
 13105    1 3   0        80   fffffe8206c528e0                ssh select
 10194    1 3   4        80   fffffe821ec3f920                 sh wait
 8968     1 3   6        80   fffffe820675a4e0                ssh select
 11039    1 3   5        80   fffffe8207ee98c0                 sh wait
 12997    1 3   6        80   fffffe82214252e0                ssh select
 10428    1 3   6        80   fffffe82228c5b80                 sh wait
 13132    1 3   5        80   fffffe820deaaae0                ssh select
 12249    1 3   7        80   fffffe821654daa0                 sh wait
 12547    1 3   0        80   fffffe8220696840                ssh select
 10612    1 3   1        80   fffffe8223545880                 sh wait
 10685    1 3   0        80   fffffe82164555e0                ssh select
 12205    1 3   1        80   fffffe820bb17a40                 sh wait
 9717     1 3   0        80   fffffe82104fb280                ssh select
 8576     1 3   6        80   fffffe821f5e1340                 sh wait
 9263     1 3   2        80   fffffe82290a06a0                ssh select
 8236     1 3   2        80   fffffe821fc844c0                 sh wait
 7629     1 3   7        80   fffffe821cfd7940                ssh select
 10678    1 3   5        80   fffffe8222b97040                 sh wait
 11663    1 3   0        80   fffffe8222d378e0                ssh select
 9517     1 3   6        80   fffffe8229b70aa0                 sh wait
 8203     1 3   0        80   fffffe8228128ae0                ssh select
 10074    1 3   5        80   fffffe821d71a9a0                 sh wait
 13318    1 3   6        80   fffffe8221d87600                ssh select
 8850     1 3   0        80   fffffe820fedeac0                 sh wait
 13166    1 3   0        80   fffffe822042c020                ssh select
 11230    1 3   0        80   fffffe822d8079c0                 sh wait
 10197    1 3   0        80   fffffe822207a900                ssh select
 9056     1 3   1        80   fffffe821b7545c0                 sh wait
 12554    1 3   7        80   fffffe82130ff360                ssh select
 9510     1 3   0        80   fffffe8220964060                 sh wait
 9456     1 3   7        80   fffffe822d1c45e0                ssh select
 11296    1 3   0        80   fffffe821d71a580                 sh wait
 10743    1 3   0        80   fffffe822a637ae0                ssh select
 13144    1 3   0        80   fffffe8226369ba0                 sh wait
 10605    1 3   4        80   fffffe8213be79a0                ssh select
 11156    1 3   0        80   fffffe822aecd580                 sh wait
 9831     1 3   0        80   fffffe821ef7d9e0                ssh select
 10696    1 3   1        80   fffffe8222a46560                 sh wait
 11143    1 3   2        80   fffffe8210b7aa40                ssh select
 10167    1 3   0        80   fffffe82242529c0                 sh wait
 10843    1 3   3        80   fffffe822d807180                ssh select
 9171     1 3   5        80   fffffe8227456760                 sh wait
 9017     1 3   0        80   fffffe8227456b80                ssh select
 8264     1 3   0        80   fffffe822207a0c0                 sh wait
 7803     1 3   7        80   fffffe82228c5760                ssh select
 9042     1 3   0        80   fffffe82203e5300                 sh wait
 8337     1 3   1        80   fffffe822aeefa80                ssh select
 8918     1 3   1        80   fffffe8221ab5680                 sh wait
 9259     1 3   0        80   fffffe8222d5b480                ssh select
 9894     1 3   1        80   fffffe8228895720                 sh wait
 10382    1 3   5        80   fffffe820deaa2a0                ssh select
 12603    1 3   6        80   fffffe822d9d8980                 sh wait
 10391    1 3   6        80   fffffe8221425700                ssh select
 10053    1 3   0        80   fffffe821e7ea4c0                 sh wait
 8569     1 3   5        80   fffffe820675a900                ssh select
 9513     1 3   5        80   fffffe821e7ea0a0                 sh wait
 7732     1 3   2        80   fffffe821e9b41e0                ssh select
 7288     1 3   5        80   fffffe810f2da300                 sh wait
 7615     1 3   3        80   fffffe8210032420                ssh select
 11516    1 3   4        80   fffffe8221bfc240                 sh wait
 6244     1 3   0        80   fffffe8227944300                ssh select
 8987     1 3   0        80   fffffe8221d87a20                 sh wait
 10342    1 3   3        80   fffffe82203e5b40                ssh select
 9118     1 3   0        80   fffffe820a3ca480                 sh wait
 10063    1 3   7        80   fffffe822bc5da60                ssh select
 14307    1 3   7        80   fffffe8220dcd540                 sh wait
 8607     1 3   4        80   fffffe82146c4140                ssh select
 8875     1 3   4        80   fffffe8220ff2620                 sh wait
 7285     1 3   3        80   fffffe82229e38c0                ssh select
 6689     1 3   4        80   fffffe82067c8920                 sh wait
 11095    1 3   4        80   fffffe821fc840a0                ssh select
 9536     1 3   5        80   fffffe821f762920                 sh wait
 8030     1 3   4        80   fffffe821f50aa00                ssh select
 11495    1 3   5        80   fffffe82130ffba0                 sh wait
 10208    1 3   2        80   fffffe820ed99040                ssh select
 8352     1 3   2        80   fffffe820e8d7260                 sh wait
 8859     1 3   5        80   fffffe8220964480                ssh select
 7714     1 3   2        80   fffffe822207c000                 sh wait
 9647     1 3   0        80   fffffe8207ee94a0                ssh select
 10611    1 3   0        80   fffffe8227b8d560                 sh wait
 7484     1 3   7        80   fffffe82228c5340                ssh select
 8137     1 3   7        80   fffffe8227b8d140                 sh wait
 9308     1 3   7        80   fffffe8222b97460                ssh select
 7474     1 3   7        80   fffffe8227456340                 sh wait
 12342    1 3   0        80   fffffe822a2b5120                ssh select
 10235    1 3   0        80   fffffe821e843100                 sh wait
 8851     1 3   7        80   fffffe82201ea2a0                ssh select
 13638    1 3   0        80   fffffe821f2c3480                 sh wait
 8869     1 3   7        80   fffffe820e438760                ssh select
 7630     1 3   7        80   fffffe82281282a0                 sh wait
 8531     1 3   5        80   fffffe8205c49a20                ssh select
 8642     1 3   7        80   fffffe8216455a00                 sh wait
 8310     1 3   5        80   fffffe822d9bd580                ssh select
 11193    1 3   7        80   fffffe822a2b5540                 sh wait
 6579     1 3   0        80   fffffe822d9d8140                ssh select
 12466    1 3   0        80   fffffe8214a1db00                 sh wait
 7694     1 3   0        80   fffffe8222b97880                ssh select
 10187    1 3   0        80   fffffe82146c4980                 sh wait
 9246     1 3   0        80   fffffe8220696000                ssh select
 10052    1 3   0        80   fffffe8221bfca80                 sh wait
 9722     1 3   3        80   fffffe8210b32220                ssh select
 11654    1 3   3        80   fffffe8223ab30e0                 sh wait
 9180     1 3   2        80   fffffe82229e3080                ssh select
 7647     1 3   3        80   fffffe822a6376c0                 sh wait
 10535    1 3   0        80   fffffe821fb6d240                ssh select
 8965     1 3   0        80   fffffe82214a25e0                 sh wait
 10010    1 3   1        80   fffffe821cddf280                ssh select
 8138     1 3   7        80   fffffe8221ab5aa0                 sh wait
 6940     1 3   0        80   fffffe8210b7a620                ssh select
 9251     1 3   0        80   fffffe82104fb6a0                 sh wait
 8556     1 3   7        80   fffffe822776d2c0               tcsh ttyraw
 8477     1 3   0        80   fffffe8222d5b060               sshd select
 7657     1 3   1        80   fffffe822aecd160               sshd select
 322      1 3   3         0   fffffe8220727b60          conserver tstile
 563      1 3   3         0   fffffe8220727740          conserver tstile
 534      1 3   3         0   fffffe821ffd7360          conserver tstile
 531      1 3   7        80   fffffe82201eaae0          conserver select
 400      1 3   3         0   fffffe821ffd7ba0          conserver tstile
 900      1 3   3         0   fffffe8220a7f720          conserver tstile
 361      1 3   3         0   fffffe82209b1640          conserver tstile
 136      1 3   7        80   fffffe8223ab3500          conserver select
 549      1 3   4         0   fffffe822d9d8560          conserver tstile
 516      1 3   3         0   fffffe82246f4840          conserver tstile
 931      1 3   5        80   fffffe822dd780e0              getty ttyraw
 857      1 3   2        80   fffffe8226369360          conserver select
 511      1 3   5        80   fffffe82286af2c0          conserver select
 892      1 3   2        80   fffffe8226369780               cron nanoslp
 890      1 3   2        80   fffffe822d0b3200          python2.7 select
 107      1 3   4        80   fffffe82290a0280              inetd kqueue
 98       1 3   6        80   fffffe822778c320              nginx kqueue
 97       1 3   0        80   fffffe822778c740              nginx pause
 589  :   1 7   3         0   fffffe82286afb00               qmgr
 775  :   1 7   4         0   fffffe82286af6e0             master
 471      1 3   5        80   fffffe8229b70680               sshd select
 460      1 3   3         0   fffffe822aeef240           tac_plus tstile
 430      1 3   7        80   fffffe822d9bd160             powerd kqueue
 420      1 3   6        80   fffffe822d0b3620               ntpd pause
 342      4 3   3        80   fffffe822d7bf1a0              dhcpd parked
 342      3 3   0        80   fffffe810f2dab40              dhcpd kqueue
 342      2 3   6        80   fffffe822d9bd9a0              dhcpd parked
 342      1 3   6        80   fffffe822d8075a0              dhcpd parked
 221      1 2   1         0   fffffe822d1c4a00            syslogd
 1        1 3   3        80   fffffe810f51a900               init wait
 0      123 3   4       200   fffffe822d1c41c0            npfgc-0 xchicv
 0      122 3   7       200   fffffe822d7bf9e0              ipmi0 ipmi0
 0      121 3   7       200   fffffe822dcee940            physiod physiod
 0      120 3   6       200   fffffe822dce5960           aiodoned aiodoned
 0      119 3   0       200   fffffe822dcee100            ioflush syncer
 0      118 3   0       200   fffffe822dcee520           pgdaemon pgdaemon
 0      115 3   0       280   fffffe822dd78500              spkr1 bellcv
 0      114 3   5       280   fffffe822dd78920           audiomix play
 0      113 3   5       280   fffffe810ed3b2a0           audiorec record
 0      112 3   0       200   fffffe810f51a0c0          atapibus0 sccomp
 0      110 3   7       200   fffffe810f20d2c0               usb5 usbevt
 0      109 3   7       200   fffffe810f2cb2e0               usb0 usbevt
 0      108 2   3       200   fffffe810f2cb700               usb4
 0      107 2   3       200   fffffe810f2cbb20               usb1
 0      106 2   3       200   fffffe810f2da720               usb3
 0      105 2   3       200   fffffe810f51a4e0               usb2
 0      104 3   0       200   fffffe810f43b0a0            rt_free rt_free
 0      103 3   0       200   fffffe810f43b4c0              unpgc unpgc
 0      102 2   4       200   fffffe810f43b8e0    key_timehandler
 0      101 3   7       200   fffffe810f45c080    icmp6_wqinput/7 icmp6_wqinput
 0      100 3   6       200   fffffe810f45c4a0    icmp6_wqinput/6 icmp6_wqinput
 0       99 3   5       200   fffffe810f45c8c0    icmp6_wqinput/5 icmp6_wqinput
 0       98 3   4       200   fffffe810f585060    icmp6_wqinput/4 icmp6_wqinput
 0       97 3   3       200   fffffe810f585480    icmp6_wqinput/3 icmp6_wqinput
 0       96 3   2       200   fffffe810f5858a0    icmp6_wqinput/2 icmp6_wqinput
 0       95 3   1       200   fffffe810f38e040    icmp6_wqinput/1 icmp6_wqinput
 0       94 3   0       200   fffffe810f38e460    icmp6_wqinput/0 icmp6_wqinput
 0    :  93 7   2       200   fffffe810f38e880          nd6_timer
 0       92 3   7       200   fffffe810f35f020     icmp_wqinput/7 icmp_wqinput
 0       91 3   6       200   fffffe810f35f440     icmp_wqinput/6 icmp_wqinput
 0       90 3   5       200   fffffe810f35f860     icmp_wqinput/5 icmp_wqinput
 0       89 3   4       200   fffffe810f370000     icmp_wqinput/4 icmp_wqinput
 0       88 3   3       200   fffffe810f370420     icmp_wqinput/3 icmp_wqinput
 0       87 3   2       200   fffffe810f370840     icmp_wqinput/2 icmp_wqinput
 0       86 3   1       200   fffffe810f33e360     icmp_wqinput/1 icmp_wqinput
 0       85 3   0       200   fffffe810f33e780     icmp_wqinput/0 icmp_wqinput
 0       84 2   3       200   fffffe810f33eba0           rt_timer
 0       83 3   7       200   fffffe810f2f1b60        vmem_rehash vmem_rehash
 0       82 3   4       200   fffffe810f2f0340          coretemp3 coretemp3
 0       81 3   6       200   fffffe810f2f0760          coretemp2 coretemp2
 0       80 3   7       200   fffffe810f2f0b80          coretemp1 coretemp1
 0       79 3   1       200   fffffe810f2f1320          coretemp0 coretemp0
 0       78 3   1       200   fffffe810f2f1740               mfi0 mfi0
 0       69 3   5       200   fffffe810f20d6e0            atabus1 atath
 0       68 3   4       200   fffffe810f20db00            atabus0 atath
 0       66 3   0       200   fffffe810ed3b6c0           scsibus0 sccomp
 0       65 3   7       200   fffffe810ed3bae0         usbtask-dr usbtsk
 0       64 3   0       200   fffffe810ecaa280         usbtask-hc usbtsk
 0       63 3   7       200   fffffe810ecaa6a0               bnx3 bnx3
 0       62 3   5       200   fffffe810ecaaac0               bnx2 bnx2
 0       61 3   7       200   fffffe810eb61260               bnx1 bnx1
 0       60 3   7       200   fffffe810eb61680               bnx0 bnx0
 0    :  59 7   5       200   fffffe810eb61aa0               ipmi
 0       58 3   7       200   fffffe810ea7c240            xcall/7 xcall
 0       57 1   7       200   fffffe810ea7c660          softser/7
 0       56 3   7       200   fffffe810ea7ca80          softclk/7 tstile
 0       55 1   7       200   fffffe810ea5d220          softbio/7
 0       54 1   7       200   fffffe810ea5d640          softnet/7
 0       53 1   7       201   fffffe810ea5da60             idle/7
 0       52 3   6       200   fffffe810ea36200            xcall/6 xcall
 0       51 1   6       200   fffffe810ea36620          softser/6
 0       50 1   6       200   fffffe810ea36a40          softclk/6
 0       49 1   6       200   fffffe810ea0f1e0          softbio/6
 0       48 1   6       200   fffffe810ea0f600          softnet/6
 0       47 1   6       201   fffffe810ea0fa20             idle/6
 0       46 3   5       200   fffffe810e9e01c0            xcall/5 xcall
 0       45 1   5       200   fffffe810e9e05e0          softser/5
 0       44 1   5       200   fffffe810e9e0a00          softclk/5
 0       43 1   5       200   fffffe810e9d11a0          softbio/5
 0       42 1   5       200   fffffe810e9d15c0          softnet/5
 0       41 1   5       201   fffffe810e9d19e0             idle/5
 0       40 3   4       200   fffffe810e9a2180            xcall/4 xcall
 0       39 1   4       200   fffffe810e9a25a0          softser/4
 0       38 1   4       200   fffffe810e9a29c0          softclk/4
 0       37 1   4       200   fffffe810e973160          softbio/4
 0       36 1   4       200   fffffe810e973580          softnet/4
 0       35 1   4       201   fffffe810e9739a0             idle/4
 0       34 3   3       200   fffffe810e94c140            xcall/3 xcall
 0       33 1   3       200   fffffe810e94c560          softser/3
 0       32 1   3       200   fffffe810e94c980          softclk/3
 0       31 1   3       200   fffffe810e93d120          softbio/3
 0       30 1   3       200   fffffe810e93d540          softnet/3
 0       29 1   3       201   fffffe810e93d960             idle/3
 0       28 3   2       200   fffffe810e906100            xcall/2 xcall
 0       27 1   2       200   fffffe810e906520          softser/2
 0       26 3   2       200   fffffe810e906940          softclk/2 tstile
 0       25 1   2       200   fffffe810e8ef0e0          softbio/2
 0       24 1   2       200   fffffe810e8ef500          softnet/2
 0       23 1   2       201   fffffe810e8ef920             idle/2
 0       22 3   1       200   fffffe810e8c50c0            xcall/1 xcall
 0       21 1   1       200   fffffe810e8c54e0          softser/1
 0       20 1   1       200   fffffe810e8c5900          softclk/1
 0       19 1   1       200   fffffe810e8b10a0          softbio/1
 0       18 1   1       200   fffffe810e8b14c0          softnet/1
 0       17 1   1       201   fffffe810e8b18e0             idle/1
 0       16 3   0       200   fffffe822de92080           lnxsyswq lnxsyswq
 0       15 3   0       200   fffffe822de924a0             sysmon smtaskq
 0       14 3   2       200   fffffe822de928c0         pmfsuspend pmfsuspend
 0       13 3   1       200   fffffe822e2b1060           pmfevent pmfevent
 0       12 3   0       200   fffffe822e2b1480         sopendfree sopendfr
 0       11 3   4       200   fffffe822e2b18a0           nfssilly nfssilly
 0       10 2   3       200   fffffe822f6d0040            cachegc
 0        9 3   1       200   fffffe822f6d0460             vdrain vdrain
 0        8 3   0       200   fffffe822f6d0880          modunload mod_unld
 0        7 3   0       200   fffffe822f6eb020            xcall/0 xcall
 0    :   6 7   0       200   fffffe822f6eb440          softser/0
 0    :   5 7   0       200   fffffe822f6eb860          softclk/0
 0        4 1   0       200   fffffe822f707000          softbio/0
 0        3 1   0       200   fffffe822f707420          softnet/0
 0        2 1   0       201   fffffe822f707840             idle/0
 0        1 2   3       200   ffffffff81481b20            swapper
 db{0}: trace/t 0t322
 trace: pid 322 lid 1 at 0xffff80009eb99b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7af9a8e3e10a:
 db{0}: trace/t 0t563
 trace: pid 563 lid 1 at 0xffff80009eb9eb00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7af9a8e3e10a:
 db{0}: trace/t 0t534
 trace: pid 534 lid 1 at 0xffff80009ec64b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7af9a8e3e10a:
 db{0}: trace/t 0t400
 trace: pid 400 lid 1 at 0xffff80009ebc9b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7af9a8e3e10a:
 db{0}: trace/t 0t900
 trace: pid 900 lid 1 at 0xffff80009eb94b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7af9a8e3e10a:
 db{0}: trace/t 0t361
 trace: pid 361 lid 1 at 0xffff80009eadcb00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7af9a8e3e10a:
 db{0}: trace/t 0t549
 trace: pid 549 lid 1 at 0xffff80009dd72b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7af9a8e3e10a:
 db{0}: trace/t 0t516
 trace: pid 516 lid 1 at 0xffff80009e8a9b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7af9a8e3e10a:
 db{0}: trace/t 0t460
 trace: pid 460 lid 1 at 0xffff80009e12eb20
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x176
 pollcommon.part.3() at netbsd:pollcommon.part.3+0x8d
 sys_poll() at netbsd:sys_poll+0x9b
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 209) ---
 7dba6203e28a:
 db{0}: trace/a fffffe810ea7ca80
 trace: pid 0 lid 56 at 0xffff80008f7e3e40
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 tcp_slowtimo() at netbsd:tcp_slowtimo+0x10
 callout_softclock() at netbsd:callout_softclock+0x201
 softint_dispatch() at netbsd:softint_dispatch+0xd3
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f7e3ff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}: trace/a fffffe810e906940
 trace: pid 0 lid 26 at 0xffff80008f725e00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 tcp_timer_rexmt() at netbsd:tcp_timer_rexmt+0x27
 callout_softclock() at netbsd:callout_softclock+0x201
 softint_dispatch() at netbsd:softint_dispatch+0xd3
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f725ff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}: call cpu_reset

From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Thu, 05 Apr 2018 09:36:04 +0200 (CEST)

 BTW, the last update is a kernel which is GENERIC + DIAGNOSTIC +
 LOCKDEBUG and with 4x default NMBCLUSTERS.

 - H=E5vard

From: Ryota Ozaki <ozaki-r@netbsd.org>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc: port-amd64-maintainer@netbsd.org, gnats-admin@netbsd.org, 
	netbsd-bugs@netbsd.org, Havard Eidnes <he@netbsd.org>
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network
 interfaces in use
Date: Thu, 5 Apr 2018 16:48:54 +0900

 On Thu, Apr 5, 2018 at 4:40 PM, Havard Eidnes <he@netbsd.org> wrote:
 > The following reply was made to PR port-amd64/53155; it has been noted by GNATS.
 >
 > From: Havard Eidnes <he@NetBSD.org>
 > To: gnats-bugs@NetBSD.org
 > Cc:
 > Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 >  network interfaces in use
 > Date: Thu, 05 Apr 2018 09:36:04 +0200 (CEST)
 >
 >  BTW, the last update is a kernel which is GENERIC + DIAGNOSTIC +
 >  LOCKDEBUG and with 4x default NMBCLUSTERS.

 If you're using latest source code of netbsd-8, you can use a new ddb command:
 show all locks [/t], which may help for locking issues.

   ozaki-r

From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Thu, 05 Apr 2018 14:22:17 +0200 (CEST)

 Following ozaki-r's hints, here's more information with the locks
 dumped on another wedge event.

 "DDB break sequence keyed in"
 fatal breakpoint trap in supervisor mode
 trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0xffff80009da69000 ilevel 0x8 rsp 0xffff80008f69bcf8
 curlwp 0xfffffe822f6eb440 pid 0.6 lowest kstack 0xffff80008f6972c0
 Stopped in pid 0.6 (system) at  netbsd:breakpoint+0x5:  leave
 db{0}: show all locks
 [Locks tracked through LWPs]
 Locks held by an LWP (expect):
 Lock 0 (initialized at fstrans_init)
 lock address : 0xffffffff815cb9b8 type     :     sleep/adaptive
 initialized  : 0xffffffff809fe690
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  3
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe821e439080
 last locked* : 0xffffffff80a00523 unlocked : 0xffffffff80a0315f
 owner field  : 0xfffffe821e439080 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c5130.
 =: No active turnstile for this lock.
 Lock 1 (initialized at fstrans_init)
 lock address : 0xffffffff815cb9b0 type     :     sleep/adaptive
 initialized  : 0xffffffff809fe6a3
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  3
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe821e439080
 last locked* : 0xffffffff809ffb6d unlocked : 0xffffffff809fecc4
 owner field  : 0xfffffe821e439080 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c5120.
 =: No active turnstile for this lock.

 Locks held by an LWP (pickup):
 Lock 0 (initialized at fork1)
 lock address : 0xfffffe821dce4a30 type     :     sleep/adaptive
 initialized  : 0xffffffff809599ab
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  2
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe81b102da80
 last locked* : 0xffffffff809580c1 unlocked : 0xffffffff80879889
 owner/count  : 0xfffffe81b102da80 flags    : 0x0000000000000004

 Turnstile chain at 0xffffffff814c4e20.
 =: No active turnstile for this lock.
 Lock 1 (initialized at filedesc_ctor)
 lock address : 0xfffffe821f62f880 type     :     sleep/adaptive
 initialized  : 0xffffffff8094d11f
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  2
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe81b102da80
 last locked* : 0xffffffff80953c76 unlocked : 0xffffffff8094e4a5
 owner field  : 0xfffffe81b102da80 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c4ec0.
 =: No active turnstile for this lock.

 Locks held by an LWP (master):
 Lock 0 (initialized at filedesc_ctor)
 lock address : 0xfffffe822870adc0 type     :     sleep/adaptive
 initialized  : 0xffffffff8094d11f
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  1
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe822962bac0
 last locked* : 0xffffffff809522e8 unlocked : 0xffffffff8094ec84
 owner field  : 0xfffffe822962bac0 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c5140.
 =: No active turnstile for this lock.

 Locks held by an LWP (ioflush):
 Lock 0 (initialized at vcache_alloc)
 lock address : 0xfffffe8061cb6fa0 type     :     sleep/adaptive
 initialized  : 0xffffffff80a01652
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  7
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe822dce8100
 last locked* : 0xffffffff80a17d69 unlocked : 0xffffffff80a17db4
 owner/count  : 0xfffffe822dce8100 flags    : 0x0000000000000004

 Turnstile chain at 0xffffffff814c5100.
 =: No active turnstile for this lock.

 Locks held by an LWP (softnet/0):
 Lock 0 (initialized at soinit)
 lock address : 0xfffffe822fb97f40 type     :     sleep/adaptive
 initialized  : 0xffffffff809db0cb
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                 14
 current cpu  :                  0 last held:                  0
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe822f707420
 last locked* : 0xffffffff806d7add unlocked : 0xffffffff80951f2c
 owner field  : 0xfffffe822f707420 wait/spin:                1/0

 Turnstile chain at 0xffffffff814c5040.
 =: Turnstile at 0xfffffe810f381d20 (wrq=0xfffffe810f381d40, rdq=0xfffffe810f381d
 50).
 =: 0 waiting readers:
 =: 14 waiting writers: 0xfffffe810f396880 0xfffffe810e9a29c0 0xfffffe810e9e0a00 0xfffffe810e94c980 0xfffffe82212fc700 0xfffffe82228e28e0 0xfffffe8222950b60 0xfffffe82212fcb20 0xfffffe822962b6a0 0xfffffe822c341200 0xfffffe8220e621e0 0xfffffe821ea02260 0xfffffe821f0986c0 0xfffffe810f2da300


 [Locks tracked through CPUs]
 Locks held on CPU 0:
 Lock 0 (initialized at main)
 lock address : 0xffffffff814c3840 type     :               spin
 initialized  : 0xffffffff80d02d91
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  6
 current cpu  :                  0 last held:                  0
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe822f707420
 last locked* : 0xffffffff806d7ae7 unlocked : 0xffffffff80a3a00f
 curcpu holds :                  1 wanted by: 000000000000000000

 Lock 1 (initialized at com_attach_subr)
 lock address : 0xfffffe810f20b1e0 type     :               spin
 initialized  : 0xffffffff805ce644
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  0
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe822f6eb440
 last locked* : 0xffffffff805cdba0 unlocked : 0xffffffff805cdd7f
 owner field  : 0x0000000000010800 wait/spin:                0/1


 db{0}: ps
 PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 6110 :   1 7   3         0   fffffe821e439080             expect
 16029    1 3   5        80   fffffe8223e0d680                 sh wait
 17269    1 3   5        80   fffffe821e4398c0               perl pipe_rd
 6258     1 3   5        80   fffffe8220e10960                 sh wait
 6160     1 3   4        80   fffffe821db62940                ssh select
 9045     2 3   2        80   fffffe8207c29900             expect select
 9045     1 3   5        80   fffffe821d8e9120             expect parked
 8628     1 3   2        80   fffffe8217495860                 sh wait
 6162     1 3   0        80   fffffe821baa01a0               perl pipe_rd
 10236    1 3   4        80   fffffe8220439aa0                 sh wait
 10536    1 3   4     40000   fffffe821ea02260                ssh tstile
 16657    2 3   4        80   fffffe822d304a00             expect select
 16657    1 3   0        80   fffffe822244f160             expect parked
 1568     1 3   2        80   fffffe821c6c44c0                 sh wait
 621      1 3   3        80   fffffe821eb29980               perl pipe_rd
 1714     1 3   5        80   fffffe82220db9a0                 sh wait
 20537    1 3   7        80   fffffe8220e62600                ssh select
 855      2 3   3        80   fffffe8220c66920             expect select
 855  :   1 7   6         0   fffffe820e54e100             expect
 2913     1 3   6        80   fffffe8220439680                 sh wait
 10978    1 3   6        80   fffffe821d9671e0                ssh select
 26960    1 3   6        80   fffffe821f312a60               perl pipe_rd
 13813    1 3   0        80   fffffe820e54e940                 sh wait
 4350     2 3   6        80   fffffe8222d32880             expect select
 4350     1 3   0        80   fffffe82220e8b80             expect parked
 22724    1 3   7        80   fffffe822a7bc960                 sh wait
 5198     1 3   0        80   fffffe822301aba0               perl pipe_rd
 3944     1 3   5        80   fffffe821fe156e0                 sh wait
 6695     1 3   5        80   fffffe8221b59560                par pause
 1700     1 3   5        80   fffffe81dc6c72a0                 sh wait
 9549     1 3   1        80   fffffe821c9580e0                 sh wait
 6024     1 3   1        80   fffffe8220975780                 sh wait
 7673     1 3   1        80   fffffe822244f580                 sh wait
 10638    1 3   1        80   fffffe821ed0c9c0               cron pipe_rd
 4839     1 3   6        80   fffffe820e0aa200               sshd select
 3563     1 3   6        80   fffffe821f7e8a60               sshd select
 11359    1 3   1        80   fffffe821f4f2b40                ssh select
 1612     1 3   1        80   fffffe81dc6c7ae0                 sh wait
 16815    1 3   1        80   fffffe821ea02aa0               bash select
 5195     1 3   0        80   fffffe820f859840               sshd select
 19300    1 3   0        80   fffffe8220e62a20               sshd select
 6945     1 3   1        80   fffffe821d1d9340               bash wait
 5497     1 3   2        80   fffffe8226609b40               sshd select
 3611     1 3   2        80   fffffe8226a51b20               sshd select
 18181    1 3   6        80   fffffe82228e20a0                ssh select
 29538    1 3   6        80   fffffe82174aa600                 sh wait
 1782     1 3   2     40000   fffffe821f0986c0               ping tstile
 23170    1 3   4        80   fffffe822345f900                ssh select
 25239    1 3   6        80   fffffe8220e10540                 sh wait
 22752    1 3   2        80   fffffe8223e968c0                ssh select
 21643    1 3   6        80   fffffe8221723ac0                 sh wait
 22354    1 3   1        80   fffffe822301a780                ssh select
 12292    1 3   6        80   fffffe8221b59140                 sh wait
 440      1 3   1        80   fffffe821f2f9760               bash select
 16624    1 3   4        80   fffffe821dd60320               sshd select
 24686    1 3   3        80   fffffe82219b8480               sshd select
 17072:   1 7   2         0   fffffe81b102da80             pickup
 17333    1 3   3        80   fffffe821e378ba0               bash wait
 12287    1 3   5        80   fffffe8223e0d260               sshd select
 28431    1 3   3        80   fffffe821f62b300               sshd select
 25668    1 3   5        80   fffffe821db62100                ssh select
 18777    1 3   7        80   fffffe821d548460                 sh wait
 23483    1 3   6        80   fffffe821dfed5e0                ssh select
 24114    1 3   1        80   fffffe82219b8060                 sh wait
 18841    1 3   2        80   fffffe821db62520                ssh select
 8541     1 3   1        80   fffffe821c347020                 sh wait
 20854    1 3   3        80   fffffe8220a11420                ssh select
 28316    1 3   7        80   fffffe821d449760                 sh wait
 21952    1 3   6        80   fffffe8223e964a0                ssh select
 21821    1 3   7        80   fffffe81b0264680                 sh wait
 22659    1 3   7        80   fffffe82220db580                ssh select
 15239    1 3   7        80   fffffe8200e69640                 sh wait
 20787    1 3   0        80   fffffe8222093620                ssh select
 25294    1 3   7        80   fffffe821e959b20                 sh wait
 20182    1 3   4        80   fffffe82205cf440                ssh select
 21140    1 3   1        80   fffffe822301a360                 sh wait
 10348    1 3   1        80   fffffe822084f900                ssh select
 19979    1 3   6        80   fffffe8219d9d9e0                 sh wait
 20748    1 3   0        80   fffffe81b102d240                ssh select
 29870    1 3   6        80   fffffe821ea84a60                 sh wait
 28154    1 3   3        80   fffffe821e6f1060                ssh select
 21713    1 3   1        80   fffffe82215225a0                 sh wait
 22666    1 3   5        80   fffffe821fe912a0                ssh select
 19782    1 3   3        80   fffffe8228f16ae0                 sh wait
 23668    1 3   4        80   fffffe822325b440                ssh select
 24824    1 3   3        80   fffffe821c347440                 sh wait
 21452    1 3   1        80   fffffe821e6f18a0                ssh select
 18115    1 3   6        80   fffffe8223c84b20                 sh wait
 23734    1 3   3        80   fffffe82101970e0                ssh select
 23642    1 3   3        80   fffffe821b7f01c0                 sh wait
 15721    1 3   2        80   fffffe822d3045e0                ssh select
 24741    1 3   6        80   fffffe8222d32040                 sh wait
 25785    1 3   4        80   fffffe820e91e8c0                ssh select
 29353    1 3   6        80   fffffe822d96d580                 sh wait
 17591    1 3   3        80   fffffe8211ded180                ssh select
 22243    1 3   6        80   fffffe821e378360                 sh wait
 11016    1 3   7        80   fffffe820d036160                ssh select
 23915    1 3   6        80   fffffe822d9f1560                 sh wait
 23755    1 3   2        80   fffffe82220e8760                ssh select
 28517    1 3   3        80   fffffe821d967600                 sh wait
 21937    1 3   1        80   fffffe8222b59420                ssh select
 24531    1 3   3        80   fffffe8222093a40                 sh wait
 21091    1 3   5        80   fffffe82101cf480                ssh select
 24050    1 3   6        80   fffffe82174aaa20                 sh wait
 21377    1 3   7        80   fffffe821dfed1c0                ssh select
 25968    1 3   6        80   fffffe822d9f1140                 sh wait
 19956    1 3   5        80   fffffe8215f17980                ssh select
 22513    1 3   1        80   fffffe821d1d9b80                 sh wait
 23875    1 3   1        80   fffffe821e2da1e0                ssh select
 20549    1 3   1        80   fffffe821eb29560                 sh wait
 21480    1 3   6        80   fffffe822281b940                ssh select
 19370    1 3   6        80   fffffe8221343200                 sh wait
 27941    1 3   3        80   fffffe821d548040                ssh select
 1308     1 3   3        80   fffffe821abcc300                 sh wait
 16930    1 3   5        80   fffffe821e2a7160                ssh select
 20374    1 3   6        80   fffffe8221b59980                 sh wait
 17114    1 3   4        80   fffffe82219c66e0                ssh select
 19866    1 3   6        80   fffffe820f859000                 sh wait
 26412    1 3   6        80   fffffe821b800780                ssh select
 21138    1 3   1        80   fffffe821c0755a0                 sh wait
 19764    1 3   7        80   fffffe8220e10120                ssh select
 19992    1 3   6        80   fffffe82219c62c0                 sh wait
 8561     1 3   4        80   fffffe822c0c4660                ssh select
 22282    1 3   5        80   fffffe820a38b0a0                 sh wait
 21455    1 3   5        80   fffffe82205446c0                ssh select
 19503    1 3   7        80   fffffe822a7bc540                 sh wait
 20237    1 3   2        80   fffffe820f859420                ssh select
 23846    1 3   7        80   fffffe821b7f05e0                 sh wait
 23390    1 3   0        80   fffffe821f62b720                ssh select
 6312     1 3   3        80   fffffe822325b860                 sh wait
 22688    1 3   6        80   fffffe8200e69220                ssh select
 26232    1 3   6        80   fffffe8220975360                 sh wait
 25797    1 3   1        80   fffffe82217cba00                ssh select
 26094    1 3   5        80   fffffe822d77e5c0                 sh wait
 18349    1 3   2        80   fffffe82224e1920                ssh select
 18457    1 3   5        80   fffffe821b800360                 sh wait
 4150     1 3   6        80   fffffe8222ec34a0                ssh select
 17097    1 3   1        80   fffffe821dfeda00                 sh wait
 21786    1 3   2        80   fffffe81b0264260                ssh select
 21190    1 3   5        80   fffffe821e959700                 sh wait
 21970    1 3   4        80   fffffe8226a51700                ssh select
 19183    1 3   5        80   fffffe821f2f9b80                 sh wait
 18769    1 3   2        80   fffffe821db454e0                ssh select
 23677    1 3   1        80   fffffe821c958500                 sh wait
 21567    1 3   1        80   fffffe822232c060                ssh select
 27597    1 3   2        80   fffffe821ed0c5a0                 sh wait
 23189    1 3   0        80   fffffe8217495440                ssh select
 24479    1 3   0        80   fffffe82205cf860                 sh wait
 19747    1 3   5        80   fffffe820e91e080                ssh select
 22977    1 3   0        80   fffffe822d7a19c0                 sh wait
 19741    1 3   6        80   fffffe8211524880                ssh select
 21217    1 3   5        80   fffffe820a38b8e0                 sh wait
 17887    1 3   1        80   fffffe821f7e8640                ssh select
 20134    1 3   7        80   fffffe821f62bb40                 sh wait
 17673    1 3   3        80   fffffe8222ec3080                ssh select
 16512    1 3   5        80   fffffe8222d32460                 sh wait
 22179    1 3   7        80   fffffe8220c66500                ssh select
 23863    1 3   3        80   fffffe821ea84220                 sh wait
 15669    1 3   5        80   fffffe820e91e4a0                ssh select
 19089    1 3   3        80   fffffe821f4f2300                 sh wait
 22480    1 3   2        80   fffffe8222b59000                ssh select
 15576    1 3   6        80   fffffe8220439260                 sh wait
 16125    1 3   2        80   fffffe821f098ae0                ssh select
 14729    1 3   6        80   fffffe810f2f1b60                 sh wait
 20188    1 3   2        80   fffffe822d77e1a0                ssh select
 23671    1 3   6        80   fffffe8219d9d1a0                 sh wait
 2378     1 3   4        80   fffffe822244f9a0                ssh select
 13692    1 3   3        80   fffffe8226a512e0                 sh wait
 16255    1 3   6        80   fffffe8224cb4940                ssh select
 13845    1 3   3        80   fffffe822084f4e0                 sh wait
 21973    1 3   3        80   fffffe821c958920                ssh select
 13867    1 3   3        80   fffffe821e6f1480                 sh wait
 13804    1 3   4        80   fffffe82212fc2e0                ssh select
 20798    1 3   6        80   fffffe822325b020                 sh wait
 17218    1 3   2        80   fffffe8229c95960                ssh select
 3258     1 3   0        80   fffffe821fe152c0                 sh wait
 21575    1 3   1        80   fffffe82200a1660                ssh select
 18008    1 3   6        80   fffffe81b102d660                 sh wait
 19955    1 3   6        80   fffffe821db45900                ssh select
 19174    1 3   6        80   fffffe822dcc7120                 sh wait
 16916    1 3   2        80   fffffe822962b280                ssh select
 2248     1 3   6        80   fffffe82224e10e0                 sh wait
 16945    1 3   0        80   fffffe8222b59840                ssh select
 21163    1 3   2        80   fffffe8211524460                 sh wait
 22509    1 3   2        80   fffffe8223c84700                ssh select
 18626    1 3   2        80   fffffe82220e8340                 sh wait
 18828    1 3   6        80   fffffe821de0a8e0                ssh select
 15805    1 3   7        80   fffffe8221343a40                 sh wait
 20827    1 3   4        80   fffffe822c0c4a80                ssh select
 14384    1 3   0        80   fffffe821e4a6a40                 sh wait
 20713    1 3   6        80   fffffe82164eab60               bash select
 9812     1 3   3        80   fffffe82280a41a0               sshd select
 13062    1 3   3        80   fffffe8220975ba0               sshd select
 17386    1 3   6        80   fffffe820e0aaa40                ssh select
 11169    1 3   2        80   fffffe82101cf060                 sh wait
 1442     1 3   0        80   fffffe821c075180               sshd select
 458      1 3   0        80   fffffe821d8e9540          conserver select
 1083     1 3   4         0   fffffe822c341200          conserver tstile
 1182     1 3   4         0   fffffe82228e28e0          conserver tstile
 1179     1 3   4         0   fffffe82212fcb20          conserver tstile
 534      1 3   0        80   fffffe8220e93660          conserver select
 900      1 3   2         0   fffffe82212fc700          conserver tstile
 935      1 3   4         0   fffffe8220e621e0          conserver tstile
 775      1 3   4        80   fffffe822345f0c0          conserver select
 871      1 3   4        80   fffffe8222950740          conserver select
 902      1 3   4         0   fffffe8222950b60          conserver tstile
 724      1 3   7        80   fffffe822dd720e0              getty ttyraw
 799      1 3   4         0   fffffe822962b6a0          conserver tstile
 894      1 3   4        80   fffffe8226609300          conserver select
 891      1 3   4        80   fffffe8226609720               cron nanoslp
 570      1 3   4        80   fffffe822d7a15a0          python2.7 select
 558      1 3   4        80   fffffe8228f166c0              inetd kqueue
 485      1 3   7        80   fffffe822c0c4240              nginx kqueue
 548      1 3   7        80   fffffe8228f162a0              nginx pause
 397      1 3   6        80   fffffe822d405a20               qmgr kqueue
 759  :   1 7   1         0   fffffe822962bac0             master
 473      1 3   0        80   fffffe822a7ffaa0           tac_plus select
 428      1 3   2        80   fffffe822d3041c0             powerd kqueue
 408      1 3   4        80   fffffe822c9dfa60               ntpd pause
 342      4 3   3        80   fffffe822c9df220              dhcpd parked
 342      3 3   7        80   fffffe822c9df640              dhcpd kqueue
 342  :   2 7   0         0   fffffe822d7a1180              dhcpd
 342      1 3   1        80   fffffe822c341a40              dhcpd parked
 83       1 3   2         0   fffffe810f2da300            syslogd tstile
 1        1 3   7        80   fffffe810f512900               init wait
 0      123 3   0       200   fffffe822d77e9e0            npfgc-0 npfgccv
 0      122 3   5       200   fffffe822d405600              ipmi0 ipmi0
 0      121 3   7       200   fffffe822dce8940            physiod physiod
 0      120 3   0       200   fffffe822dcc7960           aiodoned aiodoned
 0    : 119 7   7       200   fffffe822dce8100            ioflush
 0      118 3   0       200   fffffe822dce8520           pgdaemon pgdaemon
 0      115 3   0       280   fffffe822dd72500              spkr1 bellcv
 0      114 3   5       280   fffffe822dd72920           audiomix play
 0      113 3   5       280   fffffe810ed3b2a0           audiorec record
 0      112 3   0       200   fffffe810f5120c0          atapibus0 sccomp
 0      110 3   4       200   fffffe810f20d2c0               usb4 usbevt
 0      109 3   4       200   fffffe810f2cb2e0               usb0 usbevt
 0      108 3   4       200   fffffe810f2dab40               usb5 usbevt
 0      107 3   0       200   fffffe810f2cbb20               usb3 usbevt
 0      106 3   4       200   fffffe810f2da720               usb2 usbevt
 0      105 3   4       200   fffffe810f5124e0               usb1 usbevt
 0      104 3   0       200   fffffe810f4a30a0            rt_free rt_free
 0      103 3   0       200   fffffe810f4a34c0              unpgc unpgc
 0      102 3   0       200   fffffe810f4a38e0    key_timehandler key_timehandler
 0      101 3   7       200   fffffe810f474080    icmp6_wqinput/7 icmp6_wqinput
 0      100 3   6       200   fffffe810f4744a0    icmp6_wqinput/6 icmp6_wqinput
 0       99 3   5       200   fffffe810f4748c0    icmp6_wqinput/5 icmp6_wqinput
 0       98 3   4       200   fffffe810f495060    icmp6_wqinput/4 icmp6_wqinput
 0       97 3   3       200   fffffe810f495480    icmp6_wqinput/3 icmp6_wqinput
 0       96 3   2       200   fffffe810f4958a0    icmp6_wqinput/2 icmp6_wqinput
 0       95 3   1       200   fffffe810f396040    icmp6_wqinput/1 icmp6_wqinput
 0       94 3   0       200   fffffe810f396460    icmp6_wqinput/0 icmp6_wqinput
 0       93 3   1     40200   fffffe810f396880          nd6_timer tstile
 0       92 3   7       200   fffffe810f35f020     icmp_wqinput/7 icmp_wqinput
 0       91 3   6       200   fffffe810f35f440     icmp_wqinput/6 icmp_wqinput
 0       90 3   5       200   fffffe810f35f860     icmp_wqinput/5 icmp_wqinput
 0       89 3   4       200   fffffe810f370000     icmp_wqinput/4 icmp_wqinput
 0       88 3   3       200   fffffe810f370420     icmp_wqinput/3 icmp_wqinput
 0       87 3   2       200   fffffe810f370840     icmp_wqinput/2 icmp_wqinput
 0       86 3   1       200   fffffe810f33e360     icmp_wqinput/1 icmp_wqinput
 0       85 3   0       200   fffffe810f33e780     icmp_wqinput/0 icmp_wqinput
 0       84 3   4       200   fffffe810f33eba0           rt_timer rt_timer
 0       83 3   2       200   fffffe810f2cb700        vmem_rehash vmem_rehash
 0       82 3   2       200   fffffe810f2f0340          coretemp3 coretemp3
 0       81 3   3       200   fffffe810f2f0760          coretemp2 coretemp2
 0       80 3   4       200   fffffe810f2f0b80          coretemp1 coretemp1
 0       79 3   5       200   fffffe810f2f1320          coretemp0 coretemp0
 0       78 3   5       200   fffffe810f2f1740               mfi0 mfi0
 0       69 3   5       200   fffffe810f20d6e0            atabus1 atath
 0       68 3   7       200   fffffe810f20db00            atabus0 atath
 0       66 3   0       200   fffffe810ed3b6c0           scsibus0 sccomp
 0       65 3   2       200   fffffe810ed3bae0         usbtask-dr usbtsk
 0       64 3   0       200   fffffe810ecaa280         usbtask-hc usbtsk
 0       63 3   7       200   fffffe810ecaa6a0               bnx3 bnx3
 0       62 3   4       200   fffffe810ecaaac0               bnx2 bnx2
 0       61 3   2       200   fffffe810eb61260               bnx1 bnx1
 0       60 3   5       200   fffffe810eb61680               bnx0 bnx0
 0    :  59 7   5       200   fffffe810eb61aa0               ipmi
 0       58 3   7       200   fffffe810ea7c240            xcall/7 xcall
 0       57 1   7       200   fffffe810ea7c660          softser/7
 0       56 1   7       200   fffffe810ea7ca80          softclk/7
 0       55 1   7       200   fffffe810ea5d220          softbio/7
 0       54 1   7       200   fffffe810ea5d640          softnet/7
 0       53 1   7       201   fffffe810ea5da60             idle/7
 0       52 3   6       200   fffffe810ea36200            xcall/6 xcall
 0       51 1   6       200   fffffe810ea36620          softser/6
 0       50 1   6       200   fffffe810ea36a40          softclk/6
 0       49 1   6       200   fffffe810ea0f1e0          softbio/6
 0       48 1   6       200   fffffe810ea0f600          softnet/6
 0       47 1   6       201   fffffe810ea0fa20             idle/6
 0       46 3   5       200   fffffe810e9e01c0            xcall/5 xcall
 0       45 1   5       200   fffffe810e9e05e0          softser/5
 0       44 3   5       200   fffffe810e9e0a00          softclk/5 tstile
 0       43 1   5       200   fffffe810e9d11a0          softbio/5
 0       42 1   5       200   fffffe810e9d15c0          softnet/5
 0       41 1   5       201   fffffe810e9d19e0             idle/5
 0       40 3   4       200   fffffe810e9a2180            xcall/4 xcall
 0       39 1   4       200   fffffe810e9a25a0          softser/4
 0       38 3   4       200   fffffe810e9a29c0          softclk/4 tstile
 0       37 1   4       200   fffffe810e973160          softbio/4
 0       36 1   4       200   fffffe810e973580          softnet/4
 0    :  35 7   4       201   fffffe810e9739a0             idle/4
 0       34 3   3       200   fffffe810e94c140            xcall/3 xcall
 0       33 1   3       200   fffffe810e94c560          softser/3
 0       32 3   3       200   fffffe810e94c980          softclk/3 tstile
 0       31 1   3       200   fffffe810e93d120          softbio/3
 0       30 1   3       200   fffffe810e93d540          softnet/3
 0       29 1   3       201   fffffe810e93d960             idle/3
 0       28 3   2       200   fffffe810e906100            xcall/2 xcall
 0       27 1   2       200   fffffe810e906520          softser/2
 0       26 1   2       200   fffffe810e906940          softclk/2
 0       25 1   2       200   fffffe810e8ef0e0          softbio/2
 0       24 1   2       200   fffffe810e8ef500          softnet/2
 0       23 1   2       201   fffffe810e8ef920             idle/2
 0       22 3   1       200   fffffe810e8c50c0            xcall/1 xcall
 0       21 1   1       200   fffffe810e8c54e0          softser/1
 0       20 1   1       200   fffffe810e8c5900          softclk/1
 0       19 1   1       200   fffffe810e8b10a0          softbio/1
 0       18 1   1       200   fffffe810e8b14c0          softnet/1
 0       17 1   1       201   fffffe810e8b18e0             idle/1
 0       16 3   0       200   fffffe822de92080           lnxsyswq lnxsyswq
 0       15 3   0       200   fffffe822de924a0             sysmon smtaskq
 0       14 3   5       200   fffffe822de928c0         pmfsuspend pmfsuspend
 0       13 3   7       200   fffffe822e2b1060           pmfevent pmfevent
 0       12 3   0       200   fffffe822e2b1480         sopendfree sopendfr
 0       11 3   1       200   fffffe822e2b18a0           nfssilly nfssilly
 0       10 3   4       200   fffffe822f6d0040            cachegc cachegc
 0        9 3   5       200   fffffe822f6d0460             vdrain vdrain
 0        8 3   0       200   fffffe822f6d0880          modunload mod_unld
 0        7 3   0       200   fffffe822f6eb020            xcall/0 xcall
 0    :   6 7   0       200   fffffe822f6eb440          softser/0
 0        5 1   0       200   fffffe822f6eb860          softclk/0
 0        4 1   0       200   fffffe822f707000          softbio/0
 0    :   3 7   0       200   fffffe822f707420          softnet/0
 0        2 1   0       201   fffffe822f707840             idle/0
 0        1 3   4       200   ffffffff81481b20            swapper uvm
 db{0}: trace/t 0t10536
 trace: pid 10536 lid 1 at 0xffff80009f171b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 739a9de3e10a:
 db{0}: trace/t 0t1782
 trace: pid 1782 lid 1 at 0xffff80009f18ad00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sosetopt() at netbsd:sosetopt+0x2e
 sys_setsockopt() at netbsd:sys_setsockopt+0x91
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 105) ---
 72968e6ba14a:
 db{0}: trace/t 0t1083
 trace: pid 1083 lid 1 at 0xffff80009e0b9b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7bc97bc3e10a:
 db{0}: trace/t 0t1182
 trace: pid 1182 lid 1 at 0xffff80009e9aeb00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7bc97bc3e10a:
 db{0}: trace/t 0t1179
 trace: pid 1179 lid 1 at 0xffff80009eb5db00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7bc97bc3e10a:
 db{0}: trace/t 0t900
 trace: pid 900 lid 1 at 0xffff80009eb74b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7bc97bc3e10a:
 db{0}: trace/t 0t935
 trace: pid 935 lid 1 at 0xffff80009eab8b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7bc97bc3e10a:
 db{0}: trace/t 0t902
 trace: pid 902 lid 1 at 0xffff80009e84eb00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7bc97bc3e10a:
 db{0}: trace/t 0t799
 trace: pid 799 lid 1 at 0xffff80009e517b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7bc97bc3e10a:
 db{0}: trace/t 0t83
 trace: pid 83 lid 1 at 0xffff80009db73ae0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sosend() at netbsd:sosend+0x48
 do_sys_sendmsg_so() at netbsd:do_sys_sendmsg_so+0x272
 do_sys_sendmsg() at netbsd:do_sys_sendmsg+0x85
 sys_sendto() at netbsd:sys_sendto+0x5c
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 133) ---
 7d35dd0eea7a:
 db{0}: trace/a fffffe810f396880
 trace: pid 0 lid 93 at 0xffff80009dbc5d60
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 nd6_timer_work() at netbsd:nd6_timer_work+0x3f
 workqueue_worker() at netbsd:workqueue_worker+0xcd
 db{0}: trace/a fffffe810e9e0a00
 trace: pid 0 lid 44 at 0xffff80008f797e00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 tcp_timer_rexmt() at netbsd:tcp_timer_rexmt+0x27
 callout_softclock() at netbsd:callout_softclock+0x201
 softint_dispatch() at netbsd:softint_dispatch+0xd3
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f797ff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}: trace/a fffffe810e9a29c0
 trace: pid 0 lid 38 at 0xffff80008f771e00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 tcp_timer_rexmt() at netbsd:tcp_timer_rexmt+0x27
 callout_softclock() at netbsd:callout_softclock+0x201
 softint_dispatch() at netbsd:softint_dispatch+0xd3
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f771ff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}: 
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 DDB lost frame for netbsd:Xdoreti, trying 0xffff80008f69bd70
 Xdoreti() at netbsd:Xdoreti
 --- interrupt ---
 Bad frame pointer: 0xffff800000000000
 202:
 db{0}: trace/a fffffe810e94c980
 trace: pid 0 lid 32 at 0xffff80008f74be00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 tcp_timer_rexmt() at netbsd:tcp_timer_rexmt+0x27
 callout_softclock() at netbsd:callout_softclock+0x201
 softint_dispatch() at netbsd:softint_dispatch+0xd3
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f74bff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}: show all locks /t
 [Locks tracked through LWPs]
 Locks held by an LWP (expect):
 Lock 0 (initialized at fstrans_init)
 lock address : 0xffffffff815cb9b8 type     :     sleep/adaptive
 initialized  : 0xffffffff809fe690
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  3
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe821e439080
 last locked* : 0xffffffff80a00523 unlocked : 0xffffffff80a0315f
 owner field  : 0xfffffe821e439080 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c5130.
 =: No active turnstile for this lock.
 Lock 1 (initialized at fstrans_init)
 lock address : 0xffffffff815cb9b0 type     :     sleep/adaptive
 initialized  : 0xffffffff809fe6a3
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  3
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe821e439080
 last locked* : 0xffffffff809ffb6d unlocked : 0xffffffff809fecc4
 owner field  : 0xfffffe821e439080 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c5120.
 =: No active turnstile for this lock.
 trace: pid 6110 lid 1 at 0xffff80009ed098a0
 sleepq_block() at netbsd:sleepq_block+0x1c5
 kpause() at netbsd:kpause+0xea
 pserialize_perform() at netbsd:pserialize_perform+0xf4
 fstrans_setstate() at netbsd:fstrans_setstate+0x42
 genfs_suspendctl() at netbsd:genfs_suspendctl+0x57
 VFS_SUSPENDCTL() at netbsd:VFS_SUSPENDCTL+0x1f
 vfs_suspend() at netbsd:vfs_suspend+0x5b
 vrevoke_suspend_next() at netbsd:vrevoke_suspend_next+0x2a
 vrevoke() at netbsd:vrevoke+0x2b
 genfs_revoke() at netbsd:genfs_revoke+0x13
 VOP_REVOKE() at netbsd:VOP_REVOKE+0x54
 pty_grant_slave() at netbsd:pty_grant_slave+0xa8
 ptmioctl() at netbsd:ptmioctl+0x90
 cdev_ioctl() at netbsd:cdev_ioctl+0x88
 VOP_IOCTL() at netbsd:VOP_IOCTL+0x3b
 vn_ioctl() at netbsd:vn_ioctl+0xa6
 sys_ioctl() at netbsd:sys_ioctl+0x101
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 54) ---
 74c471efeeaa:

 Locks held by an LWP (pickup):
 Lock 0 (initialized at fork1)
 lock address : 0xfffffe821dce4a30 type     :     sleep/adaptive
 initialized  : 0xffffffff809599ab
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  2
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe81b102da80
 last locked* : 0xffffffff809580c1 unlocked : 0xffffffff80879889
 owner/count  : 0xfffffe81b102da80 flags    : 0x0000000000000004

 Turnstile chain at 0xffffffff814c4e20.
 =: No active turnstile for this lock.
 Lock 1 (initialized at filedesc_ctor)
 lock address : 0xfffffe821f62f880 type     :     sleep/adaptive
 initialized  : 0xffffffff8094d11f
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  2
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe81b102da80
 last locked* : 0xffffffff80953c76 unlocked : 0xffffffff8094e4a5
 owner field  : 0xfffffe81b102da80 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c4ec0.
 =: No active turnstile for this lock.
 trace: pid 17072 lid 1 at 0xffff80009fa24ca0
 knote_detach() at netbsd:knote_detach+0x66
 knote_fdclose() at netbsd:knote_fdclose+0x44
 fd_close() at netbsd:fd_close+0x388
 fd_free() at netbsd:fd_free+0x17e
 exit1() at netbsd:exit1+0x11d
 sys_exit() at netbsd:sys_exit+0x3d
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 1) ---
 7947772fe52a:

 Locks held by an LWP (master):
 Lock 0 (initialized at filedesc_ctor)
 lock address : 0xfffffe822870adc0 type     :     sleep/adaptive
 initialized  : 0xffffffff8094d11f
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  1
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe822962bac0
 last locked* : 0xffffffff809522e8 unlocked : 0xffffffff8094ec84
 owner field  : 0xfffffe822962bac0 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c5140.
 =: No active turnstile for this lock.
 trace: pid 759 lid 1 at 0x81051570
 ?() at 0
 address 0x81051578 is invalid
 address 0x81051570 is invalid

 Locks held by an LWP (ioflush):
 Lock 0 (initialized at vcache_alloc)
 lock address : 0xfffffe8061cb6fa0 type     :     sleep/adaptive
 initialized  : 0xffffffff80a01652
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  7
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe822dce8100
 last locked* : 0xffffffff80a17d69 unlocked : 0xffffffff80a17db4
 owner/count  : 0xfffffe822dce8100 flags    : 0x0000000000000004

 Turnstile chain at 0xffffffff814c5100.
 =: No active turnstile for this lock.
 trace: pid 0 lid 119 at 0xffff80009dce9790
 scsipi_adapter_request() at netbsd:scsipi_adapter_request+0x52
 scsipi_run_queue() at netbsd:scsipi_run_queue+0xe2
 scsipi_execute_xs() at netbsd:scsipi_execute_xs+0x1bb
 sd_diskstart() at netbsd:sd_diskstart+0x25b
 dk_start() at netbsd:dk_start+0xde
 spec_strategy() at netbsd:spec_strategy+0x9a
 VOP_STRATEGY() at netbsd:VOP_STRATEGY+0x54
 genfs_do_io() at netbsd:genfs_do_io+0x1b4
 genfs_gop_write() at netbsd:genfs_gop_write+0x52
 genfs_do_putpages() at netbsd:genfs_do_putpages+0xb9c
 VOP_PUTPAGES() at netbsd:VOP_PUTPAGES+0x3a
 ffs_full_fsync() at netbsd:ffs_full_fsync+0x116
 ffs_fsync() at netbsd:ffs_fsync+0x3a
 VOP_FSYNC() at netbsd:VOP_FSYNC+0x3e
 sched_sync() at netbsd:sched_sync+0x198

 Locks held by an LWP (softnet/0):
 Lock 0 (initialized at soinit)
 lock address : 0xfffffe822fb97f40 type     :     sleep/adaptive
 initialized  : 0xffffffff809db0cb
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                 14
 current cpu  :                  0 last held:                  0
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe822f707420
 last locked* : 0xffffffff806d7add unlocked : 0xffffffff80951f2c
 owner field  : 0xfffffe822f707420 wait/spin:                1/0

 Turnstile chain at 0xffffffff814c5040.
 =: Turnstile at 0xfffffe810f381d20 (wrq=0xfffffe810f381d40, rdq=0xfffffe810f381d50).
 =: 0 waiting readers:
 =: 14 waiting writers: 0xfffffe810f396880 0xfffffe810e9a29c0 0xfffffe810e9e0a00 0xfffffe810e94c980 0xfffffe82212fc700 0xfffffe82228e28e0 0xfffffe8222950b60 0xfffffe82212fcb20 0xfffffe822962b6a0 0xfffffe822c341200 0xfffffe8220e621e0 0xfffffe821ea02260 0xfffffe821f0986c0 0xfffffe810f2da300
 trace: pid 0 lid 3 at 0xffff80008f68cc00
 sowakeup() at netbsd:sowakeup+0x58
 tcp_input() at netbsd:tcp_input+0x2f61
 tcp6_input() at netbsd:tcp6_input+0x49
 ip6_input() at netbsd:ip6_input+0x69e
 ip6intr() at netbsd:ip6intr+0x7b
 softint_dispatch() at netbsd:softint_dispatch+0xd3
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f68cff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:


 [Locks tracked through CPUs]
 Locks held on CPU 0:
 Lock 0 (initialized at main)
 lock address : 0xffffffff814c3840 type     :               spin
 initialized  : 0xffffffff80d02d91
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  6
 current cpu  :                  0 last held:                  0
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe822f707420
 last locked* : 0xffffffff806d7ae7 unlocked : 0xffffffff80a3a00f
 curcpu holds :                  1 wanted by: 000000000000000000
 trace: pid 0 lid 6 at 0xffff80008f69bcf8
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 DDB lost frame for netbsd:Xdoreti, trying 0xffff80008f69bd70
 Xdoreti() at netbsd:Xdoreti
 --- interrupt ---
 Bad frame pointer: 0xffff800000000000
 202:

 Lock 1 (initialized at com_attach_subr)
 lock address : 0xfffffe810f20b1e0 type     :               spin
 initialized  : 0xffffffff805ce644
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  0
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe822f6eb440
 last locked* : 0xffffffff805cdba0 unlocked : 0xffffffff805cdd7f
 owner field  : 0x0000000000010800 wait/spin:                0/1
 trace: pid 0 lid 6 at 0xffff80008f69bcf8
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 DDB lost frame for netbsd:Xdoreti, trying 0xffff80008f69bd70
 Xdoreti() at netbsd:Xdoreti
 --- interrupt ---
 Bad frame pointer: 0xffff800000000000
 202:


 db{0}: call cpu_reset

From: Ryota Ozaki <ozaki-r@netbsd.org>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc: port-amd64-maintainer@netbsd.org, gnats-admin@netbsd.org, 
	netbsd-bugs@netbsd.org, Havard Eidnes <he@netbsd.org>
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network
 interfaces in use
Date: Fri, 6 Apr 2018 02:24:11 +0900

 On Thu, Apr 5, 2018 at 9:25 PM, Havard Eidnes <he@netbsd.org> wrote:
 > The following reply was made to PR port-amd64/53155; it has been noted by=
  GNATS.
 >
 > From: Havard Eidnes <he@NetBSD.org>
 > To: gnats-bugs@NetBSD.org
 > Cc:
 > Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 >  network interfaces in use
 > Date: Thu, 05 Apr 2018 14:22:17 +0200 (CEST)
 >
 >  Following ozaki-r's hints, here's more information with the locks
 >  dumped on another wedge event.

 I've tried to compose a resource dependency graph from the outputs
 (if the graph has a loop, there is a deadlock) and this is a result:

   [expect] vfs_suspend_lock, fstrans_lock =3D> pserialize_perform =3D> kpau=
 se =3D>
   [tcp_timer_rexmt (softclk/?)] softnet_lock =3D> [ip6intr (softnet/0)]
 sowakeup =3D> ???

 I don't know what is going on in sowakeup. If it gets stuck because of
 vfs_suspend_lock or fstrans_lock, a loop appears.

 vfs experts would find a missing link.

   ozaki-r




From: Joerg Sonnenberger <joerg@bec.de>
To: gnats-bugs@NetBSD.org
Cc: port-amd64-maintainer@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org, he@NetBSD.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Thu, 5 Apr 2018 21:03:46 +0200

 On Thu, Apr 05, 2018 at 07:25:00AM +0000, Havard Eidnes wrote:
 >  --- interrupt ---
 >  Bad frame pointer: 0xffff800000000000
 >  202:
 >  db{0}: machine cpu 1
 >  using CPU 1
 >  db{0}: tra
 >  x86_pause() at netbsd:x86_pause+0x2
 >  cdev_open() at netbsd:cdev_open+0xaf
 >  spec_open() at netbsd:spec_open+0x260
 >  VOP_OPEN() at netbsd:VOP_OPEN+0x33
 >  vn_open() at netbsd:vn_open+0x28b
 >  do_open() at netbsd:do_open+0x112
 >  do_sys_openat() at netbsd:do_sys_openat+0x68
 >  sys_open() at netbsd:sys_open+0x24
 >  syscall() at netbsd:syscall+0x1ed

 Any chance to find out what device this is? I wonder due to the kernel
 lock instances on the other cores...

 Joerg

From: Havard Eidnes <he@NetBSD.org>
To: joerg@bec.de
Cc: gnats-bugs@NetBSD.org, port-amd64-maintainer@netbsd.org,
 netbsd-bugs@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Fri, 06 Apr 2018 00:30:04 +0200 (CEST)

 > On Thu, Apr 05, 2018 at 07:25:00AM +0000, Havard Eidnes wrote:
 >>  --- interrupt ---
 >>  Bad frame pointer: 0xffff800000000000
 >>  202:
 >>  db{0}: machine cpu 1
 >>  using CPU 1
 >>  db{0}: tra
 >>  x86_pause() at netbsd:x86_pause+0x2
 >>  cdev_open() at netbsd:cdev_open+0xaf
 >>  spec_open() at netbsd:spec_open+0x260
 >>  VOP_OPEN() at netbsd:VOP_OPEN+0x33
 >>  vn_open() at netbsd:vn_open+0x28b
 >>  do_open() at netbsd:do_open+0x112
 >>  do_sys_openat() at netbsd:do_sys_openat+0x68
 >>  sys_open() at netbsd:sys_open+0x24
 >>  syscall() at netbsd:syscall+0x1ed
 >
 > Any chance to find out what device this is? I wonder due to the kerne=
 l
 > lock instances on the other cores...

 For one, the machine is no longer sitting in DDB, it is a
 semi-normal production machine so needs to be up most of the
 time.

 Secondly, how would I go about finding out?

 I don't even know how to identify which process this is, perhaps
 other than doing a backtrace of each and every process on the
 system at the time of the wedge.

 However, if I were going to give a guess, I would guess at one of
 the expect processes, which typically want to open a pty pair,
 which should match the cdev_open() use.

 Regards,

 - H=E5vard

From: mlelstv@serpens.de (Michael van Elst)
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network interfaces in use
Date: Fri, 6 Apr 2018 05:38:13 -0000 (UTC)

 he@NetBSD.org (Havard Eidnes) writes:

 >I don't even know how to identify which process this is, perhaps
 >other than doing a backtrace of each and every process on the
 >system at the time of the wedge.

 The ps output has marked the processes running on a CPU:

  >  PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
  >  6110 :   1 7   3         0   fffffe821e439080             expect
  >  855  :   1 7   6         0   fffffe820e54e100             expect
  >  759  :   1 7   1         0   fffffe822962bac0             master
  >  342  :   2 7   0         0   fffffe822d7a1180              dhcpd
  >  0    : 119 7   7       200   fffffe822dce8100            ioflush
  >  0    :  59 7   5       200   fffffe810eb61aa0               ipmi
  >  0    :  35 7   4       201   fffffe810e9739a0             idle/4
  >  0    :   6 7   0       200   fffffe822f6eb440          softser/0
  >  0    :   3 7   0       200   fffffe822f707420          softnet/0
          ^
          |


  >>  db{0}: machine cpu 1
  >>  using CPU 1
  >>  db{0}: tra
  >>  x86_pause() at netbsd:x86_pause+0x2
  >>  cdev_open() at netbsd:cdev_open+0xaf
  >>  spec_open() at netbsd:spec_open+0x260
  >>  VOP_OPEN() at netbsd:VOP_OPEN+0x33
  >>  vn_open() at netbsd:vn_open+0x28b
  >>  do_open() at netbsd:do_open+0x112
  >>  do_sys_openat() at netbsd:do_sys_openat+0x68
  >>  sys_open() at netbsd:sys_open+0x24
  >>  syscall() at netbsd:syscall+0x1ed

 Assuming both outputs are from the same state: CPU 1 is running PID 759 (master).
 From "netbsd:cdev_open+0xaf" I would also think it's waiting on KERNEL_LOCK
 which is held by:

  >  current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe822f6eb440

  >  0    :   6 7   0       200   fffffe822f6eb440          softser/0


 Unfortunately there is no backtrace of that LWP.

 -- 
 -- 
                                 Michael van Elst
 Internet: mlelstv@serpens.de
                                 "A potential Snark may lurk in every tree."

From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org, mlelstv@serpens.de
Cc: netbsd-bugs@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Fri, 06 Apr 2018 08:47:08 +0200 (CEST)

 >  >I don't even know how to identify which process this is, perhaps
 >  >other than doing a backtrace of each and every process on the
 >  >system at the time of the wedge.
 >
 >  The ps output has marked the processes running on a CPU:

 Ah, of course, thanks.

 >  Assuming both outputs are from the same state: CPU 1 is running PID =
 759 (master).
 >  From "netbsd:cdev_open+0xaf" I would also think it's waiting on KERN=
 EL_LOCK
 >  which is held by:
 >
 >   >  current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe822f6eb440
 >
 >   >  0    :   6 7   0       200   fffffe822f6eb440          softser/0=

 >
 >  Unfortunately there is no backtrace of that LWP.

 OK, we'll provoke another wedge and I'll make sure to collect
 this information sometime later today (I hope).

 Regards,

 - H=E5vard

From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org, mlelstv@serpens.de
Cc: netbsd-bugs@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Fri, 06 Apr 2018 14:21:00 +0200 (CEST)

 OK, now I should have at least the backtraces of all the PIDs
 active on CPUs + those in tstile state.

 "DDB key sequence typed"
 fatal breakpoint trap in supervisor mode
 trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0x75e5b2f26020 ilevel 0x8 rsp 0xffff80008f69bc30
 curlwp 0xfffffe822f6eb440 pid 0.6 lowest kstack 0xffff80008f6972c0
 Stopped in pid 0.6 (system) at  netbsd:breakpoint+0x5:  leave
 db{0}: tra
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 uhci_intr() at netbsd:uhci_intr+0x19
 intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
 Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}: show all locks /t
 [Locks tracked through LWPs]
 Locks held by an LWP (pickup):
 Lock 0 (initialized at filedesc_ctor)
 lock address : 0xfffffe821eb37240 type     :     sleep/adaptive
 initialized  : 0xffffffff8094d11f
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  3
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe81711d9960
 last locked* : 0xffffffff8095362b unlocked : 0xffffffff809536d4
 owner field  : 0xfffffe81711d9960 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c4e40.
 =: No active turnstile for this lock.
 trace: pid 22385 lid 1 at 0xffff80009ed64e70
 kevent1() at netbsd:kevent1+0x648
 sys___kevent50() at netbsd:sys___kevent50+0x33
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 435) ---
 72a5bf03e40a:

 Locks held by an LWP (master):
 Lock 0 (initialized at filedesc_ctor)
 lock address : 0xfffffe822bcb1a00 type     :     sleep/adaptive
 initialized  : 0xffffffff8094d11f
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  1
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe822d043a00
 last locked* : 0xffffffff809522e8 unlocked : 0xffffffff8094ec84
 owner field  : 0xfffffe822d043a00 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c4dc0.
 =: No active turnstile for this lock.
 trace: pid 735 lid 1 at 0xffff80009dea1bd0
 kqueue_register() at netbsd:kqueue_register+0x431
 kevent1() at netbsd:kevent1+0x150
 sys___kevent50() at netbsd:sys___kevent50+0x33
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 435) ---
 74de5623e40a:

 Locks held by an LWP (npfgc-0):
 Lock 0 (initialized at npf_config_init)
 lock address : 0xfffffe822dd5f4c8 type     :     sleep/adaptive
 initialized  : 0xffffffff8186d8cf
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  6
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe822d96f160
 last locked* : 0xffffffff81872df3 unlocked : 0xffffffff81872e03
 owner field  : 0xfffffe822d96f160 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c4f50.
 =: No active turnstile for this lock.
 trace: pid 0 lid 123 at 0xffff80009dd9edb0
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0x127
 xc_wait() at netbsd:xc_wait+0x7c
 pserialize_perform() at netbsd:pserialize_perform+0xdf
 npf_conn_gc() at npf:npf_conn_gc+0x1b5
 npf_worker() at npf:npf_worker+0x4b

 Locks held by an LWP (ioflush):
 Lock 0 (initialized at vcache_alloc)
 lock address : 0xfffffe822db39528 type     :     sleep/adaptive
 initialized  : 0xffffffff80a01652
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  5
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe822dd08520
 last locked* : 0xffffffff80a17d69 unlocked : 0xffffffff80a17db4
 owner/count  : 0xfffffe822dd08520 flags    : 0x0000000000000004

 Turnstile chain at 0xffffffff814c5010.
 =: No active turnstile for this lock.
 trace: pid 0 lid 119 at 0xffff80009dce4790
 scsipi_adapter_request() at netbsd:scsipi_adapter_request+0x52
 scsipi_run_queue() at netbsd:scsipi_run_queue+0xe2
 scsipi_execute_xs() at netbsd:scsipi_execute_xs+0x1bb
 sd_diskstart() at netbsd:sd_diskstart+0x25b
 dk_start() at netbsd:dk_start+0xde
 spec_strategy() at netbsd:spec_strategy+0x9a
 VOP_STRATEGY() at netbsd:VOP_STRATEGY+0x54
 genfs_do_io() at netbsd:genfs_do_io+0x1b4
 genfs_gop_write() at netbsd:genfs_gop_write+0x52
 genfs_do_putpages() at netbsd:genfs_do_putpages+0xb9c
 VOP_PUTPAGES() at netbsd:VOP_PUTPAGES+0x3a
 ffs_full_fsync() at netbsd:ffs_full_fsync+0x116
 ffs_fsync() at netbsd:ffs_fsync+0x3a
 VOP_FSYNC() at netbsd:VOP_FSYNC+0x3e
 sched_sync() at netbsd:sched_sync+0x198

 Locks held by an LWP (nd6_timer):
 Lock 0 (initialized at soinit)
 lock address : 0xfffffe822fb97f40 type     :     sleep/adaptive
 initialized  : 0xffffffff809db0cb
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                 12
 current cpu  :                  0 last held:                  7
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe810f38e880
 last locked* : 0xffffffff806e1987 unlocked : 0xffffffff809dce9a
 owner field  : 0xfffffe810f38e880 wait/spin:                1/0

 Turnstile chain at 0xffffffff814c5040.
 =: Turnstile at 0xfffffe810f3810c0 (wrq=0xfffffe810f3810e0, rdq=0xfffffe810f3810f0).
 =: 0 waiting readers:
 =: 12 waiting writers: 0xfffffe810e9a29c0 0xfffffe810f2da720 0xfffffe822d96f9a0 0xfffffe822a70dac0 0xfffffe8223245340 0xfffffe821f081780 0xfffffe822111e2e0 0xfffffe82216935e0 0xfffffe82235c9900 0xfffffe821e590b40 0xfffffe821f30e560 0xfffffe821dc19940
 trace: pid 0 lid 93 at 0xffff80009dbc5ea0
 nd6_timer_work() at netbsd:nd6_timer_work+0x49
 workqueue_worker() at netbsd:workqueue_worker+0xcd


 [Locks tracked through CPUs]
 Locks held on CPU 0:
 Lock 0 (initialized at main)
 lock address : 0xffffffff814c3840 type     :               spin
 initialized  : 0xffffffff80d02d91
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  7
 current cpu  :                  0 last held:                  0
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe8175a02200
 last locked* : 0xffffffff80a0ed67 unlocked : 0xffffffff80a0e3ea
 curcpu holds :                  2 wanted by: 000000000000000000
 trace: pid 0 lid 6 at 0xffff80008f69bc30
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 uhci_intr() at netbsd:uhci_intr+0x19
 intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
 Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:

 Lock 1 (initialized at com_attach_subr)
 lock address : 0xfffffe810f20b1e0 type     :               spin
 initialized  : 0xffffffff805ce644
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  0
 current lwp  : 0xfffffe822f6eb440 last held: 0xfffffe822f6eb440
 last locked* : 0xffffffff805cdba0 unlocked : 0xffffffff805cdd7f
 owner field  : 0x0000000000010800 wait/spin:                0/1
 trace: pid 0 lid 6 at 0xffff80008f69bc30
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 uhci_intr() at netbsd:uhci_intr+0x19
 intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
 Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:


 db{0}: ps
 PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 28757    1 3   2        80   fffffe821bb6f840                ssh select
 19899    2 3   1        80   fffffe816ce52980             expect select
 19899    1 2   1         0   fffffe82273c5b80             expect
 26666    1 3   2        80   fffffe82274af2a0                 sh wait
 6662     1 3   7        80   fffffe8175e19760               perl pipe_rd
 2392     1 3   7        80   fffffe8176c60700                 sh wait
 3520     1 3   1        80   fffffe8015ba2580                ssh select
 6776     2 3   7        80   fffffe821bb6f420             expect select
 6776 :   1 7   0         0   fffffe8175a02200             expect
 26945    1 3   1        80   fffffe822cddb620                 sh wait
 2043     1 3   0        80   fffffe801faf7460               perl pipe_rd
 14744    1 3   7        80   fffffe816f4a2460                 sh wait
 14052    1 3   3        80   fffffe821dc0b5a0                ssh select
 5530     2 2   3         0   fffffe8175b4e060             expect
 5530 :   1 7   4         0   fffffe805815d0e0             expect
 7604     1 3   7        80   fffffe817c8655a0                 sh wait
 934      1 2   5         0   fffffe813db3fb00               perl
 25713    1 3   7        80   fffffe821cf411c0                 sh wait
 15595    1 3   0        80   fffffe817197e9c0                ssh select
 22126:   2 7   6         0   fffffe822aac6700             expect
 22126:   1 7   2         0   fffffe81788f42a0             expect
 4276     1 3   7        80   fffffe8174860660                 sh wait
 11831    1 3   4        80   fffffe822d0431c0               perl pipe_rd
 9327     1 3   7        80   fffffe8057cc5520                 sh wait
 27698    1 3   1        80   fffffe8129e07ae0                ssh select
 1953     2 3   2        80   fffffe8175aeeaa0             expect select
 1953     1 3   0        80   fffffe8226268260             expect parked
 17149    1 3   0        80   fffffe81749f7020                 sh wait
 4019     1 3   2        80   fffffe817bd8c5e0               perl pipe_rd
 17099    1 3   0        80   fffffe821d640040                 sh wait
 26876    1 3   7        80   fffffe821f30e980                par pause
 23287    1 3   7        80   fffffe8229ce8780                 sh wait
 1075     1 3   4        80   fffffe817197e180                 sh wait
 17073    1 3   4        80   fffffe817a8eb560                 sh wait
 28968    1 3   4        80   fffffe822047f500                 sh wait
 15804    1 3   4        80   fffffe817c7f59a0               cron pipe_rd
 24695    1 3   3        80   fffffe80716ad540                ssh select
 29259    1 3   7        80   fffffe8020710020               bash wait
 4005     1 3   3        80   fffffe81787d0280               sshd select
 1624     1 3   0        80   fffffe8216428060               sshd select
 22385:   1 7   3         0   fffffe81711d9960             pickup
 14283    1 3   0        80   fffffe8228a6a6e0                ssh select
 67       1 3   0        80   fffffe8170e612c0               bash wait
 7137     1 3   2        80   fffffe8133f6ab80               sshd select
 22735    1 3   1        80   fffffe8118cd8b40               sshd select
 25974    1 3   7        80   fffffe821e0b10c0               bash select
 23420    1 3   0        80   fffffe8170e61b00               sshd select
 29656    1 3   0        80   fffffe813db3f2c0               sshd select
 28863    1 3   7        80   fffffe81797b3640               bash select
 26749    1 3   5        80   fffffe8175e19b80               tcsh pause
 27317    1 3   0        80   fffffe816f4a2880              xterm select
 22374    1 3   0        80   fffffe821e93ea20               tcsh pause
 28940    1 3   2        80   fffffe817197e5a0               sshd select
 25310    1 3   0        80   fffffe816e7a8840               sshd select
 26934    1 3   0        80   fffffe8174828360               sshd select
 28572    1 3   0        80   fffffe82273c5340               sshd select
 26471    1 3   0        80   fffffe817c8659c0                ssh select
 12600    1 3   3        80   fffffe81797b3a60               bash wait
 11468    1 3   6        80   fffffe821af06440               sshd select
 14064    1 3   4        80   fffffe8229ce8360               sshd select
 27603    1 3   0        80   fffffe8178ff7600               sshd select
 19455    1 3   0        80   fffffe8226268680                ssh select
 24504    1 3   2        80   fffffe8170e616e0               bash wait
 23735    1 3   3        80   fffffe821e7a9060               sshd select
 11768    1 3   3        80   fffffe813db3f6e0               sshd select
 10560    1 3   0        80   fffffe82164408c0               sshd select
 8508     1 3   7        80   fffffe81707941c0                ssh select
 9649     1 3   0        80   fffffe816c812a40               bash wait
 25381    1 3   0        80   fffffe822722d6a0               sshd select
 7565     1 3   1        80   fffffe82197160a0               sshd select
 2309     1 3   6        80   fffffe81745ad1a0                ssh select
 26354    1 3   7        80   fffffe8173cfc720               bash wait
 17405    1 3   5        80   fffffe81711d9120               sshd select
 751      1 3   0        80   fffffe822a70d280               sshd select
 27677    1 3   0        80   fffffe822722dac0                ssh select
 357      1 3   7        80   fffffe816c812620               bash wait
 25010    1 3   0        80   fffffe82273c5760               sshd select
 1179     1 3   0        80   fffffe8174828ba0               sshd select
 26922    1 3   0        80   fffffe821f13e9a0                ssh select
 23611    1 3   0        80   fffffe813ceb9b20               bash wait
 17211    1 3   3        80   fffffe822004d080               sshd select
 19266    1 3   5        80   fffffe81754f6920               sshd select
 4685     1 3   1        80   fffffe8118cd8720                ssh select
 8078     1 3   1        80   fffffe8174860240               tcsh pause
 16167    1 3   0        80   fffffe8173cfcb40              xterm select
 10545    1 3   4        80   fffffe821e87f540               tcsh pause
 13913    1 3   6        80   fffffe817c81d520               sshd select
 16693    1 3   4        80   fffffe821f13e160               sshd select
 25137    1 3   6        80   fffffe822b87a680               less ttyraw
 4957     1 3   6        80   fffffe81711d9540               tcsh pause
 5578     1 3   5        80   fffffe81787d0ac0              xterm select
 5847     1 3   3        80   fffffe821bb6f000               tcsh pause
 21995    1 3   7        80   fffffe8175a02a40               sshd select
 17502    1 3   3        80   fffffe8175a02620               sshd select
 7577     1 4   7   1000000   fffffe82164288a0               less
 578      1 3   6        80   fffffe821d0d2640                ssh select
 1125     1 3   3        80   fffffe821bb785c0                 sh wait
 22670    1 3   3        80   fffffe821da51a40                ssh select
 20005    1 3   7        80   fffffe821e9f5240                 sh wait
 3107     1 3   7        80   fffffe822aac6b20                ssh select
 1604     1 3   6        80   fffffe81794ca5c0                 sh wait
 19643    1 3   6        80   fffffe816ce52560                ssh select
 19274    1 3   3        80   fffffe821cf415e0                 sh wait
 19209    1 3   5        80   fffffe817a8eb980                ssh select
 1745     1 3   3        80   fffffe817beb7900                 sh wait
 20698    1 3   2        80   fffffe821cf41a00                ssh select
 2093     1 3   1        80   fffffe821da51620                 sh wait
 22181    1 3   5        80   fffffe821f13e580                ssh select
 1594     1 3   0        80   fffffe817c77d120                 sh wait
 9837     1 3   4        80   fffffe821e9f5660                ssh select
 20910    1 3   5        80   fffffe821a83d040                 sh wait
 21449    1 3   1        80   fffffe817b1cfa80                ssh select
 1321     1 3   5        80   fffffe81716b2100                 sh wait
 23060    1 3   5        80   fffffe821e9f5a80                ssh select
 1702     1 3   0        80   fffffe8176c602e0                 sh wait
 1949     1 3   2        80   fffffe821dc19520                ssh select
 18511    1 3   4        80   fffffe8219e58b40                 sh wait
 2206     1 3   3        80   fffffe82298eaae0                ssh select
 1312     1 3   2        80   fffffe821bb19320                 sh wait
 23320    1 3   5        80   fffffe821f30e140                ssh select
 19993    1 3   6        80   fffffe81716b2520                 sh wait
 555      1 3   3        80   fffffe822004d4a0                ssh select
 22754    1 3   6        80   fffffe816ce52140                 sh wait
 545      1 3   6        80   fffffe821deca4c0                ssh select
 20813    1 3   6        80   fffffe821fee3740                 sh wait
 22820    1 3   4        80   fffffe821deca8e0                ssh select
 18960    1 3   7        80   fffffe82298ea2a0                 sh wait
 20254    1 3   6        80   fffffe821e0b14e0                ssh select
 20453    1 3   4        80   fffffe81794ca1a0                 sh wait
 17259    1 3   6        80   fffffe8173ba50a0                ssh select
 19827    1 3   4        80   fffffe8220756420                 sh wait
 1052     1 3   2        80   fffffe821eed1440                ssh select
 12864    1 3   7        80   fffffe822bcd5220                 sh wait
 8700     1 3   3        80   fffffe821eed1020                ssh select
 21994    1 3   2        80   fffffe82274afae0                 sh wait
 21369    1 3   2        80   fffffe816d83b1e0                ssh select
 20846    1 3   5        80   fffffe8170794a00                 sh wait
 21204    1 3   7        80   fffffe8220756840                ssh select
 21380    1 3   1        80   fffffe817bd8ca00                 sh wait
 20422    1 3   2        80   fffffe821f081360                ssh select
 20223    1 3   2        80   fffffe8216440080                 sh wait
 12794    1 3   0        80   fffffe81790a6080               tcsh ttyraw
 4984     1 3   7        80   fffffe822004d8c0               tcsh pause
 12264    1 3   7        80   fffffe8173ba58e0              xterm select
 10847    1 3   0        80   fffffe821d0d2220               tcsh pause
 12000    1 3   6        80   fffffe817c77d960               sshd select
 12914    1 3   0        80   fffffe8216428480               sshd select
 2019     1 3   0        80   fffffe8225286340              getty ttyraw
 1589     1 3   2        80   fffffe8227ca52c0                ssh select
 816      1 3   2        80   fffffe8227ca5b00                 sh wait
 953      1 3   2         0   fffffe821f30e560          conserver tstile
 306      1 3   4         0   fffffe821dc19940          conserver tstile
 1047     1 3   6         0   fffffe821f081780          conserver tstile
 534      1 3   6         0   fffffe822111e2e0          conserver tstile
 1045     1 3   2        80   fffffe822111e700                ssh select
 916      1 3   0        80   fffffe821f081ba0                 sh wait
 915      1 3   2         0   fffffe8223245340          conserver tstile
 274      1 3   2        80   fffffe821fee3b60                ssh select
 1041     1 3   5        80   fffffe8222d56580                 sh wait
 400      1 3   0        80   fffffe8221c54640                ssh select
 527      1 3   0        80   fffffe8223245760                 sh wait
 782      1 3   4        80   fffffe8223245b80                ssh select
 1165     1 3   1        80   fffffe822200f5c0                 sh wait
 908      1 3   3        80   fffffe822200f1a0                ssh select
 267      1 3   0        80   fffffe821fee3320                 sh wait
 264      1 3   3        80   fffffe821e590300                ssh select
 903      1 3   1        80   fffffe821e590720                 sh wait
 774      1 3   6         0   fffffe821e590b40          conserver tstile
 1155     1 3   6        80   fffffe822111eb20                ssh select
 386      1 3   6        80   fffffe8221c54220                 sh wait
 385      1 3   4        80   fffffe822047b2c0                ssh select
 384      1 3   0        80   fffffe822a01e4a0                 sh wait
 146      1 3   0        80   fffffe822047b6e0                ssh select
 959      1 3   7        80   fffffe822047bb00                 sh wait
 958      1 3   7        80   fffffe8220e372a0                ssh select
 445      1 3   7        80   fffffe8220e376c0                 sh wait
 700      1 3   1        80   fffffe8220e37ae0                ssh select
 635      1 3   0        80   fffffe8220f27280                 sh wait
 698      1 3   7        80   fffffe8220f276a0                ssh select
 761      1 3   0        80   fffffe8220f27ac0                 sh wait
 696      1 3   2        80   fffffe822195e260                ssh select
 951      1 3   0        80   fffffe822195e680                 sh wait
 566      1 3   4        80   fffffe822195eaa0                ssh select
 1141     1 3   0        80   fffffe8220529240                 sh wait
 756      1 3   5        80   fffffe8220529660                ssh select
 691      1 3   0        80   fffffe8220529a80                 sh wait
 368      1 3   1        80   fffffe8221c54a60                ssh select
 559      1 3   0        80   fffffe821f738200                 sh wait
 814      1 3   4        80   fffffe821f738620                ssh select
 685      1 3   1        80   fffffe821f738a40                 sh wait
 620      1 3   5        80   fffffe8221b2f1e0                ssh select
 171      1 3   0        80   fffffe8221b2f600                 sh wait
 362      1 3   3        80   fffffe8221b2fa20                ssh select
 361      1 3   0        80   fffffe82216931c0                 sh wait
 360      1 3   2        80   fffffe822a70d6a0                ssh select
 679      1 3   6         0   fffffe82216935e0          conserver tstile
 550      1 3   1        80   fffffe8221693a00                 sh wait
 1123     1 3   4        80   fffffe822200f9e0                ssh select
 354      1 3   0        80   fffffe8222a36180                 sh wait
 161      1 3   5        80   fffffe8222c13480                ssh select
 160      1 3   6        80   fffffe822a01e080                 sh wait
 799      1 3   0        80   fffffe8222a365a0                ssh select
 990      1 3   2        80   fffffe8222c138a0                 sh wait
 285      1 3   7        80   fffffe822d8565a0                ssh select
 668      1 3   1        80   fffffe8227206740                 sh wait
 539      1 3   4        80   fffffe8222a369c0                ssh select
 538      1 3   2        80   fffffe8222d56160                 sh wait
 919      1 3   2        80   fffffe822d96b560                ssh select
 406      1 3   1        80   fffffe822da69540                 sh wait
 917      1 3   1        80   fffffe8222d569a0                ssh select
 596      1 3   5        80   fffffe8222213140                 sh wait
 723      1 3   5        80   fffffe8222213560                ssh select
 210      1 3   0        80   fffffe8222213980                 sh wait
 1105     1 3   4        80   fffffe8220d58120                ssh select
 208      1 3   0        80   fffffe8220d58540                 sh wait
 399      1 3   6        80   fffffe822cddba40                ssh select
 846      1 3   6        80   fffffe8220d58960                 sh wait
 781      1 3   7        80   fffffe82232e6100                ssh select
 524      1 3   0        80   fffffe82232e6520                 sh wait
 139      1 3   4        80   fffffe82232e6940                ssh select
 138      1 3   0        80   fffffe8223fe90e0                 sh wait
 137      1 3   1        80   fffffe8223fe9500                ssh select
 136      1 3   1        80   fffffe8223fe9920                 sh wait
 519      1 3   7        80   fffffe822dd92500                ssh select
 646      1 3   2        80   fffffe82235c90c0                 sh wait
 709      1 3   5        80   fffffe82235c94e0                ssh select
 708      1 3   6         0   fffffe82235c9900          conserver tstile
 963      1 3   0        80   fffffe82231dc0a0                 sh wait
 194      1 3   2        80   fffffe82231dc4c0                ssh select
 193      1 3   0        80   fffffe82231dc8e0                 sh wait
 73       1 3   6        80   fffffe822a01e8c0                ssh select
 72       1 3   5        80   fffffe8222c13060                 sh wait
 604      1 3   0        80   fffffe8224f36040                ssh select
 895      1 3   5        80   fffffe8224f36460                 sh wait
 636      1 3   3        80   fffffe8224f36880                ssh select
 381      1 3   5        80   fffffe8227747700                 sh wait
 728      1 3   3        80   fffffe822d96b980                ssh select
 634      1 3   5        80   fffffe822d6b49e0                 sh wait
 564      1 3   6        80   fffffe8223956020                ssh select
 949      1 3   1        80   fffffe8223956440                 sh wait
 517      1 3   0        80   fffffe8223956860                ssh select
 178      1 3   0        80   fffffe8223182000                 sh wait
 913      1 3   2        80   fffffe8223182420                ssh select
 176      1 3   0        80   fffffe822d856180                 sh wait
 495      1 3   3        80   fffffe822d8569c0                ssh select
 717      1 3   3        80   fffffe82277472e0                 sh wait
 169      1 3   5        80   fffffe822da69960                ssh select
 170      1 3   0        80   fffffe8223182840                 sh wait
 168      1 3   1        80   fffffe822d96b140                ssh select
 615      1 3   0        80   fffffe82234ca360                 sh wait
 582      1 3   1        80   fffffe82234ca780                ssh select
 516      1 3   6        80   fffffe82234caba0                 sh wait
 931      1 3   4        80   fffffe822d96f580          conserver select
 162      1 3   1         0   fffffe822d96f9a0          conserver tstile
 853      1 3   0        80   fffffe8227206b60          conserver select
 894      1 3   0        80   fffffe8225286760          conserver select
 475      1 3   4        80   fffffe8225286b80               cron nanoslp
 569      1 2   2         0   fffffe8227206320          python2.7
 460      1 3   1        80   fffffe822bd2e660              inetd kqueue
 97       1 3   5        80   fffffe8227747b20              nginx kqueue
 96       1 3   5        80   fffffe8228a6a2c0              nginx pause
 531      1 2   4         0   fffffe822d3cf1e0               qmgr
 735  :   1 7   1         0   fffffe822d043a00             master
 486      1 3   0        80   fffffe822b87a260               sshd select
 473      1 3   3         0   fffffe822a70dac0           tac_plus tstile
 428      1 3   5        80   fffffe822bd2ea80             powerd kqueue
 420      1 3   3   1000000   fffffe810f2da720               ntpd tstile
 342      4 3   3        80   fffffe822b87aaa0              dhcpd parked
 342      3 3   3        80   fffffe822bd2e240              dhcpd kqueue
 342      2 3   0        80   fffffe822bcd5a60              dhcpd parked
 342      1 3   2        80   fffffe822bcd5640              dhcpd parked
 221      1 2   0         0   fffffe822d0435e0            syslogd
 1        1 3   2        80   fffffe810f4fa900               init wait
 0      123 3   6       200   fffffe822d96f160            npfgc-0 xchicv
 0      122 3   5       200   fffffe822d6b45c0              ipmi0 ipmi0
 0      121 3   7       200   fffffe822dd920e0            physiod physiod
 0      120 3   3       200   fffffe822dd08100           aiodoned aiodoned
 0    : 119 7   5       200   fffffe822dd08520            ioflush
 0      118 3   0       200   fffffe822dd08940           pgdaemon pgdaemon
 0      115 3   0       280   fffffe822dd92920              spkr1 bellcv
 0      114 3   4       280   fffffe810f4fa0c0           audiomix play
 0      113 3   4       280   fffffe810ed3b2a0           audiorec record
 0      112 3   0       200   fffffe810f2f1b60          atapibus0 sccomp
 0      110 2   2       200   fffffe810f20d2c0               usb3
 0      109 2   6       200   fffffe810f2da300               usb0
 0      108 2   2       200   fffffe810f2cb2e0               usb2
 0      107 2   2       200   fffffe810f2dab40               usb5
 0      106 3   0       200   fffffe810f2cbb20               usb1 usbevt
 0      105 2   2       200   fffffe810f4fa4e0               usb4
 0      104 3   0       200   fffffe810f4b30a0            rt_free rt_free
 0      103 3   0       200   fffffe810f4b34c0              unpgc unpgc
 0      102 3   0       200   fffffe810f4b38e0    key_timehandler key_timehandler

 0      101 3   7       200   fffffe810f474080    icmp6_wqinput/7 icmp6_wqinput
 0      100 3   6       200   fffffe810f4744a0    icmp6_wqinput/6 icmp6_wqinput
 0       99 3   5       200   fffffe810f4748c0    icmp6_wqinput/5 icmp6_wqinput
 0       98 3   4       200   fffffe810f4b5060    icmp6_wqinput/4 icmp6_wqinput
 0       97 3   3       200   fffffe810f4b5480    icmp6_wqinput/3 icmp6_wqinput
 0       96 3   2       200   fffffe810f4b58a0    icmp6_wqinput/2 icmp6_wqinput
 0       95 3   1       200   fffffe810f38e040    icmp6_wqinput/1 icmp6_wqinput
 0       94 3   0       200   fffffe810f38e460    icmp6_wqinput/0 icmp6_wqinput
 0    :  93 7   7       200   fffffe810f38e880          nd6_timer
 0       92 3   7       200   fffffe810f35f020     icmp_wqinput/7 icmp_wqinput
 0       91 3   6       200   fffffe810f35f440     icmp_wqinput/6 icmp_wqinput
 0       90 3   5       200   fffffe810f35f860     icmp_wqinput/5 icmp_wqinput
 0       89 3   4       200   fffffe810f378000     icmp_wqinput/4 icmp_wqinput
 0       88 3   3       200   fffffe810f378420     icmp_wqinput/3 icmp_wqinput
 0       87 3   2       200   fffffe810f378840     icmp_wqinput/2 icmp_wqinput
 0       86 3   1       200   fffffe810f331360     icmp_wqinput/1 icmp_wqinput
 0       85 3   0       200   fffffe810f331780     icmp_wqinput/0 icmp_wqinput
 0       84 3   4       200   fffffe810f2cb700           rt_timer rt_timer
 0       83 3   0       200   fffffe810f331ba0        vmem_rehash vmem_rehash
 0       82 3   7       200   fffffe810f2f0340          coretemp3 coretemp3
 0       81 3   6       200   fffffe810f2f0760          coretemp2 coretemp2
 0       80 3   7       200   fffffe810f2f0b80          coretemp1 coretemp1
 0       79 3   1       200   fffffe810f2f1320          coretemp0 coretemp0
 0       78 3   1       200   fffffe810f2f1740               mfi0 mfi0
 0       69 3   5       200   fffffe810f20d6e0            atabus1 atath
 0       68 3   5       200   fffffe810f20db00            atabus0 atath
 0       66 3   0       200   fffffe810ed3b6c0           scsibus0 sccomp
 0       65 3   0       200   fffffe810ed3bae0         usbtask-dr usbtsk
 0       64 3   0       200   fffffe810ecaa280         usbtask-hc usbtsk
 0       63 3   0       200   fffffe810ecaa6a0               bnx3 bnx3
 0       62 3   2       200   fffffe810ecaaac0               bnx2 bnx2
 0       61 3   1       200   fffffe810eb61260               bnx1 bnx1
 0       60 3   7       200   fffffe810eb61680               bnx0 bnx0
 0       59 3   0       200   fffffe810eb61aa0               ipmi ipmipoll
 0       58 3   7       200   fffffe810ea7c240            xcall/7 xcall
 0       57 1   7       200   fffffe810ea7c660          softser/7
 0       56 1   7       200   fffffe810ea7ca80          softclk/7
 0       55 1   7       200   fffffe810ea5d220          softbio/7
 0       54 1   7       200   fffffe810ea5d640          softnet/7
 0       53 1   7       201   fffffe810ea5da60             idle/7
 0       52 3   6       200   fffffe810ea36200            xcall/6 xcall
 0       51 1   6       200   fffffe810ea36620          softser/6
 0       50 1   6       200   fffffe810ea36a40          softclk/6
 0       49 1   6       200   fffffe810ea0f1e0          softbio/6
 0       48 1   6       200   fffffe810ea0f600          softnet/6
 0       47 1   6       201   fffffe810ea0fa20             idle/6
 0       46 3   5       200   fffffe810e9e01c0            xcall/5 xcall
 0       45 1   5       200   fffffe810e9e05e0          softser/5
 0       44 1   5       200   fffffe810e9e0a00          softclk/5
 0       43 1   5       200   fffffe810e9d11a0          softbio/5
 0       42 1   5       200   fffffe810e9d15c0          softnet/5
 0       41 1   5       201   fffffe810e9d19e0             idle/5
 0       40 3   4       200   fffffe810e9a2180            xcall/4 xcall
 0       39 1   4       200   fffffe810e9a25a0          softser/4
 0       38 3   4       200   fffffe810e9a29c0          softclk/4 tstile
 0       37 1   4       200   fffffe810e973160          softbio/4
 0       36 1   4       200   fffffe810e973580          softnet/4
 0       35 1   4       201   fffffe810e9739a0             idle/4
 0       34 3   3       200   fffffe810e94c140            xcall/3 xcall
 0       33 1   3       200   fffffe810e94c560          softser/3
 0       32 1   3       200   fffffe810e94c980          softclk/3
 0       31 1   3       200   fffffe810e93d120          softbio/3
 0       30 1   3       200   fffffe810e93d540          softnet/3
 0       29 1   3       201   fffffe810e93d960             idle/3
 0       28 3   2       200   fffffe810e906100            xcall/2 xcall
 0       27 1   2       200   fffffe810e906520          softser/2
 0       26 1   2       200   fffffe810e906940          softclk/2
 0       25 1   2       200   fffffe810e8ef0e0          softbio/2
 0       24 1   2       200   fffffe810e8ef500          softnet/2
 0       23 1   2       201   fffffe810e8ef920             idle/2
 0       22 3   1       200   fffffe810e8c50c0            xcall/1 xcall
 0       21 1   1       200   fffffe810e8c54e0          softser/1
 0       20 1   1       200   fffffe810e8c5900          softclk/1
 0       19 1   1       200   fffffe810e8b10a0          softbio/1
 0       18 1   1       200   fffffe810e8b14c0          softnet/1
 0       17 1   1       201   fffffe810e8b18e0             idle/1
 0       16 3   1       200   fffffe822de92080           lnxsyswq lnxsyswq
 0       15 3   0       200   fffffe822de924a0             sysmon smtaskq
 0       14 3   0       200   fffffe822de928c0         pmfsuspend pmfsuspend
 0       13 3   5       200   fffffe822e2b1060           pmfevent pmfevent
 0       12 3   0       200   fffffe822e2b1480         sopendfree sopendfr
 0       11 3   7       200   fffffe822e2b18a0           nfssilly nfssilly
 0       10 2   6       200   fffffe822f6d0040            cachegc
 0        9 3   5       200   fffffe822f6d0460             vdrain vdrain
 0        8 3   0       200   fffffe822f6d0880          modunload mod_unld
 0        7 3   0       200   fffffe822f6eb020            xcall/0 xcall
 0    :   6 7   0       200   fffffe822f6eb440          softser/0
 0    :   5 7   0       200   fffffe822f6eb860          softclk/0
 0        4 1   0       200   fffffe822f707000          softbio/0
 0        3 1   0       200   fffffe822f707420          softnet/0
 0        2 1   0       201   fffffe822f707840             idle/0
 0        1 2   2       200   ffffffff81481b20            swapper
 db{0}: trace/t 0t6776
 trace: pid 6776 lid 2 at 0xffff80009f5d9c00
 sleepq_block() at netbsd:sleepq_block+0x97
 sel_do_scan() at netbsd:sel_do_scan+0x4d1
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7426b363e10a:
 db{0}: trace/t 0t5530
 trace: pid 5530 lid 2 at 0xffff80009f752c00
 sleepq_block() at netbsd:sleepq_block+0x97
 sel_do_scan() at netbsd:sel_do_scan+0x4d1
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 7237e283e10a:
 db{0}: trace/t 0t22126
 trace: pid 22126 lid 2 at 0xffff80009f826c00
 VOP_POLL() at netbsd:VOP_POLL+0x7a
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 789c3e03e10a:
 db{0}: trace/a fffffe822aac6700
 trace: pid 22126 lid 2 at 0xffff80009f826c00
 VOP_POLL() at netbsd:VOP_POLL+0x7a
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 789c3e03e10a:
 db{0}: trace/a fffffe81788f42a0
 trace: pid 22126 lid 1 at 0xffff80009f64bd80
 VOP_FCNTL() at netbsd:VOP_FCNTL+0x4b
 sys_fcntl() at netbsd:sys_fcntl+0x254
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 92) ---
 789c3e03e48a:
 db{0}: trace/t 0t22385
 trace: pid 22385 lid 1 at 0xffff80009ed64e70
 kevent1() at netbsd:kevent1+0x648
 sys___kevent50() at netbsd:sys___kevent50+0x33
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 435) ---
 72a5bf03e40a:
 db{0}: trace/t 0t735
 trace: pid 735 lid 1 at 0xffff80009dea1bd0
 kqueue_register() at netbsd:kqueue_register+0x431
 kevent1() at netbsd:kevent1+0x150
 sys___kevent50() at netbsd:sys___kevent50+0x33
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 435) ---
 74de5623e40a:
 db{0}: trace/a fffffe822dd08520
 trace: pid 0 lid 119 at 0xffff80009dce4790
 scsipi_adapter_request() at netbsd:scsipi_adapter_request+0x52
 scsipi_run_queue() at netbsd:scsipi_run_queue+0xe2
 scsipi_execute_xs() at netbsd:scsipi_execute_xs+0x1bb
 sd_diskstart() at netbsd:sd_diskstart+0x25b
 dk_start() at netbsd:dk_start+0xde
 spec_strategy() at netbsd:spec_strategy+0x9a
 VOP_STRATEGY() at netbsd:VOP_STRATEGY+0x54
 genfs_do_io() at netbsd:genfs_do_io+0x1b4
 genfs_gop_write() at netbsd:genfs_gop_write+0x52
 genfs_do_putpages() at netbsd:genfs_do_putpages+0xb9c
 VOP_PUTPAGES() at netbsd:VOP_PUTPAGES+0x3a
 ffs_full_fsync() at netbsd:ffs_full_fsync+0x116
 ffs_fsync() at netbsd:ffs_fsync+0x3a
 VOP_FSYNC() at netbsd:VOP_FSYNC+0x3e
 sched_sync() at netbsd:sched_sync+0x198
 db{0}: trace/a fffffe810f38e880
 trace: pid 0 lid 93 at 0xffff80009dbc5ea0
 nd6_timer_work() at netbsd:nd6_timer_work+0x49
 workqueue_worker() at netbsd:workqueue_worker+0xcd
 db{0}: trace/a fffffe822f6eb440
 trace: pid 0 lid 6 at 0xffff80008f69bc30
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 uhci_intr() at netbsd:uhci_intr+0x19
 intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
 Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}: trace/a fffffe822f6eb860
 trace: pid 0 lid 5 at 0xffff80009edadd60
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 ffff80009edadf00:
 db{0}: machine cpu 0
 using CPU 0
 db{0}: tra
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 uhci_intr() at netbsd:uhci_intr+0x19
 intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
 Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}: machine cpu 1
 using CPU 1
 db{0}: tra
 x86_pause() at netbsd:x86_pause
 _kernel_lock() at netbsd:_kernel_lock+0x183
 kqueue_register() at netbsd:kqueue_register+0x431
 kevent1() at netbsd:kevent1+0x150
 sys___kevent50() at netbsd:sys___kevent50+0x33
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 435) ---
 74de5623e40a:
 db{0}: machine cpu 2
 using CPU 2
 db{0}: tra
 x86_pause() at netbsd:x86_pause+0x2
 VOP_FCNTL() at netbsd:VOP_FCNTL+0x4b
 sys_fcntl() at netbsd:sys_fcntl+0x254
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 92) ---
 789c3e03e48a:
 db{0}: machine cpu 3
 using CPU 3
 db{0}: tra
 x86_pause() at netbsd:x86_pause+0x2
 kevent1() at netbsd:kevent1+0x648
 sys___kevent50() at netbsd:sys___kevent50+0x33
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 435) ---
 72a5bf03e40a:
 db{0}: machine cpu 4
 using CPU 4
 db{0}: tra
 x86_pause() at netbsd:x86_pause+0x2
 VOP_FCNTL() at netbsd:VOP_FCNTL+0x4b
 sys_fcntl() at netbsd:sys_fcntl+0x254
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 92) ---
 7237e283e48a:
 db{0}: machine cpu 5
 using CPU 5
 db{0}: tra
 x86_pause() at netbsd:x86_pause+0x2
 scsipi_adapter_request() at netbsd:scsipi_adapter_request+0x52
 scsipi_run_queue() at netbsd:scsipi_run_queue+0xe2
 scsipi_execute_xs() at netbsd:scsipi_execute_xs+0x1bb
 sd_diskstart() at netbsd:sd_diskstart+0x25b
 dk_start() at netbsd:dk_start+0xde
 spec_strategy() at netbsd:spec_strategy+0x9a
 VOP_STRATEGY() at netbsd:VOP_STRATEGY+0x54
 genfs_do_io() at netbsd:genfs_do_io+0x1b4
 genfs_gop_write() at netbsd:genfs_gop_write+0x52
 genfs_do_putpages() at netbsd:genfs_do_putpages+0xb9c
 VOP_PUTPAGES() at netbsd:VOP_PUTPAGES+0x3a
 ffs_full_fsync() at netbsd:ffs_full_fsync+0x116
 ffs_fsync() at netbsd:ffs_fsync+0x3a
 VOP_FSYNC() at netbsd:VOP_FSYNC+0x3e
 sched_sync() at netbsd:sched_sync+0x198
 db{0}: machine cpu 6
 using CPU 6
 db{0}: tra
 _kernel_lock() at netbsd:_kernel_lock+0x17e
 VOP_POLL() at netbsd:VOP_POLL+0x7a
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 789c3e03e10a:
 db{0}: machine cpu 7
 using CPU 7
 db{0}: tra
 _kernel_lock() at netbsd:_kernel_lock+0x1a1
 nd6_timer_work() at netbsd:nd6_timer_work+0x49
 workqueue_worker() at netbsd:workqueue_worker+0xcd
 db{0}: trace/a fffffe82273c5b80
 trace: pid 19899 lid 1 at 0xffff80009f20de10
 sleepq_block() at netbsd:sleepq_block+0x106
 lwp_park() at netbsd:lwp_park+0x107
 sys____lwp_park60() at netbsd:sys____lwp_park60+0x52
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 478) ---
 7b611f09998a:
 db{0}: trace/t 0t953
 trace: pid 953 lid 1 at 0xffff80009ed78b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 776936e3e10a:
 db{0}: trace/t 0t306
 trace: pid 306 lid 1 at 0xffff80009ed34b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 776936e3e10a:
 db{0}: trace/t 0t1047
 trace: pid 1047 lid 1 at 0xffff80009ec09b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 776936e3e10a:
 db{0}: trace/t 0t534
 trace: pid 534 lid 1 at 0xffff80009eb6eb00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 776936e3e10a:
 db{0}: trace/t 0t915
 trace: pid 915 lid 1 at 0xffff80009ebadb00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 776936e3e10a:
 db{0}: trace/t 0t774
 trace: pid 774 lid 1 at 0xffff80009eb73b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 776936e3e10a:
 db{0}: trace/t 0t679
 trace: pid 679 lid 1 at 0xffff80009ea98b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 776936e3e10a:
 db{0}: trace/t 0t708
 trace: pid 708 lid 1 at 0xffff80009e9c4b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 776936e3e10a:
 db{0}: trace/t 0t162
 trace: pid 162 lid 1 at 0xffff80009dd94b00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x324
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 417) ---
 776936e3e10a:
 db{0}: trace/t 0t473
 trace: pid 473 lid 1 at 0xffff80009e179b20
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sopoll() at netbsd:sopoll+0x1c
 sel_do_scan() at netbsd:sel_do_scan+0x176
 pollcommon.part.3() at netbsd:pollcommon.part.3+0x8d
 sys_poll() at netbsd:sys_poll+0x9b
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 209) ---
 720c50e3e28a:
 db{0}: trace/t 0t420
 trace: pid 420 lid 1 at 0xffff80009db6eae0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 sosend() at netbsd:sosend+0x48
 do_sys_sendmsg_so() at netbsd:do_sys_sendmsg_so+0x272
 do_sys_sendmsg() at netbsd:do_sys_sendmsg+0x85
 sys_sendto() at netbsd:sys_sendto+0x5c
 syscall() at netbsd:syscall+0x1ed
 --- syscall (number 133) ---
 7334786eea7a:
 db{0}: trace/a fffffe810e9a29c0
 trace: pid 0 lid 38 at 0xffff80008f771e00
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x3e5
 mutex_enter() at netbsd:mutex_enter+0x36d
 tcp_timer_rexmt() at netbsd:tcp_timer_rexmt+0x27
 callout_softclock() at netbsd:callout_softclock+0x201
 softint_dispatch() at netbsd:softint_dispatch+0xd3
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f771ff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}: call cpu_reset

From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Wed, 16 May 2018 11:44:47 +0200 (CEST)

 Hi,

 we provoked another wedge, and captured a kernel core dump from
 the wedging machine.  The kernel core dump and NetBSD images are
 available for looking at.


 using gdb and crash, here's a brief summary of the "interesting"
 processes in the crash dump, and below that I include backtraces
 of all the waiting processes.

 There's lots of contention for fstrans_lock.

 Offhand I don't see a deadlock which might explain the observed
 behaviour (goes totally "deaf" on the network, i.e. doesn't even
 respond to ping).

 It doesn't look like gdb can trace through interrupt frames (?),
 looking at proc 788 all I get is:

 (gdb) kvm proc 0xfffffe8220b8e360
 0xffffffff8021cfe0 in softintr_ret ()
 (gdb) where
 #0  0xffffffff8021cfe0 in softintr_ret ()
 #1  0x0000000000000000 in ?? ()
 (gdb) =


 Crash manages to do this one, though, apparently (see below).

 Furthermore, using the various gdb scripts in
 /usr/src/sys/gdbscripts/ I can look at some of the locks.

 It is quite possible that up'ing the interface in question causes
 lots of activity for opening pty pairs, and that the root cause
 of the issue is there rather than related to networking in itself(?)

 Further hints?

 Regards,

 - H=E5vard

 ------------------------------

 PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 7052     1 3   1   8020000   fffffe8220c58540               cron tstile=

   Wants fstrans_lock

 9187     1 3   6   8020000   fffffe821faa60c0             expect xchicv=

   Holds fstrans_lock, in pserialize_perform, waits on condition variabl=
 e
     after doing xc_broadcast(XC_HIGHPRI, nullop)
   Doing (roughly) pty_grant_slave -> genfs_revoke -> vfs_suspend -> =

     fstrans_setstate -> pserialize_perform -> xc_wait -> cv_wait

 3794     1 3   7   8020000   fffffe82045da540                ssh tstile=

   Wants fstrans_lock

 2853     2 3   6   8020000   fffffe8216d35140             expect tstile=

   Wants fstrans_lock

 2853     1 3   7   8020000   fffffe82214bf2c0             expect fdclos=
 e
   Doing cv_wait(&ff->ff_closing, &fdp->fd_lock);

 8327     2 3   3   8020000   fffffe8225709000             expect tstile=

   Wants fstrans_lock

 8327     1 3   1   8020000   fffffe820df34480             expect fdclos=
 e
   Doing cv_wait(&ff->ff_closing, &fdp->fd_lock);

 5687     2 3   3   8020000   fffffe8207fd0360             expect tstile=

   Wants fstrans_lock

 5687     1 3   2   8020000   fffffe822346f080             expect fdclos=
 e
   Doing cv_wait(&ff->ff_closing, &fdp->fd_lock);

 9671     2 3   5   8020000   fffffe821d987a40             expect tstile=

   Wants fstrans_lock

 9671     1 3   4   8020000   fffffe821e90c240             expect fdclos=
 e
   Doing cv_wait(&ff->ff_closing, &fdp->fd_lock);

 704      1 3   6   8020000   fffffe821d362980          conserver tstile=

   Wants rw_enter(&vip->vi_lock, op);, lock held by 788 apparently

 564      1 3   1   8020000   fffffe8229b924c0          conserver tstile=

   Wants fstrans_lock

 788  >   1 7   0   8020000   fffffe8220b8e360          conserver
   Doing softintr_ret(), gdb doesn't trace further (interrupt frame?)
   Crash, though, says:

 crash> trace/a 0xfffffe8220b8e360
 trace: pid 788 lid 1 at 0xffff80009ebfdc40
 ufs_post_write_update.isra.0() at ufs_post_write_update.isra.0+0x95
 ffs_write() at ffs_write+0x56f
 VOP_WRITE() at VOP_WRITE+0x37
 vn_write() at vn_write+0xec
 dofilewrite() at dofilewrite+0x97
 sys_write() at sys_write+0x5f
 syscall() at syscall+0x1ed
 --- syscall (number 4) ---
 764fb503e02a:
 crash> =


 275      1 3   5   8020000   fffffe8220b8e780          conserver tstile=

   Wants rw_enter(&vip->vi_lock, op);, lock held by 788 apparently

 878      1 3   2   8020000   fffffe8221e57600          conserver tstile=

   Wants fstrans_lock

 136      1 3   3   8020000   fffffe8222569900          conserver tstile=

   Wants fstrans_lock

 773      1 3   1   8020000   fffffe8225709840          conserver tstile=

   Wants fstrans_lock

 ------------------------------

 crash> ps
 PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 7052     1 3   1   8020000   fffffe8220c58540               cron tstile=

 8587     1 3   1   8020000   fffffe821d362560               cron wait
 9187     1 3   6   8020000   fffffe821faa60c0             expect xchicv=

 5064     1 2   6   8020000   fffffe8219f93540                 sh
 7813     1 2   0   8020000   fffffe8220ecf860               perl
 7354     1 2   0   8020000   fffffe8207aba240                 sh
 3794     1 3   7   8020000   fffffe82045da540                ssh tstile=

 2853     2 3   6   8020000   fffffe8216d35140             expect tstile=

 2853     1 3   7   8020000   fffffe82214bf2c0             expect fdclos=
 e
 6097     1 2   0   8020000   fffffe8221165720                 sh
 5738     1 2   0   8020000   fffffe821e90ca80               perl
 4461     1 2   0   8020000   fffffe8207a90aa0                 sh
 4310     1 2   3   8020000   fffffe8220c2c5a0                ssh
 8327     2 3   3   8020000   fffffe8225709000             expect tstile=

 8327     1 3   1   8020000   fffffe820df34480             expect fdclos=
 e
 6086     1 2   5   8020000   fffffe821d2d9a60                 sh
 3214     1 2   0   8020000   fffffe821f5d8360               perl
 5921     1 2   0   8020000   fffffe8207a90260                 sh
 2890     1 2   2   8020000   fffffe8228bacae0                ssh
 3772     1 2   0   8020000   fffffe821a27f420                 sh
 4425     1 2   0   8020000   fffffe821f5d8780                ssh
 5793     1 2   0   8020000   fffffe821d342340                 sh
 4833     1 2   0   8020000   fffffe82207f5460                ssh
 7238     1 2   0   8020000   fffffe822123b640                 sh
 6312     1 2   4   8020000   fffffe8222a9b500                ssh
 7003     1 2   0   8020000   fffffe822160e700                 sh
 6900     1 2   0   8020000   fffffe8221d82140                ssh
 4139     1 2   0   8020000   fffffe822346f8c0                 sh
 3765     1 2   0   8020000   fffffe821f94f060                ssh
 5386     1 2   1   8020000   fffffe8222a7a8c0                 sh
 4633     1 2   0   8020000   fffffe821f8969c0                ssh
 9691     1 2   0   8020000   fffffe821928d180                 sh
 6528     1 2   7   8020000   fffffe8207f4b460                ssh
 5816     1 2   0   8020000   fffffe8207f4b880                 sh
 6285     1 2   0   8020000   fffffe821e1e28c0                ssh
 8366     1 2   0   8020000   fffffe820d267520                 sh
 8041     1 2   0   8020000   fffffe821f94f8a0                ssh
 3407     1 2   0   8020000   fffffe82045da960                 sh
 2815     1 2   0   8020000   fffffe8225709420                ssh
 5687     2 3   3   8020000   fffffe8207fd0360             expect tstile=

 5687     1 3   2   8020000   fffffe822346f080             expect fdclos=
 e
 2928     1 2   0   8020000   fffffe8228bac2a0                 sh
 5127     1 2   0   8020000   fffffe821b498860               perl
 5658     1 2   0   8020000   fffffe820ea5eac0                 sh
 6746     1 2   0   8020000   fffffe821d443520                ssh
 9671     2 3   5   8020000   fffffe821d987a40             expect tstile=

 9671     1 3   4   8020000   fffffe821e90c240             expect fdclos=
 e
 9105     1 2   0   8020000   fffffe8226ea8b40                 sh
 3990     1 2   0   8020000   fffffe8227069700               perl
 4504     1 2   0   8020000   fffffe8207a90680                 sh
 9019     1 2   0   8020000   fffffe821f4c78e0                ssh
 2825     1 2   0   8020000   fffffe821f896180                 sh
 5779     1 2   0   8020000   fffffe821a27f840                ssh
 2846     1 2   0   8020000   fffffe822af57440                 sh
 3544     1 2   0   8020000   fffffe82216b1a40                ssh
 6513     1 2   0   8020000   fffffe822728b460                 sh
 5965     1 2   0   8020000   fffffe821faa6900                ssh
 6327     1 2   0   8020000   fffffe8221837a80                 sh
 2739     1 2   0   8020000   fffffe822708e740                ssh
 6361     1 2   0   8020000   fffffe82239772a0                 sh
 3534     1 2   0   8020000   fffffe821f94f480                ssh
 5988     1 2   0   8020000   fffffe8222a7a080                 sh
 7283     1 2   0   8020000   fffffe8222b99980                ssh
 6596     1 2   0   8020000   fffffe820ea5e6a0                 sh
 3343     1 2   0   8020000   fffffe8223049740                ssh
 7708     1 2   0   8020000   fffffe822da31140                 sh
 6381     1 2   0   8020000   fffffe8221d82980                ssh
 7056     1 2   0   8020000   fffffe821a542500                 sh
 4161     1 2   0   8020000   fffffe821c708100                ssh
 7144     1 2   0   8020000   fffffe8223049b60                 sh
 6276     1 2   0   8020000   fffffe821d2d9220                ssh
 4460     1 2   0   8020000   fffffe821f36a040                 sh
 6845     1 2   0   8020000   fffffe8222a9b0e0                ssh
 4669     1 2   0   8020000   fffffe821e1e24a0                 sh
 3652     1 2   0   8020000   fffffe821c708520                ssh
 5131     1 2   0   8020000   fffffe820df34060                 sh
 5704     1 2   0   8020000   fffffe8220ce6760                ssh
 6031     1 2   0   8020000   fffffe821ecaea00                 sh
 5875     1 2   0   8020000   fffffe82240071a0                ssh
 5464     1 2   0   8020000   fffffe8205110580                 sh
 2852     1 2   0   8020000   fffffe82207f5040                ssh
 3432     1 2   0   8020000   fffffe820dd5e6e0                 sh
 4917     1 2   0   8020000   fffffe8221317ac0                ssh
 2850     1 2   0   8020000   fffffe8207fd0780                 sh
 3399     1 2   0   8020000   fffffe822d92e9c0                ssh
 3497     1 2   0   8020000   fffffe821d987200                 sh
 3880     1 2   0   8020000   fffffe821d342b80                ssh
 4728     1 2   0   8020000   fffffe822188a5a0                 sh
 3904     1 2   0   8020000   fffffe820df348a0                ssh
 4847     1 2   0   8020000   fffffe8204072900                 sh
 3465     1 2   0   8020000   fffffe8207fd0ba0                ssh
 3881     1 2   0   8020000   fffffe821a27f000                 sh
 4000     1 2   0   8020000   fffffe821928d9c0                ssh
 4242     1 2   0   8020000   fffffe8221165300                 sh
 4074     1 2   0   8020000   fffffe82207f5880                ssh
 3902     1 2   0   8020000   fffffe8205110160                 sh
 3293     1 2   0   8020000   fffffe8224d918a0                ssh
 3228     1 2   0   8020000   fffffe821d362140                 sh
 4064     1 2   0   8020000   fffffe8224b0c760                ssh
 4762     1 2   0   8020000   fffffe822037f1a0                 sh
 2903     1 2   0   8020000   fffffe8221dab1a0                ssh
 2218     1 2   0   8020000   fffffe821fdf9b00                 sh
 2497     1 2   0   8020000   fffffe822728b880                ssh
 3080     1 2   0   8020000   fffffe8223563b40                 sh
 2573     1 2   0   8020000   fffffe8206fea440                ssh
 2635     1 2   0   8020000   fffffe822af57020                 sh
 2114     1 2   0   8020000   fffffe82220aa520                ssh
 4141     1 2   0   8020000   fffffe82240079e0                 sh
 4477     1 2   0   8020000   fffffe822160eb20                ssh
 3978     1 2   0   8020000   fffffe821f6f9b60                 sh
 3243     1 2   0   8020000   fffffe821f1b0680                ssh
 2652     1 2   0   8020000   fffffe821c49f5e0                 sh
 3804     1 2   0   8020000   fffffe821f37d9a0                ssh
 4348     1 2   0   8020000   fffffe822c1e7660                 sh
 4334     1 2   0   8020000   fffffe8220c2c9c0                ssh
 3781     1 2   0   8020000   fffffe82214bfb00                 sh
 3422     1 2   0   8020000   fffffe82216b1200                ssh
 3277     1 2   0   8020000   fffffe8221165b40                 sh
 3997     1 2   0   8020000   fffffe8206fea020                ssh
 3863     1 2   0   8020000   fffffe82239776c0                 sh
 2950     1 2   0   8020000   fffffe821f6f9320                par
 2644     1 2   0   8020000   fffffe822220f9a0                ssh
 2418     1 2   0   8020000   fffffe822253d1c0                 sh
 2616     1 2   0   8020000   fffffe821e1e2080                ssh
 2724     1 2   0   8020000   fffffe821b0306a0                 sh
 1968     1 2   0   8020000   fffffe8224d91060                 sh
 3042     1 2   0   8020000   fffffe8221dab5c0                 sh
 2259     1 2   0   8020000   fffffe821c4bb4e0                 sh
 2733     1 2   0   8020000   fffffe8223c97a40                 sh
 2608     1 2   0   8020000   fffffe821b498020               cron
 2402     1 2   0   8020000   fffffe822123ba60                ssh
 2346     1 2   0   8020000   fffffe821f6f9740                 sh
 3141     1 2   0   8020000   fffffe8221837240                ssh
 1970     1 2   0   8020000   fffffe821f4c70a0                 sh
 2649     1 2   0   8020000   fffffe821a5f28e0                ssh
 2129     1 2   0   8020000   fffffe8227c3cb00                 sh
 2384     1 2   0   8020000   fffffe8221afe480                ssh
 2971     1 2   0   8020000   fffffe82225690c0                 sh
 2070     1 2   0   8020000   fffffe82212562e0                ssh
 2098     1 2   0   8020000   fffffe82201c92a0                 sh
 2764     1 2   0   8020000   fffffe821bf22680                ssh
 3322     1 2   0   8020000   fffffe8221256b20                 sh
 2511     1 2   0   8020000   fffffe821f37d160                ssh
 2661     1 2   0   8020000   fffffe821ecae1c0                 sh
 3491     1 2   0   8020000   fffffe822220f160                ssh
 3195     1 2   0   8020000   fffffe822188a9c0                 sh
 3305     1 2   0   8020000   fffffe821b800600                ssh
 2197     1 2   0   8020000   fffffe821d2d9640                 sh
 3011     1 2   0   8020000   fffffe82214bf6e0                ssh
 3056     1 2   0   8020000   fffffe82288f2ac0                 sh
 2350     1 2   0   8020000   fffffe8219f93960                ssh
 2297     1 2   0   8020000   fffffe8221e571e0                 sh
 2638     1 2   0   8020000   fffffe821a5f20a0                ssh
 2169     1 2   0   8020000   fffffe821c4bb0c0                 sh
 2550     1 2   0   8020000   fffffe82213176a0                ssh
 2225     1 2   0   8020000   fffffe82220aa100                 sh
 2677     1 2   0   8020000   fffffe821e90c660                ssh
 3183     1 2   0   8020000   fffffe8229af4260                 sh
 2214     1 2   0   8020000   fffffe821f232000                ssh
 3259     1 2   0   8020000   fffffe8223563720                 sh
 2092     1 2   0   8020000   fffffe822220f580                ssh
 2331     1 2   0   8020000   fffffe8223563300                 sh
 2750     1 2   0   8020000   fffffe8226888000                ssh
 1921     1 2   0   8020000   fffffe821bf22aa0                 sh
 2636     1 2   0   8020000   fffffe822d818160                ssh
 3223     1 2   0   8020000   fffffe821d443100                 sh
 2773     1 2   0   8020000   fffffe8226888420                ssh
 2150     1 2   0   8020000   fffffe8223049320                 sh
 2310     1 2   0   8020000   fffffe8222b99560                ssh
 1587     1 2   0   8020000   fffffe8220d110e0                 sh
 2901     1 2   0   8020000   fffffe822188a180                ssh
 2627     1 2   0   8020000   fffffe821c4bb900                 sh
 2060     1 2   0   8020000   fffffe821f36a460                ssh
 2599     1 2   0   8020000   fffffe821bf22260                 sh
 2422     1 2   0   8020000   fffffe821f8965a0                ssh
 2164     1 2   0   8020000   fffffe821d443940                 sh
 2022     1 2   0   8020000   fffffe821f556960                ssh
 2000     1 2   0   8020000   fffffe821a542920                 sh
 1593     1 2   0   8020000   fffffe822d481a40               bash
 1914     1 2   0   8020000   fffffe821f5d8ba0               sshd
 2167     1 2   0   8020000   fffffe822037f5c0               sshd
 1821     1 2   0   8020000   fffffe8220d11500               sshd
 1590     1 2   0   8020000   fffffe822dcff120               sshd
 1041     1 2   0   8020000   fffffe821c708940               bash
 594      1 2   0   8020000   fffffe822037f9e0               bash
 895      1 2   0   8020000   fffffe822708eb60               sshd
 1533     1 2   0   8020000   fffffe8227069b20               sshd
 704      1 3   6   8020000   fffffe821d362980          conserver tstile=

 564      1 3   1   8020000   fffffe8229b924c0          conserver tstile=

 789      1 2   0   8020000   fffffe821f232840          conserver
 788  >   1 7   0   8020000   fffffe8220b8e360          conserver
 275      1 3   5   8020000   fffffe8220b8e780          conserver tstile=

 772      1 2   0   8020000   fffffe8221256700          conserver
 878      1 3   2   8020000   fffffe8221e57600          conserver tstile=

 136      1 3   3   8020000   fffffe8222569900          conserver tstile=

 773      1 3   1   8020000   fffffe8225709840          conserver tstile=

 643      1 2   0   8020000   fffffe82251ce360          conserver
 162      1 2   0   8020000   fffffe822dd620e0              getty
 861      1 2   0   8020000   fffffe82251ce780          conserver
 575      1 2   0   8020000   fffffe82251ceba0          conserver
 892      1 2   0   8020000   fffffe8224b0c340               cron
 570      1 2   0   8020000   fffffe8224b0cb80          python2.7
 107      1 2   0   8020000   fffffe822d7de5c0              inetd
 740      1 2   0   8020000   fffffe822cf91a00              nginx
 97       1 2   0   8020000   fffffe822d4cd600              nginx
 677      1 2   0   8020000   fffffe822d481200               qmgr
 479      1 2   0   8020000   fffffe822c312a60             pickup
 760      1 2   0   8020000   fffffe822c1e7240             master
 436      1 2   0   8020000   fffffe822d818580               sshd
 474      1 2   0   8020000   fffffe822d92e5a0           tac_plus
 428      1 2   0   8020000   fffffe822cf915e0             powerd
 431      1 2   0   8020000   fffffe822d8189a0               ntpd
 342      4 2   4   8020000   fffffe822c1e7a80              dhcpd
 342      3 2   0   8020000   fffffe822c312220              dhcpd
 342      2 2   1   8020000   fffffe822c312640              dhcpd
 342      1 2   3   8020000   fffffe822d481620              dhcpd
 83   >   1 7   6   8020000   fffffe822cf911c0            syslogd
 1        1 2   0   8020000   fffffe810f4fa900               init
 0      123 3   7       200   fffffe822d4cd1e0            npfgc-0 xchicv=

 0      122 3   7       200   fffffe822d4cda20              ipmi0 ipmi0
 0      121 3   0       200   fffffe822dd10940            physiod physio=
 d
 0      120 3   3       200   fffffe822dcff960           aiodoned aiodon=
 ed
 0      119 3   5       200   fffffe822dd10100            ioflush biowai=
 t
 0      118 3   0       200   fffffe822dd10520           pgdaemon pgdaem=
 on
 0      115 3   0       280   fffffe822dd62500              spkr1 bellcv=

 0      114 3   3       280   fffffe822dd62920           audiomix play
 0      113 3   3       280   fffffe810ed3b2a0           audiorec record=

 0      112 3   0       200   fffffe810f4fa0c0          atapibus0 sccomp=

 0      110 3   6       200   fffffe810f2e9b60               usb1 usbevt=

 0      109 3   7       200   fffffe810f2c3b20               usb3 usbevt=

 0      108 3   6       200   fffffe810f20d2c0               usb5 usbevt=

 0      107 3   0       200   fffffe810f2da300               usb0 usbevt=

 0      106 3   0       200   fffffe810f2c32e0               usb4 usbevt=

 0      105 3   0       200   fffffe810f4fa4e0               usb2 usbevt=

 0      104 3   0       200   fffffe810f4a30a0            rt_free rt_fre=
 e
 0      103 3   0       200   fffffe810f4a34c0              unpgc unpgc
 0      102 3   7       200   fffffe810f4a38e0    key_timehandler key_ti=
 mehandler

 0      101 3   7       200   fffffe810f45c080    icmp6_wqinput/7 icmp6_=
 wqinput
 0      100 3   6       200   fffffe810f45c4a0    icmp6_wqinput/6 icmp6_=
 wqinput
 0       99 3   5       200   fffffe810f45c8c0    icmp6_wqinput/5 icmp6_=
 wqinput
 0       98 3   4       200   fffffe810f4b5060    icmp6_wqinput/4 icmp6_=
 wqinput
 0       97 3   3       200   fffffe810f4b5480    icmp6_wqinput/3 icmp6_=
 wqinput
 0       96 3   2       200   fffffe810f4b58a0    icmp6_wqinput/2 icmp6_=
 wqinput
 0       95 3   1       200   fffffe810f396040    icmp6_wqinput/1 icmp6_=
 wqinput
 0       94 3   0       200   fffffe810f396460    icmp6_wqinput/0 icmp6_=
 wqinput
 0       93 3   2       200   fffffe810f396880          nd6_timer nd6_ti=
 mer
 0       92 3   7       200   fffffe810f367020     icmp_wqinput/7 icmp_w=
 qinput
 0       91 3   6       200   fffffe810f367440     icmp_wqinput/6 icmp_w=
 qinput
 0       90 3   5       200   fffffe810f367860     icmp_wqinput/5 icmp_w=
 qinput
 0       89 3   4       200   fffffe810f378000     icmp_wqinput/4 icmp_w=
 qinput
 0       88 3   3       200   fffffe810f378420     icmp_wqinput/3 icmp_w=
 qinput
 0       87 3   2       200   fffffe810f378840     icmp_wqinput/2 icmp_w=
 qinput
 0       86 3   1       200   fffffe810f329360     icmp_wqinput/1 icmp_w=
 qinput
 0       85 3   0       200   fffffe810f329780     icmp_wqinput/0 icmp_w=
 qinput
 0       84 3   0       200   fffffe810f2da720           rt_timer rt_tim=
 er
 0       83 3   7       200   fffffe810f329ba0        vmem_rehash vmem_r=
 ehash
 0       82 3   6       200   fffffe810f2f8340          coretemp3 corete=
 mp3
 0       81 3   7       200   fffffe810f2f8760          coretemp2 corete=
 mp2
 0       80 3   2       200   fffffe810f2f8b80          coretemp1 corete=
 mp1
 0       79 3   3       200   fffffe810f2e9320          coretemp0 corete=
 mp0
 0       78 3   3       200   fffffe810f2e9740               mfi0 mfi0
 0       69 3   5       200   fffffe810f20d6e0            atabus1 atath
 0       68 3   4       200   fffffe810f20db00            atabus0 atath
 0       66 3   0       200   fffffe810ed3b6c0           scsibus0 sccomp=

 0       65 3   0       200   fffffe810ed3bae0         usbtask-dr usbtsk=

 0       64 3   0       200   fffffe810ecaa280         usbtask-hc usbtsk=

 0       63 3   0       200   fffffe810ecaa6a0               bnx3 bnx3
 0       62 3   0       200   fffffe810ecaaac0               bnx2 bnx2
 0       61 3   0       200   fffffe810eb61260               bnx1 bnx1
 0       60 3   7       200   fffffe810eb61680               bnx0 bnx0
 0    >  59 7   2       200   fffffe810eb61aa0               ipmi
 0       58 3   7       200   fffffe810ea7c240            xcall/7 xcall
 0       57 1   7       200   fffffe810ea7c660          softser/7
 0       56 1   7       200   fffffe810ea7ca80          softclk/7
 0       55 1   7       200   fffffe810ea5d220          softbio/7
 0       54 1   7       200   fffffe810ea5d640          softnet/7
 0    >  53 7   7       201   fffffe810ea5da60             idle/7
 0       52 3   6       200   fffffe810ea36200            xcall/6 xcall
 0       51 1   6       200   fffffe810ea36620          softser/6
 0       50 1   6       200   fffffe810ea36a40          softclk/6
 0       49 1   6       200   fffffe810ea0f1e0          softbio/6
 0       48 1   6       200   fffffe810ea0f600          softnet/6
 0       47 1   6       201   fffffe810ea0fa20             idle/6
 0       46 3   5       200   fffffe810e9e01c0            xcall/5 xcall
 0       45 1   5       200   fffffe810e9e05e0          softser/5
 0       44 1   5       200   fffffe810e9e0a00          softclk/5
 0       43 1   5       200   fffffe810e9d11a0          softbio/5
 0       42 1   5       200   fffffe810e9d15c0          softnet/5
 0    >  41 7   5       201   fffffe810e9d19e0             idle/5
 0       40 3   4       200   fffffe810e9a2180            xcall/4 xcall
 0       39 1   4       200   fffffe810e9a25a0          softser/4
 0       38 1   4       200   fffffe810e9a29c0          softclk/4
 0       37 1   4       200   fffffe810e973160          softbio/4
 0       36 1   4       200   fffffe810e973580          softnet/4
 0    >  35 7   4       201   fffffe810e9739a0             idle/4
 0       34 3   3       200   fffffe810e94c140            xcall/3 xcall
 0       33 1   3       200   fffffe810e94c560          softser/3
 0       32 1   3       200   fffffe810e94c980          softclk/3
 0       31 1   3       200   fffffe810e93d120          softbio/3
 0       30 1   3       200   fffffe810e93d540          softnet/3
 0    >  29 7   3       201   fffffe810e93d960             idle/3
 0       28 3   2       200   fffffe810e906100            xcall/2 xcall
 0       27 1   2       200   fffffe810e906520          softser/2
 0       26 1   2       200   fffffe810e906940          softclk/2
 0       25 1   2       200   fffffe810e8ef0e0          softbio/2
 0       24 1   2       200   fffffe810e8ef500          softnet/2
 0       23 1   2       201   fffffe810e8ef920             idle/2
 0       22 3   1       200   fffffe810e8c50c0            xcall/1 xcall
 0       21 1   1       200   fffffe810e8c54e0          softser/1
 0       20 1   1       200   fffffe810e8c5900          softclk/1
 0       19 1   1       200   fffffe810e8b10a0          softbio/1
 0       18 1   1       200   fffffe810e8b14c0          softnet/1
 0    >  17 7   1       201   fffffe810e8b18e0             idle/1
 0       16 3   0       200   fffffe822de92080           lnxsyswq lnxsys=
 wq
 0       15 3   0       200   fffffe822de924a0             sysmon smtask=
 q
 0       14 3   2       200   fffffe822de928c0         pmfsuspend pmfsus=
 pend
 0       13 3   7       200   fffffe822e2b1060           pmfevent pmfeve=
 nt
 0       12 3   0       200   fffffe822e2b1480         sopendfree sopend=
 fr
 0       11 3   2       200   fffffe822e2b18a0           nfssilly nfssil=
 ly
 0       10 3   2       200   fffffe822f6d0040            cachegc cacheg=
 c
 0        9 3   3       200   fffffe822f6d0460             vdrain vdrain=

 0        8 3   0       200   fffffe822f6d0880          modunload mod_un=
 ld
 0        7 3   0       200   fffffe822f6eb020            xcall/0 xcall
 0    >   6 7   0       200   fffffe822f6eb440          softser/0
 0    >   5 7   0       200   fffffe822f6eb860          softclk/0
 0        4 1   0       200   fffffe822f707000          softbio/0
 0        3 1   0       200   fffffe822f707420          softnet/0
 0        2 1   0       201   fffffe822f707840             idle/0
 0        1 3   4       200   ffffffff81481b20            swapper uvm
 crash> =


 Backtraces for all the lwps which have non-zero "WAIT" strings:

 crash> trace/a fffffe8220c58540
 trace: pid 7052 lid 1 at 0xffff80009ea3d8f0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x460
 VOP_LOCK() at VOP_LOCK+0x4d
 vn_lock() at vn_lock+0x90
 namei_tryemulroot() at namei_tryemulroot+0x193
 namei() at namei+0x29
 vn_open() at vn_open+0x92
 do_open() at do_open+0x112
 do_sys_openat() at do_sys_openat+0x68
 sys_open() at sys_open+0x24
 syscall() at syscall+0x1ed
 --- syscall (number 5) ---
 70efab63e2ca:
 crash>

 crash> trace/a fffffe821d362560
 trace: pid 8587 lid 1 at 0xffff80009edb8d20
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 fork1() at fork1+0x774
 sys___vfork14() at sys___vfork14+0x35
 syscall() at syscall+0x1ed
 --- syscall (number 282) ---
 70efab68204c:
 crash>

 crash> trace/a fffffe821faa60c0
 trace: pid 9187 lid 1 at 0xffff80009ed4e890
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_wait() at xc_wait+0x7c
 pserialize_perform() at pserialize_perform+0xdf
 fstrans_setstate() at fstrans_setstate+0x42
 genfs_suspendctl() at genfs_suspendctl+0x57
 VFS_SUSPENDCTL() at VFS_SUSPENDCTL+0x1f
 vfs_suspend() at vfs_suspend+0x5b
 vrevoke_suspend_next() at vrevoke_suspend_next+0x2a
 vrevoke() at vrevoke+0x2b
 genfs_revoke() at genfs_revoke+0x13
 VOP_REVOKE() at VOP_REVOKE+0x54
 pty_grant_slave() at pty_grant_slave+0xa8
 ptmioctl() at ptmioctl+0x90
 cdev_ioctl() at cdev_ioctl+0x88
 VOP_IOCTL() at VOP_IOCTL+0x3b
 vn_ioctl() at vn_ioctl+0xa6
 sys_ioctl() at sys_ioctl+0x101
 syscall() at syscall+0x1ed
 --- syscall (number 54) ---
 6f9ce08feeaa:
 crash>

 crash> trace/a fffffe82045da540
 trace: pid 3794 lid 1 at 0xffff80009f6c6be0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x74d
 VOP_LOCK() at VOP_LOCK+0x4d
 vn_lock() at vn_lock+0x90
 vn_write() at vn_write+0x9b
 dofilewrite() at dofilewrite+0x97
 sys_write() at sys_write+0x5f
 syscall() at syscall+0x1ed
 --- syscall (number 4) ---
 73b7cfe3e02a:
 crash>

 crash> trace/a fffffe8216d35140
 trace: pid 2853 lid 2 at 0xffff80009f6eaab0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x460
 VOP_POLL() at VOP_POLL+0x37
 sel_do_scan() at sel_do_scan+0x324
 selcommon.part.2() at selcommon.part.2+0x18b
 sys___select50() at sys___select50+0x6f
 syscall() at syscall+0x1ed
 --- syscall (number 417) ---
 7b328823e10a:
 crash>

 crash> trace/a fffffe82214bf2c0
 trace: pid 2853 lid 1 at 0xffff80009ef1ddb0
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 fd_close() at fd_close+0x3ba
 sys_close() at sys_close+0x20
 syscall() at syscall+0x1ed
 --- syscall (number 6) ---
 7b328823e4ea:
 crash>

 crash> trace/a fffffe8225709000
 trace: pid 8327 lid 2 at 0xffff80009e90eab0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x74d
 VOP_POLL() at VOP_POLL+0x37
 sel_do_scan() at sel_do_scan+0x324
 selcommon.part.2() at selcommon.part.2+0x18b
 sys___select50() at sys___select50+0x6f
 syscall() at syscall+0x1ed
 --- syscall (number 417) ---
 7aeb6723e10a:
 crash>

 crash> trace/a fffffe820df34480
 trace: pid 8327 lid 1 at 0xffff80009f668db0
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 fd_close() at fd_close+0x3ba
 sys_close() at sys_close+0x20
 syscall() at syscall+0x1ed
 --- syscall (number 6) ---
 7aeb6723e4ea:
 crash>

 crash> trace/a fffffe8207fd0360
 trace: pid 5687 lid 2 at 0xffff80009f5f9ab0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x74d
 VOP_POLL() at VOP_POLL+0x37
 sel_do_scan() at sel_do_scan+0x324
 selcommon.part.2() at selcommon.part.2+0x18b
 sys___select50() at sys___select50+0x6f
 syscall() at syscall+0x1ed
 --- syscall (number 417) ---
 7d665343e10a:
 crash>

 crash> trace/a fffffe822346f080
 trace: pid 5687 lid 1 at 0xffff80009efd4db0
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 fd_close() at fd_close+0x3ba
 sys_close() at sys_close+0x20
 syscall() at syscall+0x1ed
 --- syscall (number 6) ---
 7d665343e4ea:
 crash>

 crash> trace/a fffffe821d987a40
 trace: pid 9671 lid 2 at 0xffff80009ee64ab0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x74d
 VOP_POLL() at VOP_POLL+0x37
 sel_do_scan() at sel_do_scan+0x324
 selcommon.part.2() at selcommon.part.2+0x18b
 sys___select50() at sys___select50+0x6f
 syscall() at syscall+0x1ed
 --- syscall (number 417) ---
 7ea7fc43e10a:
 crash>

 crash> trace/a fffffe821e90c240
 trace: pid 9671 lid 1 at 0xffff80009ee94db0
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 fd_close() at fd_close+0x3ba
 sys_close() at sys_close+0x20
 syscall() at syscall+0x1ed
 --- syscall (number 6) ---
 7ea7fc43e4ea:
 crash>

 crash> trace/a fffffe821d362980
 trace: pid 704 lid 1 at 0xffff80009edaebc0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 rw_enter() at rw_enter+0x49d
 genfs_lock() at genfs_lock+0x3c
 VOP_LOCK() at VOP_LOCK+0x67
 vn_lock() at vn_lock+0x90
 vn_write() at vn_write+0x9b
 dofilewrite() at dofilewrite+0x97
 sys_write() at sys_write+0x5f
 syscall() at syscall+0x1ed
 --- syscall (number 4) ---
 764fb503e02a:
 crash>

 crash> trace/a fffffe8229b924c0
 trace: pid 564 lid 1 at 0xffff80009e9c3ab0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x74d
 VOP_POLL() at VOP_POLL+0x37
 sel_do_scan() at sel_do_scan+0x324
 selcommon.part.2() at selcommon.part.2+0x18b
 sys___select50() at sys___select50+0x6f
 syscall() at syscall+0x1ed
 --- syscall (number 417) ---
 764fb503e10a:
 crash>

 crash> trace/a fffffe8220b8e780
 trace: pid 275 lid 1 at 0xffff80009ebf8bc0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 rw_enter() at rw_enter+0x49d
 genfs_lock() at genfs_lock+0x3c
 VOP_LOCK() at VOP_LOCK+0x67
 vn_lock() at vn_lock+0x90
 vn_write() at vn_write+0x9b
 dofilewrite() at dofilewrite+0x97
 sys_write() at sys_write+0x5f
 syscall() at syscall+0x1ed
 --- syscall (number 4) ---
 764fb503e02a:
 crash>

 crash> trace/a fffffe8221e57600
 trace: pid 878 lid 1 at 0xffff80009eae8ab0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x74d
 VOP_POLL() at VOP_POLL+0x37
 sel_do_scan() at sel_do_scan+0x324
 selcommon.part.2() at selcommon.part.2+0x18b
 sys___select50() at sys___select50+0x6f
 syscall() at syscall+0x1ed
 --- syscall (number 417) ---
 764fb503e10a:
 crash>

 crash> trace/a fffffe8222569900
 trace: pid 136 lid 1 at 0xffff80009e9cdab0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x74d
 VOP_POLL() at VOP_POLL+0x37
 sel_do_scan() at sel_do_scan+0x324
 selcommon.part.2() at selcommon.part.2+0x18b
 sys___select50() at sys___select50+0x6f
 syscall() at syscall+0x1ed
 --- syscall (number 417) ---
 764fb503e10a:
 crash>

 crash> trace/a fffffe8225709840
 trace: pid 773 lid 1 at 0xffff80009e904ab0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x74d
 VOP_POLL() at VOP_POLL+0x37
 sel_do_scan() at sel_do_scan+0x324
 selcommon.part.2() at selcommon.part.2+0x18b
 sys___select50() at sys___select50+0x6f
 syscall() at syscall+0x1ed
 --- syscall (number 417) ---
 764fb503e10a:
 crash>

 crash> trace/a fffffe822d4cd1e0
 trace: pid 0 lid 123 at 0xffff80009dee1da0
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_broadcast() at xc_broadcast+0x17a
 pserialize_perform() at pserialize_perform+0xd7
 npf_conn_gc() at npf_conn_gc+0x1b5
 npf_worker() at npf_worker+0x4b
 crash>

 crash> trace/a fffffe822d4cda20
 trace: pid 0 lid 122 at 0xffff80009dec3e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe822dd10940
 trace: pid 0 lid 121 at 0xffff80009ddc4e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe822dcff960
 trace: pid 0 lid 120 at 0xffff80009dceee60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe822dd10100
 trace: pid 0 lid 119 at 0xffff80009dce9bf0
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 biowait() at biowait+0x5d
 wapbl_buffered_flush() at wapbl_buffered_flush+0xc6
 wapbl_buffered_write() at wapbl_buffered_write+0x69
 wapbl_circ_write() at wapbl_circ_write+0xa7
 wapbl_flush() at wapbl_flush+0x354
 ffs_sync() at ffs_sync+0x221
 VFS_SYNC() at VFS_SYNC+0x1c
 sched_sync() at sched_sync+0x91
 crash>

 crash> trace/a fffffe822dd10520
 trace: pid 0 lid 118 at 0xffff80009dce4d80
 sleepq_block() at sleepq_block+0x97
 mtsleep() at mtsleep+0x102
 uvm_pageout() at uvm_pageout+0x62e
 crash>

 crash> trace/a fffffe822dd62500
 trace: pid 0 lid 115 at 0xffff80009dc8ee60
 sleepq_block() at sleepq_block+0x97
 cv_wait_sig() at cv_wait_sig+0x12b
 bell_thread() at bell_thread+0x5f
 crash>

 crash> trace/a fffffe822dd62920
 trace: pid 0 lid 114 at 0xffff80009db5fe80
 sleepq_block() at sleepq_block+0x97
 cv_wait_sig() at cv_wait_sig+0x12b
 audio_play_thread() at audio_play_thread+0x55
 crash>

 crash> trace/a fffffe810ed3b2a0
 trace: pid 0 lid 113 at 0xffff80009dc89e90
 sleepq_block() at sleepq_block+0x97
 cv_wait_sig() at cv_wait_sig+0x12b
 audio_rec_thread() at audio_rec_thread+0x3a
 crash>

 crash> trace/a fffffe810f4fa0c0
 trace: pid 0 lid 112 at 0xffff80009dc32e80
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 scsipi_completion_thread() at scsipi_completion_thread+0xf5
 crash>

 crash> trace/a fffffe810f2e9b60
 trace: pid 0 lid 110 at 0xffff80009dc27e80
 sleepq_block() at sleepq_block+0x106
 cv_timedwait() at cv_timedwait+0x131
 usb_event_thread() at usb_event_thread+0xa2
 crash>

 crash> trace/a fffffe810f2c3b20
 trace: pid 0 lid 109 at 0xffff80009dc22e80
 sleepq_block() at sleepq_block+0x106
 cv_timedwait() at cv_timedwait+0x131
 usb_event_thread() at usb_event_thread+0xa2
 crash>

 crash> trace/a fffffe810f20d2c0
 trace: pid 0 lid 108 at 0xffff80009db64e80
 sleepq_block() at sleepq_block+0x106
 cv_timedwait() at cv_timedwait+0x131
 usb_event_thread() at usb_event_thread+0xa2
 crash>

 crash> trace/a fffffe810f2da300
 trace: pid 0 lid 107 at 0xffff80009dc1de80
 sleepq_block() at sleepq_block+0x106
 cv_timedwait() at cv_timedwait+0x131
 usb_event_thread() at usb_event_thread+0xa2
 crash>

 crash> trace/a fffffe810f2c32e0
 trace: pid 0 lid 106 at 0xffff80009dc18e80
 sleepq_block() at sleepq_block+0x106
 cv_timedwait() at cv_timedwait+0x131
 usb_event_thread() at usb_event_thread+0xa2
 crash>

 crash> trace/a fffffe810f4fa4e0
 trace: pid 0 lid 105 at 0xffff80009dc13e80
 sleepq_block() at sleepq_block+0x106
 cv_timedwait() at cv_timedwait+0x131
 usb_event_thread() at usb_event_thread+0xa2
 crash>

 crash> trace/a fffffe810f4a30a0
 trace: pid 0 lid 104 at 0xffff80009dbfce60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f4a34c0
 trace: pid 0 lid 103 at 0xffff80009dbf7e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 unp_thread() at unp_thread+0x700
 crash>

 crash> trace/a fffffe810f4a38e0
 trace: pid 0 lid 102 at 0xffff80009dbf2e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f45c080
 trace: pid 0 lid 101 at 0xffff80009dbede60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f45c4a0
 trace: pid 0 lid 100 at 0xffff80009dbe8e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f45c8c0
 trace: pid 0 lid 99 at 0xffff80009dbe3e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f4b5060
 trace: pid 0 lid 98 at 0xffff80009dbdee60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f4b5480
 trace: pid 0 lid 97 at 0xffff80009dbd9e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f4b58a0
 trace: pid 0 lid 96 at 0xffff80009dbd4e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f396040
 trace: pid 0 lid 95 at 0xffff80009dbcfe60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f396460
 trace: pid 0 lid 94 at 0xffff80009dbcae60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f396880
 trace: pid 0 lid 93 at 0xffff80009dbc5e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f367020
 trace: pid 0 lid 92 at 0xffff80009dbc0e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f367440
 trace: pid 0 lid 91 at 0xffff80009dbbbe60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f367860
 trace: pid 0 lid 90 at 0xffff80009dbb6e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash> =


 crash> trace/a fffffe810f378000
 trace: pid 0 lid 89 at 0xffff80009dbb1e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f378420
 trace: pid 0 lid 88 at 0xffff80009dbace60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f378840
 trace: pid 0 lid 87 at 0xffff80009dba7e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f329360
 trace: pid 0 lid 86 at 0xffff80009dba2e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f329780
 trace: pid 0 lid 85 at 0xffff80009db6ee60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f2da720
 trace: pid 0 lid 84 at 0xffff80009db9de60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f329ba0
 trace: pid 0 lid 83 at 0xffff80009db98e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f2f8340
 trace: pid 0 lid 82 at 0xffff80009db91e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f2f8760
 trace: pid 0 lid 81 at 0xffff80009db8ce60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f2f8b80
 trace: pid 0 lid 80 at 0xffff80009db87e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f2e9320
 trace: pid 0 lid 79 at 0xffff80009db82e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f2e9740
 trace: pid 0 lid 78 at 0xffff80009db7de60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f20d6e0
 trace: pid 0 lid 69 at 0xffff80009d349eb0
 sleepq_block() at sleepq_block+0x97
 atabus_thread() at atabus_thread+0xcc
 crash>

 crash> trace/a fffffe810f20db00
 trace: pid 0 lid 68 at 0xffff80009d344eb0
 sleepq_block() at sleepq_block+0x97
 atabus_thread() at atabus_thread+0xcc
 crash>

 crash> trace/a fffffe810ed3b6c0
 trace: pid 0 lid 66 at 0xffff80009d0ffe80
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 scsipi_completion_thread() at scsipi_completion_thread+0xf5
 crash>

 crash> trace/a fffffe810ed3bae0
 trace: pid 0 lid 65 at 0xffff800099072e70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 usb_task_thread() at usb_task_thread+0x8a
 crash>

 crash> trace/a fffffe810ecaa280
 trace: pid 0 lid 64 at 0xffff80009906de70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 usb_task_thread() at usb_task_thread+0x8a
 crash>

 crash> trace/a fffffe810ecaa6a0
 trace: pid 0 lid 63 at 0xffff800099056e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810ecaaac0
 trace: pid 0 lid 62 at 0xffff800096a41e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810eb61260
 trace: pid 0 lid 61 at 0xffff80009442ae60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810eb61680
 trace: pid 0 lid 60 at 0xffff800091e15e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810eb61aa0
 trace: pid 0 lid 59 at 0xffff80008f7f8e30
 sleepq_block() at sleepq_block+0x1c5
 cv_timedwait() at cv_timedwait+0x131
 ipmi_thread() at ipmi_thread+0x2f4
 crash>

 crash> trace/a fffffe810ea7c240
 trace: pid 0 lid 58 at 0xffff80008f7ede70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_thread() at xc_thread+0x5a
 crash>

 crash> trace/a fffffe810ea36200
 trace: pid 0 lid 52 at 0xffff80008f7c7e70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_thread() at xc_thread+0x5a
 crash>

 crash> trace/a fffffe810e9e01c0
 trace: pid 0 lid 46 at 0xffff80008f7a1e70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_thread() at xc_thread+0x5a
 crash>

 crash> trace/a fffffe810e9a2180
 trace: pid 0 lid 40 at 0xffff80008f77be70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_thread() at xc_thread+0x5a
 crash>

 crash> trace/a fffffe810e94c140
 trace: pid 0 lid 34 at 0xffff80008f755e70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_thread() at xc_thread+0x5a
 crash>

 crash> trace/a fffffe810e906100
 trace: pid 0 lid 28 at 0xffff80008f72fe70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_thread() at xc_thread+0x5a
 crash>

 crash> trace/a fffffe810e8c50c0
 trace: pid 0 lid 22 at 0xffff80008f706e70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_thread() at xc_thread+0x5a
 crash>

 crash> trace/a fffffe822de92080
 trace: pid 0 lid 16 at 0xffff80008f6cde60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe822de924a0
 trace: pid 0 lid 15 at 0xffff80008f6c8e90
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 sysmon_task_queue_thread() at sysmon_task_queue_thread+0x81
 crash>

 crash> trace/a fffffe822de928c0
 trace: pid 0 lid 14 at 0xffff80008f6c3e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe822e2b1060
 trace: pid 0 lid 13 at 0xffff80008f6bee60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe822e2b1480
 trace: pid 0 lid 12 at 0xffff80008f6b9e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 sopendfree_thread() at sopendfree_thread+0x1da
 crash>

 crash> trace/a fffffe822e2b18a0
 trace: pid 0 lid 11 at 0xffff80008f6b4e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe822f6d0040
 trace: pid 0 lid 10 at 0xffff80008f6afe80
 sleepq_block() at sleepq_block+0x106
 kpause() at kpause+0xea
 cache_thread() at cache_thread+0x30
 crash>

 crash> trace/a fffffe822f6d0460
 trace: pid 0 lid 9 at 0xffff80008f6aae30
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 vdrain_thread() at vdrain_thread+0x3f1
 crash>

 crash> trace/a fffffe822f6d0880
 trace: pid 0 lid 8 at 0xffff80008f6a5e80
 sleepq_block() at sleepq_block+0x97
 cv_timedwait() at cv_timedwait+0x131
 module_thread() at module_thread+0xe9
 crash>

 crash> trace/a fffffe822f6eb020
 trace: pid 0 lid 7 at 0xffff80008f6a0e70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_thread() at xc_thread+0x5a
 crash>

 crash> trace/a ffffffff81481b20
 trace: pid 0 lid 1 at 0xffffffff817dde20
 sleepq_block() at sleepq_block+0x106
 kpause() at kpause+0xea
 uvm_scheduler() at uvm_scheduler+0x69
 sysctl_alloc() at sysctl_alloc
 crash>

 ------------------------------

 (gdb) kvm proc 0xfffffe8220c58540
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8220c58540)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8220c58540)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =

     ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =

     obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =

     sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fec40 in fstrans_alloc_lwp_info (mp=3D0xfffffe822dd7a0=
 08)
     at /usr/src/sys/kern/vfs_trans.c:274
 #5  fstrans_get_lwp_info (do_alloc=3Dtrue, mp=3D0xfffffe822dd7a008)
     at /usr/src/sys/kern/vfs_trans.c:331
 #6  _fstrans_start (wait=3D1, lock_type=3DFSTRANS_SHARED, mp=3D<optimiz=
 ed out>)
     at /usr/src/sys/kern/vfs_trans.c:375
 #7  fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
 .c:414
 #8  0xffffffff80a0ed1b in vop_pre (vp=3D0xfffffe810f2f2930, =

     vp=3D0xfffffe810f2f2930, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
 , =

     mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #9  VOP_LOCK (vp=3Dvp@entry=3D0xfffffe810f2f2930, flags=3Dflags@entry=3D=
 131074)
     at /usr/src/sys/kern/vnode_if.c:1281
 #10 0xffffffff80a05846 in vn_lock (vp=3Dvp@entry=3D0xfffffe810f2f2930, =

     flags=3Dflags@entry=3D131074) at /usr/src/sys/kern/vfs_vnops.c:1036=

 #11 0xffffffff809ef7c3 in namei_start (startdir_ret=3D0xffff80009ea3db2=
 8, =

     isnfsd=3D0, state=3D0xffff80009ea3dbd0) at /usr/src/sys/kern/vfs_lo=
 okup.c:681
 #12 namei_oneroot (isnfsd=3D0, inhibitmagic=3D0, neverfollow=3D0, =

     state=3D0xffff80009ea3dbd0) at /usr/src/sys/kern/vfs_lookup.c:1156
 #13 namei_tryemulroot (state=3Dstate@entry=3D0xffff80009ea3dbd0, =

     neverfollow=3Dneverfollow@entry=3D0, inhibitmagic=3Dinhibitmagic@en=
 try=3D0, =

     isnfsd=3Disnfsd@entry=3D0) at /usr/src/sys/kern/vfs_lookup.c:1510
 #14 0xffffffff809f1905 in namei (ndp=3Dndp@entry=3D0xffff80009ea3dd58)
     at /usr/src/sys/kern/vfs_lookup.c:1546
 #15 0xffffffff80a061fd in vn_open (ndp=3Dndp@entry=3D0xffff80009ea3dd58=
 , =

     fmode=3Dfmode@entry=3D4194305, cmode=3Dcmode@entry=3D3116)
     at /usr/src/sys/kern/vfs_vnops.c:175
 #16 0xffffffff809fb7c4 in do_open (l=3Dl@entry=3D0xfffffe8220c58540, dv=
 p=3D0x0, =

     pb=3D<optimized out>, open_flags=3Dopen_flags@entry=3D4194304, =

     open_mode=3Dopen_mode@entry=3D774778414, fd=3Dfd@entry=3D0xffff8000=
 9ea3de7c)
     at /usr/src/sys/kern/vfs_syscalls.c:1576
 #17 0xffffffff809fb915 in do_sys_openat (l=3D0xfffffe8220c58540, =

     fdat=3Dfdat@entry=3D-100, path=3D<optimized out>, flags=3D4194304, =
 mode=3D774778414, =

     fd=3Dfd@entry=3D0xffff80009ea3de7c) at /usr/src/sys/kern/vfs_syscal=
 ls.c:1656
 #18 0xffffffff809fb9d3 in sys_open (l=3D<optimized out>, uap=3D<optimiz=
 ed out>, =

     retval=3D0xffff80009ea3deb0) at /usr/src/sys/kern/vfs_syscalls.c:16=
 76
 #19 0xffffffff80248dbd in sy_call (rval=3D0xffff80009ea3deb0, =

     uap=3D0xffff80009ea3df00, l=3D0xfffffe8220c58540, =

     sy=3D0xffffffff8147da58 <sysent+120>) at /usr/src/sys/sys/syscallva=
 r.h:65
 #20 sy_invoke (code=3D5, rval=3D0xffff80009ea3deb0, uap=3D0xffff80009ea=
 3df00, =

     l=3D0xfffffe8220c58540, sy=3D0xffffffff8147da58 <sysent+120>)
     at /usr/src/sys/sys/syscallvar.h:94
 #21 syscall (frame=3D0xffff80009ea3df00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #22 0xffffffff8020768d in handle_syscall ()
 #23 0x000070efab549000 in ?? ()
 #24 0x0000000000400000 in ?? ()
 #25 0x2e2e2e2e2e2e2e2e in ?? ()
 #26 0x0000000000000000 in ?? ()
 (gdb)

 Looking at fstrans_lock:

 (gdb) lock 0xffffffff815cb9b0
 lock address : 0xffffffff815cb9b0 type     : sleep/adaptive
 initialized  : 0xffffffff809fe6a3
 shared holds :                  0 exclusive: 1
 shares wanted:                  0 exclusive:                 10
 cpu last held:                  6
 current lwp  : 000000000000000000 last held: 0xfffffe821faa60c0
 last locked  : 0xffffffff809ffb6d unlocked : 0xffffffff809fecc4
 (gdb) =


 What I don't understand is that "current lwp" is 0x0, even though
 this one seems to be held at the moment, and there's lots of
 contention for the lock judging by the "shares wanted, exclusive: 10".

 "last held" appears to point to

 9187     1 3   6   8020000   fffffe821faa60c0             expect xchicv=


 (gdb) kvm proc 0xfffffe821faa60c0
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821faa60c0)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821faa60c0)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8094afca in cv_wait (
     cv=3Dcv@entry=3D0xffffffff814c5c88 <xc_high_pri+8>, =

     mtx=3Dmtx@entry=3D0xffffffff814c5c80 <xc_high_pri>)
     at /usr/src/sys/kern/kern_condvar.c:217
 #3  0xffffffff809b1293 in xc_wait (where=3D64992)
     at /usr/src/sys/kern/subr_xcall.c:321
 #4  0xffffffff809aa122 in pserialize_perform (psz=3D0xfffffe822fbd20e8)=

     at /usr/src/sys/kern/subr_pserialize.c:187
 #5  0xffffffff809ffb7c in fstrans_setstate (mp=3D0xfffffe822c993008, =

     new_state=3Dnew_state@entry=3DFSTRANS_SUSPENDED)
     at /usr/src/sys/kern/vfs_trans.c:530
 #6  0xffffffff80a17729 in genfs_suspendctl (mp=3D0xfffffe822c993008, =

     cmd=3D<optimized out>) at /usr/src/sys/miscfs/genfs/genfs_vfsops.c:=
 86
 #7  0xffffffff809f781e in VFS_SUSPENDCTL (mp=3Dmp@entry=3D0xfffffe822c9=
 93008, =

     a=3Da@entry=3D1) at /usr/src/sys/kern/vfs_subr.c:1478
 #8  0xffffffff80a00530 in vfs_suspend (mp=3D<optimized out>, =

     mp@entry=3D0xfffffe822c993008, nowait=3Dnowait@entry=3D0)
     at /usr/src/sys/kern/vfs_trans.c:587
 #9  0xffffffff80a01769 in vrevoke_suspend_next (lastmp=3D<optimized out=
 >, =

     thismp=3D0xfffffe822c993008) at /usr/src/sys/kern/vfs_vnode.c:978
 #10 0xffffffff80a030ed in vrevoke (vp=3D0xfffffe8221e36690)
     at /usr/src/sys/kern/vfs_vnode.c:1002
 #11 0xffffffff80a17caa in genfs_revoke (v=3D<optimized out>)
     at /usr/src/sys/miscfs/genfs/genfs_vnops.c:273
 #12 0xffffffff80a0e4f7 in VOP_REVOKE (vp=3D0xfffffe8221e36690, =

     flags=3Dflags@entry=3D1) at /usr/src/sys/kern/vnode_if.c:744
 #13 0xffffffff809cfb00 in pty_grant_slave (l=3Dl@entry=3D0xfffffe821faa=
 60c0, =

     dev=3D<optimized out>, mp=3D<optimized out>) at /usr/src/sys/kern/t=
 ty_ptm.c:245
 #14 0xffffffff809cfd7b in ptmioctl (dev=3D<optimized out>, cmd=3D<optim=
 ized out>, =

     data=3D0xfffffe822d624008, flag=3D<optimized out>, l=3D0xfffffe821f=
 aa60c0)
     at /usr/src/sys/kern/tty_ptm.c:397
 #15 0xffffffff80996a52 in cdev_ioctl (dev=3D42241, cmd=3D1208513606, =

     data=3D0xfffffe822d624008, flag=3D3, l=3D0xfffffe821faa60c0)
     at /usr/src/sys/kern/subr_devsw.c:938
 #16 0xffffffff80a0e2b8 in VOP_IOCTL (vp=3Dvp@entry=3D0xfffffe822caabd20=
 , =

     command=3Dcommand@entry=3D1208513606, data=3Ddata@entry=3D0xfffffe8=
 22d624008, =

     fflag=3D<optimized out>, cred=3D<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:610
 #17 0xffffffff80a05e1e in vn_ioctl (fp=3D0xfffffe821d011a40, com=3D1208=
 513606, =

     data=3D0xfffffe822d624008) at /usr/src/sys/kern/vfs_vnops.c:765
 #18 0xffffffff809b4e81 in sys_ioctl (l=3D<optimized out>, =

     uap=3D0xffff80009ed4ef00, retval=3D<optimized out>)
     at /usr/src/sys/kern/sys_generic.c:671
 #19 0xffffffff80248dbd in sy_call (rval=3D0xffff80009ed4eeb0, =

     uap=3D0xffff80009ed4ef00, l=3D0xfffffe821faa60c0, =

     sy=3D0xffffffff8147def0 <sysent+1296>) at /usr/src/sys/sys/syscallv=
 ar.h:65
 #20 sy_invoke (code=3D54, rval=3D0xffff80009ed4eeb0, uap=3D0xffff80009e=
 d4ef00, =

     l=3D0xfffffe821faa60c0, sy=3D0xffffffff8147def0 <sysent+1296>)
     at /usr/src/sys/sys/syscallvar.h:94
 #21 syscall (frame=3D0xffff80009ed4ef00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #22 0xffffffff8020768d in handle_syscall ()
 #23 0x0000000000000003 in ?? ()
 #24 0x0000000048087446 in ?? ()
 #25 0x00007f7fff318360 in ?? ()
 #26 0x0000000000000000 in ?? ()
 (gdb)

 The xc_high_pri looks like this:

 (gdb) lock 0xffffffff814c5c80
 lock address : 0xffffffff814c5c80 type     : sleep/adaptive
 initialized  : 0xffffffff809b1053
 shared holds :                  0 exclusive: 0
 shares wanted:                  0 exclusive:                  0
 cpu last held:                  7
 current lwp  : 000000000000000000 last held: 000000000000000000
 last locked  : 0xffffffff809b17d1 unlocked : 0xffffffff8094afb7
 (gdb) =


 but that doesn't appear to lead anywhere.

 ------------------------------

 Decode of the stack backtraces of the non-kernel lwps using gdb:

 (gdb) kvm proc 0xfffffe8220c58540
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8220c58540)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8220c58540)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =

     ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =

     obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =

     sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fec40 in fstrans_alloc_lwp_info (mp=3D0xfffffe822dd7a0=
 08)
     at /usr/src/sys/kern/vfs_trans.c:274
 #5  fstrans_get_lwp_info (do_alloc=3Dtrue, mp=3D0xfffffe822dd7a008)
     at /usr/src/sys/kern/vfs_trans.c:331
 #6  _fstrans_start (wait=3D1, lock_type=3DFSTRANS_SHARED, mp=3D<optimiz=
 ed out>)
     at /usr/src/sys/kern/vfs_trans.c:375
 #7  fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
 .c:414
 #8  0xffffffff80a0ed1b in vop_pre (vp=3D0xfffffe810f2f2930, =

     vp=3D0xfffffe810f2f2930, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
 , =

     mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #9  VOP_LOCK (vp=3Dvp@entry=3D0xfffffe810f2f2930, flags=3Dflags@entry=3D=
 131074)
     at /usr/src/sys/kern/vnode_if.c:1281
 #10 0xffffffff80a05846 in vn_lock (vp=3Dvp@entry=3D0xfffffe810f2f2930, =

     flags=3Dflags@entry=3D131074) at /usr/src/sys/kern/vfs_vnops.c:1036=

 #11 0xffffffff809ef7c3 in namei_start (startdir_ret=3D0xffff80009ea3db2=
 8, =

     isnfsd=3D0, state=3D0xffff80009ea3dbd0) at /usr/src/sys/kern/vfs_lo=
 okup.c:681
 #12 namei_oneroot (isnfsd=3D0, inhibitmagic=3D0, neverfollow=3D0, =

     state=3D0xffff80009ea3dbd0) at /usr/src/sys/kern/vfs_lookup.c:1156
 #13 namei_tryemulroot (state=3Dstate@entry=3D0xffff80009ea3dbd0, =

     neverfollow=3Dneverfollow@entry=3D0, inhibitmagic=3Dinhibitmagic@en=
 try=3D0, =

     isnfsd=3Disnfsd@entry=3D0) at /usr/src/sys/kern/vfs_lookup.c:1510
 #14 0xffffffff809f1905 in namei (ndp=3Dndp@entry=3D0xffff80009ea3dd58)
     at /usr/src/sys/kern/vfs_lookup.c:1546
 #15 0xffffffff80a061fd in vn_open (ndp=3Dndp@entry=3D0xffff80009ea3dd58=
 , =

     fmode=3Dfmode@entry=3D4194305, cmode=3Dcmode@entry=3D3116)
     at /usr/src/sys/kern/vfs_vnops.c:175
 #16 0xffffffff809fb7c4 in do_open (l=3Dl@entry=3D0xfffffe8220c58540, dv=
 p=3D0x0, =

     pb=3D<optimized out>, open_flags=3Dopen_flags@entry=3D4194304, =

     open_mode=3Dopen_mode@entry=3D774778414, fd=3Dfd@entry=3D0xffff8000=
 9ea3de7c)
     at /usr/src/sys/kern/vfs_syscalls.c:1576
 #17 0xffffffff809fb915 in do_sys_openat (l=3D0xfffffe8220c58540, =

     fdat=3Dfdat@entry=3D-100, path=3D<optimized out>, flags=3D4194304, =
 mode=3D774778414, =

     fd=3Dfd@entry=3D0xffff80009ea3de7c) at /usr/src/sys/kern/vfs_syscal=
 ls.c:1656
 #18 0xffffffff809fb9d3 in sys_open (l=3D<optimized out>, uap=3D<optimiz=
 ed out>, =

     retval=3D0xffff80009ea3deb0) at /usr/src/sys/kern/vfs_syscalls.c:16=
 76
 #19 0xffffffff80248dbd in sy_call (rval=3D0xffff80009ea3deb0, =

     uap=3D0xffff80009ea3df00, l=3D0xfffffe8220c58540, =

     sy=3D0xffffffff8147da58 <sysent+120>) at /usr/src/sys/sys/syscallva=
 r.h:65
 #20 sy_invoke (code=3D5, rval=3D0xffff80009ea3deb0, uap=3D0xffff80009ea=
 3df00, =

     l=3D0xfffffe8220c58540, sy=3D0xffffffff8147da58 <sysent+120>)
     at /usr/src/sys/sys/syscallvar.h:94
 #21 syscall (frame=3D0xffff80009ea3df00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #22 0xffffffff8020768d in handle_syscall ()
 #23 0x000070efab549000 in ?? ()
 #24 0x0000000000400000 in ?? ()
 #25 0x2e2e2e2e2e2e2e2e in ?? ()
 #26 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe821d362560
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821d362560)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821d362560)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8094afca in cv_wait (cv=3Dcv@entry=3D0xfffffe8210396788, =

     mtx=3D0xfffffe822fb97f00) at /usr/src/sys/kern/kern_condvar.c:217
 #3  0xffffffff80959f24 in fork1 (l1=3D0xfffffe821d362560, flags=3Dflags=
 @entry=3D3, =

     exitsig=3Dexitsig@entry=3D20, stack=3Dstack@entry=3D0x0, =

     stacksize=3Dstacksize@entry=3D0, func=3Dfunc@entry=3D0x0, arg=3Darg=
 @entry=3D0x0, =

     retval=3D0xffff80009edb8eb0, rnewprocp=3Drnewprocp@entry=3D0x0)
     at /usr/src/sys/kern/kern_fork.c:592
 #4  0xffffffff8095a424 in sys___vfork14 (l=3D<optimized out>, v=3D<opti=
 mized out>, =

     retval=3D<optimized out>) at /usr/src/sys/kern/kern_fork.c:143
 #5  0xffffffff80248dbd in sy_call (rval=3D0xffff80009edb8eb0, =

     uap=3D0xffff80009edb8f00, l=3D0xfffffe821d362560, =

     sy=3D0xffffffff8147f450 <sysent+6768>) at /usr/src/sys/sys/syscallv=
 ar.h:65
 #6  sy_invoke (code=3D282, rval=3D0xffff80009edb8eb0, uap=3D0xffff80009=
 edb8f00, =

     l=3D0xfffffe821d362560, sy=3D0xffffffff8147f450 <sysent+6768>)
     at /usr/src/sys/sys/syscallvar.h:94
 #7  syscall (frame=3D0xffff80009edb8f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #8  0xffffffff8020768d in handle_syscall ()
 #9  0x00007f7fff9fa3d8 in ?? ()
 #10 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe821faa60c0
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821faa60c0)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821faa60c0)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8094afca in cv_wait (
     cv=3Dcv@entry=3D0xffffffff814c5c88 <xc_high_pri+8>, =

     mtx=3Dmtx@entry=3D0xffffffff814c5c80 <xc_high_pri>)
     at /usr/src/sys/kern/kern_condvar.c:217
 #3  0xffffffff809b1293 in xc_wait (where=3D64992)
     at /usr/src/sys/kern/subr_xcall.c:321
 #4  0xffffffff809aa122 in pserialize_perform (psz=3D0xfffffe822fbd20e8)=

     at /usr/src/sys/kern/subr_pserialize.c:187
 #5  0xffffffff809ffb7c in fstrans_setstate (mp=3D0xfffffe822c993008, =

     new_state=3Dnew_state@entry=3DFSTRANS_SUSPENDED)
     at /usr/src/sys/kern/vfs_trans.c:530
 #6  0xffffffff80a17729 in genfs_suspendctl (mp=3D0xfffffe822c993008, =

     cmd=3D<optimized out>) at /usr/src/sys/miscfs/genfs/genfs_vfsops.c:=
 86
 #7  0xffffffff809f781e in VFS_SUSPENDCTL (mp=3Dmp@entry=3D0xfffffe822c9=
 93008, =

     a=3Da@entry=3D1) at /usr/src/sys/kern/vfs_subr.c:1478
 #8  0xffffffff80a00530 in vfs_suspend (mp=3D<optimized out>, =

     mp@entry=3D0xfffffe822c993008, nowait=3Dnowait@entry=3D0)
     at /usr/src/sys/kern/vfs_trans.c:587
 #9  0xffffffff80a01769 in vrevoke_suspend_next (lastmp=3D<optimized out=
 >, =

     thismp=3D0xfffffe822c993008) at /usr/src/sys/kern/vfs_vnode.c:978
 #10 0xffffffff80a030ed in vrevoke (vp=3D0xfffffe8221e36690)
     at /usr/src/sys/kern/vfs_vnode.c:1002
 #11 0xffffffff80a17caa in genfs_revoke (v=3D<optimized out>)
     at /usr/src/sys/miscfs/genfs/genfs_vnops.c:273
 #12 0xffffffff80a0e4f7 in VOP_REVOKE (vp=3D0xfffffe8221e36690, =

     flags=3Dflags@entry=3D1) at /usr/src/sys/kern/vnode_if.c:744
 #13 0xffffffff809cfb00 in pty_grant_slave (l=3Dl@entry=3D0xfffffe821faa=
 60c0, =

     dev=3D<optimized out>, mp=3D<optimized out>) at /usr/src/sys/kern/t=
 ty_ptm.c:245
 #14 0xffffffff809cfd7b in ptmioctl (dev=3D<optimized out>, cmd=3D<optim=
 ized out>, =

     data=3D0xfffffe822d624008, flag=3D<optimized out>, l=3D0xfffffe821f=
 aa60c0)
     at /usr/src/sys/kern/tty_ptm.c:397
 #15 0xffffffff80996a52 in cdev_ioctl (dev=3D42241, cmd=3D1208513606, =

     data=3D0xfffffe822d624008, flag=3D3, l=3D0xfffffe821faa60c0)
     at /usr/src/sys/kern/subr_devsw.c:938
 #16 0xffffffff80a0e2b8 in VOP_IOCTL (vp=3Dvp@entry=3D0xfffffe822caabd20=
 , =

     command=3Dcommand@entry=3D1208513606, data=3Ddata@entry=3D0xfffffe8=
 22d624008, =

     fflag=3D<optimized out>, cred=3D<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:610
 #17 0xffffffff80a05e1e in vn_ioctl (fp=3D0xfffffe821d011a40, com=3D1208=
 513606, =

     data=3D0xfffffe822d624008) at /usr/src/sys/kern/vfs_vnops.c:765
 #18 0xffffffff809b4e81 in sys_ioctl (l=3D<optimized out>, =

     uap=3D0xffff80009ed4ef00, retval=3D<optimized out>)
     at /usr/src/sys/kern/sys_generic.c:671
 #19 0xffffffff80248dbd in sy_call (rval=3D0xffff80009ed4eeb0, =

     uap=3D0xffff80009ed4ef00, l=3D0xfffffe821faa60c0, =

     sy=3D0xffffffff8147def0 <sysent+1296>) at /usr/src/sys/sys/syscallv=
 ar.h:65
 #20 sy_invoke (code=3D54, rval=3D0xffff80009ed4eeb0, uap=3D0xffff80009e=
 d4ef00, =

     l=3D0xfffffe821faa60c0, sy=3D0xffffffff8147def0 <sysent+1296>)
     at /usr/src/sys/sys/syscallvar.h:94
 #21 syscall (frame=3D0xffff80009ed4ef00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #22 0xffffffff8020768d in handle_syscall ()
 #23 0x0000000000000003 in ?? ()
 #24 0x0000000048087446 in ?? ()
 #25 0x00007f7fff318360 in ?? ()
 #26 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe82045da540
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe82045da540)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe82045da540)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =

     ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =

     obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =

     sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fef2d in _fstrans_start (wait=3D1, lock_type=3DFSTRANS=
 _SHARED, =

     mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
 #5  fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
 .c:414
 #6  0xffffffff80a0ed1b in vop_pre (vp=3D0xfffffe821a9bc7e0, =

     vp=3D0xfffffe821a9bc7e0, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
 , =

     mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #7  VOP_LOCK (vp=3Dvp@entry=3D0xfffffe821a9bc7e0, flags=3Dflags@entry=3D=
 131074)
     at /usr/src/sys/kern/vnode_if.c:1281
 #8  0xffffffff80a05846 in vn_lock (vp=3Dvp@entry=3D0xfffffe821a9bc7e0, =

     flags=3Dflags@entry=3D131074) at /usr/src/sys/kern/vfs_vnops.c:1036=

 #9  0xffffffff80a0601d in vn_write (fp=3D<optimized out>, =

     offset=3D0xfffffe821ed94300, uio=3D0xffff80009f6c6df0, =

     cred=3D0xfffffe8220120e40, flags=3D1) at /usr/src/sys/kern/vfs_vnop=
 s.c:592
 #10 0xffffffff809b4c1d in dofilewrite (fd=3Dfd@entry=3D2, fp=3D0xfffffe=
 821ed94300, =

     buf=3D0x7f7fff0d1230, nbyte=3D102, offset=3D<optimized out>, =

     flags=3Dflags@entry=3D1, retval=3Dretval@entry=3D0xffff80009f6c6eb0=
 )
     at /usr/src/sys/kern/sys_generic.c:350
 #11 0xffffffff809b4d22 in sys_write (l=3D<optimized out>, =

     uap=3D0xffff80009f6c6f00, retval=3D0xffff80009f6c6eb0)
     at /usr/src/sys/kern/sys_generic.c:318
 #12 0xffffffff80248dbd in sy_call (rval=3D0xffff80009f6c6eb0, =

     uap=3D0xffff80009f6c6f00, l=3D0xfffffe82045da540, =

     sy=3D0xffffffff8147da40 <sysent+96>) at /usr/src/sys/sys/syscallvar=
 .h:65
 #13 sy_invoke (code=3D4, rval=3D0xffff80009f6c6eb0, uap=3D0xffff80009f6=
 c6f00, =

     l=3D0xfffffe82045da540, sy=3D0xffffffff8147da40 <sysent+96>)
     at /usr/src/sys/sys/syscallvar.h:94
 #14 syscall (frame=3D0xffff80009f6c6f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #15 0xffffffff8020768d in handle_syscall ()
 #16 0x0000000000000002 in ?? ()
 #17 0x00007f7fff0d1230 in ?? ()
 #18 0x0000000000000066 in ?? ()
 #19 0x00007f7fff0d129a in ?? ()
 #20 0x0101010101010101 in ?? ()
 #21 0x00007f7fff0d128c in ?? ()
 #22 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe8216d35140
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8216d35140)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8216d35140)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, ts@ent=
 ry=3D0x0, =

     q=3Dq@entry=3D1, obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock=
 >, =

     sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fec40 in fstrans_alloc_lwp_info (mp=3D0xfffffe822c9930=
 08)
     at /usr/src/sys/kern/vfs_trans.c:274
 #5  fstrans_get_lwp_info (do_alloc=3Dtrue, mp=3D0xfffffe822c993008)
     at /usr/src/sys/kern/vfs_trans.c:331
 #6  _fstrans_start (wait=3D1, lock_type=3DFSTRANS_SHARED, mp=3D<optimiz=
 ed out>)
     at /usr/src/sys/kern/vfs_trans.c:375
 #7  fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
 .c:414
 #8  0xffffffff80a0e399 in vop_pre (vp=3D0xfffffe82226253f0, =

     vp=3D0xfffffe82226253f0, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
 , =

     mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #9  VOP_POLL (vp=3D0xfffffe82226253f0, events=3D<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:677
 #10 0xffffffff809bd7e8 in selscan (retval=3D0xffff80009f6eaeb0, ni=3D0,=
  nfd=3D5, =

     bits=3D0xffff80009f6ead48 "\030") at /usr/src/sys/kern/sys_select.c=
 :402
 #11 sel_do_scan (op=3Dop@entry=3D1, fds=3Dfds@entry=3D0xffff80009f6ead4=
 8, =

     nf=3Dnf@entry=3D5, ni=3Dni@entry=3D4, ts=3Dts@entry=3D0x0, mask=3Dm=
 ask@entry=3D0x0, =

     retval=3Dretval@entry=3D0xffff80009f6eaeb0)
     at /usr/src/sys/kern/sys_select.c:264
 #12 0xffffffff809bdb96 in selcommon (retval=3Dretval@entry=3D0xffff8000=
 9f6eaeb0, =

     nd=3D5, u_in=3D0x7b32875fff40, u_ou=3D0x7b32875fff60, u_ex=3D0x7b32=
 875fff80, =

     ts=3D0x0, mask=3Dmask@entry=3D0x0) at /usr/src/sys/kern/sys_select.=
 c:362
 #13 0xffffffff809bdeab in selcommon (mask=3D0x0, ts=3D<optimized out>, =

     u_ex=3D<optimized out>, u_ou=3D<optimized out>, u_in=3D<optimized o=
 ut>, =

     nd=3D<optimized out>, retval=3D0xffff80009f6eaeb0)
     at /usr/src/sys/kern/sys_select.c:337
 #14 sys___select50 (l=3D<optimized out>, uap=3D0xffff80009f6eaf00, =

     retval=3D0xffff80009f6eaeb0) at /usr/src/sys/kern/sys_select.c:212
 #15 0xffffffff80248dbd in sy_call (rval=3D0xffff80009f6eaeb0, =

     uap=3D0xffff80009f6eaf00, l=3D0xfffffe8216d35140, =

     sy=3D0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscall=
 var.h:65
 #16 sy_invoke (code=3D417, rval=3D0xffff80009f6eaeb0, uap=3D0xffff80009=
 f6eaf00, =

     l=3D0xfffffe8216d35140, sy=3D0xffffffff814800f8 <sysent+10008>)
     at /usr/src/sys/sys/syscallvar.h:94
 #17 syscall (frame=3D0xffff80009f6eaf00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #18 0xffffffff8020768d in handle_syscall ()
 #19 0x0000000000000005 in ?? ()
 #20 0x00007b32875fff40 in ?? ()
 #21 0x00007b32875fff60 in ?? ()
 #22 0x00007b32875fff80 in ?? ()
 #23 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe82214bf2c0
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe82214bf2c0)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe82214bf2c0)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8094afca in cv_wait (cv=3Dcv@entry=3D0xfffffe8221620998, =

     mtx=3Dmtx@entry=3D0xfffffe8221620a00) at /usr/src/sys/kern/kern_con=
 dvar.c:217
 #3  0xffffffff8094e4e5 in fd_close (fd=3D<optimized out>)
     at /usr/src/sys/kern/kern_descrip.c:684
 #4  0xffffffff809b3c88 in sys_close (l=3D0xfffffe82214bf2c0, =

     uap=3D0xffff80009ef1df00, retval=3D<optimized out>)
     at /usr/src/sys/kern/sys_descrip.c:488
 #5  0xffffffff80248dbd in sy_call (rval=3D0xffff80009ef1deb0, =

     uap=3D0xffff80009ef1df00, l=3D0xfffffe82214bf2c0, =

     sy=3D0xffffffff8147da70 <sysent+144>) at /usr/src/sys/sys/syscallva=
 r.h:65
 #6  sy_invoke (code=3D6, rval=3D0xffff80009ef1deb0, uap=3D0xffff80009ef=
 1df00, =

     l=3D0xfffffe82214bf2c0, sy=3D0xffffffff8147da70 <sysent+144>)
     at /usr/src/sys/sys/syscallvar.h:94
 #7  syscall (frame=3D0xffff80009ef1df00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #8  0xffffffff8020768d in handle_syscall ()
 #9  0x0000000000000004 in ?? ()
 #10 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe8225709000
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8225709000)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8225709000)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =

     ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =

     obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =

     sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fef2d in _fstrans_start (wait=3D1, lock_type=3DFSTRANS=
 _SHARED, =

     mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
 #5  fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
 .c:414
 #6  0xffffffff80a0e399 in vop_pre (vp=3D0xfffffe82220a8690, =

     vp=3D0xfffffe82220a8690, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
 , =

     mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #7  VOP_POLL (vp=3D0xfffffe82220a8690, events=3D<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:677
 #8  0xffffffff809bd7e8 in selscan (retval=3D0xffff80009e90eeb0, ni=3D0,=
  nfd=3D5, =

     bits=3D0xffff80009e90ed48 "\030") at /usr/src/sys/kern/sys_select.c=
 :402
 #9  sel_do_scan (op=3Dop@entry=3D1, fds=3Dfds@entry=3D0xffff80009e90ed4=
 8, =

     nf=3Dnf@entry=3D5, ni=3Dni@entry=3D4, ts=3Dts@entry=3D0x0, mask=3Dm=
 ask@entry=3D0x0, =

     retval=3Dretval@entry=3D0xffff80009e90eeb0)
     at /usr/src/sys/kern/sys_select.c:264
 #10 0xffffffff809bdb96 in selcommon (retval=3Dretval@entry=3D0xffff8000=
 9e90eeb0, =

     nd=3D5, u_in=3D0x7aeb665fff40, u_ou=3D0x7aeb665fff60, u_ex=3D0x7aeb=
 665fff80, =

     ts=3D0x0, mask=3Dmask@entry=3D0x0) at /usr/src/sys/kern/sys_select.=
 c:362
 #11 0xffffffff809bdeab in selcommon (mask=3D0x0, ts=3D<optimized out>, =

     u_ex=3D<optimized out>, u_ou=3D<optimized out>, u_in=3D<optimized o=
 ut>, =

     nd=3D<optimized out>, retval=3D0xffff80009e90eeb0)
     at /usr/src/sys/kern/sys_select.c:337
 #12 sys___select50 (l=3D<optimized out>, uap=3D0xffff80009e90ef00, =

     retval=3D0xffff80009e90eeb0) at /usr/src/sys/kern/sys_select.c:212
 #13 0xffffffff80248dbd in sy_call (rval=3D0xffff80009e90eeb0, =

     uap=3D0xffff80009e90ef00, l=3D0xfffffe8225709000, =

     sy=3D0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscall=
 var.h:65
 #14 sy_invoke (code=3D417, rval=3D0xffff80009e90eeb0, uap=3D0xffff80009=
 e90ef00, =

     l=3D0xfffffe8225709000, sy=3D0xffffffff814800f8 <sysent+10008>)
     at /usr/src/sys/sys/syscallvar.h:94
 #15 syscall (frame=3D0xffff80009e90ef00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #16 0xffffffff8020768d in handle_syscall ()
 #17 0x0000000000000005 in ?? ()
 #18 0x00007aeb665fff40 in ?? ()
 #19 0x00007aeb665fff60 in ?? ()
 #20 0x00007aeb665fff80 in ?? ()
 #21 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe820df34480
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe820df34480)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe820df34480)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8094afca in cv_wait (cv=3Dcv@entry=3D0xfffffe8207461358, =

     mtx=3Dmtx@entry=3D0xfffffe82074613c0) at /usr/src/sys/kern/kern_con=
 dvar.c:217
 #3  0xffffffff8094e4e5 in fd_close (fd=3D<optimized out>)
     at /usr/src/sys/kern/kern_descrip.c:684
 #4  0xffffffff809b3c88 in sys_close (l=3D0xfffffe820df34480, =

     uap=3D0xffff80009f668f00, retval=3D<optimized out>)
     at /usr/src/sys/kern/sys_descrip.c:488
 #5  0xffffffff80248dbd in sy_call (rval=3D0xffff80009f668eb0, =

     uap=3D0xffff80009f668f00, l=3D0xfffffe820df34480, =

     sy=3D0xffffffff8147da70 <sysent+144>) at /usr/src/sys/sys/syscallva=
 r.h:65
 #6  sy_invoke (code=3D6, rval=3D0xffff80009f668eb0, uap=3D0xffff80009f6=
 68f00, =

     l=3D0xfffffe820df34480, sy=3D0xffffffff8147da70 <sysent+144>)
     at /usr/src/sys/sys/syscallvar.h:94
 #7  syscall (frame=3D0xffff80009f668f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #8  0xffffffff8020768d in handle_syscall ()
 #9  0x0000000000000004 in ?? ()
 #10 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe8207fd0360
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8207fd0360)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8207fd0360)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =

     ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =

     obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =

     sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fef2d in _fstrans_start (wait=3D1, lock_type=3DFSTRANS=
 _SHARED, =

     mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
 #5  fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
 .c:414
 #6  0xffffffff80a0e399 in vop_pre (vp=3D0xfffffe822122fa88, =

     vp=3D0xfffffe822122fa88, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
 , =

     mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #7  VOP_POLL (vp=3D0xfffffe822122fa88, events=3D<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:677
 #8  0xffffffff809bd7e8 in selscan (retval=3D0xffff80009f5f9eb0, ni=3D0,=
  nfd=3D5, =

     bits=3D0xffff80009f5f9d48 "\030") at /usr/src/sys/kern/sys_select.c=
 :402
 #9  sel_do_scan (op=3Dop@entry=3D1, fds=3Dfds@entry=3D0xffff80009f5f9d4=
 8, =

     nf=3Dnf@entry=3D5, ni=3Dni@entry=3D4, ts=3Dts@entry=3D0x0, mask=3Dm=
 ask@entry=3D0x0, =

     retval=3Dretval@entry=3D0xffff80009f5f9eb0)
     at /usr/src/sys/kern/sys_select.c:264
 #10 0xffffffff809bdb96 in selcommon (retval=3Dretval@entry=3D0xffff8000=
 9f5f9eb0, =

     nd=3D5, u_in=3D0x7d66527fff40, u_ou=3D0x7d66527fff60, u_ex=3D0x7d66=
 527fff80, =

     ts=3D0x0, mask=3Dmask@entry=3D0x0) at /usr/src/sys/kern/sys_select.=
 c:362
 #11 0xffffffff809bdeab in selcommon (mask=3D0x0, ts=3D<optimized out>, =

     u_ex=3D<optimized out>, u_ou=3D<optimized out>, u_in=3D<optimized o=
 ut>, =

     nd=3D<optimized out>, retval=3D0xffff80009f5f9eb0)
     at /usr/src/sys/kern/sys_select.c:337
 #12 sys___select50 (l=3D<optimized out>, uap=3D0xffff80009f5f9f00, =

     retval=3D0xffff80009f5f9eb0) at /usr/src/sys/kern/sys_select.c:212
 #13 0xffffffff80248dbd in sy_call (rval=3D0xffff80009f5f9eb0, =

     uap=3D0xffff80009f5f9f00, l=3D0xfffffe8207fd0360, =

     sy=3D0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscall=
 var.h:65
 #14 sy_invoke (code=3D417, rval=3D0xffff80009f5f9eb0, uap=3D0xffff80009=
 f5f9f00, =

     l=3D0xfffffe8207fd0360, sy=3D0xffffffff814800f8 <sysent+10008>)
     at /usr/src/sys/sys/syscallvar.h:94
 #15 syscall (frame=3D0xffff80009f5f9f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #16 0xffffffff8020768d in handle_syscall ()
 #17 0x0000000000000005 in ?? ()
 #18 0x00007d66527fff40 in ?? ()
 #19 0x00007d66527fff60 in ?? ()
 #20 0x00007d66527fff80 in ?? ()
 #21 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe822346f080
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe822346f080)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe822346f080)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8094afca in cv_wait (cv=3Dcv@entry=3D0xfffffe82243bfc18, =

     mtx=3Dmtx@entry=3D0xfffffe82243bfc80) at /usr/src/sys/kern/kern_con=
 dvar.c:217
 #3  0xffffffff8094e4e5 in fd_close (fd=3D<optimized out>)
     at /usr/src/sys/kern/kern_descrip.c:684
 #4  0xffffffff809b3c88 in sys_close (l=3D0xfffffe822346f080, =

     uap=3D0xffff80009efd4f00, retval=3D<optimized out>)
     at /usr/src/sys/kern/sys_descrip.c:488
 #5  0xffffffff80248dbd in sy_call (rval=3D0xffff80009efd4eb0, =

     uap=3D0xffff80009efd4f00, l=3D0xfffffe822346f080, =

     sy=3D0xffffffff8147da70 <sysent+144>) at /usr/src/sys/sys/syscallva=
 r.h:65
 #6  sy_invoke (code=3D6, rval=3D0xffff80009efd4eb0, uap=3D0xffff80009ef=
 d4f00, =

     l=3D0xfffffe822346f080, sy=3D0xffffffff8147da70 <sysent+144>)
     at /usr/src/sys/sys/syscallvar.h:94
 #7  syscall (frame=3D0xffff80009efd4f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #8  0xffffffff8020768d in handle_syscall ()
 #9  0x0000000000000004 in ?? ()
 #10 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe821d987a40
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821d987a40)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821d987a40)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =

     ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =

     obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =

     sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fef2d in _fstrans_start (wait=3D1, lock_type=3DFSTRANS=
 _SHARED, =

     mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
 #5  fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
 .c:414
 #6  0xffffffff80a0e399 in vop_pre (vp=3D0xfffffe820ecde930, =

     vp=3D0xfffffe820ecde930, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
 , =

     mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #7  VOP_POLL (vp=3D0xfffffe820ecde930, events=3D<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:677
 #8  0xffffffff809bd7e8 in selscan (retval=3D0xffff80009ee64eb0, ni=3D0,=
  nfd=3D5, =

     bits=3D0xffff80009ee64d48 "\030") at /usr/src/sys/kern/sys_select.c=
 :402
 #9  sel_do_scan (op=3Dop@entry=3D1, fds=3Dfds@entry=3D0xffff80009ee64d4=
 8, =

     nf=3Dnf@entry=3D5, ni=3Dni@entry=3D4, ts=3Dts@entry=3D0x0, mask=3Dm=
 ask@entry=3D0x0, =

     retval=3Dretval@entry=3D0xffff80009ee64eb0)
     at /usr/src/sys/kern/sys_select.c:264
 #10 0xffffffff809bdb96 in selcommon (retval=3Dretval@entry=3D0xffff8000=
 9ee64eb0, =

     nd=3D5, u_in=3D0x7ea7fb7fff40, u_ou=3D0x7ea7fb7fff60, u_ex=3D0x7ea7=
 fb7fff80, =

     ts=3D0x0, mask=3Dmask@entry=3D0x0) at /usr/src/sys/kern/sys_select.=
 c:362
 #11 0xffffffff809bdeab in selcommon (mask=3D0x0, ts=3D<optimized out>, =

     u_ex=3D<optimized out>, u_ou=3D<optimized out>, u_in=3D<optimized o=
 ut>, =

     nd=3D<optimized out>, retval=3D0xffff80009ee64eb0)
     at /usr/src/sys/kern/sys_select.c:337
 #12 sys___select50 (l=3D<optimized out>, uap=3D0xffff80009ee64f00, =

     retval=3D0xffff80009ee64eb0) at /usr/src/sys/kern/sys_select.c:212
 #13 0xffffffff80248dbd in sy_call (rval=3D0xffff80009ee64eb0, =

     uap=3D0xffff80009ee64f00, l=3D0xfffffe821d987a40, =

     sy=3D0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscall=
 var.h:65
 #14 sy_invoke (code=3D417, rval=3D0xffff80009ee64eb0, uap=3D0xffff80009=
 ee64f00, =

     l=3D0xfffffe821d987a40, sy=3D0xffffffff814800f8 <sysent+10008>)
     at /usr/src/sys/sys/syscallvar.h:94
 #15 syscall (frame=3D0xffff80009ee64f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #16 0xffffffff8020768d in handle_syscall ()
 #17 0x0000000000000005 in ?? ()
 #18 0x00007ea7fb7fff40 in ?? ()
 #19 0x00007ea7fb7fff60 in ?? ()
 #20 0x00007ea7fb7fff80 in ?? ()
 #21 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe821e90c240
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821e90c240)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821e90c240)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8094afca in cv_wait (cv=3Dcv@entry=3D0xfffffe822400e918, =

     mtx=3Dmtx@entry=3D0xfffffe822400e980) at /usr/src/sys/kern/kern_con=
 dvar.c:217
 #3  0xffffffff8094e4e5 in fd_close (fd=3D<optimized out>)
     at /usr/src/sys/kern/kern_descrip.c:684
 #4  0xffffffff809b3c88 in sys_close (l=3D0xfffffe821e90c240, =

     uap=3D0xffff80009ee94f00, retval=3D<optimized out>)
     at /usr/src/sys/kern/sys_descrip.c:488
 #5  0xffffffff80248dbd in sy_call (rval=3D0xffff80009ee94eb0, =

     uap=3D0xffff80009ee94f00, l=3D0xfffffe821e90c240, =

     sy=3D0xffffffff8147da70 <sysent+144>) at /usr/src/sys/sys/syscallva=
 r.h:65
 #6  sy_invoke (code=3D6, rval=3D0xffff80009ee94eb0, uap=3D0xffff80009ee=
 94f00, =

     l=3D0xfffffe821e90c240, sy=3D0xffffffff8147da70 <sysent+144>)
     at /usr/src/sys/sys/syscallvar.h:94
 #7  syscall (frame=3D0xffff80009ee94f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #8  0xffffffff8020768d in handle_syscall ()
 #9  0x0000000000000004 in ?? ()
 #10 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe821d362980
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821d362980)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe821d362980)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =

     ts@entry=3D0xfffffe821f9db480, q=3Dq@entry=3D1, =

     obj=3Dobj@entry=3D0xfffffe8224a7b130, =

     sobj=3Dsobj@entry=3D0xffffffff81482720 <rw_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80977cb3 in rw_vector_enter (rw=3D0xfffffe8224a7b130, op=3D=
 RW_WRITER)
     at /usr/src/sys/kern/kern_rwlock.c:395
 #4  0xffffffff80a17d7e in genfs_lock (v=3D<optimized out>)
     at /usr/src/sys/miscfs/genfs/genfs_vnops.c:343
 #5  0xffffffff80a0ed35 in VOP_LOCK (vp=3Dvp@entry=3D0xfffffe8224a7b000,=
  =

     flags=3Dflags@entry=3D131074) at /usr/src/sys/kern/vnode_if.c:1284
 #6  0xffffffff80a05846 in vn_lock (vp=3Dvp@entry=3D0xfffffe8224a7b000, =

     flags=3Dflags@entry=3D131074) at /usr/src/sys/kern/vfs_vnops.c:1036=

 #7  0xffffffff80a0601d in vn_write (fp=3D<optimized out>, =

     offset=3D0xfffffe8225588b00, uio=3D0xffff80009edaedf0, =

     cred=3D0xfffffe822f724f00, flags=3D1) at /usr/src/sys/kern/vfs_vnop=
 s.c:592
 #8  0xffffffff809b4c1d in dofilewrite (fd=3Dfd@entry=3D2, fp=3D0xfffffe=
 8225588b00, =

     buf=3D0x764fb6902000, nbyte=3D94, offset=3D<optimized out>, flags=3D=
 flags@entry=3D1, =

     retval=3Dretval@entry=3D0xffff80009edaeeb0)
     at /usr/src/sys/kern/sys_generic.c:350
 #9  0xffffffff809b4d22 in sys_write (l=3D<optimized out>, =

     uap=3D0xffff80009edaef00, retval=3D0xffff80009edaeeb0)
     at /usr/src/sys/kern/sys_generic.c:318
 #10 0xffffffff80248dbd in sy_call (rval=3D0xffff80009edaeeb0, =

     uap=3D0xffff80009edaef00, l=3D0xfffffe821d362980, =

     sy=3D0xffffffff8147da40 <sysent+96>) at /usr/src/sys/sys/syscallvar=
 .h:65
 #11 sy_invoke (code=3D4, rval=3D0xffff80009edaeeb0, uap=3D0xffff80009ed=
 aef00, =

     l=3D0xfffffe821d362980, sy=3D0xffffffff8147da40 <sysent+96>)
     at /usr/src/sys/sys/syscallvar.h:94
 #12 syscall (frame=3D0xffff80009edaef00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #13 0xffffffff8020768d in handle_syscall ()
 #14 0x0000000000000002 in ?? ()
 #15 0x0000764fb6902000 in ?? ()
 #16 0x000000000000005e in ?? ()
 #17 0x0000764fb690205e in ?? ()
 #18 0x000000000000001f in ?? ()
 #19 0x0000000000000001 in ?? ()
 #20 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe8229b924c0
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8229b924c0)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8229b924c0)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =

     ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =

     obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =

     sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fef2d in _fstrans_start (wait=3D1, lock_type=3DFSTRANS=
 _SHARED, =

     mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
 #5  fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
 .c:414
 #6  0xffffffff80a0e399 in vop_pre (vp=3D0xfffffe821d7d1690, =

     vp=3D0xfffffe821d7d1690, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
 , =

     mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #7  VOP_POLL (vp=3D0xfffffe821d7d1690, events=3D<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:677
 #8  0xffffffff809bd7e8 in selscan (retval=3D0xffff80009e9c3eb0, ni=3D36=
 39083008, =

     nfd=3D48, bits=3D0xffff80009e9c3d48 "\b@\350\330(")
     at /usr/src/sys/kern/sys_select.c:402
 #9  sel_do_scan (op=3Dop@entry=3D1, fds=3Dfds@entry=3D0xffff80009e9c3d4=
 8, =

     nf=3Dnf@entry=3D48, ni=3Dni@entry=3D8, ts=3Dts@entry=3D0xffff80009e=
 9c3e70, =

     mask=3Dmask@entry=3D0x0, retval=3Dretval@entry=3D0xffff80009e9c3eb0=
 )
     at /usr/src/sys/kern/sys_select.c:264
 #10 0xffffffff809bdb96 in selcommon (retval=3Dretval@entry=3D0xffff8000=
 9e9c3eb0, =

     nd=3D48, u_in=3D0x7f7fff4c5a10, u_ou=3D0x7f7fff4c5a30, u_ex=3D0x0, =

     ts=3D0xffff80009e9c3e70, mask=3Dmask@entry=3D0x0)
     at /usr/src/sys/kern/sys_select.c:362
 #11 0xffffffff809bdeab in selcommon (mask=3D0x0, ts=3D<optimized out>, =

     u_ex=3D<optimized out>, u_ou=3D<optimized out>, u_in=3D<optimized o=
 ut>, =

     nd=3D<optimized out>, retval=3D0xffff80009e9c3eb0)
     at /usr/src/sys/kern/sys_select.c:337
 #12 sys___select50 (l=3D<optimized out>, uap=3D0xffff80009e9c3f00, =

     retval=3D0xffff80009e9c3eb0) at /usr/src/sys/kern/sys_select.c:212
 #13 0xffffffff80248dbd in sy_call (rval=3D0xffff80009e9c3eb0, =

     uap=3D0xffff80009e9c3f00, l=3D0xfffffe8229b924c0, =

     sy=3D0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscall=
 var.h:65
 #14 sy_invoke (code=3D417, rval=3D0xffff80009e9c3eb0, uap=3D0xffff80009=
 e9c3f00, =

     l=3D0xfffffe8229b924c0, sy=3D0xffffffff814800f8 <sysent+10008>)
     at /usr/src/sys/sys/syscallvar.h:94
 #15 syscall (frame=3D0xffff80009e9c3f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #16 0xffffffff8020768d in handle_syscall ()
 #17 0x0000000000000030 in ?? ()
 #18 0x00007f7fff4c5a10 in ?? ()
 #19 0x00007f7fff4c5a30 in ?? ()
 #20 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe8220b8e780
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8220b8e780)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8220b8e780)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, ts@ent=
 ry=3D0x0, =

     q=3Dq@entry=3D1, obj=3Dobj@entry=3D0xfffffe8224a7b130, =

     sobj=3Dsobj@entry=3D0xffffffff81482720 <rw_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80977cb3 in rw_vector_enter (rw=3D0xfffffe8224a7b130, op=3D=
 RW_WRITER)
     at /usr/src/sys/kern/kern_rwlock.c:395
 #4  0xffffffff80a17d7e in genfs_lock (v=3D<optimized out>)
     at /usr/src/sys/miscfs/genfs/genfs_vnops.c:343
 #5  0xffffffff80a0ed35 in VOP_LOCK (vp=3Dvp@entry=3D0xfffffe8224a7b000,=
  =

     flags=3Dflags@entry=3D131074) at /usr/src/sys/kern/vnode_if.c:1284
 #6  0xffffffff80a05846 in vn_lock (vp=3Dvp@entry=3D0xfffffe8224a7b000, =

     flags=3Dflags@entry=3D131074) at /usr/src/sys/kern/vfs_vnops.c:1036=

 #7  0xffffffff80a0601d in vn_write (fp=3D<optimized out>, =

     offset=3D0xfffffe8225588b00, uio=3D0xffff80009ebf8df0, =

     cred=3D0xfffffe822f724f00, flags=3D1) at /usr/src/sys/kern/vfs_vnop=
 s.c:592
 #8  0xffffffff809b4c1d in dofilewrite (fd=3Dfd@entry=3D2, fp=3D0xfffffe=
 8225588b00, =

     buf=3D0x764fb6902000, nbyte=3D90, offset=3D<optimized out>, flags=3D=
 flags@entry=3D1, =

     retval=3Dretval@entry=3D0xffff80009ebf8eb0)
     at /usr/src/sys/kern/sys_generic.c:350
 #9  0xffffffff809b4d22 in sys_write (l=3D<optimized out>, =

     uap=3D0xffff80009ebf8f00, retval=3D0xffff80009ebf8eb0)
     at /usr/src/sys/kern/sys_generic.c:318
 #10 0xffffffff80248dbd in sy_call (rval=3D0xffff80009ebf8eb0, =

     uap=3D0xffff80009ebf8f00, l=3D0xfffffe8220b8e780, =

     sy=3D0xffffffff8147da40 <sysent+96>) at /usr/src/sys/sys/syscallvar=
 .h:65
 #11 sy_invoke (code=3D4, rval=3D0xffff80009ebf8eb0, uap=3D0xffff80009eb=
 f8f00, =

     l=3D0xfffffe8220b8e780, sy=3D0xffffffff8147da40 <sysent+96>)
     at /usr/src/sys/sys/syscallvar.h:94
 #12 syscall (frame=3D0xffff80009ebf8f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #13 0xffffffff8020768d in handle_syscall ()
 #14 0x0000000000000002 in ?? ()
 #15 0x0000764fb6902000 in ?? ()
 #16 0x000000000000005a in ?? ()
 #17 0x0000764fb690205a in ?? ()
 #18 0x000000000000001e in ?? ()
 #19 0x0000000000000001 in ?? ()
 #20 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe8221e57600
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8221e57600)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8221e57600)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =

     ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =

     obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =

     sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fef2d in _fstrans_start (wait=3D1, lock_type=3DFSTRANS=
 _SHARED, =

     mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
 #5  fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
 .c:414
 #6  0xffffffff80a0e399 in vop_pre (vp=3D0xfffffe8221e36bd0, =

     vp=3D0xfffffe8221e36bd0, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
 , =

     mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #7  VOP_POLL (vp=3D0xfffffe8221e36bd0, events=3D<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:677
 #8  0xffffffff809bd7e8 in selscan (retval=3D0xffff80009eae8eb0, ni=3D20=
 46820352, =

     nfd=3D52, bits=3D0xffff80009eae8d48 "H") at /usr/src/sys/kern/sys_s=
 elect.c:402
 #9  sel_do_scan (op=3Dop@entry=3D1, fds=3Dfds@entry=3D0xffff80009eae8d4=
 8, =

     nf=3Dnf@entry=3D52, ni=3Dni@entry=3D8, ts=3Dts@entry=3D0xffff80009e=
 ae8e70, =

     mask=3Dmask@entry=3D0x0, retval=3Dretval@entry=3D0xffff80009eae8eb0=
 )
     at /usr/src/sys/kern/sys_select.c:264
 #10 0xffffffff809bdb96 in selcommon (retval=3Dretval@entry=3D0xffff8000=
 9eae8eb0, =

     nd=3D52, u_in=3D0x7f7fff4c5a10, u_ou=3D0x7f7fff4c5a30, u_ex=3D0x0, =

     ts=3D0xffff80009eae8e70, mask=3Dmask@entry=3D0x0)
     at /usr/src/sys/kern/sys_select.c:362
 #11 0xffffffff809bdeab in selcommon (mask=3D0x0, ts=3D<optimized out>, =

     u_ex=3D<optimized out>, u_ou=3D<optimized out>, u_in=3D<optimized o=
 ut>, =

     nd=3D<optimized out>, retval=3D0xffff80009eae8eb0)
     at /usr/src/sys/kern/sys_select.c:337
 #12 sys___select50 (l=3D<optimized out>, uap=3D0xffff80009eae8f00, =

     retval=3D0xffff80009eae8eb0) at /usr/src/sys/kern/sys_select.c:212
 #13 0xffffffff80248dbd in sy_call (rval=3D0xffff80009eae8eb0, =

     uap=3D0xffff80009eae8f00, l=3D0xfffffe8221e57600, =

     sy=3D0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscall=
 var.h:65
 #14 sy_invoke (code=3D417, rval=3D0xffff80009eae8eb0, uap=3D0xffff80009=
 eae8f00, =

     l=3D0xfffffe8221e57600, sy=3D0xffffffff814800f8 <sysent+10008>)
     at /usr/src/sys/sys/syscallvar.h:94
 #15 syscall (frame=3D0xffff80009eae8f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #16 0xffffffff8020768d in handle_syscall ()
 #17 0x0000000000000034 in ?? ()
 #18 0x00007f7fff4c5a10 in ?? ()
 #19 0x00007f7fff4c5a30 in ?? ()
 #20 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe8222569900
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8222569900)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8222569900)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =

     ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =

     obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =

     sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fef2d in _fstrans_start (wait=3D1, lock_type=3DFSTRANS=
 _SHARED, =

     mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
 #5  fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
 .c:414
 #6  0xffffffff80a0e399 in vop_pre (vp=3D0xfffffe8223aa62a0, =

     vp=3D0xfffffe8223aa62a0, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
 , =

     mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #7  VOP_POLL (vp=3D0xfffffe8223aa62a0, events=3D<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:677
 #8  0xffffffff809bd7e8 in selscan (retval=3D0xffff80009e9cdeb0, ni=3D79=
 6917760, =

     nfd=3D53, bits=3D0xffff80009e9cdd48 "\b") at /usr/src/sys/kern/sys_=
 select.c:402
 #9  sel_do_scan (op=3Dop@entry=3D1, fds=3Dfds@entry=3D0xffff80009e9cdd4=
 8, =

     nf=3Dnf@entry=3D53, ni=3Dni@entry=3D8, ts=3Dts@entry=3D0xffff80009e=
 9cde70, =

     mask=3Dmask@entry=3D0x0, retval=3Dretval@entry=3D0xffff80009e9cdeb0=
 )
     at /usr/src/sys/kern/sys_select.c:264
 #10 0xffffffff809bdb96 in selcommon (retval=3Dretval@entry=3D0xffff8000=
 9e9cdeb0, =

     nd=3D53, u_in=3D0x7f7fff4c5a10, u_ou=3D0x7f7fff4c5a30, u_ex=3D0x0, =

     ts=3D0xffff80009e9cde70, mask=3Dmask@entry=3D0x0)
     at /usr/src/sys/kern/sys_select.c:362
 #11 0xffffffff809bdeab in selcommon (mask=3D0x0, ts=3D<optimized out>, =

     u_ex=3D<optimized out>, u_ou=3D<optimized out>, u_in=3D<optimized o=
 ut>, =

     nd=3D<optimized out>, retval=3D0xffff80009e9cdeb0)
     at /usr/src/sys/kern/sys_select.c:337
 #12 sys___select50 (l=3D<optimized out>, uap=3D0xffff80009e9cdf00, =

     retval=3D0xffff80009e9cdeb0) at /usr/src/sys/kern/sys_select.c:212
 #13 0xffffffff80248dbd in sy_call (rval=3D0xffff80009e9cdeb0, =

     uap=3D0xffff80009e9cdf00, l=3D0xfffffe8222569900, =

     sy=3D0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscall=
 var.h:65
 #14 sy_invoke (code=3D417, rval=3D0xffff80009e9cdeb0, uap=3D0xffff80009=
 e9cdf00, =

     l=3D0xfffffe8222569900, sy=3D0xffffffff814800f8 <sysent+10008>)
     at /usr/src/sys/sys/syscallvar.h:94
 #15 syscall (frame=3D0xffff80009e9cdf00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #16 0xffffffff8020768d in handle_syscall ()
 #17 0x0000000000000035 in ?? ()
 #18 0x00007f7fff4c5a10 in ?? ()
 #19 0x00007f7fff4c5a30 in ?? ()
 #20 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe8225709840
 0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8225709840)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp =3D cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=3Dl@entry=3D0xfffffe8225709840)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=3Dtimo@entry=3D0, =

     catch_p=3Dcatch_p@entry=3Dfalse) at /usr/src/sys/kern/kern_sleepq.c=
 :264
 #2  0xffffffff8098b761 in turnstile_block (ts=3D<optimized out>, =

     ts@entry=3D0xfffffe82123900c8, q=3Dq@entry=3D1, =

     obj=3Dobj@entry=3D0xffffffff815cb9b0 <fstrans_lock>, =

     sobj=3Dsobj@entry=3D0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=3D0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fef2d in _fstrans_start (wait=3D1, lock_type=3DFSTRANS=
 _SHARED, =

     mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
 #5  fstrans_start (mp=3D<optimized out>) at /usr/src/sys/kern/vfs_trans=
 .c:414
 #6  0xffffffff80a0e399 in vop_pre (vp=3D0xfffffe8222fe4bd8, =

     vp=3D0xfffffe8222fe4bd8, op=3DFST_YES, mpsafe=3D<synthetic pointer>=
 , =

     mp=3D<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #7  VOP_POLL (vp=3D0xfffffe8222fe4bd8, events=3D<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:677
 #8  0xffffffff809bd7e8 in selscan (retval=3D0xffff80009e904eb0, ni=3D26=
 25634304, =

     nfd=3D53, bits=3D0xffff80009e904d48 "\b") at /usr/src/sys/kern/sys_=
 select.c:402
 #9  sel_do_scan (op=3Dop@entry=3D1, fds=3Dfds@entry=3D0xffff80009e904d4=
 8, =

     nf=3Dnf@entry=3D53, ni=3Dni@entry=3D8, ts=3Dts@entry=3D0xffff80009e=
 904e70, =

     mask=3Dmask@entry=3D0x0, retval=3Dretval@entry=3D0xffff80009e904eb0=
 )
     at /usr/src/sys/kern/sys_select.c:264
 #10 0xffffffff809bdb96 in selcommon (retval=3Dretval@entry=3D0xffff8000=
 9e904eb0, =

     nd=3D53, u_in=3D0x7f7fff4c5a10, u_ou=3D0x7f7fff4c5a30, u_ex=3D0x0, =

     ts=3D0xffff80009e904e70, mask=3Dmask@entry=3D0x0)
     at /usr/src/sys/kern/sys_select.c:362
 #11 0xffffffff809bdeab in selcommon (mask=3D0x0, ts=3D<optimized out>, =

     u_ex=3D<optimized out>, u_ou=3D<optimized out>, u_in=3D<optimized o=
 ut>, =

     nd=3D<optimized out>, retval=3D0xffff80009e904eb0)
     at /usr/src/sys/kern/sys_select.c:337
 #12 sys___select50 (l=3D<optimized out>, uap=3D0xffff80009e904f00, =

     retval=3D0xffff80009e904eb0) at /usr/src/sys/kern/sys_select.c:212
 #13 0xffffffff80248dbd in sy_call (rval=3D0xffff80009e904eb0, =

     uap=3D0xffff80009e904f00, l=3D0xfffffe8225709840, =

     sy=3D0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscall=
 var.h:65
 #14 sy_invoke (code=3D417, rval=3D0xffff80009e904eb0, uap=3D0xffff80009=
 e904f00, =

     l=3D0xfffffe8225709840, sy=3D0xffffffff814800f8 <sysent+10008>)
     at /usr/src/sys/sys/syscallvar.h:94
 #15 syscall (frame=3D0xffff80009e904f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #16 0xffffffff8020768d in handle_syscall ()
 #17 0x0000000000000035 in ?? ()
 #18 0x00007f7fff4c5a10 in ?? ()
 #19 0x00007f7fff4c5a30 in ?? ()
 #20 0x0000000000000000 in ?? ()
 (gdb)

From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Wed, 16 May 2018 13:37:28 +0200 (CEST)

 [[ Hmm, here's a non-quoted-printable version of the previous ]]

 Hi,

 we provoked another wedge, and captured a kernel core dump from
 the wedging machine.  The kernel core dump and NetBSD images are
 available for looking at.


 using gdb and crash, here's a brief summary of the "interesting"
 processes in the crash dump, and below that I include backtraces
 of all the waiting processes.

 There's lots of contention for fstrans_lock.

 Offhand I don't see a deadlock which might explain the observed
 behaviour (goes totally "deaf" on the network, i.e. doesn't even
 respond to ping).

 It doesn't look like gdb can trace through interrupt frames (?),
 looking at proc 788 all I get is:

 (gdb) kvm proc 0xfffffe8220b8e360
 0xffffffff8021cfe0 in softintr_ret ()
 (gdb) where
 #0  0xffffffff8021cfe0 in softintr_ret ()
 #1  0x0000000000000000 in ?? ()
 (gdb) 

 Crash manages to do this one, though, apparently (see below).

 Furthermore, using the various gdb scripts in
 /usr/src/sys/gdbscripts/ I can look at some of the locks.

 It is quite possible that up'ing the interface in question causes
 lots of activity for opening pty pairs, and that the root cause
 of the issue is there rather than related to networking in itself(?)

 Further hints?

 Regards,

 - Havard

 ------------------------------

 PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 7052     1 3   1   8020000   fffffe8220c58540               cron tstile
   Wants fstrans_lock

 9187     1 3   6   8020000   fffffe821faa60c0             expect xchicv
   Holds fstrans_lock, in pserialize_perform, waits on condition variable
     after doing xc_broadcast(XC_HIGHPRI, nullop)
   Doing (roughly) pty_grant_slave -> genfs_revoke -> vfs_suspend -> 
     fstrans_setstate -> pserialize_perform -> xc_wait -> cv_wait

 3794     1 3   7   8020000   fffffe82045da540                ssh tstile
   Wants fstrans_lock

 2853     2 3   6   8020000   fffffe8216d35140             expect tstile
   Wants fstrans_lock

 2853     1 3   7   8020000   fffffe82214bf2c0             expect fdclose
   Doing cv_wait(&ff->ff_closing, &fdp->fd_lock);

 8327     2 3   3   8020000   fffffe8225709000             expect tstile
   Wants fstrans_lock

 8327     1 3   1   8020000   fffffe820df34480             expect fdclose
   Doing cv_wait(&ff->ff_closing, &fdp->fd_lock);

 5687     2 3   3   8020000   fffffe8207fd0360             expect tstile
   Wants fstrans_lock

 5687     1 3   2   8020000   fffffe822346f080             expect fdclose
   Doing cv_wait(&ff->ff_closing, &fdp->fd_lock);

 9671     2 3   5   8020000   fffffe821d987a40             expect tstile
   Wants fstrans_lock

 9671     1 3   4   8020000   fffffe821e90c240             expect fdclose
   Doing cv_wait(&ff->ff_closing, &fdp->fd_lock);

 704      1 3   6   8020000   fffffe821d362980          conserver tstile
   Wants rw_enter(&vip->vi_lock, op);, lock held by 788 apparently

 564      1 3   1   8020000   fffffe8229b924c0          conserver tstile
   Wants fstrans_lock

 788  >   1 7   0   8020000   fffffe8220b8e360          conserver
   Doing softintr_ret(), gdb doesn't trace further (interrupt frame?)
   Crash, though, says:

 crash> trace/a 0xfffffe8220b8e360
 trace: pid 788 lid 1 at 0xffff80009ebfdc40
 ufs_post_write_update.isra.0() at ufs_post_write_update.isra.0+0x95
 ffs_write() at ffs_write+0x56f
 VOP_WRITE() at VOP_WRITE+0x37
 vn_write() at vn_write+0xec
 dofilewrite() at dofilewrite+0x97
 sys_write() at sys_write+0x5f
 syscall() at syscall+0x1ed
 --- syscall (number 4) ---
 764fb503e02a:
 crash> 

 275      1 3   5   8020000   fffffe8220b8e780          conserver tstile
   Wants rw_enter(&vip->vi_lock, op);, lock held by 788 apparently

 878      1 3   2   8020000   fffffe8221e57600          conserver tstile
   Wants fstrans_lock

 136      1 3   3   8020000   fffffe8222569900          conserver tstile
   Wants fstrans_lock

 773      1 3   1   8020000   fffffe8225709840          conserver tstile
   Wants fstrans_lock

 ------------------------------

 crash> ps
 PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 7052     1 3   1   8020000   fffffe8220c58540               cron tstile
 8587     1 3   1   8020000   fffffe821d362560               cron wait
 9187     1 3   6   8020000   fffffe821faa60c0             expect xchicv
 5064     1 2   6   8020000   fffffe8219f93540                 sh
 7813     1 2   0   8020000   fffffe8220ecf860               perl
 7354     1 2   0   8020000   fffffe8207aba240                 sh
 3794     1 3   7   8020000   fffffe82045da540                ssh tstile
 2853     2 3   6   8020000   fffffe8216d35140             expect tstile
 2853     1 3   7   8020000   fffffe82214bf2c0             expect fdclose
 6097     1 2   0   8020000   fffffe8221165720                 sh
 5738     1 2   0   8020000   fffffe821e90ca80               perl
 4461     1 2   0   8020000   fffffe8207a90aa0                 sh
 4310     1 2   3   8020000   fffffe8220c2c5a0                ssh
 8327     2 3   3   8020000   fffffe8225709000             expect tstile
 8327     1 3   1   8020000   fffffe820df34480             expect fdclose
 6086     1 2   5   8020000   fffffe821d2d9a60                 sh
 3214     1 2   0   8020000   fffffe821f5d8360               perl
 5921     1 2   0   8020000   fffffe8207a90260                 sh
 2890     1 2   2   8020000   fffffe8228bacae0                ssh
 3772     1 2   0   8020000   fffffe821a27f420                 sh
 4425     1 2   0   8020000   fffffe821f5d8780                ssh
 5793     1 2   0   8020000   fffffe821d342340                 sh
 4833     1 2   0   8020000   fffffe82207f5460                ssh
 7238     1 2   0   8020000   fffffe822123b640                 sh
 6312     1 2   4   8020000   fffffe8222a9b500                ssh
 7003     1 2   0   8020000   fffffe822160e700                 sh
 6900     1 2   0   8020000   fffffe8221d82140                ssh
 4139     1 2   0   8020000   fffffe822346f8c0                 sh
 3765     1 2   0   8020000   fffffe821f94f060                ssh
 5386     1 2   1   8020000   fffffe8222a7a8c0                 sh
 4633     1 2   0   8020000   fffffe821f8969c0                ssh
 9691     1 2   0   8020000   fffffe821928d180                 sh
 6528     1 2   7   8020000   fffffe8207f4b460                ssh
 5816     1 2   0   8020000   fffffe8207f4b880                 sh
 6285     1 2   0   8020000   fffffe821e1e28c0                ssh
 8366     1 2   0   8020000   fffffe820d267520                 sh
 8041     1 2   0   8020000   fffffe821f94f8a0                ssh
 3407     1 2   0   8020000   fffffe82045da960                 sh
 2815     1 2   0   8020000   fffffe8225709420                ssh
 5687     2 3   3   8020000   fffffe8207fd0360             expect tstile
 5687     1 3   2   8020000   fffffe822346f080             expect fdclose
 2928     1 2   0   8020000   fffffe8228bac2a0                 sh
 5127     1 2   0   8020000   fffffe821b498860               perl
 5658     1 2   0   8020000   fffffe820ea5eac0                 sh
 6746     1 2   0   8020000   fffffe821d443520                ssh
 9671     2 3   5   8020000   fffffe821d987a40             expect tstile
 9671     1 3   4   8020000   fffffe821e90c240             expect fdclose
 9105     1 2   0   8020000   fffffe8226ea8b40                 sh
 3990     1 2   0   8020000   fffffe8227069700               perl
 4504     1 2   0   8020000   fffffe8207a90680                 sh
 9019     1 2   0   8020000   fffffe821f4c78e0                ssh
 2825     1 2   0   8020000   fffffe821f896180                 sh
 5779     1 2   0   8020000   fffffe821a27f840                ssh
 2846     1 2   0   8020000   fffffe822af57440                 sh
 3544     1 2   0   8020000   fffffe82216b1a40                ssh
 6513     1 2   0   8020000   fffffe822728b460                 sh
 5965     1 2   0   8020000   fffffe821faa6900                ssh
 6327     1 2   0   8020000   fffffe8221837a80                 sh
 2739     1 2   0   8020000   fffffe822708e740                ssh
 6361     1 2   0   8020000   fffffe82239772a0                 sh
 3534     1 2   0   8020000   fffffe821f94f480                ssh
 5988     1 2   0   8020000   fffffe8222a7a080                 sh
 7283     1 2   0   8020000   fffffe8222b99980                ssh
 6596     1 2   0   8020000   fffffe820ea5e6a0                 sh
 3343     1 2   0   8020000   fffffe8223049740                ssh
 7708     1 2   0   8020000   fffffe822da31140                 sh
 6381     1 2   0   8020000   fffffe8221d82980                ssh
 7056     1 2   0   8020000   fffffe821a542500                 sh
 4161     1 2   0   8020000   fffffe821c708100                ssh
 7144     1 2   0   8020000   fffffe8223049b60                 sh
 6276     1 2   0   8020000   fffffe821d2d9220                ssh
 4460     1 2   0   8020000   fffffe821f36a040                 sh
 6845     1 2   0   8020000   fffffe8222a9b0e0                ssh
 4669     1 2   0   8020000   fffffe821e1e24a0                 sh
 3652     1 2   0   8020000   fffffe821c708520                ssh
 5131     1 2   0   8020000   fffffe820df34060                 sh
 5704     1 2   0   8020000   fffffe8220ce6760                ssh
 6031     1 2   0   8020000   fffffe821ecaea00                 sh
 5875     1 2   0   8020000   fffffe82240071a0                ssh
 5464     1 2   0   8020000   fffffe8205110580                 sh
 2852     1 2   0   8020000   fffffe82207f5040                ssh
 3432     1 2   0   8020000   fffffe820dd5e6e0                 sh
 4917     1 2   0   8020000   fffffe8221317ac0                ssh
 2850     1 2   0   8020000   fffffe8207fd0780                 sh
 3399     1 2   0   8020000   fffffe822d92e9c0                ssh
 3497     1 2   0   8020000   fffffe821d987200                 sh
 3880     1 2   0   8020000   fffffe821d342b80                ssh
 4728     1 2   0   8020000   fffffe822188a5a0                 sh
 3904     1 2   0   8020000   fffffe820df348a0                ssh
 4847     1 2   0   8020000   fffffe8204072900                 sh
 3465     1 2   0   8020000   fffffe8207fd0ba0                ssh
 3881     1 2   0   8020000   fffffe821a27f000                 sh
 4000     1 2   0   8020000   fffffe821928d9c0                ssh
 4242     1 2   0   8020000   fffffe8221165300                 sh
 4074     1 2   0   8020000   fffffe82207f5880                ssh
 3902     1 2   0   8020000   fffffe8205110160                 sh
 3293     1 2   0   8020000   fffffe8224d918a0                ssh
 3228     1 2   0   8020000   fffffe821d362140                 sh
 4064     1 2   0   8020000   fffffe8224b0c760                ssh
 4762     1 2   0   8020000   fffffe822037f1a0                 sh
 2903     1 2   0   8020000   fffffe8221dab1a0                ssh
 2218     1 2   0   8020000   fffffe821fdf9b00                 sh
 2497     1 2   0   8020000   fffffe822728b880                ssh
 3080     1 2   0   8020000   fffffe8223563b40                 sh
 2573     1 2   0   8020000   fffffe8206fea440                ssh
 2635     1 2   0   8020000   fffffe822af57020                 sh
 2114     1 2   0   8020000   fffffe82220aa520                ssh
 4141     1 2   0   8020000   fffffe82240079e0                 sh
 4477     1 2   0   8020000   fffffe822160eb20                ssh
 3978     1 2   0   8020000   fffffe821f6f9b60                 sh
 3243     1 2   0   8020000   fffffe821f1b0680                ssh
 2652     1 2   0   8020000   fffffe821c49f5e0                 sh
 3804     1 2   0   8020000   fffffe821f37d9a0                ssh
 4348     1 2   0   8020000   fffffe822c1e7660                 sh
 4334     1 2   0   8020000   fffffe8220c2c9c0                ssh
 3781     1 2   0   8020000   fffffe82214bfb00                 sh
 3422     1 2   0   8020000   fffffe82216b1200                ssh
 3277     1 2   0   8020000   fffffe8221165b40                 sh
 3997     1 2   0   8020000   fffffe8206fea020                ssh
 3863     1 2   0   8020000   fffffe82239776c0                 sh
 2950     1 2   0   8020000   fffffe821f6f9320                par
 2644     1 2   0   8020000   fffffe822220f9a0                ssh
 2418     1 2   0   8020000   fffffe822253d1c0                 sh
 2616     1 2   0   8020000   fffffe821e1e2080                ssh
 2724     1 2   0   8020000   fffffe821b0306a0                 sh
 1968     1 2   0   8020000   fffffe8224d91060                 sh
 3042     1 2   0   8020000   fffffe8221dab5c0                 sh
 2259     1 2   0   8020000   fffffe821c4bb4e0                 sh
 2733     1 2   0   8020000   fffffe8223c97a40                 sh
 2608     1 2   0   8020000   fffffe821b498020               cron
 2402     1 2   0   8020000   fffffe822123ba60                ssh
 2346     1 2   0   8020000   fffffe821f6f9740                 sh
 3141     1 2   0   8020000   fffffe8221837240                ssh
 1970     1 2   0   8020000   fffffe821f4c70a0                 sh
 2649     1 2   0   8020000   fffffe821a5f28e0                ssh
 2129     1 2   0   8020000   fffffe8227c3cb00                 sh
 2384     1 2   0   8020000   fffffe8221afe480                ssh
 2971     1 2   0   8020000   fffffe82225690c0                 sh
 2070     1 2   0   8020000   fffffe82212562e0                ssh
 2098     1 2   0   8020000   fffffe82201c92a0                 sh
 2764     1 2   0   8020000   fffffe821bf22680                ssh
 3322     1 2   0   8020000   fffffe8221256b20                 sh
 2511     1 2   0   8020000   fffffe821f37d160                ssh
 2661     1 2   0   8020000   fffffe821ecae1c0                 sh
 3491     1 2   0   8020000   fffffe822220f160                ssh
 3195     1 2   0   8020000   fffffe822188a9c0                 sh
 3305     1 2   0   8020000   fffffe821b800600                ssh
 2197     1 2   0   8020000   fffffe821d2d9640                 sh
 3011     1 2   0   8020000   fffffe82214bf6e0                ssh
 3056     1 2   0   8020000   fffffe82288f2ac0                 sh
 2350     1 2   0   8020000   fffffe8219f93960                ssh
 2297     1 2   0   8020000   fffffe8221e571e0                 sh
 2638     1 2   0   8020000   fffffe821a5f20a0                ssh
 2169     1 2   0   8020000   fffffe821c4bb0c0                 sh
 2550     1 2   0   8020000   fffffe82213176a0                ssh
 2225     1 2   0   8020000   fffffe82220aa100                 sh
 2677     1 2   0   8020000   fffffe821e90c660                ssh
 3183     1 2   0   8020000   fffffe8229af4260                 sh
 2214     1 2   0   8020000   fffffe821f232000                ssh
 3259     1 2   0   8020000   fffffe8223563720                 sh
 2092     1 2   0   8020000   fffffe822220f580                ssh
 2331     1 2   0   8020000   fffffe8223563300                 sh
 2750     1 2   0   8020000   fffffe8226888000                ssh
 1921     1 2   0   8020000   fffffe821bf22aa0                 sh
 2636     1 2   0   8020000   fffffe822d818160                ssh
 3223     1 2   0   8020000   fffffe821d443100                 sh
 2773     1 2   0   8020000   fffffe8226888420                ssh
 2150     1 2   0   8020000   fffffe8223049320                 sh
 2310     1 2   0   8020000   fffffe8222b99560                ssh
 1587     1 2   0   8020000   fffffe8220d110e0                 sh
 2901     1 2   0   8020000   fffffe822188a180                ssh
 2627     1 2   0   8020000   fffffe821c4bb900                 sh
 2060     1 2   0   8020000   fffffe821f36a460                ssh
 2599     1 2   0   8020000   fffffe821bf22260                 sh
 2422     1 2   0   8020000   fffffe821f8965a0                ssh
 2164     1 2   0   8020000   fffffe821d443940                 sh
 2022     1 2   0   8020000   fffffe821f556960                ssh
 2000     1 2   0   8020000   fffffe821a542920                 sh
 1593     1 2   0   8020000   fffffe822d481a40               bash
 1914     1 2   0   8020000   fffffe821f5d8ba0               sshd
 2167     1 2   0   8020000   fffffe822037f5c0               sshd
 1821     1 2   0   8020000   fffffe8220d11500               sshd
 1590     1 2   0   8020000   fffffe822dcff120               sshd
 1041     1 2   0   8020000   fffffe821c708940               bash
 594      1 2   0   8020000   fffffe822037f9e0               bash
 895      1 2   0   8020000   fffffe822708eb60               sshd
 1533     1 2   0   8020000   fffffe8227069b20               sshd
 704      1 3   6   8020000   fffffe821d362980          conserver tstile
 564      1 3   1   8020000   fffffe8229b924c0          conserver tstile
 789      1 2   0   8020000   fffffe821f232840          conserver
 788  >   1 7   0   8020000   fffffe8220b8e360          conserver
 275      1 3   5   8020000   fffffe8220b8e780          conserver tstile
 772      1 2   0   8020000   fffffe8221256700          conserver
 878      1 3   2   8020000   fffffe8221e57600          conserver tstile
 136      1 3   3   8020000   fffffe8222569900          conserver tstile
 773      1 3   1   8020000   fffffe8225709840          conserver tstile
 643      1 2   0   8020000   fffffe82251ce360          conserver
 162      1 2   0   8020000   fffffe822dd620e0              getty
 861      1 2   0   8020000   fffffe82251ce780          conserver
 575      1 2   0   8020000   fffffe82251ceba0          conserver
 892      1 2   0   8020000   fffffe8224b0c340               cron
 570      1 2   0   8020000   fffffe8224b0cb80          python2.7
 107      1 2   0   8020000   fffffe822d7de5c0              inetd
 740      1 2   0   8020000   fffffe822cf91a00              nginx
 97       1 2   0   8020000   fffffe822d4cd600              nginx
 677      1 2   0   8020000   fffffe822d481200               qmgr
 479      1 2   0   8020000   fffffe822c312a60             pickup
 760      1 2   0   8020000   fffffe822c1e7240             master
 436      1 2   0   8020000   fffffe822d818580               sshd
 474      1 2   0   8020000   fffffe822d92e5a0           tac_plus
 428      1 2   0   8020000   fffffe822cf915e0             powerd
 431      1 2   0   8020000   fffffe822d8189a0               ntpd
 342      4 2   4   8020000   fffffe822c1e7a80              dhcpd
 342      3 2   0   8020000   fffffe822c312220              dhcpd
 342      2 2   1   8020000   fffffe822c312640              dhcpd
 342      1 2   3   8020000   fffffe822d481620              dhcpd
 83   >   1 7   6   8020000   fffffe822cf911c0            syslogd
 1        1 2   0   8020000   fffffe810f4fa900               init
 0      123 3   7       200   fffffe822d4cd1e0            npfgc-0 xchicv
 0      122 3   7       200   fffffe822d4cda20              ipmi0 ipmi0
 0      121 3   0       200   fffffe822dd10940            physiod physiod
 0      120 3   3       200   fffffe822dcff960           aiodoned aiodoned
 0      119 3   5       200   fffffe822dd10100            ioflush biowait
 0      118 3   0       200   fffffe822dd10520           pgdaemon pgdaemon
 0      115 3   0       280   fffffe822dd62500              spkr1 bellcv
 0      114 3   3       280   fffffe822dd62920           audiomix play
 0      113 3   3       280   fffffe810ed3b2a0           audiorec record
 0      112 3   0       200   fffffe810f4fa0c0          atapibus0 sccomp
 0      110 3   6       200   fffffe810f2e9b60               usb1 usbevt
 0      109 3   7       200   fffffe810f2c3b20               usb3 usbevt
 0      108 3   6       200   fffffe810f20d2c0               usb5 usbevt
 0      107 3   0       200   fffffe810f2da300               usb0 usbevt
 0      106 3   0       200   fffffe810f2c32e0               usb4 usbevt
 0      105 3   0       200   fffffe810f4fa4e0               usb2 usbevt
 0      104 3   0       200   fffffe810f4a30a0            rt_free rt_free
 0      103 3   0       200   fffffe810f4a34c0              unpgc unpgc
 0      102 3   7       200   fffffe810f4a38e0    key_timehandler key_timehandler

 0      101 3   7       200   fffffe810f45c080    icmp6_wqinput/7 icmp6_wqinput
 0      100 3   6       200   fffffe810f45c4a0    icmp6_wqinput/6 icmp6_wqinput
 0       99 3   5       200   fffffe810f45c8c0    icmp6_wqinput/5 icmp6_wqinput
 0       98 3   4       200   fffffe810f4b5060    icmp6_wqinput/4 icmp6_wqinput
 0       97 3   3       200   fffffe810f4b5480    icmp6_wqinput/3 icmp6_wqinput
 0       96 3   2       200   fffffe810f4b58a0    icmp6_wqinput/2 icmp6_wqinput
 0       95 3   1       200   fffffe810f396040    icmp6_wqinput/1 icmp6_wqinput
 0       94 3   0       200   fffffe810f396460    icmp6_wqinput/0 icmp6_wqinput
 0       93 3   2       200   fffffe810f396880          nd6_timer nd6_timer
 0       92 3   7       200   fffffe810f367020     icmp_wqinput/7 icmp_wqinput
 0       91 3   6       200   fffffe810f367440     icmp_wqinput/6 icmp_wqinput
 0       90 3   5       200   fffffe810f367860     icmp_wqinput/5 icmp_wqinput
 0       89 3   4       200   fffffe810f378000     icmp_wqinput/4 icmp_wqinput
 0       88 3   3       200   fffffe810f378420     icmp_wqinput/3 icmp_wqinput
 0       87 3   2       200   fffffe810f378840     icmp_wqinput/2 icmp_wqinput
 0       86 3   1       200   fffffe810f329360     icmp_wqinput/1 icmp_wqinput
 0       85 3   0       200   fffffe810f329780     icmp_wqinput/0 icmp_wqinput
 0       84 3   0       200   fffffe810f2da720           rt_timer rt_timer
 0       83 3   7       200   fffffe810f329ba0        vmem_rehash vmem_rehash
 0       82 3   6       200   fffffe810f2f8340          coretemp3 coretemp3
 0       81 3   7       200   fffffe810f2f8760          coretemp2 coretemp2
 0       80 3   2       200   fffffe810f2f8b80          coretemp1 coretemp1
 0       79 3   3       200   fffffe810f2e9320          coretemp0 coretemp0
 0       78 3   3       200   fffffe810f2e9740               mfi0 mfi0
 0       69 3   5       200   fffffe810f20d6e0            atabus1 atath
 0       68 3   4       200   fffffe810f20db00            atabus0 atath
 0       66 3   0       200   fffffe810ed3b6c0           scsibus0 sccomp
 0       65 3   0       200   fffffe810ed3bae0         usbtask-dr usbtsk
 0       64 3   0       200   fffffe810ecaa280         usbtask-hc usbtsk
 0       63 3   0       200   fffffe810ecaa6a0               bnx3 bnx3
 0       62 3   0       200   fffffe810ecaaac0               bnx2 bnx2
 0       61 3   0       200   fffffe810eb61260               bnx1 bnx1
 0       60 3   7       200   fffffe810eb61680               bnx0 bnx0
 0    >  59 7   2       200   fffffe810eb61aa0               ipmi
 0       58 3   7       200   fffffe810ea7c240            xcall/7 xcall
 0       57 1   7       200   fffffe810ea7c660          softser/7
 0       56 1   7       200   fffffe810ea7ca80          softclk/7
 0       55 1   7       200   fffffe810ea5d220          softbio/7
 0       54 1   7       200   fffffe810ea5d640          softnet/7
 0    >  53 7   7       201   fffffe810ea5da60             idle/7
 0       52 3   6       200   fffffe810ea36200            xcall/6 xcall
 0       51 1   6       200   fffffe810ea36620          softser/6
 0       50 1   6       200   fffffe810ea36a40          softclk/6
 0       49 1   6       200   fffffe810ea0f1e0          softbio/6
 0       48 1   6       200   fffffe810ea0f600          softnet/6
 0       47 1   6       201   fffffe810ea0fa20             idle/6
 0       46 3   5       200   fffffe810e9e01c0            xcall/5 xcall
 0       45 1   5       200   fffffe810e9e05e0          softser/5
 0       44 1   5       200   fffffe810e9e0a00          softclk/5
 0       43 1   5       200   fffffe810e9d11a0          softbio/5
 0       42 1   5       200   fffffe810e9d15c0          softnet/5
 0    >  41 7   5       201   fffffe810e9d19e0             idle/5
 0       40 3   4       200   fffffe810e9a2180            xcall/4 xcall
 0       39 1   4       200   fffffe810e9a25a0          softser/4
 0       38 1   4       200   fffffe810e9a29c0          softclk/4
 0       37 1   4       200   fffffe810e973160          softbio/4
 0       36 1   4       200   fffffe810e973580          softnet/4
 0    >  35 7   4       201   fffffe810e9739a0             idle/4
 0       34 3   3       200   fffffe810e94c140            xcall/3 xcall
 0       33 1   3       200   fffffe810e94c560          softser/3
 0       32 1   3       200   fffffe810e94c980          softclk/3
 0       31 1   3       200   fffffe810e93d120          softbio/3
 0       30 1   3       200   fffffe810e93d540          softnet/3
 0    >  29 7   3       201   fffffe810e93d960             idle/3
 0       28 3   2       200   fffffe810e906100            xcall/2 xcall
 0       27 1   2       200   fffffe810e906520          softser/2
 0       26 1   2       200   fffffe810e906940          softclk/2
 0       25 1   2       200   fffffe810e8ef0e0          softbio/2
 0       24 1   2       200   fffffe810e8ef500          softnet/2
 0       23 1   2       201   fffffe810e8ef920             idle/2
 0       22 3   1       200   fffffe810e8c50c0            xcall/1 xcall
 0       21 1   1       200   fffffe810e8c54e0          softser/1
 0       20 1   1       200   fffffe810e8c5900          softclk/1
 0       19 1   1       200   fffffe810e8b10a0          softbio/1
 0       18 1   1       200   fffffe810e8b14c0          softnet/1
 0    >  17 7   1       201   fffffe810e8b18e0             idle/1
 0       16 3   0       200   fffffe822de92080           lnxsyswq lnxsyswq
 0       15 3   0       200   fffffe822de924a0             sysmon smtaskq
 0       14 3   2       200   fffffe822de928c0         pmfsuspend pmfsuspend
 0       13 3   7       200   fffffe822e2b1060           pmfevent pmfevent
 0       12 3   0       200   fffffe822e2b1480         sopendfree sopendfr
 0       11 3   2       200   fffffe822e2b18a0           nfssilly nfssilly
 0       10 3   2       200   fffffe822f6d0040            cachegc cachegc
 0        9 3   3       200   fffffe822f6d0460             vdrain vdrain
 0        8 3   0       200   fffffe822f6d0880          modunload mod_unld
 0        7 3   0       200   fffffe822f6eb020            xcall/0 xcall
 0    >   6 7   0       200   fffffe822f6eb440          softser/0
 0    >   5 7   0       200   fffffe822f6eb860          softclk/0
 0        4 1   0       200   fffffe822f707000          softbio/0
 0        3 1   0       200   fffffe822f707420          softnet/0
 0        2 1   0       201   fffffe822f707840             idle/0
 0        1 3   4       200   ffffffff81481b20            swapper uvm
 crash> 

 Backtraces for all the lwps which have non-zero "WAIT" strings:

 crash> trace/a fffffe8220c58540
 trace: pid 7052 lid 1 at 0xffff80009ea3d8f0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x460
 VOP_LOCK() at VOP_LOCK+0x4d
 vn_lock() at vn_lock+0x90
 namei_tryemulroot() at namei_tryemulroot+0x193
 namei() at namei+0x29
 vn_open() at vn_open+0x92
 do_open() at do_open+0x112
 do_sys_openat() at do_sys_openat+0x68
 sys_open() at sys_open+0x24
 syscall() at syscall+0x1ed
 --- syscall (number 5) ---
 70efab63e2ca:
 crash>

 crash> trace/a fffffe821d362560
 trace: pid 8587 lid 1 at 0xffff80009edb8d20
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 fork1() at fork1+0x774
 sys___vfork14() at sys___vfork14+0x35
 syscall() at syscall+0x1ed
 --- syscall (number 282) ---
 70efab68204c:
 crash>

 crash> trace/a fffffe821faa60c0
 trace: pid 9187 lid 1 at 0xffff80009ed4e890
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_wait() at xc_wait+0x7c
 pserialize_perform() at pserialize_perform+0xdf
 fstrans_setstate() at fstrans_setstate+0x42
 genfs_suspendctl() at genfs_suspendctl+0x57
 VFS_SUSPENDCTL() at VFS_SUSPENDCTL+0x1f
 vfs_suspend() at vfs_suspend+0x5b
 vrevoke_suspend_next() at vrevoke_suspend_next+0x2a
 vrevoke() at vrevoke+0x2b
 genfs_revoke() at genfs_revoke+0x13
 VOP_REVOKE() at VOP_REVOKE+0x54
 pty_grant_slave() at pty_grant_slave+0xa8
 ptmioctl() at ptmioctl+0x90
 cdev_ioctl() at cdev_ioctl+0x88
 VOP_IOCTL() at VOP_IOCTL+0x3b
 vn_ioctl() at vn_ioctl+0xa6
 sys_ioctl() at sys_ioctl+0x101
 syscall() at syscall+0x1ed
 --- syscall (number 54) ---
 6f9ce08feeaa:
 crash>

 crash> trace/a fffffe82045da540
 trace: pid 3794 lid 1 at 0xffff80009f6c6be0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x74d
 VOP_LOCK() at VOP_LOCK+0x4d
 vn_lock() at vn_lock+0x90
 vn_write() at vn_write+0x9b
 dofilewrite() at dofilewrite+0x97
 sys_write() at sys_write+0x5f
 syscall() at syscall+0x1ed
 --- syscall (number 4) ---
 73b7cfe3e02a:
 crash>

 crash> trace/a fffffe8216d35140
 trace: pid 2853 lid 2 at 0xffff80009f6eaab0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x460
 VOP_POLL() at VOP_POLL+0x37
 sel_do_scan() at sel_do_scan+0x324
 selcommon.part.2() at selcommon.part.2+0x18b
 sys___select50() at sys___select50+0x6f
 syscall() at syscall+0x1ed
 --- syscall (number 417) ---
 7b328823e10a:
 crash>

 crash> trace/a fffffe82214bf2c0
 trace: pid 2853 lid 1 at 0xffff80009ef1ddb0
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 fd_close() at fd_close+0x3ba
 sys_close() at sys_close+0x20
 syscall() at syscall+0x1ed
 --- syscall (number 6) ---
 7b328823e4ea:
 crash>

 crash> trace/a fffffe8225709000
 trace: pid 8327 lid 2 at 0xffff80009e90eab0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x74d
 VOP_POLL() at VOP_POLL+0x37
 sel_do_scan() at sel_do_scan+0x324
 selcommon.part.2() at selcommon.part.2+0x18b
 sys___select50() at sys___select50+0x6f
 syscall() at syscall+0x1ed
 --- syscall (number 417) ---
 7aeb6723e10a:
 crash>

 crash> trace/a fffffe820df34480
 trace: pid 8327 lid 1 at 0xffff80009f668db0
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 fd_close() at fd_close+0x3ba
 sys_close() at sys_close+0x20
 syscall() at syscall+0x1ed
 --- syscall (number 6) ---
 7aeb6723e4ea:
 crash>

 crash> trace/a fffffe8207fd0360
 trace: pid 5687 lid 2 at 0xffff80009f5f9ab0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x74d
 VOP_POLL() at VOP_POLL+0x37
 sel_do_scan() at sel_do_scan+0x324
 selcommon.part.2() at selcommon.part.2+0x18b
 sys___select50() at sys___select50+0x6f
 syscall() at syscall+0x1ed
 --- syscall (number 417) ---
 7d665343e10a:
 crash>

 crash> trace/a fffffe822346f080
 trace: pid 5687 lid 1 at 0xffff80009efd4db0
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 fd_close() at fd_close+0x3ba
 sys_close() at sys_close+0x20
 syscall() at syscall+0x1ed
 --- syscall (number 6) ---
 7d665343e4ea:
 crash>

 crash> trace/a fffffe821d987a40
 trace: pid 9671 lid 2 at 0xffff80009ee64ab0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x74d
 VOP_POLL() at VOP_POLL+0x37
 sel_do_scan() at sel_do_scan+0x324
 selcommon.part.2() at selcommon.part.2+0x18b
 sys___select50() at sys___select50+0x6f
 syscall() at syscall+0x1ed
 --- syscall (number 417) ---
 7ea7fc43e10a:
 crash>

 crash> trace/a fffffe821e90c240
 trace: pid 9671 lid 1 at 0xffff80009ee94db0
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 fd_close() at fd_close+0x3ba
 sys_close() at sys_close+0x20
 syscall() at syscall+0x1ed
 --- syscall (number 6) ---
 7ea7fc43e4ea:
 crash>

 crash> trace/a fffffe821d362980
 trace: pid 704 lid 1 at 0xffff80009edaebc0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 rw_enter() at rw_enter+0x49d
 genfs_lock() at genfs_lock+0x3c
 VOP_LOCK() at VOP_LOCK+0x67
 vn_lock() at vn_lock+0x90
 vn_write() at vn_write+0x9b
 dofilewrite() at dofilewrite+0x97
 sys_write() at sys_write+0x5f
 syscall() at syscall+0x1ed
 --- syscall (number 4) ---
 764fb503e02a:
 crash>

 crash> trace/a fffffe8229b924c0
 trace: pid 564 lid 1 at 0xffff80009e9c3ab0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x74d
 VOP_POLL() at VOP_POLL+0x37
 sel_do_scan() at sel_do_scan+0x324
 selcommon.part.2() at selcommon.part.2+0x18b
 sys___select50() at sys___select50+0x6f
 syscall() at syscall+0x1ed
 --- syscall (number 417) ---
 764fb503e10a:
 crash>

 crash> trace/a fffffe8220b8e780
 trace: pid 275 lid 1 at 0xffff80009ebf8bc0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 rw_enter() at rw_enter+0x49d
 genfs_lock() at genfs_lock+0x3c
 VOP_LOCK() at VOP_LOCK+0x67
 vn_lock() at vn_lock+0x90
 vn_write() at vn_write+0x9b
 dofilewrite() at dofilewrite+0x97
 sys_write() at sys_write+0x5f
 syscall() at syscall+0x1ed
 --- syscall (number 4) ---
 764fb503e02a:
 crash>

 crash> trace/a fffffe8221e57600
 trace: pid 878 lid 1 at 0xffff80009eae8ab0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x74d
 VOP_POLL() at VOP_POLL+0x37
 sel_do_scan() at sel_do_scan+0x324
 selcommon.part.2() at selcommon.part.2+0x18b
 sys___select50() at sys___select50+0x6f
 syscall() at syscall+0x1ed
 --- syscall (number 417) ---
 764fb503e10a:
 crash>

 crash> trace/a fffffe8222569900
 trace: pid 136 lid 1 at 0xffff80009e9cdab0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x74d
 VOP_POLL() at VOP_POLL+0x37
 sel_do_scan() at sel_do_scan+0x324
 selcommon.part.2() at selcommon.part.2+0x18b
 sys___select50() at sys___select50+0x6f
 syscall() at syscall+0x1ed
 --- syscall (number 417) ---
 764fb503e10a:
 crash>

 crash> trace/a fffffe8225709840
 trace: pid 773 lid 1 at 0xffff80009e904ab0
 sleepq_block() at sleepq_block+0x97
 turnstile_block() at turnstile_block+0x3e5
 mutex_enter() at mutex_enter+0x36d
 fstrans_start() at fstrans_start+0x74d
 VOP_POLL() at VOP_POLL+0x37
 sel_do_scan() at sel_do_scan+0x324
 selcommon.part.2() at selcommon.part.2+0x18b
 sys___select50() at sys___select50+0x6f
 syscall() at syscall+0x1ed
 --- syscall (number 417) ---
 764fb503e10a:
 crash>

 crash> trace/a fffffe822d4cd1e0
 trace: pid 0 lid 123 at 0xffff80009dee1da0
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_broadcast() at xc_broadcast+0x17a
 pserialize_perform() at pserialize_perform+0xd7
 npf_conn_gc() at npf_conn_gc+0x1b5
 npf_worker() at npf_worker+0x4b
 crash>

 crash> trace/a fffffe822d4cda20
 trace: pid 0 lid 122 at 0xffff80009dec3e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe822dd10940
 trace: pid 0 lid 121 at 0xffff80009ddc4e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe822dcff960
 trace: pid 0 lid 120 at 0xffff80009dceee60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe822dd10100
 trace: pid 0 lid 119 at 0xffff80009dce9bf0
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 biowait() at biowait+0x5d
 wapbl_buffered_flush() at wapbl_buffered_flush+0xc6
 wapbl_buffered_write() at wapbl_buffered_write+0x69
 wapbl_circ_write() at wapbl_circ_write+0xa7
 wapbl_flush() at wapbl_flush+0x354
 ffs_sync() at ffs_sync+0x221
 VFS_SYNC() at VFS_SYNC+0x1c
 sched_sync() at sched_sync+0x91
 crash>

 crash> trace/a fffffe822dd10520
 trace: pid 0 lid 118 at 0xffff80009dce4d80
 sleepq_block() at sleepq_block+0x97
 mtsleep() at mtsleep+0x102
 uvm_pageout() at uvm_pageout+0x62e
 crash>

 crash> trace/a fffffe822dd62500
 trace: pid 0 lid 115 at 0xffff80009dc8ee60
 sleepq_block() at sleepq_block+0x97
 cv_wait_sig() at cv_wait_sig+0x12b
 bell_thread() at bell_thread+0x5f
 crash>

 crash> trace/a fffffe822dd62920
 trace: pid 0 lid 114 at 0xffff80009db5fe80
 sleepq_block() at sleepq_block+0x97
 cv_wait_sig() at cv_wait_sig+0x12b
 audio_play_thread() at audio_play_thread+0x55
 crash>

 crash> trace/a fffffe810ed3b2a0
 trace: pid 0 lid 113 at 0xffff80009dc89e90
 sleepq_block() at sleepq_block+0x97
 cv_wait_sig() at cv_wait_sig+0x12b
 audio_rec_thread() at audio_rec_thread+0x3a
 crash>

 crash> trace/a fffffe810f4fa0c0
 trace: pid 0 lid 112 at 0xffff80009dc32e80
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 scsipi_completion_thread() at scsipi_completion_thread+0xf5
 crash>

 crash> trace/a fffffe810f2e9b60
 trace: pid 0 lid 110 at 0xffff80009dc27e80
 sleepq_block() at sleepq_block+0x106
 cv_timedwait() at cv_timedwait+0x131
 usb_event_thread() at usb_event_thread+0xa2
 crash>

 crash> trace/a fffffe810f2c3b20
 trace: pid 0 lid 109 at 0xffff80009dc22e80
 sleepq_block() at sleepq_block+0x106
 cv_timedwait() at cv_timedwait+0x131
 usb_event_thread() at usb_event_thread+0xa2
 crash>

 crash> trace/a fffffe810f20d2c0
 trace: pid 0 lid 108 at 0xffff80009db64e80
 sleepq_block() at sleepq_block+0x106
 cv_timedwait() at cv_timedwait+0x131
 usb_event_thread() at usb_event_thread+0xa2
 crash>

 crash> trace/a fffffe810f2da300
 trace: pid 0 lid 107 at 0xffff80009dc1de80
 sleepq_block() at sleepq_block+0x106
 cv_timedwait() at cv_timedwait+0x131
 usb_event_thread() at usb_event_thread+0xa2
 crash>

 crash> trace/a fffffe810f2c32e0
 trace: pid 0 lid 106 at 0xffff80009dc18e80
 sleepq_block() at sleepq_block+0x106
 cv_timedwait() at cv_timedwait+0x131
 usb_event_thread() at usb_event_thread+0xa2
 crash>

 crash> trace/a fffffe810f4fa4e0
 trace: pid 0 lid 105 at 0xffff80009dc13e80
 sleepq_block() at sleepq_block+0x106
 cv_timedwait() at cv_timedwait+0x131
 usb_event_thread() at usb_event_thread+0xa2
 crash>

 crash> trace/a fffffe810f4a30a0
 trace: pid 0 lid 104 at 0xffff80009dbfce60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f4a34c0
 trace: pid 0 lid 103 at 0xffff80009dbf7e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 unp_thread() at unp_thread+0x700
 crash>

 crash> trace/a fffffe810f4a38e0
 trace: pid 0 lid 102 at 0xffff80009dbf2e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f45c080
 trace: pid 0 lid 101 at 0xffff80009dbede60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f45c4a0
 trace: pid 0 lid 100 at 0xffff80009dbe8e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f45c8c0
 trace: pid 0 lid 99 at 0xffff80009dbe3e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f4b5060
 trace: pid 0 lid 98 at 0xffff80009dbdee60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f4b5480
 trace: pid 0 lid 97 at 0xffff80009dbd9e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f4b58a0
 trace: pid 0 lid 96 at 0xffff80009dbd4e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f396040
 trace: pid 0 lid 95 at 0xffff80009dbcfe60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f396460
 trace: pid 0 lid 94 at 0xffff80009dbcae60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f396880
 trace: pid 0 lid 93 at 0xffff80009dbc5e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f367020
 trace: pid 0 lid 92 at 0xffff80009dbc0e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f367440
 trace: pid 0 lid 91 at 0xffff80009dbbbe60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f367860
 trace: pid 0 lid 90 at 0xffff80009dbb6e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash> 

 crash> trace/a fffffe810f378000
 trace: pid 0 lid 89 at 0xffff80009dbb1e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f378420
 trace: pid 0 lid 88 at 0xffff80009dbace60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f378840
 trace: pid 0 lid 87 at 0xffff80009dba7e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f329360
 trace: pid 0 lid 86 at 0xffff80009dba2e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f329780
 trace: pid 0 lid 85 at 0xffff80009db6ee60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f2da720
 trace: pid 0 lid 84 at 0xffff80009db9de60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f329ba0
 trace: pid 0 lid 83 at 0xffff80009db98e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f2f8340
 trace: pid 0 lid 82 at 0xffff80009db91e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f2f8760
 trace: pid 0 lid 81 at 0xffff80009db8ce60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f2f8b80
 trace: pid 0 lid 80 at 0xffff80009db87e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f2e9320
 trace: pid 0 lid 79 at 0xffff80009db82e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f2e9740
 trace: pid 0 lid 78 at 0xffff80009db7de60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810f20d6e0
 trace: pid 0 lid 69 at 0xffff80009d349eb0
 sleepq_block() at sleepq_block+0x97
 atabus_thread() at atabus_thread+0xcc
 crash>

 crash> trace/a fffffe810f20db00
 trace: pid 0 lid 68 at 0xffff80009d344eb0
 sleepq_block() at sleepq_block+0x97
 atabus_thread() at atabus_thread+0xcc
 crash>

 crash> trace/a fffffe810ed3b6c0
 trace: pid 0 lid 66 at 0xffff80009d0ffe80
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 scsipi_completion_thread() at scsipi_completion_thread+0xf5
 crash>

 crash> trace/a fffffe810ed3bae0
 trace: pid 0 lid 65 at 0xffff800099072e70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 usb_task_thread() at usb_task_thread+0x8a
 crash>

 crash> trace/a fffffe810ecaa280
 trace: pid 0 lid 64 at 0xffff80009906de70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 usb_task_thread() at usb_task_thread+0x8a
 crash>

 crash> trace/a fffffe810ecaa6a0
 trace: pid 0 lid 63 at 0xffff800099056e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810ecaaac0
 trace: pid 0 lid 62 at 0xffff800096a41e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810eb61260
 trace: pid 0 lid 61 at 0xffff80009442ae60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810eb61680
 trace: pid 0 lid 60 at 0xffff800091e15e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe810eb61aa0
 trace: pid 0 lid 59 at 0xffff80008f7f8e30
 sleepq_block() at sleepq_block+0x1c5
 cv_timedwait() at cv_timedwait+0x131
 ipmi_thread() at ipmi_thread+0x2f4
 crash>

 crash> trace/a fffffe810ea7c240
 trace: pid 0 lid 58 at 0xffff80008f7ede70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_thread() at xc_thread+0x5a
 crash>

 crash> trace/a fffffe810ea36200
 trace: pid 0 lid 52 at 0xffff80008f7c7e70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_thread() at xc_thread+0x5a
 crash>

 crash> trace/a fffffe810e9e01c0
 trace: pid 0 lid 46 at 0xffff80008f7a1e70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_thread() at xc_thread+0x5a
 crash>

 crash> trace/a fffffe810e9a2180
 trace: pid 0 lid 40 at 0xffff80008f77be70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_thread() at xc_thread+0x5a
 crash>

 crash> trace/a fffffe810e94c140
 trace: pid 0 lid 34 at 0xffff80008f755e70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_thread() at xc_thread+0x5a
 crash>

 crash> trace/a fffffe810e906100
 trace: pid 0 lid 28 at 0xffff80008f72fe70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_thread() at xc_thread+0x5a
 crash>

 crash> trace/a fffffe810e8c50c0
 trace: pid 0 lid 22 at 0xffff80008f706e70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_thread() at xc_thread+0x5a
 crash>

 crash> trace/a fffffe822de92080
 trace: pid 0 lid 16 at 0xffff80008f6cde60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe822de924a0
 trace: pid 0 lid 15 at 0xffff80008f6c8e90
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 sysmon_task_queue_thread() at sysmon_task_queue_thread+0x81
 crash>

 crash> trace/a fffffe822de928c0
 trace: pid 0 lid 14 at 0xffff80008f6c3e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe822e2b1060
 trace: pid 0 lid 13 at 0xffff80008f6bee60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe822e2b1480
 trace: pid 0 lid 12 at 0xffff80008f6b9e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 sopendfree_thread() at sopendfree_thread+0x1da
 crash>

 crash> trace/a fffffe822e2b18a0
 trace: pid 0 lid 11 at 0xffff80008f6b4e60
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 workqueue_worker() at workqueue_worker+0x8f
 crash>

 crash> trace/a fffffe822f6d0040
 trace: pid 0 lid 10 at 0xffff80008f6afe80
 sleepq_block() at sleepq_block+0x106
 kpause() at kpause+0xea
 cache_thread() at cache_thread+0x30
 crash>

 crash> trace/a fffffe822f6d0460
 trace: pid 0 lid 9 at 0xffff80008f6aae30
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 vdrain_thread() at vdrain_thread+0x3f1
 crash>

 crash> trace/a fffffe822f6d0880
 trace: pid 0 lid 8 at 0xffff80008f6a5e80
 sleepq_block() at sleepq_block+0x97
 cv_timedwait() at cv_timedwait+0x131
 module_thread() at module_thread+0xe9
 crash>

 crash> trace/a fffffe822f6eb020
 trace: pid 0 lid 7 at 0xffff80008f6a0e70
 sleepq_block() at sleepq_block+0x97
 cv_wait() at cv_wait+0x127
 xc_thread() at xc_thread+0x5a
 crash>

 crash> trace/a ffffffff81481b20
 trace: pid 0 lid 1 at 0xffffffff817dde20
 sleepq_block() at sleepq_block+0x106
 kpause() at kpause+0xea
 uvm_scheduler() at uvm_scheduler+0x69
 sysctl_alloc() at sysctl_alloc
 crash>

 ------------------------------

 (gdb) kvm proc 0xfffffe8220c58540
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8220c58540)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8220c58540)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8098b761 in turnstile_block (ts=<optimized out>, 
     ts@entry=0xfffffe82123900c8, q=q@entry=1, 
     obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>, 
     sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fec40 in fstrans_alloc_lwp_info (mp=0xfffffe822dd7a008)
     at /usr/src/sys/kern/vfs_trans.c:274
 #5  fstrans_get_lwp_info (do_alloc=true, mp=0xfffffe822dd7a008)
     at /usr/src/sys/kern/vfs_trans.c:331
 #6  _fstrans_start (wait=1, lock_type=FSTRANS_SHARED, mp=<optimized out>)
     at /usr/src/sys/kern/vfs_trans.c:375
 #7  fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
 #8  0xffffffff80a0ed1b in vop_pre (vp=0xfffffe810f2f2930, 
     vp=0xfffffe810f2f2930, op=FST_YES, mpsafe=<synthetic pointer>, 
     mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #9  VOP_LOCK (vp=vp@entry=0xfffffe810f2f2930, flags=flags@entry=131074)
     at /usr/src/sys/kern/vnode_if.c:1281
 #10 0xffffffff80a05846 in vn_lock (vp=vp@entry=0xfffffe810f2f2930, 
     flags=flags@entry=131074) at /usr/src/sys/kern/vfs_vnops.c:1036
 #11 0xffffffff809ef7c3 in namei_start (startdir_ret=0xffff80009ea3db28, 
     isnfsd=0, state=0xffff80009ea3dbd0) at /usr/src/sys/kern/vfs_lookup.c:681
 #12 namei_oneroot (isnfsd=0, inhibitmagic=0, neverfollow=0, 
     state=0xffff80009ea3dbd0) at /usr/src/sys/kern/vfs_lookup.c:1156
 #13 namei_tryemulroot (state=state@entry=0xffff80009ea3dbd0, 
     neverfollow=neverfollow@entry=0, inhibitmagic=inhibitmagic@entry=0, 
     isnfsd=isnfsd@entry=0) at /usr/src/sys/kern/vfs_lookup.c:1510
 #14 0xffffffff809f1905 in namei (ndp=ndp@entry=0xffff80009ea3dd58)
     at /usr/src/sys/kern/vfs_lookup.c:1546
 #15 0xffffffff80a061fd in vn_open (ndp=ndp@entry=0xffff80009ea3dd58, 
     fmode=fmode@entry=4194305, cmode=cmode@entry=3116)
     at /usr/src/sys/kern/vfs_vnops.c:175
 #16 0xffffffff809fb7c4 in do_open (l=l@entry=0xfffffe8220c58540, dvp=0x0, 
     pb=<optimized out>, open_flags=open_flags@entry=4194304, 
     open_mode=open_mode@entry=774778414, fd=fd@entry=0xffff80009ea3de7c)
     at /usr/src/sys/kern/vfs_syscalls.c:1576
 #17 0xffffffff809fb915 in do_sys_openat (l=0xfffffe8220c58540, 
     fdat=fdat@entry=-100, path=<optimized out>, flags=4194304, mode=774778414, 
     fd=fd@entry=0xffff80009ea3de7c) at /usr/src/sys/kern/vfs_syscalls.c:1656
 #18 0xffffffff809fb9d3 in sys_open (l=<optimized out>, uap=<optimized out>, 
     retval=0xffff80009ea3deb0) at /usr/src/sys/kern/vfs_syscalls.c:1676
 #19 0xffffffff80248dbd in sy_call (rval=0xffff80009ea3deb0, 
     uap=0xffff80009ea3df00, l=0xfffffe8220c58540, 
     sy=0xffffffff8147da58 <sysent+120>) at /usr/src/sys/sys/syscallvar.h:65
 #20 sy_invoke (code=5, rval=0xffff80009ea3deb0, uap=0xffff80009ea3df00, 
     l=0xfffffe8220c58540, sy=0xffffffff8147da58 <sysent+120>)
     at /usr/src/sys/sys/syscallvar.h:94
 #21 syscall (frame=0xffff80009ea3df00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #22 0xffffffff8020768d in handle_syscall ()
 #23 0x000070efab549000 in ?? ()
 #24 0x0000000000400000 in ?? ()
 #25 0x2e2e2e2e2e2e2e2e in ?? ()
 #26 0x0000000000000000 in ?? ()
 (gdb)

 Looking at fstrans_lock:

 (gdb) lock 0xffffffff815cb9b0
 lock address : 0xffffffff815cb9b0 type     : sleep/adaptive
 initialized  : 0xffffffff809fe6a3
 shared holds :                  0 exclusive: 1
 shares wanted:                  0 exclusive:                 10
 cpu last held:                  6
 current lwp  : 000000000000000000 last held: 0xfffffe821faa60c0
 last locked  : 0xffffffff809ffb6d unlocked : 0xffffffff809fecc4
 (gdb) 

 What I don't understand is that "current lwp" is 0x0, even though
 this one seems to be held at the moment, and there's lots of
 contention for the lock judging by the "shares wanted, exclusive: 10".

 "last held" appears to point to

 9187     1 3   6   8020000   fffffe821faa60c0             expect xchicv

 (gdb) kvm proc 0xfffffe821faa60c0
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821faa60c0)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821faa60c0)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8094afca in cv_wait (
     cv=cv@entry=0xffffffff814c5c88 <xc_high_pri+8>, 
     mtx=mtx@entry=0xffffffff814c5c80 <xc_high_pri>)
     at /usr/src/sys/kern/kern_condvar.c:217
 #3  0xffffffff809b1293 in xc_wait (where=64992)
     at /usr/src/sys/kern/subr_xcall.c:321
 #4  0xffffffff809aa122 in pserialize_perform (psz=0xfffffe822fbd20e8)
     at /usr/src/sys/kern/subr_pserialize.c:187
 #5  0xffffffff809ffb7c in fstrans_setstate (mp=0xfffffe822c993008, 
     new_state=new_state@entry=FSTRANS_SUSPENDED)
     at /usr/src/sys/kern/vfs_trans.c:530
 #6  0xffffffff80a17729 in genfs_suspendctl (mp=0xfffffe822c993008, 
     cmd=<optimized out>) at /usr/src/sys/miscfs/genfs/genfs_vfsops.c:86
 #7  0xffffffff809f781e in VFS_SUSPENDCTL (mp=mp@entry=0xfffffe822c993008, 
     a=a@entry=1) at /usr/src/sys/kern/vfs_subr.c:1478
 #8  0xffffffff80a00530 in vfs_suspend (mp=<optimized out>, 
     mp@entry=0xfffffe822c993008, nowait=nowait@entry=0)
     at /usr/src/sys/kern/vfs_trans.c:587
 #9  0xffffffff80a01769 in vrevoke_suspend_next (lastmp=<optimized out>, 
     thismp=0xfffffe822c993008) at /usr/src/sys/kern/vfs_vnode.c:978
 #10 0xffffffff80a030ed in vrevoke (vp=0xfffffe8221e36690)
     at /usr/src/sys/kern/vfs_vnode.c:1002
 #11 0xffffffff80a17caa in genfs_revoke (v=<optimized out>)
     at /usr/src/sys/miscfs/genfs/genfs_vnops.c:273
 #12 0xffffffff80a0e4f7 in VOP_REVOKE (vp=0xfffffe8221e36690, 
     flags=flags@entry=1) at /usr/src/sys/kern/vnode_if.c:744
 #13 0xffffffff809cfb00 in pty_grant_slave (l=l@entry=0xfffffe821faa60c0, 
     dev=<optimized out>, mp=<optimized out>) at /usr/src/sys/kern/tty_ptm.c:245
 #14 0xffffffff809cfd7b in ptmioctl (dev=<optimized out>, cmd=<optimized out>, 
     data=0xfffffe822d624008, flag=<optimized out>, l=0xfffffe821faa60c0)
     at /usr/src/sys/kern/tty_ptm.c:397
 #15 0xffffffff80996a52 in cdev_ioctl (dev=42241, cmd=1208513606, 
     data=0xfffffe822d624008, flag=3, l=0xfffffe821faa60c0)
     at /usr/src/sys/kern/subr_devsw.c:938
 #16 0xffffffff80a0e2b8 in VOP_IOCTL (vp=vp@entry=0xfffffe822caabd20, 
     command=command@entry=1208513606, data=data@entry=0xfffffe822d624008, 
     fflag=<optimized out>, cred=<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:610
 #17 0xffffffff80a05e1e in vn_ioctl (fp=0xfffffe821d011a40, com=1208513606, 
     data=0xfffffe822d624008) at /usr/src/sys/kern/vfs_vnops.c:765
 #18 0xffffffff809b4e81 in sys_ioctl (l=<optimized out>, 
     uap=0xffff80009ed4ef00, retval=<optimized out>)
     at /usr/src/sys/kern/sys_generic.c:671
 #19 0xffffffff80248dbd in sy_call (rval=0xffff80009ed4eeb0, 
     uap=0xffff80009ed4ef00, l=0xfffffe821faa60c0, 
     sy=0xffffffff8147def0 <sysent+1296>) at /usr/src/sys/sys/syscallvar.h:65
 #20 sy_invoke (code=54, rval=0xffff80009ed4eeb0, uap=0xffff80009ed4ef00, 
     l=0xfffffe821faa60c0, sy=0xffffffff8147def0 <sysent+1296>)
     at /usr/src/sys/sys/syscallvar.h:94
 #21 syscall (frame=0xffff80009ed4ef00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #22 0xffffffff8020768d in handle_syscall ()
 #23 0x0000000000000003 in ?? ()
 #24 0x0000000048087446 in ?? ()
 #25 0x00007f7fff318360 in ?? ()
 #26 0x0000000000000000 in ?? ()
 (gdb)

 The xc_high_pri looks like this:

 (gdb) lock 0xffffffff814c5c80
 lock address : 0xffffffff814c5c80 type     : sleep/adaptive
 initialized  : 0xffffffff809b1053
 shared holds :                  0 exclusive: 0
 shares wanted:                  0 exclusive:                  0
 cpu last held:                  7
 current lwp  : 000000000000000000 last held: 000000000000000000
 last locked  : 0xffffffff809b17d1 unlocked : 0xffffffff8094afb7
 (gdb) 

 but that doesn't appear to lead anywhere.

 ------------------------------

 Decode of the stack backtraces of the non-kernel lwps using gdb:

 (gdb) kvm proc 0xfffffe8220c58540
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8220c58540)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8220c58540)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8098b761 in turnstile_block (ts=<optimized out>, 
     ts@entry=0xfffffe82123900c8, q=q@entry=1, 
     obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>, 
     sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fec40 in fstrans_alloc_lwp_info (mp=0xfffffe822dd7a008)
     at /usr/src/sys/kern/vfs_trans.c:274
 #5  fstrans_get_lwp_info (do_alloc=true, mp=0xfffffe822dd7a008)
     at /usr/src/sys/kern/vfs_trans.c:331
 #6  _fstrans_start (wait=1, lock_type=FSTRANS_SHARED, mp=<optimized out>)
     at /usr/src/sys/kern/vfs_trans.c:375
 #7  fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
 #8  0xffffffff80a0ed1b in vop_pre (vp=0xfffffe810f2f2930, 
     vp=0xfffffe810f2f2930, op=FST_YES, mpsafe=<synthetic pointer>, 
     mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #9  VOP_LOCK (vp=vp@entry=0xfffffe810f2f2930, flags=flags@entry=131074)
     at /usr/src/sys/kern/vnode_if.c:1281
 #10 0xffffffff80a05846 in vn_lock (vp=vp@entry=0xfffffe810f2f2930, 
     flags=flags@entry=131074) at /usr/src/sys/kern/vfs_vnops.c:1036
 #11 0xffffffff809ef7c3 in namei_start (startdir_ret=0xffff80009ea3db28, 
     isnfsd=0, state=0xffff80009ea3dbd0) at /usr/src/sys/kern/vfs_lookup.c:681
 #12 namei_oneroot (isnfsd=0, inhibitmagic=0, neverfollow=0, 
     state=0xffff80009ea3dbd0) at /usr/src/sys/kern/vfs_lookup.c:1156
 #13 namei_tryemulroot (state=state@entry=0xffff80009ea3dbd0, 
     neverfollow=neverfollow@entry=0, inhibitmagic=inhibitmagic@entry=0, 
     isnfsd=isnfsd@entry=0) at /usr/src/sys/kern/vfs_lookup.c:1510
 #14 0xffffffff809f1905 in namei (ndp=ndp@entry=0xffff80009ea3dd58)
     at /usr/src/sys/kern/vfs_lookup.c:1546
 #15 0xffffffff80a061fd in vn_open (ndp=ndp@entry=0xffff80009ea3dd58, 
     fmode=fmode@entry=4194305, cmode=cmode@entry=3116)
     at /usr/src/sys/kern/vfs_vnops.c:175
 #16 0xffffffff809fb7c4 in do_open (l=l@entry=0xfffffe8220c58540, dvp=0x0, 
     pb=<optimized out>, open_flags=open_flags@entry=4194304, 
     open_mode=open_mode@entry=774778414, fd=fd@entry=0xffff80009ea3de7c)
     at /usr/src/sys/kern/vfs_syscalls.c:1576
 #17 0xffffffff809fb915 in do_sys_openat (l=0xfffffe8220c58540, 
     fdat=fdat@entry=-100, path=<optimized out>, flags=4194304, mode=774778414, 
     fd=fd@entry=0xffff80009ea3de7c) at /usr/src/sys/kern/vfs_syscalls.c:1656
 #18 0xffffffff809fb9d3 in sys_open (l=<optimized out>, uap=<optimized out>, 
     retval=0xffff80009ea3deb0) at /usr/src/sys/kern/vfs_syscalls.c:1676
 #19 0xffffffff80248dbd in sy_call (rval=0xffff80009ea3deb0, 
     uap=0xffff80009ea3df00, l=0xfffffe8220c58540, 
     sy=0xffffffff8147da58 <sysent+120>) at /usr/src/sys/sys/syscallvar.h:65
 #20 sy_invoke (code=5, rval=0xffff80009ea3deb0, uap=0xffff80009ea3df00, 
     l=0xfffffe8220c58540, sy=0xffffffff8147da58 <sysent+120>)
     at /usr/src/sys/sys/syscallvar.h:94
 #21 syscall (frame=0xffff80009ea3df00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #22 0xffffffff8020768d in handle_syscall ()
 #23 0x000070efab549000 in ?? ()
 #24 0x0000000000400000 in ?? ()
 #25 0x2e2e2e2e2e2e2e2e in ?? ()
 #26 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe821d362560
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821d362560)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821d362560)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8094afca in cv_wait (cv=cv@entry=0xfffffe8210396788, 
     mtx=0xfffffe822fb97f00) at /usr/src/sys/kern/kern_condvar.c:217
 #3  0xffffffff80959f24 in fork1 (l1=0xfffffe821d362560, flags=flags@entry=3, 
     exitsig=exitsig@entry=20, stack=stack@entry=0x0, 
     stacksize=stacksize@entry=0, func=func@entry=0x0, arg=arg@entry=0x0, 
     retval=0xffff80009edb8eb0, rnewprocp=rnewprocp@entry=0x0)
     at /usr/src/sys/kern/kern_fork.c:592
 #4  0xffffffff8095a424 in sys___vfork14 (l=<optimized out>, v=<optimized out>, 
     retval=<optimized out>) at /usr/src/sys/kern/kern_fork.c:143
 #5  0xffffffff80248dbd in sy_call (rval=0xffff80009edb8eb0, 
     uap=0xffff80009edb8f00, l=0xfffffe821d362560, 
     sy=0xffffffff8147f450 <sysent+6768>) at /usr/src/sys/sys/syscallvar.h:65
 #6  sy_invoke (code=282, rval=0xffff80009edb8eb0, uap=0xffff80009edb8f00, 
     l=0xfffffe821d362560, sy=0xffffffff8147f450 <sysent+6768>)
     at /usr/src/sys/sys/syscallvar.h:94
 #7  syscall (frame=0xffff80009edb8f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #8  0xffffffff8020768d in handle_syscall ()
 #9  0x00007f7fff9fa3d8 in ?? ()
 #10 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe821faa60c0
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821faa60c0)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821faa60c0)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8094afca in cv_wait (
     cv=cv@entry=0xffffffff814c5c88 <xc_high_pri+8>, 
     mtx=mtx@entry=0xffffffff814c5c80 <xc_high_pri>)
     at /usr/src/sys/kern/kern_condvar.c:217
 #3  0xffffffff809b1293 in xc_wait (where=64992)
     at /usr/src/sys/kern/subr_xcall.c:321
 #4  0xffffffff809aa122 in pserialize_perform (psz=0xfffffe822fbd20e8)
     at /usr/src/sys/kern/subr_pserialize.c:187
 #5  0xffffffff809ffb7c in fstrans_setstate (mp=0xfffffe822c993008, 
     new_state=new_state@entry=FSTRANS_SUSPENDED)
     at /usr/src/sys/kern/vfs_trans.c:530
 #6  0xffffffff80a17729 in genfs_suspendctl (mp=0xfffffe822c993008, 
     cmd=<optimized out>) at /usr/src/sys/miscfs/genfs/genfs_vfsops.c:86
 #7  0xffffffff809f781e in VFS_SUSPENDCTL (mp=mp@entry=0xfffffe822c993008, 
     a=a@entry=1) at /usr/src/sys/kern/vfs_subr.c:1478
 #8  0xffffffff80a00530 in vfs_suspend (mp=<optimized out>, 
     mp@entry=0xfffffe822c993008, nowait=nowait@entry=0)
     at /usr/src/sys/kern/vfs_trans.c:587
 #9  0xffffffff80a01769 in vrevoke_suspend_next (lastmp=<optimized out>, 
     thismp=0xfffffe822c993008) at /usr/src/sys/kern/vfs_vnode.c:978
 #10 0xffffffff80a030ed in vrevoke (vp=0xfffffe8221e36690)
     at /usr/src/sys/kern/vfs_vnode.c:1002
 #11 0xffffffff80a17caa in genfs_revoke (v=<optimized out>)
     at /usr/src/sys/miscfs/genfs/genfs_vnops.c:273
 #12 0xffffffff80a0e4f7 in VOP_REVOKE (vp=0xfffffe8221e36690, 
     flags=flags@entry=1) at /usr/src/sys/kern/vnode_if.c:744
 #13 0xffffffff809cfb00 in pty_grant_slave (l=l@entry=0xfffffe821faa60c0, 
     dev=<optimized out>, mp=<optimized out>) at /usr/src/sys/kern/tty_ptm.c:245
 #14 0xffffffff809cfd7b in ptmioctl (dev=<optimized out>, cmd=<optimized out>, 
     data=0xfffffe822d624008, flag=<optimized out>, l=0xfffffe821faa60c0)
     at /usr/src/sys/kern/tty_ptm.c:397
 #15 0xffffffff80996a52 in cdev_ioctl (dev=42241, cmd=1208513606, 
     data=0xfffffe822d624008, flag=3, l=0xfffffe821faa60c0)
     at /usr/src/sys/kern/subr_devsw.c:938
 #16 0xffffffff80a0e2b8 in VOP_IOCTL (vp=vp@entry=0xfffffe822caabd20, 
     command=command@entry=1208513606, data=data@entry=0xfffffe822d624008, 
     fflag=<optimized out>, cred=<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:610
 #17 0xffffffff80a05e1e in vn_ioctl (fp=0xfffffe821d011a40, com=1208513606, 
     data=0xfffffe822d624008) at /usr/src/sys/kern/vfs_vnops.c:765
 #18 0xffffffff809b4e81 in sys_ioctl (l=<optimized out>, 
     uap=0xffff80009ed4ef00, retval=<optimized out>)
     at /usr/src/sys/kern/sys_generic.c:671
 #19 0xffffffff80248dbd in sy_call (rval=0xffff80009ed4eeb0, 
     uap=0xffff80009ed4ef00, l=0xfffffe821faa60c0, 
     sy=0xffffffff8147def0 <sysent+1296>) at /usr/src/sys/sys/syscallvar.h:65
 #20 sy_invoke (code=54, rval=0xffff80009ed4eeb0, uap=0xffff80009ed4ef00, 
     l=0xfffffe821faa60c0, sy=0xffffffff8147def0 <sysent+1296>)
     at /usr/src/sys/sys/syscallvar.h:94
 #21 syscall (frame=0xffff80009ed4ef00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #22 0xffffffff8020768d in handle_syscall ()
 #23 0x0000000000000003 in ?? ()
 #24 0x0000000048087446 in ?? ()
 #25 0x00007f7fff318360 in ?? ()
 #26 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe82045da540
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe82045da540)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe82045da540)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8098b761 in turnstile_block (ts=<optimized out>, 
     ts@entry=0xfffffe82123900c8, q=q@entry=1, 
     obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>, 
     sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fef2d in _fstrans_start (wait=1, lock_type=FSTRANS_SHARED, 
     mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
 #5  fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
 #6  0xffffffff80a0ed1b in vop_pre (vp=0xfffffe821a9bc7e0, 
     vp=0xfffffe821a9bc7e0, op=FST_YES, mpsafe=<synthetic pointer>, 
     mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #7  VOP_LOCK (vp=vp@entry=0xfffffe821a9bc7e0, flags=flags@entry=131074)
     at /usr/src/sys/kern/vnode_if.c:1281
 #8  0xffffffff80a05846 in vn_lock (vp=vp@entry=0xfffffe821a9bc7e0, 
     flags=flags@entry=131074) at /usr/src/sys/kern/vfs_vnops.c:1036
 #9  0xffffffff80a0601d in vn_write (fp=<optimized out>, 
     offset=0xfffffe821ed94300, uio=0xffff80009f6c6df0, 
     cred=0xfffffe8220120e40, flags=1) at /usr/src/sys/kern/vfs_vnops.c:592
 #10 0xffffffff809b4c1d in dofilewrite (fd=fd@entry=2, fp=0xfffffe821ed94300, 
     buf=0x7f7fff0d1230, nbyte=102, offset=<optimized out>, 
     flags=flags@entry=1, retval=retval@entry=0xffff80009f6c6eb0)
     at /usr/src/sys/kern/sys_generic.c:350
 #11 0xffffffff809b4d22 in sys_write (l=<optimized out>, 
     uap=0xffff80009f6c6f00, retval=0xffff80009f6c6eb0)
     at /usr/src/sys/kern/sys_generic.c:318
 #12 0xffffffff80248dbd in sy_call (rval=0xffff80009f6c6eb0, 
     uap=0xffff80009f6c6f00, l=0xfffffe82045da540, 
     sy=0xffffffff8147da40 <sysent+96>) at /usr/src/sys/sys/syscallvar.h:65
 #13 sy_invoke (code=4, rval=0xffff80009f6c6eb0, uap=0xffff80009f6c6f00, 
     l=0xfffffe82045da540, sy=0xffffffff8147da40 <sysent+96>)
     at /usr/src/sys/sys/syscallvar.h:94
 #14 syscall (frame=0xffff80009f6c6f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #15 0xffffffff8020768d in handle_syscall ()
 #16 0x0000000000000002 in ?? ()
 #17 0x00007f7fff0d1230 in ?? ()
 #18 0x0000000000000066 in ?? ()
 #19 0x00007f7fff0d129a in ?? ()
 #20 0x0101010101010101 in ?? ()
 #21 0x00007f7fff0d128c in ?? ()
 #22 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe8216d35140
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8216d35140)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8216d35140)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8098b761 in turnstile_block (ts=<optimized out>, ts@entry=0x0, 
     q=q@entry=1, obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>, 
     sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fec40 in fstrans_alloc_lwp_info (mp=0xfffffe822c993008)
     at /usr/src/sys/kern/vfs_trans.c:274
 #5  fstrans_get_lwp_info (do_alloc=true, mp=0xfffffe822c993008)
     at /usr/src/sys/kern/vfs_trans.c:331
 #6  _fstrans_start (wait=1, lock_type=FSTRANS_SHARED, mp=<optimized out>)
     at /usr/src/sys/kern/vfs_trans.c:375
 #7  fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
 #8  0xffffffff80a0e399 in vop_pre (vp=0xfffffe82226253f0, 
     vp=0xfffffe82226253f0, op=FST_YES, mpsafe=<synthetic pointer>, 
     mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #9  VOP_POLL (vp=0xfffffe82226253f0, events=<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:677
 #10 0xffffffff809bd7e8 in selscan (retval=0xffff80009f6eaeb0, ni=0, nfd=5, 
     bits=0xffff80009f6ead48 "\030") at /usr/src/sys/kern/sys_select.c:402
 #11 sel_do_scan (op=op@entry=1, fds=fds@entry=0xffff80009f6ead48, 
     nf=nf@entry=5, ni=ni@entry=4, ts=ts@entry=0x0, mask=mask@entry=0x0, 
     retval=retval@entry=0xffff80009f6eaeb0)
     at /usr/src/sys/kern/sys_select.c:264
 #12 0xffffffff809bdb96 in selcommon (retval=retval@entry=0xffff80009f6eaeb0, 
     nd=5, u_in=0x7b32875fff40, u_ou=0x7b32875fff60, u_ex=0x7b32875fff80, 
     ts=0x0, mask=mask@entry=0x0) at /usr/src/sys/kern/sys_select.c:362
 #13 0xffffffff809bdeab in selcommon (mask=0x0, ts=<optimized out>, 
     u_ex=<optimized out>, u_ou=<optimized out>, u_in=<optimized out>, 
     nd=<optimized out>, retval=0xffff80009f6eaeb0)
     at /usr/src/sys/kern/sys_select.c:337
 #14 sys___select50 (l=<optimized out>, uap=0xffff80009f6eaf00, 
     retval=0xffff80009f6eaeb0) at /usr/src/sys/kern/sys_select.c:212
 #15 0xffffffff80248dbd in sy_call (rval=0xffff80009f6eaeb0, 
     uap=0xffff80009f6eaf00, l=0xfffffe8216d35140, 
     sy=0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscallvar.h:65
 #16 sy_invoke (code=417, rval=0xffff80009f6eaeb0, uap=0xffff80009f6eaf00, 
     l=0xfffffe8216d35140, sy=0xffffffff814800f8 <sysent+10008>)
     at /usr/src/sys/sys/syscallvar.h:94
 #17 syscall (frame=0xffff80009f6eaf00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #18 0xffffffff8020768d in handle_syscall ()
 #19 0x0000000000000005 in ?? ()
 #20 0x00007b32875fff40 in ?? ()
 #21 0x00007b32875fff60 in ?? ()
 #22 0x00007b32875fff80 in ?? ()
 #23 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe82214bf2c0
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe82214bf2c0)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe82214bf2c0)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8094afca in cv_wait (cv=cv@entry=0xfffffe8221620998, 
     mtx=mtx@entry=0xfffffe8221620a00) at /usr/src/sys/kern/kern_condvar.c:217
 #3  0xffffffff8094e4e5 in fd_close (fd=<optimized out>)
     at /usr/src/sys/kern/kern_descrip.c:684
 #4  0xffffffff809b3c88 in sys_close (l=0xfffffe82214bf2c0, 
     uap=0xffff80009ef1df00, retval=<optimized out>)
     at /usr/src/sys/kern/sys_descrip.c:488
 #5  0xffffffff80248dbd in sy_call (rval=0xffff80009ef1deb0, 
     uap=0xffff80009ef1df00, l=0xfffffe82214bf2c0, 
     sy=0xffffffff8147da70 <sysent+144>) at /usr/src/sys/sys/syscallvar.h:65
 #6  sy_invoke (code=6, rval=0xffff80009ef1deb0, uap=0xffff80009ef1df00, 
     l=0xfffffe82214bf2c0, sy=0xffffffff8147da70 <sysent+144>)
     at /usr/src/sys/sys/syscallvar.h:94
 #7  syscall (frame=0xffff80009ef1df00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #8  0xffffffff8020768d in handle_syscall ()
 #9  0x0000000000000004 in ?? ()
 #10 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe8225709000
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8225709000)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8225709000)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8098b761 in turnstile_block (ts=<optimized out>, 
     ts@entry=0xfffffe82123900c8, q=q@entry=1, 
     obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>, 
     sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fef2d in _fstrans_start (wait=1, lock_type=FSTRANS_SHARED, 
     mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
 #5  fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
 #6  0xffffffff80a0e399 in vop_pre (vp=0xfffffe82220a8690, 
     vp=0xfffffe82220a8690, op=FST_YES, mpsafe=<synthetic pointer>, 
     mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #7  VOP_POLL (vp=0xfffffe82220a8690, events=<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:677
 #8  0xffffffff809bd7e8 in selscan (retval=0xffff80009e90eeb0, ni=0, nfd=5, 
     bits=0xffff80009e90ed48 "\030") at /usr/src/sys/kern/sys_select.c:402
 #9  sel_do_scan (op=op@entry=1, fds=fds@entry=0xffff80009e90ed48, 
     nf=nf@entry=5, ni=ni@entry=4, ts=ts@entry=0x0, mask=mask@entry=0x0, 
     retval=retval@entry=0xffff80009e90eeb0)
     at /usr/src/sys/kern/sys_select.c:264
 #10 0xffffffff809bdb96 in selcommon (retval=retval@entry=0xffff80009e90eeb0, 
     nd=5, u_in=0x7aeb665fff40, u_ou=0x7aeb665fff60, u_ex=0x7aeb665fff80, 
     ts=0x0, mask=mask@entry=0x0) at /usr/src/sys/kern/sys_select.c:362
 #11 0xffffffff809bdeab in selcommon (mask=0x0, ts=<optimized out>, 
     u_ex=<optimized out>, u_ou=<optimized out>, u_in=<optimized out>, 
     nd=<optimized out>, retval=0xffff80009e90eeb0)
     at /usr/src/sys/kern/sys_select.c:337
 #12 sys___select50 (l=<optimized out>, uap=0xffff80009e90ef00, 
     retval=0xffff80009e90eeb0) at /usr/src/sys/kern/sys_select.c:212
 #13 0xffffffff80248dbd in sy_call (rval=0xffff80009e90eeb0, 
     uap=0xffff80009e90ef00, l=0xfffffe8225709000, 
     sy=0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscallvar.h:65
 #14 sy_invoke (code=417, rval=0xffff80009e90eeb0, uap=0xffff80009e90ef00, 
     l=0xfffffe8225709000, sy=0xffffffff814800f8 <sysent+10008>)
     at /usr/src/sys/sys/syscallvar.h:94
 #15 syscall (frame=0xffff80009e90ef00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #16 0xffffffff8020768d in handle_syscall ()
 #17 0x0000000000000005 in ?? ()
 #18 0x00007aeb665fff40 in ?? ()
 #19 0x00007aeb665fff60 in ?? ()
 #20 0x00007aeb665fff80 in ?? ()
 #21 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe820df34480
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe820df34480)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe820df34480)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8094afca in cv_wait (cv=cv@entry=0xfffffe8207461358, 
     mtx=mtx@entry=0xfffffe82074613c0) at /usr/src/sys/kern/kern_condvar.c:217
 #3  0xffffffff8094e4e5 in fd_close (fd=<optimized out>)
     at /usr/src/sys/kern/kern_descrip.c:684
 #4  0xffffffff809b3c88 in sys_close (l=0xfffffe820df34480, 
     uap=0xffff80009f668f00, retval=<optimized out>)
     at /usr/src/sys/kern/sys_descrip.c:488
 #5  0xffffffff80248dbd in sy_call (rval=0xffff80009f668eb0, 
     uap=0xffff80009f668f00, l=0xfffffe820df34480, 
     sy=0xffffffff8147da70 <sysent+144>) at /usr/src/sys/sys/syscallvar.h:65
 #6  sy_invoke (code=6, rval=0xffff80009f668eb0, uap=0xffff80009f668f00, 
     l=0xfffffe820df34480, sy=0xffffffff8147da70 <sysent+144>)
     at /usr/src/sys/sys/syscallvar.h:94
 #7  syscall (frame=0xffff80009f668f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #8  0xffffffff8020768d in handle_syscall ()
 #9  0x0000000000000004 in ?? ()
 #10 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe8207fd0360
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8207fd0360)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8207fd0360)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8098b761 in turnstile_block (ts=<optimized out>, 
     ts@entry=0xfffffe82123900c8, q=q@entry=1, 
     obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>, 
     sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fef2d in _fstrans_start (wait=1, lock_type=FSTRANS_SHARED, 
     mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
 #5  fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
 #6  0xffffffff80a0e399 in vop_pre (vp=0xfffffe822122fa88, 
     vp=0xfffffe822122fa88, op=FST_YES, mpsafe=<synthetic pointer>, 
     mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #7  VOP_POLL (vp=0xfffffe822122fa88, events=<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:677
 #8  0xffffffff809bd7e8 in selscan (retval=0xffff80009f5f9eb0, ni=0, nfd=5, 
     bits=0xffff80009f5f9d48 "\030") at /usr/src/sys/kern/sys_select.c:402
 #9  sel_do_scan (op=op@entry=1, fds=fds@entry=0xffff80009f5f9d48, 
     nf=nf@entry=5, ni=ni@entry=4, ts=ts@entry=0x0, mask=mask@entry=0x0, 
     retval=retval@entry=0xffff80009f5f9eb0)
     at /usr/src/sys/kern/sys_select.c:264
 #10 0xffffffff809bdb96 in selcommon (retval=retval@entry=0xffff80009f5f9eb0, 
     nd=5, u_in=0x7d66527fff40, u_ou=0x7d66527fff60, u_ex=0x7d66527fff80, 
     ts=0x0, mask=mask@entry=0x0) at /usr/src/sys/kern/sys_select.c:362
 #11 0xffffffff809bdeab in selcommon (mask=0x0, ts=<optimized out>, 
     u_ex=<optimized out>, u_ou=<optimized out>, u_in=<optimized out>, 
     nd=<optimized out>, retval=0xffff80009f5f9eb0)
     at /usr/src/sys/kern/sys_select.c:337
 #12 sys___select50 (l=<optimized out>, uap=0xffff80009f5f9f00, 
     retval=0xffff80009f5f9eb0) at /usr/src/sys/kern/sys_select.c:212
 #13 0xffffffff80248dbd in sy_call (rval=0xffff80009f5f9eb0, 
     uap=0xffff80009f5f9f00, l=0xfffffe8207fd0360, 
     sy=0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscallvar.h:65
 #14 sy_invoke (code=417, rval=0xffff80009f5f9eb0, uap=0xffff80009f5f9f00, 
     l=0xfffffe8207fd0360, sy=0xffffffff814800f8 <sysent+10008>)
     at /usr/src/sys/sys/syscallvar.h:94
 #15 syscall (frame=0xffff80009f5f9f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #16 0xffffffff8020768d in handle_syscall ()
 #17 0x0000000000000005 in ?? ()
 #18 0x00007d66527fff40 in ?? ()
 #19 0x00007d66527fff60 in ?? ()
 #20 0x00007d66527fff80 in ?? ()
 #21 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe822346f080
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe822346f080)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe822346f080)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8094afca in cv_wait (cv=cv@entry=0xfffffe82243bfc18, 
     mtx=mtx@entry=0xfffffe82243bfc80) at /usr/src/sys/kern/kern_condvar.c:217
 #3  0xffffffff8094e4e5 in fd_close (fd=<optimized out>)
     at /usr/src/sys/kern/kern_descrip.c:684
 #4  0xffffffff809b3c88 in sys_close (l=0xfffffe822346f080, 
     uap=0xffff80009efd4f00, retval=<optimized out>)
     at /usr/src/sys/kern/sys_descrip.c:488
 #5  0xffffffff80248dbd in sy_call (rval=0xffff80009efd4eb0, 
     uap=0xffff80009efd4f00, l=0xfffffe822346f080, 
     sy=0xffffffff8147da70 <sysent+144>) at /usr/src/sys/sys/syscallvar.h:65
 #6  sy_invoke (code=6, rval=0xffff80009efd4eb0, uap=0xffff80009efd4f00, 
     l=0xfffffe822346f080, sy=0xffffffff8147da70 <sysent+144>)
     at /usr/src/sys/sys/syscallvar.h:94
 #7  syscall (frame=0xffff80009efd4f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #8  0xffffffff8020768d in handle_syscall ()
 #9  0x0000000000000004 in ?? ()
 #10 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe821d987a40
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821d987a40)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821d987a40)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8098b761 in turnstile_block (ts=<optimized out>, 
     ts@entry=0xfffffe82123900c8, q=q@entry=1, 
     obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>, 
     sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fef2d in _fstrans_start (wait=1, lock_type=FSTRANS_SHARED, 
     mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
 #5  fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
 #6  0xffffffff80a0e399 in vop_pre (vp=0xfffffe820ecde930, 
     vp=0xfffffe820ecde930, op=FST_YES, mpsafe=<synthetic pointer>, 
     mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #7  VOP_POLL (vp=0xfffffe820ecde930, events=<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:677
 #8  0xffffffff809bd7e8 in selscan (retval=0xffff80009ee64eb0, ni=0, nfd=5, 
     bits=0xffff80009ee64d48 "\030") at /usr/src/sys/kern/sys_select.c:402
 #9  sel_do_scan (op=op@entry=1, fds=fds@entry=0xffff80009ee64d48, 
     nf=nf@entry=5, ni=ni@entry=4, ts=ts@entry=0x0, mask=mask@entry=0x0, 
     retval=retval@entry=0xffff80009ee64eb0)
     at /usr/src/sys/kern/sys_select.c:264
 #10 0xffffffff809bdb96 in selcommon (retval=retval@entry=0xffff80009ee64eb0, 
     nd=5, u_in=0x7ea7fb7fff40, u_ou=0x7ea7fb7fff60, u_ex=0x7ea7fb7fff80, 
     ts=0x0, mask=mask@entry=0x0) at /usr/src/sys/kern/sys_select.c:362
 #11 0xffffffff809bdeab in selcommon (mask=0x0, ts=<optimized out>, 
     u_ex=<optimized out>, u_ou=<optimized out>, u_in=<optimized out>, 
     nd=<optimized out>, retval=0xffff80009ee64eb0)
     at /usr/src/sys/kern/sys_select.c:337
 #12 sys___select50 (l=<optimized out>, uap=0xffff80009ee64f00, 
     retval=0xffff80009ee64eb0) at /usr/src/sys/kern/sys_select.c:212
 #13 0xffffffff80248dbd in sy_call (rval=0xffff80009ee64eb0, 
     uap=0xffff80009ee64f00, l=0xfffffe821d987a40, 
     sy=0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscallvar.h:65
 #14 sy_invoke (code=417, rval=0xffff80009ee64eb0, uap=0xffff80009ee64f00, 
     l=0xfffffe821d987a40, sy=0xffffffff814800f8 <sysent+10008>)
     at /usr/src/sys/sys/syscallvar.h:94
 #15 syscall (frame=0xffff80009ee64f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #16 0xffffffff8020768d in handle_syscall ()
 #17 0x0000000000000005 in ?? ()
 #18 0x00007ea7fb7fff40 in ?? ()
 #19 0x00007ea7fb7fff60 in ?? ()
 #20 0x00007ea7fb7fff80 in ?? ()
 #21 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe821e90c240
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821e90c240)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821e90c240)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8094afca in cv_wait (cv=cv@entry=0xfffffe822400e918, 
     mtx=mtx@entry=0xfffffe822400e980) at /usr/src/sys/kern/kern_condvar.c:217
 #3  0xffffffff8094e4e5 in fd_close (fd=<optimized out>)
     at /usr/src/sys/kern/kern_descrip.c:684
 #4  0xffffffff809b3c88 in sys_close (l=0xfffffe821e90c240, 
     uap=0xffff80009ee94f00, retval=<optimized out>)
     at /usr/src/sys/kern/sys_descrip.c:488
 #5  0xffffffff80248dbd in sy_call (rval=0xffff80009ee94eb0, 
     uap=0xffff80009ee94f00, l=0xfffffe821e90c240, 
     sy=0xffffffff8147da70 <sysent+144>) at /usr/src/sys/sys/syscallvar.h:65
 #6  sy_invoke (code=6, rval=0xffff80009ee94eb0, uap=0xffff80009ee94f00, 
     l=0xfffffe821e90c240, sy=0xffffffff8147da70 <sysent+144>)
     at /usr/src/sys/sys/syscallvar.h:94
 #7  syscall (frame=0xffff80009ee94f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #8  0xffffffff8020768d in handle_syscall ()
 #9  0x0000000000000004 in ?? ()
 #10 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe821d362980
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821d362980)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe821d362980)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8098b761 in turnstile_block (ts=<optimized out>, 
     ts@entry=0xfffffe821f9db480, q=q@entry=1, 
     obj=obj@entry=0xfffffe8224a7b130, 
     sobj=sobj@entry=0xffffffff81482720 <rw_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80977cb3 in rw_vector_enter (rw=0xfffffe8224a7b130, op=RW_WRITER)
     at /usr/src/sys/kern/kern_rwlock.c:395
 #4  0xffffffff80a17d7e in genfs_lock (v=<optimized out>)
     at /usr/src/sys/miscfs/genfs/genfs_vnops.c:343
 #5  0xffffffff80a0ed35 in VOP_LOCK (vp=vp@entry=0xfffffe8224a7b000, 
     flags=flags@entry=131074) at /usr/src/sys/kern/vnode_if.c:1284
 #6  0xffffffff80a05846 in vn_lock (vp=vp@entry=0xfffffe8224a7b000, 
     flags=flags@entry=131074) at /usr/src/sys/kern/vfs_vnops.c:1036
 #7  0xffffffff80a0601d in vn_write (fp=<optimized out>, 
     offset=0xfffffe8225588b00, uio=0xffff80009edaedf0, 
     cred=0xfffffe822f724f00, flags=1) at /usr/src/sys/kern/vfs_vnops.c:592
 #8  0xffffffff809b4c1d in dofilewrite (fd=fd@entry=2, fp=0xfffffe8225588b00, 
     buf=0x764fb6902000, nbyte=94, offset=<optimized out>, flags=flags@entry=1, 
     retval=retval@entry=0xffff80009edaeeb0)
     at /usr/src/sys/kern/sys_generic.c:350
 #9  0xffffffff809b4d22 in sys_write (l=<optimized out>, 
     uap=0xffff80009edaef00, retval=0xffff80009edaeeb0)
     at /usr/src/sys/kern/sys_generic.c:318
 #10 0xffffffff80248dbd in sy_call (rval=0xffff80009edaeeb0, 
     uap=0xffff80009edaef00, l=0xfffffe821d362980, 
     sy=0xffffffff8147da40 <sysent+96>) at /usr/src/sys/sys/syscallvar.h:65
 #11 sy_invoke (code=4, rval=0xffff80009edaeeb0, uap=0xffff80009edaef00, 
     l=0xfffffe821d362980, sy=0xffffffff8147da40 <sysent+96>)
     at /usr/src/sys/sys/syscallvar.h:94
 #12 syscall (frame=0xffff80009edaef00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #13 0xffffffff8020768d in handle_syscall ()
 #14 0x0000000000000002 in ?? ()
 #15 0x0000764fb6902000 in ?? ()
 #16 0x000000000000005e in ?? ()
 #17 0x0000764fb690205e in ?? ()
 #18 0x000000000000001f in ?? ()
 #19 0x0000000000000001 in ?? ()
 #20 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe8229b924c0
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8229b924c0)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8229b924c0)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8098b761 in turnstile_block (ts=<optimized out>, 
     ts@entry=0xfffffe82123900c8, q=q@entry=1, 
     obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>, 
     sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fef2d in _fstrans_start (wait=1, lock_type=FSTRANS_SHARED, 
     mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
 #5  fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
 #6  0xffffffff80a0e399 in vop_pre (vp=0xfffffe821d7d1690, 
     vp=0xfffffe821d7d1690, op=FST_YES, mpsafe=<synthetic pointer>, 
     mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #7  VOP_POLL (vp=0xfffffe821d7d1690, events=<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:677
 #8  0xffffffff809bd7e8 in selscan (retval=0xffff80009e9c3eb0, ni=3639083008, 
     nfd=48, bits=0xffff80009e9c3d48 "\b@\350\330(")
     at /usr/src/sys/kern/sys_select.c:402
 #9  sel_do_scan (op=op@entry=1, fds=fds@entry=0xffff80009e9c3d48, 
     nf=nf@entry=48, ni=ni@entry=8, ts=ts@entry=0xffff80009e9c3e70, 
     mask=mask@entry=0x0, retval=retval@entry=0xffff80009e9c3eb0)
     at /usr/src/sys/kern/sys_select.c:264
 #10 0xffffffff809bdb96 in selcommon (retval=retval@entry=0xffff80009e9c3eb0, 
     nd=48, u_in=0x7f7fff4c5a10, u_ou=0x7f7fff4c5a30, u_ex=0x0, 
     ts=0xffff80009e9c3e70, mask=mask@entry=0x0)
     at /usr/src/sys/kern/sys_select.c:362
 #11 0xffffffff809bdeab in selcommon (mask=0x0, ts=<optimized out>, 
     u_ex=<optimized out>, u_ou=<optimized out>, u_in=<optimized out>, 
     nd=<optimized out>, retval=0xffff80009e9c3eb0)
     at /usr/src/sys/kern/sys_select.c:337
 #12 sys___select50 (l=<optimized out>, uap=0xffff80009e9c3f00, 
     retval=0xffff80009e9c3eb0) at /usr/src/sys/kern/sys_select.c:212
 #13 0xffffffff80248dbd in sy_call (rval=0xffff80009e9c3eb0, 
     uap=0xffff80009e9c3f00, l=0xfffffe8229b924c0, 
     sy=0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscallvar.h:65
 #14 sy_invoke (code=417, rval=0xffff80009e9c3eb0, uap=0xffff80009e9c3f00, 
     l=0xfffffe8229b924c0, sy=0xffffffff814800f8 <sysent+10008>)
     at /usr/src/sys/sys/syscallvar.h:94
 #15 syscall (frame=0xffff80009e9c3f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #16 0xffffffff8020768d in handle_syscall ()
 #17 0x0000000000000030 in ?? ()
 #18 0x00007f7fff4c5a10 in ?? ()
 #19 0x00007f7fff4c5a30 in ?? ()
 #20 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe8220b8e780
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8220b8e780)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8220b8e780)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8098b761 in turnstile_block (ts=<optimized out>, ts@entry=0x0, 
     q=q@entry=1, obj=obj@entry=0xfffffe8224a7b130, 
     sobj=sobj@entry=0xffffffff81482720 <rw_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80977cb3 in rw_vector_enter (rw=0xfffffe8224a7b130, op=RW_WRITER)
     at /usr/src/sys/kern/kern_rwlock.c:395
 #4  0xffffffff80a17d7e in genfs_lock (v=<optimized out>)
     at /usr/src/sys/miscfs/genfs/genfs_vnops.c:343
 #5  0xffffffff80a0ed35 in VOP_LOCK (vp=vp@entry=0xfffffe8224a7b000, 
     flags=flags@entry=131074) at /usr/src/sys/kern/vnode_if.c:1284
 #6  0xffffffff80a05846 in vn_lock (vp=vp@entry=0xfffffe8224a7b000, 
     flags=flags@entry=131074) at /usr/src/sys/kern/vfs_vnops.c:1036
 #7  0xffffffff80a0601d in vn_write (fp=<optimized out>, 
     offset=0xfffffe8225588b00, uio=0xffff80009ebf8df0, 
     cred=0xfffffe822f724f00, flags=1) at /usr/src/sys/kern/vfs_vnops.c:592
 #8  0xffffffff809b4c1d in dofilewrite (fd=fd@entry=2, fp=0xfffffe8225588b00, 
     buf=0x764fb6902000, nbyte=90, offset=<optimized out>, flags=flags@entry=1, 
     retval=retval@entry=0xffff80009ebf8eb0)
     at /usr/src/sys/kern/sys_generic.c:350
 #9  0xffffffff809b4d22 in sys_write (l=<optimized out>, 
     uap=0xffff80009ebf8f00, retval=0xffff80009ebf8eb0)
     at /usr/src/sys/kern/sys_generic.c:318
 #10 0xffffffff80248dbd in sy_call (rval=0xffff80009ebf8eb0, 
     uap=0xffff80009ebf8f00, l=0xfffffe8220b8e780, 
     sy=0xffffffff8147da40 <sysent+96>) at /usr/src/sys/sys/syscallvar.h:65
 #11 sy_invoke (code=4, rval=0xffff80009ebf8eb0, uap=0xffff80009ebf8f00, 
     l=0xfffffe8220b8e780, sy=0xffffffff8147da40 <sysent+96>)
     at /usr/src/sys/sys/syscallvar.h:94
 #12 syscall (frame=0xffff80009ebf8f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #13 0xffffffff8020768d in handle_syscall ()
 #14 0x0000000000000002 in ?? ()
 #15 0x0000764fb6902000 in ?? ()
 #16 0x000000000000005a in ?? ()
 #17 0x0000764fb690205a in ?? ()
 #18 0x000000000000001e in ?? ()
 #19 0x0000000000000001 in ?? ()
 #20 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe8221e57600
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8221e57600)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8221e57600)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8098b761 in turnstile_block (ts=<optimized out>, 
     ts@entry=0xfffffe82123900c8, q=q@entry=1, 
     obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>, 
     sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fef2d in _fstrans_start (wait=1, lock_type=FSTRANS_SHARED, 
     mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
 #5  fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
 #6  0xffffffff80a0e399 in vop_pre (vp=0xfffffe8221e36bd0, 
     vp=0xfffffe8221e36bd0, op=FST_YES, mpsafe=<synthetic pointer>, 
     mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #7  VOP_POLL (vp=0xfffffe8221e36bd0, events=<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:677
 #8  0xffffffff809bd7e8 in selscan (retval=0xffff80009eae8eb0, ni=2046820352, 
     nfd=52, bits=0xffff80009eae8d48 "H") at /usr/src/sys/kern/sys_select.c:402
 #9  sel_do_scan (op=op@entry=1, fds=fds@entry=0xffff80009eae8d48, 
     nf=nf@entry=52, ni=ni@entry=8, ts=ts@entry=0xffff80009eae8e70, 
     mask=mask@entry=0x0, retval=retval@entry=0xffff80009eae8eb0)
     at /usr/src/sys/kern/sys_select.c:264
 #10 0xffffffff809bdb96 in selcommon (retval=retval@entry=0xffff80009eae8eb0, 
     nd=52, u_in=0x7f7fff4c5a10, u_ou=0x7f7fff4c5a30, u_ex=0x0, 
     ts=0xffff80009eae8e70, mask=mask@entry=0x0)
     at /usr/src/sys/kern/sys_select.c:362
 #11 0xffffffff809bdeab in selcommon (mask=0x0, ts=<optimized out>, 
     u_ex=<optimized out>, u_ou=<optimized out>, u_in=<optimized out>, 
     nd=<optimized out>, retval=0xffff80009eae8eb0)
     at /usr/src/sys/kern/sys_select.c:337
 #12 sys___select50 (l=<optimized out>, uap=0xffff80009eae8f00, 
     retval=0xffff80009eae8eb0) at /usr/src/sys/kern/sys_select.c:212
 #13 0xffffffff80248dbd in sy_call (rval=0xffff80009eae8eb0, 
     uap=0xffff80009eae8f00, l=0xfffffe8221e57600, 
     sy=0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscallvar.h:65
 #14 sy_invoke (code=417, rval=0xffff80009eae8eb0, uap=0xffff80009eae8f00, 
     l=0xfffffe8221e57600, sy=0xffffffff814800f8 <sysent+10008>)
     at /usr/src/sys/sys/syscallvar.h:94
 #15 syscall (frame=0xffff80009eae8f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #16 0xffffffff8020768d in handle_syscall ()
 #17 0x0000000000000034 in ?? ()
 #18 0x00007f7fff4c5a10 in ?? ()
 #19 0x00007f7fff4c5a30 in ?? ()
 #20 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe8222569900
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8222569900)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8222569900)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8098b761 in turnstile_block (ts=<optimized out>, 
     ts@entry=0xfffffe82123900c8, q=q@entry=1, 
     obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>, 
     sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fef2d in _fstrans_start (wait=1, lock_type=FSTRANS_SHARED, 
     mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
 #5  fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
 #6  0xffffffff80a0e399 in vop_pre (vp=0xfffffe8223aa62a0, 
     vp=0xfffffe8223aa62a0, op=FST_YES, mpsafe=<synthetic pointer>, 
     mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #7  VOP_POLL (vp=0xfffffe8223aa62a0, events=<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:677
 #8  0xffffffff809bd7e8 in selscan (retval=0xffff80009e9cdeb0, ni=796917760, 
     nfd=53, bits=0xffff80009e9cdd48 "\b") at /usr/src/sys/kern/sys_select.c:402
 #9  sel_do_scan (op=op@entry=1, fds=fds@entry=0xffff80009e9cdd48, 
     nf=nf@entry=53, ni=ni@entry=8, ts=ts@entry=0xffff80009e9cde70, 
     mask=mask@entry=0x0, retval=retval@entry=0xffff80009e9cdeb0)
     at /usr/src/sys/kern/sys_select.c:264
 #10 0xffffffff809bdb96 in selcommon (retval=retval@entry=0xffff80009e9cdeb0, 
     nd=53, u_in=0x7f7fff4c5a10, u_ou=0x7f7fff4c5a30, u_ex=0x0, 
     ts=0xffff80009e9cde70, mask=mask@entry=0x0)
     at /usr/src/sys/kern/sys_select.c:362
 #11 0xffffffff809bdeab in selcommon (mask=0x0, ts=<optimized out>, 
     u_ex=<optimized out>, u_ou=<optimized out>, u_in=<optimized out>, 
     nd=<optimized out>, retval=0xffff80009e9cdeb0)
     at /usr/src/sys/kern/sys_select.c:337
 #12 sys___select50 (l=<optimized out>, uap=0xffff80009e9cdf00, 
     retval=0xffff80009e9cdeb0) at /usr/src/sys/kern/sys_select.c:212
 #13 0xffffffff80248dbd in sy_call (rval=0xffff80009e9cdeb0, 
     uap=0xffff80009e9cdf00, l=0xfffffe8222569900, 
     sy=0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscallvar.h:65
 #14 sy_invoke (code=417, rval=0xffff80009e9cdeb0, uap=0xffff80009e9cdf00, 
     l=0xfffffe8222569900, sy=0xffffffff814800f8 <sysent+10008>)
     at /usr/src/sys/sys/syscallvar.h:94
 #15 syscall (frame=0xffff80009e9cdf00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #16 0xffffffff8020768d in handle_syscall ()
 #17 0x0000000000000035 in ?? ()
 #18 0x00007f7fff4c5a10 in ?? ()
 #19 0x00007f7fff4c5a30 in ?? ()
 #20 0x0000000000000000 in ?? ()
 (gdb)

 (gdb) kvm proc 0xfffffe8225709840
 0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8225709840)
     at /usr/src/sys/kern/kern_synch.c:720
 720                     prevlwp = cpu_switchto(l, newl, returning);
 (gdb) where
 #0  0xffffffff80980841 in mi_switch (l=l@entry=0xfffffe8225709840)
     at /usr/src/sys/kern/kern_synch.c:720
 #1  0xffffffff8097d5fd in sleepq_block (timo=timo@entry=0, 
     catch_p=catch_p@entry=false) at /usr/src/sys/kern/kern_sleepq.c:264
 #2  0xffffffff8098b761 in turnstile_block (ts=<optimized out>, 
     ts@entry=0xfffffe82123900c8, q=q@entry=1, 
     obj=obj@entry=0xffffffff815cb9b0 <fstrans_lock>, 
     sobj=sobj@entry=0xffffffff814821a0 <mutex_syncobj>)
     at /usr/src/sys/kern/kern_turnstile.c:430
 #3  0xffffffff80965a59 in mutex_vector_enter (
     mtx=0xffffffff815cb9b0 <fstrans_lock>)
     at /usr/src/sys/kern/kern_mutex.c:693
 #4  0xffffffff809fef2d in _fstrans_start (wait=1, lock_type=FSTRANS_SHARED, 
     mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:399
 #5  fstrans_start (mp=<optimized out>) at /usr/src/sys/kern/vfs_trans.c:414
 #6  0xffffffff80a0e399 in vop_pre (vp=0xfffffe8222fe4bd8, 
     vp=0xfffffe8222fe4bd8, op=FST_YES, mpsafe=<synthetic pointer>, 
     mp=<synthetic pointer>) at /usr/src/sys/kern/vnode_if.c:77
 #7  VOP_POLL (vp=0xfffffe8222fe4bd8, events=<optimized out>)
     at /usr/src/sys/kern/vnode_if.c:677
 #8  0xffffffff809bd7e8 in selscan (retval=0xffff80009e904eb0, ni=2625634304, 
     nfd=53, bits=0xffff80009e904d48 "\b") at /usr/src/sys/kern/sys_select.c:402
 #9  sel_do_scan (op=op@entry=1, fds=fds@entry=0xffff80009e904d48, 
     nf=nf@entry=53, ni=ni@entry=8, ts=ts@entry=0xffff80009e904e70, 
     mask=mask@entry=0x0, retval=retval@entry=0xffff80009e904eb0)
     at /usr/src/sys/kern/sys_select.c:264
 #10 0xffffffff809bdb96 in selcommon (retval=retval@entry=0xffff80009e904eb0, 
     nd=53, u_in=0x7f7fff4c5a10, u_ou=0x7f7fff4c5a30, u_ex=0x0, 
     ts=0xffff80009e904e70, mask=mask@entry=0x0)
     at /usr/src/sys/kern/sys_select.c:362
 #11 0xffffffff809bdeab in selcommon (mask=0x0, ts=<optimized out>, 
     u_ex=<optimized out>, u_ou=<optimized out>, u_in=<optimized out>, 
     nd=<optimized out>, retval=0xffff80009e904eb0)
     at /usr/src/sys/kern/sys_select.c:337
 #12 sys___select50 (l=<optimized out>, uap=0xffff80009e904f00, 
     retval=0xffff80009e904eb0) at /usr/src/sys/kern/sys_select.c:212
 #13 0xffffffff80248dbd in sy_call (rval=0xffff80009e904eb0, 
     uap=0xffff80009e904f00, l=0xfffffe8225709840, 
     sy=0xffffffff814800f8 <sysent+10008>) at /usr/src/sys/sys/syscallvar.h:65
 #14 sy_invoke (code=417, rval=0xffff80009e904eb0, uap=0xffff80009e904f00, 
     l=0xfffffe8225709840, sy=0xffffffff814800f8 <sysent+10008>)
     at /usr/src/sys/sys/syscallvar.h:94
 #15 syscall (frame=0xffff80009e904f00)
     at /usr/src/sys/arch/x86/x86/syscall.c:144
 #16 0xffffffff8020768d in handle_syscall ()
 #17 0x0000000000000035 in ?? ()
 #18 0x00007f7fff4c5a10 in ?? ()
 #19 0x00007f7fff4c5a30 in ?? ()
 #20 0x0000000000000000 in ?? ()
 (gdb)

From: Ryota Ozaki <ozaki-r@netbsd.org>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc: port-amd64-maintainer@netbsd.org, gnats-admin@netbsd.org, 
	netbsd-bugs@netbsd.org, Havard Eidnes <he@netbsd.org>
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network
 interfaces in use
Date: Thu, 17 May 2018 10:50:59 +0900

 On Wed, May 16, 2018 at 8:41 PM Havard Eidnes <he@netbsd.org> wrote:

 > The following reply was made to PR port-amd64/53155; it has been noted by
 GNATS.

 > From: Havard Eidnes <he@NetBSD.org>
 > To: gnats-bugs@NetBSD.org
 > Cc:
 > Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 >    network interfaces in use
 > Date: Wed, 16 May 2018 13:37:28 +0200 (CEST)

 >    [[ Hmm, here's a non-quoted-printable version of the previous ]]

 >    Hi,

 >    we provoked another wedge, and captured a kernel core dump from
 >    the wedging machine.  The kernel core dump and NetBSD images are
 >    available for looking at.


 >    using gdb and crash, here's a brief summary of the "interesting"
 >    processes in the crash dump, and below that I include backtraces
 >    of all the waiting processes.

 >    There's lots of contention for fstrans_lock.

 >    Offhand I don't see a deadlock which might explain the observed
 >    behaviour (goes totally "deaf" on the network, i.e. doesn't even
 >    respond to ping).

 >    It doesn't look like gdb can trace through interrupt frames (?),
 >    looking at proc 788 all I get is:

 >    (gdb) kvm proc 0xfffffe8220b8e360
 >    0xffffffff8021cfe0 in softintr_ret ()
 >    (gdb) where
 >    #0  0xffffffff8021cfe0 in softintr_ret ()
 >    #1  0x0000000000000000 in ?? ()
 >    (gdb)

 >    Crash manages to do this one, though, apparently (see below).

 >    Furthermore, using the various gdb scripts in
 >    /usr/src/sys/gdbscripts/ I can look at some of the locks.

 >    It is quite possible that up'ing the interface in question causes
 >    lots of activity for opening pty pairs, and that the root cause
 >    of the issue is there rather than related to networking in itself(?)

 >    Further hints?

 >    Regards,

 >    - Havard

 >    ------------------------------

 >    PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 >    7052     1 3   1   8020000   fffffe8220c58540               cron tstile
 >      Wants fstrans_lock

 >    9187     1 3   6   8020000   fffffe821faa60c0             expect xchicv
 >      Holds fstrans_lock, in pserialize_perform, waits on condition variable
 >        after doing xc_broadcast(XC_HIGHPRI, nullop)
 >      Doing (roughly) pty_grant_slave -> genfs_revoke -> vfs_suspend ->
 >        fstrans_setstate -> pserialize_perform -> xc_wait -> cv_wait

 This xcall requires that the softint of SOFTINT_SERIAL (softser/N)
 on all CPUs processes a callback of the xcall. If any of the softints
 get stuck for some reason, the xcall never finish.

 Could you show the stack trace of each softser/N? In particular softser/0
 looks running and is a suspect.

 Thanks,
       ozaki-r

From: Havard Eidnes <he@NetBSD.org>
To: ozaki-r@netbsd.org
Cc: gnats-bugs@netbsd.org, port-amd64-maintainer@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Thu, 17 May 2018 12:34:28 +0200 (CEST)

 >>    PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 >>    7052     1 3   1   8020000   fffffe8220c58540               cron tstile
 >>      Wants fstrans_lock
 >>
 >>    9187     1 3   6   8020000   fffffe821faa60c0             expect xchicv
 >>      Holds fstrans_lock, in pserialize_perform, waits on condition variable
 >>        after doing xc_broadcast(XC_HIGHPRI, nullop)
 >>      Doing (roughly) pty_grant_slave -> genfs_revoke -> vfs_suspend ->
 >>        fstrans_setstate -> pserialize_perform -> xc_wait -> cv_wait
 >
 > This xcall requires that the softint of SOFTINT_SERIAL (softser/N)
 > on all CPUs processes a callback of the xcall. If any of the softints
 > get stuck for some reason, the xcall never finish.
 >
 > Could you show the stack trace of each softser/N? In particular softser/0
 > looks running and is a suspect.

 Getting the tracebacks of these LWPs appears to be somewhat
 problematic; I don't get much useful info from crash:

 0       78 3   3       200   fffffe810f2e9740               mfi0 mfi0
 0       69 3   5       200   fffffe810f20d6e0            atabus1 atath
 0       68 3   4       200   fffffe810f20db00            atabus0 atath
 0       66 3   0       200   fffffe810ed3b6c0           scsibus0 sccomp
 0       65 3   0       200   fffffe810ed3bae0         usbtask-dr usbtsk
 0       64 3   0       200   fffffe810ecaa280         usbtask-hc usbtsk
 0       63 3   0       200   fffffe810ecaa6a0               bnx3 bnx3
 0       62 3   0       200   fffffe810ecaaac0               bnx2 bnx2
 0       61 3   0       200   fffffe810eb61260               bnx1 bnx1
 0       60 3   7       200   fffffe810eb61680               bnx0 bnx0
 0    >  59 7   2       200   fffffe810eb61aa0               ipmi
 0       58 3   7       200   fffffe810ea7c240            xcall/7 xcall
 0       57 1   7       200   fffffe810ea7c660          softser/7
 0       56 1   7       200   fffffe810ea7ca80          softclk/7
 0       55 1   7       200   fffffe810ea5d220          softbio/7
 0       54 1   7       200   fffffe810ea5d640          softnet/7
 0    >  53 7   7       201   fffffe810ea5da60             idle/7
 0       52 3   6       200   fffffe810ea36200            xcall/6 xcall
 0       51 1   6       200   fffffe810ea36620          softser/6
 0       50 1   6       200   fffffe810ea36a40          softclk/6
 0       49 1   6       200   fffffe810ea0f1e0          softbio/6
 0       48 1   6       200   fffffe810ea0f600          softnet/6
 0       47 1   6       201   fffffe810ea0fa20             idle/6
 0       46 3   5       200   fffffe810e9e01c0            xcall/5 xcall
 0       45 1   5       200   fffffe810e9e05e0          softser/5
 0       44 1   5       200   fffffe810e9e0a00          softclk/5
 0       43 1   5       200   fffffe810e9d11a0          softbio/5
 0       42 1   5       200   fffffe810e9d15c0          softnet/5
 0    >  41 7   5       201   fffffe810e9d19e0             idle/5
 0       40 3   4       200   fffffe810e9a2180            xcall/4 xcall
 0       39 1   4       200   fffffe810e9a25a0          softser/4
 0       38 1   4       200   fffffe810e9a29c0          softclk/4
 0       37 1   4       200   fffffe810e973160          softbio/4
 0       36 1   4       200   fffffe810e973580          softnet/4
 0    >  35 7   4       201   fffffe810e9739a0             idle/4
 0       34 3   3       200   fffffe810e94c140            xcall/3 xcall
 0       33 1   3       200   fffffe810e94c560          softser/3
 0       32 1   3       200   fffffe810e94c980          softclk/3
 0       31 1   3       200   fffffe810e93d120          softbio/3
 0       30 1   3       200   fffffe810e93d540          softnet/3
 0    >  29 7   3       201   fffffe810e93d960             idle/3
 0       28 3   2       200   fffffe810e906100            xcall/2 xcall
 0       27 1   2       200   fffffe810e906520          softser/2
 0       26 1   2       200   fffffe810e906940          softclk/2
 0       25 1   2       200   fffffe810e8ef0e0          softbio/2
 0       24 1   2       200   fffffe810e8ef500          softnet/2
 0       23 1   2       201   fffffe810e8ef920             idle/2
 0       22 3   1       200   fffffe810e8c50c0            xcall/1 xcall
 0       21 1   1       200   fffffe810e8c54e0          softser/1
 0       20 1   1       200   fffffe810e8c5900          softclk/1
 0       19 1   1       200   fffffe810e8b10a0          softbio/1
 0       18 1   1       200   fffffe810e8b14c0          softnet/1
 0    >  17 7   1       201   fffffe810e8b18e0             idle/1
 0       16 3   0       200   fffffe822de92080           lnxsyswq lnxsyswq
 0       15 3   0       200   fffffe822de924a0             sysmon smtaskq
 0       14 3   2       200   fffffe822de928c0         pmfsuspend pmfsuspend
 0       13 3   7       200   fffffe822e2b1060           pmfevent pmfevent
 0       12 3   0       200   fffffe822e2b1480         sopendfree sopendfr
 0       11 3   2       200   fffffe822e2b18a0           nfssilly nfssilly
 0       10 3   2       200   fffffe822f6d0040            cachegc cachegc
 0        9 3   3       200   fffffe822f6d0460             vdrain vdrain
 0        8 3   0       200   fffffe822f6d0880          modunload mod_unld
 0        7 3   0       200   fffffe822f6eb020            xcall/0 xcall
 0    >   6 7   0       200   fffffe822f6eb440          softser/0
 0    >   5 7   0       200   fffffe822f6eb860          softclk/0
 0        4 1   0       200   fffffe822f707000          softbio/0
 0        3 1   0       200   fffffe822f707420          softnet/0
 0        2 1   0       201   fffffe822f707840             idle/0
 0        1 3   4       200   ffffffff81481b20            swapper uvm
 crash> trace/a fffffe810ea7c660
 trace: pid 0 lid 57 at 0xffff80008f7e8e80
 lockdebug_wantlock() at lockdebug_wantlock+0xf9
 _KERNEL_OPT_WSDISPLAY_SCROLLBACK_LINES() at _KERNEL_OPT_WSDISPLAY_SCROLLBACK_LIN
 ES+0x173
 crash> trace/a fffffe810ea36620
 trace: pid 0 lid 51 at 0x0
 crash> trace/a fffffe810e9e05e0
 trace: pid 0 lid 45 at 0x0
 0:
 crash> trace/a fffffe810e9a25a0
 trace: pid 0 lid 39 at 0x0
 0:
 crash> trace/a fffffe810e94c560
 trace: pid 0 lid 33 at 0x0
 0:
 crash> trace/a fffffe810e906520
 trace: pid 0 lid 27 at 0xffff80008f72ae80
 lockdebug_wantlock() at lockdebug_wantlock+0xf9
 _KERNEL_OPT_WSDISPLAY_SCROLLBACK_LINES() at _KERNEL_OPT_WSDISPLAY_SCROLLBACK_LIN
 ES+0x173
 crash> trace/a fffffe810e8c54e0
 trace: pid 0 lid 21 at 0x0
 crash> trace/a fffffe822f6eb440
 trace: pid 0 lid 6 at 0x0
 crash> 

 GDB doesn't look a lot better either:

 (gdb) kvm proc 0xfffffe822f6eb440
 0x0000000000000005 in ?? ()
 (gdb) where
 #0  0x0000000000000005 in ?? ()
 #1  0xfffffe810ed5be08 in ?? ()
 #2  0xffffffff809a2791 in percpu_getref (pc=0xffff80008f69bfe0)
     at /usr/src/sys/kern/subr_percpu.c:293
 #3  0x0000000000000000 in ?? ()
 (gdb) 
 (gdb) x/i 0xfffffe810ed5be08
    0xfffffe810ed5be08:  (bad)  
 (gdb) 
 (gdb) up
 #1  0xfffffe810ed5be08 in ?? ()
 (gdb) up
 #2  0xffffffff809a2791 in percpu_getref (pc=0xffff80008f69bfe0)
     at /usr/src/sys/kern/subr_percpu.c:293
 293             kpreempt_disable();
 (gdb) 
 (gdb) list
 288
 289     void *
 290     percpu_getref(percpu_t *pc)
 291     {
 292
 293             kpreempt_disable();
 294             return percpu_getptr_remote(pc, curcpu());
 295     }
 296
 297     /*
 (gdb) 

 Regards,

 - Havard

From: "J. Hannken-Illjes" <hannken@eis.cs.tu-bs.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network
 interfaces in use
Date: Thu, 17 May 2018 12:58:45 +0200

 >> Could you show the stack trace of each softser/N? In particular softser/0
 >> looks running and is a suspect.
 > 
 > Getting the tracebacks of these LWPs appears to be somewhat
 > problematic; I don't get much useful info from crash:
 <snip>
 > 
 > 0    >   6 7   0       200   fffffe822f6eb440          softser/0
 > 0    >   5 7   0       200   fffffe822f6eb860          softclk/0

 Crash reports two threads on cpu 0.

 If you get this type of hang again please try to get the corresponding
 stack traces from DDB: if softXXX/N is on cpu (>) use "mach cpu N; bt"

 --
 J. Hannken-Illjes - hannken@eis.cs.tu-bs.de - TU Braunschweig (Germany)

From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org, hannken@eis.cs.tu-bs.de
Cc: port-amd64-maintainer@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Thu, 17 May 2018 15:54:45 +0200 (CEST)

 >  >> Could you show the stack trace of each softser/N? In particular softser/0
 >  >> looks running and is a suspect.
 >  >
 >  > Getting the tracebacks of these LWPs appears to be somewhat
 >  > problematic; I don't get much useful info from crash:
 >  <snip>
 >  >
 >  > 0    >   6 7   0       200   fffffe822f6eb440          softser/0
 >  > 0    >   5 7   0       200   fffffe822f6eb860          softclk/0
 >
 >  Crash reports two threads on cpu 0.
 >
 >  If you get this type of hang again please try to get the corresponding
 >  stack traces from DDB: if softXXX/N is on cpu (>) use "mach cpu N; bt"

 Hmm, looking back at earlier submitted information where I dupmd
 various information from DDB, I find this identical situation:

 0    >   6 7   0       200   fffffe822f6eb440          softser/0
 0    >   5 7   0       200   fffffe822f6eb860          softclk/0

 and the backtraces for these were:

 db{0}> trace/a fffffe822f6eb440
 trace: pid 0 lid 6 at 0xffff80008f69bc30
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 uhci_intr() at netbsd:uhci_intr+0x19
 intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
 Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}> 
 db{0}> trace/a fffffe822f6eb860
 trace: pid 0 lid 5 at 0xffff80009edadd60
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 ffff80009edadf00:
 db{0}> 

 Not sure this provides any more useful information, though...

 Regards,

 - Havard

From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org, hannken@eis.cs.tu-bs.de
Cc: port-amd64-maintainer@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Thu, 17 May 2018 16:28:56 +0200 (CEST)

 And here's similar of a live kernel (I've updated to 8.0_RC1 with
 this kernel), where there's apparently no less than 3 lwps active
 on CPU 0(!).  It takes 5-10 seconds to provoke the hang after I
 "up" the third network interface...

 fatal breakpoint trap in supervisor mode
 trap type 1 code 0 rip 0xffffffff8021ff65 cs 0x8 rflags 0x202 cr2 0x7f7fff36db08 ilevel 0x8 rsp 0xffff8000ae6a2d78
 curlwp 0xfffffe822f5eb440 pid 0.6 lowest kstack 0xffff8000ae69f2c0
 Stopped in pid 0.6 (system) at  netbsd:breakpoint+0x5:  leave
 db{0}> ps
 PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 16020    1 3   4         0   fffffe820b650b00                  w tstile
 12   >   1 7   2         0   fffffe820a0d92a0             pickup
 21344    1 3   0        80   fffffe820ec515a0               sshd select
 9214     1 3   2        80   fffffe821780f9c0               sshd select
 5857     1 3   2        80   fffffe81c7c8d020               sshd select
 17622    1 3   2        80   fffffe8220a260a0               sshd select
 6326     1 3   4        80   fffffe82181fb8e0               bash select
 5489     1 3   4        80   fffffe82187b8340               sshd select
 11299    1 3   0        80   fffffe80b2611580               sshd select
 25883    1 3   6        80   fffffe820aa4da40               sshd select
 2169     1 3   4        80   fffffe821aa8d700               sshd select
 11556    1 3   6        80   fffffe82193e5900               bash select
 29717    1 3   4        80   fffffe81d20356a0               sshd select
 27130    1 3   7        80   fffffe822297f2c0               sshd select
 2956     1 3   0        80   fffffe8212a8a760               tcsh pause
 20252    1 3   6        80   fffffe821a113280               tcsh pause
 24846    1 3   3        80   fffffe82060d0580              xterm select
 24052    1 3   1        80   fffffe81bc2e02c0               tcsh pause
 18703    1 3   2        80   fffffe8211a67120               sshd select
 29547    1 3   1        80   fffffe820de4f780               sshd select
 585      1 3   0        80   fffffe822d7b20c0               tcsh ttyraw
 11254    1 3   0        80   fffffe820ad30460              xterm select
 21566    1 3   0        80   fffffe820cd798c0               tcsh pause
 29317    1 3   5        80   fffffe820fa32900               sshd select
 19621    1 3   0        80   fffffe8220a264c0               sshd select
 23570    1 3   2        80   fffffe821b5bd1a0                ssh select
 14266    1 3   2        80   fffffe821971d640                 sh wait
 20244    1 3   5        80   fffffe82190d4aa0                ssh select
 20184    1 3   5        80   fffffe82213f5b80                 sh wait
 14347    1 3   2        80   fffffe8206800100                ssh select
 22513    1 3   2        80   fffffe820fa320c0                 sh wait
 25132    1 3   2        80   fffffe8211a67960                ssh select
 24207    1 3   2        80   fffffe821a8e19c0                 sh wait
 18230    1 3   6        80   fffffe822a6789e0                ssh select
 15149    1 3   6        80   fffffe821f3e3860                 sh wait
 20547    1 3   2        80   fffffe820f5bb140                ssh select
 28126    1 3   2        80   fffffe8219d9fb60                 sh wait
 21642    1 3   7        80   fffffe8219cc1aa0                ssh select
 22234    1 3   7        80   fffffe82187b8760                 sh wait
 23629    1 3   7        80   fffffe8219973860                ssh select
 17367    1 3   7        80   fffffe8208a40600                 sh wait
 20259    1 3   2        80   fffffe8212a8ab80                ssh select
 26419    1 3   2        80   fffffe8211b7b420                 sh wait
 20593    1 3   7        80   fffffe822a314a40                ssh select
 22687    1 3   7        80   fffffe81bc2e06e0                 sh wait
 29573    1 3   7        80   fffffe820ec519c0                ssh select
 22337    1 3   7        80   fffffe820b8ff2e0                 sh wait
 29331    1 3   2        80   fffffe82061ab120                ssh select
 20688    1 3   2        80   fffffe8217196680                 sh wait
 24552    1 3   7        80   fffffe82179df580                ssh select
 16014    1 3   7        80   fffffe8209b005e0                 sh wait
 14412    1 3   2        80   fffffe8218d7f8a0                ssh select
 16926    1 3   2        80   fffffe821a716720                 sh wait
 15537    1 3   7        80   fffffe81c5ad6880                ssh select
 18282    1 3   7        80   fffffe8206800520                 sh wait
 14420    1 3   2        80   fffffe82060d0160                ssh select
 17970    1 3   2        80   fffffe82193e54e0                 sh wait
 21443    1 3   7        80   fffffe8214d12ac0                ssh select
 20009    1 3   7        80   fffffe82124940e0                 sh wait
 18174    1 3   0        80   fffffe820b6502c0                ssh select
 25056    1 3   2        80   fffffe82215f3b60                 sh wait
 22994    1 3   2        80   fffffe821971da60                ssh select
 25609    1 3   2        80   fffffe81c4e9f360                 sh wait
 22696    1 3   2        80   fffffe8222716b40                ssh select
 17593    1 3   2        80   fffffe82251301e0                 sh wait
 16480    1 3   7        80   fffffe82215f3320                ssh select
 20589    1 3   7        80   fffffe8216624ae0                 sh wait
 20596    1 3   7        80   fffffe822125a700                ssh select
 10969    1 3   7        80   fffffe81bad31340                 sh wait
 14044    1 3   7        80   fffffe8218f87480                ssh select
 17878    1 3   7        80   fffffe8217de6a80                 sh wait
 15657    1 3   0        80   fffffe8208a401e0                ssh select
 21421    1 3   0        80   fffffe81c53a00c0                 sh wait
 14576    1 3   5        80   fffffe822125a2e0                ssh select
 19912    1 3   2        80   fffffe82220202a0                 sh wait
 17455    1 3   2        80   fffffe822d8cd140                ssh select
 16887    1 3   2        80   fffffe821f3e3440                 sh wait
 18158    1 3   2        80   fffffe8215e19200                ssh select
 17985    1 3   2        80   fffffe820b8e8200                 sh wait
 16716    1 3   0        80   fffffe822d9a7120                ssh select
 19884    1 3   5        80   fffffe820b8ffb20                 sh wait
 25660    1 3   0        80   fffffe8217da1320                ssh select
 22067    1 3   0        80   fffffe820c5741e0                 sh wait
 17620    1 3   5        80   fffffe81c69ad4c0                ssh select
 18165    1 3   5        80   fffffe82183b70e0                 sh wait
 22113    1 3   5        80   fffffe821b76b300                ssh select
 18292    1 3   6        80   fffffe82183e59e0                ssh select
 18510    1 3   5        80   fffffe8217b226a0                 sh wait
 20654    1 3   0        80   fffffe820de4f360                 sh wait
 21304    1 3   0        80   fffffe821995d480                ssh select
 19551    1 3   0        80   fffffe820e4f9740                 sh wait
 27093    1 3   0        80   fffffe8208a7da00                ssh select
 25978    1 3   0        80   fffffe821a716300                 sh wait
 17223    1 3   5        80   fffffe82183e51a0                ssh select
 24635    1 3   0        80   fffffe820de4fba0                 sh wait
 22126    1 3   5        80   fffffe8219d9f320                ssh select
 18244    1 3   0        80   fffffe81cffaeaa0                 sh wait
 22357    1 3   5        80   fffffe8225130a20                ssh select
 17566    1 3   0        80   fffffe82179df160                 sh wait
 17650    1 3   5        80   fffffe81c5e1e8c0                ssh select
 17943    1 3   0        80   fffffe821780f180                 sh wait
 21570    1 3   0        80   fffffe8212f7e4c0                ssh select
 20536    1 3   0        80   fffffe820a0d9ae0                 sh wait
 18404    1 3   5        80   fffffe822d7055e0                ssh select
 16408    1 3   0        80   fffffe8207a97180                 sh wait
 17272    1 3   0        80   fffffe8204fc9980                ssh select
 9560     1 3   0        80   fffffe822d7051c0                 sh wait
 16215    1 3   0        80   fffffe820ad30040                ssh select
 23670    1 3   0        80   fffffe821ac5db00                 sh wait
 16842    1 3   0        80   fffffe821d40c620                ssh select
 12548    1 3   0        80   fffffe81c71ce700                 sh wait
 23637    1 3   5        80   fffffe81c69ad8e0                ssh select
 17966    1 3   0        80   fffffe8207a979c0                 sh wait
 19110    1 3   0        80   fffffe821bd5c0e0                ssh select
 17002    1 3   0        80   fffffe8214c6e360                 sh wait
 23925    1 3   0        80   fffffe8219e3b6c0                ssh select
 19616    1 3   0        80   fffffe8218434560                 sh wait
 21852    1 3   5        80   fffffe82179cb100                ssh select
 16322    1 3   0        80   fffffe82190bc240                 sh wait
 14599    1 3   1        80   fffffe8219b33540                ssh select
 11614    1 3   1        80   fffffe8229aff220                 sh wait
 10527    1 3   0        80   fffffe82220206c0                ssh select
 14101    1 3   1        80   fffffe8219cc1680                 sh wait
 13441    1 3   0        80   fffffe8219d9f740                ssh select
 17221    1 3   1        80   fffffe8209b001c0                 sh wait
 14873    1 3   0        80   fffffe821ae039a0                ssh select
 15011    1 3   0        80   fffffe822d7b2900                 sh wait
 14055    1 3   0        80   fffffe8212a8a340                ssh select
 16749    1 3   1        80   fffffe82207c9780                 sh wait
 18012    1 3   1        80   fffffe8212787080                ssh select
 10499    1 3   1        80   fffffe821a1fa240                 sh wait
 13041    1 3   0        80   fffffe821981c460                ssh select
 15374    1 3   0        80   fffffe821bf6d4a0                 sh wait
 9803     1 3   5        80   fffffe820f3865c0                ssh select
 12534    1 3   0        80   fffffe820f3861a0                 sh wait
 11674    1 3   2        80   fffffe82193e50c0                ssh select
 17698    1 3   0        80   fffffe81c4e9f780                 sh wait
 13811    1 3   0        80   fffffe820cc7d440                ssh select
 11034    1 3   0        80   fffffe821aa2b220                 sh wait
 6294     1 3   0        80   fffffe8218d7f480                ssh select
 12577    1 3   0        80   fffffe82183e55c0                 sh wait
 10701    1 3   5        80   fffffe8217196aa0                ssh select
 6593     1 3   0        80   fffffe8222020ae0                 sh wait
 12702    1 3   1        80   fffffe82188ea360                ssh select
 10806    1 3   0        80   fffffe8208a7d5e0                 sh wait
 6638     1 3   4        80   fffffe820cc7d860                ssh select
 14569    1 3   0        80   fffffe8215e19620                 sh wait
 5202     1 3   3        80   fffffe820c2a6900                ssh select
 10543    1 3   0        80   fffffe82166242a0                 sh wait
 17177    1 3   5        80   fffffe820c574600                ssh select
 11624    1 3   0        80   fffffe822d7b24e0                 sh wait
 16329    1 3   0        80   fffffe81cfe7f660                ssh select
 9943     1 3   0        80   fffffe8222123b20                 sh wait
 16096    1 3   5        80   fffffe82183b7920                ssh select
 11557    1 3   0        80   fffffe81c4e9fba0                 sh wait
 14572    1 3   0        80   fffffe8220f0a5e0                ssh select
 13731    1 3   0        80   fffffe81c84acb60                 sh wait
 10907    1 3   5        80   fffffe822d6c45a0                ssh select
 10785    1 3   0        80   fffffe81c53a04e0                 sh wait
 10925    1 3   5        80   fffffe8211b7b840                ssh select
 10573    1 3   0        80   fffffe8219973020                 sh wait
 7697     1 3   5        80   fffffe821a084340                ssh select
 16269    1 3   0        80   fffffe8217da1b60                 sh wait
 13970    1 3   0        80   fffffe82190d4680                ssh select
 7270     1 3   0        80   fffffe820b7e4000                 sh wait
 16358    1 3   0        80   fffffe820e4f9320                ssh select
 13381    1 3   0        80   fffffe821c93d040                 sh wait
 4222     1 3   0        80   fffffe820b6506e0               less ttyraw
 2320     1 3   6        80   fffffe821bf6d8c0             dumpfs pipe_wr
 7456     1 3   6        80   fffffe821b5bd5c0               tcsh pause
 1250     1 3   0        80   fffffe821b76b720               tcsh pause
 1607     1 3   4        80   fffffe8214c6eba0              xterm select
 330      1 3   0        80   fffffe82169cdb00               tcsh pause
 1457     1 3   5        80   fffffe8214d12280               sshd select
 787      1 3   0        80   fffffe8214d126a0               sshd select
 1251     1 3   0        80   fffffe82166246c0               tcsh ttyraw
 746      1 3   1        80   fffffe8215cf9220               tcsh pause
 1503     1 3   5        80   fffffe8215cf9640              xterm select
 603      1 3   6        80   fffffe8215cf9a60               tcsh pause
 1112     1 3   2        80   fffffe8215e19a40               sshd select
 338      1 3   0        80   fffffe8215dea1e0               sshd select
 1105     1 3   3         0   fffffe8215dea600          conserver tstile
 1210     1 3   4         0   fffffe82179cb520          conserver tstile
 921      1 3   7         0   fffffe82232b56a0          conserver tstile
 918      1 3   7         0   fffffe8219973440          conserver tstile
 527      1 3   3         0   fffffe8218bbb420          conserver tstile
 264      1 3   3         0   fffffe82188ea780          conserver tstile
 869      1 3   2         0   fffffe821a1fa660          conserver tstile
 708      1 3   6         0   fffffe821b925560          conserver tstile
 959      1 3   2         0   fffffe822297fb00          conserver tstile
 1052     1 3   3         0   fffffe822a314200          conserver tstile
 763      1 3   0        80   fffffe822dc72500              getty ttyraw
 1149     1 3   4         0   fffffe822d8cd560          conserver tstile
 435      1 3   6        80   fffffe82207c9ba0          conserver select
 361      1 3   0        80   fffffe821d42a4c0               cron nanoslp
 370      1 3   6        80   fffffe821bc42060          python2.7 select
 646      1 3   5        80   fffffe821c93d460               tcsh ttyraw
 475      1 3   0        80   fffffe821c93d880               tcsh pause
 692      1 3   0        80   fffffe821cdfe020              xterm select
 557      1 3   2        80   fffffe821cdfe440               tcsh pause
 495      1 3   7        80   fffffe821cdfe860               sshd select
 170      1 3   0        80   fffffe822026b000               sshd select
 112      1 3   6        80   fffffe822026b420              inetd kqueue
 709      1 3   6        80   fffffe822026b840              nginx kqueue
 516      1 3   1        80   fffffe8222716720              nginx pause
 643      1 3   1        80   fffffe8222716300               qmgr kqueue
 785  >   1 7   6         0   fffffe8222123700             master
 438      1 3   0        80   fffffe82232b5ac0               sshd select
 335      1 3   6         0   fffffe822d6c49c0           tac_plus tstile
 439      1 3   6        80   fffffe822555c260             powerd kqueue
 446      1 3   3        80   fffffe822555c680               ntpd pause
 398      4 3   2        80   fffffe822579d660              dhcpd parked
 398      3 3   2        80   fffffe822579da80              dhcpd kqueue
 398      2 3   0        80   fffffe822a8d3600              dhcpd parked
 398      1 3   1        80   fffffe822a6785c0              dhcpd parked
 83   >   1 7   7         0   fffffe810f34a300            syslogd
 1        1 3   2        80   fffffe810f57a900               init wait
 0      123 3   7       200   fffffe822a8d3a20            npfgc-0 xchicv
 0      122 3   2       200   fffffe822d705a00              ipmi0 ipmi0
 0      121 3   0       200   fffffe822dc720e0            physiod physiod
 0      120 3   0       200   fffffe822dc08100           aiodoned aiodoned
 0    > 119 7   5       200   fffffe822dc08520            ioflush
 0      118 3   0       200   fffffe822dc08940           pgdaemon pgdaemon
 0      115 3   0       280   fffffe822dc72920              spkr1 bellcv
 0      114 3   5       280   fffffe810f57a0c0           audiomix play
 0      113 3   5       280   fffffe810edcb2a0           audiorec record
 0      112 3   0       200   fffffe810f23d2c0          atapibus0 sccomp
 0      110 3   2       200   fffffe810f319b60               usb4 usbevt
 0      109 3   3       200   fffffe810f34ab40               usb3 usbevt
 0      108 3   6       200   fffffe810f33b700               usb1 usbevt
 0      107 3   6       200   fffffe810f33b2e0               usb2 usbevt
 0      106 3   2       200   fffffe810f34a720               usb0 usbevt
 0      105 3   2       200   fffffe810f57a4e0               usb5 usbevt
 0      104 3   0       200   fffffe810f4f30a0            rt_free rt_free
 0      103 3   0       200   fffffe810f4f34c0              unpgc unpgc
 0      102 3   0       200   fffffe810f4f38e0    key_timehandler key_timehandler

 0      101 3   7       200   fffffe810f494080    icmp6_wqinput/7 icmp6_wqinput
 0      100 3   6       200   fffffe810f4944a0    icmp6_wqinput/6 icmp6_wqinput
 0       99 3   5       200   fffffe810f4948c0    icmp6_wqinput/5 icmp6_wqinput
 0       98 3   4       200   fffffe810f4f5060    icmp6_wqinput/4 icmp6_wqinput
 0       97 3   3       200   fffffe810f4f5480    icmp6_wqinput/3 icmp6_wqinput
 0       96 3   2       200   fffffe810f4f58a0    icmp6_wqinput/2 icmp6_wqinput
 0       95 3   1       200   fffffe810f3ce040    icmp6_wqinput/1 icmp6_wqinput
 0       94 3   0       200   fffffe810f3ce460    icmp6_wqinput/0 icmp6_wqinput
 0       93 3   6       200   fffffe810f3ce880          nd6_timer tstile
 0       92 3   7       200   fffffe810f38f020     icmp_wqinput/7 icmp_wqinput
 0       91 3   6       200   fffffe810f38f440     icmp_wqinput/6 icmp_wqinput
 0       90 3   5       200   fffffe810f38f860     icmp_wqinput/5 icmp_wqinput
 0       89 3   4       200   fffffe810f3a0000     icmp_wqinput/4 icmp_wqinput
 0       88 3   3       200   fffffe810f3a0420     icmp_wqinput/3 icmp_wqinput
 0       87 3   2       200   fffffe810f3a0840     icmp_wqinput/2 icmp_wqinput
 0       86 3   1       200   fffffe810f361360     icmp_wqinput/1 icmp_wqinput
 0       85 3   0       200   fffffe810f361780     icmp_wqinput/0 icmp_wqinput
 0       84 3   4       200   fffffe810f33bb20           rt_timer rt_timer
 0       83 3   0       200   fffffe810f361ba0        vmem_rehash vmem_rehash
 0       82 3   0       200   fffffe810f320340          coretemp3 coretemp3
 0       81 3   3       200   fffffe810f320760          coretemp2 coretemp2
 0       80 3   4       200   fffffe810f320b80          coretemp1 coretemp1
 0       79 3   5       200   fffffe810f319320          coretemp0 coretemp0
 0       78 3   5       200   fffffe810f319740               mfi0 mfi0
 0       69 3   5       200   fffffe810f23d6e0            atabus1 atath
 0       68 3   1       200   fffffe810f23db00            atabus0 atath
 0       66 3   0       200   fffffe810edcb6c0           scsibus0 sccomp
 0       65 3   6       200   fffffe810edcbae0         usbtask-dr usbtsk
 0       64 3   0       200   fffffe810ed12280         usbtask-hc usbtsk
 0       63 3   7       200   fffffe810ed126a0               bnx3 bnx3
 0       62 3   5       200   fffffe810ed12ac0               bnx2 bnx2
 0       61 3   4       200   fffffe810eb81260               bnx1 bnx1
 0       60 3   6       200   fffffe810eb81680               bnx0 bnx0
 0    >  59 7   1       200   fffffe810eb81aa0               ipmi
 0       58 3   7       200   fffffe810ead2240            xcall/7 xcall
 0       57 1   7       200   fffffe810ead2660          softser/7
 0       56 1   7       200   fffffe810ead2a80          softclk/7
 0       55 1   7       200   fffffe810eacb220          softbio/7
 0       54 1   7       200   fffffe810eacb640          softnet/7
 0       53 1   7       201   fffffe810eacba60             idle/7
 0       52 3   6       200   fffffe810ea7a200            xcall/6 xcall
 0       51 1   6       200   fffffe810ea7a620          softser/6
 0       50 1   6       200   fffffe810ea7aa40          softclk/6
 0       49 1   6       200   fffffe810ea731e0          softbio/6
 0       48 1   6       200   fffffe810ea73600          softnet/6
 0       47 1   6       201   fffffe810ea73a20             idle/6
 0       46 3   5       200   fffffe810ea2a1c0            xcall/5 xcall
 0       45 1   5       200   fffffe810ea2a5e0          softser/5
 0       44 1   5       200   fffffe810ea2aa00          softclk/5
 0       43 1   5       200   fffffe810ea131a0          softbio/5
 0       42 1   5       200   fffffe810ea135c0          softnet/5
 0       41 1   5       201   fffffe810ea139e0             idle/5
 0       40 3   4       200   fffffe810e9d2180            xcall/4 xcall
 0       39 1   4       200   fffffe810e9d25a0          softser/4
 0       38 1   4       200   fffffe810e9d29c0          softclk/4
 0       37 1   4       200   fffffe810e9bb160          softbio/4
 0       36 1   4       200   fffffe810e9bb580          softnet/4
 0    >  35 7   4       201   fffffe810e9bb9a0             idle/4
 0       34 3   3       200   fffffe810e97a140            xcall/3 xcall
 0       33 1   3       200   fffffe810e97a560          softser/3
 0       32 1   3       200   fffffe810e97a980          softclk/3
 0       31 1   3       200   fffffe810e963120          softbio/3
 0       30 1   3       200   fffffe810e963540          softnet/3
 0    >  29 7   3       201   fffffe810e963960             idle/3
 0       28 3   2       200   fffffe810e92a100            xcall/2 xcall
 0       27 1   2       200   fffffe810e92a520          softser/2
 0       26 1   2       200   fffffe810e92a940          softclk/2
 0       25 1   2       200   fffffe810e90b0e0          softbio/2
 0       24 1   2       200   fffffe810e90b500          softnet/2
 0       23 1   2       201   fffffe810e90b920             idle/2
 0       22 3   1       200   fffffe810e8d20c0            xcall/1 xcall
 0       21 1   1       200   fffffe810e8d24e0          softser/1
 0       20 1   1       200   fffffe810e8d2900          softclk/1
 0       19 1   1       200   fffffe810e8ab0a0          softbio/1
 0       18 1   1       200   fffffe810e8ab4c0          softnet/1
 0       17 1   1       201   fffffe810e8ab8e0             idle/1
 0       16 3   7       200   fffffe822dd92080           lnxsyswq lnxsyswq
 0       15 3   0       200   fffffe822dd924a0             sysmon smtaskq
 0       14 3   0       200   fffffe822dd928c0         pmfsuspend pmfsuspend
 0       13 3   7       200   fffffe822e1b1060           pmfevent pmfevent
 0       12 3   0       200   fffffe822e1b1480         sopendfree sopendfr
 0       11 3   6       200   fffffe822e1b18a0           nfssilly nfssilly
 0       10 3   3       200   fffffe822f5d0040            cachegc cachegc
 0        9 3   7       200   fffffe822f5d0460             vdrain vdrain
 0        8 3   0       200   fffffe822f5d0880          modunload mod_unld
 0        7 3   0       200   fffffe822f5eb020            xcall/0 xcall
 0    >   6 7   0       200   fffffe822f5eb440          softser/0
 0        5 1   0       200   fffffe822f5eb860          softclk/0
 0        4 1   0       200   fffffe822f607000          softbio/0
 0    >   3 7   0       200   fffffe822f607420          softnet/0
 0    >   2 7   0       201   fffffe822f607840             idle/0
 0        1 3   7       200   ffffffff81481b80            swapper uvm

 db{0}> trace/a fffffe822f5eb440
 trace: pid 0 lid 6 at 0xffff8000ae6a2d78
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 bus_space_read_4() at netbsd:bus_space_read_4+0xa
 mfi_intr() at netbsd:mfi_intr+0x37
 intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x36
 Xhandle_ioapic_level8() at netbsd:Xhandle_ioapic_level8+0x6a
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff8000ae6a30f0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:

 db{0}> trace/a fffffe822f607420
 trace: pid 0 lid 3 at 0xffff8000ae693e20
 sowakeup() at netbsd:sowakeup+0x58
 udp4_sendup() at netbsd:udp4_sendup+0x15d
 udp_input() at netbsd:udp_input+0x4ed
 ipintr() at netbsd:ipintr+0x9e1
 softint_dispatch() at netbsd:softint_dispatch+0xeb
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff8000ae6940f0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:

 db{0}> trace/a fffffe822f607840
 trace: pid 0 lid 2 at 0xffff8000ae645f90
 acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
 acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
 idle_loop() at netbsd:idle_loop+0x1b5
 db{0}> 

 and all the locks:

 db{0}> show all locks
 [Locks tracked through LWPs]
 Locks held by an LWP (pickup):
 Lock 0 (initialized at filedesc_ctor)
 lock address : 0xfffffe821a4f6c40 type     :     sleep/adaptive
 initialized  : 0xffffffff8098863f
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  2
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe820a0d92a0
 last locked* : 0xffffffff8098ec35 unlocked : 0xffffffff8098ecde
 owner field  : 0xfffffe820a0d92a0 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c4f40.
 => No active turnstile for this lock.

 Locks held by an LWP (master):
 Lock 0 (initialized at filedesc_ctor)
 lock address : 0xfffffe822a3a4980 type     :     sleep/adaptive
 initialized  : 0xffffffff8098863f
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  6
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe8222123700
 last locked* : 0xffffffff8098d870 unlocked : 0xffffffff8098a1d6
 owner field  : 0xfffffe8222123700 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c51c0.
 => No active turnstile for this lock.

 Locks held by an LWP (syslogd):
 Lock 0 (initialized at filedesc_ctor)
 lock address : 0xfffffe822a3a4640 type     :     sleep/adaptive
 initialized  : 0xffffffff8098863f
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  7
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe810f34a300
 last locked* : 0xffffffff8098ec35 unlocked : 0xffffffff8098ecde
 owner field  : 0xfffffe810f34a300 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c4f40.
 => No active turnstile for this lock.

 Locks held by an LWP (npfgc-0):
 Lock 0 (initialized at npf_config_init)
 lock address : 0xfffffe822dc5f608 type     :     sleep/adaptive
 initialized  : 0xffffffff8186e8cf
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  7
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe822a8d3a20
 last locked* : 0xffffffff81873df3 unlocked : 0xffffffff81873e03
 owner field  : 0xfffffe822a8d3a20 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c4ed0.
 => No active turnstile for this lock.

 Locks held by an LWP (ioflush):
 Lock 0 (initialized at vcache_alloc)
 lock address : 0xfffffe82278a2130 type     :     sleep/adaptive
 initialized  : 0xffffffff80a402a2
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  5
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe822dc08520
 last locked* : 0xffffffff80a57979 unlocked : 0xffffffff80a579c4
 owner/count  : 0xfffffe822dc08520 flags    : 0x0000000000000004

 Turnstile chain at 0xffffffff814c5120.
 => No active turnstile for this lock.

 Locks held by an LWP (softnet/0):
 Lock 0 (initialized at soinit)
 lock address : 0xfffffe822fa90f40 type     :     sleep/adaptive
 initialized  : 0xffffffff80a190ce
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                 14
 current cpu  :                  0 last held:                  0
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe822f607420
 last locked* : 0xffffffff806ce6bc unlocked : 0xffffffff806ceb1f
 owner field  : 0xfffffe822f607420 wait/spin:                1/0

 Turnstile chain at 0xffffffff814c5140.
 => Turnstile at 0xfffffe8208bfe7e8 (wrq=0xfffffe8208bfe808, rdq=0xfffffe8208bfe8
 18).
 => 0 waiting readers:
 => 14 waiting writers: 0xfffffe810f3ce880 0xfffffe820b650b00 0xfffffe821a1fa660 0xfffffe821b925560 0xfffffe822297fb00 0xfffffe822a314200 0xfffffe82188ea780 0xfffffe8215dea600 0xfffffe82179cb520 0xfffffe8218bbb420 0xfffffe82232b56a0 0xfffffe822d6c49c0 0xfffffe8219973440 0xfffffe822d8cd560


 [Locks tracked through CPUs]
 Locks held on CPU 0:
 Lock 0 (initialized at main)
 lock address : 0xffffffff814c3940 type     :               spin
 initialized  : 0xffffffff80d77491
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  5
 current cpu  :                  0 last held:                  0
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe822f607420
 last locked* : 0xffffffff806ce6c6 unlocked : 0xffffffff809ba51e
 curcpu holds :                  2 wanted by: 000000000000000000

 Lock 1 (initialized at com_attach_subr)
 lock address : 0xfffffe810f27b1e0 type     :               spin
 initialized  : 0xffffffff805f4924
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  0
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe822f5eb440
 last locked* : 0xffffffff805f3e84 unlocked : 0xffffffff805f4063
 owner field  : 0x0000000000010800 wait/spin:                0/1


 db{0}> 

From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, port-amd64-maintainer@netbsd.org, 
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, he@NetBSD.org
Cc: 
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network interfaces in use
Date: Thu, 17 May 2018 14:28:54 -0400

 On May 17,  2:30pm, he@NetBSD.org (Havard Eidnes) wrote:
 -- Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx net

 | The following reply was made to PR port-amd64/53155; it has been noted by GNATS.
 | 
 | From: Havard Eidnes <he@NetBSD.org>
 | To: gnats-bugs@NetBSD.org, hannken@eis.cs.tu-bs.de
 | Cc: port-amd64-maintainer@netbsd.org
 | Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 |  network interfaces in use
 | Date: Thu, 17 May 2018 16:28:56 +0200 (CEST)

 Try this?

 christos

 Index: if_bnx.c
 ===================================================================
 RCS file: /cvsroot/src/sys/dev/pci/if_bnx.c,v
 retrieving revision 1.64
 diff -u -u -r1.64 if_bnx.c
 --- if_bnx.c	8 May 2018 04:11:09 -0000	1.64
 +++ if_bnx.c	17 May 2018 18:28:21 -0000
 @@ -3942,6 +3942,7 @@
  	struct ifnet *ifp = &sc->bnx_ec.ec_if;
  	struct bnx_pkt *pkt;
  	int i, s;
 +	bool got = false;

  	for (i = 0; i < 4; i++) { /* magic! */
  		pkt = pool_get(bnx_tx_pool, PR_WAITOK);
 @@ -3961,17 +3962,20 @@
  		TAILQ_INSERT_TAIL(&sc->tx_free_pkts, pkt, pkt_entry);
  		sc->tx_pkt_count++;
  		mutex_exit(&sc->tx_pkt_mtx);
 +		got = true;
  	}

  	mutex_enter(&sc->tx_pkt_mtx);
  	CLR(sc->bnx_flags, BNX_ALLOC_PKTS_FLAG);
  	mutex_exit(&sc->tx_pkt_mtx);

 -	/* fire-up TX now that allocations have been done */
 -	s = splnet();
 -	if (!IFQ_IS_EMPTY(&ifp->if_snd))
 -		bnx_start(ifp);
 -	splx(s);
 +	if (got) {
 +		/* fire-up TX now that allocations have been done */
 +		s = splnet();
 +		if (!IFQ_IS_EMPTY(&ifp->if_snd))
 +			bnx_start(ifp);
 +		splx(s);
 +	}

  	return;

 @@ -3979,6 +3983,10 @@
  	bus_dmamap_destroy(sc->bnx_dmatag, pkt->pkt_dmamap);
  put:
  	pool_put(bnx_tx_pool, pkt);
 +	mutex_enter(&sc->tx_pkt_mtx);
 +	CLR(sc->bnx_flags, BNX_ALLOC_PKTS_FLAG);
 +	mutex_exit(&sc->tx_pkt_mtx);
 +
  	return;
  }

 @@ -4949,8 +4957,8 @@

  		if (sc->tx_pkt_count <= TOTAL_TX_BD &&
  		    !ISSET(sc->bnx_flags, BNX_ALLOC_PKTS_FLAG)) {
 -			workqueue_enqueue(sc->bnx_wq, &bnx_wk, NULL);
  			SET(sc->bnx_flags, BNX_ALLOC_PKTS_FLAG);
 +			workqueue_enqueue(sc->bnx_wq, &bnx_wk, NULL);
  		}

  		mutex_exit(&sc->tx_pkt_mtx);

From: Havard Eidnes <he@NetBSD.org>
To: christos@zoulas.com
Cc: gnats-bugs@NetBSD.org, port-amd64-maintainer@netbsd.org,
 gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Thu, 17 May 2018 21:26:02 +0200 (CEST)

 > Try this?

 Thanks, did that, but it didn't fix the problem.

 Additionally, when I tried breaking into DDB , all I got was
 "fata" (first part of the text I usually get) and it hung,
 requiring a hard reset.  Luckily we have remote management of the
 host so it'll be ready for another try shortly.

 I did a second try, and this time I got into DDB; here's the
 console log:

 fatal breakpoint trap in supervisor mode
 trap type 1 code 0 rip 0xffffffff8021ff65 cs 0x8 rflags 0x202 cr2 0x7ece618de000
 curlwp 0xfffffe822f5eb440 pid 0.6 lowest kstack 0xffff8000ae6972c0
 Stopped in pid 0.6 (system) at  netbsd:breakpoint+0x5:  leave
 db{0}> ps
 PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 1212 >   1 7   0         0   fffffe8220810940                ssh
 829      1 3   1        80   fffffe821cd5dac0                 sh wait
 490      1 3   5        80   fffffe821e5d40e0               tcsh ttyraw
 745      1 3   6        80   fffffe821c260680             systat ttyraw
 1251     1 3   5        80   fffffe821c260260               tcsh pause
 861      1 3   7        80   fffffe821c260aa0               tcsh pause
 860      1 3   6        80   fffffe821dad51c0                ssh select
 1243     1 3   6        80   fffffe821dd75240                 sh wait
 602      1 3   6        80   fffffe821dc2d9e0                ssh select
 985      1 3   6        80   fffffe821dd75660                 sh wait
 1240     1 3   0        80   fffffe821cb1b160                ssh select
 1239     1 3   6        80   fffffe821dd75a80                 sh wait
 854      1 3   5        80   fffffe821e10d140                ssh select
 1237     1 3   6        80   fffffe821dce7220                 sh wait
 340      1 3   0        80   fffffe821e26d540                ssh select
 979      1 3   6        80   fffffe821dce7640                 sh wait
 338      1 3   0        80   fffffe82223d3920                ssh select
 721      1 3   0        80   fffffe821fd2f2a0                 sh wait
 336      1 3   2        80   fffffe821e8f64e0                ssh select
 591      1 3   2        80   fffffe821dce7a60                 sh wait
 1230     1 3   0        80   fffffe821d394200          conserver select
 1229     1 3   4        80   fffffe821d394620                ssh select
 844      1 3   6        80   fffffe821d394a40                ssh select
 459      1 3   3        80   fffffe821d87d1e0                 sh wait
 457      1 3   1        80   fffffe821d87da20                 sh wait
 1223     1 3   0        80   fffffe821dad55e0                ssh select
 838      1 3   2        80   fffffe821dad5a00                 sh wait
 580      1 3   7        80   fffffe821dc2d5c0                ssh select
 322      1 3   1        80   fffffe821c99b180                 sh wait
 320      1 3   0        80   fffffe821c99b9c0                ssh select
 830      1 3   5        80   fffffe821cb1b580                 sh wait
 1211     1 3   1        80   fffffe821e10d560                ssh select
 954      1 3   0        80   fffffe821e10d980                 sh wait
 1207     1 3   7        80   fffffe821e26d960          conserver select
 950      1 3   2        80   fffffe821d3f3100                ssh select
 1077     1 3   2        80   fffffe821d3f3520                 sh wait
 692      1 3   0        80   fffffe821d3f3940                ssh select
 306      1 3   5        80   fffffe821e5d4500                 sh wait
 945      1 3   4        80   fffffe821e5d4920                ssh select
 304      1 3   0        80   fffffe821e8f60c0                ssh select
 942      1 3   4        80   fffffe821e8f6900                 sh wait
 813      1 3   4        80   fffffe82214745c0                ssh select
 556      1 3   0        80   fffffe821d97b0a0                 sh wait
 299      1 3   7        80   fffffe821d97b4c0                 sh wait
 298      1 3   7        80   fffffe821d97b8e0                ssh select
 297      1 3   6        80   fffffe821ed7d080                 sh wait
 680      1 3   0        80   fffffe821ed7d4a0                ssh select
 1063     1 3   6        80   fffffe821ed7d8c0                 sh wait
 550      1 3   5        80   fffffe821dd63060                ssh select
 933      1 3   6        80   fffffe821dd63480                 sh wait
 804      1 3   3        80   fffffe821dd638a0                ssh select
 803      1 3   6        80   fffffe821e8b7040                 sh wait
 674      1 3   7        80   fffffe821e8b7460                ssh select
 289      1 3   0        80   fffffe821e8b7880                 sh wait
 288      1 3   0        80   fffffe821e163020                ssh select
 1183     1 3   5        80   fffffe821e163440                ssh select
 1182     1 3   4        80   fffffe821e163860                 sh wait
 925      1 3   5        80   fffffe822023e6e0                 sh wait
 1180     1 3   2        80   fffffe8220116000          conserver select
 1051     1 3   1        80   fffffe82216345a0                ssh select
 922      1 3   6        80   fffffe8220116420                 sh wait
 793      1 3   4        80   fffffe8220116840          conserver select
 918      1 3   7        80   fffffe821f6f5360          conserver select
 1045     1 3   0        80   fffffe82218ba4e0                ssh select
 660      1 3   3        80   fffffe821f6f5780                 sh wait
 531      1 3   0        80   fffffe821f6f5ba0          conserver select
 402      1 3   7        80   fffffe821fa76340                ssh select
 529      1 3   2        80   fffffe821fa76760                ssh select
 400      1 3   7        80   fffffe821fa76b80                 sh wait
 911      1 3   0        80   fffffe821f4bf320                ssh select
 782      1 3   4        80   fffffe821f4bf740                 sh wait
 1037     1 3   0        80   fffffe821f4bfb60                 sh wait
 908      1 3   1        80   fffffe821eb92300                ssh select
 267      1 3   4        80   fffffe821eb92720                ssh select
 266      1 3   3        80   fffffe821eb92b40                 sh wait
 265      1 3   0        80   fffffe82200462e0                 sh wait
 264      1 3   6        80   fffffe8220046700                ssh select
 775      1 3   1        80   fffffe8220046b20                 sh wait
 902      1 3   3        80   fffffe822023e2c0                ssh select
 900      1 3   6        80   fffffe822023eb00                 sh wait
 386      1 3   4        80   fffffe821fd2f6c0                ssh select
 385      1 3   4        80   fffffe821fd2fae0                 sh wait
 384      1 3   2        80   fffffe822057e280                ssh select
 762      1 3   2        80   fffffe822057e6a0                 sh wait
 827      1 3   7        80   fffffe822057eac0                ssh select
 889      1 3   1        80   fffffe82207e6260                 sh wait
 824      1 3   0        80   fffffe82207e6680                ssh select
 823      1 3   2        80   fffffe82207e6aa0                 sh wait
 566      1 3   5        80   fffffe82208fe240                ssh select
 949      1 3   0        80   fffffe82208fe660                ssh select
 500      1 3   5        80   fffffe82208fea80                 sh wait
 755      1 3   7        80   fffffe822099d220                 sh wait
 370      1 3   5        80   fffffe822099d640                ssh select
 753      1 3   0        80   fffffe822099da60                 sh wait
 368      1 3   1        80   fffffe821ff42200          conserver select
 751      1 3   5        80   fffffe821ff42620                ssh select
 558      1 3   0        80   fffffe821ff42a40                ssh select
 877      1 3   1        80   fffffe82211651e0                 sh wait
 620      1 3   6        80   fffffe8221165600                ssh select
 170      1 3   0        80   fffffe822d985980                 sh wait
 171      1 3   2        80   fffffe8221165a20                ssh select
 361      1 3   7        80   fffffe822d65e5c0                 sh wait
 296      1 3   5        80   fffffe822079b1c0                ssh select
 935      1 3   2        80   fffffe822c1fc200                 sh wait
 614      1 3   1        80   fffffe822079b5e0                ssh select
 741      1 3   3        80   fffffe822c3a6660                 sh wait
 868      1 3   4        80   fffffe822079ba00                ssh select
 867      1 3   4        80   fffffe82214741a0                 sh wait
 161      1 3   2        80   fffffe82214749e0                 sh wait
 160      1 3   7        80   fffffe8221634180                ssh select
 798      1 3   3        80   fffffe82216349c0                ssh select
 733      1 3   6        80   fffffe8221afd160                ssh select
 796      1 3   3        80   fffffe8221afd580                 sh wait
 923      1 3   4        80   fffffe8221afd9a0                 sh wait
 538      1 3   7        80   fffffe822c8105e0                ssh select
 665      1 3   6        80   fffffe82219fc140                 sh wait
 920      1 3   1        80   fffffe82288c56a0                ssh select
 855      1 3   7        80   fffffe82219fc560                 sh wait
 726      1 3   1        80   fffffe82219fc980                ssh select
 917      1 3   1        80   fffffe82211da120                 sh wait
 276      1 3   2        80   fffffe82211da540                 sh wait
 595      1 3   0        80   fffffe82211da960                ssh select
 210      1 3   5        80   fffffe8220810100                 sh wait
 527      1 3   1        80   fffffe82223d30e0          conserver select
 846      1 3   1        80   fffffe82223d3500                ssh select
 909      1 3   3        80   fffffe82243a3780                 sh wait
 716      1 3   0        80   fffffe822d9659c0                ssh select
 138      1 3   3        80   fffffe82218ba0c0                 sh wait
 136      1 3   3        80   fffffe82218ba900                ssh select
 839      1 3   0        80   fffffe8222b380a0                 sh wait
 710      1 3   0        80   fffffe8222b384c0                ssh select
 581      1 3   6        80   fffffe8222b388e0                 sh wait
 516      1 3   4        80   fffffe822d8e6160                ssh select
 643      1 3   7        80   fffffe822594f340                ssh select
 194      1 3   3        80   fffffe822646c080                 sh wait
 193      1 3   1        80   fffffe822646c4a0                ssh select
 192      1 3   0        80   fffffe822646c8c0                 sh wait
 856      1 3   4        80   fffffe8227d1b060                 sh wait
 73       1 3   0        80   fffffe822d8e69a0                ssh select
 72       1 3   1        80   fffffe8227d1b480                 sh wait
 75       1 3   5        80   fffffe8227d1b8a0                ssh select
 74       1 3   5        80   fffffe822d763040                 sh wait
 579      1 3   0        80   fffffe822d763460                ssh select
 561      1 3   6        80   fffffe82269e6b60                 sh wait
 845      1 3   1        80   fffffe810f2352c0                ssh select
 654      1 3   7        80   fffffe8223d7b840                 sh wait
 871      1 3   6        80   fffffe822d985140                ssh select
 232      1 3   7        80   fffffe822d985560                 sh wait
 178      1 3   6        80   fffffe822dbc9540                ssh select
 563      1 3   0        80   fffffe822dbc9120                 sh wait
 676      1 3   4        80   fffffe822d763880                ssh select
 226      1 3   2        80   fffffe822bcfb020                 sh wait
 675      1 3   4        80   fffffe822c1fc620                ssh select
 534      1 3   5        80   fffffe822bcfb440                 sh wait
 541      1 3   3        80   fffffe822bcfb860          conserver select
 859      1 3   1        80   fffffe8223d7b000          conserver select
 732      1 3   2        80   fffffe822dc740e0              getty ttyraw
 759      1 3   5        80   fffffe8223d7b420          conserver select
 176      1 3   1        80   fffffe822c8101c0          conserver select
 524      1 3   7        80   fffffe82243a3360               cron nanoslp
 169      1 3   4        80   fffffe82243a3ba0          python2.7 select
 549      1 3   0        80   fffffe822594f760              xterm select
 853      1 3   0        80   fffffe82288c5280               tcsh pause
 663      1 3   1        80   fffffe82288c5ac0               sshd select
 895      1 3   5        80   fffffe822d65e1a0               sshd select
 734      1 3   2        80   fffffe822594fb80              xterm select
 570      1 3   0        80   fffffe822d9655a0               tcsh pause
 697      1 3   5        80   fffffe822cdf5a60               sshd select
 502      1 3   0        80   fffffe822c3a6a80               sshd select
 107      1 3   0        80   fffffe82273e8720              inetd kqueue
 97       1 3   6        80   fffffe82271af2e0              nginx kqueue
 96       1 3   0        80   fffffe82271af700              nginx pause
 647      1 3   0        80   fffffe8226ea06c0    trivial-rewrite kqueue
 601      1 3   0        80   fffffe8228feb680               qmgr kqueue
 702      1 3   6        80   fffffe822d65e9e0             pickup kqueue
 763      1 3   2        80   fffffe8226ea0ae0             master kqueue
 474      1 3   7        80   fffffe8228febaa0               sshd select
 463      1 3   4        80   fffffe822c3a6240           tac_plus select
 428      1 3   6        80   fffffe822cdf5220             powerd kqueue
 422      1 3   7        80   fffffe822cdf5640               ntpd pause
 337      4 3   7        80   fffffe822c1fca40              dhcpd parked
 337      3 3   7        80   fffffe822c810a00              dhcpd kqueue
 337      2 3   7         0   fffffe822d8e6580              dhcpd biowait
 337      1 3   0        80   fffffe822c7601e0              dhcpd parked
 212      1 2   7         0   fffffe810f33a300            syslogd
 1        1 3   0        80   fffffe810f572900               init wait
 0      123 3   3       200   fffffe822c760a20            npfgc-0 xchicv
 0      122 3   3       200   fffffe822d965180              ipmi0 ipmi0
 0      121 3   7       200   fffffe822dbf2940            physiod physiod
 0      120 3   4       200   fffffe822dbc9960           aiodoned aiodoned
 0      119 3   5       200   fffffe822dbf2100            ioflush tstile
 0      118 3   0       200   fffffe822dbf2520           pgdaemon pgdaemon
 0      115 3   0       280   fffffe822dc74500              spkr1 bellcv
 0      114 3   5       280   fffffe822dc74920           audiomix play
 0      113 3   5       280   fffffe810edc32a0           audiorec record
 0      112 3   5       200   fffffe810f5720c0          atapibus0 sccomp
 0      110 3   0       200   fffffe810f33ab40               usb1 usbevt
 0      109 3   6       200   fffffe810f3332e0               usb3 usbevt
 0      108 3   5       200   fffffe810f333700               usb2 usbevt
 0      107 3   0       200   fffffe810f33a720               usb0 usbevt
 0      106 3   5       200   fffffe810f333b20               usb5 usbevt
 0      105 3   5       200   fffffe810f5724e0               usb4 usbevt
 0      104 3   0       200   fffffe810f50b0a0            rt_free rt_free
 0      103 3   0       200   fffffe810f50b4c0              unpgc unpgc
 0      102 3   5       200   fffffe810f50b8e0    key_timehandler key_timehandler

 0      101 3   7       200   fffffe810f494080    icmp6_wqinput/7 icmp6_wqinput
 0      100 3   6       200   fffffe810f4944a0    icmp6_wqinput/6 icmp6_wqinput
 0       99 3   5       200   fffffe810f4948c0    icmp6_wqinput/5 icmp6_wqinput
 0       98 3   4       200   fffffe810f4d5060    icmp6_wqinput/4 icmp6_wqinput
 0       97 3   3       200   fffffe810f4d5480    icmp6_wqinput/3 icmp6_wqinput
 0       96 3   2       200   fffffe810f4d58a0    icmp6_wqinput/2 icmp6_wqinput
 0       95 3   1       200   fffffe810f3c4040    icmp6_wqinput/1 icmp6_wqinput
 0       94 3   0       200   fffffe810f3c4460    icmp6_wqinput/0 icmp6_wqinput
 0       93 3   7       200   fffffe810f3c4880          nd6_timer nd6_timer
 0       92 3   7       200   fffffe810f38c020     icmp_wqinput/7 icmp_wqinput
 0       91 3   6       200   fffffe810f38c440     icmp_wqinput/6 icmp_wqinput
 0       90 3   5       200   fffffe810f38c860     icmp_wqinput/5 icmp_wqinput
 0       89 3   4       200   fffffe810f385000     icmp_wqinput/4 icmp_wqinput
 0       88 3   3       200   fffffe810f385420     icmp_wqinput/3 icmp_wqinput
 0       87 3   2       200   fffffe810f385840     icmp_wqinput/2 icmp_wqinput
 0       86 3   1       200   fffffe810f36e360     icmp_wqinput/1 icmp_wqinput
 0       85 3   0       200   fffffe810f36e780     icmp_wqinput/0 icmp_wqinput
 0       84 3   5       200   fffffe810f36eba0           rt_timer rt_timer
 0       83 3   0       200   fffffe810f311b60        vmem_rehash vmem_rehash
 0       82 3   5       200   fffffe810f318340          coretemp3 coretemp3
 0       81 3   4       200   fffffe810f318760          coretemp2 coretemp2
 0       80 3   5       200   fffffe810f318b80          coretemp1 coretemp1
 0       79 3   7       200   fffffe810f311320          coretemp0 coretemp0
 0       78 3   7       200   fffffe810f311740               mfi0 mfi0
 0       69 3   5       200   fffffe810f2356e0            atabus1 atath
 0       68 3   3       200   fffffe810f235b00            atabus0 atath
 0       66 3   0       200   fffffe810edc36c0           scsibus0 sccomp
 0       65 3   0       200   fffffe810edc3ae0         usbtask-dr usbtsk
 0       64 3   0       200   fffffe810ed02280         usbtask-hc usbtsk
 0       63 3   7       200   fffffe810ed026a0               bnx3 bnx3
 0       62 3   6       200   fffffe810ed02ac0               bnx2 bnx2
 0       61 3   4       200   fffffe810eb79260               bnx1 bnx1
 0       60 3   1       200   fffffe810eb79680               bnx0 bnx0
 0       59 3   5       200   fffffe810eb79aa0               ipmi ipmipoll
 0       58 3   7       200   fffffe810eaca240            xcall/7 xcall
 0       57 1   7       200   fffffe810eaca660          softser/7
 0       56 1   7       200   fffffe810eacaa80          softclk/7
 0       55 1   7       200   fffffe810eac3220          softbio/7
 0       54 1   7       200   fffffe810eac3640          softnet/7
 0    >  53 7   7       201   fffffe810eac3a60             idle/7
 0       52 3   6       200   fffffe810ea72200            xcall/6 xcall
 0       51 1   6       200   fffffe810ea72620          softser/6
 0       50 1   6       200   fffffe810ea72a40          softclk/6
 0       49 1   6       200   fffffe810ea6b1e0          softbio/6
 0       48 1   6       200   fffffe810ea6b600          softnet/6
 0    >  47 7   6       201   fffffe810ea6ba20             idle/6
 0       46 3   5       200   fffffe810ea221c0            xcall/5 xcall
 0       45 1   5       200   fffffe810ea225e0          softser/5
 0       44 1   5       200   fffffe810ea22a00          softclk/5
 0       43 1   5       200   fffffe810ea0b1a0          softbio/5
 0       42 1   5       200   fffffe810ea0b5c0          softnet/5
 0    >  41 7   5       201   fffffe810ea0b9e0             idle/5
 0       40 3   4       200   fffffe810e9ca180            xcall/4 xcall
 0       39 1   4       200   fffffe810e9ca5a0          softser/4
 0       38 1   4       200   fffffe810e9ca9c0          softclk/4
 0       37 1   4       200   fffffe810e9b3160          softbio/4
 0       36 1   4       200   fffffe810e9b3580          softnet/4
 0    >  35 7   4       201   fffffe810e9b39a0             idle/4
 0       34 3   3       200   fffffe810e972140            xcall/3 xcall
 0       33 1   3       200   fffffe810e972560          softser/3
 0       32 1   3       200   fffffe810e972980          softclk/3
 0       31 1   3       200   fffffe810e95b120          softbio/3
 0       30 1   3       200   fffffe810e95b540          softnet/3
 0    >  29 7   3       201   fffffe810e95b960             idle/3
 0       28 3   2       200   fffffe810e922100            xcall/2 xcall
 0       27 1   2       200   fffffe810e922520          softser/2
 0       26 1   2       200   fffffe810e922940          softclk/2
 0       25 1   2       200   fffffe810e9030e0          softbio/2
 0       24 1   2       200   fffffe810e903500          softnet/2
 0    >  23 7   2       201   fffffe810e903920             idle/2
 0       22 3   1       200   fffffe810e8ca0c0            xcall/1 xcall
 0       21 1   1       200   fffffe810e8ca4e0          softser/1
 0       20 1   1       200   fffffe810e8ca900          softclk/1
 0       19 1   1       200   fffffe810e8a30a0          softbio/1
 0       18 1   1       200   fffffe810e8a34c0          softnet/1
 0    >  17 7   1       201   fffffe810e8a38e0             idle/1
 0       16 3   1       200   fffffe822dd92080           lnxsyswq lnxsyswq
 0       15 3   0       200   fffffe822dd924a0             sysmon smtaskq
 0       14 3   7       200   fffffe822dd928c0         pmfsuspend pmfsuspend
 0       13 3   2       200   fffffe822e1b1060           pmfevent pmfevent
 0       12 3   0       200   fffffe822e1b1480         sopendfree sopendfr
 0       11 3   0       200   fffffe822e1b18a0           nfssilly nfssilly
 0       10 3   1       200   fffffe822f5d0040            cachegc cachegc
 0        9 3   0       200   fffffe822f5d0460             vdrain vdrain
 0        8 3   6       200   fffffe822f5d0880          modunload mod_unld
 0        7 3   0       200   fffffe822f5eb020            xcall/0 xcall
 0    >   6 7   0       200   fffffe822f5eb440          softser/0
 0    >   5 7   0       200   fffffe822f5eb860          softclk/0
 0        4 1   0       200   fffffe822f607000          softbio/0
 0    >   3 7   0       200   fffffe822f607420          softnet/0
 0        2 1   0       201   fffffe822f607840             idle/0
 0        1 3   3       200   ffffffff81481b80            swapper uvm
 db{0}> trace/a fffffe822f5eb440
 trace: pid 0 lid 6 at 0xffff8000ae69adf8
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 DDB lost frame for netbsd:Xdoreti, trying 0xffff8000ae69ae70
 Xdoreti() at netbsd:Xdoreti
 --- interrupt ---
 Bad frame pointer: 0xffff800000000000
 206:
 db{0}> trace/a fffffe822f5eb860
 trace: pid 0 lid 5 at 0xffff8000bdab8ba0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 7f8000000000:
 db{0}> trace/a fffffe822f607420
 trace: pid 0 lid 3 at 0xffff8000ae6960e0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}> show all locks
 [Locks tracked through LWPs]
 Locks held by an LWP (ssh):
 Lock 0 (initialized at uvm_obj_init)
 lock address : 0xfffffe822cf2df00 type     :     sleep/adaptive
 initialized  : 0xffffffff8096aed9
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  1
 current cpu  :                  0 last held:                  0
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe8220810940
 last locked* : 0xffffffff8095cf70 unlocked : 0xffffffff80a41ef5
 owner field  : 0xfffffe8220810940 wait/spin:                1/0

 Turnstile chain at 0xffffffff814c50c0.
 => Turnstile at 0xfffffe810f3865a0 (wrq=0xfffffe810f3865c0, rdq=0xfffffe810f3865d0).
 => 0 waiting readers:
 => 1 waiting writers: 0xfffffe822dbf2100
 Lock 1 (initialized at pmap_create)
 lock address : 0xfffffe8226aa4760 type     :     sleep/adaptive
 initialized  : 0xffffffff802463c2
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  0
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe8220810940
 last locked* : 0xffffffff80247877 unlocked : 0xffffffff80249014
 owner field  : 0xfffffe8220810940 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c5180.
 => No active turnstile for this lock.

 Locks held by an LWP (dhcpd):
 Lock 0 (initialized at vcache_alloc)
 lock address : 0xfffffe822ce13e58 type     :     sleep/adaptive
 initialized  : 0xffffffff80a402e2
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  7
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe822d8e6580
 last locked* : 0xffffffff80a579ce unlocked : 0xffffffff80a57a04
 owner/count  : 0xfffffe822d8e6580 flags    : 0x0000000000000004

 Turnstile chain at 0xffffffff814c4f70.
 => No active turnstile for this lock.

 Locks held by an LWP (npfgc-0):
 Lock 0 (initialized at npf_config_init)
 lock address : 0xfffffe822ca22f08 type     :     sleep/adaptive
 initialized  : 0xffffffff8196d8cf
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  3
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe822c760a20
 last locked* : 0xffffffff81972df3 unlocked : 0xffffffff81972e03
 owner field  : 0xfffffe822c760a20 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c50d0.
 => No active turnstile for this lock.

 Locks held by an LWP (ioflush):
 Lock 0 (initialized at vfs_mount_sysinit)
 lock address : 0xffffffff815cc8d8 type     :     sleep/adaptive
 initialized  : 0xffffffff80a30a85
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  5
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe822dbf2100
 last locked* : 0xffffffff80a30864 unlocked : 0xffffffff80a30966
 owner field  : 0xfffffe822dbf2100 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c5070.
 => No active turnstile for this lock.


 [Locks tracked through CPUs]
 Locks held on CPU 0:
 Lock 0 (initialized at com_attach_subr)
 lock address : 0xfffffe810f2731e0 type     :               spin
 initialized  : 0xffffffff805f4964
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  0
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe822f5eb440
 last locked* : 0xffffffff805f3ec4 unlocked : 0xffffffff805f40a3
 owner field  : 0x0000000000010800 wait/spin:                0/1


 db{0}> 

 Regards,

 - Havard

From: Ryota Ozaki <ozaki-r@netbsd.org>
To: Havard Eidnes <he@netbsd.org>
Cc: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>, port-amd64-maintainer@netbsd.org, 
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network
 interfaces in use
Date: Fri, 18 May 2018 18:07:29 +0900

 On Fri, May 18, 2018 at 4:26 AM Havard Eidnes <he@netbsd.org> wrote:

 > > Try this?

 > Thanks, did that, but it didn't fix the problem.

 > Additionally, when I tried breaking into DDB , all I got was
 > "fata" (first part of the text I usually get) and it hung,
 > requiring a hard reset.  Luckily we have remote management of the
 > host so it'll be ready for another try shortly.

 > I did a second try, and this time I got into DDB; here's the
 > console log:

 Please add '/t' option to show all locks next time. It provides stack traces
 of related LWPs in addition.

 BTW softser/0 I suspected looks just a trace of DDB, so it's not a stopper
 of pserialize_perform...

 Thanks,
    ozaki-r

From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: port-amd64-maintainer@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Sun, 27 May 2018 23:22:16 +0200 (CEST)

 Hi,

 another wedge.  This time with "show all locks/t".

 This time I did not assign an IP address to the bnx3 interface
 which I took up right before it wedged.

 I have the matching kernel core dump.

 Regards,

 - Havard

 ------------------------------


 db{0}> show all locks/t
 [Locks tracked through LWPs]
 Locks held by an LWP (npfgc-0):
 Lock 0 (initialized at npf_config_init)
 lock address : 0xfffffe822d7e54c8 type     :     sleep/adaptive
 initialized  : 0xffffffff8196d8cf
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  5
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe822cf3d1c0
 last locked* : 0xffffffff81972df3 unlocked : 0xffffffff81972e03
 owner field  : 0xfffffe822cf3d1c0 wait/spin:                0/0

 Turnstile chain at 0xffffffff814c5050.
 => No active turnstile for this lock.
 trace: pid 0 lid 123 at 0xffff8000bcec6eb0
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0x127
 xc_wait() at netbsd:xc_wait+0x7c
 pserialize_perform() at netbsd:pserialize_perform+0x102
 npf_conn_gc() at npf:npf_conn_gc+0x1b5
 npf_worker() at npf:npf_worker+0x4b

 Locks held by an LWP (ioflush):
 Lock 0 (initialized at wapbl_start)
 lock address : 0xfffffe810f4fd480 type     :     sleep/adaptive
 initialized  : 0xffffffff80a4a13c
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  2
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe822dc08940
 last locked* : 0xffffffff80a480ce unlocked : 0xffffffff8091396b
 owner/count  : 0xfffffe822dc08940 flags    : 0x0000000000000004

 Turnstile chain at 0xffffffff814c4fc0.
 => No active turnstile for this lock.
 trace: pid 0 lid 119 at 0xffff8000bcceecf0
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0x127
 biowait() at netbsd:biowait+0x61
 wapbl_buffered_flush() at netbsd:wapbl_buffered_flush+0xc6
 wapbl_buffered_write() at netbsd:wapbl_buffered_write+0x69
 wapbl_circ_write() at netbsd:wapbl_circ_write+0xa7
 wapbl_flush() at netbsd:wapbl_flush+0x386
 ffs_sync() at netbsd:ffs_sync+0x220
 VFS_SYNC() at netbsd:VFS_SYNC+0x35
 sched_sync() at netbsd:sched_sync+0x91


 [Locks tracked through CPUs]
 Locks held on CPU 0:
 Lock 0 (initialized at com_attach_subr)
 lock address : 0xfffffe810f2731e0 type     :               spin
 initialized  : 0xffffffff805f4964
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  0 last held:                  0
 current lwp  : 0xfffffe822f5eb440 last held: 0xfffffe822f5eb440
 last locked* : 0xffffffff805f3ec4 unlocked : 0xffffffff805f40a3
 owner field  : 0x0000000000010800 wait/spin:                0/1
 trace: pid 0 lid 6 at 0xffff8000ae69ad18
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 splraise() at netbsd:splraise
 _kernel_lock() at netbsd:_kernel_lock+0x129
 intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x16
 Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff8000ae69b0f0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:


 db{0}>

 db{0}> ps
 PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 28127    1 3   0        80   fffffe821b45c660               sshd select
 3765     1 3   0        80   fffffe821c860980               sshd select
 10510    1 3   0        80   fffffe820c1a95a0               sshd select
 12095    1 3   0        80   fffffe8201ea4880               sshd select
 24917    1 3   2        80   fffffe82143e10c0               sshd select
 14706    1 3   2        80   fffffe81f95ab000               sshd select
 24718    1 3   4        80   fffffe822d981420               tcsh ttyraw
 14768    1 3   6        80   fffffe8127c869c0             pickup kqueue
 17173    1 3   7        80   fffffe821b7d1300               tcsh ttyraw
 22683    1 3   0        80   fffffe821e368720               tcsh pause
 6496     1 3   2        80   fffffe81fdf25980              xterm select
 8713     1 3   4        80   fffffe8223f352e0               tcsh pause
 18715    1 3   1        80   fffffe8203885760               sshd select
 27311    1 3   0        80   fffffe821f2322c0               sshd select
 12100    1 3   1        80   fffffe82208de4a0               tcsh ttyraw
 19259    1 3   7        80   fffffe820b1e85c0              xterm select
 27176    1 3   1        80   fffffe81bf43b260               tcsh pause
 17081    1 3   0        80   fffffe82261ef260               sshd select
 20646    1 3   0        80   fffffe821e413100               sshd select
 12410    1 3   0        80   fffffe821c761640               tcsh ttyraw
 17135    1 3   7        80   fffffe821e413520              xterm select
 5138     1 3   1        80   fffffe810f791ba0               tcsh pause
 4590     1 3   5        80   fffffe82261efaa0               sshd select
 14049    1 3   7        80   fffffe81f72641a0               sshd select
 13260    1 3   2        80   fffffe8218a44280                ssh select
 12084    1 3   5        80   fffffe8126a67a60               tcsh pause
 20039    1 3   0        80   fffffe8214096100              xterm select
 12456    1 3   5        80   fffffe820b1e81a0               tcsh pause
 16421    1 3   3        80   fffffe81afe42600               sshd select
 2132     1 3   5        80   fffffe8200185120               sshd select
 28308    1 3   6        80   fffffe81fdf25140                ssh select
 153      1 3   6        80   fffffe81cfc24240                 sh wait
 29602    1 3   6        80   fffffe81f77cd640                ssh select
 26803    1 3   6        80   fffffe820c1a99c0                 sh wait
 25619    1 3   5        80   fffffe822d908120                ssh select
 24644    1 3   6        80   fffffe821acfb480                 sh wait
 28650    1 3   6        80   fffffe82208de8c0                ssh select
 28765    1 3   6        80   fffffe822d7f8140                 sh wait
 26866    1 3   0        80   fffffe8218a446a0                ssh select
 28190    1 3   6        80   fffffe821a21e600                 sh wait
 23270    1 3   0        80   fffffe820f1bc640                ssh select
 29000    1 3   0        80   fffffe820bb76b60                 sh wait
 556      1 3   0        80   fffffe822478cb00                ssh select
 27618    1 3   0        80   fffffe81ffea40e0                 sh wait
 25866    1 3   0        80   fffffe821c341260                ssh select
 29240    1 3   0        80   fffffe82222d0a00                 sh wait
 27689    1 3   0        80   fffffe82206a4340                ssh select
 2714     1 3   0        80   fffffe81f95db780                 sh wait
 1755     1 3   5        80   fffffe821d1b00e0                ssh select
 27303    1 3   6        80   fffffe821fb47060                 sh wait
 25810    1 3   6        80   fffffe821f3a70c0                ssh select
 28383    1 3   6        80   fffffe81f77cd220                 sh wait
 21350    1 3   6        80   fffffe8219cd4740                ssh select
 24176    1 3   6        80   fffffe8201ea4040                 sh wait
 25377    1 3   0        80   fffffe82251f3280                ssh select
 28753    1 3   6        80   fffffe8208905280                 sh wait
 28152    1 3   0        80   fffffe820ad5b080                ssh select
 22480    1 3   6        80   fffffe820bb76320                 sh wait
 397      1 3   6        80   fffffe8208905ac0                ssh select
 27414    1 3   6        80   fffffe82134910a0                 sh wait
 25934    1 3   1        80   fffffe82102b8500                ssh select
 1603     1 3   0        80   fffffe81ad7c26a0                 sh wait
 28109    1 3   0        80   fffffe822d981000                ssh select
 29687    1 3   0        80   fffffe8214096940                 sh wait
 22723    1 3   7        80   fffffe821630a060                ssh select
 936      1 3   7        80   fffffe81f9e538a0                 sh wait
 28299    1 3   7        80   fffffe822c2c2860                ssh select
 25799    1 3   7        80   fffffe821a3b8b20                 sh wait
 25713    1 3   6        80   fffffe81be3c16c0                ssh select
 28263    1 3   6        80   fffffe82207b6ba0                 sh wait
 28024    1 3   6        80   fffffe821a21e1e0                ssh select
 27324    1 3   6        80   fffffe81f95dbba0                 sh wait
 23843    1 3   6        80   fffffe821e368b40                ssh select
 25187    1 3   6        80   fffffe8220b67760                 sh wait
 26257    1 3   6        80   fffffe822d60d9c0                ssh select
 28946    1 3   6        80   fffffe822d887580                 sh wait
 29668    1 3   6        80   fffffe821e33b860                ssh select
 28070    1 3   6        80   fffffe821f2326e0                 sh wait
 8919     1 3   5        80   fffffe821a852020               tcsh ttyraw
 792      1 3   0        80   fffffe8218a44ac0               tcsh pause
 1018     1 3   0        80   fffffe82174cb680               tcsh pause
 1284     1 3   0        80   fffffe82174cbaa0                ssh select
 1017     1 3   5        80   fffffe821b45c240                 sh wait
 1143     1 3   4        80   fffffe821b45ca80              xterm select
 563      1 3   5        80   fffffe8219b27220               tcsh pause
 1076     1 3   1        80   fffffe8219b27640               sshd select
 626      1 3   5        80   fffffe8219b27a60               sshd select
 272      1 3   0        80   fffffe8218a621e0              xterm select
 1086     1 3   6        80   fffffe821af99b80               tcsh pause
 893      1 3   3        80   fffffe82184f89a0               sshd select
 639      1 3   0        80   fffffe821c0181a0               sshd select
 884      1 3   4        80   fffffe8218bb2a00                ssh select
 371      1 3   4        80   fffffe8218bb25e0                 sh wait
 878      1 3   1        80   fffffe821d22e280                ssh select
 877      1 3   1        80   fffffe821a2e9200                 sh wait
 1260     1 3   7        80   fffffe821a2e9620                ssh select
 747      1 3   2        80   fffffe821d7af100                 sh wait
 354      1 3   3        80   fffffe8218ff04c0                ssh select
 481      1 3   0        80   fffffe821a2e9a40                 sh wait
 1118     1 3   1        80   fffffe8218a62600                ssh select
 989      1 3   4        80   fffffe8218a62a20          conserver select
 988      1 3   1        80   fffffe8218bb21c0                 sh wait
 601      1 3   3        80   fffffe821c0185c0                ssh select
 984      1 3   0        80   fffffe82196e11a0                ssh select
 855      1 3   3        80   fffffe82196e15c0                 sh wait
 598      1 3   6        80   fffffe82196e19e0                 sh wait
 597      1 3   7        80   fffffe821af99340                ssh select
 1236     1 3   3        80   fffffe821a3a7180                ssh select
 467      1 3   5        80   fffffe821a3a75a0                 sh wait
 338      1 3   7        80   fffffe821a3a79c0                ssh select
 1233     1 3   1        80   fffffe82184f8160                 sh wait
 336      1 3   4        80   fffffe82184f8580                 sh wait
 1230     1 3   6        80   fffffe821be405e0                ssh select
 844      1 3   5        80   fffffe821c064140                 sh wait
 459      1 3   4        80   fffffe821c064560                ssh select
 458      1 3   2        80   fffffe821c064980                 sh wait
 457      1 3   3        80   fffffe8219be1120                ssh select
 456      1 3   7        80   fffffe8219be1540                ssh select
 1222     1 3   6        80   fffffe8219be1960                 sh wait
 1220     1 3   0        80   fffffe821d7af940                 sh wait
 321      1 3   0        80   fffffe821af99760                ssh select
 320      1 3   0        80   fffffe821d7af520                 sh wait
 957      1 3   4        80   fffffe821a0110e0          conserver select
 954      1 3   0        80   fffffe821c8b5320                ssh select
 953      1 3   0        80   fffffe821a011500                 sh wait
 952      1 3   5        80   fffffe821a011920                ssh select
 566      1 3   7        80   fffffe821a1424e0                ssh select
 949      1 3   5        80   fffffe821a142900                 sh wait
 1204     1 3   7        80   fffffe8218ff00a0                 sh wait
 306      1 3   4        80   fffffe8218ff08e0                ssh select
 1201     1 3   4        80   fffffe821ab0b080                 sh wait
 304      1 3   6        80   fffffe821ab0b4a0                ssh select
 943      1 3   1        80   fffffe821a9b8b00                ssh select
 1198     1 3   6        80   fffffe821ab0b8c0                 sh wait
 1197     1 3   1        80   fffffe821acfb060                 sh wait
 1182     1 3   4        80   fffffe821b9b9360          conserver select
 797      1 3   5        80   fffffe821b7d1b40                ssh select
 540      1 3   0        80   fffffe821b9b9780                 sh wait
 539      1 3   0        80   fffffe821b9b9ba0          conserver select
 918      1 3   0        80   fffffe821d22e6a0                ssh select
 917      1 3   0        80   fffffe821c341aa0                 sh wait
 1044     1 3   4        80   fffffe821c341680          conserver select
 1041     1 3   0        80   fffffe821c761220                ssh select
 400      1 3   0        80   fffffe821d107a80                 sh wait
 911      1 3   4        80   fffffe821c8b1200                ssh select
 1038     1 3   0        80   fffffe821c8b5740                 sh wait
 781      1 3   4        80   fffffe821c8b5b60          conserver select
 267      1 3   4        80   fffffe821b7d1720                ssh select
 264      1 3   4        80   fffffe821cc7e2e0                 sh wait
 265      1 3   6        80   fffffe821cc7e700                ssh select
 1415     1 3   6        80   fffffe821cc7eb20                 sh wait
 774      1 3   7        80   fffffe821a9b82c0                ssh select
 901      1 3   0        80   fffffe821a9b86e0                 sh wait
 386      1 3   2        80   fffffe821bd2a6c0                ssh select
 385      1 3   2        80   fffffe821bd2aae0                 sh wait
 630      1 3   4        80   fffffe821d107240                ssh select
 757      1 3   4        80   fffffe821d107660                 sh wait
 1009     1 3   6        80   fffffe821c761a60                ssh select
 559      1 3   6        80   fffffe821c8b1620                ssh select
 814      1 3   0        80   fffffe821c8b1a40                 sh wait
 813      1 3   6        80   fffffe821ce4b1e0                 sh wait
 876      1 3   7        80   fffffe821ce4b600                ssh select
 363      1 3   7        80   fffffe821ce4ba20                 sh wait
 362      1 3   2        80   fffffe821be401c0          conserver select
 296      1 3   0        80   fffffe821be40a00                ssh select
 613      1 3   1        80   fffffe821c0189e0                ssh select
 995      1 3   0        80   fffffe821d462180                 sh wait
 290      1 3   0        80   fffffe821d4625a0                ssh select
 161      1 3   1        80   fffffe821d4629c0                 sh wait
 160      1 3   4        80   fffffe82251f3ac0                 sh wait
 671      1 3   0        80   fffffe821e38c160                ssh select
 990      1 3   6        80   fffffe821e38c580                ssh select
 669      1 3   2        80   fffffe821e38c9a0                 sh wait
 604      1 3   6        80   fffffe821c860140                 sh wait
 603      1 3   0        80   fffffe821c860560                ssh select
 537      1 3   3        80   fffffe821dba9120                 sh wait
 137      1 3   3        80   fffffe821f3a74e0          conserver select
 136      1 3   0        80   fffffe82209a0320                ssh select
 1031     1 3   7        80   fffffe822cf3da00                 sh wait
 838      1 3   2        80   fffffe821f3a7900                ssh select
 517      1 3   4        80   fffffe821ed310a0                ssh select
 900      1 3   2        80   fffffe822d887160                 sh wait
 899      1 3   4        80   fffffe82285b7640                 sh wait
 194      1 3   6        80   fffffe8224659ae0                ssh select
 193      1 3   3        80   fffffe821ed314c0                 sh wait
 192      1 3   5        80   fffffe821ed318e0                ssh select
 473      1 3   0        80   fffffe821e330080                 sh wait
 73       1 3   1        80   fffffe822d908540                ssh select
 569      1 3   7        80   fffffe821e3304a0                 sh wait
 75       1 3   3        80   fffffe822d7f8980                ssh select
 232      1 3   6        80   fffffe821fb47480                 sh wait
 749      1 3   5        80   fffffe822dc08100          conserver select
 170      1 3   6        80   fffffe82285b7a60          conserver select
 171      1 3   0        80   fffffe822dc01920              getty ttyraw
 169      1 3   4        80   fffffe822d908960          conserver select
 646      1 3   7        80   fffffe82207b6780          conserver select
 766      1 3   0        80   fffffe82209a0740               cron nanoslp
 716      1 3   1        80   fffffe82234a5b40              inetd kqueue
 97       1 3   4        80   fffffe822cff7200              nginx kqueue
 96       1 3   4        80   fffffe8223f35700              nginx pause
 589      1 3   4        80   fffffe822478c6e0               qmgr kqueue
 756      1 3   6        80   fffffe82246592a0             master kqueue
 408      1 3   0        80   fffffe822d34d5c0               sshd select
 469      1 3   0        80   fffffe822705ea80           tac_plus select
 438      1 3   0        80   fffffe82285b7220             powerd kqueue
 421      1 3   3        80   fffffe822d60d5a0               ntpd pause
 380      4 3   4        80   fffffe822cfce600              dhcpd parked
 380      3 3   0        80   fffffe822d34d9e0              dhcpd kqueue
 380      2 3   7        80   fffffe822d7f8560              dhcpd parked
 380      1 3   4        80   fffffe822cff7620              dhcpd parked
 83       1 2   6         0   fffffe822cfce1e0            syslogd
 1        1 3   7        80   fffffe810f56a900               init wait
 0      124 5   0       200   fffffe81ffea4920           (zombie)
 0      123 3   5       200   fffffe822cf3d1c0            npfgc-0 xchicv
 0      122 3   7       200   fffffe822cfcea20              ipmi0 ipmi0
 0      121 3   3       200   fffffe822dc01500            physiod physiod
 0      120 3   0       200   fffffe822dc08520           aiodoned aiodoned
 0      119 3   2       200   fffffe822dc08940            ioflush biowait
 0      118 3   0       200   fffffe822dc010e0           pgdaemon pgdaemon
 0      115 3   0       280   fffffe810f56a0c0              spkr1 bellcv
 0      114 3   5       280   fffffe810f56a4e0           audiomix play
 0      113 3   5       280   fffffe810edc32a0           audiorec record
 0      112 3   0       200   fffffe810f33a300          atapibus0 sccomp
 0      110 3   1       200   fffffe810f3332e0               usb2 usbevt
 0      109 3   3       200   fffffe810f333b20               usb5 usbevt
 0      108 3   3       200   fffffe810f333700               usb0 usbevt
 0      107 3   1       200   fffffe810f33ab40               usb3 usbevt
 0      106 3   1       200   fffffe810f2352c0               usb1 usbevt
 0      105 3   3       200   fffffe810f311b60               usb4 usbevt
 0      104 3   0       200   fffffe810f99b0a0            rt_free rt_free
 0      103 3   0       200   fffffe810f99b4c0              unpgc unpgc
 0      102 3   0       200   fffffe810f99b8e0    key_timehandler key_timehandler

 0      101 3   7       200   fffffe810f484080    icmp6_wqinput/7 icmp6_wqinput
 0      100 3   6       200   fffffe810f4844a0    icmp6_wqinput/6 icmp6_wqinput
 0       99 3   5       200   fffffe810f4848c0    icmp6_wqinput/5 icmp6_wqinput
 0       98 3   4       200   fffffe810f495060    icmp6_wqinput/4 icmp6_wqinput
 0       97 3   3       200   fffffe810f495480    icmp6_wqinput/3 icmp6_wqinput
 0       96 3   2       200   fffffe810f4958a0    icmp6_wqinput/2 icmp6_wqinput
 0       95 3   1       200   fffffe810f3be040    icmp6_wqinput/1 icmp6_wqinput
 0       94 3   0       200   fffffe810f3be460    icmp6_wqinput/0 icmp6_wqinput
 0       93 3   0       200   fffffe810f3be880          nd6_timer nd6_timer
 0       92 3   7       200   fffffe810f387020     icmp_wqinput/7 icmp_wqinput
 0       91 3   6       200   fffffe810f387440     icmp_wqinput/6 icmp_wqinput
 0       90 3   5       200   fffffe810f387860     icmp_wqinput/5 icmp_wqinput
 0       89 3   4       200   fffffe810f3a8000     icmp_wqinput/4 icmp_wqinput
 0       88 3   3       200   fffffe810f3a8420     icmp_wqinput/3 icmp_wqinput
 0       87 3   2       200   fffffe810f3a8840     icmp_wqinput/2 icmp_wqinput
 0       86 3   1       200   fffffe810f36e360     icmp_wqinput/1 icmp_wqinput
 0       85 3   0       200   fffffe810f36e780     icmp_wqinput/0 icmp_wqinput
 0       84 3   0       200   fffffe810f36eba0           rt_timer rt_timer
 0       83 3   0       200   fffffe810f33a720        vmem_rehash vmem_rehash
 0       82 3   7       200   fffffe810f318340          coretemp3 coretemp3
 0       81 3   7       200   fffffe810f318760          coretemp2 coretemp2
 0       80 3   5       200   fffffe810f318b80          coretemp1 coretemp1
 0       79 3   1       200   fffffe810f311320          coretemp0 coretemp0
 0       78 3   1       200   fffffe810f311740               mfi0 mfi0
 0       69 3   5       200   fffffe810f2356e0            atabus1 atath
 0       68 3   5       200   fffffe810f235b00            atabus0 atath
 0       66 3   0       200   fffffe810edc36c0           scsibus0 sccomp
 0       65 3   3       200   fffffe810edc3ae0         usbtask-dr usbtsk
 0       64 3   0       200   fffffe810ed02280         usbtask-hc usbtsk
 0       63 3   1       200   fffffe810ed026a0               bnx3 bnx3
 0       62 3   7       200   fffffe810ed02ac0               bnx2 bnx2
 0       61 3   2       200   fffffe810eb79260               bnx1 bnx1
 0       60 3   7       200   fffffe810eb79680               bnx0 bnx0
 0       59 3   5       200   fffffe810eb79aa0               ipmi ipmipoll
 0       58 3   7       200   fffffe810eaca240            xcall/7 xcall
 0       57 1   7       200   fffffe810eaca660          softser/7
 0       56 1   7       200   fffffe810eacaa80          softclk/7
 0       55 1   7       200   fffffe810eac3220          softbio/7
 0       54 1   7       200   fffffe810eac3640          softnet/7
 0    >  53 7   7       201   fffffe810eac3a60             idle/7
 0       52 3   6       200   fffffe810ea72200            xcall/6 xcall
 0       51 1   6       200   fffffe810ea72620          softser/6
 0       50 1   6       200   fffffe810ea72a40          softclk/6
 0       49 1   6       200   fffffe810ea6b1e0          softbio/6
 0       48 1   6       200   fffffe810ea6b600          softnet/6
 0    >  47 7   6       201   fffffe810ea6ba20             idle/6
 0       46 3   5       200   fffffe810ea221c0            xcall/5 xcall
 0       45 1   5       200   fffffe810ea225e0          softser/5
 0       44 1   5       200   fffffe810ea22a00          softclk/5
 0       43 1   5       200   fffffe810ea0b1a0          softbio/5
 0       42 1   5       200   fffffe810ea0b5c0          softnet/5
 0    >  41 7   5       201   fffffe810ea0b9e0             idle/5
 0       40 3   4       200   fffffe810e9ca180            xcall/4 xcall
 0       39 1   4       200   fffffe810e9ca5a0          softser/4
 0       38 1   4       200   fffffe810e9ca9c0          softclk/4
 0       37 1   4       200   fffffe810e9b3160          softbio/4
 0       36 1   4       200   fffffe810e9b3580          softnet/4
 0    >  35 7   4       201   fffffe810e9b39a0             idle/4
 0       34 3   3       200   fffffe810e972140            xcall/3 xcall
 0       33 1   3       200   fffffe810e972560          softser/3
 0       32 1   3       200   fffffe810e972980          softclk/3
 0       31 1   3       200   fffffe810e95b120          softbio/3
 0       30 1   3       200   fffffe810e95b540          softnet/3
 0    >  29 7   3       201   fffffe810e95b960             idle/3
 0       28 3   2       200   fffffe810e922100            xcall/2 xcall
 0       27 1   2       200   fffffe810e922520          softser/2
 0       26 1   2       200   fffffe810e922940          softclk/2
 0       25 1   2       200   fffffe810e9030e0          softbio/2
 0       24 1   2       200   fffffe810e903500          softnet/2
 0    >  23 7   2       201   fffffe810e903920             idle/2
 0       22 3   1       200   fffffe810e8ca0c0            xcall/1 xcall
 0       21 1   1       200   fffffe810e8ca4e0          softser/1
 0       20 1   1       200   fffffe810e8ca900          softclk/1
 0       19 1   1       200   fffffe810e8a30a0          softbio/1
 0       18 1   1       200   fffffe810e8a34c0          softnet/1
 0    >  17 7   1       201   fffffe810e8a38e0             idle/1
 0       16 3   0       200   fffffe822dd92080           lnxsyswq lnxsyswq
 0       15 3   2       200   fffffe822dd924a0             sysmon smtaskq
 0       14 3   6       200   fffffe822dd928c0         pmfsuspend pmfsuspend
 0       13 3   5       200   fffffe822e1b1060           pmfevent pmfevent
 0       12 3   0       200   fffffe822e1b1480         sopendfree sopendfr
 0       11 3   7       200   fffffe822e1b18a0           nfssilly nfssilly
 0       10 3   0       200   fffffe822f5d0040            cachegc cachegc
 0        9 3   0       200   fffffe822f5d0460             vdrain vdrain
 0        8 3   0       200   fffffe822f5d0880          modunload mod_unld
 0        7 3   0       200   fffffe822f5eb020            xcall/0 xcall
 0    >   6 7   0       200   fffffe822f5eb440          softser/0
 0    >   5 7   0       200   fffffe822f5eb860          softclk/0
 0        4 1   0       200   fffffe822f607000          softbio/0
 0    >   3 7   0       200   fffffe822f607420          softnet/0
 0    >   2 7   0       201   fffffe822f607840             idle/0
 0        1 3   0       200   ffffffff81481b80            swapper uvm
 db{0}>

 db{0}> trace/a fffffe822f607420
 trace: pid 0 lid 3 at 0xffff8000ae6960e0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}> trace/a fffffe822f5eb860
 trace: pid 0 lid 5 at 0xffff8000ae63df50
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 ?() at 0
 db{0}> trace/a fffffe822f5eb440
 trace: pid 0 lid 6 at 0xffff8000ae69ad18
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 splraise() at netbsd:splraise
 _kernel_lock() at netbsd:_kernel_lock+0x129
 intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x16
 Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff8000ae69b0f0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}>

 db{0}> show all callout
 hardclock_ticks now: 86821384
     ticks  wheel               arg  func
      -863 -1/-256                0  pffasttimo
      -861 -1/-256                0  pfslowtimo
      -861 -1/-256                0  tcp_slowtimo
      -851 -1/-256                0  sched_balance
      -849 -1/-256 fffffe812d614ad0  tcp_delack
      -848 -1/-256 fffffe810f382e10  uhci_poll_hub
      -833 -1/-256                0  key_timehandler
      -828 -1/-256 ffffffff81481b80  sleepq_timeout
      -822 -1/-256                0  nd6_timer
      -819 -1/-256                0  rt_timer_timer
      -811 -1/-256 ffff80000e836000  bnx_tick
      -811 -1/-256 ffff80000e830008  if_slowtimo
      -811 -1/-256 ffff80000e833008  if_slowtimo
      -811 -1/-256 ffff80000e836008  if_slowtimo
      -811 -1/-256 ffff80000e839008  if_slowtimo
      -811 -1/-256 ffff80000e830000  bnx_tick
      -798 -1/-256 ffff80000e839000  bnx_tick
      -784 -1/-256 fffffe822f5d0040  sleepq_timeout
      -561 -1/-256                0  vmem_rehash_all_kick
      -430 -1/-256 fffffe81f8e8f5d8  tcp_timer_rexmt
        13  0/107  fffffe810f372e10  uhci_poll_hub
        14  0/129  fffffe810f330e10  uhci_poll_hub
        43  0/163  fffffe810eb79aa0  sleepq_timeout
        13  0/182  fffffe810f351e10  uhci_poll_hub
        50  0/220  fffffe822c048e70  realtimerexpire
     30475  1/321  fffffe822c048d68  realtimerexpire
     36544  1/344  fffffe8222de5218  realtimerexpire
     58821  1/431  fffffe8222de5950  realtimerexpire
       511  1/460  fffffe822705ea80  sleepq_timeout
       606  1/460  fffffe821f08ba00  tcp_timer_rexmt
       608  1/460  fffffe812d614790  tcp_timer_rexmt
       969  1/462  fffffe822478c6e0  sleepq_timeout
       975  1/462  fffffe82246592a0  sleepq_timeout
      1713  1/464  fffffe810ed76188  sme_events_check
      1713  1/464  fffffe810f24e708  sme_events_check
      1713  1/464  fffffe810f2afec8  sme_events_check
      1713  1/464  fffffe810f2af748  sme_events_check
      1713  1/464  fffffe810f2f2dc8  sme_events_check
      1773  1/465  fffffe810f5711c8  sme_events_check
      1821  1/465  fffffe821f3a74e0  sleepq_timeout
      1943  1/466  fffffe821be401c0  sleepq_timeout
      1976  1/466  fffffe821c341680  sleepq_timeout
      2134  1/466  fffffe822cfce1e0  sleepq_timeout
      2407  1/467  fffffe82209a0740  sleepq_timeout
      2891  1/469  fffffe82285b7a60  sleepq_timeout
      3302  1/470  fffffe821b565120  tcp_timer_rexmt
      3075  1/470  fffffe8218a62a20  sleepq_timeout
      2982  1/470  fffffe821b9b9360  sleepq_timeout
      3251  1/471  fffffe822d908960  sleepq_timeout
      3409  1/471  fffffe822dc08100  sleepq_timeout
      3553  1/472  fffffe821c8b5b60  sleepq_timeout
      3685  1/472  fffffe821a0110e0  sleepq_timeout
      4352  1/475  fffffe821b9b9ba0  sleepq_timeout
      4269  1/475  fffffe822c048c60  realtimerexpire
      4641  1/476  fffffe810f2352c0  sleepq_timeout
      4652  1/476  fffffe810f333700  sleepq_timeout
      4685  1/476  fffffe810f333b20  sleepq_timeout
      4906  1/477  fffffe821b466498  tcp_timer_rexmt
      4852  1/477  fffffe810f3332e0  sleepq_timeout
      4974  1/477  fffffe810f33ab40  sleepq_timeout
      4718  1/477  fffffe810f311b60  sleepq_timeout
      5146  1/478  fffffe81c2dc3b10  tcp_timer_rexmt
      5434  1/479  fffffe8222de5008  realtimerexpire
      6033  1/482  fffffe82068d5b60  tcp_timer_rexmt
      6368  1/483  fffffe821f08b380  tcp_timer_rexmt
      7175  1/486  fffffe8127c869c0  sleepq_timeout
     34620  2/557  fffffe821a7580d0  tcp_timer_keep
     34639  2/557  fffffe8210a5d1f0  tcp_timer_keep
     34665  2/557  fffffe81c2dc37d0  tcp_timer_keep
     34686  2/557  fffffe82068d54e0  tcp_timer_keep
     18442  2/557  fffffe81de573330  arptimer
     30184  2/557  fffffe822cdcc990  arptimer
     73369  2/557  fffffe81de573220  arptimer
     34607  2/557  fffffe81d8010030  tcp_timer_keep
     34626  2/557  fffffe821b96a778  tcp_timer_keep
     39854  2/557  fffffe821ada5480  tcp_timer_keep
     41498  2/557  fffffe821cb3a3c8  tcp_timer_keep
     63262  2/557  fffffe82173407f0  tcp_timer_keep
     83173  2/558  fffffe81dbf21228  arptimer
     99760  2/558  fffffe822cdcc110  arptimer
    102304  2/558  fffffe822cdcc880  arptimer
    110475  2/558  fffffe822cdccee0  arptimer
    113194  2/558  fffffe822cdcc550  arptimer
    119123  2/558  fffffe822cdcc440  arptimer
     83485  2/558  fffffe822cfce600  sleepq_timeout
    170902  2/559  fffffe821ed4a6c8  tcp_timer_keep
    178082  2/559  fffffe8210a5d530  tcp_timer_keep
    178095  2/559  fffffe821b4667d8  tcp_timer_keep
    178100  2/559  fffffe820cd5cb98  tcp_timer_keep
    178167  2/559  fffffe821f039038  tcp_timer_keep
    180463  2/559  fffffe821b937760  tcp_timer_keep
    180673  2/559  fffffe8217340b30  tcp_timer_keep
    178141  2/559  fffffe81f4ad9208  tcp_timer_keep
    178182  2/559  fffffe82068d51a0  tcp_timer_keep
    178181  2/559  fffffe821f08b040  tcp_timer_keep
    183651  2/559  fffffe820cd5c1d8  tcp_timer_keep
    185007  2/559  fffffe82173404b0  tcp_timer_keep
    186205  2/559  fffffe81f4ad9888  tcp_timer_keep
    186244  2/559  fffffe81f4ad9bc8  tcp_timer_keep
    180493  2/559  fffffe81f8e8f918  tcp_timer_keep
    180504  2/559  fffffe82173384a8  tcp_timer_keep
    180556  2/559  fffffe81f8e8fc58  tcp_timer_keep
    186851  2/559  fffffe8210a5dbb0  tcp_timer_keep
    188512  2/559  fffffe821c046428  tcp_timer_keep
    188800  2/559  fffffe8219e307a8  tcp_timer_keep
    189621  2/559  fffffe821b565460  tcp_timer_keep
    189932  2/559  fffffe821ada5b00  tcp_timer_keep
    190283  2/559  fffffe81f30d3578  tcp_timer_keep
    190659  2/559  fffffe821ae20430  tcp_timer_keep
    266181  2/560  fffffe821ed4aa08  tcp_timer_keep
    266494  2/560  fffffe821ed4a048  tcp_timer_keep
    267094  2/560  fffffe821b4d2a68  tcp_timer_keep
    267288  2/560  fffffe821ae20770  tcp_timer_keep
    267310  2/560  fffffe821ae200f0  tcp_timer_keep
    267318  2/560  fffffe821b96a438  tcp_timer_keep
    267334  2/560  fffffe821ab59ac0  tcp_timer_keep
    267334  2/560  fffffe821ab59100  tcp_timer_keep
    267406  2/560  fffffe8210a5d870  tcp_timer_keep
    275093  2/560  fffffe8219e30468  tcp_timer_keep
    267330  2/560  fffffe821cc7d798  tcp_timer_keep
    267349  2/560  fffffe8219e30ae8  tcp_timer_keep
    267064  2/560  fffffe821a758750  tcp_timer_keep
    267068  2/560  fffffe821ba920a0  tcp_timer_keep
    267070  2/560  fffffe821c0c2740  tcp_timer_keep
    267070  2/560  fffffe821c0c2400  tcp_timer_keep
    267068  2/560  fffffe821b9c20c8  tcp_timer_keep
    267071  2/560  fffffe821b9c2748  tcp_timer_keep
    267071  2/560  fffffe821c4f9418  tcp_timer_keep
    267072  2/560  fffffe821c4f9a98  tcp_timer_keep
    267124  2/560  fffffe821b937420  tcp_timer_keep
    267263  2/560  fffffe821c0460e8  tcp_timer_keep
    268769  2/560  fffffe821ebc26d0  tcp_timer_keep
    266964  2/560  fffffe821b9c2408  tcp_timer_keep
    267662  2/560  fffffe81f30d3bf8  tcp_timer_keep
    268000  2/560  fffffe81f8e8f298  tcp_timer_keep
    268341  2/560  fffffe8219a92b20  tcp_timer_keep
    266820  2/560  fffffe821caaa090  tcp_timer_keep
    267068  2/560  fffffe821b339a78  tcp_timer_keep
    267068  2/560  fffffe821ba92a60  tcp_timer_keep
    267068  2/560  fffffe821b3393f8  tcp_timer_keep
    267070  2/560  fffffe821ba923e0  tcp_timer_keep
    277109  2/561  fffffe8219e30128  tcp_timer_keep
    278611  2/561  fffffe820cd5c858  tcp_timer_keep
    278987  2/561  fffffe821b2023f0  tcp_timer_keep
    280166  2/561  fffffe821b339738  tcp_timer_keep
    280644  2/561  fffffe821b9c2a88  tcp_timer_keep
    281529  2/561  fffffe821cb3a088  tcp_timer_keep
    281820  2/561  fffffe821b0c8130  tcp_timer_keep
    284392  2/561  fffffe821b0c8af0  tcp_timer_keep
    298629  2/561                 0  nd6_slowtimo
    278048  2/561  fffffe821b565ae0  tcp_timer_keep
    278713  2/561  fffffe821ebc2390  tcp_timer_keep
    281860  2/561  fffffe821cb3a708  tcp_timer_keep
    282331  2/561  fffffe821ab59440  tcp_timer_keep
    283391  2/561  fffffe821cc7dad8  tcp_timer_keep
    278133  2/561  fffffe821b4d2728  tcp_timer_keep
    279419  2/561  fffffe821e59a058  tcp_timer_keep
    279801  2/561  fffffe821aa0d148  tcp_timer_keep
    281290  2/561  fffffe821cce2080  tcp_timer_keep
    276329  2/561  fffffe821ada57c0  tcp_timer_keep
    277660  2/561  fffffe821ebc2a10  tcp_timer_keep
    279103  2/561  fffffe8217920150  tcp_timer_keep
    279484  2/561  fffffe821b466158  tcp_timer_keep
    308870  2/561  fffffe821c0c20c0  tcp_timer_keep
    277014  2/561  fffffe821cc7d458  tcp_timer_keep
    278388  2/561  fffffe821ada5140  tcp_timer_keep
    279047  2/561  fffffe821e59a398  tcp_timer_keep
    279340  2/561  fffffe821b202730  tcp_timer_keep
    279751  2/561  fffffe821c192098  tcp_timer_keep
    282696  2/561  fffffe821b466b18  tcp_timer_keep
    276424  2/561  fffffe821cce23c0  tcp_timer_keep
    277575  2/561  fffffe821b5657a0  tcp_timer_keep
    278335  2/561  fffffe821e59aa18  tcp_timer_keep
    281000  2/561  fffffe821b96aab8  tcp_timer_keep
    281394  2/561  fffffe821b0c87b0  tcp_timer_keep
    281520  2/561  fffffe8217920490  tcp_timer_keep
    276779  2/561  fffffe821c1923d8  tcp_timer_keep
    277337  2/561  fffffe821c192718  tcp_timer_keep
    277983  2/561  fffffe821e59a6d8  tcp_timer_keep
    278977  2/561  fffffe821b202a70  tcp_timer_keep
    279866  2/561  fffffe81f4ad9548  tcp_timer_keep
    280139  2/561  fffffe821cce2700  tcp_timer_keep
    280193  2/561  fffffe821aa0d488  tcp_timer_keep
    280586  2/561  fffffe81f30d38b8  tcp_timer_keep
    280886  2/561  fffffe821caaaa50  tcp_timer_keep
    280948  2/561  fffffe821c0c2a80  tcp_timer_keep
    276057  2/561  fffffe821c192a58  tcp_timer_keep
    278567  2/561  fffffe821b4d20a8  tcp_timer_keep
    279767  2/561  fffffe821e51aa20  tcp_timer_keep
    280560  2/561  fffffe821cc7d118  tcp_timer_keep
    281960  2/561  fffffe82179207d0  tcp_timer_keep
    282179  2/561  fffffe8219a927e0  tcp_timer_keep
    282625  2/561  fffffe821cb3aa48  tcp_timer_keep
    283796  2/561  fffffe821aa0d7c8  tcp_timer_keep
    352289  2/562  fffffe821c046768  tcp_timer_keep
    345110  2/562  fffffe81f30d3238  tcp_timer_keep
    717815  2/563  fffffe8222cba9d0  tcp_timer_keep
    717812  2/563  fffffe8222cba690  tcp_timer_keep
    714678  2/563  fffffe8222cba350  tcp_timer_keep
    717813  2/563  fffffe8222cba010  tcp_timer_keep
    717814  2/563  fffffe821e1189d8  tcp_timer_keep
    714584  2/563  fffffe821e118358  tcp_timer_keep
    714678  2/563  fffffe821e118018  tcp_timer_keep
    714678  2/563  fffffe821f87c9e0  tcp_timer_keep
    717813  2/563  fffffe821f87c6a0  tcp_timer_keep
    719074  2/563  fffffe821f87c360  tcp_timer_keep
    714966  2/563  fffffe821f87c020  tcp_timer_keep
    714678  2/563  fffffe821f8549e8  tcp_timer_keep
    714967  2/563  fffffe821f8546a8  tcp_timer_keep
    714678  2/563  fffffe821f854368  tcp_timer_keep
    714678  2/563  fffffe821f854028  tcp_timer_keep
    714967  2/563  fffffe821fc8e9f0  tcp_timer_keep
    714678  2/563  fffffe821fc8e6b0  tcp_timer_keep
    714147  2/563  fffffe821fc8e370  tcp_timer_keep
    717812  2/563  fffffe821f0399f8  tcp_timer_keep
    717815  2/563  fffffe821f039378  tcp_timer_keep
    714678  2/563  fffffe821ebc2050  tcp_timer_keep
    717812  2/563  fffffe821e51a3a0  tcp_timer_keep
    714966  2/563  fffffe821e51a060  tcp_timer_keep
    714966  2/563  fffffe821d280a28  tcp_timer_keep
    714147  2/563  fffffe821d2806e8  tcp_timer_keep
    714678  2/563  fffffe821d280068  tcp_timer_keep
    714678  2/563  fffffe821cce2a40  tcp_timer_keep
    717791  2/563  fffffe821f0396b8  tcp_timer_keep
    714654  2/563  fffffe821ed4a388  tcp_timer_keep
    717789  2/563  fffffe821e51a6e0  tcp_timer_keep
    714943  2/563  fffffe821caaa710  tcp_timer_keep
    717791  2/563  fffffe82255b6008  tcp_timer_keep
    474660  2/564  fffffe821c046aa8  tcp_timer_keep
    719129  2/565  fffffe812d614790  tcp_timer_keep
    716178  2/565  fffffe821b0c8470  tcp_timer_keep
    575467  2/565  fffffe81ba3c3918  tcp_timer_keep
    716183  2/565  fffffe821c4f9758  tcp_timer_keep
    716203  2/565  fffffe821aa0db08  tcp_timer_keep
    716203  2/565  fffffe82173387e8  tcp_timer_keep
    716180  2/565  fffffe8219a924a0  tcp_timer_keep
    719131  2/565  fffffe821f08ba00  tcp_timer_keep
    575487  2/565  fffffe81966cb070  tcp_timer_keep
    582001  2/565  fffffe821ba92720  tcp_timer_keep
    719107  2/565  fffffe812d614ad0  tcp_timer_keep
    716173  2/565  fffffe821b937aa0  tcp_timer_keep
    575462  2/565  fffffe81ba3c3c58  tcp_timer_keep
    597175  2/565  fffffe8222de5e78  realtimerexpire
    620866  2/566  fffffe821b466498  tcp_timer_keep
    657207  2/566  fffffe82068d5b60  tcp_timer_keep
    666161  2/566  fffffe820cd5c518  tcp_timer_keep
    697102  2/567  fffffe821b565120  tcp_timer_keep
    718814  2/567  fffffe81f8e8f5d8  tcp_timer_keep
    680446  2/567  fffffe81c2dc3b10  tcp_timer_keep
    710518  2/567  fffffe821f08b380  tcp_timer_keep
    696427  2/567  fffffe81966cb6f0  tcp_timer_keep
    938912  2/571  fffffe81de573770  nd6_llinfo_timer
   3549262  2/610  fffffe81de573dd0  nd6_llinfo_timer
   8163629  2/681                 0  in6_tmpaddrtimer
   8269307  2/682  fffffe81de573000  nd6_llinfo_timer
   8271207  2/682  fffffe81de573aa0  nd6_llinfo_timer
   8271207  2/682  fffffe81de573110  nd6_llinfo_timer
 db{0}>

 db{0}> trace/a fffffe822f607840
 trace: pid 0 lid 2 at 0xffff8000ae63df90
 acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
 acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
 idle_loop() at netbsd:idle_loop+0x1b5
 db{0}>
 db{0}> trace/a fffffe810e8a38e0
 trace: pid 0 lid 17 at 0xffff8000ae4f1f30
 acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0xdb
 acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0xb6
 idle_loop() at netbsd:idle_loop+0x1b5
 cpu_hatch() at netbsd:cpu_hatch+0x17a
 md_root_setconf() at netbsd:md_root_setconf
 db{0}>

From: "J. Hannken-Illjes" <hannken@eis.cs.tu-bs.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network
 interfaces in use
Date: Fri, 1 Jun 2018 12:40:37 +0200

 > db{0}>
 > 
 > db{0}> ps
 > PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 <snip>
 > 0    >   6 7   0       200   fffffe822f5eb440          softser/0
 > 0    >   5 7   0       200   fffffe822f5eb860          softclk/0
 > 0        4 1   0       200   fffffe822f607000          softbio/0
 > 0    >   3 7   0       200   fffffe822f607420          softnet/0
 > 0    >   2 7   0       201   fffffe822f607840             idle/0
 > 0        1 3   0       200   ffffffff81481b80            swapper uvm
 <snip>
 > db{0}> show all callout
 > hardclock_ticks now: 86821384
 >     ticks  wheel               arg  func
 >      -863 -1/-256                0  pffasttimo
 >      -861 -1/-256                0  pfslowtimo
 >      -861 -1/-256                0  tcp_slowtimo
 >      -851 -1/-256                0  sched_balance
 >      -849 -1/-256 fffffe812d614ad0  tcp_delack
 >      -848 -1/-256 fffffe810f382e10  uhci_poll_hub
 >      -833 -1/-256                0  key_timehandler
 >      -828 -1/-256 ffffffff81481b80  sleepq_timeout
 >      -822 -1/-256                0  nd6_timer
 >      -819 -1/-256                0  rt_timer_timer
 >      -811 -1/-256 ffff80000e836000  bnx_tick
 >      -811 -1/-256 ffff80000e830008  if_slowtimo
 >      -811 -1/-256 ffff80000e833008  if_slowtimo
 >      -811 -1/-256 ffff80000e836008  if_slowtimo
 >      -811 -1/-256 ffff80000e839008  if_slowtimo
 >      -811 -1/-256 ffff80000e830000  bnx_tick
 >      -798 -1/-256 ffff80000e839000  bnx_tick
 >      -784 -1/-256 fffffe822f5d0040  sleepq_timeout
 >      -561 -1/-256                0  vmem_rehash_all_kick
 >      -430 -1/-256 fffffe81f8e8f5d8  tcp_timer_rexmt
 >        13  0/107  fffffe810f372e10  uhci_poll_hub
 >        14  0/129  fffffe810f330e10  uhci_poll_hub
 >        43  0/163  fffffe810eb79aa0  sleepq_timeout
 >        13  0/182  fffffe810f351e10  uhci_poll_hub

 Suppose negative ticks means callout processing has stopped.

 Examined all softhandler on cpu0 and got:

 index             func  flags   func        ACTIVE,PENDING      LEVEL
 108 0xffffffff809c5cf0 0x3101   callout_softclock       AP      CLOCK
 113 0xffffffff809ee1ce 0x2102   xc__highpri_intr        A       SERIAL
 148 0xffffffff80a25ec8 0x1100   biointr                  P      BIO
 268 0xffffffff805f39e9 0x1002   comsoft                  P      SERIAL
 303 0xffffffff80705118 0x3303   ip6intr                 AP      NET

 If I get it right we have callout_softclock and ip6intr busy and with
 more work and we have xc__highpri_intr busy (there is a
 pserialize_perform running).

 Is it right the running ip6intr will block all other handlers as NET
 is the highest priority here?

 Any idea how to debug the current state of "softnet/0"

 --
 J. Hannken-Illjes - hannken@eis.cs.tu-bs.de - TU Braunschweig (Germany)

From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: port-amd64-maintainer@netbsd.org
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Sun, 17 Jun 2018 13:18:11 +0200 (CEST)

 Hi,

 two additional observations on this case:

 1) When provoking the wedge, it is sufficient to only "up" the
    third interface on the box.  It's not required to configure
    any IP addresses on the third interface to cause the wedge.

 2) We have also provoked the wedge by using a quad wm(4) card,
    and trying to use three interfaces on that card will also
    provoke the wedge, so whatever is causing this issue is
    unlikely to be a device driver bug.

 Best regards,

 - H=E5vard

>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.43 2018/01/16 07:36:43 maya Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2017 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.