NetBSD Problem Report #57580
From kardel@Kardel.name Sun Aug 13 11:33:22 2023
Return-Path: <kardel@Kardel.name>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id E450D1A9238
for <gnats-bugs@gnats.NetBSD.org>; Sun, 13 Aug 2023 11:33:21 +0000 (UTC)
Message-Id: <20230813101439.82AA344B33@Andromeda.Kardel.name>
Date: Sun, 13 Aug 2023 12:14:39 +0200 (CEST)
From: kardel@netbsd.org
Reply-To: kardel@netbsd.org
To: gnats-bugs@NetBSD.org
Subject: evbarm/earmv7hf RPI2 scheduler/cpu stall
X-Send-Pr-Version: 3.95
>Number: 57580
>Category: kern
>Synopsis: evbarm/earmv7hf RPI2 scheduler/cpu stall
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: kern-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sun Aug 13 11:35:00 +0000 2023
>Last-Modified: Fri Aug 25 09:30:03 +0000 2023
>Originator: Frank Kardel
>Release: NetBSD 10.0_BETA
>Organization:
>Environment:
System: NetBSD rpi 10.0_BETA NetBSD 10.0_BETA (GENERIC) #3: Wed Aug 2 22:32:24 CEST 2023 kardel@:/src/NetBSD/n10/src/obj.evbarm/sys/arch/evbarm/compile/GENERIC evbarm
Architecture: earmv7hf
Machine: evbarm
>Description:
After a while (~ 4 days) a router+webserver system on RPI2 ceases to
operate properly.
Many processes are in a RUN/1 (to be scheduled on CPU 1. CPU 1 does
not pick up processes. crash lists one process as active ('>') and
the stack of that is sort of empty (0). but no progress is made.
In this state "cpuctl offline 1" gets stuck waiting on "xclocv"
wait channel.
So why is CPU 1 not running processes anymore?
Collected information:
ps alxww:
UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
0 0 0 0 125 0 0 7600 - RKl ? 120:57.85 [system]
0 1 0 0 85 0 5952 1056 wait Ss ? 0:00.06 init
0 113 14075 25600 79 0 7284 2008 select Ss ? 0:00.28 SCREEN -R (screen)
0 169 1 0 85 0 5356 52 nanoslp Ss ? 0:02.63 /sbin/wdogctl -u bcmpmwdog0
0 304 1 34510 85 0 25644 1256 kqueue Isl ? 0:00.08 /usr/pkg/sbin/fscd -c /usr/pkg/etc/fscd.conf
1001 329 1 10860 83 0 10748 988 poll Is ? 0:00.05 /usr/pkg/sbin/staticd -d
0 385 1 7489 40 0 69152 4388 parked Ssl ? 1:44.83 /usr/sbin/dhcpd -q
0 414 1 0 85 0 10120 9176 - Rs ? 0:42.43 /usr/sbin/ntpd -p /var/run/ntpd.pid -g
0 421 1 9366 83 0 6100 880 devmon Is ? 0:00.03 /sbin/devpubd
0 643 1 26133 79 0 5924 972 kqueue Is ? 0:00.01 /usr/sbin/inetd -l
1002 657 1 0 85 0 163600 2344 - Rs ? 0:01.63 /usr/pkg/bin/postgres -D /usr/pkg/pgsql/data
1002 805 657 0 85 0 163744 1884 poll Ss ? 0:05.26 postgres: background writer
32767 828 1 0 85 0 6336 1016 - Rs ? 0:08.82 /usr/sbin/mdnsd
0 853 1 0 85 0 18224 2404 poll Ssl ? 0:17.29 /usr/pkg/sbin/bareos-fd -g bareos
0 885 1 0 85 0 14104 1836 kqueue Ss ? 0:04.43 /usr/libexec/postfix/master -w
12 888 885 0 85 0 13796 2400 kqueue S ? 0:01.13 qmgr -l -t unix -u
0 952 1 12409 85 0 130676 3128 - Rsl ? 8:00.60 /usr/sbin/named
1004 1019 1172 584527 43 0 130232 2400 - Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
0 1134 1 30290 78 0 9744 1768 kqueue Ss ? 5:01.79 /usr/sbin/syslogd -s
0 1172 1 0 85 0 10616 1376 select Ss ? 0:39.18 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 79 0 128104 2496 pipe_rd Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
0 1435 1 0 85 0 15112 1792 poll Ss ? 0:00.04 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd)
0 1460 1 0 85 0 5560 632 - Rs ? 1:27.24 /usr/pkg/sbin/gpiomon
1001 1518 1 1024 85 0 11172 1780 poll Ss ? 0:29.16 /usr/pkg/sbin/ripd -P 0 -d
1004 1524 1172 344575 85 0 128020 1112 - Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
0 1677 1 8 85 0 6088 1048 - Rs ? 0:04.30 /usr/sbin/cron
1005 1716 2860 0 85 0 77468 2876 - R ? 5:56.62 /usr/bin/perl /usr/pkg/fhem/fhem.pl /usr/pkg/fhem/fhem.cfg
1005 1717 2860 0 85 0 77640 2900 - R ? 9:44.65 /usr/bin/perl /usr/pkg/fhem/fhem.pl /usr/pkg/fhem/fhem.cfg
1002 1828 657 0 85 0 163908 2248 poll Ss ? 0:00.31 postgres: checkpointer
0 1853 1 36619 85 0 16244 3556 select Ss ? 0:55.32 /usr/pkg/bin/perl -wT /usr/pkg/sbin/munin-node
1005 1940 1 0 85 0 51076 1520 nanoslp S ? 3:44.99 /usr/bin/perl /usr/pkg/fhem/fhem.pl /usr/pkg/fhem/fhem.cfg
1001 2088 1 35690 85 0 44436 3280 poll Ssl ? 0:00.89 /usr/pkg/sbin/zebra -P 0 -d
1005 2197 2860 1035 85 0 70020 3268 nanoslp S ? 87:12.66 /usr/bin/perl /usr/pkg/fhem/fhem.pl /usr/pkg/fhem/fhem.cfg
1002 2728 657 5120 84 0 22772 2228 poll Ss ? 0:12.10 postgres: stats collector
1002 2854 657 0 85 0 163824 1824 poll Ss ? 0:05.15 postgres: walwriter
1002 2983 657 1024 85 0 164056 3712 poll Ss ? 0:09.56 postgres: autovacuum launcher
1002 2987 657 0 85 0 163908 2876 poll Ss ? 0:00.56 postgres: logical replication launcher
1005 2998 2860 9216 83 0 77872 20856 netio S ? 8:09.65 /usr/bin/perl /usr/pkg/fhem/fhem.pl /usr/pkg/fhem/fhem.cfg
0 3037 10374 15808 82 0 6468 1068 wait I ? 0:00.08 /bin/sh /etc/daily
1004 3107 1172 36864 76 0 10288 1540 socket S ? 0:15.69 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 79 0 128104 2480 pipe_rd Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1000 3719 1 580 85 0 9416 912 poll Is ? 0:00.01 ssh-agent
1000 3903 5069 99 85 0 16276 2196 - R ? 1:56.00 sshd: kardel@pts/2 (sshd)
1000 4148 1 2822 85 0 8732 912 poll Is ? 0:00.01 ssh-agent
1008 4666 1 96796 43 0 190376 38892 - Rsl ? 3:06.96 /usr/pkg/bin/node /usr/pkg/zigbee2mqtt/index.js
0 4939 1 0 0 0 16008 5880 - Rs ? 0:00.65 (perl)
0 5069 1435 1065 85 0 16012 2760 poll Is ? 0:00.58 sshd: kardel [priv]
0 5611 10374 0 85 0 13588 1828 pipe_rd I ? 0:00.07 sendmail -t
0 5664 4954 0 85 0 7168 1352 select Ss ? 1:23.66 SCREEN -R (screen)
0 10374 16429 331 85 0 6020 1068 wait Is ? 0:00.02 /bin/sh -c /bin/sh /etc/daily 2>&1 | tee /var/log/daily.out | sendmail -t
0 10885 3037 1179 85 0 6624 1200 - R ? 0:00.27 sh /etc/security
0 11561 1 0 0 0 16008 5880 - Rs ? 0:00.65 (perl)
0 15499 5611 654 85 0 13564 1828 netio I ? 0:00.10 postdrop -r
12 16001 885 0 85 0 13656 1880 - R ? 0:00.09 pickup -l -t unix -u
0 16429 1677 331 85 0 12520 1704 pipe_rd I ? 0:00.05 cron: running job
0 16846 1 0 0 0 16008 5880 - Rs ? 0:00.65 (perl)
0 17255 1435 13312 82 0 16008 5880 poll Ss ? 0:00.65 sshd: kardel [priv]
0 17312 10374 0 85 0 5464 892 pipe_rd I ? 0:00.01 tee /var/log/daily.out
0 19005 10885 0 0 0 0 0 - Z ? 0:00.00 (pkg_admin)
1000 22307 17255 19456 81 0 16148 4800 poll S ? 0:00.46 sshd: kardel@pts/0 (sshd)
1000 26845 1 20480 80 0 8732 1496 poll Ss ? 0:00.01 ssh-agent
0 28610 1 36864 76 0 14368 3132 select Ss ? 6:04.20 /usr/sbin/racoon
1000 11906 22307 20480 81 0 6748 1976 pause Ss pts/0 0:00.13 -csh
1000 14075 11906 20480 81 0 6928 2032 pause S+ pts/0 0:00.03 screen -R
1000 17710 113 36864 80 0 7412 2652 pause Ss pts/1 0:00.23 -usr/pkg/bin/tcsh
1000 24950 17710 36864 25 0 6300 1620 - O+ pts/1 0:00.02 ps -alxww
1000 4928 3903 580 85 0 6748 1216 pause Is pts/2 0:00.11 -csh
1000 4954 4928 0 85 0 6928 1244 - R+ pts/2 0:03.05 screen -R
0 1915 8499 0 85 0 6120 1340 - R+ pts/3 3:19.01 ping gateway
1000 4951 5664 0 85 0 7412 1332 pause Is pts/3 0:00.14 -usr/pkg/bin/tcsh
0 8499 4951 9 85 0 7384 1272 pause I pts/3 0:00.60 -csh (tcsh)
0 1939 1 30536 78 0 5744 1128 ttyraw Is+ ttyE0 0:00.02 /usr/libexec/getty default constty
1005 2860 1 2613 85 0 85084 3272 - R ttyE0- 171:49.22 /usr/bin/perl /usr/pkg/fhem/fhem.pl /usr/pkg/fhem/fhem.cfg
ps alsww
UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
0 0 0 0 96 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 96 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 96 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 96 0 0 7600 hytcv RKl ? 120:57.98 [system]
0 0 0 0 96 0 0 7600 hytcv RKl ? 120:57.98 [system]
0 0 0 0 123 0 0 7600 physiod RKl ? 120:57.98 [system]
0 0 0 0 125 0 0 7600 xclocv RKl ? 120:57.98 [system]
0 0 0 0 124 0 0 7600 syncer RKl ? 120:57.98 [system]
0 0 0 0 126 0 0 7600 pgdaemon RKl ? 120:57.98 [system]
0 0 0 0 96 0 0 7600 lnxcmplt RKl ? 120:57.98 [system]
0 0 0 0 123 0 0 7600 data RKl ? 120:57.98 [system]
0 0 0 0 96 0 0 7600 semacv RKl ? 120:57.98 [system]
0 0 0 0 96 0 0 7600 semacv RKl ? 120:57.98 [system]
0 0 0 0 96 0 0 7600 semacv RKl ? 120:57.98 [system]
0 0 0 0 43 0 0 7600 swwreboo RKl ? 120:57.98 [system]
0 0 0 0 96 0 0 7600 npfgcw RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 rt_free RKl ? 120:57.98 [system]
0 0 0 0 96 0 0 7600 unpgc RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 xclocv RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 icmp6_wq RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 icmp6_wq RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 icmp6_wq RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 icmp6_wq RKl ? 120:57.98 [system]
0 0 0 0 96 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 nd6_time RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 carp6_wq RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 carp6_wq RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 carp6_wq RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 carp6_wq RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 carp_wqi RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 carp_wqi RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 carp_wqi RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 carp_wqi RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 icmp_wqi RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 icmp_wqi RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 icmp_wqi RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 icmp_wqi RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 rt_timer RKl ? 120:57.98 [system]
0 0 0 0 125 0 0 7600 vmem_reh RKl ? 120:57.98 [system]
0 0 0 0 43 0 0 7600 vcmbox0 RKl ? 120:57.98 [system]
0 0 0 0 96 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 96 0 0 7600 usbtsk RKl ? 120:57.98 [system]
0 0 0 0 43 0 0 7600 dwc2 RKl ? 120:57.98 [system]
0 0 0 0 221 0 0 7600 mmctaskq RKl ? 120:57.98 [system]
0 0 0 0 43 0 0 7600 entropy RKl ? 120:57.98 [system]
0 0 0 0 127 0 0 7600 xcall RKl ? 120:57.98 [system]
0 0 0 0 223 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 220 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 221 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 0 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 127 0 0 7600 xcall RKl ? 120:57.98 [system]
0 0 0 0 223 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 220 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 221 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 0 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 127 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 223 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 220 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 221 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 0 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 43 0 0 7600 lnxsyswq RKl ? 120:57.98 [system]
0 0 0 0 43 0 0 7600 lnxubdwq RKl ? 120:57.98 [system]
0 0 0 0 43 0 0 7600 lnxpwrwq RKl ? 120:57.98 [system]
0 0 0 0 43 0 0 7600 lnxlngwq RKl ? 120:57.98 [system]
0 0 0 0 43 0 0 7600 lnxhipwq RKl ? 120:57.98 [system]
0 0 0 0 43 0 0 7600 lnxrcugc RKl ? 120:57.98 [system]
0 0 0 0 96 0 0 7600 smtaskq RKl ? 120:57.98 [system]
0 0 0 0 43 0 0 7600 pmfsuspe RKl ? 120:57.98 [system]
0 0 0 0 43 0 0 7600 pmfevent RKl ? 120:57.98 [system]
0 0 0 0 96 0 0 7600 sopendfr RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 ifwdog RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 iflnkst RKl ? 120:57.98 [system]
0 0 0 0 43 0 0 7600 nfssilly RKl ? 120:57.98 [system]
0 0 0 0 125 0 0 7600 vdrain RKl ? 120:57.98 [system]
0 0 0 0 125 0 0 7600 mod_unld RKl ? 120:57.98 [system]
0 0 0 0 127 0 0 7600 xcall RKl ? 120:57.98 [system]
0 0 0 0 223 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 220 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 221 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 222 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 0 0 0 7600 - RKl ? 120:57.98 [system]
0 0 0 0 125 0 0 7600 - RKl ? 120:57.98 [system]
0 1 0 0 85 0 5952 1056 wait Ss ? 0:00.06 init
0 113 14075 26624 79 0 7284 2008 select Ss ? 0:00.37 SCREEN -R (screen)
0 169 1 0 85 0 5356 52 nanoslp Ss ? 0:02.63 /sbin/wdogctl -u bcmpmwdog0
0 304 1 34510 43 0 25644 1256 - Isl ? 0:00.08 /usr/pkg/sbin/fscd -c /usr/pkg/etc/fscd.conf
0 304 1 34510 43 0 25644 1256 - Isl ? 0:00.08 /usr/pkg/sbin/fscd -c /usr/pkg/etc/fscd.conf
0 304 1 34510 43 0 25644 1256 - Isl ? 0:00.08 /usr/pkg/sbin/fscd -c /usr/pkg/etc/fscd.conf
0 304 1 34510 82 0 25644 1256 socket Isl ? 0:00.08 /usr/pkg/sbin/fscd -c /usr/pkg/etc/fscd.conf
0 304 1 34510 85 0 25644 1256 kqueue Isl ? 0:00.08 /usr/pkg/sbin/fscd -c /usr/pkg/etc/fscd.conf
1001 329 1 10860 83 0 10748 988 poll Is ? 0:00.05 /usr/pkg/sbin/staticd -d
0 385 1 7489 43 0 69152 4388 parked Ssl ? 1:44.83 /usr/sbin/dhcpd -q
0 385 1 7489 85 0 69152 4388 kqueue Ssl ? 1:44.83 /usr/sbin/dhcpd -q
0 385 1 7489 85 0 69152 4388 kqueue Ssl ? 1:44.83 /usr/sbin/dhcpd -q
0 385 1 7489 85 0 69152 4388 kqueue Ssl ? 1:44.83 /usr/sbin/dhcpd -q
0 385 1 7489 40 0 69152 4388 parked Ssl ? 1:44.83 /usr/sbin/dhcpd -q
0 414 1 0 85 0 10120 9176 - Rs ? 0:42.43 /usr/sbin/ntpd -p /var/run/ntpd.pid -g
0 421 1 9366 83 0 6100 880 devmon Is ? 0:00.03 /sbin/devpubd
0 643 1 26133 79 0 5924 972 kqueue Is ? 0:00.01 /usr/sbin/inetd -l
1002 657 1 0 85 0 163600 2344 - Rs ? 0:01.63 /usr/pkg/bin/postgres -D /usr/pkg/pgsql/data
1002 805 657 0 85 0 163744 1884 poll Ss ? 0:05.26 postgres: background writer
32767 828 1 0 85 0 6336 1016 - Rs ? 0:08.82 /usr/sbin/mdnsd
0 853 1 0 43 0 18224 2404 parked Ssl ? 0:17.29 /usr/pkg/sbin/bareos-fd -g bareos
0 853 1 0 85 0 18224 2404 poll Ssl ? 0:17.29 /usr/pkg/sbin/bareos-fd -g bareos
0 885 1 0 85 0 14104 1836 kqueue Ss ? 0:04.43 /usr/libexec/postfix/master -w
12 888 885 0 85 0 13796 2400 kqueue S ? 0:01.13 qmgr -l -t unix -u
0 952 1 12409 43 0 130676 3128 parked Rsl ? 8:00.60 /usr/sbin/named
0 952 1 12409 43 0 130676 3128 parked Rsl ? 8:00.60 /usr/sbin/named
0 952 1 12409 43 0 130676 3128 parked Rsl ? 8:00.60 /usr/sbin/named
0 952 1 12409 43 0 130676 3128 parked Rsl ? 8:00.60 /usr/sbin/named
0 952 1 12409 85 0 130676 3128 kqueue Rsl ? 8:00.60 /usr/sbin/named
0 952 1 12409 85 0 130676 3128 kqueue Rsl ? 8:00.60 /usr/sbin/named
0 952 1 12409 85 0 130676 3128 kqueue Rsl ? 8:00.60 /usr/sbin/named
0 952 1 12409 85 0 130676 3128 kqueue Rsl ? 8:00.60 /usr/sbin/named
0 952 1 12409 43 0 130676 3128 parked Rsl ? 8:00.60 /usr/sbin/named
0 952 1 12409 43 0 130676 3128 parked Rsl ? 8:00.60 /usr/sbin/named
0 952 1 12409 43 0 130676 3128 parked Rsl ? 8:00.60 /usr/sbin/named
0 952 1 12409 85 0 130676 3128 - Rsl ? 8:00.60 /usr/sbin/named
0 952 1 12409 43 0 130676 3128 parked Rsl ? 8:00.60 /usr/sbin/named
0 952 1 12409 83 0 130676 3128 sigwait Rsl ? 8:00.60 /usr/sbin/named
1004 1019 1172 584527 85 0 130232 2400 kqueue Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 43 0 130232 2400 - Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 43 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 34 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 34 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 34 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 34 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 34 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 31 0 130232 2400 parked Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 43 0 130232 2400 - Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
1004 1019 1172 584527 79 0 130232 2400 pipe_rd Rl ? 0:11.11 /usr/pkg/sbin/httpd -k start
0 1134 1 30290 78 0 9744 1768 kqueue Ss ? 5:01.79 /usr/sbin/syslogd -s
0 1172 1 0 85 0 10616 1376 select Ss ? 0:39.18 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 84 0 128104 2496 kqueue Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 42 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 33 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 33 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 33 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 33 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 33 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 33 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 33 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 33 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 33 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 31 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 31 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 31 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 31 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 31 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 31 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 31 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 31 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 31 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 31 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 31 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 31 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 31 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 31 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 31 0 128104 2496 parked Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 43 0 128104 2496 - Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
1004 1175 1172 614742 79 0 128104 2496 pipe_rd Sl ? 0:08.94 /usr/pkg/sbin/httpd -k start
0 1435 1 0 85 0 15112 1792 poll Ss ? 0:00.04 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd)
0 1460 1 0 85 0 5560 632 - Rs ? 1:27.24 /usr/pkg/sbin/gpiomon
1001 1518 1 1024 85 0 11172 1780 poll Ss ? 0:29.16 /usr/pkg/sbin/ripd -P 0 -d
1004 1524 1172 344575 85 0 128020 1112 - Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 43 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 37 0 128020 1112 parked Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 43 0 128020 1112 - Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
1004 1524 1172 344575 82 0 128020 1112 pipe_rd Rl ? 0:05.02 /usr/pkg/sbin/httpd -k start
0 1677 1 8 85 0 6088 1048 - Rs ? 0:04.30 /usr/sbin/cron
1005 1716 2860 0 85 0 77468 2876 - R ? 5:56.62 /usr/bin/perl /usr/pkg/fhem/fhem.pl /usr/pkg/fhem/fhem.cfg
1005 1717 2860 0 85 0 77640 2900 - R ? 9:44.65 /usr/bin/perl /usr/pkg/fhem/fhem.pl /usr/pkg/fhem/fhem.cfg
1002 1828 657 0 85 0 163908 2248 poll Ss ? 0:00.31 postgres: checkpointer
0 1853 1 36619 85 0 16244 3556 select Ss ? 0:55.32 /usr/pkg/bin/perl -wT /usr/pkg/sbin/munin-node
1005 1940 1 0 85 0 51076 1520 nanoslp S ? 3:45.01 /usr/bin/perl /usr/pkg/fhem/fhem.pl /usr/pkg/fhem/fhem.cfg
1001 2088 1 35690 85 0 44436 3280 poll Ssl ? 0:00.89 /usr/pkg/sbin/zebra -P 0 -d
1001 2088 1 35690 84 0 44436 3280 poll Ssl ? 0:00.89 /usr/pkg/sbin/zebra -P 0 -d
1001 2088 1 35690 82 0 44436 3280 poll Ssl ? 0:00.89 /usr/pkg/sbin/zebra -P 0 -d
1001 2088 1 35690 85 0 44436 3280 poll Ssl ? 0:00.89 /usr/pkg/sbin/zebra -P 0 -d
1001 2088 1 35690 36 0 44436 3280 parked Ssl ? 0:00.89 /usr/pkg/sbin/zebra -P 0 -d
1001 2088 1 35690 85 0 44436 3280 poll Ssl ? 0:00.89 /usr/pkg/sbin/zebra -P 0 -d
1005 2197 2860 1035 85 0 70020 3268 nanoslp S ? 87:12.67 /usr/bin/perl /usr/pkg/fhem/fhem.pl /usr/pkg/fhem/fhem.cfg
1002 2728 657 5120 84 0 22772 2228 poll Ss ? 0:12.10 postgres: stats collector
1002 2854 657 0 85 0 163824 1824 poll Ss ? 0:05.15 postgres: walwriter
1002 2983 657 1024 85 0 164056 3712 poll Ss ? 0:09.56 postgres: autovacuum launcher
1002 2987 657 0 85 0 163908 2876 poll Ss ? 0:00.56 postgres: logical replication launcher
1005 2998 2860 9216 83 0 77872 20856 netio S ? 8:09.65 /usr/bin/perl /usr/pkg/fhem/fhem.pl /usr/pkg/fhem/fhem.cfg
0 3037 10374 15808 82 0 6468 1068 wait I ? 0:00.08 /bin/sh /etc/daily
1004 3107 1172 36864 76 0 10288 1540 socket S ? 0:15.69 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 85 0 128104 2480 kqueue Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 43 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 34 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 34 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 34 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 34 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 34 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 34 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 34 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 34 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 31 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 31 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 31 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 31 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 31 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 31 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 31 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 31 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 31 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 31 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 31 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 31 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 31 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 31 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 31 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 31 0 128104 2480 parked Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 43 0 128104 2480 - Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1004 3500 1172 595247 79 0 128104 2480 pipe_rd Sl ? 0:05.71 /usr/pkg/sbin/httpd -k start
1000 3719 1 580 85 0 9416 912 poll Is ? 0:00.01 ssh-agent
1000 3903 5069 99 85 0 16276 2196 - R ? 1:56.00 sshd: kardel@pts/2 (sshd)
1000 4148 1 2822 85 0 8732 912 poll Is ? 0:00.01 ssh-agent
1008 4666 1 96796 42 0 190376 38920 parked Rsl ? 3:06.97 /usr/pkg/bin/node /usr/pkg/zigbee2mqtt/index.js
1008 4666 1 96796 40 0 190376 38920 parked Rsl ? 3:06.97 /usr/pkg/bin/node /usr/pkg/zigbee2mqtt/index.js
1008 4666 1 96796 38 0 190376 38920 parked Rsl ? 3:06.97 /usr/pkg/bin/node /usr/pkg/zigbee2mqtt/index.js
1008 4666 1 96796 38 0 190376 38920 parked Rsl ? 3:06.97 /usr/pkg/bin/node /usr/pkg/zigbee2mqtt/index.js
1008 4666 1 96796 84 0 190376 38920 psem Rsl ? 3:06.97 /usr/pkg/bin/node /usr/pkg/zigbee2mqtt/index.js
1008 4666 1 96796 43 0 190376 38920 parked Rsl ? 3:06.97 /usr/pkg/bin/node /usr/pkg/zigbee2mqtt/index.js
1008 4666 1 96796 42 0 190376 38920 parked Rsl ? 3:06.97 /usr/pkg/bin/node /usr/pkg/zigbee2mqtt/index.js
1008 4666 1 96796 43 0 190376 38920 parked Rsl ? 3:06.97 /usr/pkg/bin/node /usr/pkg/zigbee2mqtt/index.js
1008 4666 1 96796 43 0 190376 38920 - Rsl ? 3:06.97 /usr/pkg/bin/node /usr/pkg/zigbee2mqtt/index.js
1008 4666 1 96796 85 0 190376 38920 kqueue Rsl ? 3:06.97 /usr/pkg/bin/node /usr/pkg/zigbee2mqtt/index.js
1008 4666 1 96796 76 0 190376 38920 kqueue Rsl ? 3:06.97 /usr/pkg/bin/node /usr/pkg/zigbee2mqtt/index.js
0 5069 1435 1065 85 0 16012 2760 poll Is ? 0:00.58 sshd: kardel [priv]
0 5611 10374 0 85 0 13588 1828 pipe_rd I ? 0:00.07 sendmail -t
0 5664 4954 0 85 0 7168 1352 select Ss ? 1:23.66 SCREEN -R (screen)
0 10374 16429 331 85 0 6020 1068 wait Is ? 0:00.02 /bin/sh -c /bin/sh /etc/daily 2>&1 | tee /var/log/daily.out | sendmail -t
0 10885 3037 1179 85 0 6624 1200 - R ? 0:00.27 sh /etc/security
0 15499 5611 654 85 0 13564 1828 netio I ? 0:00.10 postdrop -r
12 16001 885 0 85 0 13656 1880 - R ? 0:00.09 pickup -l -t unix -u
0 16429 1677 331 85 0 12520 1704 pipe_rd I ? 0:00.05 cron: running job
0 17255 1435 13312 82 0 16008 5880 poll Ss ? 0:00.65 sshd: kardel [priv]
0 17312 10374 0 85 0 5464 892 pipe_rd I ? 0:00.01 tee /var/log/daily.out
1000 22307 17255 21504 80 0 16148 4800 poll S ? 0:00.55 sshd: kardel@pts/0 (sshd)
1000 26845 1 20480 80 0 8732 1496 poll Ss ? 0:00.01 ssh-agent
0 28610 1 36864 76 0 14368 3132 select Ss ? 6:04.20 /usr/sbin/racoon
1000 11906 22307 20480 81 0 6748 1976 pause Ss pts/0 0:00.13 -csh
1000 14075 11906 20480 81 0 6928 2032 pause S+ pts/0 0:00.03 screen -R
1000 8151 17710 36864 25 0 6300 1612 - O+ pts/1 0:00.02 ps -alxs
1000 17710 113 36864 76 0 7412 2656 pause Ss pts/1 0:00.24 -usr/pkg/bin/tcsh
1000 4928 3903 580 85 0 6748 1216 pause Is pts/2 0:00.11 -csh
1000 4954 4928 0 85 0 6928 1244 - R+ pts/2 0:03.05 screen -R
0 1915 8499 0 85 0 6120 1340 - R+ pts/3 3:19.01 ping gateway
1000 4951 5664 0 85 0 7412 1332 pause Is pts/3 0:00.14 -usr/pkg/bin/tcsh
0 8499 4951 9 85 0 7384 1272 pause I pts/3 0:00.60 -csh (tcsh)
0 1939 1 30536 78 0 5744 1128 ttyraw Is+ ttyE0 0:00.02 /usr/libexec/getty default constty
1005 2860 1 2613 85 0 85084 3272 - R ttyE0- 171:49.22 /usr/bin/perl /usr/pkg/fhem/fhem.pl /usr/pkg/fhem/fhem.cfg
crash ps:
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
24191>24191 7 0 0 93ce9340 crash
4453 4453 3 3 180 94097100 tcsh pause
17710 17710 3 2 180 91dfd700 tcsh pause
113 113 3 2 180 91fd3240 screen select
14075 14075 3 0 180 93f75d00 screen pause
26845 26845 3 0 180 93fd7600 ssh-agent poll
11906 11906 3 0 180 940bb400 csh pause
22307 22307 3 3 180 940bb6c0 sshd poll
17255 17255 3 0 180 940bb140 sshd poll
11561 11561 3 3 1000000 91fd37c0 perl tstile
16846 16846 3 0 1000000 93ce9600 perl tstile
4939 >4939 7 1 1000000 93f75200 perl
10885 10885 2 1 100 934bd3c0 sh
15499 15499 3 0 180 93ce9b80 postdrop netio
5611 5611 3 1 180 96630480 sendmail pipe_rd
17312 17312 3 3 180 9404abc0 tee pipe_rd
3037 3037 3 0 180 930a4400 sh wait
10374 10374 3 2 180 93faa040 sh wait
16429 16429 3 2 180 91fb9780 cron pipe_rd
16001 16001 2 1 100 94097c00 pickup
1915 1915 2 1 140 91a21900 ping
28610 28610 3 0 180 93f95d40 racoon select
8499 8499 3 0 180 93fd7b80 tcsh pause
4951 4951 3 0 180 93f957c0 tcsh pause
5664 5664 3 2 180 9404a380 screen select
4954 4954 2 1 1000100 93fd7080 screen
3719 3719 3 0 180 93faab40 ssh-agent poll
4928 4928 3 2 180 93f33740 csh pause
3903 3903 2 1 100 93fd7340 sshd
5069 5069 3 1 180 93f95500 sshd poll
4148 4148 3 2 180 93faa300 ssh-agent poll
4666 2608 3 0 180 93f75a40 node parked
4666 2847 3 3 180 93f33480 node parked
4666 4152 3 2 180 93ee1400 node parked
4666 5942 3 0 180 93f169c0 node parked
4666 4130 3 2 180 93f754c0 node psem
4666 4248 3 2 180 93f331c0 node parked
4666 2842 3 0 180 93e62c00 node parked
4666 3677 3 0 180 93e5e0c0 node parked
4666 3582 2 1 100 93e62680 node
4666 5848 3 0 180 93e623c0 node kqueue
4666 4666 3 3 180 93e62100 node kqueue
2998 2998 3 2 1c0 93ce9080 perl netio
1717 1717 2 1 140 93469bc0 perl
1716 1716 2 1 140 9350ad40 perl
2197 2197 3 0 180 9201e880 perl nanoslp
1940 1940 3 3 180 9184d240 perl nanoslp
1939 1939 3 1 180 91772c80 getty ttyraw
1677 1677 2 1 140 934bd680 cron
643 643 3 0 180 93369180 inetd kqueue
888 888 3 2 180 934690c0 qmgr kqueue
885 885 3 0 180 93469900 master kqueue
1524 791 2 1 140 9350a7c0 httpd
1524 790 3 2 1c0 9350a500 httpd parked
1524 789 3 2 180 9350a240 httpd parked
1524 788 3 2 180 93507d00 httpd parked
1524 787 3 2 180 93507a40 httpd parked
1524 786 3 2 180 93507780 httpd parked
1524 785 3 2 180 935074c0 httpd parked
1524 784 3 2 180 93507200 httpd parked
1524 783 3 2 180 93503cc0 httpd parked
1524 782 3 2 180 93503a00 httpd parked
1524 781 3 3 180 93503740 httpd parked
1524 780 3 3 180 93503480 httpd parked
1524 779 3 3 180 935031c0 httpd parked
1524 778 3 3 180 934fdc80 httpd parked
1524 777 3 2 180 934fd9c0 httpd parked
1524 776 3 2 180 934fd700 httpd parked
1524 775 3 2 180 934fd440 httpd parked
1524 774 3 2 180 934fd180 httpd parked
1524 773 3 2 180 934fac40 httpd parked
1524 772 3 2 180 934fa980 httpd parked
1524 771 3 2 180 934fa6c0 httpd parked
1524 770 3 2 180 934fa400 httpd parked
1524 769 3 2 180 934fa140 httpd parked
1524 768 3 2 180 934bdc00 httpd parked
1524 1826 3 3 180 933efb80 httpd parked
1524 1948 3 2 180 93469640 httpd parked
1524 1837 5 0 0 934bd940 httpd
1524 1524 3 2 180 93469380 httpd pipe_rd
1435 1435 3 0 180 930a4980 sshd poll
1175 1276 3 2 1c0 933de780 httpd kqueue
1175 3583 3 0 1c0 933de4c0 httpd parked
1175 1272 3 3 180 933de200 httpd parked
1175 1395 3 3 180 933d7cc0 httpd parked
1175 1274 3 3 180 933d7a00 httpd parked
1175 1273 3 3 180 933d7480 httpd parked
1175 3568 3 3 180 933d71c0 httpd parked
1175 1108 3 3 180 933d1c80 httpd parked
1175 1271 3 3 180 933d19c0 httpd parked
1175 1394 3 3 180 933d1700 httpd parked
1175 2265 3 3 180 933d1440 httpd parked
1175 1253 3 3 180 933c3100 httpd parked
1175 1248 3 2 180 933bc640 httpd parked
1175 3001 3 3 180 933b6880 httpd parked
1175 1103 3 1 180 933b65c0 httpd parked
1175 1110 3 3 180 933b6300 httpd parked
1175 1071 3 3 180 933b6040 httpd parked
1175 1854 3 3 180 933b0d40 httpd parked
1175 3278 3 3 180 933b0a80 httpd parked
1175 1106 3 3 180 933b07c0 httpd parked
1175 1095 3 3 180 933b0500 httpd parked
1175 1104 3 3 180 933b0240 httpd parked
1175 849 3 3 180 933aba00 httpd parked
1175 1093 3 3 180 933ab740 httpd parked
1175 1066 3 3 180 933ab480 httpd parked
1175 1215 3 3 180 933ab1c0 httpd parked
1175 3126 5 1 0 922260c0 httpd
1175 1175 3 1 180 93369700 httpd pipe_rd
3500 1421 3 2 1c0 933ef8c0 httpd kqueue
3500 922 3 3 1c0 933ef600 httpd parked
3500 1418 3 3 180 933ef340 httpd parked
3500 1417 3 3 180 933ef080 httpd parked
3500 1416 3 3 180 933e9b40 httpd parked
3500 3163 3 3 180 933e9880 httpd parked
3500 1415 3 3 180 933e95c0 httpd parked
3500 1098 3 3 180 933e9300 httpd parked
3500 3511 3 3 180 933e9040 httpd parked
3500 2652 3 1 180 933e2d40 httpd parked
3500 1517 3 3 180 933d1180 httpd parked
3500 1250 3 1 180 933bcbc0 httpd parked
3500 1118 3 2 180 933bc0c0 httpd parked
3500 2253 3 3 180 933add00 httpd parked
3500 1091 3 3 180 933ada40 httpd parked
3500 2251 3 3 180 933ad780 httpd parked
3500 1073 3 3 180 933ad4c0 httpd parked
3500 1092 3 3 180 933ad200 httpd parked
3500 1090 3 3 180 933abcc0 httpd parked
3500 1980 3 2 180 9184da80 httpd parked
3500 3129 3 2 180 91c5f0c0 httpd parked
3500 1088 3 1 180 91cc4100 httpd parked
3500 1848 3 2 180 919c9080 httpd parked
3500 1171 3 0 180 930a4c40 httpd parked
3500 1461 3 1 180 924cec00 httpd parked
3500 1014 3 2 180 9201eb40 httpd parked
3500 1167 5 1 0 92226bc0 httpd
3500 3500 3 2 180 93369440 httpd pipe_rd
1019 3440 3 0 180 933e2a80 httpd kqueue
1019 1096 2 1 140 933e27c0 httpd
1019 1277 3 3 1c0 933e2500 httpd parked
1019 1278 3 2 180 933e2240 httpd parked
1019 2336 3 2 180 933ded00 httpd parked
1019 3455 3 2 180 933dea40 httpd parked
1019 1269 3 2 180 933d7740 httpd parked
1019 3308 3 3 180 933c9c40 httpd parked
1019 1259 3 3 180 933c9980 httpd parked
1019 2287 3 3 180 933c96c0 httpd parked
1019 1255 3 3 180 933c9400 httpd parked
1019 1252 3 3 180 933c9140 httpd parked
1019 1257 3 3 180 933c3c00 httpd parked
1019 3306 3 3 180 933c3940 httpd parked
1019 2875 3 3 180 933c3680 httpd parked
1019 1256 3 3 180 933c33c0 httpd parked
1019 1254 3 2 180 933bc900 httpd parked
1019 1100 3 2 180 933bc380 httpd parked
1019 2266 3 2 180 933b9b80 httpd parked
1019 1111 3 2 180 933b98c0 httpd parked
1019 2904 3 2 180 933b9600 httpd parked
1019 2780 3 3 180 933b9340 httpd parked
1019 3291 3 3 180 933b9080 httpd parked
1019 1152 3 3 180 933b6b40 httpd parked
1019 1089 3 2 180 93369c80 httpd parked
1019 1065 3 2 180 933699c0 httpd parked
1019 3383 5 1 0 924ce100 httpd
1019 1019 3 2 180 930a4140 httpd pipe_rd
3107 3107 3 0 180 91dfd9c0 httpd socket
1172 1172 3 2 180 930a46c0 httpd select
853 993 3 2 180 92226900 bareos-fd parked
853 853 3 2 180 9203db80 bareos-fd poll
2860 2860 2 1 100 91f4c740 perl
1460 1460 2 1 140 919c98c0 gpiomon
1853 1853 3 2 180 91d68c40 perl select
2987 2987 3 3 180 924ce940 postgres poll
2728 2728 3 3 180 924ce680 postgres poll
2983 2983 3 2 180 924ce3c0 postgres poll
2854 2854 3 2 180 92226380 postgres poll
805 805 3 2 180 92226640 postgres poll
1828 1828 3 0 180 91dfdc80 postgres poll
657 657 2 1 140 91f4ccc0 postgres
1518 1518 3 0 180 91dfd440 ripd poll
329 329 3 1 180 91fb94c0 staticd poll
304 3410 5 2 140 93f33cc0 fscd
304 1208 5 2 140 91809d00 fscd
304 2454 5 2 100 9184d7c0 fscd
304 2482 3 3 180 9203d600 fscd socket
304 304 3 0 180 919c9b80 fscd kqueue
414 414 2 1 1000100 9203d340 ntpd
385 389 3 3 180 9201e300 timer parked
385 388 3 0 1c0 9201e040 sock-0 kqueue
385 387 3 3 180 91fd3d40 net-1 kqueue
385 386 3 3 180 91fd3a80 net-0 kqueue
385 385 3 2 180 91f4c1c0 dhcpd parked
2088 2801 3 2 180 91c5fbc0 zebra_apic poll
2088 339 3 1 180 91f4c480 zebra_apic poll
2088 1979 3 0 180 91fb9200 zebra_opaque poll
2088 2211 3 2 180 918d85c0 zebra_dplane poll
2088 2417 3 1 180 91dfd180 RCU sweeper parked
2088 2088 3 2 1c0 919c9600 zebra poll
952 7985 3 1 180 940bf180 named parked
952 8001 3 0 180 9404a900 named parked
952 8254 3 1 180 94097680 named parked
952 8004 3 0 180 940bbc40 named parked
952 820 3 0 180 91d68980 sock-3 kqueue
952 1119 3 0 180 91d686c0 sock-2 kqueue
952 1099 3 3 180 91d68400 sock-1 kqueue
952 1113 3 2 180 91d68140 sock-0 kqueue
952 1114 3 2 180 91cc4c00 timer parked
952 954 3 3 180 916207c0 net-3 parked
952 1101 3 0 180 91624040 net-2 parked
952 958 2 1 100 91c5f900 net-1
952 1135 3 2 180 91620a80 net-0 parked
952 952 3 3 180 91c5f640 named sigwait
1134 1134 3 2 180 91cc4940 syslogd kqueue
828 828 2 1 100 91cc4680 mdnsd
421 421 3 1 180 918d8b40 devpubd devmon
169 169 3 2 180 918d8300 wdogctl nanoslp
1 1 3 0 180 91772440 init wait
0 20474 5 2 200 91fb9a40 (zombie)
0 1444 2 1 240 919c9340 onewire0
0 309 2 1 240 918d8880 hythygtemp2
0 499 3 2 200 9184dd40 hythygtemp1 hytcv
0 498 3 2 200 918d8040 hythygtemp0 hytcv
0 365 3 0 200 918061c0 physiod physiod
0 214 3 0 240 91809a40 pooldrain xclocv
0 213 3 3 200 91809780 ioflush syncer
0 212 3 2 200 918094c0 pgdaemon pgdaemon
0 211 3 2 380 91809200 VCHIQka-0 lnxcmplt
0 210 3 3 380 91806cc0 vcaudio data
0 209 3 3 380 91806a00 VCHIQs-0 semacv
0 208 3 3 380 91806740 VCHIQr-0 semacv
0 207 3 3 380 91806480 VCHIQ-0 semacv
0 204 3 2 200 917729c0 swwreboot swwreboot
0 202 3 2 200 91772700 npfgc0 npfgcw
0 201 3 0 200 91772180 rt_free rt_free
0 200 3 0 200 91762c40 unpgc unpgc
0 199 3 3 200 91762980 key_timehandler xclocv
0 198 3 3 200 917626c0 icmp6_wqinput/3 icmp6_wqinput
0 197 3 2 200 91762400 icmp6_wqinput/2 icmp6_wqinput
0 196 3 1 200 91624300 icmp6_wqinput/1 icmp6_wqinput
0 195 3 0 200 91620500 icmp6_wqinput/0 icmp6_wqinput
0 194 2 1 240 914e4d00 usb0
0 193 3 2 200 91620d40 nd6_timer nd6_timer
0 192 3 3 200 91762140 carp6_wqinput/3 carp6_wqinput
0 175 3 2 200 9175dc00 carp6_wqinput/2 carp6_wqinput
0 177 3 1 200 9175d940 carp6_wqinput/1 carp6_wqinput
0 176 3 0 200 9175d680 carp6_wqinput/0 carp6_wqinput
0 170 3 3 200 9175d3c0 carp_wqinput/3 carp_wqinput
0 173 3 2 200 9175d100 carp_wqinput/2 carp_wqinput
0 31 3 1 200 91752bc0 carp_wqinput/1 carp_wqinput
0 63 3 0 200 91752900 carp_wqinput/0 carp_wqinput
0 126 3 3 200 91752640 icmp_wqinput/3 icmp_wqinput
0 125 3 2 200 91752380 icmp_wqinput/2 icmp_wqinput
0 124 3 1 200 917520c0 icmp_wqinput/1 icmp_wqinput
0 123 3 0 200 91628b80 icmp_wqinput/0 icmp_wqinput
0 122 3 2 200 916288c0 rt_timer rt_timer
0 121 3 0 200 916245c0 vmem_rehash vmem_rehash
0 120 3 3 200 91624880 vcmbox0 vcmbox0
0 119 2 1 200 91628600 usbtask-dr
0 118 3 2 200 91628340 usbtask-hc usbtsk
0 117 3 3 200 91628080 dwc2 dwc2
0 116 3 2 200 91624b40 sdmmc0 mmctaskq
0 107 3 0 200 91620240 entbutler entropy
0 105 3 3 200 914e4a40 xcall/3 xcall
0 104 1 3 200 914e4780 softser/3
0 103 1 3 200 914e44c0 softclk/3
0 102 1 3 200 914e4200 softbio/3
0 101 1 3 200 914e1cc0 softnet/3
0 > 100 1 3 201 914e1a00 idle/3
0 99 3 2 200 914e1740 xcall/2 xcall
0 98 1 2 200 914e1480 softser/2
0 97 1 2 200 914e11c0 softclk/2
0 96 1 2 200 914ddc80 softbio/2
0 30 1 2 200 914dd9c0 softnet/2
0 > 29 1 2 201 914dd700 idle/2
0 28 2 1 200 914dd440 xcall/1
0 27 1 1 200 914dd180 softser/1
0 26 1 1 200 914c3c40 softclk/1
0 25 1 1 200 914c3980 softbio/1
0 24 1 1 200 914c36c0 softnet/1
0 23 1 1 201 914c3400 idle/1
0 22 3 3 200 914c3140 lnxsyswq lnxsyswq
0 21 3 0 200 914c2c00 lnxubdwq lnxubdwq
0 20 3 0 200 914c2940 lnxpwrwq lnxpwrwq
0 19 3 1 200 914c2680 lnxlngwq lnxlngwq
0 18 3 2 200 914c23c0 lnxhipwq lnxhipwq
0 17 3 0 200 914c2100 lnxrcugc lnxrcugc
0 16 3 2 200 914bbbc0 sysmon smtaskq
0 15 3 2 200 914bb900 pmfsuspend pmfsuspend
0 14 3 0 200 914bb640 pmfevent pmfevent
0 13 3 0 200 914bb380 sopendfree sopendfr
0 12 3 3 200 914bb0c0 ifwdog ifwdog
0 11 3 2 200 914aeb80 iflnkst iflnkst
0 10 3 0 200 914ae8c0 nfssilly nfssilly
0 9 3 3 240 914ae600 vdrain vdrain
0 8 3 2 240 914ae340 modunload mod_unld
0 7 3 0 200 914ae080 xcall/0 xcall
0 6 1 0 200 914acb40 softser/0
0 5 1 0 200 914ac880 softclk/0
0 4 1 0 200 914ac5c0 softbio/0
0 3 1 0 200 914ac300 softnet/0
0 2 1 0 201 914ac040 idle/0
0 0 2 1 240 8094a880 swapper
crash stacks:
trace: pid 9624 lid 9624 at 0xbb52fdc4
0xbb52fdc4: vop_lock_desc
Bad frame pointer: 0x805a0700
trace: pid 20132 lid 20132 at 0xbc0b1dc4
0xbc0b1dc4: $d+0x994
Bad frame pointer: 0x807a8a98
trace: pid 10232 lid 10232
trace: pid 23079 lid 23079
trace: pid 5478 lid 5478
trace: pid 4453 lid 4453 at 0xbb687e34
0xbb687e34: mi_switch+0x10
0xbb687e64: sleepq_block+0xb4
0xbb687ec4: kpause+0x164
0xbb687eec: sys___sigsuspend14+0x6c
0xbb687fac: syscall+0x188
--- tf 0xbb687fb0 ---
trace: pid 17710 lid 17710 at 0xbb6cde34
0xbb6cde34: mi_switch+0x10
0xbb6cde64: sleepq_block+0xb4
0xbb6cdec4: kpause+0x164
0xbb6cdeec: sys___sigsuspend14+0x6c
0xbb6cdfac: syscall+0x188
--- tf 0xbb6cdfb0 ---
trace: pid 113 lid 113 at 0xbb6cbca4
0xbb6cbca4: mi_switch+0x10
0xbb6cbcd4: sleepq_block+0xb4
0xbb6cbd84: sel_do_scan+0x5ec
0xbb6cbe9c: selcommon+0x124
0xbb6cbeec: sys___select50+0x80
0xbb6cbfac: syscall+0x188
--- tf 0xbb6cbfb0 ---
trace: pid 14075 lid 14075 at 0xbb68be34
0xbb68be34: mi_switch+0x10
0xbb68be64: sleepq_block+0xb4
0xbb68bec4: kpause+0x164
0xbb68beec: sys___sigsuspend14+0x6c
0xbb68bfac: syscall+0x188
--- tf 0xbb68bfb0 ---
trace: pid 26845 lid 26845 at 0xbb719c94
0xbb719c94: mi_switch+0x10
0xbb719cc4: sleepq_block+0xb4
0xbb719d74: sel_do_scan+0x5ec
0xbb719ebc: pollcommon+0xa8
0xbb719eec: sys_poll+0x70
0xbb719fac: syscall+0x188
--- tf 0xbb719fb0 ---
trace: pid 11906 lid 11906 at 0xbb70fe34
0xbb70fe34: mi_switch+0x10
0xbb70fe64: sleepq_block+0xb4
0xbb70fec4: kpause+0x164
0xbb70feec: sys___sigsuspend14+0x6c
0xbb70ffac: syscall+0x188
--- tf 0xbb70ffb0 ---
trace: pid 22307 lid 22307 at 0xbb56fc7c
0xbb56fc7c: mi_switch+0x10
0xbb56fcac: sleepq_block+0xb4
0xbb56fd5c: sel_do_scan+0x5ec
0xbb56fea4: pollcommon+0xa8
0xbb56feec: sys___pollts50+0x7c
0xbb56ffac: syscall+0x188
--- tf 0xbb56ffb0 ---
trace: pid 17255 lid 17255 at 0xbb6cfc94
0xbb6cfc94: mi_switch+0x10
0xbb6cfcc4: sleepq_block+0xb4
0xbb6cfd74: sel_do_scan+0x5ec
0xbb6cfebc: pollcommon+0xa8
0xbb6cfeec: sys_poll+0x70
0xbb6cffac: syscall+0x188
--- tf 0xbb6cffb0 ---
trace: pid 11561 lid 11561 at 0xbb6f1844
0xbb6f1844: mi_switch+0x10
0xbb6f1874: sleepq_block+0xb4
0xbb6f18fc: turnstile_block+0x58c
0xbb6f197c: rw_enter+0x17c
0xbb6f199c: genfs_lock+0xa4
0xbb6f19dc: VOP_LOCK+0x84
0xbb6f1a04: vn_lock+0xb0
0xbb6f1af4: namei_tryemulroot+0x9b0
0xbb6f1b2c: namei+0x34
0xbb6f1bd4: check_exec+0x68
0xbb6f1c14: exec_script_makecmds+0x32c
0xbb6f1cbc: check_exec+0x350
0xbb6f1d34: execve_loadvm+0x258
0xbb6f1ec4: execve1+0x38
0xbb6f1eec: sys_execve+0x34
0xbb6f1fac: syscall+0x188
--- tf 0xbb6f1fb0 ---
trace: pid 16846 lid 16846 at 0xbb6c3844
0xbb6c3844: mi_switch+0x10
0xbb6c3874: sleepq_block+0xb4
0xbb6c38fc: turnstile_block+0x58c
0xbb6c397c: rw_enter+0x17c
0xbb6c399c: genfs_lock+0xa4
0xbb6c39dc: VOP_LOCK+0x84
0xbb6c3a04: vn_lock+0xb0
0xbb6c3af4: namei_tryemulroot+0x9b0
0xbb6c3b2c: namei+0x34
0xbb6c3bd4: check_exec+0x68
0xbb6c3c14: exec_script_makecmds+0x32c
0xbb6c3cbc: check_exec+0x350
0xbb6c3d34: execve_loadvm+0x258
0xbb6c3ec4: execve1+0x38
0xbb6c3eec: sys_execve+0x34
0xbb6c3fac: syscall+0x188
--- tf 0xbb6c3fb0 ---
trace: pid 4939 lid 4939 at 0xbb511914
0xbb511914: 0
trace: pid 10885 lid 10885 at 0xbb6f7c5c
0xbb6f7c5c: mi_switch+0x10
0xbb6f7c8c: sleepq_block+0xb4
0xbb6f7cb4: cv_wait_sig+0x54
0xbb6f7d9c: do_sys_waitid+0x138
0xbb6f7e74: do_sys_wait+0xb8
0xbb6f7eec: sys___wait450+0x3c
0xbb6f7fac: syscall+0x188
--- tf 0xbb6f7fb0 ---
trace: pid 15499 lid 15499 at 0xbb6f3cfc
0xbb6f3cfc: mi_switch+0x10
0xbb6f3d2c: sleepq_block+0xb4
0xbb6f3d54: cv_timedwait_sig+0x58
0xbb6f3d84: sbwait+0x80
0xbb6f3e4c: soreceive+0xc94
0xbb6f3e6c: soo_read+0x2c
0xbb6f3ebc: dofileread+0x7c
0xbb6f3eec: sys_read+0x50
0xbb6f3fac: syscall+0x188
--- tf 0xbb6f3fb0 ---
trace: pid 5611 lid 5611 at 0xbb6fbdc4
0xbb6fbdc4: mi_switch+0x10
0xbb6fbdf4: sleepq_block+0xb4
0xbb6fbe1c: cv_wait_sig+0x54
0xbb6fbe6c: pipe_read+0x32c
0xbb6fbebc: dofileread+0x7c
0xbb6fbeec: sys_read+0x50
0xbb6fbfac: syscall+0x188
--- tf 0xbb6fbfb0 ---
trace: pid 17312 lid 17312 at 0xbb5d7dc4
0xbb5d7dc4: mi_switch+0x10
0xbb5d7df4: sleepq_block+0xb4
0xbb5d7e1c: cv_wait_sig+0x54
0xbb5d7e6c: pipe_read+0x32c
0xbb5d7ebc: dofileread+0x7c
0xbb5d7eec: sys_read+0x50
0xbb5d7fac: syscall+0x188
trace: pid 3037 lid 3037 at 0xbb5d9c5c
0xbb5d9c5c: mi_switch+0x10
0xbb5d9c8c: sleepq_block+0xb4
0xbb5d9cb4: cv_wait_sig+0x54
0xbb5d9d9c: do_sys_waitid+0x138
0xbb5d9e74: do_sys_wait+0xb8
0xbb5d9eec: sys___wait450+0x3c
0xbb5d9fac: syscall+0x188
--- tf 0xbb5d9fb0 ---
trace: pid 10374 lid 10374 at 0xbb5dbc5c
0xbb5dbc5c: mi_switch+0x10
0xbb5dbc8c: sleepq_block+0xb4
0xbb5dbcb4: cv_wait_sig+0x54
0xbb5dbd9c: do_sys_waitid+0x138
0xbb5dbe74: do_sys_wait+0xb8
0xbb5dbeec: sys___wait450+0x3c
0xbb5dbfac: syscall+0x188
--- tf 0xbb5dbfb0 ---
trace: pid 16429 lid 16429 at 0xbc2b5dc4
0xbc2b5dc4: mi_switch+0x10
0xbc2b5df4: sleepq_block+0xb4
0xbc2b5e1c: cv_wait_sig+0x54
0xbc2b5e6c: pipe_read+0x32c
0xbc2b5ebc: dofileread+0x7c
0xbc2b5eec: sys_read+0x50
0xbc2b5fac: syscall+0x188
--- tf 0xbc2b5fb0 ---
trace: pid 16001 lid 16001 at 0xbb27fc5c
0xbb27fc5c: mi_switch+0x10
0xbb27fc8c: sleepq_block+0x124
0xbb27fcb4: cv_timedwait_sig+0x58
0xbb27fec4: kevent1+0xcfc
0xbb27feec: sys___kevent50+0x3c
0xbb27ffac: syscall+0x188
--- tf 0xbb27ffb0 ---
trace: pid 1915 lid 1915 at 0xbc583c94
0xbc583c94: mi_switch+0x10
0xbc583cc4: sleepq_block+0x124
0xbc583d74: sel_do_scan+0x5ec
0xbc583ebc: pollcommon+0xa8
0xbc583eec: sys_poll+0x70
0xbc583fac: syscall+0x188
--- tf 0xbc583fb0 ---
trace: pid 28610 lid 28610 at 0xbc56bca4
0xbc56bca4: mi_switch+0x10
0xbc56bcd4: sleepq_block+0x124
0xbc56bd84: sel_do_scan+0x5ec
0xbc56be9c: selcommon+0x124
0xbc56beec: sys___select50+0x80
0xbc56bfac: syscall+0x188
--- tf 0xbc56bfb0 ---
trace: pid 8499 lid 8499 at 0xbc9ffe34
0xbc9ffe34: mi_switch+0x10
0xbc9ffe64: sleepq_block+0xb4
0xbc9ffec4: kpause+0x164
0xbc9ffeec: sys___sigsuspend14+0x6c
0xbc9fffac: syscall+0x188
--- tf 0xbc9fffb0 ---
trace: pid 4951 lid 4951 at 0xbc9cfe34
0xbc9cfe34: mi_switch+0x10
0xbc9cfe64: sleepq_block+0xb4
0xbc9cfec4: kpause+0x164
0xbc9cfeec: sys___sigsuspend14+0x6c
0xbc9cffac: syscall+0x188
--- tf 0xbc9cffb0 ---
trace: pid 5664 lid 5664 at 0xbca83ca4
0xbca83ca4: mi_switch+0x10
0xbca83cd4: sleepq_block+0xb4
0xbca83d84: sel_do_scan+0x5ec
0xbca83e9c: selcommon+0x124
0xbca83eec: sys___select50+0x80
0xbca83fac: syscall+0x188
--- tf 0xbca83fb0 ---
trace: pid 4954 lid 4954 at 0xbc9f7e34
0xbc9f7e34: mi_switch+0x10
0xbc9f7e64: sleepq_block+0xb4
0xbc9f7ec4: kpause+0x164
0xbc9f7eec: sys___sigsuspend14+0x6c
0xbc9f7fac: syscall+0x188
--- tf 0xbc9f7fb0 ---
trace: pid 3719 lid 3719 at 0xbc9f5c94
0xbc9f5c94: mi_switch+0x10
0xbc9f5cc4: sleepq_block+0xb4
0xbc9f5d74: sel_do_scan+0x5ec
0xbc9f5ebc: pollcommon+0xa8
0xbc9f5eec: sys_poll+0x70
0xbc9f5fac: syscall+0x188
--- tf 0xbc9f5fb0 ---
trace: pid 4928 lid 4928 at 0xbc9afe34
0xbc9afe34: mi_switch+0x10
0xbc9afe64: sleepq_block+0xb4
0xbc9afec4: kpause+0x164
0xbc9afeec: sys___sigsuspend14+0x6c
0xbc9affac: syscall+0x188
--- tf 0xbc9affb0 ---
trace: pid 3903 lid 3903 at 0xbc9f9c7c
0xbc9f9c7c: mi_switch+0x10
0xbc9f9cac: sleepq_block+0xb4
0xbc9f9d5c: sel_do_scan+0x5ec
0xbc9f9ea4: pollcommon+0xa8
0xbc9f9eec: sys___pollts50+0x7c
0xbc9f9fac: syscall+0x188
--- tf 0xbc9f9fb0 ---
trace: pid 5069 lid 5069 at 0xbc9cdc94
0xbc9cdc94: mi_switch+0x10
0xbc9cdcc4: sleepq_block+0xb4
0xbc9cdd74: sel_do_scan+0x5ec
0xbc9cdebc: pollcommon+0xa8
0xbc9cdeec: sys_poll+0x70
0xbc9cdfac: syscall+0x188
--- tf 0xbc9cdfb0 ---
trace: pid 4148 lid 4148 at 0xbc9efc94
0xbc9efc94: mi_switch+0x10
0xbc9efcc4: sleepq_block+0xb4
0xbc9efd74: sel_do_scan+0x5ec
0xbc9efebc: pollcommon+0xa8
0xbc9efeec: sys_poll+0x70
0xbc9effac: syscall+0x188
--- tf 0xbc9effb0 ---
trace: pid 4666 lid 2608 at 0xbc9c7e24
0xbc9c7e24: mi_switch+0x10
0xbc9c7e54: sleepq_block+0xb4
0xbc9c7ebc: lwp_park+0x134
0xbc9c7eec: sys____lwp_park60+0x68
0xbc9c7fac: syscall+0x188
--- tf 0xbc9c7fb0 ---
trace: pid 4666 lid 2847 at 0xbc979e24
0xbc979e24: mi_switch+0x10
0xbc979e54: sleepq_block+0xb4
0xbc979ebc: lwp_park+0x134
0xbc979eec: sys____lwp_park60+0x68
0xbc979fac: syscall+0x188
--- tf 0xbc979fb0 ---
trace: pid 4666 lid 4152 at 0xbc90de24
0xbc90de24: mi_switch+0x10
0xbc90de54: sleepq_block+0xb4
0xbc90debc: lwp_park+0x134
0xbc90deec: sys____lwp_park60+0x68
0xbc90dfac: syscall+0x188
--- tf 0xbc90dfb0 ---
trace: pid 4666 lid 5942 at 0xbc96be24
0xbc96be24: mi_switch+0x10
0xbc96be54: sleepq_block+0xb4
0xbc96bebc: lwp_park+0x134
0xbc96beec: sys____lwp_park60+0x68
0xbc96bfac: syscall+0x188
--- tf 0xbc96bfb0 ---
trace: pid 4666 lid 4130 at 0xbc9c3e5c
0xbc9c3e5c: mi_switch+0x10
0xbc9c3e8c: sleepq_block+0xb4
0xbc9c3eb4: cv_timedwait_sig+0x58
0xbc9c3eec: do_ksem_wait+0xe0
0xbc9c3fac: syscall+0x188
--- tf 0xbc9c3fb0 ---
trace: pid 4666 lid 4248 at 0xbc973e24
0xbc973e24: mi_switch+0x10
0xbc973e54: sleepq_block+0xb4
0xbc973ebc: lwp_park+0x134
0xbc973eec: sys____lwp_park60+0x68
0xbc973fac: syscall+0x188
--- tf 0xbc973fb0 ---
trace: pid 4666 lid 2842 at 0xbc905e24
0xbc905e24: mi_switch+0x10
0xbc905e54: sleepq_block+0xb4
0xbc905ebc: lwp_park+0x134
0xbc905eec: sys____lwp_park60+0x68
0xbc905fac: syscall+0x188
--- tf 0xbc905fb0 ---
trace: pid 4666 lid 3677 at 0xbc8e7e24
0xbc8e7e24: mi_switch+0x10
0xbc8e7e54: sleepq_block+0xb4
0xbc8e7ebc: lwp_park+0x134
0xbc8e7eec: sys____lwp_park60+0x68
0xbc8e7fac: syscall+0x188
--- tf 0xbc8e7fb0 ---
trace: pid 4666 lid 3582 at 0xbc901e24
0xbc901e24: mi_switch+0x10
0xbc901e54: sleepq_block+0xb4
0xbc901ebc: lwp_park+0x134
0xbc901eec: sys____lwp_park60+0x68
0xbc901fac: syscall+0x188
--- tf 0xbc901fb0 ---
trace: pid 4666 lid 5848 at 0xbc8ffc5c
0xbc8ffc5c: mi_switch+0x10
0xbc8ffc8c: sleepq_block+0xb4
0xbc8ffcb4: cv_timedwait_sig+0x58
0xbc8ffec4: kevent1+0xcfc
0xbc8ffeec: sys___kevent50+0x3c
0xbc8fffac: syscall+0x188
--- tf 0xbc8fffb0 ---
trace: pid 4666 lid 4666 at 0xbc8fdc5c
0xbc8fdc5c: mi_switch+0x10
0xbc8fdc8c: sleepq_block+0x124
0xbc8fdcb4: cv_timedwait_sig+0x58
0xbc8fdec4: kevent1+0xcfc
0xbc8fdeec: sys___kevent50+0x3c
0xbc8fdfac: syscall+0x188
--- tf 0xbc8fdfb0 ---
trace: pid 2998 lid 2998 at 0xbc4fbd04
0xbc4fbd04: mi_switch+0x10
0xbc4fbd34: sleepq_block+0xb4
0xbc4fbd5c: cv_timedwait_sig+0x58
0xbc4fbd8c: sbwait+0x80
0xbc4fbe3c: sosend+0xab0
0xbc4fbe64: soo_write+0x34
0xbc4fbebc: dofilewrite+0x80
0xbc4fbeec: sys_write+0x50
0xbc4fbfac: syscall+0x188
--- tf 0xbc4fbfb0 ---
trace: pid 1717 lid 1717 at 0xbc409ca4
0xbc409ca4: mi_switch+0x10
0xbc409cd4: sleepq_block+0x124
0xbc409d84: sel_do_scan+0x5ec
0xbc409e9c: selcommon+0x124
0xbc409eec: sys___select50+0x80
0xbc409fac: syscall+0x188
--- tf 0xbc409fb0 ---
trace: pid 1716 lid 1716 at 0xbc4e7ca4
0xbc4e7ca4: mi_switch+0x10
0xbc4e7cd4: sleepq_block+0x124
0xbc4e7d84: sel_do_scan+0x5ec
0xbc4e7e9c: selcommon+0x124
0xbc4e7eec: sys___select50+0x80
0xbc4e7fac: syscall+0x188
--- tf 0xbc4e7fb0 ---
trace: pid 2197 lid 2197 at 0xbb968d9c
0xbb968d9c: mi_switch+0x10
0xbb968dcc: sleepq_block+0x124
0xbb968e2c: kpause+0x164
0xbb968ea4: nanosleep1+0xf8
0xbb968eec: sys___nanosleep50+0x54
0xbb968fac: syscall+0x188
trace: pid 1940 lid 1940 at 0xbb22fd9c
0xbb22fd9c: mi_switch+0x10
0xbb22fdcc: sleepq_block+0x124
0xbb22fe2c: kpause+0x164
0xbb22fea4: nanosleep1+0xf8
0xbb22feec: sys___nanosleep50+0x54
0xbb22ffac: syscall+0x188
trace: pid 1939 lid 1939 at 0xbb1c4bcc
0xbb1c4bcc: mi_switch+0x10
0xbb1c4bfc: sleepq_block+0xb4
0xbb1c4c24: cv_timedwait_sig+0x58
0xbb1c4c54: ttysleep+0x8c
0xbb1c4cd4: ttread+0x184
0xbb1c4d14: cdev_read+0xa4
0xbb1c4d54: cdev_read+0xa4
0xbb1c4e04: spec_read+0x368
0xbb1c4e34: VOP_READ+0x44
0xbb1c4e6c: vn_read+0x148
0xbb1c4ebc: dofileread+0x7c
0xbb1c4eec: sys_read+0x50
0xbb1c4fac: syscall+0x188
trace: pid 1677 lid 1677 at 0xbc40fd9c
0xbc40fd9c: mi_switch+0x10
0xbc40fdcc: sleepq_block+0x124
0xbc40fe2c: kpause+0x164
0xbc40fea4: nanosleep1+0xf8
0xbc40feec: sys___nanosleep50+0x54
0xbc40ffac: syscall+0x188
--- tf 0xbc40ffb0 ---
trace: pid 643 lid 643 at 0xbc355c5c
0xbc355c5c: mi_switch+0x10
0xbc355c8c: sleepq_block+0xb4
0xbc355cb4: cv_timedwait_sig+0x58
0xbc355ec4: kevent1+0xcfc
0xbc355eec: sys___kevent50+0x3c
0xbc355fac: syscall+0x188
--- tf 0xbc355fb0 ---
trace: pid 888 lid 888 at 0xbc401c5c
0xbc401c5c: mi_switch+0x10
0xbc401c8c: sleepq_block+0x124
0xbc401cb4: cv_timedwait_sig+0x58
0xbc401ec4: kevent1+0xcfc
0xbc401eec: sys___kevent50+0x3c
0xbc401fac: syscall+0x188
--- tf 0xbc401fb0 ---
trace: pid 885 lid 885 at 0xbc407c5c
0xbc407c5c: mi_switch+0x10
0xbc407c8c: sleepq_block+0x124
0xbc407cb4: cv_timedwait_sig+0x58
0xbc407ec4: kevent1+0xcfc
0xbc407eec: sys___kevent50+0x3c
0xbc407fac: syscall+0x188
--- tf 0xbc407fb0 ---
trace: pid 1524 lid 791 at 0xbc4e2c5c
0xbc4e2c5c: mi_switch+0x10
0xbc4e2c8c: sleepq_block+0xb4
0xbc4e2cb4: cv_timedwait_sig+0x58
0xbc4e2ec4: kevent1+0xcfc
0xbc4e2eec: sys___kevent50+0x3c
0xbc4e2fac: syscall+0x188
trace: pid 1524 lid 790 at 0xbc4e0e24
0xbc4e0e24: mi_switch+0x10
0xbc4e0e54: sleepq_block+0xb4
0xbc4e0ebc: lwp_park+0x134
0xbc4e0eec: sys____lwp_park60+0x68
0xbc4e0fac: syscall+0x188
--- tf 0xbc4e0fb0 ---
trace: pid 1524 lid 789 at 0xbc4dee24
0xbc4dee24: mi_switch+0x10
0xbc4dee54: sleepq_block+0xb4
0xbc4deebc: lwp_park+0x134
0xbc4deeec: sys____lwp_park60+0x68
0xbc4defac: syscall+0x188
--- tf 0xbc4defb0 ---
trace: pid 1524 lid 788 at 0xbc4dce24
0xbc4dce24: mi_switch+0x10
0xbc4dce54: sleepq_block+0xb4
0xbc4dcebc: lwp_park+0x134
0xbc4dceec: sys____lwp_park60+0x68
0xbc4dcfac: syscall+0x188
--- tf 0xbc4dcfb0 ---
trace: pid 1524 lid 787 at 0xbc4dae24
0xbc4dae24: mi_switch+0x10
0xbc4dae54: sleepq_block+0xb4
0xbc4daebc: lwp_park+0x134
0xbc4daeec: sys____lwp_park60+0x68
0xbc4dafac: syscall+0x188
--- tf 0xbc4dafb0 ---
trace: pid 1524 lid 786 at 0xbc4d8e24
0xbc4d8e24: mi_switch+0x10
0xbc4d8e54: sleepq_block+0xb4
0xbc4d8ebc: lwp_park+0x134
0xbc4d8eec: sys____lwp_park60+0x68
0xbc4d8fac: syscall+0x188
--- tf 0xbc4d8fb0 ---
trace: pid 1524 lid 785 at 0xbc4d6e24
0xbc4d6e24: mi_switch+0x10
0xbc4d6e54: sleepq_block+0xb4
0xbc4d6ebc: lwp_park+0x134
0xbc4d6eec: sys____lwp_park60+0x68
0xbc4d6fac: syscall+0x188
--- tf 0xbc4d6fb0 ---
trace: pid 1524 lid 784 at 0xbc4d4e24
0xbc4d4e24: mi_switch+0x10
0xbc4d4e54: sleepq_block+0xb4
0xbc4d4ebc: lwp_park+0x134
0xbc4d4eec: sys____lwp_park60+0x68
0xbc4d4fac: syscall+0x188
--- tf 0xbc4d4fb0 ---
trace: pid 1524 lid 783 at 0xbc4d2e24
0xbc4d2e24: mi_switch+0x10
0xbc4d2e54: sleepq_block+0xb4
0xbc4d2ebc: lwp_park+0x134
0xbc4d2eec: sys____lwp_park60+0x68
0xbc4d2fac: syscall+0x188
--- tf 0xbc4d2fb0 ---
trace: pid 1524 lid 782 at 0xbc4d0e24
0xbc4d0e24: mi_switch+0x10
0xbc4d0e54: sleepq_block+0xb4
0xbc4d0ebc: lwp_park+0x134
0xbc4d0eec: sys____lwp_park60+0x68
0xbc4d0fac: syscall+0x188
--- tf 0xbc4d0fb0 ---
trace: pid 1524 lid 781 at 0xbc4cee24
0xbc4cee24: mi_switch+0x10
0xbc4cee54: sleepq_block+0xb4
0xbc4ceebc: lwp_park+0x134
0xbc4ceeec: sys____lwp_park60+0x68
0xbc4cefac: syscall+0x188
--- tf 0xbc4cefb0 ---
trace: pid 1524 lid 780 at 0xbc4cce24
0xbc4cce24: mi_switch+0x10
0xbc4cce54: sleepq_block+0xb4
0xbc4ccebc: lwp_park+0x134
0xbc4cceec: sys____lwp_park60+0x68
0xbc4ccfac: syscall+0x188
--- tf 0xbc4ccfb0 ---
trace: pid 1524 lid 779 at 0xbc4cae24
0xbc4cae24: mi_switch+0x10
0xbc4cae54: sleepq_block+0xb4
0xbc4caebc: lwp_park+0x134
0xbc4caeec: sys____lwp_park60+0x68
0xbc4cafac: syscall+0x188
--- tf 0xbc4cafb0 ---
trace: pid 1524 lid 778 at 0xbc4c8e24
0xbc4c8e24: mi_switch+0x10
0xbc4c8e54: sleepq_block+0xb4
0xbc4c8ebc: lwp_park+0x134
0xbc4c8eec: sys____lwp_park60+0x68
0xbc4c8fac: syscall+0x188
--- tf 0xbc4c8fb0 ---
trace: pid 1524 lid 777 at 0xbc4c6e24
0xbc4c6e24: mi_switch+0x10
0xbc4c6e54: sleepq_block+0xb4
0xbc4c6ebc: lwp_park+0x134
0xbc4c6eec: sys____lwp_park60+0x68
0xbc4c6fac: syscall+0x188
--- tf 0xbc4c6fb0 ---
trace: pid 1524 lid 776 at 0xbc4c4e24
0xbc4c4e24: mi_switch+0x10
0xbc4c4e54: sleepq_block+0xb4
0xbc4c4ebc: lwp_park+0x134
0xbc4c4eec: sys____lwp_park60+0x68
0xbc4c4fac: syscall+0x188
--- tf 0xbc4c4fb0 ---
trace: pid 1524 lid 775 at 0xbc4c2e24
0xbc4c2e24: mi_switch+0x10
0xbc4c2e54: sleepq_block+0xb4
0xbc4c2ebc: lwp_park+0x134
0xbc4c2eec: sys____lwp_park60+0x68
0xbc4c2fac: syscall+0x188
--- tf 0xbc4c2fb0 ---
trace: pid 1524 lid 774 at 0xbc4c0e24
0xbc4c0e24: mi_switch+0x10
0xbc4c0e54: sleepq_block+0xb4
0xbc4c0ebc: lwp_park+0x134
0xbc4c0eec: sys____lwp_park60+0x68
0xbc4c0fac: syscall+0x188
--- tf 0xbc4c0fb0 ---
trace: pid 1524 lid 773 at 0xbc4bee24
0xbc4bee24: mi_switch+0x10
0xbc4bee54: sleepq_block+0xb4
0xbc4beebc: lwp_park+0x134
0xbc4beeec: sys____lwp_park60+0x68
0xbc4befac: syscall+0x188
--- tf 0xbc4befb0 ---
trace: pid 1524 lid 772 at 0xbc4bce24
0xbc4bce24: mi_switch+0x10
0xbc4bce54: sleepq_block+0xb4
0xbc4bcebc: lwp_park+0x134
0xbc4bceec: sys____lwp_park60+0x68
0xbc4bcfac: syscall+0x188
--- tf 0xbc4bcfb0 ---
trace: pid 1524 lid 771 at 0xbc4bae24
0xbc4bae24: mi_switch+0x10
0xbc4bae54: sleepq_block+0xb4
0xbc4baebc: lwp_park+0x134
0xbc4baeec: sys____lwp_park60+0x68
0xbc4bafac: syscall+0x188
--- tf 0xbc4bafb0 ---
trace: pid 1524 lid 770 at 0xbc4b8e24
0xbc4b8e24: mi_switch+0x10
0xbc4b8e54: sleepq_block+0xb4
0xbc4b8ebc: lwp_park+0x134
0xbc4b8eec: sys____lwp_park60+0x68
0xbc4b8fac: syscall+0x188
--- tf 0xbc4b8fb0 ---
trace: pid 1524 lid 769 at 0xbc4b6e24
0xbc4b6e24: mi_switch+0x10
0xbc4b6e54: sleepq_block+0xb4
0xbc4b6ebc: lwp_park+0x134
0xbc4b6eec: sys____lwp_park60+0x68
0xbc4b6fac: syscall+0x188
--- tf 0xbc4b6fb0 ---
trace: pid 1524 lid 768 at 0xbc4b4e24
0xbc4b4e24: mi_switch+0x10
0xbc4b4e54: sleepq_block+0xb4
0xbc4b4ebc: lwp_park+0x134
0xbc4b4eec: sys____lwp_park60+0x68
0xbc4b4fac: syscall+0x188
--- tf 0xbc4b4fb0 ---
trace: pid 1524 lid 1826 at 0xbc3ffe24
0xbc3ffe24: mi_switch+0x10
0xbc3ffe54: sleepq_block+0xb4
0xbc3ffebc: lwp_park+0x134
0xbc3ffeec: sys____lwp_park60+0x68
0xbc3fffac: syscall+0x188
--- tf 0xbc3fffb0 ---
trace: pid 1524 lid 1948 at 0xbc405e24
0xbc405e24: mi_switch+0x10
0xbc405e54: sleepq_block+0xb4
0xbc405ebc: lwp_park+0x134
0xbc405eec: sys____lwp_park60+0x68
0xbc405fac: syscall+0x188
--- tf 0xbc405fb0 ---
trace: pid 1524 lid 1837 at 0xbc4b2e6c
0xbc4b2e6c: mi_switch+0x10
0xbc4b2edc: lwp_exit+0x69c
0xbc4b2eec: sys__lwp_exit+0x10
0xbc4b2fac: syscall+0x188
--- tf 0xbc4b2fb0 ---
trace: pid 1524 lid 1524 at 0xbc403dc4
0xbc403dc4: mi_switch+0x10
0xbc403df4: sleepq_block+0xb4
0xbc403e1c: cv_wait_sig+0x54
0xbc403e6c: pipe_read+0x32c
0xbc403ebc: dofileread+0x7c
0xbc403eec: sys_read+0x50
0xbc403fac: syscall+0x188
--- tf 0xbc403fb0 ---
trace: pid 1435 lid 1435 at 0xbc2b1c7c
0xbc2b1c7c: mi_switch+0x10
0xbc2b1cac: sleepq_block+0xb4
0xbc2b1d5c: sel_do_scan+0x5ec
0xbc2b1ea4: pollcommon+0xa8
0xbc2b1eec: sys___pollts50+0x7c
0xbc2b1fac: syscall+0x188
--- tf 0xbc2b1fb0 ---
trace: pid 1175 lid 1276 at 0xbc3ddc5c
0xbc3ddc5c: mi_switch+0x10
0xbc3ddc8c: sleepq_block+0xb4
0xbc3ddcb4: cv_timedwait_sig+0x58
0xbc3ddec4: kevent1+0xcfc
0xbc3ddeec: sys___kevent50+0x3c
0xbc3ddfac: syscall+0x188
trace: pid 1175 lid 3583 at 0xbc3dbe24
0xbc3dbe24: mi_switch+0x10
0xbc3dbe54: sleepq_block+0xb4
0xbc3dbebc: lwp_park+0x134
0xbc3dbeec: sys____lwp_park60+0x68
0xbc3dbfac: syscall+0x188
--- tf 0xbc3dbfb0 ---
trace: pid 1175 lid 1272 at 0xbc3d9e24
0xbc3d9e24: mi_switch+0x10
0xbc3d9e54: sleepq_block+0xb4
0xbc3d9ebc: lwp_park+0x134
0xbc3d9eec: sys____lwp_park60+0x68
0xbc3d9fac: syscall+0x188
--- tf 0xbc3d9fb0 ---
trace: pid 1175 lid 1395 at 0xbc3d7e24
0xbc3d7e24: mi_switch+0x10
0xbc3d7e54: sleepq_block+0xb4
0xbc3d7ebc: lwp_park+0x134
0xbc3d7eec: sys____lwp_park60+0x68
0xbc3d7fac: syscall+0x188
--- tf 0xbc3d7fb0 ---
trace: pid 1175 lid 1274 at 0xbc3d5e24
0xbc3d5e24: mi_switch+0x10
0xbc3d5e54: sleepq_block+0xb4
0xbc3d5ebc: lwp_park+0x134
0xbc3d5eec: sys____lwp_park60+0x68
0xbc3d5fac: syscall+0x188
--- tf 0xbc3d5fb0 ---
trace: pid 1175 lid 1273 at 0xbc3d1e24
0xbc3d1e24: mi_switch+0x10
0xbc3d1e54: sleepq_block+0xb4
0xbc3d1ebc: lwp_park+0x134
0xbc3d1eec: sys____lwp_park60+0x68
0xbc3d1fac: syscall+0x188
--- tf 0xbc3d1fb0 ---
trace: pid 1175 lid 3568 at 0xbc3cfe24
0xbc3cfe24: mi_switch+0x10
0xbc3cfe54: sleepq_block+0xb4
0xbc3cfebc: lwp_park+0x134
0xbc3cfeec: sys____lwp_park60+0x68
0xbc3cffac: syscall+0x188
--- tf 0xbc3cffb0 ---
trace: pid 1175 lid 1108 at 0xbc3cde24
0xbc3cde24: mi_switch+0x10
0xbc3cde54: sleepq_block+0xb4
0xbc3cdebc: lwp_park+0x134
0xbc3cdeec: sys____lwp_park60+0x68
0xbc3cdfac: syscall+0x188
--- tf 0xbc3cdfb0 ---
trace: pid 1175 lid 1271 at 0xbc3cbe24
0xbc3cbe24: mi_switch+0x10
0xbc3cbe54: sleepq_block+0xb4
0xbc3cbebc: lwp_park+0x134
0xbc3cbeec: sys____lwp_park60+0x68
0xbc3cbfac: syscall+0x188
--- tf 0xbc3cbfb0 ---
trace: pid 1175 lid 1394 at 0xbc3c9e24
0xbc3c9e24: mi_switch+0x10
0xbc3c9e54: sleepq_block+0xb4
0xbc3c9ebc: lwp_park+0x134
0xbc3c9eec: sys____lwp_park60+0x68
0xbc3c9fac: syscall+0x188
--- tf 0xbc3c9fb0 ---
trace: pid 1175 lid 2265 at 0xbc3c7e24
0xbc3c7e24: mi_switch+0x10
0xbc3c7e54: sleepq_block+0xb4
0xbc3c7ebc: lwp_park+0x134
0xbc3c7eec: sys____lwp_park60+0x68
0xbc3c7fac: syscall+0x188
--- tf 0xbc3c7fb0 ---
trace: pid 1175 lid 1253 at 0xbc3b1e24
0xbc3b1e24: mi_switch+0x10
0xbc3b1e54: sleepq_block+0xb4
0xbc3b1ebc: lwp_park+0x134
0xbc3b1eec: sys____lwp_park60+0x68
0xbc3b1fac: syscall+0x188
--- tf 0xbc3b1fb0 ---
trace: pid 1175 lid 1248 at 0xbc3abe24
0xbc3abe24: mi_switch+0x10
0xbc3abe54: sleepq_block+0xb4
0xbc3abebc: lwp_park+0x134
0xbc3abeec: sys____lwp_park60+0x68
0xbc3abfac: syscall+0x188
--- tf 0xbc3abfb0 ---
trace: pid 1175 lid 3001 at 0xbc399e24
0xbc399e24: mi_switch+0x10
0xbc399e54: sleepq_block+0xb4
0xbc399ebc: lwp_park+0x134
0xbc399eec: sys____lwp_park60+0x68
0xbc399fac: syscall+0x188
--- tf 0xbc399fb0 ---
trace: pid 1175 lid 1103 at 0xbc397e24
0xbc397e24: mi_switch+0x10
0xbc397e54: sleepq_block+0xb4
0xbc397ebc: lwp_park+0x134
0xbc397eec: sys____lwp_park60+0x68
0xbc397fac: syscall+0x188
--- tf 0xbc397fb0 ---
trace: pid 1175 lid 1110 at 0xbc395e24
0xbc395e24: mi_switch+0x10
0xbc395e54: sleepq_block+0xb4
0xbc395ebc: lwp_park+0x134
0xbc395eec: sys____lwp_park60+0x68
0xbc395fac: syscall+0x188
--- tf 0xbc395fb0 ---
trace: pid 1175 lid 1071 at 0xbc393e24
0xbc393e24: mi_switch+0x10
0xbc393e54: sleepq_block+0xb4
0xbc393ebc: lwp_park+0x134
0xbc393eec: sys____lwp_park60+0x68
0xbc393fac: syscall+0x188
--- tf 0xbc393fb0 ---
trace: pid 1175 lid 1854 at 0xbc391e24
0xbc391e24: mi_switch+0x10
0xbc391e54: sleepq_block+0xb4
0xbc391ebc: lwp_park+0x134
0xbc391eec: sys____lwp_park60+0x68
0xbc391fac: syscall+0x188
--- tf 0xbc391fb0 ---
trace: pid 1175 lid 3278 at 0xbc38fe24
0xbc38fe24: mi_switch+0x10
0xbc38fe54: sleepq_block+0xb4
0xbc38febc: lwp_park+0x134
0xbc38feec: sys____lwp_park60+0x68
0xbc38ffac: syscall+0x188
--- tf 0xbc38ffb0 ---
trace: pid 1175 lid 1106 at 0xbc38de24
0xbc38de24: mi_switch+0x10
0xbc38de54: sleepq_block+0xb4
0xbc38debc: lwp_park+0x134
0xbc38deec: sys____lwp_park60+0x68
0xbc38dfac: syscall+0x188
--- tf 0xbc38dfb0 ---
trace: pid 1175 lid 1095 at 0xbc38be24
0xbc38be24: mi_switch+0x10
0xbc38be54: sleepq_block+0xb4
0xbc38bebc: lwp_park+0x134
0xbc38beec: sys____lwp_park60+0x68
0xbc38bfac: syscall+0x188
--- tf 0xbc38bfb0 ---
trace: pid 1175 lid 1104 at 0xbc389e24
0xbc389e24: mi_switch+0x10
0xbc389e54: sleepq_block+0xb4
0xbc389ebc: lwp_park+0x134
0xbc389eec: sys____lwp_park60+0x68
0xbc389fac: syscall+0x188
--- tf 0xbc389fb0 ---
trace: pid 1175 lid 849 at 0xbc37be24
0xbc37be24: mi_switch+0x10
0xbc37be54: sleepq_block+0xb4
0xbc37bebc: lwp_park+0x134
0xbc37beec: sys____lwp_park60+0x68
0xbc37bfac: syscall+0x188
--- tf 0xbc37bfb0 ---
trace: pid 1175 lid 1093 at 0xbc379e24
0xbc379e24: mi_switch+0x10
0xbc379e54: sleepq_block+0xb4
0xbc379ebc: lwp_park+0x134
0xbc379eec: sys____lwp_park60+0x68
0xbc379fac: syscall+0x188
--- tf 0xbc379fb0 ---
trace: pid 1175 lid 1066 at 0xbc377e24
0xbc377e24: mi_switch+0x10
0xbc377e54: sleepq_block+0xb4
0xbc377ebc: lwp_park+0x134
0xbc377eec: sys____lwp_park60+0x68
0xbc377fac: syscall+0x188
--- tf 0xbc377fb0 ---
trace: pid 1175 lid 1215 at 0xbc375e24
0xbc375e24: mi_switch+0x10
0xbc375e54: sleepq_block+0xb4
0xbc375ebc: lwp_park+0x134
0xbc375eec: sys____lwp_park60+0x68
0xbc375fac: syscall+0x188
--- tf 0xbc375fb0 ---
trace: pid 1175 lid 3126 at 0xbb9fbe6c
0xbb9fbe6c: mi_switch+0x10
0xbb9fbedc: lwp_exit+0x69c
0xbb9fbeec: sys__lwp_exit+0x10
0xbb9fbfac: syscall+0x188
--- tf 0xbb9fbfb0 ---
trace: pid 1175 lid 1175 at 0xbc35ddc4
0xbc35ddc4: mi_switch+0x10
0xbc35ddf4: sleepq_block+0xb4
0xbc35de1c: cv_wait_sig+0x54
0xbc35de6c: pipe_read+0x32c
0xbc35debc: dofileread+0x7c
0xbc35deec: sys_read+0x50
0xbc35dfac: syscall+0x188
--- tf 0xbc35dfb0 ---
trace: pid 3500 lid 1421 at 0xbc3fdc5c
0xbc3fdc5c: mi_switch+0x10
0xbc3fdc8c: sleepq_block+0xb4
0xbc3fdcb4: cv_timedwait_sig+0x58
0xbc3fdec4: kevent1+0xcfc
0xbc3fdeec: sys___kevent50+0x3c
0xbc3fdfac: syscall+0x188
trace: pid 3500 lid 922 at 0xbc3fbe24
0xbc3fbe24: mi_switch+0x10
0xbc3fbe54: sleepq_block+0xb4
0xbc3fbebc: lwp_park+0x134
0xbc3fbeec: sys____lwp_park60+0x68
0xbc3fbfac: syscall+0x188
--- tf 0xbc3fbfb0 ---
trace: pid 3500 lid 1418 at 0xbc3f9e24
0xbc3f9e24: mi_switch+0x10
0xbc3f9e54: sleepq_block+0xb4
0xbc3f9ebc: lwp_park+0x134
0xbc3f9eec: sys____lwp_park60+0x68
0xbc3f9fac: syscall+0x188
--- tf 0xbc3f9fb0 ---
trace: pid 3500 lid 1417 at 0xbc3f7e24
0xbc3f7e24: mi_switch+0x10
0xbc3f7e54: sleepq_block+0xb4
0xbc3f7ebc: lwp_park+0x134
0xbc3f7eec: sys____lwp_park60+0x68
0xbc3f7fac: syscall+0x188
--- tf 0xbc3f7fb0 ---
trace: pid 3500 lid 1416 at 0xbc3f5e24
0xbc3f5e24: mi_switch+0x10
0xbc3f5e54: sleepq_block+0xb4
0xbc3f5ebc: lwp_park+0x134
0xbc3f5eec: sys____lwp_park60+0x68
0xbc3f5fac: syscall+0x188
--- tf 0xbc3f5fb0 ---
trace: pid 3500 lid 3163 at 0xbc3f3e24
0xbc3f3e24: mi_switch+0x10
0xbc3f3e54: sleepq_block+0xb4
0xbc3f3ebc: lwp_park+0x134
0xbc3f3eec: sys____lwp_park60+0x68
0xbc3f3fac: syscall+0x188
--- tf 0xbc3f3fb0 ---
trace: pid 3500 lid 1415 at 0xbc3f1e24
0xbc3f1e24: mi_switch+0x10
0xbc3f1e54: sleepq_block+0xb4
0xbc3f1ebc: lwp_park+0x134
0xbc3f1eec: sys____lwp_park60+0x68
0xbc3f1fac: syscall+0x188
--- tf 0xbc3f1fb0 ---
trace: pid 3500 lid 1098 at 0xbc3efe24
0xbc3efe24: mi_switch+0x10
0xbc3efe54: sleepq_block+0xb4
0xbc3efebc: lwp_park+0x134
0xbc3efeec: sys____lwp_park60+0x68
0xbc3effac: syscall+0x188
--- tf 0xbc3effb0 ---
trace: pid 3500 lid 3511 at 0xbc3ede24
0xbc3ede24: mi_switch+0x10
0xbc3ede54: sleepq_block+0xb4
0xbc3edebc: lwp_park+0x134
0xbc3edeec: sys____lwp_park60+0x68
0xbc3edfac: syscall+0x188
--- tf 0xbc3edfb0 ---
trace: pid 3500 lid 2652 at 0xbc3ebe24
0xbc3ebe24: mi_switch+0x10
0xbc3ebe54: sleepq_block+0xb4
0xbc3ebebc: lwp_park+0x134
0xbc3ebeec: sys____lwp_park60+0x68
0xbc3ebfac: syscall+0x188
--- tf 0xbc3ebfb0 ---
trace: pid 3500 lid 1517 at 0xbc3c5e24
0xbc3c5e24: mi_switch+0x10
0xbc3c5e54: sleepq_block+0xb4
0xbc3c5ebc: lwp_park+0x134
0xbc3c5eec: sys____lwp_park60+0x68
0xbc3c5fac: syscall+0x188
--- tf 0xbc3c5fb0 ---
trace: pid 3500 lid 1250 at 0xbc3afe24
0xbc3afe24: mi_switch+0x10
0xbc3afe54: sleepq_block+0xb4
0xbc3afebc: lwp_park+0x134
0xbc3afeec: sys____lwp_park60+0x68
0xbc3affac: syscall+0x188
--- tf 0xbc3affb0 ---
trace: pid 3500 lid 1118 at 0xbc3a7e24
0xbc3a7e24: mi_switch+0x10
0xbc3a7e54: sleepq_block+0xb4
0xbc3a7ebc: lwp_park+0x134
0xbc3a7eec: sys____lwp_park60+0x68
0xbc3a7fac: syscall+0x188
--- tf 0xbc3a7fb0 ---
trace: pid 3500 lid 2253 at 0xbc387e24
0xbc387e24: mi_switch+0x10
0xbc387e54: sleepq_block+0xb4
0xbc387ebc: lwp_park+0x134
0xbc387eec: sys____lwp_park60+0x68
0xbc387fac: syscall+0x188
--- tf 0xbc387fb0 ---
trace: pid 3500 lid 1091 at 0xbc385e24
0xbc385e24: mi_switch+0x10
0xbc385e54: sleepq_block+0xb4
0xbc385ebc: lwp_park+0x134
0xbc385eec: sys____lwp_park60+0x68
0xbc385fac: syscall+0x188
--- tf 0xbc385fb0 ---
trace: pid 3500 lid 2251 at 0xbc383e24
0xbc383e24: mi_switch+0x10
0xbc383e54: sleepq_block+0xb4
0xbc383ebc: lwp_park+0x134
0xbc383eec: sys____lwp_park60+0x68
0xbc383fac: syscall+0x188
--- tf 0xbc383fb0 ---
trace: pid 3500 lid 1073 at 0xbc381e24
0xbc381e24: mi_switch+0x10
0xbc381e54: sleepq_block+0xb4
0xbc381ebc: lwp_park+0x134
0xbc381eec: sys____lwp_park60+0x68
0xbc381fac: syscall+0x188
--- tf 0xbc381fb0 ---
trace: pid 3500 lid 1092 at 0xbc37fe24
0xbc37fe24: mi_switch+0x10
0xbc37fe54: sleepq_block+0xb4
0xbc37febc: lwp_park+0x134
0xbc37feec: sys____lwp_park60+0x68
0xbc37ffac: syscall+0x188
--- tf 0xbc37ffb0 ---
trace: pid 3500 lid 1090 at 0xbc37de24
0xbc37de24: mi_switch+0x10
0xbc37de54: sleepq_block+0xb4
0xbc37debc: lwp_park+0x134
0xbc37deec: sys____lwp_park60+0x68
0xbc37dfac: syscall+0x188
--- tf 0xbc37dfb0 ---
trace: pid 3500 lid 1980 at 0xbb299e24
0xbb299e24: mi_switch+0x10
0xbb299e54: sleepq_block+0xb4
0xbb299ebc: lwp_park+0x134
0xbb299eec: sys____lwp_park60+0x68
0xbb299fac: syscall+0x188
--- tf 0xbb299fb0 ---
trace: pid 3500 lid 3129 at 0xbb4e5e24
0xbb4e5e24: mi_switch+0x10
0xbb4e5e54: sleepq_block+0xb4
0xbb4e5ebc: lwp_park+0x134
0xbb4e5eec: sys____lwp_park60+0x68
0xbb4e5fac: syscall+0x188
--- tf 0xbb4e5fb0 ---
trace: pid 3500 lid 1088 at 0xbb581e24
0xbb581e24: mi_switch+0x10
0xbb581e54: sleepq_block+0xb4
0xbb581ebc: lwp_park+0x134
0xbb581eec: sys____lwp_park60+0x68
0xbb581fac: syscall+0x188
--- tf 0xbb581fb0 ---
trace: pid 3500 lid 1848 at 0xbb42ee24
0xbb42ee24: mi_switch+0x10
0xbb42ee54: sleepq_block+0xb4
0xbb42eebc: lwp_park+0x134
0xbb42eeec: sys____lwp_park60+0x68
0xbb42efac: syscall+0x188
--- tf 0xbb42efb0 ---
trace: pid 3500 lid 1171 at 0xbc2b3e24
0xbc2b3e24: mi_switch+0x10
0xbc2b3e54: sleepq_block+0xb4
0xbc2b3ebc: lwp_park+0x134
0xbc2b3eec: sys____lwp_park60+0x68
0xbc2b3fac: syscall+0x188
--- tf 0xbc2b3fb0 ---
trace: pid 3500 lid 1461 at 0xbc098e24
0xbc098e24: mi_switch+0x10
0xbc098e54: sleepq_block+0xb4
0xbc098ebc: lwp_park+0x134
0xbc098eec: sys____lwp_park60+0x68
0xbc098fac: syscall+0x188
--- tf 0xbc098fb0 ---
trace: pid 3500 lid 1014 at 0xbb96ae24
0xbb96ae24: mi_switch+0x10
0xbb96ae54: sleepq_block+0xb4
0xbb96aebc: lwp_park+0x134
0xbb96aeec: sys____lwp_park60+0x68
0xbb96afac: syscall+0x188
--- tf 0xbb96afb0 ---
trace: pid 3500 lid 1167 at 0xbbc99e6c
0xbbc99e6c: mi_switch+0x10
0xbbc99edc: lwp_exit+0x69c
0xbbc99eec: sys__lwp_exit+0x10
0xbbc99fac: syscall+0x188
--- tf 0xbbc99fb0 ---
trace: pid 3500 lid 3500 at 0xbc35bdc4
0xbc35bdc4: mi_switch+0x10
0xbc35bdf4: sleepq_block+0xb4
0xbc35be1c: cv_wait_sig+0x54
0xbc35be6c: pipe_read+0x32c
0xbc35bebc: dofileread+0x7c
0xbc35beec: sys_read+0x50
0xbc35bfac: syscall+0x188
--- tf 0xbc35bfb0 ---
trace: pid 1019 lid 3440 at 0xbc3e9c5c
0xbc3e9c5c: mi_switch+0x10
0xbc3e9c8c: sleepq_block+0xb4
0xbc3e9cb4: cv_timedwait_sig+0x58
0xbc3e9ec4: kevent1+0xcfc
0xbc3e9eec: sys___kevent50+0x3c
0xbc3e9fac: syscall+0x188
trace: pid 1019 lid 1096 at 0xbc3e7e24
0xbc3e7e24: mi_switch+0x10
0xbc3e7e54: sleepq_block+0xb4
0xbc3e7ebc: lwp_park+0x134
0xbc3e7eec: sys____lwp_park60+0x68
0xbc3e7fac: syscall+0x188
--- tf 0xbc3e7fb0 ---
trace: pid 1019 lid 1277 at 0xbc3e5e24
0xbc3e5e24: mi_switch+0x10
0xbc3e5e54: sleepq_block+0xb4
0xbc3e5ebc: lwp_park+0x134
0xbc3e5eec: sys____lwp_park60+0x68
0xbc3e5fac: syscall+0x188
--- tf 0xbc3e5fb0 ---
trace: pid 1019 lid 1278 at 0xbc3e3e24
0xbc3e3e24: mi_switch+0x10
0xbc3e3e54: sleepq_block+0xb4
0xbc3e3ebc: lwp_park+0x134
0xbc3e3eec: sys____lwp_park60+0x68
0xbc3e3fac: syscall+0x188
--- tf 0xbc3e3fb0 ---
trace: pid 1019 lid 2336 at 0xbc3e1e24
0xbc3e1e24: mi_switch+0x10
0xbc3e1e54: sleepq_block+0xb4
0xbc3e1ebc: lwp_park+0x134
0xbc3e1eec: sys____lwp_park60+0x68
0xbc3e1fac: syscall+0x188
--- tf 0xbc3e1fb0 ---
trace: pid 1019 lid 3455 at 0xbc3dfe24
0xbc3dfe24: mi_switch+0x10
0xbc3dfe54: sleepq_block+0xb4
0xbc3dfebc: lwp_park+0x134
0xbc3dfeec: sys____lwp_park60+0x68
0xbc3dffac: syscall+0x188
--- tf 0xbc3dffb0 ---
trace: pid 1019 lid 1269 at 0xbc3d3e24
0xbc3d3e24: mi_switch+0x10
0xbc3d3e54: sleepq_block+0xb4
0xbc3d3ebc: lwp_park+0x134
0xbc3d3eec: sys____lwp_park60+0x68
0xbc3d3fac: syscall+0x188
--- tf 0xbc3d3fb0 ---
trace: pid 1019 lid 3308 at 0xbc3c3e24
0xbc3c3e24: mi_switch+0x10
0xbc3c3e54: sleepq_block+0xb4
0xbc3c3ebc: lwp_park+0x134
0xbc3c3eec: sys____lwp_park60+0x68
0xbc3c3fac: syscall+0x188
--- tf 0xbc3c3fb0 ---
trace: pid 1019 lid 1259 at 0xbc3c1e24
0xbc3c1e24: mi_switch+0x10
0xbc3c1e54: sleepq_block+0xb4
0xbc3c1ebc: lwp_park+0x134
0xbc3c1eec: sys____lwp_park60+0x68
0xbc3c1fac: syscall+0x188
--- tf 0xbc3c1fb0 ---
trace: pid 1019 lid 2287 at 0xbc3bfe24
0xbc3bfe24: mi_switch+0x10
0xbc3bfe54: sleepq_block+0xb4
0xbc3bfebc: lwp_park+0x134
0xbc3bfeec: sys____lwp_park60+0x68
0xbc3bffac: syscall+0x188
--- tf 0xbc3bffb0 ---
trace: pid 1019 lid 1255 at 0xbc3bde24
0xbc3bde24: mi_switch+0x10
0xbc3bde54: sleepq_block+0xb4
0xbc3bdebc: lwp_park+0x134
0xbc3bdeec: sys____lwp_park60+0x68
0xbc3bdfac: syscall+0x188
--- tf 0xbc3bdfb0 ---
trace: pid 1019 lid 1252 at 0xbc3bbe24
0xbc3bbe24: mi_switch+0x10
0xbc3bbe54: sleepq_block+0xb4
0xbc3bbebc: lwp_park+0x134
0xbc3bbeec: sys____lwp_park60+0x68
0xbc3bbfac: syscall+0x188
--- tf 0xbc3bbfb0 ---
trace: pid 1019 lid 1257 at 0xbc3b9e24
0xbc3b9e24: mi_switch+0x10
0xbc3b9e54: sleepq_block+0xb4
0xbc3b9ebc: lwp_park+0x134
0xbc3b9eec: sys____lwp_park60+0x68
0xbc3b9fac: syscall+0x188
--- tf 0xbc3b9fb0 ---
trace: pid 1019 lid 3306 at 0xbc3b7e24
0xbc3b7e24: mi_switch+0x10
0xbc3b7e54: sleepq_block+0xb4
0xbc3b7ebc: lwp_park+0x134
0xbc3b7eec: sys____lwp_park60+0x68
0xbc3b7fac: syscall+0x188
--- tf 0xbc3b7fb0 ---
trace: pid 1019 lid 2875 at 0xbc3b5e24
0xbc3b5e24: mi_switch+0x10
0xbc3b5e54: sleepq_block+0xb4
0xbc3b5ebc: lwp_park+0x134
0xbc3b5eec: sys____lwp_park60+0x68
0xbc3b5fac: syscall+0x188
--- tf 0xbc3b5fb0 ---
trace: pid 1019 lid 1256 at 0xbc3b3e24
0xbc3b3e24: mi_switch+0x10
0xbc3b3e54: sleepq_block+0xb4
0xbc3b3ebc: lwp_park+0x134
0xbc3b3eec: sys____lwp_park60+0x68
0xbc3b3fac: syscall+0x188
--- tf 0xbc3b3fb0 ---
trace: pid 1019 lid 1254 at 0xbc3ade24
0xbc3ade24: mi_switch+0x10
0xbc3ade54: sleepq_block+0xb4
0xbc3adebc: lwp_park+0x134
0xbc3adeec: sys____lwp_park60+0x68
0xbc3adfac: syscall+0x188
--- tf 0xbc3adfb0 ---
trace: pid 1019 lid 1100 at 0xbc3a9e24
0xbc3a9e24: mi_switch+0x10
0xbc3a9e54: sleepq_block+0xb4
0xbc3a9ebc: lwp_park+0x134
0xbc3a9eec: sys____lwp_park60+0x68
0xbc3a9fac: syscall+0x188
--- tf 0xbc3a9fb0 ---
trace: pid 1019 lid 2266 at 0xbc3a5e24
0xbc3a5e24: mi_switch+0x10
0xbc3a5e54: sleepq_block+0xb4
0xbc3a5ebc: lwp_park+0x134
0xbc3a5eec: sys____lwp_park60+0x68
0xbc3a5fac: syscall+0x188
--- tf 0xbc3a5fb0 ---
trace: pid 1019 lid 1111 at 0xbc3a3e24
0xbc3a3e24: mi_switch+0x10
0xbc3a3e54: sleepq_block+0xb4
0xbc3a3ebc: lwp_park+0x134
0xbc3a3eec: sys____lwp_park60+0x68
0xbc3a3fac: syscall+0x188
--- tf 0xbc3a3fb0 ---
trace: pid 1019 lid 2904 at 0xbc3a1e24
0xbc3a1e24: mi_switch+0x10
0xbc3a1e54: sleepq_block+0xb4
0xbc3a1ebc: lwp_park+0x134
0xbc3a1eec: sys____lwp_park60+0x68
0xbc3a1fac: syscall+0x188
--- tf 0xbc3a1fb0 ---
trace: pid 1019 lid 2780 at 0xbc39fe24
0xbc39fe24: mi_switch+0x10
0xbc39fe54: sleepq_block+0xb4
0xbc39febc: lwp_park+0x134
0xbc39feec: sys____lwp_park60+0x68
0xbc39ffac: syscall+0x188
--- tf 0xbc39ffb0 ---
trace: pid 1019 lid 3291 at 0xbc39de24
0xbc39de24: mi_switch+0x10
0xbc39de54: sleepq_block+0xb4
0xbc39debc: lwp_park+0x134
0xbc39deec: sys____lwp_park60+0x68
0xbc39dfac: syscall+0x188
--- tf 0xbc39dfb0 ---
trace: pid 1019 lid 1152 at 0xbc39be24
0xbc39be24: mi_switch+0x10
0xbc39be54: sleepq_block+0xb4
0xbc39bebc: lwp_park+0x134
0xbc39beec: sys____lwp_park60+0x68
0xbc39bfac: syscall+0x188
--- tf 0xbc39bfb0 ---
trace: pid 1019 lid 1089 at 0xbc373e24
0xbc373e24: mi_switch+0x10
0xbc373e54: sleepq_block+0xb4
0xbc373ebc: lwp_park+0x134
0xbc373eec: sys____lwp_park60+0x68
0xbc373fac: syscall+0x188
--- tf 0xbc373fb0 ---
trace: pid 1019 lid 1065 at 0xbc371e24
0xbc371e24: mi_switch+0x10
0xbc371e54: sleepq_block+0xb4
0xbc371ebc: lwp_park+0x134
0xbc371eec: sys____lwp_park60+0x68
0xbc371fac: syscall+0x188
--- tf 0xbc371fb0 ---
trace: pid 1019 lid 3383 at 0xbbc9be6c
0xbbc9be6c: mi_switch+0x10
0xbbc9bedc: lwp_exit+0x69c
0xbbc9beec: sys__lwp_exit+0x10
0xbbc9bfac: syscall+0x188
--- tf 0xbbc9bfb0 ---
trace: pid 1019 lid 1019 at 0xbc09edc4
0xbc09edc4: mi_switch+0x10
0xbc09edf4: sleepq_block+0xb4
0xbc09ee1c: cv_wait_sig+0x54
0xbc09ee6c: pipe_read+0x32c
0xbc09eebc: dofileread+0x7c
0xbc09eeec: sys_read+0x50
0xbc09efac: syscall+0x188
--- tf 0xbc09efb0 ---
trace: pid 3107 lid 3107 at 0xbb789cec
0xbb789cec: mi_switch+0x10
0xbb789d1c: sleepq_block+0xb4
0xbb789d44: cv_timedwait_sig+0x58
0xbb789d74: sowait+0x60
0xbb789dbc: do_sys_accept+0x274
0xbb789eec: sys_accept+0x40
0xbb789fac: syscall+0x188
--- tf 0xbb789fb0 ---
trace: pid 1172 lid 1172 at 0xbc26fca4
0xbc26fca4: mi_switch+0x10
0xbc26fcd4: sleepq_block+0x124
0xbc26fd84: sel_do_scan+0x5ec
0xbc26fe9c: selcommon+0x124
0xbc26feec: sys___select50+0x80
0xbc26ffac: syscall+0x188
--- tf 0xbc26ffb0 ---
trace: pid 853 lid 993 at 0xbbc97e24
0xbbc97e24: mi_switch+0x10
0xbbc97e54: sleepq_block+0x124
0xbbc97ebc: lwp_park+0x134
0xbbc97eec: sys____lwp_park60+0x68
0xbbc97fac: syscall+0x188
--- tf 0xbbc97fb0 ---
trace: pid 853 lid 853 at 0xbb9f9c94
0xbb9f9c94: mi_switch+0x10
0xbb9f9cc4: sleepq_block+0x124
0xbb9f9d74: sel_do_scan+0x5ec
0xbb9f9ebc: pollcommon+0xa8
0xbb9f9eec: sys_poll+0x70
0xbb9f9fac: syscall+0x188
--- tf 0xbb9f9fb0 ---
trace: pid 2860 lid 2860 at 0xbb799ca4
0xbb799ca4: mi_switch+0x10
0xbb799cd4: sleepq_block+0x124
0xbb799d84: sel_do_scan+0x5ec
0xbb799e9c: selcommon+0x124
0xbb799eec: sys___select50+0x80
0xbb799fac: syscall+0x188
--- tf 0xbb799fb0 ---
trace: pid 1460 lid 1460 at 0xbb499d9c
0xbb499d9c: mi_switch+0x10
0xbb499dcc: sleepq_block+0x124
0xbb499e2c: kpause+0x164
0xbb499ea4: nanosleep1+0xf8
0xbb499eec: sys___nanosleep50+0x54
0xbb499fac: syscall+0x188
trace: pid 1853 lid 1853 at 0xbb635ca4
0xbb635ca4: mi_switch+0x10
0xbb635cd4: sleepq_block+0x124
0xbb635d84: sel_do_scan+0x5ec
0xbb635e9c: selcommon+0x124
0xbb635eec: sys___select50+0x80
0xbb635fac: syscall+0x188
--- tf 0xbb635fb0 ---
trace: pid 2987 lid 2987 at 0xbbf2dc94
0xbbf2dc94: mi_switch+0x10
0xbbf2dcc4: sleepq_block+0x124
0xbbf2dd74: sel_do_scan+0x5ec
0xbbf2debc: pollcommon+0xa8
0xbbf2deec: sys_poll+0x70
0xbbf2dfac: syscall+0x188
--- tf 0xbbf2dfb0 ---
trace: pid 2728 lid 2728 at 0xbbf2ac94
0xbbf2ac94: mi_switch+0x10
0xbbf2acc4: sleepq_block+0xb4
0xbbf2ad74: sel_do_scan+0x5ec
0xbbf2aebc: pollcommon+0xa8
0xbbf2aeec: sys_poll+0x70
0xbbf2afac: syscall+0x188
--- tf 0xbbf2afb0 ---
trace: pid 2983 lid 2983 at 0xbbf28c94
0xbbf28c94: mi_switch+0x10
0xbbf28cc4: sleepq_block+0x124
0xbbf28d74: sel_do_scan+0x5ec
0xbbf28ebc: pollcommon+0xa8
0xbbf28eec: sys_poll+0x70
0xbbf28fac: syscall+0x188
--- tf 0xbbf28fb0 ---
trace: pid 2854 lid 2854 at 0xbb9fdc94
0xbb9fdc94: mi_switch+0x10
0xbb9fdcc4: sleepq_block+0x124
0xbb9fdd74: sel_do_scan+0x5ec
0xbb9fdebc: pollcommon+0xa8
0xbb9fdeec: sys_poll+0x70
0xbb9fdfac: syscall+0x188
--- tf 0xbb9fdfb0 ---
trace: pid 805 lid 805 at 0xbbc95c94
0xbbc95c94: mi_switch+0x10
0xbbc95cc4: sleepq_block+0x124
0xbbc95d74: sel_do_scan+0x5ec
0xbbc95ebc: pollcommon+0xa8
0xbbc95eec: sys_poll+0x70
0xbbc95fac: syscall+0x188
--- tf 0xbbc95fb0 ---
trace: pid 1828 lid 1828 at 0xbb78fc94
0xbb78fc94: mi_switch+0x10
0xbb78fcc4: sleepq_block+0x124
0xbb78fd74: sel_do_scan+0x5ec
0xbb78febc: pollcommon+0xa8
0xbb78feec: sys_poll+0x70
0xbb78ffac: syscall+0x188
--- tf 0xbb78ffb0 ---
trace: pid 657 lid 657 at 0xbb7b1ca4
0xbb7b1ca4: mi_switch+0x10
0xbb7b1cd4: sleepq_block+0x124
0xbb7b1d84: sel_do_scan+0x5ec
0xbb7b1e9c: selcommon+0x124
0xbb7b1eec: sys___select50+0x80
0xbb7b1fac: syscall+0x188
--- tf 0xbb7b1fb0 ---
trace: pid 1518 lid 1518 at 0xbb639c7c
0xbb639c7c: mi_switch+0x10
0xbb639cac: sleepq_block+0x124
0xbb639d5c: sel_do_scan+0x5ec
0xbb639ea4: pollcommon+0xa8
0xbb639eec: sys___pollts50+0x7c
0xbb639fac: syscall+0x188
--- tf 0xbb639fb0 ---
trace: pid 329 lid 329 at 0xbb7bec7c
0xbb7bec7c: mi_switch+0x10
0xbb7becac: sleepq_block+0xb4
0xbb7bed5c: sel_do_scan+0x5ec
0xbb7beea4: pollcommon+0xa8
0xbb7beeec: sys___pollts50+0x7c
0xbb7befac: syscall+0x188
--- tf 0xbb7befb0 ---
trace: pid 304 lid 3410 at 0xbc9b7e6c
0xbc9b7e6c: mi_switch+0x10
0xbc9b7edc: lwp_exit+0x69c
0xbc9b7eec: sys__lwp_exit+0x10
0xbc9b7fac: syscall+0x188
--- tf 0xbc9b7fb0 ---
trace: pid 304 lid 1208 at 0xbb22de6c
0xbb22de6c: mi_switch+0x10
0xbb22dedc: lwp_exit+0x69c
0xbb22deec: sys__lwp_exit+0x10
0xbb22dfac: syscall+0x188
--- tf 0xbb22dfb0 ---
trace: pid 304 lid 2454 at 0xbb297e6c
0xbb297e6c: mi_switch+0x10
0xbb297edc: lwp_exit+0x69c
0xbb297eec: sys__lwp_exit+0x10
0xbb297fac: syscall+0x188
--- tf 0xbb297fb0 ---
trace: pid 304 lid 2482 at 0xbb9f5cec
0xbb9f5cec: mi_switch+0x10
0xbb9f5d1c: sleepq_block+0xb4
0xbb9f5d44: cv_timedwait_sig+0x58
0xbb9f5d74: sowait+0x60
0xbb9f5dbc: do_sys_accept+0x274
0xbb9f5eec: sys_accept+0x40
0xbb9f5fac: syscall+0x188
--- tf 0xbb9f5fb0 ---
trace: pid 304 lid 304 at 0xbb49fc5c
0xbb49fc5c: mi_switch+0x10
0xbb49fc8c: sleepq_block+0xb4
0xbb49fcb4: cv_timedwait_sig+0x58
0xbb49fec4: kevent1+0xcfc
0xbb49feec: sys___kevent50+0x3c
0xbb49ffac: syscall+0x188
--- tf 0xbb49ffb0 ---
trace: pid 414 lid 414 at 0xbb96ee34
0xbb96ee34: mi_switch+0x10
0xbb96ee64: sleepq_block+0xb4
0xbb96eec4: kpause+0x164
0xbb96eeec: sys___sigsuspend14+0x6c
0xbb96efac: syscall+0x188
trace: pid 385 lid 389 at 0xbb964e24
0xbb964e24: mi_switch+0x10
0xbb964e54: sleepq_block+0x124
0xbb964ebc: lwp_park+0x134
0xbb964eec: sys____lwp_park60+0x68
0xbb964fac: syscall+0x188
--- tf 0xbb964fb0 ---
trace: pid 385 lid 388 at 0xbb962c5c
0xbb962c5c: mi_switch+0x10
0xbb962c8c: sleepq_block+0xb4
0xbb962cb4: cv_timedwait_sig+0x58
0xbb962ec4: kevent1+0xcfc
0xbb962eec: sys___kevent50+0x3c
0xbb962fac: syscall+0x188
--- tf 0xbb962fb0 ---
trace: pid 385 lid 387 at 0xbb95cc5c
0xbb95cc5c: mi_switch+0x10
0xbb95cc8c: sleepq_block+0xb4
0xbb95ccb4: cv_timedwait_sig+0x58
0xbb95cec4: kevent1+0xcfc
0xbb95ceec: sys___kevent50+0x3c
0xbb95cfac: syscall+0x188
--- tf 0xbb95cfb0 ---
trace: pid 385 lid 386 at 0xbb952c5c
0xbb952c5c: mi_switch+0x10
0xbb952c8c: sleepq_block+0xb4
0xbb952cb4: cv_timedwait_sig+0x58
0xbb952ec4: kevent1+0xcfc
0xbb952eec: sys___kevent50+0x3c
0xbb952fac: syscall+0x188
--- tf 0xbb952fb0 ---
trace: pid 385 lid 385 at 0xbb795e24
0xbb795e24: mi_switch+0x10
0xbb795e54: sleepq_block+0xb4
0xbb795ebc: lwp_park+0x134
0xbb795eec: sys____lwp_park60+0x68
0xbb795fac: syscall+0x188
--- tf 0xbb795fb0 ---
trace: pid 2088 lid 2801 at 0xbb57fc7c
0xbb57fc7c: mi_switch+0x10
0xbb57fcac: sleepq_block+0xb4
0xbb57fd5c: sel_do_scan+0x5ec
0xbb57fea4: pollcommon+0xa8
0xbb57feec: sys___pollts50+0x7c
0xbb57ffac: syscall+0x188
--- tf 0xbb57ffb0 ---
trace: pid 2088 lid 339 at 0xbb797c7c
0xbb797c7c: mi_switch+0x10
0xbb797cac: sleepq_block+0xb4
0xbb797d5c: sel_do_scan+0x5ec
0xbb797ea4: pollcommon+0xa8
0xbb797eec: sys___pollts50+0x7c
0xbb797fac: syscall+0x188
--- tf 0xbb797fb0 ---
trace: pid 2088 lid 1979 at 0xbb7b8c7c
0xbb7b8c7c: mi_switch+0x10
0xbb7b8cac: sleepq_block+0xb4
0xbb7b8d5c: sel_do_scan+0x5ec
0xbb7b8ea4: pollcommon+0xa8
0xbb7b8eec: sys___pollts50+0x7c
0xbb7b8fac: syscall+0x188
--- tf 0xbb7b8fb0 ---
trace: pid 2088 lid 2211 at 0xbb351c7c
0xbb351c7c: mi_switch+0x10
0xbb351cac: sleepq_block+0xb4
0xbb351d5c: sel_do_scan+0x5ec
0xbb351ea4: pollcommon+0xa8
0xbb351eec: sys___pollts50+0x7c
0xbb351fac: syscall+0x188
--- tf 0xbb351fb0 ---
trace: pid 2088 lid 2417 at 0xbb637e24
0xbb637e24: mi_switch+0x10
0xbb637e54: sleepq_block+0xb4
0xbb637ebc: lwp_park+0x134
0xbb637eec: sys____lwp_park60+0x68
0xbb637fac: syscall+0x188
--- tf 0xbb637fb0 ---
trace: pid 2088 lid 2088 at 0xbb497c7c
0xbb497c7c: mi_switch+0x10
0xbb497cac: sleepq_block+0xb4
0xbb497d5c: sel_do_scan+0x5ec
0xbb497ea4: pollcommon+0xa8
0xbb497eec: sys___pollts50+0x7c
0xbb497fac: syscall+0x188
--- tf 0xbb497fb0 ---
trace: pid 952 lid 7985 at 0xbcacfe24
0xbcacfe24: mi_switch+0x10
0xbcacfe54: sleepq_block+0xb4
0xbcacfebc: lwp_park+0x134
0xbcacfeec: sys____lwp_park60+0x68
0xbcacffac: syscall+0x188
--- tf 0xbcacffb0 ---
trace: pid 952 lid 8001 at 0xbca87e24
0xbca87e24: mi_switch+0x10
0xbca87e54: sleepq_block+0xb4
0xbca87ebc: lwp_park+0x134
0xbca87eec: sys____lwp_park60+0x68
0xbca87fac: syscall+0x188
--- tf 0xbca87fb0 ---
trace: pid 952 lid 8254 at 0xbcabfe24
0xbcabfe24: mi_switch+0x10
0xbcabfe54: sleepq_block+0xb4
0xbcabfebc: lwp_park+0x134
0xbcabfeec: sys____lwp_park60+0x68
0xbcabffac: syscall+0x188
--- tf 0xbcabffb0 ---
trace: pid 952 lid 8004 at 0xbcacde24
0xbcacde24: mi_switch+0x10
0xbcacde54: sleepq_block+0xb4
0xbcacdebc: lwp_park+0x134
0xbcacdeec: sys____lwp_park60+0x68
0xbcacdfac: syscall+0x188
--- tf 0xbcacdfb0 ---
trace: pid 952 lid 820 at 0xbb633c5c
0xbb633c5c: mi_switch+0x10
0xbb633c8c: sleepq_block+0xb4
0xbb633cb4: cv_timedwait_sig+0x58
0xbb633ec4: kevent1+0xcfc
0xbb633eec: sys___kevent50+0x3c
0xbb633fac: syscall+0x188
--- tf 0xbb633fb0 ---
trace: pid 952 lid 1119 at 0xbb62dc5c
0xbb62dc5c: mi_switch+0x10
0xbb62dc8c: sleepq_block+0xb4
0xbb62dcb4: cv_timedwait_sig+0x58
0xbb62dec4: kevent1+0xcfc
0xbb62deec: sys___kevent50+0x3c
0xbb62dfac: syscall+0x188
--- tf 0xbb62dfb0 ---
trace: pid 952 lid 1099 at 0xbb627c5c
0xbb627c5c: mi_switch+0x10
0xbb627c8c: sleepq_block+0xb4
0xbb627cb4: cv_timedwait_sig+0x58
0xbb627ec4: kevent1+0xcfc
0xbb627eec: sys___kevent50+0x3c
0xbb627fac: syscall+0x188
--- tf 0xbb627fb0 ---
trace: pid 952 lid 1113 at 0xbb621c5c
0xbb621c5c: mi_switch+0x10
0xbb621c8c: sleepq_block+0xb4
0xbb621cb4: cv_timedwait_sig+0x58
0xbb621ec4: kevent1+0xcfc
0xbb621eec: sys___kevent50+0x3c
0xbb621fac: syscall+0x188
--- tf 0xbb621fb0 ---
trace: pid 952 lid 1114 at 0xbb58ee24
0xbb58ee24: mi_switch+0x10
0xbb58ee54: sleepq_block+0x124
0xbb58eebc: lwp_park+0x134
0xbb58eeec: sys____lwp_park60+0x68
0xbb58efac: syscall+0x188
--- tf 0xbb58efb0 ---
trace: pid 952 lid 954 at 0xbad67e24
0xbad67e24: mi_switch+0x10
0xbad67e54: sleepq_block+0xb4
0xbad67ebc: lwp_park+0x134
0xbad67eec: sys____lwp_park60+0x68
0xbad67fac: syscall+0x188
--- tf 0xbad67fb0 ---
trace: pid 952 lid 1101 at 0xbad6de24
0xbad6de24: mi_switch+0x10
0xbad6de54: sleepq_block+0xb4
0xbad6debc: lwp_park+0x134
0xbad6deec: sys____lwp_park60+0x68
0xbad6dfac: syscall+0x188
--- tf 0xbad6dfb0 ---
trace: pid 952 lid 958 at 0xbb571c5c
0xbb571c5c: mi_switch+0x10
0xbb571c8c: sleepq_block+0xb4
0xbb571cb4: cv_timedwait_sig+0x58
0xbb571ec4: kevent1+0xcfc
0xbb571eec: sys___kevent50+0x3c
0xbb571fac: syscall+0x188
--- tf 0xbb571fb0 ---
trace: pid 952 lid 1135 at 0xbad69e24
0xbad69e24: mi_switch+0x10
0xbad69e54: sleepq_block+0xb4
0xbad69ebc: lwp_park+0x134
0xbad69eec: sys____lwp_park60+0x68
0xbad69fac: syscall+0x188
--- tf 0xbad69fb0 ---
trace: pid 952 lid 952 at 0xbb56bd8c
0xbb56bd8c: mi_switch+0x10
0xbb56bdbc: sleepq_block+0xb4
0xbb56bde4: cv_timedwait_sig+0x58
0xbb56becc: sigtimedwait1+0x224
0xbb56beec: sys_____sigtimedwait50+0x30
0xbb56bfac: syscall+0x188
--- tf 0xbb56bfb0 ---
trace: pid 1134 lid 1134 at 0xbb587c5c
0xbb587c5c: mi_switch+0x10
0xbb587c8c: sleepq_block+0x124
0xbb587cb4: cv_timedwait_sig+0x58
0xbb587ec4: kevent1+0xcfc
0xbb587eec: sys___kevent50+0x3c
0xbb587fac: syscall+0x188
trace: pid 828 lid 828 at 0xbb585ca4
0xbb585ca4: mi_switch+0x10
0xbb585cd4: sleepq_block+0x124
0xbb585d84: sel_do_scan+0x5ec
0xbb585e9c: selcommon+0x124
0xbb585eec: sys___select50+0x80
0xbb585fac: syscall+0x188
--- tf 0xbb585fb0 ---
trace: pid 421 lid 421 at 0xbb42cd84
0xbb42cd84: mi_switch+0x10
0xbb42cdb4: sleepq_block+0xb4
0xbb42cddc: cv_wait_sig+0x54
0xbb42ce24: drvctl_ioctl+0x2b0
0xbb42ceec: sys_ioctl+0x3e4
0xbb42cfac: syscall+0x188
trace: pid 169 lid 169 at 0xbb29fd9c
0xbb29fd9c: mi_switch+0x10
0xbb29fdcc: sleepq_block+0x124
0xbb29fe2c: kpause+0x164
0xbb29fea4: nanosleep1+0xf8
0xbb29feec: sys___nanosleep50+0x54
0xbb29ffac: syscall+0x188
--- tf 0xbb29ffb0 ---
trace: pid 1 lid 1 at 0xbb1a3c5c
0xbb1a3c5c: mi_switch+0x10
0xbb1a3c8c: sleepq_block+0xb4
0xbb1a3cb4: cv_wait_sig+0x54
0xbb1a3d9c: do_sys_waitid+0x138
0xbb1a3e74: do_sys_wait+0xb8
0xbb1a3eec: sys___wait450+0x3c
0xbb1a3fac: syscall+0x188
--- tf 0xbb1a3fb0 ---
trace: pid 0 lid 20474 at 0xbb3c5ef4
0xbb3c5ef4: mi_switch+0x10
0xbb3c5f64: lwp_exit+0x69c
0xbb3c5f74: kthread_exit+0x5c
0xbb3c5fac: ktrace_thread+0x110
trace: pid 0 lid 1444 at 0xbb491ee4
0xbb491ee4: mi_switch+0x10
0xbb491f14: sleepq_block+0x124
0xbb491f3c: cv_timedwait+0x58
Bad frame pointer: 0x91cb8b88
trace: pid 0 lid 309 at 0xbb1c6f3c
0xbb1c6f3c: mi_switch+0x10
0xbb1c6f6c: sleepq_block+0x124
0xbb1c6f94: cv_timedwait+0x58
trace: pid 0 lid 499 at 0xbb29bf3c
0xbb29bf3c: mi_switch+0x10
0xbb29bf6c: sleepq_block+0x124
0xbb29bf94: cv_timedwait+0x58
trace: pid 0 lid 498 at 0xbb29df3c
0xbb29df3c: mi_switch+0x10
0xbb29df6c: sleepq_block+0x124
0xbb29df94: cv_timedwait+0x58
trace: pid 0 lid 365 at 0xbb353f04
0xbb353f04: mi_switch+0x10
0xbb353f34: sleepq_block+0xb4
0xbb353f5c: cv_wait+0x54
0xbb353fac: workqueue_worker+0xd4
trace: pid 0 lid 214 at 0xbb227e44
0xbb227e44: mi_switch+0x10
0xbb227e74: sleepq_block+0xb4
0xbb227e9c: cv_wait+0x54
0xbb227ec4: xc_wait+0xa0
0xbb227ee4: pool_cache_invalidate+0x16c
0xbb227f34: pool_reclaim+0xac
0xbb227f5c: pool_drain+0x84
0xbb227fac: $a+0x138
trace: pid 0 lid 213 at 0xbb225eb4
0xbb225eb4: mi_switch+0x10
0xbb225ee4: sleepq_block+0x124
0xbb225f44: kpause+0x164
0xbb225fac: sched_sync+0x250
trace: pid 0 lid 212 at 0xbb223e3c
0xbb223e3c: mi_switch+0x10
0xbb223e6c: sleepq_block+0xb4
0xbb223ed4: mtsleep+0x1b0
0xbb223fac: uvm_pageout+0x66c
trace: pid 0 lid 211 at 0xbb1ceee4
0xbb1ceee4: mi_switch+0x10
0xbb1cef14: sleepq_block+0xb4
0xbb1cef3c: cv_wait_sig+0x54
0xbb1cefac: $a+0x16c
trace: pid 0 lid 210 at 0xbb1ccee4
0xbb1ccee4: mi_switch+0x10
0xbb1ccf14: sleepq_block+0xb4
0xbb1ccf3c: cv_wait_sig+0x54
0xbb1ccfac: vcaudio_worker+0x164
trace: pid 0 lid 209 at 0xbb1caec4
0xbb1caec4: mi_switch+0x10
0xbb1caef4: sleepq_block+0xb4
0xbb1caf1c: cv_wait_sig+0x54
0xbb1caf34: down_interruptible+0x2c
0xbb1cafac: sync_func+0x2f8
trace: pid 0 lid 208 at 0xbb1c8f14
0xbb1c8f14: mi_switch+0x10
0xbb1c8f44: sleepq_block+0xb4
0xbb1c8f6c: cv_wait_sig+0x54
0xbb1c8f84: down_interruptible+0x2c
0xbb1c8fac: $a+0x60
trace: pid 0 lid 207 at 0xbb1b3e9c
0xbb1b3e9c: mi_switch+0x10
0xbb1b3ecc: sleepq_block+0xb4
0xbb1b3ef4: cv_wait_sig+0x54
0xbb1b3f0c: down_interruptible+0x2c
0xbb1b3fac: $a+0x1210
trace: pid 0 lid 204 at 0xbb1c2f04
0xbb1c2f04: mi_switch+0x10
0xbb1c2f34: sleepq_block+0xb4
0xbb1c2f5c: cv_wait+0x54
0xbb1c2fac: workqueue_worker+0xd4
trace: pid 0 lid 202 at 0xbb1b1f1c
0xbb1b1f1c: mi_switch+0x10
0xbb1b1f4c: sleepq_block+0x124
0xbb1b1f74: cv_timedwait+0x58
0xbb1b1fac: $a+0x118
trace: pid 0 lid 201 at 0xbb1a1f04
0xbb1a1f04: mi_switch+0x10
0xbb1a1f34: sleepq_block+0xb4
0xbb1a1f5c: cv_wait+0x54
0xbb1a1fac: workqueue_worker+0xd4
trace: pid 0 lid 200 at 0xbb19fee4
0xbb19fee4: mi_switch+0x10
0xbb19ff14: sleepq_block+0xb4
0xbb19ff3c: cv_wait+0x54
0xbb19ffac: unp_thread+0xaf4
trace: pid 0 lid 199 at 0xbb19dd94
0xbb19dd94: mi_switch+0x10
0xbb19ddc4: sleepq_block+0xb4
0xbb19ddec: cv_wait+0x54
0xbb19de2c: xc_broadcast+0x168
0xbb19de74: localcount_drain+0x6c
0xbb19debc: key_sa_chgstate.part.0+0x20
0xbb19df5c: $a+0xf04
0xbb19dfac: workqueue_worker+0x130
trace: pid 0 lid 198 at 0xbb19bf04
0xbb19bf04: mi_switch+0x10
0xbb19bf34: sleepq_block+0xb4
0xbb19bf5c: cv_wait+0x54
0xbb19bfac: workqueue_worker+0xd4
trace: pid 0 lid 197 at 0xbb199f04
0xbb199f04: mi_switch+0x10
0xbb199f34: sleepq_block+0xb4
0xbb199f5c: cv_wait+0x54
0xbb199fac: workqueue_worker+0xd4
trace: pid 0 lid 196 at 0xbad65f04
0xbad65f04: mi_switch+0x10
0xbad65f34: sleepq_block+0xb4
0xbad65f5c: cv_wait+0x54
0xbad65fac: workqueue_worker+0xd4
trace: pid 0 lid 195 at 0xbad6ff04
0xbad6ff04: mi_switch+0x10
0xbad6ff34: sleepq_block+0xb4
0xbad6ff5c: cv_wait+0x54
0xbad6ffac: workqueue_worker+0xd4
trace: pid 0 lid 194 at 0xba55ff24
0xba55ff24: mi_switch+0x10
0xba55ff54: sleepq_block+0x124
0xba55ff7c: cv_timedwait+0x58
0xba55ffac: usb_event_thread+0x9c
trace: pid 0 lid 193 at 0xbad73f04
0xbad73f04: mi_switch+0x10
0xbad73f34: sleepq_block+0xb4
0xbad73f5c: cv_wait+0x54
0xbad73fac: workqueue_worker+0xd4
trace: pid 0 lid 192 at 0xbb197f04
0xbb197f04: mi_switch+0x10
0xbb197f34: sleepq_block+0xb4
0xbb197f5c: cv_wait+0x54
0xbb197fac: workqueue_worker+0xd4
trace: pid 0 lid 175 at 0xbb195f04
0xbb195f04: mi_switch+0x10
0xbb195f34: sleepq_block+0xb4
0xbb195f5c: cv_wait+0x54
0xbb195fac: workqueue_worker+0xd4
trace: pid 0 lid 177 at 0xbb193f04
0xbb193f04: mi_switch+0x10
0xbb193f34: sleepq_block+0xb4
0xbb193f5c: cv_wait+0x54
0xbb193fac: workqueue_worker+0xd4
trace: pid 0 lid 176 at 0xbb191f04
0xbb191f04: mi_switch+0x10
0xbb191f34: sleepq_block+0xb4
0xbb191f5c: cv_wait+0x54
0xbb191fac: workqueue_worker+0xd4
trace: pid 0 lid 170 at 0xbb18ff04
0xbb18ff04: mi_switch+0x10
0xbb18ff34: sleepq_block+0xb4
0xbb18ff5c: cv_wait+0x54
0xbb18ffac: workqueue_worker+0xd4
trace: pid 0 lid 173 at 0xbb18df04
0xbb18df04: mi_switch+0x10
0xbb18df34: sleepq_block+0xb4
0xbb18df5c: cv_wait+0x54
0xbb18dfac: workqueue_worker+0xd4
trace: pid 0 lid 31 at 0xbb18bf04
0xbb18bf04: mi_switch+0x10
0xbb18bf34: sleepq_block+0xb4
0xbb18bf5c: cv_wait+0x54
0xbb18bfac: workqueue_worker+0xd4
trace: pid 0 lid 63 at 0xbb189f04
0xbb189f04: mi_switch+0x10
0xbb189f34: sleepq_block+0xb4
0xbb189f5c: cv_wait+0x54
0xbb189fac: workqueue_worker+0xd4
trace: pid 0 lid 126 at 0xbb187f04
0xbb187f04: mi_switch+0x10
0xbb187f34: sleepq_block+0xb4
0xbb187f5c: cv_wait+0x54
0xbb187fac: workqueue_worker+0xd4
trace: pid 0 lid 125 at 0xbb185f04
0xbb185f04: mi_switch+0x10
0xbb185f34: sleepq_block+0xb4
0xbb185f5c: cv_wait+0x54
0xbb185fac: workqueue_worker+0xd4
trace: pid 0 lid 124 at 0xbb183f04
0xbb183f04: mi_switch+0x10
0xbb183f34: sleepq_block+0xb4
0xbb183f5c: cv_wait+0x54
0xbb183fac: workqueue_worker+0xd4
trace: pid 0 lid 123 at 0xbb181f04
0xbb181f04: mi_switch+0x10
0xbb181f34: sleepq_block+0xb4
0xbb181f5c: cv_wait+0x54
0xbb181fac: workqueue_worker+0xd4
trace: pid 0 lid 122 at 0xbad71f04
0xbad71f04: mi_switch+0x10
0xbad71f34: sleepq_block+0xb4
0xbad71f5c: cv_wait+0x54
0xbad71fac: workqueue_worker+0xd4
trace: pid 0 lid 121 at 0xbad7ff04
0xbad7ff04: mi_switch+0x10
0xbad7ff34: sleepq_block+0xb4
0xbad7ff5c: cv_wait+0x54
0xbad7ffac: workqueue_worker+0xd4
trace: pid 0 lid 120 at 0xbad6bf04
0xbad6bf04: mi_switch+0x10
0xbad6bf34: sleepq_block+0xb4
0xbad6bf5c: cv_wait+0x54
0xbad6bfac: workqueue_worker+0xd4
trace: pid 0 lid 119 at 0xbad7df0c
0xbad7df0c: mi_switch+0x10
0xbad7df3c: sleepq_block+0xb4
0xbad7df64: cv_wait+0x54
0xbad7dfac: usb_task_thread+0x1d4
trace: pid 0 lid 118 at 0xbad7bf0c
0xbad7bf0c: mi_switch+0x10
0xbad7bf3c: sleepq_block+0xb4
0xbad7bf64: cv_wait+0x54
0xbad7bfac: usb_task_thread+0x1d4
trace: pid 0 lid 117 at 0xbad78efc
0xbad78efc: mi_switch+0x10
0xbad78f2c: sleepq_block+0xb4
0xbad78f54: cv_wait+0x54
0xbad78fac: linux_workqueue_thread+0x468
trace: pid 0 lid 116 at 0xbad76f14
0xbad76f14: mi_switch+0x10
0xbad76f44: sleepq_block+0xb4
0xbad76f6c: cv_wait+0x54
0xbad76fac: sdmmc_task_thread+0x120
trace: pid 0 lid 107 at 0xba561e1c
0xba561e1c: mi_switch+0x10
0xba561e4c: sleepq_block+0x124
0xba561e74: cv_timedwait+0x58
0xba561fac: entropy_thread+0x1ac
trace: pid 0 lid 105 at 0xba1d9f0c
0xba1d9f0c: mi_switch+0x10
0xba1d9f3c: sleepq_block+0xb4
0xba1d9f64: cv_wait+0x54
0xba1d9fac: xc_thread+0x90
trace: pid 0 lid 104 at 0xba1d7f0c
0xba1d7f0c: _splraise+0xc
0xba1d7fac: softint_dispatch+0x1a0
Bad frame pointer: 0x80cb7f1c
trace: pid 0 lid 103 at 0xba1d5f0c
0xba1d5f0c: _splraise+0xc
0xba1d5fac: softint_dispatch+0x1a0
Bad frame pointer: 0x80cb7f1c
trace: pid 0 lid 102 at 0x0
trace: pid 0 lid 101 at 0xba1d1f0c
0xba1d1f0c: _splraise+0xc
0xba1d1fac: softint_dispatch+0x1a0
Bad frame pointer: 0x80cb7f1c
trace: pid 0 lid 100 at 0x80cb7f1c
0x80cb7f1c: kpreempt_enable+0xc
0x80cb7f34: sched_curcpu_runnable_p+0x28
0x80cb7f94: idle_loop+0x1b8
--- tf 0x80cb7fb0 ---
trace: pid 0 lid 99 at 0xba1c7f0c
0xba1c7f0c: mi_switch+0x10
0xba1c7f3c: sleepq_block+0xb4
0xba1c7f64: cv_wait+0x54
0xba1c7fac: xc_thread+0x90
trace: pid 0 lid 98 at 0xba1c5f0c
0xba1c5f0c: _splraise+0xc
0xba1c5fac: softint_dispatch+0x1a0
Bad frame pointer: 0x80cb5f1c
trace: pid 0 lid 97 at 0xba1c3f0c
0xba1c3f0c: _splraise+0xc
0xba1c3fac: softint_dispatch+0x1a0
Bad frame pointer: 0x80cb5f1c
trace: pid 0 lid 96 at 0x0
trace: pid 0 lid 30 at 0xba1bff0c
0xba1bff0c: _splraise+0xc
0xba1bffac: softint_dispatch+0x1a0
Bad frame pointer: 0x80cb5f1c
trace: pid 0 lid 29 at 0x80cb5f1c
0x80cb5f1c: kpreempt_enable+0xc
0x80cb5f34: sched_curcpu_runnable_p+0x28
0x80cb5f94: idle_loop+0x1b8
--- tf 0x80cb5fb0 ---
trace: pid 0 lid 28 at 0xba1b5f0c
0xba1b5f0c: mi_switch+0x10
0xba1b5f3c: sleepq_block+0xb4
0xba1b5f64: cv_wait+0x54
0xba1b5fac: xc_thread+0x90
trace: pid 0 lid 27 at 0xba1b3f0c
0xba1b3f0c: _splraise+0xc
0xba1b3fac: softint_dispatch+0x1a0
0xbb5118ec: softint_switch+0x60
trace: pid 0 lid 26 at 0xba1b1f0c
0xba1b1f0c: _splraise+0xc
0xba1b1fac: softint_dispatch+0x1a0
0xbb5118bc: softint_switch+0x60
--- tf 0xba1b1fb0 ---
trace: pid 0 lid 25 at 0x0
trace: pid 0 lid 24 at 0xba1adf0c
0xba1adf0c: _splraise+0xc
0xba1adfac: softint_dispatch+0x1a0
0xbb511944: softint_switch+0x60
Bad frame pointer: 0x800b0e08
trace: pid 0 lid 23 at 0x80cb3f34
0x80cb3f34: mi_switch+0x10
0x80cb3f94: idle_loop+0x244
--- tf 0x80cb3fb0 ---
trace: pid 0 lid 22 at 0xba1a3efc
0xba1a3efc: mi_switch+0x10
0xba1a3f2c: sleepq_block+0xb4
0xba1a3f54: cv_wait+0x54
0xba1a3fac: linux_workqueue_thread+0x468
trace: pid 0 lid 21 at 0xba1a1efc
0xba1a1efc: mi_switch+0x10
0xba1a1f2c: sleepq_block+0xb4
0xba1a1f54: cv_wait+0x54
0xba1a1fac: linux_workqueue_thread+0x468
trace: pid 0 lid 20 at 0xba19fefc
0xba19fefc: mi_switch+0x10
0xba19ff2c: sleepq_block+0xb4
0xba19ff54: cv_wait+0x54
0xba19ffac: linux_workqueue_thread+0x468
trace: pid 0 lid 19 at 0xba19defc
0xba19defc: mi_switch+0x10
0xba19df2c: sleepq_block+0xb4
0xba19df54: cv_wait+0x54
0xba19dfac: linux_workqueue_thread+0x468
trace: pid 0 lid 18 at 0xba19befc
0xba19befc: mi_switch+0x10
0xba19bf2c: sleepq_block+0xb4
0xba19bf54: cv_wait+0x54
0xba19bfac: linux_workqueue_thread+0x468
trace: pid 0 lid 17 at 0xba199f14
0xba199f14: mi_switch+0x10
0xba199f44: sleepq_block+0xb4
0xba199f6c: cv_wait+0x54
0xba199fac: gc_thread+0x21c
trace: pid 0 lid 16 at 0xba197f2c
0xba197f2c: mi_switch+0x10
0xba197f5c: sleepq_block+0xb4
0xba197f84: cv_wait+0x54
0xba197fac: sysmon_task_queue_thread+0xc0
trace: pid 0 lid 15 at 0xba195f04
0xba195f04: mi_switch+0x10
0xba195f34: sleepq_block+0xb4
0xba195f5c: cv_wait+0x54
0xba195fac: workqueue_worker+0xd4
trace: pid 0 lid 14 at 0xba193f04
0xba193f04: mi_switch+0x10
0xba193f34: sleepq_block+0xb4
0xba193f5c: cv_wait+0x54
0xba193fac: workqueue_worker+0xd4
trace: pid 0 lid 13 at 0xba191ef4
0xba191ef4: mi_switch+0x10
0xba191f24: sleepq_block+0xb4
0xba191f4c: cv_wait+0x54
0xba191fac: sopendfree_thread+0x1c8
trace: pid 0 lid 12 at 0xba18ff04
0xba18ff04: mi_switch+0x10
0xba18ff34: sleepq_block+0xb4
0xba18ff5c: cv_wait+0x54
0xba18ffac: workqueue_worker+0xd4
trace: pid 0 lid 11 at 0xba18df04
0xba18df04: mi_switch+0x10
0xba18df34: sleepq_block+0xb4
0xba18df5c: cv_wait+0x54
0xba18dfac: workqueue_worker+0xd4
trace: pid 0 lid 10 at 0xba18bf04
0xba18bf04: mi_switch+0x10
0xba18bf34: sleepq_block+0xb4
0xba18bf5c: cv_wait+0x54
0xba18bfac: workqueue_worker+0xd4
trace: pid 0 lid 9 at 0xba189ef4
0xba189ef4: mi_switch+0x10
0xba189f24: sleepq_block+0xb4
0xba189f4c: cv_wait+0x54
0xba189fac: $a+0x514
trace: pid 0 lid 8 at 0xba187f1c
0xba187f1c: mi_switch+0x10
0xba187f4c: sleepq_block+0xb4
0xba187f74: cv_timedwait+0x58
0xba187fac: $a+0xf8
trace: pid 0 lid 7 at 0xba185f0c
0xba185f0c: mi_switch+0x10
0xba185f3c: sleepq_block+0xb4
0xba185f64: cv_wait+0x54
0xba185fac: xc_thread+0x90
trace: pid 0 lid 6 at 0xba183f0c
0xba183f0c: _splraise+0xc
0xba183fac: softint_dispatch+0x1a0
Bad frame pointer: 0xba181e7c
trace: pid 0 lid 5 at 0xba181e7c
0xba181e7c: ba181e88
--- tf 0xba181fb0 ---
trace: pid 0 lid 4 at 0x0
trace: pid 0 lid 3 at 0xba0fcd7c
0xba0fcd7c: fffffff7
trace: pid 0 lid 2 at 0x80cb1f4c
0x80cb1f4c: mi_switch+0x10
0x80cb1fac: idle_loop+0x244
trace: pid 0 lid 0 at 0x80cb9e54
0x80cb9e54: mi_switch+0x10
0x80cb9e84: sleepq_block+0x124
0x80cb9ee4: kpause+0x164
0x80cb9f14: uvm_scheduler+0x100
0x80cb9f9c: main+0x8c0
0x80cb9fac: kernel_text+0x54
top -t
load averages: 1.60, 1.03, 0.61; up 4+03:11:20 07:39:06
299 threads: 20 idle, 23 runnable, 246 sleeping, 8 zombie, 2 on CPU
CPU states: 0.0% user, 0.0% nice, 25.5% system, 2.9% interrupt, 71.5% idle
Memory: 407M Act, 209M Inact, 9292K Wired, 48M Exec, 265M File, 24M Free
Swap: / Pools: 265M Used
PID LID USERNAME PRI STATE TIME WCPU CPU NAME COMMAND
0 116 root 221 mmctas/3 82:35 4.74% 4.74% sdmmc0 [system]
0 212 root 126 pgdaem/2 1:06 4.44% 4.44% pgdaemon [system]
0 9 root 125 vdrain/3 2:42 2.64% 2.64% vdrain [system]
2197 2197 fhem 85 nanosl/2 87:16 2.15% 2.15% - perl
2860 2860 fhem 85 RUN/1 171:49 0.00% 0.00% - perl
0 213 root 124 CPU/0 15:46 0.00% 0.00% ioflush [system]
1717 1717 fhem 85 RUN/1 9:44 0.00% 0.00% - perl
0 1444 root 96 RUN/1 9:04 0.00% 0.00% onewire0 [system]
2998 2998 fhem 83 netio/2 8:09 0.00% 0.00% - perl
28610 28610 root 76 select/2 6:04 0.00% 0.00% - racoon
1716 1716 fhem 85 RUN/1 5:56 0.00% 0.00% - perl
1134 1134 root 78 kqueue/2 5:01 0.00% 0.00% - syslogd
0 119 root 96 RUN/1 4:06 0.00% 0.00% usbtask-d [system]
1940 1940 fhem 85 nanosl/2 3:49 0.00% 0.00% - perl
1915 1915 root 85 RUN/1 3:19 0.00% 0.00% - ping
0 0 root 125 RUN/1 3:01 0.00% 0.00% swapper [system]
4666 4666 zigbee 76 kqueue/3 2:44 0.00% 0.00% - node
952 1135 root 43 parked/2 2:11 0.00% 0.00% net-0 named
3903 3903 kardel 85 RUN/1 1:55 0.00% 0.00% - sshd
952 1101 root 43 parked/0 1:54 0.00% 0.00% net-2 named
952 958 root 85 RUN/1 1:52 0.00% 0.00% net-1 named
385 388 root 85 kqueue/0 1:44 0.00% 0.00% sock-0 dhcpd
952 954 root 43 parked/3 1:37 0.00% 0.00% net-3 named
1460 1460 root 85 RUN/1 1:27 0.00% 0.00% - gpiomon
5664 5664 kardel 85 select/2 1:23 0.00% 0.00% - screen
0 123 root 222 icmp_w/0 1:08 0.00% 0.00% icmp_wq*0 [system]
0 199 root 222 xclocv/3 0:59 0.00% 0.00% key_timeh [system]
1853 1853 root 85 select/2 0:56 0.00% 0.00% - perl
414 414 root 85 RUN/1 0:42 0.00% 0.00% - ntpd
1172 1172 root 85 select/2 0:39 0.00% 0.00% - httpd
1518 1518 root 85 poll/0 0:29 0.00% 0.00% - ripd
0 193 root 222 nd6_ti/2 0:22 0.00% 0.00% nd6_timer [system]
853 853 root 85 poll/2 0:17 0.00% 0.00% - bareos-fd
3107 3107 www 76 socket/0 0:16 0.00% 0.00% - httpd
952 1113 root 85 kqueue/2 0:12 0.00% 0.00% sock-0 named
2728 2728 pgsql 83 poll/0 0:12 0.00% 0.00% - postgres
0 3 root 222 IDLE/0 0:12 0.00% 0.00% softnet*0 [system]
2983 2983 pgsql 85 poll/2 0:09 0.00% 0.00% - postgres
828 828 nobody 85 RUN/1 0:08 0.00% 0.00% - mdnsd
4666 2842 zigbee 39 parked/3 0:08 0.00% 0.00% - node
1019 1096 www 43 RUN/1 0:07 0.00% 0.00% - httpd
4666 3582 zigbee 43 RUN/1 0:07 0.00% 0.00% - node
113 113 kardel 76 select/3 0:07 0.00% 0.00% - screen
4666 3677 zigbee 38 parked/0 0:07 0.00% 0.00% - node
0 122 root 222 rt_tim/2 0:06 0.00% 0.00% rt_timer [system]
952 1099 root 85 kqueue/3 0:06 0.00% 0.00% sock-1 named
22307 22307 kardel 76 poll/0 0:06 0.00% 0.00% - sshd
0 214 root 125 xclocv/0 0:05 0.00% 0.00% pooldrain [system]
0 365 root 123 physio/0 0:05 0.00% 0.00% physiod [system]
2854 2854 pgsql 85 poll/0 0:05 0.00% 0.00% - postgres
805 805 pgsql 85 poll/3 0:05 0.00% 0.00% - postgres
1175 3583 www 42 parked/3 0:05 0.00% 0.00% - httpd
4666 4248 zigbee 38 parked/2 0:05 0.00% 0.00% - node
1677 1677 root 85 RUN/1 0:04 0.00% 0.00% - cron
885 885 root 85 kqueue/0 0:04 0.00% 0.00% - master
4954 4954 kardel 85 RUN/1 0:03 0.00% 0.00% - screen
1019 3440 www 84 kqueue/2 0:03 0.00% 0.00% - httpd
1175 1276 www 84 kqueue/2 0:03 0.00% 0.00% - httpd
952 1114 root 43 parked/0 0:03 0.00% 0.00% timer named
3500 922 www 40 parked/0 0:03 0.00% 0.00% - httpd
1524 791 www 85 RUN/1 0:02 0.00% 0.00% - httpd
3500 1421 www 85 kqueue/0 0:02 0.00% 0.00% - httpd
169 169 root 85 nanosl/0 0:02 0.00% 0.00% - wdogctl
0 120 root 43 vcmbox/0 0:02 0.00% 0.00% vcmbox0 [system]
1524 790 www 43 parked/2 0:02 0.00% 0.00% - httpd
4666 2608 zigbee 38 parked/0 0:02 0.00% 0.00% - node
0 309 root 96 RUN/1 0:01 0.00% 0.00% hythygt*2 [system]
657 657 pgsql 85 RUN/1 0:01 0.00% 0.00% - postgres
0 202 root 96 npfgcw/0 0:01 0.00% 0.00% npfgc0 [system]
952 820 root 85 kqueue/0 0:01 0.00% 0.00% sock-3 named
952 1119 root 85 kqueue/0 0:01 0.00% 0.00% sock-2 named
888 888 postfix 85 kqueue/3 0:01 0.00% 0.00% - qmgr
0 97 root 220 IDLE/2 0:01 0.00% 0.00% softclk*2 [system]
7007 7007 root 25 CPU/2 0:00 0.00% 0.00% - top
0 28 root 127 RUN/1 0:00 0.00% 0.00% xcall/1 [system]
0 194 root 96 RUN/1 0:00 0.00% 0.00% usb0 [system]
10885 10885 root 85 RUN/1 0:00 0.00% 0.00% - sh
16001 16001 postfix 85 RUN/1 0:00 0.00% 0.00% - pickup
0 170 root 222 carp_w/3 0:00 0.00% 0.00% carp_wq*3 [system]
0 173 root 222 carp_w/2 0:00 0.00% 0.00% carp_wq*2 [system]
0 31 root 222 carp_w/1 0:00 0.00% 0.00% carp_wq*1 [system]
0 63 root 222 carp_w/0 0:00 0.00% 0.00% carp_wq*0 [system]
0 195 root 222 icmp6_/0 0:00 0.00% 0.00% icmp6_w*0 [system]
0 126 root 222 icmp_w/3 0:00 0.00% 0.00% icmp_wq*3 [system]
0 125 root 222 icmp_w/2 0:00 0.00% 0.00% icmp_wq*2 [system]
0 196 root 222 icmp6_/1 0:00 0.00% 0.00% icmp6_w*1 [system]
0 197 root 222 icmp6_/2 0:00 0.00% 0.00% icmp6_w*2 [system]
0 124 root 222 icmp_w/1 0:00 0.00% 0.00% icmp_wq*1 [system]
0 198 root 222 icmp6_/3 0:00 0.00% 0.00% icmp6_w*3 [system]
0 175 root 222 carp6_/2 0:00 0.00% 0.00% carp6_w*2 [system]
0 177 root 222 carp6_/1 0:00 0.00% 0.00% carp6_w*1 [system]
0 192 root 222 carp6_/3 0:00 0.00% 0.00% carp6_w*3 [system]
0 176 root 222 carp6_/0 0:00 0.00% 0.00% carp6_w*0 [system]
0 201 root 222 rt_fre/0 0:00 0.00% 0.00% rt_free [system]
0 12 root 222 ifwdog/3 0:00 0.00% 0.00% ifwdog [system]
0 11 root 222 iflnks/2 0:00 0.00% 0.00% iflnkst [system]
0 7 root 127 xcall/0 0:00 0.00% 0.00% xcall/0 [system]
0 105 root 127 xcall/3 0:00 0.00% 0.00% xcall/3 [system]
0 99 root 127 xcall/2 0:00 0.00% 0.00% xcall/2 [system]
0 121 root 125 vmem_r/3 0:00 0.00% 0.00% vmem_reha [system]
0 8 root 125 mod_un/2 0:00 0.00% 0.00% modunload [system]
0 210 root 123 data/3 0:00 0.00% 0.00% vcaudio [system]
0 208 root 96 semacv/3 0:00 0.00% 0.00% VCHIQr-0 [system]
0 499 root 96 hytcv/2 0:00 0.00% 0.00% hythygt*1 [system]
0 498 root 96 hytcv/2 0:00 0.00% 0.00% hythygt*0 [system]
0 211 root 96 lnxcmp/2 0:00 0.00% 0.00% VCHIQka*0 [system]
0 209 root 96 semacv/3 0:00 0.00% 0.00% VCHIQs-0 [system]
0 207 root 96 semacv/3 0:00 0.00% 0.00% VCHIQ-0 [system]
0 200 root 96 unpgc/0 0:00 0.00% 0.00% unpgc [system]
0 118 root 96 usbtsk/2 0:00 0.00% 0.00% usbtask-h [system]
0 16 root 96 smtask/2 0:00 0.00% 0.00% sysmon [system]
0 13 root 96 sopend/0 0:00 0.00% 0.00% sopendfre [system]
2987 2987 pgsql 85 poll/3 0:00 0.00% 0.00% - postgres
4666 5848 zigbee 85 kqueue/0 0:00 0.00% 0.00% - node
5611 5611 root 85 pipe_r/1 0:00 0.00% 0.00% - sendmail
17312 17312 root 85 pipe_r/3 0:00 0.00% 0.00% - tee
10374 10374 root 85 wait/2 0:00 0.00% 0.00% - sh
304 304 root 85 kqueue/0 0:00 0.00% 0.00% - fscd
15499 15499 root 85 netio/0 0:00 0.00% 0.00% - postdrop
16429 16429 root 85 pipe_r/2 0:00 0.00% 0.00% - cron
1828 1828 pgsql 85 poll/0 0:00 0.00% 0.00% - postgres
8499 8499 root 85 pause/0 0:00 0.00% 0.00% - tcsh
4951 4951 kardel 85 pause/0 0:00 0.00% 0.00% - tcsh
3719 3719 kardel 85 poll/0 0:00 0.00% 0.00% - ssh-agent
4928 4928 kardel 85 pause/2 0:00 0.00% 0.00% - csh
5069 5069 root 85 poll/1 0:00 0.00% 0.00% - sshd
4148 4148 kardel 85 poll/2 0:00 0.00% 0.00% - ssh-agent
1 1 root 85 wait/0 0:00 0.00% 0.00% - init
1435 1435 root 85 poll/0 0:00 0.00% 0.00% - sshd
2088 2088 root 85 poll/2 0:00 0.00% 0.00% - zebra
2088 2211 root 85 poll/2 0:00 0.00% 0.00% zebra_dpl zebra
2088 2801 root 85 poll/2 0:00 0.00% 0.00% zebra_api zebra
385 386 root 85 kqueue/3 0:00 0.00% 0.00% net-0 dhcpd
385 387 root 85 kqueue/3 0:00 0.00% 0.00% net-1 dhcpd
4666 4130 zigbee 84 psem/2 0:00 0.00% 0.00% - node
2088 339 root 84 poll/1 0:00 0.00% 0.00% zebra_api zebra
329 329 root 83 poll/1 0:00 0.00% 0.00% - staticd
421 421 root 83 devmon/1 0:00 0.00% 0.00% - devpubd
952 952 root 83 sigwai/3 0:00 0.00% 0.00% - named
17255 17255 root 82 poll/0 0:00 0.00% 0.00% - sshd
3037 3037 root 82 wait/0 0:00 0.00% 0.00% - sh
1524 1524 www 82 pipe_r/2 0:00 0.00% 0.00% - httpd
2088 1979 root 82 poll/0 0:00 0.00% 0.00% zebra_opa zebra
304 2482 root 82 socket/3 0:00 0.00% 0.00% - fscd
11906 11906 kardel 81 pause/0 0:00 0.00% 0.00% - csh
14075 14075 kardel 81 pause/0 0:00 0.00% 0.00% - screen
26845 26845 kardel 80 poll/0 0:00 0.00% 0.00% - ssh-agent
1019 1019 www 79 pipe_r/2 0:00 0.00% 0.00% - httpd
3500 3500 www 79 pipe_r/2 0:00 0.00% 0.00% - httpd
1175 1175 www 79 pipe_r/1 0:00 0.00% 0.00% - httpd
643 643 root 79 kqueue/0 0:00 0.00% 0.00% - inetd
1939 1939 root 78 ttyraw/1 0:00 0.00% 0.00% - getty
17710 17710 kardel 76 pause/2 0:00 0.00% 0.00% - tcsh
4453 4453 root 76 pause/3 0:00 0.00% 0.00% - tcsh
952 8254 root 43 parked/1 0:00 0.00% 0.00% - named
952 8001 root 43 parked/0 0:00 0.00% 0.00% - named
952 7985 root 43 parked/1 0:00 0.00% 0.00% - named
0 15 root 43 pmfsus/2 0:00 0.00% 0.00% pmfsuspen [system]
853 993 root 43 parked/3 0:00 0.00% 0.00% - bareos-fd
0 17 root 43 lnxrcu/0 0:00 0.00% 0.00% lnxrcugc [system]
0 18 root 43 lnxhip/2 0:00 0.00% 0.00% lnxhipwq [system]
0 19 root 43 lnxlng/1 0:00 0.00% 0.00% lnxlngwq [system]
0 20 root 43 lnxpwr/0 0:00 0.00% 0.00% lnxpwrwq [system]
0 10 root 43 nfssil/0 0:00 0.00% 0.00% nfssilly [system]
0 14 root 43 pmfeve/0 0:00 0.00% 0.00% pmfevent [system]
952 8004 root 43 parked/0 0:00 0.00% 0.00% - named
0 204 root 43 swwreb/2 0:00 0.00% 0.00% swwreboot [system]
385 389 root 43 parked/3 0:00 0.00% 0.00% timer dhcpd
0 117 root 43 dwc2/3 0:00 0.00% 0.00% dwc2 [system]
0 21 root 43 lnxubd/0 0:00 0.00% 0.00% lnxubdwq [system]
0 22 root 43 lnxsys/3 0:00 0.00% 0.00% lnxsyswq [system]
0 107 root 43 entrop/3 0:00 0.00% 0.00% entbutler [system]
1019 1277 www 42 parked/3 0:00 0.00% 0.00% - httpd
385 385 root 40 parked/2 0:00 0.00% 0.00% - dhcpd
4666 2847 zigbee 40 parked/3 0:00 0.00% 0.00% - node
4666 5942 zigbee 38 parked/0 0:00 0.00% 0.00% - node
4666 4152 zigbee 38 parked/2 0:00 0.00% 0.00% - node
1524 776 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 778 www 37 parked/3 0:00 0.00% 0.00% - httpd
1524 779 www 37 parked/3 0:00 0.00% 0.00% - httpd
1524 780 www 37 parked/3 0:00 0.00% 0.00% - httpd
1524 781 www 37 parked/3 0:00 0.00% 0.00% - httpd
1524 782 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 783 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 784 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 785 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 777 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 786 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 787 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 1826 www 37 parked/3 0:00 0.00% 0.00% - httpd
1524 788 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 789 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 775 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 774 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 773 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 772 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 768 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 771 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 770 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 769 www 37 parked/2 0:00 0.00% 0.00% - httpd
1524 1948 www 37 parked/2 0:00 0.00% 0.00% - httpd
2088 2417 root 36 parked/1 0:00 0.00% 0.00% RCU sweep zebra
3500 1098 www 34 parked/3 0:00 0.00% 0.00% - httpd
3500 2652 www 34 parked/1 0:00 0.00% 0.00% - httpd
1019 1278 www 34 parked/2 0:00 0.00% 0.00% - httpd
1019 2336 www 34 parked/2 0:00 0.00% 0.00% - httpd
1019 3455 www 34 parked/2 0:00 0.00% 0.00% - httpd
1019 1269 www 34 parked/2 0:00 0.00% 0.00% - httpd
1019 1254 www 34 parked/2 0:00 0.00% 0.00% - httpd
3500 1418 www 34 parked/3 0:00 0.00% 0.00% - httpd
3500 1417 www 34 parked/3 0:00 0.00% 0.00% - httpd
3500 1416 www 34 parked/3 0:00 0.00% 0.00% - httpd
3500 3163 www 34 parked/3 0:00 0.00% 0.00% - httpd
3500 1415 www 34 parked/3 0:00 0.00% 0.00% - httpd
3500 3511 www 34 parked/3 0:00 0.00% 0.00% - httpd
1175 1272 www 33 parked/3 0:00 0.00% 0.00% - httpd
1175 3568 www 33 parked/3 0:00 0.00% 0.00% - httpd
1175 1394 www 33 parked/3 0:00 0.00% 0.00% - httpd
1175 1108 www 33 parked/3 0:00 0.00% 0.00% - httpd
1175 2265 www 33 parked/3 0:00 0.00% 0.00% - httpd
1175 1271 www 33 parked/3 0:00 0.00% 0.00% - httpd
1175 1274 www 33 parked/3 0:00 0.00% 0.00% - httpd
1175 1395 www 33 parked/3 0:00 0.00% 0.00% - httpd
1175 1273 www 33 parked/3 0:00 0.00% 0.00% - httpd
1175 1106 www 31 parked/3 0:00 0.00% 0.00% - httpd
1175 1095 www 31 parked/3 0:00 0.00% 0.00% - httpd
1175 1104 www 31 parked/3 0:00 0.00% 0.00% - httpd
1019 1065 www 31 parked/2 0:00 0.00% 0.00% - httpd
1019 1152 www 31 parked/3 0:00 0.00% 0.00% - httpd
1019 3291 www 31 parked/3 0:00 0.00% 0.00% - httpd
1019 2780 www 31 parked/3 0:00 0.00% 0.00% - httpd
1019 2904 www 31 parked/2 0:00 0.00% 0.00% - httpd
1019 1111 www 31 parked/2 0:00 0.00% 0.00% - httpd
1019 2266 www 31 parked/2 0:00 0.00% 0.00% - httpd
1019 1100 www 31 parked/2 0:00 0.00% 0.00% - httpd
1019 1256 www 31 parked/3 0:00 0.00% 0.00% - httpd
1019 2875 www 31 parked/3 0:00 0.00% 0.00% - httpd
1019 3306 www 31 parked/3 0:00 0.00% 0.00% - httpd
1019 1257 www 31 parked/3 0:00 0.00% 0.00% - httpd
1019 1252 www 31 parked/3 0:00 0.00% 0.00% - httpd
1019 1255 www 31 parked/3 0:00 0.00% 0.00% - httpd
1019 2287 www 31 parked/3 0:00 0.00% 0.00% - httpd
1019 1259 www 31 parked/3 0:00 0.00% 0.00% - httpd
1019 3308 www 31 parked/3 0:00 0.00% 0.00% - httpd
1175 849 www 31 parked/3 0:00 0.00% 0.00% - httpd
1175 1093 www 31 parked/3 0:00 0.00% 0.00% - httpd
1019 1089 www 31 parked/2 0:00 0.00% 0.00% - httpd
1175 1253 www 31 parked/3 0:00 0.00% 0.00% - httpd
1175 1248 www 31 parked/2 0:00 0.00% 0.00% - httpd
1175 3001 www 31 parked/3 0:00 0.00% 0.00% - httpd
1175 1103 www 31 parked/1 0:00 0.00% 0.00% - httpd
1175 1110 www 31 parked/3 0:00 0.00% 0.00% - httpd
1175 1071 www 31 parked/3 0:00 0.00% 0.00% - httpd
1175 1854 www 31 parked/3 0:00 0.00% 0.00% - httpd
1175 3278 www 31 parked/3 0:00 0.00% 0.00% - httpd
3500 1014 www 31 parked/2 0:00 0.00% 0.00% - httpd
3500 1461 www 31 parked/1 0:00 0.00% 0.00% - httpd
3500 1171 www 31 parked/0 0:00 0.00% 0.00% - httpd
3500 1848 www 31 parked/2 0:00 0.00% 0.00% - httpd
3500 1088 www 31 parked/1 0:00 0.00% 0.00% - httpd
3500 3129 www 31 parked/2 0:00 0.00% 0.00% - httpd
3500 1980 www 31 parked/2 0:00 0.00% 0.00% - httpd
3500 1090 www 31 parked/3 0:00 0.00% 0.00% - httpd
3500 1092 www 31 parked/3 0:00 0.00% 0.00% - httpd
3500 1073 www 31 parked/3 0:00 0.00% 0.00% - httpd
3500 2251 www 31 parked/3 0:00 0.00% 0.00% - httpd
3500 1091 www 31 parked/3 0:00 0.00% 0.00% - httpd
3500 2253 www 31 parked/3 0:00 0.00% 0.00% - httpd
3500 1118 www 31 parked/2 0:00 0.00% 0.00% - httpd
3500 1250 www 31 parked/1 0:00 0.00% 0.00% - httpd
3500 1517 www 31 parked/3 0:00 0.00% 0.00% - httpd
1175 1215 www 31 parked/3 0:00 0.00% 0.00% - httpd
1175 1066 www 31 parked/3 0:00 0.00% 0.00% - httpd
0 104 root 223 IDLE/3 0:00 0.00% 0.00% softser*3 [system]
0 6 root 223 IDLE/0 0:00 0.00% 0.00% softser*0 [system]
0 27 root 223 IDLE/1 0:00 0.00% 0.00% softser*1 [system]
0 98 root 223 IDLE/2 0:00 0.00% 0.00% softser*2 [system]
0 101 root 222 IDLE/3 0:00 0.00% 0.00% softnet*3 [system]
0 30 root 222 IDLE/2 0:00 0.00% 0.00% softnet*2 [system]
0 24 root 222 IDLE/1 0:00 0.00% 0.00% softnet*1 [system]
0 25 root 221 IDLE/1 0:00 0.00% 0.00% softbio*1 [system]
0 102 root 221 IDLE/3 0:00 0.00% 0.00% softbio*3 [system]
0 96 root 221 IDLE/2 0:00 0.00% 0.00% softbio*2 [system]
0 4 root 221 IDLE/0 0:00 0.00% 0.00% softbio*0 [system]
0 26 root 220 IDLE/1 0:00 0.00% 0.00% softclk*1 [system]
0 5 root 220 IDLE/0 0:00 0.00% 0.00% softclk*0 [system]
0 103 root 220 IDLE/3 0:00 0.00% 0.00% softclk*3 [system]
0 100 root 0 IDLE/3 0:00 0.00% 0.00% idle/3 [system]
0 2 root 0 IDLE/0 0:00 0.00% 0.00% idle/0 [system]
0 23 root 0 IDLE/1 0:00 0.00% 0.00% idle/1 [system]
0 29 root 0 IDLE/2 0:00 0.00% 0.00% idle/2 [system]
vmstat -m
Memory resource pool statistics
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
amappl 56 9539 0 5731 57 4 53 53 0 inf 0
anonpl 24 114815 0 33292 493 0 493 493 0 inf 0
biopl 192 2786 0 2690 29 13 16 16 0 inf 11
buf16k 16896 641 0 609 60 49 11 11 0 inf 0
buf1k 1536 324 0 324 6 5 1 2 0 inf 1
buf2k 2560 5 0 5 1 0 1 1 0 inf 1
buf32k 32768 181 0 175 26 21 5 5 0 inf 2
buf512b 1024 2 0 2 1 0 1 1 0 inf 1
buf64k 65536 10333 0 9894 1511 1071 440 440 0 inf 1
buf8k 8704 279549 0 271168 9205 6953 2252 2256 0 inf 1047
bufpl 192 90804 0 76678 1886 1125 761 761 0 inf 4
cryptdesc 64 1928 0 1880 18 11 7 7 0 inf 6
cryptop 256 973 0 949 39 25 14 14 0 inf 12
cwdi 192 1250 0 1112 9 2 7 7 0 inf 0
dirhash 344 1297 0 1129 38 19 19 20 0 inf 0
dirhashblk 2048 6610 0 5808 1460 838 622 622 0 inf 185
dwc2qh 160 16 0 0 1 0 1 1 0 inf 0
dwc2qtd 48 1359 0 1305 1 0 1 1 0 inf 0
dwc2xfer 192 410 0 400 12 9 3 4 0 inf 2
esp_tdb_crypto 128 1043 0 1025 18 11 7 7 0 inf 6
execargs 262144 317113 0 317112 38 34 4 4 0 16 3
fcrpl 136 40 0 34 3 0 3 3 3 inf 2
fdfile 64 6752 0 4537 37 1 36 36 0 inf 0
ffsdino1 136 119273 0 70730 1677 1 1676 1676 0 inf 0
ffsino 216 131639 0 83096 2701 1 2700 2700 0 inf 0
file 128 1991 0 1353 21 0 21 21 0 inf 0
filedesc 704 1339 0 1218 44 15 29 29 0 inf 4
fstlwp 64 1198 0 1028 3 0 3 3 0 inf 0
icmp 24 493026 0 493026 3 2 1 1 0 inf 1
icmp6 24 4684 0 4684 13 12 1 1 0 inf 1
in4pcbpl 192 1028 0 906 16 6 10 15 0 inf 0
in6pcbpl 256 639 0 575 14 6 8 9 0 inf 1
inmltpl 40 15 0 0 1 0 1 1 0 inf 0
kcpuset 64 1630 0 1422 4 0 4 4 0 inf 0
kcredpl 192 1149 0 760 20 0 20 20 0 inf 0
kmem-00016 16 6488 0 3191 13 0 13 13 0 inf 0
kmem-00032 32 24446 0 19441 57 17 40 40 0 inf 0
kmem-00064 128 71538 0 65851 1050 575 475 575 0 inf 93
kmem-00128 192 18594 0 13874 348 123 225 225 0 inf 0
kmem-00192 256 10252 0 8325 256 135 121 121 0 inf 0
kmem-00256 320 1829 0 1295 49 4 45 46 0 inf 0
kmem-00320 384 2746 0 1579 120 2 118 118 0 inf 1
kmem-00384 448 1667 0 1354 41 5 36 36 0 inf 1
kmem-00448 512 1685 0 1283 67 16 51 51 0 inf 0
kmem-00512 576 1563 0 1347 42 9 33 33 0 inf 2
kmem-00768 832 2015 0 1591 134 27 107 107 0 inf 1
kmem-01024 1088 2817 0 1351 547 35 512 512 0 inf 23
kmem-02048 2048 1541 0 1167 344 155 189 189 0 inf 2
kmem-04096 4096 141 0 109 64 26 38 38 0 inf 6
ksiginfo 128 1211 0 1157 9 6 3 3 0 inf 1
kva-12288 12288 17 0 5 1 0 1 1 0 inf 0
kva-16384 16384 2 0 0 1 0 1 1 0 inf 0
kva-20480 20480 8 0 4 1 0 1 1 0 inf 0
kva-24576 24576 5 0 0 1 0 1 1 0 inf 0
kva-32768 32768 8 0 0 1 0 1 1 0 inf 0
kva-4096 4096 24568 0 2856 340 0 340 340 0 inf 0
kva-65536 65536 1 0 0 1 0 1 1 0 inf 0
kva-8192 8192 70 0 62 1 0 1 1 0 inf 0
l1ttpl 8192 1297 0 1162 532 0 532 532 532 532 397
l2dtblpl 200 1871 0 1390 44 17 27 28 0 inf 2
l2ptppl 2048 3564 0 2012 975 191 784 865 0 inf 8
llentrypl 192 568 0 547 2 1 1 2 0 inf 0
lockf 72 835 0 786 4 2 2 2 0 inf 0
lwppl 704 1134 0 803 90 16 74 74 0 inf 4
mbpl 264 7130 0 3533 272 31 241 241 0 inf 1
mclpl 2048 2742 0 2584 596 400 196 200 0 15168 117
msdosnopl 128 3015 0 3015 104 84 20 20 0 inf 20
mutex 64 113641 0 64780 776 0 776 776 0 inf 0
namecache 128 85922 0 37385 1567 0 1567 1567 0 inf 0
pcache 576 86 0 4 12 0 12 12 0 inf 0
pcachecpu 64 273 0 0 5 0 5 5 0 inf 0
pcglarge 512 5440 0 5179 156 115 41 41 0 inf 8
pcgnormal 128 316527 0 312765 599 457 142 216 0 inf 20
pdict16 48 293 0 249 1 0 1 1 0 inf 0
pdict32 64 11 0 1 1 0 1 1 0 inf 0
phpool-128 48 57 0 17 1 0 1 1 0 inf 0
phpool-256 64 13 0 0 1 0 1 1 0 inf 0
phpool-32 40 12901 0 3404 96 1 95 95 0 inf 0
phpool-64 40 341 0 0 4 0 4 4 0 inf 0
piperd 176 978 0 852 10 3 7 7 0 inf 1
pipewr 176 1079 0 935 11 4 7 7 0 inf 0
plimitpl 224 95 0 71 5 3 2 2 0 inf 0
pmappl 1088 1146 0 1008 110 63 47 48 0 inf 1
pnbufpl 1032 1276 0 1183 137 106 31 31 0 inf 0
procpl 704 838 0 736 56 28 28 29 0 inf 6
proparay 40 79 0 4 1 0 1 1 0 inf 0
propdata 24 1 0 0 1 0 1 1 0 inf 0
propdict 40 383 0 137 3 0 3 3 0 inf 0
propnmbr 48 61 0 8 1 0 1 1 0 inf 0
propstng 40 743 0 562 2 0 2 2 0 inf 0
pstatspl 320 830 0 728 19 7 12 12 0 inf 1
pvepl 24 526848110 0 526794322 839 64 775 826 0 inf 107
ractx 40 183157 0 143763 1516 1125 391 446 0 inf 0
radixnode 128 75086 0 56363 1003 301 702 759 0 inf 21
rtentpl 216 117 0 66 5 0 5 5 0 inf 2
rwlock 64 120601 1 69153 817 0 817 817 0 inf 0
sigacts 2056 1180 0 1044 453 314 139 140 0 inf 3
socket 352 1261 0 834 49 10 39 41 0 inf 0
synpl 216 15846 0 15846 11 10 1 1 0 inf 1
tcpcbpl 528 22599 0 22514 52 37 15 37 0 inf 0
tcpipqepl 40 12989 0 12989 14 13 1 1 0 inf 1
tmpfs_dirent 32 1 0 0 1 0 1 1 0 inf 0
tmpfs_node 176 2 0 0 1 0 1 1 0 inf 0
tstile 64 1134 0 803 6 0 6 6 0 inf 0
uarea 8192 1064 0 737 716 352 364 366 0 inf 37
ufsdir 272 509 0 489 23 17 6 6 0 inf 4
vcachepl 512 125869 0 77283 6077 2 6075 6075 0 inf 0
vmembt 64 3221 0 891 39 0 39 39 0 inf 0
vmmpepl 128 19273 0 10529 292 9 283 283 0 inf 0
vmsppl 216 1185 0 1048 10 1 9 9 0 inf 0
Totals 529776967 1 529175013 41072 15153 25919
In use 183439K, total allocated 270764K; utilization 67.7%
vmstat -s
4096 bytes per page
1 page color
234846 pages managed
6893 pages free
103383 pages active
53772 pages inactive
0 pages paging
2323 pages wired
1 reserve pagedaemon pages
20 reserve kernel pages
4587 boot kernel pages
67722 kernel pool pages
79679 anonymous pages
67655 cached file pages
12276 cached executable pages
1024 minimum free pages
1365 target free pages
78282 maximum wired pages
0 swap devices
0 swap pages
0 swap pages in use
0 swap allocations
157182774 total faults taken
3814757442 traps
3066133099 device interrupts
73289005 CPU context switches
171760891 software interrupts
188092725 system calls
0 pagein requests
0 pageout requests
0 pages swapped in
0 pages swapped out
509380 forks total
6573 forks blocked parent
6573 forks shared address space with parent
69878952 pagealloc desired color avail
0 pagealloc desired color not avail
59525592 pagealloc local cpu avail
10353360 pagealloc local cpu not avail
7748 faults with no memory
0 faults with no anons
0 faults had to wait on pages
0 faults found released page
132734 faults relock (132724 ok)
60825277 anon page faults
0 anon retry faults
18274044 amap copy faults
91028685 neighbour anon page faults
277269213 neighbour object page faults
57880559 locked pager get faults
132734 unlocked pager get faults
30259694 anon faults
30565583 anon copy on write faults
53132695 object faults
4747020 promote copy faults
29357930 promote zero fill faults
3057 faults upgraded lock
44 faults couldn't upgrade lock
26375 times daemon wokeup
26377 revolutions of the clock hand
1911163 pages freed by daemon
4175778 pages scanned by daemon
415098 anonymous pages scanned by daemon
1914218 object pages scanned by daemon
39533 pages reactivated
47 pages found busy by daemon
0 total pending pageouts
3873444 pages deactivated
3717312 per-cpu stats synced
20 anon pages possibly dirty
79659 anon pages dirty
0 anon pages clean
0 file pages possibly dirty
56 file pages dirty
79875 file pages clean
51343083 total name lookups
33782276 good hits
3128614 negative hits
19604 bad hits
0 false hits
14412589 miss
0 too long
146868 pass2 hits
156678 2passes
3895 reverse hits
0 reverse miss
0 access denied
cache hits (65% pos + 6% neg) system 0% per-process
deletions 0%, falsehits 0%, toolong 0%
uptime
8:06AM up 4 days, 3:38, 3 users, load averages: 1.60, 1.03, 0.61
>How-To-Repeat:
run NetBSD 10.0_BETA evbarm/earmv7hf with routing, ipsec, webserver, perl etc
for about 4 days until the system gets partially stalled as a cpu should be
running runnable processes but does not.
>Fix:
no known
>Audit-Trail:
From: Taylor R Campbell <riastradh@NetBSD.org>
To: Frank Kardel <kardel@netbsd.org>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/57580: evbarm/earmv7hf RPI2 scheduler/cpu stall
Date: Sun, 13 Aug 2023 11:56:13 +0000
> In this state "cpuctl offline 1" gets stuck waiting on "xclocv"
> wait channel.
This means it's waiting for a low-priority xcall(9) to complete on
another CPU, presumably cpu1. The xcall thread for cpu1 is here:
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
0 28 2 1 200 914dd440 xcall/1
S = 2 means LSRUN, meaning the process is runnable but not on the CPU.
FLAGS = 0x200 means LW_SYSTEM, meaning this is a kernel thread, as
expected -- this is the thread that handles low-priority xcalls.
> So why is CPU 1 not running processes anymore?
Not sure. It appears to be here:
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
4939 >4939 7 1 1000000 93f75200 perl
S = 7 means LSONPROC, meaning the process is currently executing on
the CPU.
FLAGS = 0x1000000 means LW_PENDSIG, meaning a signal is pending.
But what's the perl process doing and why can't it make progress?
Unclear.
Can you enter ddb in this state (not crash(8) -- use C-A-ESC in wskbd
or break at serial console or (set and) type the hw.cnmagic sequence),
and do `mach cpu 1', and then `bt'? Once you get output, you can do
`continue' to return from ddb.
If not, can you try enabling `options HEARTBEAT' and `options
HEARTBEAT_MAX_PERIOD=15' in our kernel config and see if you get any
diagnostics out of that?
From: Frank Kardel <kardel@netbsd.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/57580: evbarm/earmv7hf RPI2 scheduler/cpu stall
Date: Sun, 13 Aug 2023 16:58:12 +0200
Hi Taylor,
ktruss -p 4939 does not seem to do anything, but is interruptible.
perl does not collect an CPU time and if all would be working it should be
yielding the CPU due to scheduling. Somehow the CPU seems be 'lost' before
returning from kernel to user level - just a guess.
As I rebooted, I need to wait again before I can get into the debugger.
HEARTBEAT is in -current not in 10.0_BETA or did I miss a pullup-10?
Frenk
On 08/13/23 14:00, Taylor R Campbell wrote:
> > So why is CPU 1 not running processes anymore?
>
> Not sure. It appears to be here:
>
> PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
> 4939 >4939 7 1 1000000 93f75200 perl
>
> S = 7 means LSONPROC, meaning the process is currently executing on
> the CPU.
>
> FLAGS = 0x1000000 means LW_PENDSIG, meaning a signal is pending.
>
> But what's the perl process doing and why can't it make progress?
> Unclear.
>
> Can you enter ddb in this state (not crash(8) -- use C-A-ESC in wskbd
> or break at serial console or (set and) type the hw.cnmagic sequence),
> and do `mach cpu 1', and then `bt'? Once you get output, you can do
> `continue' to return from ddb.
>
> If not, can you try enabling `options HEARTBEAT' and `options
> HEARTBEAT_MAX_PERIOD=15' in our kernel config and see if you get any
> diagnostics out of that?
>
From: Frank Kardel <kardel@netbsd.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/57580: evbarm/earmv7hf RPI2 scheduler/cpu stall
Date: Thu, 17 Aug 2023 07:45:58 +0200
So I got a partial success with DDB.
Stopped in pid 0.5 (system) at netbsd:cpu_Debugger+0x4: bx r14
db{0}> mach cpu 1
kdb_trap: switching to cpu1
[ 85689.2095237] Mutex error: mutex_vector_enter,516: assertion failed:
!cpu_intr_p()
[ 85689.2095237] lock address : 913d3940
[ 85689.2095237] current cpu : 1
[ 85689.2095237] current lwp : 0x00000000914c3400
[ 85689.2095237] owner field : 000000000000000000
wait/spin: 0/0
[ 85689.2095237] panic: lock error: Mutex: mutex_vector_enter,516:
assertion failed: !cpu_intr_p(): lock 0x913d3940 cpu 1 lwp 0x914c3400
[ 85689.2095237] cpu1: Begin traceback...
[ 85689.2095237] 0x80cb3b44: netbsd:db_panic+0x14
[ 85689.2095237] 0x80cb3b64: netbsd:vpanic+0x114
[ 85689.2095237] 0x80cb3b7c: netbsd:panic+0x24
[ 85689.2095237] 0x80cb3c44: netbsd:lockdebug_abort+0xe8
[ 85689.2095237] 0x80cb3c5c: netbsd:mutex_abort+0x30
[ 85689.2095237] 0x80cb3cc4: netbsd:mutex_enter+0x48c
[ 85689.2095237] 0x80cb3cdc: netbsd:usbd_set_polling+0x34
[ 85689.2095237] 0x80cb3cfc: netbsd:ukbd_cnpollc+0x5c
[ 85689.2095237] 0x80cb3d14: netbsd:wsdisplay_pollc+0x60
[ 85689.2095237] 0x80cb3d2c: netbsd:cnpollc+0x4c
[ 85689.2095237] 0x80cb3dbc: netbsd:kdb_trap+0x19c
[ 85689.2095237] 0x80cb3dcc: netbsd:pic_ipi_ddb+0x18
[ 85689.2095237] 0x80cb3df4: netbsd:bcm2836mp_ipi_handler+0x11c
[ 85689.2095237] 0x80cb3e44: netbsd:pic_dispatch+0x54
[ 85689.2095237] 0x80cb3ecc: netbsd:pic_do_pending_ints+0x434
[ 85689.2095237] 0x80cb3f34: netbsd:irq_idle_entry+0x38
[ 85689.2095237] 0x80cb3f94: netbsd:idle_loop+0x1b8
[ 85689.2095237] cpu1: End traceback...
[ 85689.2095237] dump to dev 92,1 not possible
[ 85689.2095237] rebooting...
Additionally getting stacks from crash got unlucky and stuck.
The currently runnable processes from ps are:
UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TTY
TIME COMMAND
0 0 0 0 222 0 0 8724 - RKl ? 32:48.28
[system]
1002 585 3387 0 85 0 164508 3084 - Rs ? 0:00.14
postgres: logical replication launcher
1002 588 3387 0 85 0 164508 3268 - Rs ? 0:02.57
postgres: autovacuum launcher
1005 594 1 0 85 0 100024 74656 - R ? 37:17.08
/usr/bin/perl /usr/pkg/fhem/fhem.pl /usr/pkg/fhem/fhem.cfg
0 736 1 0 85 0 15780 4604 - Rs ? 0:13.95
/usr/pkg/bin/perl -wT /usr/pkg/sbin/munin-node
0 969 1 0 85 0 13716 2336 - Rs ? 0:01.12
/usr/libexec/postfix/master -w
0 1249 1 0 85 0 6396 1444 - Rs ? 0:01.19
/usr/sbin/cron
0 1297 1296 0 85 0 7156 1896 select Is ? 0:00.13
SCREEN -R (screen)
0 1454 1 0 85 0 9744 1684 - Rs ? 1:13.53
/usr/sbin/syslogd -s
0 1646 1 7048 85 0 112428 10004 - Rsl ? 2:08.08
/usr/sbin/named
1002 1739 3387 0 85 0 163884 2564 - Rs ? 0:01.31
postgres: walwriter
1001 1882 1 42491 85 0 44568 3800 - Rsl ? 0:00.63
/usr/pkg/sbin/zebra -P 0 -d
1008 3288 1 102890 85 0 187872 39556 - Rsl ? 1:19.25
/usr/pkg/bin/node /usr/pkg/zigbee2mqtt/index.js
1002 3387 1 0 85 0 163600 3988 - Rs ? 0:00.57
/usr/pkg/bin/postgres -D /usr/pkg/pgsql/data
0 11982 8414 0 0 0 11476 3564 - Rs ? 0:07.43
(perl)
As Hearbeat does not seem to be supported in NetBSD-10, I try to
run a -current kernel and see how far I get.
On 08/13/23 14:00, Taylor R Campbell wrote:
> Can you enter ddb in this state (not crash(8) -- use C-A-ESC in wskbd
> or break at serial console or (set and) type the hw.cnmagic sequence),
> and do `mach cpu 1', and then `bt'? Once you get output, you can do
> `continue' to return from ddb.
>
> If not, can you try enabling `options HEARTBEAT' and `options
> HEARTBEAT_MAX_PERIOD=15' in our kernel config and see if you get any
> diagnostics out of that?
From: Frank Kardel <kardel@netbsd.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/57580: evbarm/earmv7hf RPI2 scheduler/cpu stall
Date: Thu, 17 Aug 2023 08:02:39 +0200
Looks like I overlooked the the stuck CPU was CPU 0. Oh well.
So this might not be too informative. Good thing is that it
gets stuck within about 1-2 days.
Another round.
> The currently runnable processes from ps are:
> UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TTY
> TIME COMMAND
> 0 0 0 0 222 0 0 8724 - RKl ? 32:48.28
> [system]
> 1002 585 3387 0 85 0 164508 3084 - Rs ? 0:00.14
> postgres: logical replication launcher
> 1002 588 3387 0 85 0 164508 3268 - Rs ? 0:02.57
> postgres: autovacuum launcher
> 1005 594 1 0 85 0 100024 74656 - R ? 37:17.08
> /usr/bin/perl /usr/pkg/fhem/fhem.pl /usr/pkg/fhem/fhem.cfg
> 0 736 1 0 85 0 15780 4604 - Rs ? 0:13.95
> /usr/pkg/bin/perl -wT /usr/pkg/sbin/munin-node
> 0 969 1 0 85 0 13716 2336 - Rs ? 0:01.12
> /usr/libexec/postfix/master -w
> 0 1249 1 0 85 0 6396 1444 - Rs ? 0:01.19
> /usr/sbin/cron
> 0 1297 1296 0 85 0 7156 1896 select Is ? 0:00.13
> SCREEN -R (screen)
> 0 1454 1 0 85 0 9744 1684 - Rs ? 1:13.53
> /usr/sbin/syslogd -s
> 0 1646 1 7048 85 0 112428 10004 - Rsl ? 2:08.08
> /usr/sbin/named
> 1002 1739 3387 0 85 0 163884 2564 - Rs ? 0:01.31
> postgres: walwriter
> 1001 1882 1 42491 85 0 44568 3800 - Rsl ? 0:00.63
> /usr/pkg/sbin/zebra -P 0 -d
> 1008 3288 1 102890 85 0 187872 39556 - Rsl ? 1:19.25
> /usr/pkg/bin/node /usr/pkg/zigbee2mqtt/index.js
> 1002 3387 1 0 85 0 163600 3988 - Rs ? 0:00.57
> /usr/pkg/bin/postgres -D /usr/pkg/pgsql/data
> 0 11982 8414 0 0 0 11476 3564 - Rs ? 0:07.43
> (perl)
>
From: Frank Kardel <kardel@netbsd.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/57580: evbarm/earmv7hf RPI2 scheduler/cpu stall
Date: Thu, 17 Aug 2023 08:11:23 +0200
As I managed to get a ps from crash it looks like the same scenarion.
perl is on cpu but not making any progress.
crash ps with on cpu processes and runnable processes.
17744>17744 7 1 0 941db940 crash
11982>11982 7 0 1000000 9186f240 perl
1249 1249 2 0 140 934d63c0 cron
969 969 2 0 1000140 9356b400 master
736 736 2 0 140 923fcbc0 perl
594 594 2 0 140 924b5940 perl
585 585 2 0 140 91dbd700 postgres
588 588 2 0 140 923fc900 postgres
1739 1739 2 0 140 92277080 postgres
3387 3387 2 0 140 9262a140 postgres
1882 3108 2 0 100 91f4e1c0 zebra_dplane
3288 3288 2 0 100 91809a40 node
1646 1497 2 0 100 91cfbc00 net-2
1454 1454 2 0 100 918dbb40 syslogd
0 309 2 0 240 918db880 hythygtemp2
0 119 2 0 200 91628600 usbtask-dr
0 107 2 0 240 91620240 entbutler
0 > 100 1 3 201 914e1a00 idle/3
0 > 29 1 2 201 914dd700 idle/2
0 7 2 0 200 914ae080 xcall/0
0 3 2 0 200 914ac300 softnet/0
From: Frank Kardel <kardel@netbsd.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/57580: evbarm/earmv7hf RPI2 scheduler/cpu stall
Date: Fri, 25 Aug 2023 11:27:09 +0200
I now ran -current with option HEARTBEAT for 635730 seconds.
Then HEARTBEAT checks triggered:
Aug 24 19:42:22 rpi /netbsd: [ 635730.0017400] cpu0: found cpu1 heart
stopped beating after 16 seconds
Aug 24 19:42:22 rpi /netbsd: [ 635730.0068278] panic: cpu1[29860 perl]:
heart stopped beating
Aug 24 19:42:22 rpi /netbsd: [ 635730.0125393] cpu1: Begin traceback...
Aug 24 19:42:22 rpi /netbsd: [ 635730.0163590] 0xbb29b71c:
netbsd:db_panic+0x14
Aug 24 19:42:22 rpi /netbsd: [ 635730.0208674] 0xbb29b73c:
netbsd:vpanic+0x114
Aug 24 19:42:22 rpi /netbsd: [ 635730.0252987] 0xbb29b754: netbsd:panic+0x24
Aug 24 19:42:22 rpi /netbsd: [ 635730.0295481] 0xbb29b774:
netbsd:defibrillator+0x54
Aug 24 19:42:22 rpi /netbsd: [ 635730.0360377] 0xbb29b7bc:
netbsd:ipi_msg_cpu_handler+0x9c
Aug 24 19:42:22 rpi /netbsd: [ 635730.0416531] 0xbb29b804:
netbsd:ipi_cpu_handler+0xdc
Aug 24 19:42:22 rpi /netbsd: [ 635730.0467708] 0xbb29b814:
netbsd:pic_ipi_generic+0x10
Aug 24 19:42:22 rpi /netbsd: [ 635730.0518910] 0xbb29b83c:
netbsd:bcm2836mp_ipi_handler+0xbc
Aug 24 19:42:22 rpi /netbsd: [ 635730.0575454] 0xbb29b88c:
netbsd:pic_dispatch+0x54
Aug 24 19:42:22 rpi /netbsd: [ 635730.0624438] 0xbb29b914:
netbsd:pic_do_pending_ints+0x434
Aug 24 19:42:22 rpi /netbsd: [ 635730.0679535] 0xbb29ba64:
netbsd:irq_entry+0x94
Aug 24 19:42:22 rpi /netbsd: [ 635730.0725627] cpu1: End traceback...
Aug 24 19:42:22 rpi /netbsd: [ 635730.0764817] Mutex error:
mutex_vector_enter,515: assertion failed: !cpu_intr_p()
Aug 24 19:42:22 rpi /netbsd:
Aug 24 19:42:22 rpi /netbsd: [ 635730.0840143] lock address : 917cf07c
Aug 24 19:42:22 rpi /netbsd: [ 635730.0877541] current cpu
: 1
Aug 24 19:42:22 rpi /netbsd: [ 635730.0923754] current lwp :
0x00000000918dd300
Aug 24 19:42:22 rpi /netbsd: [ 635730.0969675] owner field :
000000000000000000 wait/spin: 0/0
Aug 24 19:42:22 rpi /netbsd:
Aug 24 19:42:22 rpi /netbsd: [ 635730.1043366] Skipping crash dump on
recursive panic
Aug 24 19:42:22 rpi /netbsd: [ 635730.1093675] panic: lock error: Mutex:
mutex_vector_enter,515: assertion failed: !cpu_intr_p(): lock 0x917cf07c
cpu 1 lwp 0x918dd300
Aug 24 19:42:22 rpi /netbsd: [ 635730.1214484] cpu1: Begin traceback...
Aug 24 19:42:22 rpi /netbsd: [ 635730.1252763] 0xbb29b414:
netbsd:db_panic+0x14
Aug 24 19:42:22 rpi /netbsd: [ 635730.1297859] 0xbb29b434:
netbsd:vpanic+0x114
Aug 24 19:42:22 rpi /netbsd: [ 635730.1341996] 0xbb29b44c: netbsd:panic+0x24
Aug 24 19:42:22 rpi /netbsd: [ 635730.1384551] 0xbb29b514:
netbsd:lockdebug_abort+0xe8
Aug 24 19:42:22 rpi /netbsd: [ 635730.1443758] 0xbb29b52c:
netbsd:mutex_abort+0x30
Aug 24 19:42:22 rpi /netbsd: [ 635730.1492993] 0xbb29b59c:
netbsd:mutex_enter+0x4a0
Aug 24 19:42:22 rpi /netbsd: [ 635730.1542221] 0xbb29b5d4:
netbsd:dk_open+0x3c
Aug 24 19:42:22 rpi /netbsd: [ 635730.1586363] 0xbb29b604:
netbsd:ldopen+0x78
Aug 24 19:42:22 rpi /netbsd: [ 635730.1632125] 0xbb29b634:
netbsd:dk_size+0x64
Aug 24 19:42:22 rpi /netbsd: [ 635730.1676452] 0xbb29b674:
netbsd:bdev_size+0xac
Aug 24 19:42:22 rpi /netbsd: [ 635730.1722500] 0xbb29b684:
netbsd:cpu_dumpconf+0x28
Aug 24 19:42:22 rpi /netbsd: [ 635730.1771197] 0xbb29b6d4:
netbsd:dodumpsys+0x1fc
Aug 24 19:42:22 rpi /netbsd: [ 635730.1817926] 0xbb29b704:
netbsd:dumpsys+0x18
Aug 24 19:42:22 rpi /netbsd: [ 635730.1862678] 0xbb29b71c:
netbsd:kern_reboot+0x80
Aug 24 19:42:22 rpi /netbsd: [ 635730.1910014] 0xbb29b73c:
netbsd:vpanic+0x120
Aug 24 19:42:22 rpi /netbsd: [ 635730.1954316] 0xbb29b754: netbsd:panic+0x24
Aug 24 19:42:22 rpi /netbsd: [ 635730.1996777] 0xbb29b774:
netbsd:defibrillator+0x54
... stack continues as in first stack dump
So current status is
It happens on evbarm RPI2.
It is not too frequent
It happens with NetBSD-10.0_BETA
It happens with -current (seems to take longer)
It looks like perl always the victim that is set runnable and then
cpu1 gets a silent heart attack
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.47 2022/09/11 19:34:41 kim Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2023
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home