NetBSD Problem Report #53043
From nick@jetsontk1.localdomain Wed Feb 21 08:52:13 2018
Return-Path: <nick@jetsontk1.localdomain>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id DD17F7A104
for <gnats-bugs@gnats.NetBSD.org>; Wed, 21 Feb 2018 08:52:13 +0000 (UTC)
Message-Id: <20180221085209.B7296476BA5@jetsontk1.localdomain>
Date: Wed, 21 Feb 2018 08:52:09 +0000 (GMT)
From: skrll@netbsd.org
Reply-To: skrll@netbsd.org
To: gnats-bugs@NetBSD.org
Subject: deadlock on evbarm/TEGRA with netbsd-8
X-Send-Pr-Version: 3.95
>Number: 53043
>Category: kern
>Synopsis: deadlock on evbarm/TEGRA with netbsd-8
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: ozaki-r
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Wed Feb 21 08:55:00 +0000 2018
>Closed-Date: Mon Mar 05 16:52:17 +0000 2018
>Last-Modified: Wed May 29 16:05:01 +0000 2019
>Originator: Nick Hudson
>Release: NetBSD 8.0_BETA
>Organization:
>Environment:
System: NetBSD jetsontk1 8.0_BETA NetBSD 8.0_BETA (TEGRA) #1: Wed Feb 21 06:39:18 GMT 2018 nick@jetsontk1:/var/obj/usr/src/sys/arch/evbarm/compile/TEGRA evbarm
Architecture: earmv7hf
Machine: evbarm
>Description:
db{2}> ps/l
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
25433 1 3 0 80 936e19e0 setkey kqueue
5053 1 3 0 80 935596e0 atf-check wait
24095 36 3 3 80 936d2be0 shmif kqueue
24095 35 3 2 80 936ace80 rump_server nanoslp
24095 34 3 2 80 937208a0 rump_server select
24095 33 3 1 80 91d096e0 vmem_rehash parked
24095 32 3 2 80 935536a0 aiodoned parked
24095 29 3 1 80 936db180 pmfsuspend parked
24095 28 3 3 80 92f565e0 pmfevent parked
24095 27 3 3 80 93728b80 rt_free parked
24095 26 3 1 80 92d286e0 key_timehandler parked
24095 25 3 0 80 936a8ba0 carp6_wqinput/0 parked
24095 24 3 3 80 936ac380 carp6_wqinput/1 parked
24095 23 3 2 80 93558660 carp_wqinput/0 parked
24095 22 3 3 80 93744be0 carp_wqinput/1 parked
24095 21 3 3 80 935a51c0 icmp_wqinput/0 parked
24095 20 3 3 80 93684b00 icmp_wqinput/1 parked
24095 19 3 3 80 9372f960 rt_timer parked
24095 18 3 1 80 93719900 ipflow_slowtimo parked
24095 17 3 0 80 9368c300 rsi1/3 parked
24095 16 3 2 80 92cd6140 rsi0/3 parked
24095 15 3 1 80 93760160 ioflush nanoslp
24095 14 3 3 80 9355a940 cachegc nanoslp
24095 13 3 0 80 92d28420 vdrain parked
24095 12 3 3 80 9369ce40 pdaemon parked
24095 11 3 3 80 9355cee0 xcall/1 parked
24095 10 3 2 80 937456e0 sipbnc parked
24095 9 3 1 80 9370d360 rumpclk1 nanoslp
24095 8 3 2 80 936dac00 xcall/0 parked
24095 7 3 3 80 93729940 rsi1/2 parked
24095 6 3 3 80 935599a0 rsi0/2 parked
24095 5 3 2 80 9355d9e0 sipbnc parked
24095 4 3 1 80 9355ac00 rumpclk0 nanoslp
24095 3 3 3 80 937609a0 rsi1/1 parked
24095 2 3 2 80 936f11e0 rsi0/1 parked
24095 1 3 2 80 935a4980 rump_server psem
4771 36 3 2 80 936834e0 shmif kqueue
4771 35 3 3 80 9370c2e0 rump_server parked
4771 34 3 1 80 93553960 rump_server select
4771 33 3 3 80 9370d0a0 vmem_rehash parked
4771 32 3 3 80 92eb82e0 aiodoned parked
4771 29 3 0 80 936821a0 pmfsuspend parked
4771 28 3 3 80 92cd66c0 pmfevent parked
4771 27 3 3 80 93554640 rt_free parked
4771 26 3 1 80 9368c040 key_timehandler parked
4771 25 3 3 80 936f0c60 carp6_wqinput/0 parked
4771 24 3 3 80 93737400 carp6_wqinput/1 parked
4771 23 3 1 80 93695be0 carp_wqinput/0 parked
4771 22 3 3 80 93709b80 carp_wqinput/1 parked
4771 21 3 1 80 92eb8de0 icmp_wqinput/0 parked
4771 20 3 3 80 936a9120 icmp_wqinput/1 parked
4771 19 3 3 80 935a8f20 rt_timer parked
4771 18 3 2 80 93719640 ipflow_slowtimo parked
4771 17 3 2 80 91beee80 rsi1/3 parked
4771 16 3 1 80 93750100 rsi0/3 parked
4771 15 3 3 80 936e03e0 ioflush nanoslp
4771 14 3 3 80 92189720 cachegc nanoslp
4771 13 3 2 80 936e6c40 vdrain parked
4771 12 3 3 80 92f57920 pdaemon parked
4771 11 3 2 80 92f5d940 xcall/1 parked
4771 10 3 2 80 93768440 sipbnc parked
4771 9 3 1 80 935580e0 rumpclk1 nanoslp
4771 8 3 3 80 9375c980 xcall/0 parked
4771 7 3 2 80 93728080 rsi1/2 parked
4771 6 3 3 80 9370cde0 rsi0/2 parked
4771 5 3 1 80 92d3da40 sipbnc parked
4771 4 3 1 80 91bee0c0 rumpclk0 nanoslp
4771 3 3 3 80 92d3c9c0 rsi1/1 parked
4771 2 3 3 80 93553c20 rsi0/1 parked
4771 1 3 0 80 936898e0 rump_server psem
17 1 3 0 80 92d3d200 sh wait
1976 1 3 2 80 91d09420 ksh ttyraw
1437 1 3 0 80 92eb9ba0 ksh pause
22492 1 3 0 80 92cd6c40 sshd select
23450 1 3 2 80 92f32040 sshd select
854 1 3 1 80 91bee380 atf-run select
938 1 3 3 80 91dbcec0 ksh pause
1003 1 3 3 80 91a7b940 login wait
923 1 3 1 80 92189460 cron nanoslp
815 1 3 2 80 91d08be0 inetd kqueue
762 1 3 1 80 91d09160 qmgr kqueue
572 1 3 2 80 91d08ea0 pickup kqueue
812 1 3 0 80 92188960 master kqueue
627 1 3 3 80 92188ee0 sshd select
516 1 3 0 1000000 91dbc100 ntpd tstile
313 1 3 3 80 91dbd180 syslogd kqueue
255 1 3 2 0 91dbd700 mdnsd tstile
247 1 3 0 0 91d080e0 dhcpcd tstile
1 1 3 3 80 91b9a8e0 init wait
0 86 5 2 200 91d099a0 (zombie)
0 83 3 0 200 91b9a0a0 physiod physiod
0 82 3 1 200 91bef140 usb3 usbevt
0 81 3 2 200 91bef400 usb2 usbevt
0 80 3 1 200 91bef6c0 aiodoned aiodoned
0 79 3 0 200 91bef980 ioflush syncer
0 78 3 1 200 91befc40 pgdaemon pgdaemon
0 74 3 3 200 91a7bc00 usb1 usbevt
0 73 3 0 200 91b9a360 usb0 usbevt
0 72 3 1 200 91b9a620 npfgc-0 npfgccv
0 71 3 0 200 91b9aba0 rt_free rt_free
0 70 3 0 200 91b9ae60 unpgc unpgc
0 69 3 3 200 91b9b6a0 key_timehandler key_timehandler
0 68 3 3 200 91b9b120 icmp6_wqinput/3 icmp6_wqinput
0 67 3 2 200 91a7b100 icmp6_wqinput/2 icmp6_wqinput
0 66 3 1 200 91b9b3e0 icmp6_wqinput/1 icmp6_wqinput
0 65 3 0 200 91a7b3c0 icmp6_wqinput/0 icmp6_wqinput
0 63 3 3 200 91a7ae40 nd6_timer tstile
0 61 3 3 200 91b9b960 icmp_wqinput/3 icmp_wqinput
0 60 3 2 200 91b9bc20 icmp_wqinput/2 icmp_wqinput
0 59 3 1 200 91a7a080 icmp_wqinput/1 icmp_wqinput
0 58 3 0 200 91a7b680 icmp_wqinput/0 icmp_wqinput
0 57 3 3 200 91a7ab80 rt_timer rt_timer
0 56 3 2 200 91a7a8c0 vmem_rehash vmem_rehash
0 55 3 3 200 91a7a340 sdmmc1 mmctaskq
0 54 3 1 200 91a7a600 sdmmc0 mmctaskq
0 45 3 0 200 91926060 usbtask-dr usbtsk
0 44 3 0 200 91926320 usbtask-hc usbtsk
0 43 3 0 280 919265e0 spkr0 bellcv
0 42 3 1 280 919268a0 audiomix play
0 41 3 1 280 91926b60 audiorec record
0 40 3 1 200 91926e20 atabus0 atath
0 39 3 1 200 919270e0 iic4 iicintr
0 38 3 1 200 919273a0 iic3 iicintr
0 37 3 3 200 91927660 iic2 iicintr
0 36 3 3 200 91927920 iic1 iicintr
0 35 3 3 200 91927be0 iic0 iicintr
0 34 3 3 200 918fe040 xcall/3 xcall
0 33 1 3 200 918fe300 softser/3
0 32 1 3 200 918fe5c0 softclk/3
0 31 1 3 200 918fe880 softbio/3
0 30 3 3 200 918feb40 softnet/3 tstile
0 > 29 7 3 201 918fee00 idle/3
0 28 3 2 200 918ff0c0 xcall/2 xcall
0 27 1 2 200 918ff380 softser/2
0 26 3 2 200 918ff640 softclk/2 tstile
0 25 1 2 200 918ff900 softbio/2
0 24 3 2 200 918ffbc0 softnet/2 tstile
0 > 23 7 2 201 915a8020 idle/2
0 22 3 1 200 915a82e0 xcall/1 xcall
0 21 1 1 200 915a85a0 softser/1
0 20 3 1 200 915a8860 softclk/1 tstile
0 19 1 1 200 915a8b20 softbio/1
0 18 3 1 200 915a8de0 softnet/1 tstile
0 > 17 7 1 201 915a90a0 idle/1
0 16 3 0 200 915a9360 lnxsyswq lnxsyswq
0 15 3 0 200 915a9620 sysmon smtaskq
0 14 3 2 200 915a98e0 pmfsuspend pmfsuspend
0 13 3 0 200 915a9ba0 pmfevent pmfevent
0 12 3 0 200 91590000 sopendfree sopendfr
0 11 3 1 200 915902c0 nfssilly nfssilly
0 10 3 1 200 91590580 cachegc cachegc
0 9 3 0 200 91590840 vdrain vdrain
0 8 3 0 200 91590b00 modunload mod_unld
0 7 3 0 200 91590dc0 xcall/0 xcall
0 6 1 0 200 91591080 softser/0
0 5 1 0 200 91591340 softclk/0
0 4 1 0 200 91591600 softbio/0
0 3 3 0 200 915918c0 softnet/0 tstile
0 > 2 7 0 201 91591b80 idle/0
0 1 3 0 200 81768020 swapper uvm
db{2}> ps/w
PID LID COMMAND EMUL PRI WAIT-MSG WAIT-CHANNEL
25433 1 setkey netbsd 33 kqueue 92cde0f4
5053 1 atf-check netbsd 33 wait 92d1a810
24095 36 rump_server netbsd 36 kqueue 9211ffdc
24095 35 rump_server netbsd 38 nanoslp 936ace80
24095 34 rump_server netbsd 38 select 91572480
24095 33 rump_server netbsd 33 parked c9340170
24095 32 rump_server netbsd 33 parked ce55c070
24095 29 rump_server netbsd 33 parked f12c3a70
24095 28 rump_server netbsd 33 parked f12c39f0
24095 27 rump_server netbsd 33 parked f12c3670
24095 26 rump_server netbsd 33 parked f12c3530
24095 25 rump_server netbsd 33 parked f12c31f0
24095 24 rump_server netbsd 33 parked f12c3170
24095 23 rump_server netbsd 33 parked f12c30b0
24095 22 rump_server netbsd 33 parked f12c3030
24095 21 rump_server netbsd 33 parked f12c2db0
24095 20 rump_server netbsd 36 parked f12c2d30
24095 19 rump_server netbsd 33 parked f12c2c70
24095 18 rump_server netbsd 33 parked f12c2bb0
24095 17 rump_server netbsd 36 parked f126e870
24095 16 rump_server netbsd 33 parked f126e7f0
24095 15 rump_server netbsd 43 nanoslp 93760160
24095 14 rump_server netbsd 43 nanoslp 9355a940
24095 13 rump_server netbsd 33 parked f126f830
24095 12 rump_server netbsd 34 parked f12542f0
24095 11 rump_server netbsd 38 parked f126ebb0
24095 10 rump_server netbsd 34 parked f126edb0
24095 9 rump_server netbsd 34 nanoslp 9370d360
24095 8 rump_server netbsd 38 parked f126ebb0
24095 7 rump_server netbsd 33 parked f126e870
24095 6 rump_server netbsd 34 parked f126e7f0
24095 5 rump_server netbsd 34 parked f126edb0
24095 4 rump_server netbsd 34 nanoslp 9355ac00
24095 3 rump_server netbsd 34 parked f126e870
24095 2 rump_server netbsd 34 parked f126e7f0
24095 1 rump_server netbsd 33 psem 9358e584
4771 36 rump_server netbsd 36 kqueue 9371d03c
4771 35 rump_server netbsd 38 parked fec51bcc
4771 34 rump_server netbsd 38 select 91572680
4771 33 rump_server netbsd 30 parked f6a01298
4771 32 rump_server netbsd 30 parked f939ec18
4771 29 rump_server netbsd 30 parked f939e998
4771 28 rump_server netbsd 30 parked f939ea18
4771 27 rump_server netbsd 30 parked f939e598
4771 26 rump_server netbsd 30 parked f939e6d8
4771 25 rump_server netbsd 30 parked f939e218
4771 24 rump_server netbsd 30 parked f939e298
4771 23 rump_server netbsd 30 parked f939e358
4771 22 rump_server netbsd 30 parked f939e3d8
4771 21 rump_server netbsd 36 parked f939fe58
4771 20 rump_server netbsd 30 parked f939fed8
4771 19 rump_server netbsd 30 parked f939ff98
4771 18 rump_server netbsd 30 parked f939f858
4771 17 rump_server netbsd 36 parked f9333b98
4771 16 rump_server netbsd 36 parked f9333418
4771 15 rump_server netbsd 43 nanoslp 936e03e0
4771 14 rump_server netbsd 43 nanoslp 92189720
4771 13 rump_server netbsd 30 parked f9332bd8
4771 12 rump_server netbsd 30 parked f9309118
4771 11 rump_server netbsd 38 parked f9333858
4771 10 rump_server netbsd 30 parked f9333e58
4771 9 rump_server netbsd 34 nanoslp 935580e0
4771 8 rump_server netbsd 38 parked f9333858
4771 7 rump_server netbsd 34 parked f9333b98
4771 6 rump_server netbsd 30 parked f9333418
4771 5 rump_server netbsd 30 parked f9333e58
4771 4 rump_server netbsd 30 nanoslp 91bee0c0
4771 3 rump_server netbsd 30 parked f9333b98
4771 2 rump_server netbsd 30 parked f9333418
4771 1 rump_server netbsd 30 psem 9369e6a4
17 1 sh netbsd 34 wait 92e92a18
1976 1 ksh netbsd 37 ttyraw 92ecfa30
1437 1 ksh netbsd 43 pause 92eb9ba0
22492 1 sshd netbsd 43 select 91573a80
23450 1 sshd netbsd 43 select 91572480
854 1 atf-run netbsd 40 select 91572680
938 1 ksh netbsd 43 pause 91dbcec0
1003 1 login netbsd 43 wait 91bbdba8
923 1 cron netbsd 43 nanoslp 92189460
815 1 inetd netbsd 43 kqueue 9211fbcc
762 1 qmgr netbsd 43 kqueue 9211f21c
572 1 pickup netbsd 43 kqueue 9211fa8c
812 1 master netbsd 43 kqueue 9211f08c
627 1 sshd netbsd 43 select 915722c0
516 1 ntpd netbsd 43 tstile 91567f40
313 1 syslogd netbsd 43 kqueue 91a44774
255 1 mdnsd netbsd 43 tstile 9158c900
247 1 dhcpcd netbsd 43 tstile 91567f40
1 1 init netbsd 43 wait 91bbdda0
0 86 system netbsd 96 0
0 83 system netbsd 123 physiod 91d49f04
0 82 system netbsd 96 usbevt 91a654b0
0 81 system netbsd 96 usbevt 91a6525c
0 80 system netbsd 125 aiodoned 915a6784
0 79 system netbsd 124 syncer 91bef980
0 78 system netbsd 126 pgdaemon 818c7a00
0 74 system netbsd 96 usbevt 91a64254
0 73 system netbsd 96 usbevt 91a64a54
0 72 system netbsd 96 npfgccv 91a4070c
0 71 system netbsd 222 rt_free 915a6b04
0 70 system netbsd 96 unpgc 8193a1b8
0 69 system netbsd 222 key_timehandler 915a6bc4
0 68 system netbsd 222 icmp6_wqinput 91b97684
0 67 system netbsd 222 icmp6_wqinput 91b97644
0 66 system netbsd 222 icmp6_wqinput 91b97604
0 65 system netbsd 222 icmp6_wqinput 91b975c4
0 63 system netbsd 222 tstile 91567f40
0 61 system netbsd 222 icmp_wqinput 91b97f44
0 60 system netbsd 222 icmp_wqinput 91b97f04
0 59 system netbsd 222 icmp_wqinput 91b97ec4
0 58 system netbsd 222 icmp_wqinput 91b97e84
0 57 system netbsd 222 rt_timer 915a7904
0 56 system netbsd 125 vmem_rehash 915a79c4
0 55 system netbsd 123 mmctaskq 91a523d4
0 54 system netbsd 123 mmctaskq 91a526d4
0 45 system netbsd 96 usbtsk 818b7124
0 44 system netbsd 96 usbtsk 818b7104
0 43 system netbsd 123 bellcv 9191657c
0 42 system netbsd 223 play 91a5a660
0 41 system netbsd 223 record 91a5a670
0 40 system netbsd 96 atath 915d0230
0 39 system netbsd 96 iicintr 91583b24
0 38 system netbsd 96 iicintr 91583ba4
0 37 system netbsd 96 iicintr 91583da4
0 36 system netbsd 96 iicintr 91583e24
0 35 system netbsd 96 iicintr 91583f24
0 34 system netbsd 127 xcall 91586880
0 33 system netbsd 223 0
0 32 system netbsd 220 0
0 31 system netbsd 221 0
0 30 system netbsd 222 tstile 91567f40
0 > 29 system netbsd 0 0
0 28 system netbsd 127 xcall 91587080
0 27 system netbsd 223 0
0 26 system netbsd 220 tstile 91567f40
0 25 system netbsd 221 0
0 24 system netbsd 222 tstile 91567f40
0 > 23 system netbsd 0 0
0 22 system netbsd 127 xcall 9155d880
0 21 system netbsd 223 0
0 20 system netbsd 220 tstile 91567f40
0 19 system netbsd 221 0
0 18 system netbsd 222 tstile 91567f40
0 > 17 system netbsd 0 0
0 16 system netbsd 43 lnxsyswq 915a7d44
0 15 system netbsd 96 smtaskq 818b8408
0 14 system netbsd 43 pmfsuspend 915a7e44
0 13 system netbsd 43 pmfevent 915a7f04
0 12 system netbsd 96 sopendfr 8193a17c
0 11 system netbsd 43 nfssilly 91594684
0 10 system netbsd 125 cachegc 91590580
0 9 system netbsd 125 vdrain 81891e00
0 8 system netbsd 125 mod_unld 81929c64
0 7 system netbsd 127 xcall 81732618
0 6 system netbsd 223 0
0 5 system netbsd 220 0
0 4 system netbsd 221 0
0 3 system netbsd 222 tstile 91567f40
0 > 2 system netbsd 0 0
0 1 system netbsd 125 uvm 81768020
db{2}>
516 1 3 0 1000000 91dbc100 ntpd tstile
255 1 3 2 0 91dbd700 mdnsd tstile
247 1 3 0 0 91d080e0 dhcpcd tstile
0 63 3 3 200 91a7ae40 nd6_timer tstile
0 30 3 3 200 918feb40 softnet/3 tstile
0 26 3 2 200 918ff640 softclk/2 tstile
0 24 3 2 200 918ffbc0 softnet/2 tstile
0 20 3 1 200 915a8860 softclk/1 tstile
0 18 3 1 200 915a8de0 softnet/1 tstile
0 3 3 0 200 915918c0 softnet/0 tstile
516 1 ntpd netbsd 43 tstile 91567f40
255 1 mdnsd netbsd 43 tstile 9158c900
247 1 dhcpcd netbsd 43 tstile 91567f40
0 63 system netbsd 222 tstile 91567f40
0 30 system netbsd 222 tstile 91567f40
0 26 system netbsd 220 tstile 91567f40
0 24 system netbsd 222 tstile 91567f40
0 20 system netbsd 220 tstile 91567f40
0 18 system netbsd 222 tstile 91567f40
0 3 system netbsd 222 tstile 91567f40
db{2}> x/x 91567f40
91567f40: 91dbd702
db{2}> x/x 9158c900
9158c900: 91d080e2
db{2}>
db{2}> bt/a 91dbd700
trace: pid 255 lid 1 at 0x9c997d04
0x9c997d04: netbsd:mi_switch+0x10
0x9c997d34: netbsd:sleepq_block+0xb4
0x9c997d74: netbsd:turnstile_block+0x3a0
0x9c997dd4: netbsd:mutex_enter+0x2e0
0x9c997dfc: netbsd:ip_freemoptions+0x3c
0x9c997e1c: netbsd:in_pcbdetach+0xd0
0x9c997e3c: netbsd:udp_detach_wrapper+0x40
0x9c997e64: netbsd:soclose+0x80
0x9c997e7c: netbsd:soo_close+0x20
0x9c997ebc: netbsd:closef+0x6c
0x9c997f1c: netbsd:fd_close+0x298
0x9c997f34: netbsd:sys_close+0x30
0x9c997fac: netbsd:syscall+0x104
--- tf 0x9c997fb0 ---
db{2}> print netbsd:ip_freemoptions+0x3c
81173cf8
db{2}> print netbsd:in_pcbdetach+0xd0
8116a7b8
db{2}> print netbsd:udp_detach_wrapper+0x40
8118db68
db{2}> print netbsd:soclose+0x80
813efc48
$ addr2line -e netbsd.gdb -f 813efc48
soclose
/usr/src/sys/kern/uipc_socket.c:778
$ addr2line -e netbsd.gdb -f 81173cf8
ip_freemoptions
/usr/src/sys/netinet/ip_output.c:2039 (discriminator 3)
$
$ addr2line -e netbsd.gdb -f 8116a7b8
in_pcbdetach
/usr/src/sys/netinet/in_pcb.c:645
$
$ addr2line -e netbsd.gdb -f 8118db68
udp_detach_wrapper
/usr/src/sys/netinet/udp_usrreq.c:1379
$
db{2}> bt/a 91d080e0
trace: pid 247 lid 1 at 0x9c909c14
0x9c909c14: netbsd:mi_switch+0x10
0x9c909c44: netbsd:sleepq_block+0xb4
0x9c909c84: netbsd:turnstile_block+0x3a0
0x9c909ce4: netbsd:mutex_enter+0x2e0
0x9c909d44: netbsd:in6_control+0x84
0x9c909d64: netbsd:udp6_ioctl_wrapper+0x38
0x9c909e6c: netbsd:doifioctl+0xbbc
0x9c909f34: netbsd:sys_ioctl+0x1b8
0x9c909fac: netbsd:syscall+0x104
--- tf 0x9c909fb0 ---
db{2}>
db{2}> print netbsd:in6_control+0x84
8119aaf4
db{2}> print netbsd:udp6_ioctl_wrapper+0x38
811bd650
db{2}> print netbsd:doifioctl+0xbbc
814466e4
db{2}> show lock 91567f40
lock address : 0x0000000091567f40 type : sleep/adaptive
initialized : 0x0000000081377dac
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 9
current cpu : 2 last held: 2
current lwp : 0x00000000915a8020 last held: 0x0000000091dbd700
last locked* : 0x00000000813efbe8 unlocked : 0x00000000813f0e5c
owner field : 0x0000000091dbd700 wait/spin: 1/0
Turnstile chain at 0x81890e80.
=> Turnstile at 0x91592c78 (wrq=0x91592c88, rdq=0x91592c90).
=> 0 waiting readers:
=> 9 waiting writers: 0x915918c0 0x91d080e0 0x91a7ae40 0x918ffbc0 0x918feb40 0x915a8de0 0x915a8860 0x918ff640 0x91dbc100
$ addr2line -e netbsd.gdb -f 0x00000000813efbe8
solock
/usr/src/sys/sys/socketvar.h:495
$
489: solock(struct socket *so)
490: {
491: kmutex_t *lock;
492:
493: lock = so->so_lock;
494: mutex_enter(lock);
db{2}> show lock 9158c900
lock address : 0x000000009158c900 type : sleep/adaptive
initialized : 0x0000000081377dac
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 1
current cpu : 2 last held: 0
current lwp : 0x00000000915a8020 last held: 0x0000000091d080e0
last locked* : 0x0000000081445f5c unlocked : 0x0000000081445ff0
owner field : 0x0000000091d080e0 wait/spin: 1/0
Turnstile chain at 0x81890e40.
=> Turnstile at 0x91592bd0 (wrq=0x91592be0, rdq=0x91592be8).
=> 0 waiting readers:
=> 1 waiting writers: 0x91dbd700
$ addr2line -e netbsd.gdb -f 0x0000000081445f5c
doifioctl
/usr/src/sys/net/if.c:3226
1.1 cgd 3222:
1.394.2.4 snj 3223: KERNEL_LOCK_UNLESS_IFP_MPSAFE(ifp);
3224: IFNET_LOCK(ifp);
1.336 ozaki-r 3225:
1.231 dyoung 3226: error = (*ifp->if_ioctl)(ifp, cmd, data);
so, IFNET_LOCK(ifp)
>How-To-Repeat:
run atf tests
>Fix:
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: kern-bug-people->ozaki-r
Responsible-Changed-By: ozaki-r@NetBSD.org
Responsible-Changed-When: Thu, 22 Feb 2018 07:23:09 +0000
Responsible-Changed-Why:
take
From: Ryota Ozaki <ozaki-r@netbsd.org>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/53043: deadlock on evbarm/TEGRA with netbsd-8
Date: Thu, 22 Feb 2018 17:24:13 +0900
This is a deadlock that occurs between softnet_lock and IFNET_LOCK;
the locks can be held in different orders.
softnet_lock in in6_control (and in_control) had been introduced to
address PR 51356 that was a race condition between ioctls and packet
inputs and forwarding (ipintr and ip6intr). However holding it
in6_control and in_control was not a good idea because softnet_lock
should be basically held at the very beginning of call paths to comply
the locking order and the functions were in the middle of call paths.
I think we have two options to solve the issue:
(1) Give up relying on softnet_lock to protect the network stack and
remove softnet_lock from in6_control/in_control and add some
KERNEL_LOCK to the network stack, e.g., ipintr and ip6intr.
(2) Just get rid of softnet_lock from in6_control/in_control.
(1) is safer than (2) but add some performance penalty. (2) sounds
awful but the situation is the same as netbsd-7 and netbsd-6, i.e.,
it's enough safe in practice...
A patch for (1) is here: http://www.netbsd.org/~ozaki-r/fix-pr53043.diff
Note that I gave up moving softnet_lock to doifioctl because it
just introduced other deadlocks and was more problematic.
Comments?
ozaki-r
From: Nick Hudson <skrll@netbsd.org>
To: gnats-bugs@NetBSD.org, ozaki-r@NetBSD.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, skrll@netbsd.org
Cc:
Subject: Re: kern/53043: deadlock on evbarm/TEGRA with netbsd-8
Date: Thu, 22 Feb 2018 20:59:22 +0000
On 02/22/18 08:25, Ryota Ozaki wrote:
> The following reply was made to PR kern/53043; it has been noted by GNATS.
>
> From: Ryota Ozaki <ozaki-r@netbsd.org>
> To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
> Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
> Subject: Re: kern/53043: deadlock on evbarm/TEGRA with netbsd-8
> Date: Thu, 22 Feb 2018 17:24:13 +0900
>
> This is a deadlock that occurs between softnet_lock and IFNET_LOCK;
> the locks can be held in different orders.
>
> softnet_lock in in6_control (and in_control) had been introduced to
> address PR 51356 that was a race condition between ioctls and packet
> inputs and forwarding (ipintr and ip6intr). However holding it
> in6_control and in_control was not a good idea because softnet_lock
> should be basically held at the very beginning of call paths to comply
> the locking order and the functions were in the middle of call paths.
>
> I think we have two options to solve the issue:
> (1) Give up relying on softnet_lock to protect the network stack and
> remove softnet_lock from in6_control/in_control and add some
> KERNEL_LOCK to the network stack, e.g., ipintr and ip6intr.
> (2) Just get rid of softnet_lock from in6_control/in_control.
>
> (1) is safer than (2) but add some performance penalty. (2) sounds
> awful but the situation is the same as netbsd-7 and netbsd-6, i.e.,
> it's enough safe in practice...
>
> A patch for (1) is here: http://www.netbsd.org/~ozaki-r/fix-pr53043.diff
>
> Note that I gave up moving softnet_lock to doifioctl because it
> just introduced other deadlocks and was more problematic.
>
> Comments?
>
I think 1) is the only real option for now. Hopefully someone can
address finer grained locking soon.
I tested your patch and my tegra can now complete an atf-run.
Thanks,
Nick
From: Ryota Ozaki <ozaki-r@netbsd.org>
To: Nick Hudson <skrll@netbsd.org>
Cc: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/53043: deadlock on evbarm/TEGRA with netbsd-8
Date: Fri, 23 Feb 2018 18:02:02 +0900
On Fri, Feb 23, 2018 at 5:59 AM, Nick Hudson <skrll@netbsd.org> wrote:
> On 02/22/18 08:25, Ryota Ozaki wrote:
>>
>> The following reply was made to PR kern/53043; it has been noted by GNATS.
>>
>> From: Ryota Ozaki <ozaki-r@netbsd.org>
>> To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
>> Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
>> netbsd-bugs@netbsd.org
>> Subject: Re: kern/53043: deadlock on evbarm/TEGRA with netbsd-8
>> Date: Thu, 22 Feb 2018 17:24:13 +0900
>>
>> This is a deadlock that occurs between softnet_lock and IFNET_LOCK;
>> the locks can be held in different orders.
>> softnet_lock in in6_control (and in_control) had been introduced to
>> address PR 51356 that was a race condition between ioctls and packet
>> inputs and forwarding (ipintr and ip6intr). However holding it
>> in6_control and in_control was not a good idea because softnet_lock
>> should be basically held at the very beginning of call paths to comply
>> the locking order and the functions were in the middle of call paths.
>> I think we have two options to solve the issue:
>> (1) Give up relying on softnet_lock to protect the network stack and
>> remove softnet_lock from in6_control/in_control and add some
>> KERNEL_LOCK to the network stack, e.g., ipintr and ip6intr.
>> (2) Just get rid of softnet_lock from in6_control/in_control.
>> (1) is safer than (2) but add some performance penalty. (2) sounds
>> awful but the situation is the same as netbsd-7 and netbsd-6, i.e.,
>> it's enough safe in practice...
>> A patch for (1) is here:
>> http://www.netbsd.org/~ozaki-r/fix-pr53043.diff
>> Note that I gave up moving softnet_lock to doifioctl because it
>> just introduced other deadlocks and was more problematic.
>> Comments?
>>
>
>
> I think 1) is the only real option for now. Hopefully someone can address
> finer grained locking soon.
>
> I tested your patch and my tegra can now complete an atf-run.
Thanks.
I think we should think of a way to enable NET_MPSAFE coexisting with
non-MP-safe components...
ozaki-r
From: "Ryota Ozaki" <ozaki-r@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/53043 CVS commit: src/sys
Date: Sat, 24 Feb 2018 07:37:09 +0000
Module Name: src
Committed By: ozaki-r
Date: Sat Feb 24 07:37:09 UTC 2018
Modified Files:
src/sys/netinet: in.c ip_input.c wqinput.c
src/sys/netinet6: in6.c ip6_input.c
src/sys/rump/net/lib/libnetinet: netinet_component.c
Log Message:
Avoid a deadlock between softnet_lock and IFNET_LOCK
A deadlock occurs because there is a violation of the rule of lock ordering;
softnet_lock is held with hodling IFNET_LOCK, which violates the rule.
To avoid the deadlock, replace softnet_lock in in_control and in6_control
with KERNEL_LOCK.
We also need to add some KERNEL_LOCKs to protect the network stack surely.
This is required, for example, for PR kern/51356.
Fix PR kern/53043
To generate a diff of this commit:
cvs rdiff -u -r1.218 -r1.219 src/sys/netinet/in.c
cvs rdiff -u -r1.375 -r1.376 src/sys/netinet/ip_input.c
cvs rdiff -u -r1.3 -r1.4 src/sys/netinet/wqinput.c
cvs rdiff -u -r1.259 -r1.260 src/sys/netinet6/in6.c
cvs rdiff -u -r1.192 -r1.193 src/sys/netinet6/ip6_input.c
cvs rdiff -u -r1.10 -r1.11 \
src/sys/rump/net/lib/libnetinet/netinet_component.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->pending-pullups
State-Changed-By: ozaki-r@NetBSD.org
State-Changed-When: Mon, 26 Feb 2018 06:30:02 +0000
State-Changed-Why:
pullup-8 #588
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/53043 CVS commit: [netbsd-8] src/sys
Date: Mon, 26 Feb 2018 13:32:01 +0000
Module Name: src
Committed By: martin
Date: Mon Feb 26 13:32:01 UTC 2018
Modified Files:
src/sys/netinet [netbsd-8]: in.c ip_input.c wqinput.c
src/sys/netinet6 [netbsd-8]: in6.c ip6_input.c
src/sys/rump/net/lib/libnetinet [netbsd-8]: netinet_component.c
Log Message:
Pull up following revision(s) (requested by ozaki-r in ticket #588):
sys/netinet6/in6.c: revision 1.260
sys/netinet/in.c: revision 1.219
sys/netinet/wqinput.c: revision 1.4
sys/rump/net/lib/libnetinet/netinet_component.c: revision 1.11
sys/netinet/ip_input.c: revision 1.376
sys/netinet6/ip6_input.c: revision 1.193
Avoid a deadlock between softnet_lock and IFNET_LOCK
A deadlock occurs because there is a violation of the rule of lock ordering;
softnet_lock is held with hodling IFNET_LOCK, which violates the rule.
To avoid the deadlock, replace softnet_lock in in_control and in6_control
with KERNEL_LOCK.
We also need to add some KERNEL_LOCKs to protect the network stack surely.
This is required, for example, for PR kern/51356.
Fix PR kern/53043
To generate a diff of this commit:
cvs rdiff -u -r1.203.2.9 -r1.203.2.10 src/sys/netinet/in.c
cvs rdiff -u -r1.355.2.4 -r1.355.2.5 src/sys/netinet/ip_input.c
cvs rdiff -u -r1.3 -r1.3.2.1 src/sys/netinet/wqinput.c
cvs rdiff -u -r1.245.2.7 -r1.245.2.8 src/sys/netinet6/in6.c
cvs rdiff -u -r1.178.2.5 -r1.178.2.6 src/sys/netinet6/ip6_input.c
cvs rdiff -u -r1.8.6.1 -r1.8.6.2 \
src/sys/rump/net/lib/libnetinet/netinet_component.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: pending-pullups->closed
State-Changed-By: maxv@NetBSD.org
State-Changed-When: Mon, 05 Mar 2018 16:52:17 +0000
State-Changed-Why:
pulled up
From: "Ryota Ozaki" <ozaki-r@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/53043 CVS commit: src/sys/net
Date: Mon, 27 May 2019 05:33:48 +0000
Module Name: src
Committed By: ozaki-r
Date: Mon May 27 05:33:48 UTC 2019
Modified Files:
src/sys/net: rtsock.c
Log Message:
Don't take softnet_lock in sysctl_rtable
Taking softnet_lock there can cause a deadlock with nfs sosend, so we don't.
Having only KERNEL_LOCK is enough because now the routing table is protected by
KERNEL_LOCK that was introduced by the fix for PR 53043.
PR kern/54227 from Paul Ripke
To generate a diff of this commit:
cvs rdiff -u -r1.249 -r1.250 src/sys/net/rtsock.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/53043 CVS commit: [netbsd-8] src/sys/net
Date: Wed, 29 May 2019 16:01:51 +0000
Module Name: src
Committed By: martin
Date: Wed May 29 16:01:51 UTC 2019
Modified Files:
src/sys/net [netbsd-8]: rtsock.c
Log Message:
Pull up following revision(s) (requested by ozaki-r in ticket #1276):
sys/net/rtsock.c: revision 1.250
Don't take softnet_lock in sysctl_rtable
Taking softnet_lock there can cause a deadlock with nfs sosend, so we don't.
Having only KERNEL_LOCK is enough because now the routing table is protected by
KERNEL_LOCK that was introduced by the fix for PR 53043.
PR kern/54227 from Paul Ripke
To generate a diff of this commit:
cvs rdiff -u -r1.213.2.12 -r1.213.2.13 src/sys/net/rtsock.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.43 2018/01/16 07:36:43 maya Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2017
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home