NetBSD Problem Report #57460
From www@netbsd.org Fri Jun 9 23:16:54 2023
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 4BA841A923D
for <gnats-bugs@gnats.NetBSD.org>; Fri, 9 Jun 2023 23:16:54 +0000 (UTC)
Message-Id: <20230609231621.CF5D81A9241@mollari.NetBSD.org>
Date: Fri, 9 Jun 2023 23:16:21 +0000 (UTC)
From: campbell+netbsd@mumble.net
Reply-To: campbell+netbsd@mumble.net
To: gnats-bugs@NetBSD.org
Subject: OCTEON MP kernel crash in TLB miss trap
X-Send-Pr-Version: www-1.0
>Number: 57460
>Category: kern
>Synopsis: OCTEON MP kernel crash in TLB miss trap
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: kern-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Fri Jun 09 23:20:00 +0000 2023
>Originator: Taylor R Campbell
>Release: current
>Organization:
The NetTLB Foundation
>Environment:
OCTEON + options MULTIPROCESSOR
>Description:
In the middle of a git clone:
Receiving objects: 47% (1932881/4069070), 635.55 MiB | 1.05 MiBpid 0(system): trap: cpu1, TLB miss (load or instr. fetch) in kernel mode
[ 1408.7100030] status=0xffa3, cause=0x8, epc=0x7228, vaddr=0
[ 1408.7100030] tf=0x980000041003bcb0 ksp=0x980000041003bdf0 ra=0x7228 ppl=0
[ 1408.7100030] kernel: TLB miss (load or instr. fetch) trap
[ 1408.7100030] panic: kernel diagnostic assertion "ci->ci_cpl == IPL_HIGH" fail
ed: file "/usr/src/current/sys/arch/evbmips/evbmips/interrupt.c", line 61
[ 1408.7100030] cpu0: Begin traceback...
[ 1408.7100030] pid 175344592 not found
[ 1408.7100030] cpu0: End traceback...
[ 1408.7100030] kernel: breakpoint trap
Stopped in pid 0.16 (system) at 7228:pid 0(system): trap: cpu1, TLB miss (load o
r instr. fetch) in kernel mode
[ 1408.7100030] status=0xa3, cause=0x40008008, epc=0xffffffff80253144, vaddr=0
[ 1408.7100030] tf=0x9800000410038150 ksp=0x9800000410038290 ra=0xffffffff80252b00 ppl=0
[ 1408.7100030] kernel: TLB miss (load or instr. fetch) trap
Stopped in pid 0.16 (system) at netbsd:t
rap+0x994: ld a1,0(a1)
db{1}> bt
0x9800000410038290: trap+0x994 (0xffffffff800015c0,0,0x78,0xffffffff80253144) ra 0xffffffff80242814 sz 528
0x98000004100384a0: mips64r2_kern_gen_exception+0xd8 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80252b00 sz 352
0x9800000410038600: trap+0x350 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80242814 sz 528
0x9800000410038810: mips64r2_kern_gen_exception+0xd8 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80252b00 sz 352
0x9800000410038970: trap+0x350 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80242814 sz 528
0x9800000410038b80: mips64r2_kern_gen_exception+0xd8 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80252b00 sz 352
0x9800000410038ce0: trap+0x350 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80242814 sz 528
0x9800000410038ef0: mips64r2_kern_gen_exception+0xd8 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80252b00 sz 352
0x9800000410039050: trap+0x350 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80242814 sz 528
0x9800000410039260: mips64r2_kern_gen_exception+0xd8 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80252b00 sz 352
0x98000004100393c0: trap+0x350 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80242814 sz 528
0x98000004100395d0: mips64r2_kern_gen_exception+0xd8 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80252b00 sz 352
0x9800000410039730: trap+0x350 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80242814 sz 528
0x9800000410039940: mips64r2_kern_gen_exception+0xd8 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80252b00 sz 352
0x9800000410039aa0: trap+0x350 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80242814 sz 528
0x9800000410039cb0: mips64r2_kern_gen_exception+0xd8 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80252b00 sz 352
0x9800000410039e10: trap+0x350 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80242814 sz 528
0x980000041003a020: mips64r2_kern_gen_exception+0xd8 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80252b00 sz 352
0x980000041003a180: trap+0x350 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80242814 sz 528
0x980000041003a390: mips64r2_kern_gen_exception+0xd8 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80252b00 sz 352
0x980000041003a4f0: trap+0x350 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80242814 sz 528
0x980000041003a700: mips64r2_kern_gen_exception+0xd8 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80252b00 sz 352
0x980000041003a860: trap+0x350 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80242814 sz 528
0x980000041003aa70: mips64r2_kern_gen_exception+0xd8 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80252b00 sz 352
0x980000041003abd0: trap+0x350 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80242814 sz 528
0x980000041003ade0: mips64r2_kern_gen_exception+0xd8 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80252b00 sz 352
0x980000041003af40: trap+0x350 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80242814 sz 528
0x980000041003b150: mips64r2_kern_gen_exception+0xd8 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80252b00 sz 352
0x980000041003b2b0: trap+0x350 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80242814 sz 528
0x980000041003b4c0: mips64r2_kern_gen_exception+0xd8 (0xa3,0x40008008,0x78,0xffffffff80253144) ra 0xffffffff80252b00 sz 352
0x980000041003b620: trap+0x350 (0xa3,0x40008008,0x78,0xffffffff80627ff0) ra 0xffffffff80242814 sz 528
0x980000041003b830: mips64r2_kern_gen_exception+0xd8 (0xa3,0x40008008,0x4,0xffffffff80627ff0) ra 0xffffffff802435a8 sz 352
0x980000041003b990: copyin+0x2c (0xa3,0x40008008,0x4,0xffffffff80627ff0) ra 0xffffffff80248b2c sz 32
0x980000041003b9b0: db_read_bytes+0xa4 (0xa3,0x40008008,0x4,0xffffffff80627ff0) ra 0xffffffff80247694 sz 32
0x980000041003b9d0: db_disasm+0x24 (0xa3,0x40008008,0x4,0xffffffff80627ff0) ra 0xffffffff80321120 sz 48
0x980000041003ba00: db_trap+0xa0 (0xa3,0x40008008,0x4,0xffffffff80627ff0) ra 0xffffffff80248838 sz 48
0x980000041003ba30: kdb_trap+0x140 (0xa3,0x40008008,0x4,0xffffffff80627ff0) ra 0xffffffff80252974 sz 80
0x980000041003ba80: trap+0x1c4 (0xa3,0x40008008,0x4,0x7228) ra 0xffffffff80242814 sz 528
0x980000041003bc90: mips64r2_kern_gen_exception+0xd8 (0xffa3,0x8,0xffffffff80690000,0x7228) ra 0x7228 sz 352
PC 0x7228: not in kernel space
0x980000041003bdf0: 0+0x7228 (0xffa3,0x8,0xffffffff80690000,0x7228) ra 0 sz 0
User-level: pid 0.16
db{1}> ps
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
5327 5327 2 0 40000 980000041f6c0500 git
5328 5328 3 1 0 980000041d959700 git-remote-https nfsrcv
5162 5162 3 0 180 980000041d7e1280 git wait
5370 5180 3 0 180 980000041d1e3b80 git pipe_rd
5370 5370 3 1 180 980000041d1e2980 git pipe_rd
5349 5349 3 1 180 980000041d958980 time wait
4830 4830 3 1 180 980000041d958500 dhcpcd poll
4963 4963 3 1 180 980000041d7e0e00 dhcpcd poll
4964 4964 3 0 180 980000041d1e2e00 dhcpcd poll
4827 4827 3 0 180 980000041d1e2500 dhcpcd poll
193 193 3 1 180 980000041fc67280 sh wait
1 1 3 1 180 980000041fcd0e00 init wait
0 5331 3 1 200 980000041fc67700 nfskqpoll nfskqpw
0 160 3 1 200 980000041fc58500 pooldrain pooldrain
0 167 3 1 200 980000041fc58080 ioflush syncer
0 166 3 1 200 980000041fc67b80 pgdaemon pgdaemon
0 > 63 7 0 200 980000041fc66e00 nfsio
0 126 3 0 200 980000041fc66980 nfsio nfsiod
0 125 3 1 200 980000041fc66500 nfsio nfsiod
0 124 3 1 200 980000041fc66080 nfsio nfsiod
0 123 3 1 200 980000041fcd1b80 swwreboot swwreboot
0 120 3 1 200 980000041fcd1700 usb0 usbevt
0 119 3 1 200 980000041fcd1280 npfgc0 npfgcw
0 118 3 1 200 980000041fcd0980 rt_free rt_free
0 117 3 1 200 980000041fcd0500 unpgc unpgc
0 116 3 1 200 980000041fd23b80 icmp6_wqinput/1 icmp6_wqinput
0 115 3 0 200 980000041fcfa080 icmp6_wqinput/0 icmp6_wqinput
0 114 3 1 200 980000041fcd0080 ip6flow ip6flow
0 113 3 1 200 980000041fcfbb80 nd6_timer nd6_timer
0 112 3 1 200 980000041fcfb700 carp6_wqinput/1 carp6_wqinput
0 111 3 0 200 980000041fcfb280 carp6_wqinput/0 carp6_wqinput
0 110 3 1 200 980000041fcfae00 carp_wqinput/1 carp_wqinput
0 109 3 0 200 980000041fcfa980 carp_wqinput/0 carp_wqinput
0 108 3 1 200 980000041fcfa500 icmp_wqinput/1 icmp_wqinput
0 107 3 0 200 980000041fd22500 icmp_wqinput/0 icmp_wqinput
0 106 3 1 200 980000041fd22e00 rt_timer rt_timer
0 105 3 1 200 980000041fd23280 ipflow_slowtimo ipflow_slowtimo
0 104 3 1 200 980000041fd23700 vmem_rehash vmem_rehash
0 30 3 1 200 980000041fd22080 entbutler entropy
0 29 3 0 200 980000041fd29b80 usbtask-dr usbtsk
0 28 3 0 200 980000041fd29700 usbtask-hc usbtsk
0 27 3 0 200 980000041fd29280 dwc2 dwc2
0 25 3 0 200 980000041fd28980 lnxubdwq lnxubdwq
0 24 3 0 200 980000041fd28500 lnxpwrwq lnxpwrwq
0 23 3 0 200 980000041fd28080 lnxlngwq lnxlngwq
0 22 3 0 200 980000041fd65b80 lnxhipwq lnxhipwq
0 21 3 1 200 980000041fd65700 xcall/1 xcall
0 20 1 1 200 980000041fd65280 softser/1
0 19 1 1 200 980000041fd64e00 softclk/1
0 18 1 1 200 980000041fd64980 softbio/1
0 17 1 1 200 980000041fd64500 softnet/1
0 > 16 1 1 201 980000041fd64080 idle/1
0 15 3 0 200 980000041fdcbb80 pmfsuspend pmfsuspend
0 14 3 0 200 980000041fdcb700 pmfevent pmfevent
0 13 3 0 200 980000041fdcb280 sopendfree sopendfr
0 12 3 0 200 980000041fdcae00 ifwdog ifwdog
0 11 3 1 200 980000041fdca980 iflnkst iflnkst
0 10 3 0 200 980000041fdca500 nfssilly nfssilly
0 9 3 0 200 980000041fdca080 vdrain vdrain
0 8 3 0 200 980000041fe27b80 modunload mod_unld
0 7 3 0 200 980000041fe27700 xcall/0 xcall
0 6 1 0 200 980000041fe27280 softser/0
0 5 1 0 200 980000041fe26e00 softclk/0
0 4 1 0 200 980000041fe26980 softbio/0
0 > 3 1 0 200 980000041fe26500 softnet/0
0 2 1 0 201 980000041fe26080 idle/0
0 0 2 1 240 ffffffff80761480 swapper
db{1}> ps/w
PID LID COMMAND EMUL PRI WAIT-MSG WAIT-CHANNEL
5327 5327 git netbsd32 28 0
5328 5328 git-remote-https netbsd32 40 nfsrcv 980000041fc76510
5162 5162 git netbsd32 43 wait 980000041d740018
5370 5180 git netbsd32 43 pipe_rd 980000041f9d9840
5370 5370 git netbsd32 43 pipe_rd 980000041f9d8e40
5349 5349 time netbsd32 43 wait 980000041d8ce018
4830 4830 dhcpcd netbsd32 43 poll 980000041fd6d200
4963 4963 dhcpcd netbsd32 43 poll 980000041fd6d200
4964 4964 dhcpcd netbsd32 42 poll 980000041fef1000
4827 4827 dhcpcd netbsd32 43 poll 980000041fef1000
193 193 sh netbsd32 43 wait 980000041fcba418
1 1 init netbsd32 40 wait 980000041fcba018
0 5331 system netbsd 96 nfskqpw ffffffff808722f8
0 160 system netbsd 125 pooldrain ffffffff80878b08
0 167 system netbsd 124 syncer 980000041fc58080
0 166 system netbsd 126 pgdaemon ffffffff80877350
0 > 63 system netbsd 96 0
0 126 system netbsd 96 nfsiod 980000041fc82710
0 125 system netbsd 96 nfsiod 980000041fc82610
0 124 system netbsd 96 nfsiod 980000041fc82410
0 123 system netbsd 43 swwreboot 980000041fcd2d90
0 121 system netbsd 96 sccomp 980000041fcd3888
0 120 system netbsd 96 usbevt 980000041fd2f920
0 119 system netbsd 96 npfgcw 980000041fcab810
0 118 system netbsd 222 rt_free 980000041fcd2610
0 117 system netbsd 96 unpgc ffffffff80886b20
0 116 system netbsd 222 icmp6_wqinput 980000041fcd8890
0 115 system netbsd 222 icmp6_wqinput 980000041fcd8810
0 114 system netbsd 222 ip6flow 980000041fcd2110
0 113 system netbsd 222 nd6_timer 980000041fd7fe10
0 112 system netbsd 222 carp6_wqinput 980000041fcd8510
0 111 system netbsd 222 carp6_wqinput 980000041fcd8490
0 110 system netbsd 222 carp_wqinput 980000041fcd8190
0 109 system netbsd 222 carp_wqinput 980000041fcd8110
0 108 system netbsd 222 icmp_wqinput 980000041fe31690
0 107 system netbsd 222 icmp_wqinput 980000041fe31610
0 106 system netbsd 222 rt_timer 980000041fd7f410
0 105 system netbsd 222 ipflow_slowtimo 980000041fd7f190
0 104 system netbsd 125 vmem_rehash 980000041fd7ef10
0 30 system netbsd 43 entropy ffffffff80812cf0
0 29 system netbsd 96 usbtsk ffffffff80863698
0 28 system netbsd 96 usbtsk ffffffff80863650
0 26 system netbsd 43 lnxsyswq 980000041fd2ab10
0 25 system netbsd 43 lnxubdwq 980000041fd2a990
0 24 system netbsd 43 lnxpwrwq 980000041fd2a810
0 23 system netbsd 43 lnxlngwq 980000041fd2a690
0 22 system netbsd 43 lnxhipwq 980000041fd2a510
0 21 system netbsd 127 xcall ffffffff800015d0
0 20 system netbsd 223 0
0 19 system netbsd 220 0
0 18 system netbsd 221 0
0 17 system netbsd 222 0
0 > 16 system netbsd 0 0
0 15 system netbsd 43 pmfsuspend 980000041fd7ea10
0 14 system netbsd 43 pmfevent 980000041fd7e790
0 13 system netbsd 96 sopendfr ffffffff80886a98
0 12 system netbsd 222 ifwdog 980000041fd7e510
0 11 system netbsd 222 iflnkst 980000041fd7e290
0 10 system netbsd 43 nfssilly 980000041fd84490
0 9 system netbsd 125 vdrain ffffffff80887cd8
0 8 system netbsd 125 mod_unld ffffffff8087ad20
0 7 system netbsd 127 xcall ffffffff80755950
0 6 system netbsd 223 0
0 5 system netbsd 220 0
0 4 system netbsd 221 0
0 > 3 system netbsd 222 0
0 2 system netbsd 0 0
0 0 system netbsd 125 0
No place for a crash dump, so this is it, sorry.
>How-To-Repeat:
git clone https://github.com/NetBSD/pkgsrc
(not sure if reproducible)
>Fix:
Yes, please!
(Contact us)
$NetBSD: query-full-pr,v 1.47 2022/09/11 19:34:41 kim Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2023
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home