NetBSD Problem Report #60664

From www@netbsd.org  Fri Aug 28 23:48:24 2026
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "mail.netbsd.org", Issuer "YR2" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 2997F1A923B
	for <gnats-bugs@gnats.NetBSD.org>; Fri, 28 Aug 2026 23:48:24 +0000 (UTC)
Message-Id: <20260828234822.C65211A923C@mollari.NetBSD.org>
Date: Fri, 28 Aug 2026 23:48:22 +0000 (UTC)
From: kaloz@dune.hu
Reply-To: kaloz@dune.hu
To: gnats-bugs@NetBSD.org
Subject: mips: rdhwr emulation corrupts v0 on nested TLB miss, random userland SIGSEGV
X-Send-Pr-Version: www-1.0
X-From4GNATS: "kaloz@dune.hu via gnats" <gnats-admin@NetBSD.org>

>Number:         60664
>Category:       port-mips
>Synopsis:       mips: rdhwr emulation corrupts v0 on nested TLB miss, random userland SIGSEGV
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    port-mips-maintainer
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Aug 28 23:50:01 +0000 2026
>Closed-Date:    Sun Oct 11 05:30:49 +0000 2026
>Last-Modified:  Sun Oct 11 05:30:49 +0000 2026
>Originator:     Imre Kaloz
>Release:        NetBSD 11.0 (also -current)
>Organization:
>Environment:
NetBSD indigo 11.0 NetBSD 11.0 (GENERIC32_IP2x) #0: Thu Jul 30 15:23:12 UTC 2026
mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/sgimips/compile/GENERIC32_IP2x sgimips

>Description:
On CPUs without RDHWR every TLS access traps into MIPSX(user_reserved_insn), which re-reads the trapping instruction from user space with a plain inline load while EXL is still set.  If that read takes a TLB miss (the entry can be evicted by a refill for the mapped kernel stack moments earlier), the exception dispatch, which only looks at KSU, treats it as a fresh trap from userland and clobbers the v0 save slot the routine is still using. The emulation then returns the trap's own exception PC to the program as its TLS pointer, and the process later crashes at some unrelated address. From the outside it looks like random memory corruption, took a while to pin down.
>How-To-Repeat:
Run a few concurrent loops of dynamically linked commands (while :; do /bin/ls >/dev/null; done) next to a memory hog bigger than RAM, so paging keeps the TLB busy. I get a SIGSEGV rate of about 1 in 7.
>Fix:
Check the page with TLBP before the inline read and fall back to the existing protected slow path if it is not mapped.

Signed-off-by: Imre Kaloz <kaloz@dune.hu>

--- a/sys/arch/mips/mips/mipsX_subr.S
+++ b/sys/arch/mips/mips/mipsX_subr.S
@@ -1320,11 +1320,43 @@ NESTED_NOPROFILE(MIPSX(user_reserved_insn), CALLFRAME_SIZ, ra)
 	 nop

 	/*
-	 * Get exception PC and fetch the instruction.  We know we can do
-	 * this since the instruction actually got read.
+	 * Get exception PC and fetch the instruction.
+	 *
+	 * Probe the page with TLBP first, and take the slow path if the
+	 * entry isn't there.  A nested miss on the inline read below would
+	 * otherwise land in user_gen_exception and clobber this routine's
+	 * own v0, since the exception PC isn't saved again while EXL is set.
 	 */
 	_MFC0	v0, MIPS_COP_0_EXC_PC
 	MFC0_HAZARD
+
+	/* Need v1 and a0 too */
+	REG_S	v1, CALLFRAME_SIZ+TF_REG_V1(k0)
+	REG_S	a0, CALLFRAME_SIZ+TF_REG_A0(k0)
+	_MFC0	v1, MIPS_COP_0_TLB_HI		# save current ASID
+	MFC0_HAZARD
+	andi	a0, v1, MIPS3_PG_ASID
+	move	AT, v0
+#ifdef MIPSNNR2
+	_INS	AT, a0, 0, V_MIPS3_PG_HVPN
+#else
+	_SRA	AT, V_MIPS3_PG_HVPN		# clear bottom bits of VA
+	_SLL	AT, V_MIPS3_PG_HVPN		# clear bottom bits of VA
+	or	AT, AT, a0
+#endif
+	_MTC0	AT, MIPS_COP_0_TLB_HI		# look for the vaddr & ASID
+	COP0_SYNC
+	tlbp					# probe the entry in question
+	COP0_SYNC
+	mfc0	AT, MIPS_COP_0_TLB_INDEX	# see what we got
+	MFC0_HAZARD
+	_MTC0	v1, MIPS_COP_0_TLB_HI		# restore ASID
+	COP0_SYNC
+	/* Restore v1 and a0 for either outcome */
+	REG_L	v1, CALLFRAME_SIZ+TF_REG_V1(k0)
+	bltz	AT, MIPSX(user_gen_exception_common) # index < 0 => !found
+	 REG_L	a0, CALLFRAME_SIZ+TF_REG_A0(k0)
+
 	INT_L	AT, 0(v0)

 	/*

>Release-Note:

>Audit-Trail:
From: Nick Hudson <nick.hudson@gmx.co.uk>
To: gnats-bugs@netbsd.org,
 port-mips-maintainer@netbsd.org,
 gnats-admin@netbsd.org,
 netbsd-bugs@netbsd.org
Cc: 
Subject: Re: port-mips/60664: mips: rdhwr emulation corrupts v0 on nested TLB
 miss, random userland SIGSEGV
Date: Tue, 22 Sep 2026 18:58:42 +0100

 --Apple-Mail=_1E0D089D-F651-40AE-98A7-C6C6B197948D
 Content-Transfer-Encoding: quoted-printable
 Content-Type: text/plain;
 	charset=us-ascii

 Thanks for the PR.

 I changed the patch a little to improve register usage in both the fast =
 and slow paths.



 --Apple-Mail=_1E0D089D-F651-40AE-98A7-C6C6B197948D
 Content-Disposition: attachment;
 	filename=mipsX_subr.S.diff
 Content-Type: application/octet-stream;
 	x-unix-mode=0644;
 	name="mipsX_subr.S.diff"
 Content-Transfer-Encoding: quoted-printable

 Index: sys/arch/mips/mips/mipsX_subr.S
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 RCS file: /cvsroot/src/sys/arch/mips/mips/mipsX_subr.S,v
 retrieving revision 1.118
 diff -u -p -r1.118 mipsX_subr.S
 =2D-- sys/arch/mips/mips/mipsX_subr.S	27 Jun 2026 07:44:56 -0000	1.118
 +++ sys/arch/mips/mips/mipsX_subr.S	22 Sep 2026 15:52:31 -0000
 @@ -1320,32 +1320,63 @@ NESTED_NOPROFILE(MIPSX(user_reserved_ins
  	 nop
 =20
  	/*
 -	 * Get exception PC and fetch the instruction.  We know we can do
 -	 * this since the instruction actually got read.
 +	 * Get exception PC and fetch the instruction.
 +	 *
 +	 * Probe the page with TLBP first, and take the slow path if the
 +	 * entry isn't there.  A nested miss on the inline read below would
 +	 * otherwise land in user_gen_exception and clobber this routine's
 +	 * own v0, since the exception PC isn't saved again while EXL is set.
  	 */
  	_MFC0	v0, MIPS_COP_0_EXC_PC
  	MFC0_HAZARD
 +
 +	/* Need v1 and a0 too */
 +	REG_S	v1, CALLFRAME_SIZ+TF_REG_V1(k0)
 +	REG_S	a0, CALLFRAME_SIZ+TF_REG_A0(k0)
 +
 +	_MFC0	v1, MIPS_COP_0_TLB_HI		# grab current ASID
 +	MFC0_HAZARD
 +	andi	a0, v1, MIPS3_PG_ASID
 +	move	AT, v0
 +#ifdef MIPSNNR2
 +	_INS	AT, a0, 0, V_MIPS3_PG_HVPN
 +#else
 +	_SRA	AT, V_MIPS3_PG_HVPN		# clear bottom bits of VA
 +	_SLL	AT, V_MIPS3_PG_HVPN		# clear bottom bits of VA
 +	or	AT, AT, a0
 +#endif
 +	_MTC0	AT, MIPS_COP_0_TLB_HI		# look for the vaddr & ASID
 +	COP0_SYNC
 +	tlbp					# probe the entry in question
 +	COP0_SYNC
 +	mfc0	AT, MIPS_COP_0_TLB_INDEX	# see what we got
 +	MFC0_HAZARD
 +	_MTC0	v1, MIPS_COP_0_TLB_HI		# restore HI
 +	COP0_SYNC
 +	bltz	AT, MIPSX(user_gen_exception_common2) # index < 0 =3D> !found
 +	 nop
 +
 +	// Load instruction
  	INT_L	AT, 0(v0)
 =20
  	/*
  	 * Was this rdhwr $3,$29?
  	 */
 -	lui	v0, %hi(0x7c03e83b)	# 0x7c03e83b =3D> rdhwr $3,$29
 -	addiu	v0, %lo(0x7c03e83b)	#        or ... rdhwr v1,ulr
 -	bne	AT, v0, MIPSX(user_gen_exception_common)
 +	lui	v1, %hi(0x7c03e83b)	# 0x7c03e83b =3D> rdhwr $3,$29
 +	addiu	v1, %lo(0x7c03e83b)	#        or ... rdhwr v1,ulr
 +	bne	AT, v1, MIPSX(user_gen_exception_common2)
  	 nop
 =20
  	/*
  	 * Advance the PC (don't want to restart at the rdhwr).
  	 */
 -	_MFC0	v0, MIPS_COP_0_EXC_PC
 -	MFC0_HAZARD
  	PTR_ADDIU v0, 4
  	_MTC0	v0, MIPS_COP_0_EXC_PC
  	COP0_SYNC
 =20
  	PTR_L	v1, L_PRIVATE(k1)		# rdhwr $3,$29 updates v1
 =20
 +	REG_L	a0, CALLFRAME_SIZ+TF_REG_A0(k0) # restore reg
  	REG_L	AT, CALLFRAME_SIZ+TF_REG_AST(k0)# restore reg
  	REG_L	v0, CALLFRAME_SIZ+TF_REG_V0(k0) # restore reg
  	eret
 @@ -1374,34 +1405,35 @@ NESTED_NOPROFILE(MIPSX(user_gen_exceptio
  	REG_S	v0, CALLFRAME_SIZ+TF_REG_V0(k0)
  MIPSX(user_gen_exception_common):
  	REG_S	v1, CALLFRAME_SIZ+TF_REG_V1(k0)
 -	mflo	v0
  	REG_S	a0, CALLFRAME_SIZ+TF_REG_A0(k0)
 +MIPSX(user_gen_exception_common2):
  	REG_S	a1, CALLFRAME_SIZ+TF_REG_A1(k0)
 +	mflo	v0
  	REG_S	a2, CALLFRAME_SIZ+TF_REG_A2(k0)
  	REG_S	a3, CALLFRAME_SIZ+TF_REG_A3(k0)
 -	mfhi	v1
  	REG_S	t0, CALLFRAME_SIZ+TF_REG_T0(k0)
  	REG_S	t1, CALLFRAME_SIZ+TF_REG_T1(k0)
 +	mfhi	v1
  	REG_S	t2, CALLFRAME_SIZ+TF_REG_T2(k0)
  	REG_S	t3, CALLFRAME_SIZ+TF_REG_T3(k0)
 -	mfc0	a0, MIPS_COP_0_STATUS		# 1st arg is STATUS
  	REG_S	ta0, CALLFRAME_SIZ+TF_REG_TA0(k0)
  	REG_S	ta1, CALLFRAME_SIZ+TF_REG_TA1(k0)
 +	mfc0	a0, MIPS_COP_0_STATUS		# 1st arg is STATUS
  	REG_S	ta2, CALLFRAME_SIZ+TF_REG_TA2(k0)
  	REG_S	ta3, CALLFRAME_SIZ+TF_REG_TA3(k0)
 -	mfc0	a1, MIPS_COP_0_CAUSE		# 2nd arg is CAUSE
  	REG_S	s0, CALLFRAME_SIZ+TF_REG_S0(k0)
  	REG_S	s1, CALLFRAME_SIZ+TF_REG_S1(k0)
 +	mfc0	a1, MIPS_COP_0_CAUSE		# 2nd arg is CAUSE
  	REG_S	s2, CALLFRAME_SIZ+TF_REG_S2(k0)
  	REG_S	s3, CALLFRAME_SIZ+TF_REG_S3(k0)
 -	_MFC0	a2, MIPS_COP_0_BAD_VADDR	# 3rd arg is fault address
  	REG_S	s4, CALLFRAME_SIZ+TF_REG_S4(k0)
  	REG_S	s5, CALLFRAME_SIZ+TF_REG_S5(k0)
 +	_MFC0	a2, MIPS_COP_0_BAD_VADDR	# 3rd arg is fault address
  	REG_S	s6, CALLFRAME_SIZ+TF_REG_S6(k0)
  	REG_S	s7, CALLFRAME_SIZ+TF_REG_S7(k0)
 -	_MFC0	a3, MIPS_COP_0_EXC_PC		# 4th arg is exception PC
  	REG_S	t8, CALLFRAME_SIZ+TF_REG_T8(k0)	# will be MIPS_CURLWP
  	REG_S	t9, CALLFRAME_SIZ+TF_REG_T9(k0)
 +	_MFC0	a3, MIPS_COP_0_EXC_PC		# 4th arg is exception PC
  	REG_S	v0, CALLFRAME_SIZ+TF_REG_MULLO(k0)
  	REG_S	v1, CALLFRAME_SIZ+TF_REG_MULHI(k0)
  	REG_S	gp, CALLFRAME_SIZ+TF_REG_GP(k0)

 --Apple-Mail=_1E0D089D-F651-40AE-98A7-C6C6B197948D--

From: "Nick Hudson" <skrll@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/60664 CVS commit: src/sys/arch/mips/mips
Date: Fri, 25 Sep 2026 16:30:31 +0000

 Module Name:	src
 Committed By:	skrll
 Date:		Fri Sep 25 16:30:31 UTC 2026

 Modified Files:
 	src/sys/arch/mips/mips: mipsX_subr.S

 Log Message:
 mips: fix rdhwr emulation wrt TLB misses on loading the faulting instruction.

 Use tlbp to probe the TLB for an entry for the page containing the
 instruction to emulate, and fail to slow path if it doesn't exist.

 A TLB miss (nested exception) on the instruction load would otherwise
 land in user_gen_exception and clobber this routine's v0, since the
 exception PC isn't saved again while EXL is set.

 PR/60664 mips: rdhwr emulation corrupts v0 on nested TLB miss...

 This patch is based on the diff included in the PR, but updated by me
 to improve register save / restore in both the fast and slow paths.

 The mf* instructions in user_gen_exception_common are shuffled down
 to maintain the same number of stores (4) between each mf* in
 user_gen_exception_common and user_gen_exception_common2

 The symptoms that this change fixes match those described in

 port-evbmips/59236: Multiple segfaults in erlite3 boot


 To generate a diff of this commit:
 cvs rdiff -u -r1.118 -r1.119 src/sys/arch/mips/mips/mipsX_subr.S

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Nick Hudson" <skrll@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/60664 CVS commit: src/sys/arch/mips/mips
Date: Mon, 28 Sep 2026 10:53:42 +0000

 Module Name:	src
 Committed By:	skrll
 Date:		Mon Sep 28 10:53:41 UTC 2026

 Modified Files:
 	src/sys/arch/mips/mips: mipsX_subr.S

 Log Message:
 mips: remove the OCTEON workaround for PR/59236 as fixed by PR/60664

 The rdhwr emulation bug observed in PR/59236 was fixed by the patch in
 PR/60664 so the workaround applied for PR/59236 can be safely removed.


 To generate a diff of this commit:
 cvs rdiff -u -r1.119 -r1.120 src/sys/arch/mips/mips/mipsX_subr.S

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->pending-pullups
State-Changed-By: skrll@NetBSD.org
State-Changed-When: Mon, 28 Sep 2026 12:05:13 +0000
State-Changed-Why:
[pullup-11 #500] PR/60664 mips: rdhwr emulation corrupts v0 on nested TLB miss...
[pullup-10 #1347] PR/60664 mips: rdhwr emulation corrupts v0 on nested TLB miss...


From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/60664 CVS commit: [netbsd-11] src/sys/arch/mips/mips
Date: Fri, 2 Oct 2026 10:27:03 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Fri Oct  2 10:27:02 UTC 2026

 Modified Files:
 	src/sys/arch/mips/mips [netbsd-11]: mipsX_subr.S

 Log Message:
 Pull up following revision(s) (requested by skrll in ticket #500):

 	sys/arch/mips/mips/mipsX_subr.S: revision 1.120
 	sys/arch/mips/mips/mipsX_subr.S: revision 1.119

 mips: fix rdhwr emulation wrt TLB misses on loading the faulting instruction.

 Use tlbp to probe the TLB for an entry for the page containing the
 instruction to emulate, and fail to slow path if it doesn't exist.

 A TLB miss (nested exception) on the instruction load would otherwise
 land in user_gen_exception and clobber this routine's v0, since the
 exception PC isn't saved again while EXL is set.

 PR/60664 mips: rdhwr emulation corrupts v0 on nested TLB miss...

 This patch is based on the diff included in the PR, but updated by me
 to improve register save / restore in both the fast and slow paths.

 The mf* instructions in user_gen_exception_common are shuffled down
 to maintain the same number of stores (4) between each mf* in
 user_gen_exception_common and user_gen_exception_common2

 The symptoms that this change fixes match those described in
 port-evbmips/59236: Multiple segfaults in erlite3 boot
 mips: remove the OCTEON workaround for PR/59236 as fixed by PR/60664

 The rdhwr emulation bug observed in PR/59236 was fixed by the patch in
 PR/60664 so the workaround applied for PR/59236 can be safely removed.


 To generate a diff of this commit:
 cvs rdiff -u -r1.116 -r1.116.2.1 src/sys/arch/mips/mips/mipsX_subr.S

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/60664 CVS commit: [netbsd-10] src/sys/arch/mips/mips
Date: Fri, 2 Oct 2026 10:29:34 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Fri Oct  2 10:29:34 UTC 2026

 Modified Files:
 	src/sys/arch/mips/mips [netbsd-10]: mipsX_subr.S

 Log Message:
 Pull up following revision(s) (requested by skrll in ticket #1347):

 	sys/arch/mips/mips/mipsX_subr.S: revision 1.119

 mips: fix rdhwr emulation wrt TLB misses on loading the faulting instruction.

 Use tlbp to probe the TLB for an entry for the page containing the
 instruction to emulate, and fail to slow path if it doesn't exist.

 A TLB miss (nested exception) on the instruction load would otherwise
 land in user_gen_exception and clobber this routine's v0, since the
 exception PC isn't saved again while EXL is set.

 PR/60664 mips: rdhwr emulation corrupts v0 on nested TLB miss...

 This patch is based on the diff included in the PR, but updated by me
 to improve register save / restore in both the fast and slow paths.

 The mf* instructions in user_gen_exception_common are shuffled down
 to maintain the same number of stores (4) between each mf* in
 user_gen_exception_common and user_gen_exception_common2

 The symptoms that this change fixes match those described in
 port-evbmips/59236: Multiple segfaults in erlite3 boot


 To generate a diff of this commit:
 cvs rdiff -u -r1.115 -r1.115.4.1 src/sys/arch/mips/mips/mipsX_subr.S

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: pending-pullups->feedback
State-Changed-By: skrll@NetBSD.org
State-Changed-When: Fri, 02 Oct 2026 17:05:58 +0000
State-Changed-Why:
ok to close?


From: Imre Kaloz <kaloz@dune.hu>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@NetBSD.org>
Cc: 
Subject: Re: port-mips/60664: mips: rdhwr emulation corrupts v0 on nested TLB
 miss, random userland SIGSEGV
Date: Sat, 10 Oct 2026 17:55:39 +0000

 Hi Nick,=0A=
 =0A=
 Yes, OK to close from my side, thanks for picking it up and for the=0A=
 pullups.=0A=
 =0A=
 I re-tested with the GENERIC32_IP2x kernels from the 20261009 HEAD and=0A=
 20261008 netbsd-11 daily builds, on an 11.0 userland.=0A=
 =0A=
 Each run is 400 invocations of expr, sed, date and sysctl from a single-use=
 r=0A=
 shell, counting SIGSEGVs:=0A=
 =0A=
                         11.0        HEAD      netbsd-11=0A=
 IP20 (R4400)    43/400   0/400      0/400=0A=
 IP24 (R5000)    52/400   0/400      0/400=0A=
 =0A=
 =0A=
 Imre=

State-Changed-From-To: feedback->closed
State-Changed-By: skrll@NetBSD.org
State-Changed-When: Sun, 11 Oct 2026 05:30:49 +0000
State-Changed-Why:
confirmed fixed.


>Unformatted:

Home	PR Database Search