NetBSD Problem Report #57258

From www@netbsd.org  Sat Mar  4 07:41:36 2023
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 7DB331A923B
	for <gnats-bugs@gnats.NetBSD.org>; Sat,  4 Mar 2023 07:41:36 +0000 (UTC)
Message-Id: <20230304074134.AEE671A923C@mollari.NetBSD.org>
Date: Sat,  4 Mar 2023 07:41:34 +0000 (UTC)
From: prlw1@cam.ac.uk
Reply-To: prlw1@cam.ac.uk
To: gnats-bugs@NetBSD.org
Subject: kthread_fpu_enter/exit problem
X-Send-Pr-Version: www-1.0

>Number:         57258
>Category:       kern
>Synopsis:       kthread_fpu_enter/exit problem
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Mar 04 07:45:00 +0000 2023
>Last-Modified:  Thu May 15 18:20:02 +0000 2025
>Originator:     Patrick Welche
>Release:        NetBSD-10.99.2/amd64
>Organization:
>Environment:
>Description:
A kernel built with amd64/locore.S 1.216 having AMD Ryzen 7 5700X cpu panics on boot with:

Configuring CGD
[   4.4533308] fatal protection fault in supervisor mode
[   4.4633307] trap type 4 code 0 rip 0xffffffff802363c4 cs 0x8 rflags 0x10256 cr2 0 ilevel 0x8 rsp 0xffffc510aada2d58
[   4.4733305] curlwp 0xffffe98c5ae24340 pid 0.877 lowest kstack 0xffffc510aad9e2c0

kernel: protection fault trap, code=0                                           
Stopped in pid 0.1371 (system) at       netbsd:xrstor64+0xa:    fxsaveq         
xrstor64() at netbsd:xrstor64+0xa                                               
sleepq_block() at netbsd:sleepq_block+0x13a                                     
cv_wait() at netbsd:cv_wait+0xb7                                                
workqueue_worker() at netbsd:workqueue_worker+0x125                             

crash> show proc/a 0xffffe98c5ae24340
lwp_t ffffe98c5ae24340
system: pid 0 proc ffffffff81069140 vmspace/map ffffffff81122420 flags 20002
> lwp 877 [cgd/5] ffffe98c5ae24340 pcb ffffc510aad9e000
    stat 7 flags 600 cpu 5 pri 123 ref 0

(gdb) print *(struct pcb *)0xffffc510aad9e000
$1 = {pcb_flags = 0, pcb_cr0 = 2147811379, pcb_rsp0 = 18446679274109415408, pcb_cr2 = 0, 
  pcb_cr3 = 21565440, pcb_rsp = 18446679274109414752, pcb_rbp = 18446679274109414896, 
  pcb_onfault = 0x0, pcb_fs = 0, pcb_gs = 0, pcb_dbregs = 0x0, pcb_fpu_dflt_cw = 0, pcb_iopl = 0, 
  pcb_unused = {0, 0, 0, 0, 0, 0, 0, 0}, pcb_savefpu = {sv_87 = {s87_cw = 59904, s87_sw = 32639, 
      s87_tw = 37712, s87_ip = {fa_64 = 32639, fa_32 = {fa_off = 32639, fa_seg = 0, fa_opcode = 0}}, 
      s87_dp = {fa_64 = 17865305007633465344, fa_32 = {fa_off = 0, fa_seg = 20544, fa_opcode = 63470}}, 
      s87_ac = {{f87_mantissa = 17865304732755591039, f87_exp_sign = 32639}, {f87_mantissa = 268435456, 
          f87_exp_sign = 0}, {f87_mantissa = 51539607554, f87_exp_sign = 12}, {f87_mantissa = 0, 
          f87_exp_sign = 0}, {f87_mantissa = 17865307069217767424, f87_exp_sign = 32639}, {
          f87_mantissa = 0, f87_exp_sign = 0}, {f87_mantissa = 140187597193216, f87_exp_sign = 0}, {
          f87_mantissa = 0, f87_exp_sign = 0}}}, sv_xmm = {fx_cw = 59904, fx_sw = 65535, 
      fx_tw = 127 '\177', fx_zero = 127 '\177', fx_opcode = 0, fx_ip = {fa_64 = 140187597247312, 
        fa_32 = {fa_off = 4159673168, fa_seg = 32639, fa_opcode = 0}}, fx_dp = {fa_64 = 0, fa_32 = {
          fa_off = 0, fa_seg = 0, fa_opcode = 0}}, fx_mxcsr = 4159590464, fx_mxcsr_mask = 32639, 
      fx_87_ac = {{r = {f87_mantissa = 140187597164544, f87_exp_sign = 4096}}, {r = {
            f87_mantissa = 51539607554, f87_exp_sign = 12}}, {r = {f87_mantissa = 0, 
            f87_exp_sign = 21024}}, {r = {f87_mantissa = 0, f87_exp_sign = 49152}}, {r = {
            f87_mantissa = 0, f87_exp_sign = 0}}, {r = {f87_mantissa = 0, f87_exp_sign = 49152}}, {r = {
            f87_mantissa = 140187597266656, f87_exp_sign = 0}}, {r = {f87_mantissa = 140187597193280, 
            f87_exp_sign = 504}}}, fx_xmm = {{xmm_bytes = '\000' <repeats 15 times>}, {
          xmm_bytes = "\000\000\000\000\000\000\000\000\240\305\356\367\177\177\000"}, {
          xmm_bytes = "@\310\356\367\177\177\000\000\000\000\000\000\000\000\000"}, {
          xmm_bytes = '\000' <repeats 15 times>}, {
          xmm_bytes = "\000\000\000\000\000\000\000\000\340\302\356\367\177\177\000"}, {
          xmm_bytes = "\360\304\356\367\177\177\000\000\255\000\000\000\000\000\000"}, {
          xmm_bytes = "@\302\356\367\177\177\000\000\000\000\000\000\000\000\000"}, {
          xmm_bytes = "\204\302\356\367\177\177\000\000\026\000\000\000\000\000\000"}, {
          xmm_bytes = '\000' <repeats 15 times>}, {xmm_bytes = '\000' <repeats 15 times>}, {
          xmm_bytes = "\n\000\004\000\000\000\000\000\000\300\356\367\177\177\000"}, {
          xmm_bytes = " R\356\367\177\177\000\000\340\336\357\367\177\177\000"}, {
          xmm_bytes = "\000\000\000\000\000\000\000\000\b\275\356\367\177\177\000"}, {
          xmm_bytes = '\000' <repeats 15 times>}, {xmm_bytes = '\000' <repeats 15 times>}, {
          xmm_bytes = '\000' <repeats 15 times>}}, 
      fx_rsvd = '\000' <repeats 16 times>, "\021\000\000\000\342\341\341\341\001\004", '\000' <repeats 62 times>, "\022\000\000\000\000\000\000"}, sv_xsave_hdr = {
      xsh_fxsave = "\000\352\377\377\177\177\000\000P\223\357\367\177\177\000\000\000\000\000\000\000\000\000\000@P\356\367\177\177\000\000\000P\356\367\177\177\000\000\000\020\000\000\000\000\000\000\002\000\000\000\f\000\000\000\f", '\000' <repeats 15 times>, " R\356\367\177\177", '\000' <repeats 11 times>, "\300\356\367\177\177", '\000' <repeats 27 times>, "\300\356\367\177\177\000\000\340\336\357\367\177\177\000\000\000\000\000\000\000\000\000\000@\300\356\367\177\177\000\000\370\001", '\000' <repeats 30 times>, "\240\305\356\367\177\177\000\000@\310\356\367\177\177", '\000' <repeats 34 times>..., xsh_xstate_bv = 0, 
      xsh_xcomp_bv = 0, xsh_rsrvd = "\000\000\000\000\000\000\000", 
      xsh_reserved = '\000' <repeats 39 times>}}}

>How-To-Repeat:

>Fix:
riastradh@ has already reverted this.

>Audit-Trail:
From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/57258 CVS commit: src/sys/arch
Date: Thu, 24 Apr 2025 01:50:40 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Thu Apr 24 01:50:39 UTC 2025

 Modified Files:
 	src/sys/arch/amd64/amd64: genassym.cf
 	src/sys/arch/amd64/include: pcb.h
 	src/sys/arch/i386/i386: genassym.cf
 	src/sys/arch/i386/include: pcb.h
 	src/sys/arch/x86/include: cpu.h cpu_extended_state.h
 	src/sys/arch/x86/x86: fpu.c vm_machdep.c

 Log Message:
 amd64: Allocate FPU save state outside pcb if it's too large.

 We have seen x86_fpu_save_size values (CPUID[EAX=0x0d, ECX=0].ECX) as
 large as 11008 bytes, notably with Intel AMX TILEDATA's 8192-byte
 state.

 We only do this for user threads, and only on machines where it's
 necessary, to avoid incurring much overhead.  There is still a tiny
 bit of overhead when saving and restoring the FPU state by using a
 pointer indirection instead of arithmetic indirection for access to
 struct pcb::pcb_savefpu, but this is probably a drop in the bucket
 compared to the memory traffic incurred by the FPU state save/restore
 anyway.

 For now, these paths are mostly disabled on i386.  We could enable
 them but it will require either rewriting cpu_uarea_alloc/free for
 i386, or adopting a guard page like amd64 does, which might be costly
 and so should be undertaken only with some thought and care.  And
 since Intel AMX instructions only work in 64-bit mode, it's not
 likely to be useful on i386.

 PR port-amd64/57661: Crash when booting on Xeon Silver 4416+ in
 KVM/Qemu

 These changes, as a side effect, may fix:

 PR kern/57258: kthread_fpu_enter/exit problem

 by making sure to allocate an FPU save space that is large enough to
 guarantee fpu_kern_enter/leave work safely, instead of just using a
 union savefpu object on the stack (which, at 576 bytes, may be too
 small on some machines, particularly with AVX512 requiring ~2.5K).
 (But we'll have to do some extra work with kthread_fpu_enter/exit_md
 -- if we try doing them again on x86 -- to actually allocate the
 separate pcb on these machines!)


 To generate a diff of this commit:
 cvs rdiff -u -r1.98 -r1.99 src/sys/arch/amd64/amd64/genassym.cf
 cvs rdiff -u -r1.32 -r1.33 src/sys/arch/amd64/include/pcb.h
 cvs rdiff -u -r1.136 -r1.137 src/sys/arch/i386/i386/genassym.cf
 cvs rdiff -u -r1.59 -r1.60 src/sys/arch/i386/include/pcb.h
 cvs rdiff -u -r1.139 -r1.140 src/sys/arch/x86/include/cpu.h
 cvs rdiff -u -r1.18 -r1.19 src/sys/arch/x86/include/cpu_extended_state.h
 cvs rdiff -u -r1.89 -r1.90 src/sys/arch/x86/x86/fpu.c
 cvs rdiff -u -r1.46 -r1.47 src/sys/arch/x86/x86/vm_machdep.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/57258 CVS commit: [netbsd-10] src/sys/arch/x86/x86
Date: Thu, 15 May 2025 18:17:07 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Thu May 15 18:17:07 UTC 2025

 Modified Files:
 	src/sys/arch/x86/x86 [netbsd-10]: fpu.c

 Log Message:
 Pull up following revision(s) (requested by riastradh in ticket #1119):

 	sys/arch/x86/x86/fpu.c: revision 1.81
 	sys/arch/x86/x86/fpu.c: revision 1.90 (partial, via patch)

 x86: Fix fpu_kern_enter/leave for machines with big fpu state.

 fpu_kern_enter/leave are used for FPU/SIMD access from the kernel,
 for cgd(4) crypto and similar.  Currently, we use a statically
 allocated union savefpu object with particular content to put the
 FPU into a safe state or an all-zero state.

 This doesn't work (reliably, anyway) when the FPU state is larger
 than 576 bytes, e.g. with AVX-512 register state (another ~2 KiB)
 or Intel AMX TILECFG/TILEDATA (another ~10 KiB).  For machines
 with larger FPU state, this change dynamically allocates a larger
 area for the safe/zero FPU states.

 (This change doesn't add support for Intel AMX TILECFG/TILEDATA;
 it just avoids the failure mode.  The part of
 https://mail-index.netbsd.org/source-changes/2025/04/24/msg156552.html
 it applies is limited.)
 This may improve the situation for the following PRs:

 PR kern/57258: kthread_fpu_enter/exit problem
 PR kern/58650: unable to install v10.0 - kernel does not boot, sysinst does not start


 To generate a diff of this commit:
 cvs rdiff -u -r1.79.4.4 -r1.79.4.5 src/sys/arch/x86/x86/fpu.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

Home	PR Database Search