NetBSD Problem Report #49150
From jnemeth@CornerstoneService.ca Mon Aug 25 05:23:55 2014
Return-Path: <jnemeth@CornerstoneService.ca>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
by mollari.NetBSD.org (Postfix) with ESMTPS id EBF7DAF165
for <gnats-bugs@gnats.NetBSD.org>; Mon, 25 Aug 2014 05:23:54 +0000 (UTC)
Message-Id: <201408250523.s7P5Nm62003095@server.cornerstoneservice.ca>
Date: Sun, 24 Aug 2014 22:23:48 -0700 (PDT)
From: jnemeth@CornerstoneService.ca
Reply-To: jnemeth@CornerstoneService.ca
To: gnats-bugs@gnats.NetBSD.org
Subject: xrstor is privileged in Xen
X-Send-Pr-Version: 3.95
>Number: 49150
>Category: port-amd64
>Synopsis: xrstor instruction is privileged in Xen
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: port-amd64-maintainer
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Mon Aug 25 05:25:00 +0000 2014
>Closed-Date: Thu Jan 15 22:45:57 +0000 2015
>Last-Modified: Sun Nov 05 17:25:00 +0000 2017
>Originator: John Nemeth
>Release: NetBSD 7.0 BETA and -current of same time frame
>Organization:
NetBSD
>Environment:
>Description:
The xrstor instruction is privileged in Xen and the use of
it leads to a panic, "fatal privileged instruction fault in supervisor
mode".
>How-To-Repeat:
Attempt to boot a NetBSD 7 BETA domu kernel and watch it go
boom at mountroot time.
>Fix:
>Release-Note:
>Audit-Trail:
From: Manuel Bouyer <bouyer@antioche.eu.org>
To: gnats-bugs@NetBSD.org
Cc: port-amd64-maintainer@NetBSD.org, gnats-admin@NetBSD.org,
netbsd-bugs@NetBSD.org
Subject: Re: port-amd64/49150: xrstor is privileged in Xen
Date: Mon, 25 Aug 2014 09:54:46 +0200
On Mon, Aug 25, 2014 at 05:25:00AM +0000, jnemeth@CornerstoneService.ca wrote:
> >Description:
> The xrstor instruction is privileged in Xen and the use of
> it leads to a panic, "fatal privileged instruction fault in supervisor
> mode".
> >How-To-Repeat:
> Attempt to boot a NetBSD 7 BETA domu kernel and watch it go
> boom at mountroot time.
that's strange, amd64 XEN3_DOMU boots fine here:
http://www-soc.lip6.fr/~bouyer/NetBSD-tests/xen/netbsd-7/
Can you give more details about your setup (CPU, xen version) ?
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
From: John Nemeth <jnemeth@cue.bc.ca>
To: Manuel Bouyer <bouyer@antioche.eu.org>, gnats-bugs@NetBSD.org
Cc: port-amd64-maintainer@NetBSD.org, gnats-admin@NetBSD.org,
netbsd-bugs@NetBSD.org
Subject: Re: port-amd64/49150: xrstor is privileged in Xen
Date: Mon, 25 Aug 2014 01:24:43 -0700
On Aug 25, 9:54am, Manuel Bouyer wrote:
} On Mon, Aug 25, 2014 at 05:25:00AM +0000, jnemeth@CornerstoneService.ca wrote:
} > >Description:
} > The xrstor instruction is privileged in Xen and the use of
} > it leads to a panic, "fatal privileged instruction fault in supervisor
} > mode".
} > >How-To-Repeat:
} > Attempt to boot a NetBSD 7 BETA domu kernel and watch it go
} > boom at mountroot time.
}
} that's strange, amd64 XEN3_DOMU boots fine here:
} http://www-soc.lip6.fr/~bouyer/NetBSD-tests/xen/netbsd-7/
}
} Can you give more details about your setup (CPU, xen version) ?
It's Xen Kernel 4.1.2. I see that it's out of date. However,
it is a production mission critical server. I can't reboot it
randomly; I have to plan downtime.
The details on the CPU are:
cpu0: AMD Family 15h (686-class), id 0x600f12
cpu0: features 0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features 0x178bfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2>
cpu0: features 0x178bfbff<HTT>
cpu0: features2 0x1698220b<SSE3,PCLMULQDQ,MONITOR,SSSE3,CX16,SSE41,SSE42>
cpu0: features2 0x1698220b<POPCNT,AES,XSAVE,AVX>
cpu0: features3 0x2fd3fbff<SYSCALL/SYSRET,NOX,MXX,FFXSR,P1GB,RDTSCP,LONG>
cpu0: features4 0x1c9bfff<LAHF,CMPLEGACY,SVM,EAPIC,ALTMOVCR0,LZCNT,SSE4A>
cpu0: features4 0x1c9bfff<MISALIGNSSE,3DNOWPREFETCH,OSVW,IBS,XOP,SKINIT,WDT>
cpu0: features4 0x1c9bfff<LWP,FMA4,NodeID,TopoExt,B23,B24>
cpu0: "AMD Opteron(TM) Processor 6272 "
cpu0: I-cache 64KB 64B/line 2-way, D-cache 16KB 64B/line 4-way
cpu0: L2 cache 2MB 64B/line 16-way
cpu0: L3 cache 12MB 64B/line 128-way
cpu0: ITLB 48 4KB entries fully associative, 24 2MB entries fully associative
cpu0: DTLB 32 4KB entries fully associative, 32 2MB entries fully associative
cpu0: L2 ITLB 512 4KB entries 4-way
cpu0: L2 DTLB 1024 4KB entries 8-way, 1024 2MB entries 8-way
cpu0: L1 1GB page ITLB 24 1GB entries fully associative
cpu0: L1 1GB page DTLB 32 1GB entries fully associative
cpu0: L2 1GB page DTLB 1024 1GB entries 8-way
cpu0: Initial APIC ID 0
cpu0: AMD Power Management features: 0x3d9<TS,TTP,HTC,100,HWP,TSC,CPB>
cpu0: SVM Rev. 1
cpu0: SVM NASID 65536
cpu0: SVM features 0x14ff<NP,LbrVirt,SVML,NRIPS,TSCRate,VMCBCleanBits>
cpu0: SVM features 0x14ff<FlushByASID,DecodeAssist,PauseFilter,B12>
cpu0: family 0f model 01 extfamily 06 extmodel 00 stepping 02
cpu0: UCode version: 0x6000629
Some quick googling seems to indicate that it is related to
XSA-52 / CVE-2013-2076. In particular, you need to have an AMD
cpu that is family 15h and up.
}-- End of excerpt from Manuel Bouyer
From: Manuel Bouyer <bouyer@antioche.eu.org>
To: John Nemeth <jnemeth@cue.bc.ca>
Cc: gnats-bugs@NetBSD.org, port-amd64-maintainer@NetBSD.org,
gnats-admin@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: port-amd64/49150: xrstor is privileged in Xen
Date: Mon, 25 Aug 2014 10:36:43 +0200
On Mon, Aug 25, 2014 at 01:24:43AM -0700, John Nemeth wrote:
> It's Xen Kernel 4.1.2. I see that it's out of date. However,
> it is a production mission critical server. I can't reboot it
> randomly; I have to plan downtime.
Mine is 4.1.4, but I'm not sure it'll make a difference.
If you send me your xen.gz kernel I can test it.
>
> The details on the CPU are:
>
> cpu0: AMD Family 15h (686-class), id 0x600f12
> cpu0: features 0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
> cpu0: features 0x178bfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2>
> cpu0: features 0x178bfbff<HTT>
> cpu0: features2 0x1698220b<SSE3,PCLMULQDQ,MONITOR,SSSE3,CX16,SSE41,SSE42>
> cpu0: features2 0x1698220b<POPCNT,AES,XSAVE,AVX>
> cpu0: features3 0x2fd3fbff<SYSCALL/SYSRET,NOX,MXX,FFXSR,P1GB,RDTSCP,LONG>
> cpu0: features4 0x1c9bfff<LAHF,CMPLEGACY,SVM,EAPIC,ALTMOVCR0,LZCNT,SSE4A>
> cpu0: features4 0x1c9bfff<MISALIGNSSE,3DNOWPREFETCH,OSVW,IBS,XOP,SKINIT,WDT>
> cpu0: features4 0x1c9bfff<LWP,FMA4,NodeID,TopoExt,B23,B24>
> cpu0: "AMD Opteron(TM) Processor 6272 "
Mine is:
cpu0: Intel (686-class), id 0xf64
cpu0: features 0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features 0xbfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu0: features 0xbfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu0: features2 0xe4bd<SSE3,DTES64,MONITOR,DS-CPL,VMX,EST,CID,CX16,xTPR,PDCM>
cpu0: features3 0x20100800<SYSCALL/SYSRET,XD,EM64T>
cpu0: "Intel(R) Xeon(TM) CPU 3.00GHz"
this may be what makes the difference.
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
From: David Laight <david@l8s.co.uk>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: port-amd64/49150: xrstor is privileged in Xen
Date: Thu, 11 Sep 2014 21:26:43 +0100
On Mon, Aug 25, 2014 at 05:25:00AM +0000, jnemeth@CornerstoneService.ca wrote:
> >Number: 49150
> >Category: port-amd64
> >Synopsis: xrstor instruction is privileged in Xen
> >Confidential: no
> >Severity: critical
> >Priority: high
> >Responsible: port-amd64-maintainer
> >State: open
> >Class: sw-bug
> >Submitter-Id: net
> >Arrival-Date: Mon Aug 25 05:25:00 +0000 2014
> >Originator: John Nemeth
> >Release: NetBSD 7.0 BETA and -current of same time frame
> >Organization:
> NetBSD
> >Environment:
> >Description:
> The xrstor instruction is privileged in Xen and the use of
> it leads to a panic, "fatal privileged instruction fault in supervisor
> mode".
> >How-To-Repeat:
> Attempt to boot a NetBSD 7 BETA domu kernel and watch it go
> boom at mountroot time.
ISTR this is in 'fpu' setup deecting avx??
IIRC the kernel only calls xrstor instruction if one of the other
registers says it is available.
So xen is failing to lie....
Possibly the instruction needs a 'trap detect' added and to disable avx
if it faults.
David
--
David Laight: david@l8s.co.uk
From: "John Nemeth" <jnemeth@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/49150 CVS commit: src/sys/arch/x86/x86
Date: Tue, 14 Oct 2014 03:16:56 +0000
Module Name: src
Committed By: jnemeth
Date: Tue Oct 14 03:16:56 UTC 2014
Modified Files:
src/sys/arch/x86/x86: identcpu.c
Log Message:
Force x86_xsave_features to 0 when running under XEN for AMD
processors. This prevents the use of xsave and xrstor thus fixing
the problem in PR/49150. The basic problem is that the way AMD
implements those instructions means that information can leak
between domains so XEN treats them as privileged.
XXX If anybody else comes up with a better / more "proper" fix, go
for it. However, this solves the problem I was having. And, given
that XEN being broken is pretty much a show-stopper for a release,
something needed to be done.
To generate a diff of this commit:
cvs rdiff -u -r1.45 -r1.46 src/sys/arch/x86/x86/identcpu.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Jeff Rizzo <riz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: port-amd64/49150: xrstor is privileged in Xen
Date: Mon, 27 Oct 2014 16:09:42 -0700
On 9/11/14 1:10 PM, David Laight wrote:
>
> ISTR this is in 'fpu' setup deecting avx??
>
> IIRC the kernel only calls xrstor instruction if one of the other
> registers says it is available.
> So xen is failing to lie....
>
> Possibly the instruction needs a 'trap detect' added and to disable avx
> if it faults.
>
I've discovered another processor for which it fails:
NetBSD 7.0_BETA (XEN3_DOMU.201410252300Z)
total memory = 615 MB
avail memory = 586 MB
kern.module.path=/stand/amd64/7.0/modules
mainbus0 (root)
hypervisor0 at mainbus0: Xen version 3.4.3.amazon
vcpu0 at hypervisor0: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz, id 0x206d7
xenbus0 at hypervisor0: Xen Virtual Bus Interface
xencons0 at hypervisor0: Xen Virtual Console Driver
xbd0 at xenbus0 id 2049: Xen Virtual Block Device Interface
xbd1 at xenbus0 id 2050: Xen Virtual Block Device Interface
xennet0 at xenbus0 id 0: Xen Virtual Network Interface
xennet0: MAC address 22:00:0a:5b:94:c4
balloon0 at xenbus0 id 0: Xen Balloon driver
balloon0: current reservation: 629760 KiB
xennet0: using RX copy mode
boot device: xbd1
root on xbd1a dumps on xbd1b
balloon0: current reservation: 157440 pages => target: 157440 pages
root file system type: ffs
fatal privileged instruction fault in supervisor mode
trap type 0 code 0 rip ffffffff8012e3a2 cs e030 rflags 10246 cr2
7f7ff781db1c ilevel 8 rsp ffffa0001c7feec8
curlwp 0xffffa000017c4120 pid 14.1 lowest kstack 0xffffa0001c7fc2c0
kernel: privileged instruction fault trap, code=0
Stopped in pid 14.1 (ps) at netbsd:xrstor+0xa: fxsavel
xrstor() at netbsd:xrstor+0xa
ds 4120
es 95c0
fs 0
gs d000
rdi ffffa0001c7fc080
rsi 7
rbp ffffa0001c7feef0
rbx ffffa000017c4120
rdx 0
rcx 0
rax 7
r8 3ff
r9 7f7fffffdc50
r10 7f7ff7c0c19a
r11 246
r12 ffffffff805d4600 cpu_info_primary
r13 ffffa0001c7fc000
r14 0
r15 4001c8
rip ffffffff8012e3a2 xrstor+0xa
cs e030
rflags 10246
rsp ffffa0001c7feec8
ss e02b
netbsd:xrstor+0xa: fxsavel
db{0}>
Given that these are in use on Amazon EC2 instances, I'm going to work
around the problem in a similar fashion for now.
From: Manuel Bouyer <bouyer@antioche.eu.org>
To: gnats-bugs@NetBSD.org
Cc: port-amd64-maintainer@NetBSD.org, gnats-admin@NetBSD.org,
netbsd-bugs@NetBSD.org, jnemeth@CornerstoneService.ca
Subject: Re: port-amd64/49150: xrstor is privileged in Xen
Date: Tue, 28 Oct 2014 08:34:49 +0100
On Mon, Oct 27, 2014 at 04:39:01PM -0700, Jeff Rizzo wrote:
> On 10/27/14 4:10 PM, Jeff Rizzo wrote:
> > Given that these are in use on Amazon EC2 instances, I'm going to work
> > around the problem in a similar fashion for now.
>
> I just looked a little more closely at the existing workaround, and now I'm
> not sure what the right thing to do is - should I set x86_xsave_features to
> 0 for Xen in general?
As it shows up on Intel CPUs too, I guess it's a reasonable
woraround for now.
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
From: Jeff Rizzo <riz@NetBSD.org>
To: Manuel Bouyer <bouyer@antioche.eu.org>, gnats-bugs@NetBSD.org
Cc: port-amd64-maintainer@NetBSD.org, gnats-admin@NetBSD.org,
netbsd-bugs@NetBSD.org, jnemeth@CornerstoneService.ca
Subject: Re: port-amd64/49150: xrstor is privileged in Xen
Date: Tue, 28 Oct 2014 10:34:59 -0700
On 10/28/14 12:34 AM, Manuel Bouyer wrote:
> On Mon, Oct 27, 2014 at 04:39:01PM -0700, Jeff Rizzo wrote:
>> On 10/27/14 4:10 PM, Jeff Rizzo wrote:
>>> Given that these are in use on Amazon EC2 instances, I'm going to work
>>> around the problem in a similar fashion for now.
>> I just looked a little more closely at the existing workaround, and now I'm
>> not sure what the right thing to do is - should I set x86_xsave_features to
>> 0 for Xen in general?
> As it shows up on Intel CPUs too, I guess it's a reasonable
> woraround for now.
>
I can confirm that if I set x86_xsave_features to 0 for Xen, the problem
AMIs boot on the problem CPUs. I will commit the change later today.
From: "Jeff Rizzo" <riz@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/49150 CVS commit: src/sys/arch/x86/x86
Date: Tue, 28 Oct 2014 17:44:48 +0000
Module Name: src
Committed By: riz
Date: Tue Oct 28 17:44:47 UTC 2014
Modified Files:
src/sys/arch/x86/x86: identcpu.c
Log Message:
Work around the problem in PR port-amd64/49150 for all CPUs under Xen.
The problem (calling xrstor, which is privileged in Xen) has appeared
on some Intel CPUs as well, so implement the workaround (ensure that
x86_xsave_features is 0) for all CPUs, not just AMD CPUs.
XXX pullup to 7
To generate a diff of this commit:
cvs rdiff -u -r1.46 -r1.47 src/sys/arch/x86/x86/identcpu.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/49150 CVS commit: [netbsd-7] src/sys/arch/x86/x86
Date: Thu, 30 Oct 2014 18:58:45 +0000
Module Name: src
Committed By: martin
Date: Thu Oct 30 18:58:45 UTC 2014
Modified Files:
src/sys/arch/x86/x86 [netbsd-7]: identcpu.c
Log Message:
Pull up following revision(s) (requested by riz in ticket #171):
sys/arch/x86/x86/identcpu.c: revision 1.46
sys/arch/x86/x86/identcpu.c: revision 1.47
Force x86_xsave_features to 0 when running under XEN.
This prevents the use of xsave and xrstor thus fixing
the problem in PR/49150. The basic problem is that the way AMD
implements those instructions means that information can leak
between domains so XEN treats them as privileged.
To generate a diff of this commit:
cvs rdiff -u -r1.45 -r1.45.2.1 src/sys/arch/x86/x86/identcpu.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->closed
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Thu, 15 Jan 2015 22:45:57 +0000
State-Changed-Why:
fixed, thanks
From: coypu@sdf.org
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: port-amd64/49150: xrstor is privileged in Xen
Date: Sun, 5 Nov 2017 17:23:31 +0000
The reason it was seen in Intel CPUs is that Xen made a no-xsave flag
and allowed people to set it (and mentioned it in a security advisory as
a mitigation so perhaps people were confused).
Now they have had a fix for this issue for a long time. Can we revert it
and allow netbsd/xen to use xsave?
This means removing the xen case in cpu_probe_fpu, if anyone wants to
try.
Index: identcpu.c
===================================================================
RCS file: /cvsroot/src/sys/arch/x86/x86/identcpu.c,v
retrieving revision 1.64
diff -u -r1.64 identcpu.c
--- identcpu.c 3 Nov 2017 16:21:01 -0000 1.64
+++ identcpu.c 5 Nov 2017 17:11:12 -0000
@@ -771,12 +771,7 @@
if (descs[2] > 512)
x86_fpu_save_size = descs[2];
-#ifdef XEN
- /* Don't use xsave, force fxsave with x86_xsave_features = 0. */
- x86_fpu_save = FPU_SAVE_FXSAVE;
-#else
x86_xsave_features = (uint64_t)descs[3] << 32 | descs[0];
-#endif
}
void
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home