NetBSD Problem Report #50332

From www@NetBSD.org  Mon Oct 12 20:13:21 2015
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 46AC5A5B2E
	for <gnats-bugs@gnats.NetBSD.org>; Mon, 12 Oct 2015 20:13:21 +0000 (UTC)
Message-Id: <20151012201320.48AFAA65BA@mollari.NetBSD.org>
Date: Mon, 12 Oct 2015 20:13:20 +0000 (UTC)
From: disabled.due.to.submitter.request
To: gnats-bugs@NetBSD.org
Subject: AVX instructions don't work but OSXSAVE flag is set
X-Send-Pr-Version: www-1.0

>Number:         50332
>Category:       port-xen
>Synopsis:       AVX instructions don't work but OSXSAVE flag is set
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    jdolecek
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Oct 12 20:15:00 +0000 2015
>Closed-Date:    Mon Nov 19 22:22:08 +0000 2018
>Last-Modified:  Mon Nov 19 22:25:00 +0000 2018
>Originator:     TG
>Release:        7.0
>Organization:
>Environment:
NetBSD hannah.gmplib.org 7.0 NetBSD 7.0 (XEN3_DOM0.201509250726Z) amd64

>Description:
Apparently, NetBSD sets the OSXSAVE flag without supporting AVX instructions, at least under Xen 4.5.1.  I cannot test this on bare metal, since all my NetBSD machines run Xen.

For what it is worth, the problem happens in both the Dom0 and DomU.
>How-To-Repeat:
cat >foo.c <<EOF
#include <stdio.h>
int main() {
  unsigned int eax, ebx, edx, ecx;
  __asm__ ("cpuid" : "=a" (eax), "=b" (ebx), "=d" (edx), "=c" (ecx) : "0" (1));
  if ((ecx >> 27) & 1) {
    printf("OS claims AVX is supported\n");
    printf("Testing an AVX instruction..."); fflush(stdout);
    __asm__ volatile (".byte 0xc5,0xeb,0x10,0xd9");
    printf("it seems to have worked\n");
  }
  return 0;
}
EOF
gcc foo.c && a.out
OS claims AVX is supported
Testing an AVX instruction...Illegal instruction

>Fix:

>Release-Note:

>Audit-Trail:
From: "Jonathan A. Kollasch" <jakllsch@kollasch.net>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org
Subject: Re: kern/50332: AVX instructions don't work but OSXSAVE flag is set
Date: Mon, 12 Oct 2015 20:19:05 -0500

 If this is a bug in NetBSD, it's also a bug on this Linux:

 Linux eternium 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u4 (2015-09-19) x86_64 GNU/Linux

 on a:

 model name	: Intel(R) Core(TM)2 Duo CPU     E8600  @ 3.33GHz

 CPU.

 Nothing I see says that OSXSAVE implies that AVX instructions are
 implemented and working.  According to:

 https://software.intel.com/en-us/blogs/2011/04/14/is-avx-enabled/

 the AVX flag (bit 28) also needs to be asserted.

From: tg@gmplib.org (=?utf-8?Q?Torbj=C3=B6rn?= Granlund)
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/50332: AVX instructions don't work but OSXSAVE flag is set
Date: Tue, 13 Oct 2015 09:09:44 +0200

 --=-=-=


    If this is a bug in NetBSD, it's also a bug on this Linux:

    Linux eternium 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u4 (2015-09-19) x86_64 GNU/Linux

    on a:

    model name	: Intel(R) Core(TM)2 Duo CPU     E8600  @ 3.33GHz

    CPU.

 This CPU does not support AVX.

    Nothing I see says that OSXSAVE implies that AVX instructions are
    implemented and working.  According to:

    https://software.intel.com/en-us/blogs/2011/04/14/is-avx-enabled/

    the AVX flag (bit 28) also needs to be asserted.

 The test case demos a real bug, please don't dismiss it because a CPU
 without AVX cannot execute AVX instructions.

 In practice, one of course also tests for AVX.  I modified the testcase
 to do that in order to clarify things.  Attached.

 This does not fail on Linux, FreeBSD, OpenBSD, etc.  It does fail on
 NetBSD.  Let me be clear: To see this AVX error, one needs a computer
 with AVX hardware.

 I've reproduced this on an Intel haswell system, an AMD bulldozer
 system, and an AMD piledriver system.  All these systems run some
 version of NetBSD 7.0 as Dom0, either a prerelease or the release.

 I've also reproduced it under NetBSD 6.1.x and NetBSD 7.0 in a DomU.  On
 the same machines, Linux and FreeBSD DomUs run the testcase fine.



 --=-=-=
 Content-Type: application/octet-stream
 Content-Disposition: attachment; filename=osxsave.c
 Content-Transfer-Encoding: base64

 I2luY2x1ZGUgPHN0ZGlvLmg+CgppbnQgbWFpbigpIHsKICB1bnNpZ25lZCBpbnQgZWF4LCBlYngs
 IGVkeCwgZWN4OwoKICBfX2FzbV9fICgiY3B1aWQiIDogIj1hIiAoZWF4KSwgIj1iIiAoZWJ4KSwg
 Ij1kIiAoZWR4KSwgIj1jIiAoZWN4KSA6ICIwIiAoMSkpOwogIGlmICgoZWN4ID4+IDI3KSAmIDEp
 IHsKICAgIHByaW50ZigiT1MgY2xhaW1zIEFWWCBpcyBzdXBwb3J0ZWRcbiIpOwoKICAgIHByaW50
 ZigiVGVzdGluZyBhbiBBVlggaW5zdHJ1Y3Rpb24uLi4iKTsgZmZsdXNoKHN0ZG91dCk7CiAgICBf
 X2FzbV9fIHZvbGF0aWxlICgiLmJ5dGUgMHhjNSwweGViLDB4MTAsMHhkOSIpOwogICAgcHJpbnRm
 KCJpdCBzZWVtcyB0byBoYXZlIHdvcmtlZFxuIik7CiAgfQoKICByZXR1cm4gMDsKfQo=
 --=-=-=
 Content-Type: text/plain; charset=utf-8
 Content-Transfer-Encoding: quoted-printable

 =20=20

 --=20
 Torbj=C3=B6rn
 Please encrypt, key id 0xC8601622

 --=-=-=--

From: David Laight <david@l8s.co.uk>
To: Torbj?rn Granlund <tg@gmplib.org>
Cc: gnats-bugs@NetBSD.org, kern-bug-people@netbsd.org,
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/50332: AVX instructions don't work but OSXSAVE flag is set
Date: Sun, 18 Oct 2015 21:41:39 +0100

 On Tue, Oct 13, 2015 at 09:09:44AM +0200, Torbj?rn Granlund wrote:
 >   
 >    If this is a bug in NetBSD, it's also a bug on this Linux:
 >    
 >    Linux eternium 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u4 (2015-09-19) x86_64 GNU/Linux
 >    
 >    on a:
 >    
 >    model name	: Intel(R) Core(TM)2 Duo CPU     E8600  @ 3.33GHz
 >    
 >    CPU.
 >    
 > This CPU does not support AVX.
 > 
 >    Nothing I see says that OSXSAVE implies that AVX instructions are
 >    implemented and working.  According to:
 >    
 >    https://software.intel.com/en-us/blogs/2011/04/14/is-avx-enabled/
 >    
 >    the AVX flag (bit 28) also needs to be asserted.
 >    
 > The test case demos a real bug, please don't dismiss it because a CPU
 > without AVX cannot execute AVX instructions.
 > 
 > In practice, one of course also tests for AVX.  I modified the testcase
 > to do that in order to clarify things.  Attached.

 No, there is no reason why you can't have a cpu (and os) that supports xsave
 (etc) but doesn't support avx.
 You have to look at the bitmap of supported extensions.
 IIRC Only the x87 fpu registers are mandatory.

 However that probably isn't the issue here.
 More likely is that something in xen should stop avx being used,
 or xen needs to do something different from bare-metal to support avx.

 	David

 -- 
 David Laight: david@l8s.co.uk

From: tg@gmplib.org (=?utf-8?Q?Torbj=C3=B6rn?= Granlund)
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/50332: AVX instructions don't work but OSXSAVE flag is set
Date: Tue, 20 Oct 2015 18:41:54 +0200

 David Laight <david@l8s.co.uk> writes:

    No, there is no reason why you can't have a cpu (and os) that supports x=
 save
    (etc) but doesn't support avx.
    You have to look at the bitmap of supported extensions.
    IIRC Only the x87 fpu registers are mandatory.
 =20=20=20
 Note that my updated test case tests both flags to make this a moot
 point.

    However that probably isn't the issue here.
    More likely is that something in xen should stop avx being used,
    or xen needs to do something different from bare-metal to support avx.
 =20=20=20
 Please note that non-NetBSD guest systems have no problem.  Also, NetBSD
 with no Xen around works fine.

 That makes me suspect Xen is innocent and that NetBSD gets this wrong
 when run with Xen.

 My complaint is not that AVX does not work under Xen+NetBSD, but that it
 doesn't work in spite of that NetBSD sets OSXSAVE.

 --=20
 Torbj=C3=B6rn
 Please encrypt, key id 0xC8601622

From: Manuel Bouyer <bouyer@antioche.eu.org>
To: =?iso-8859-1?Q?Torbj=F6rn?= Granlund <tg@gmplib.org>
Cc: gnats-bugs@netbsd.org, kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
        netbsd-bugs@netbsd.org
Subject: Re: kern/50332: AVX instructions don't work but OSXSAVE flag is set
Date: Tue, 20 Oct 2015 21:23:48 +0200

 On Tue, Oct 20, 2015 at 06:41:54PM +0200, Torbjörn Granlund wrote:
 > David Laight <david@l8s.co.uk> writes:
 > 
 >    No, there is no reason why you can't have a cpu (and os) that supports xsave
 >    (etc) but doesn't support avx.
 >    You have to look at the bitmap of supported extensions.
 >    IIRC Only the x87 fpu registers are mandatory.
 >    
 > Note that my updated test case tests both flags to make this a moot
 > point.
 > 
 >    However that probably isn't the issue here.
 >    More likely is that something in xen should stop avx being used,
 >    or xen needs to do something different from bare-metal to support avx.
 >    
 > Please note that non-NetBSD guest systems have no problem.  Also, NetBSD
 > with no Xen around works fine.
 > 
 > That makes me suspect Xen is innocent and that NetBSD gets this wrong
 > when run with Xen.
 > 
 > My complaint is not that AVX does not work under Xen+NetBSD, but that it
 > doesn't work in spite of that NetBSD sets OSXSAVE.

 AFAIK NetBSD is not setting OSXSAVE. This comes from the cpuid instruction,
 and I don't think the OS can change what cpuid repports.

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

Responsible-Changed-From-To: kern-bug-people->jdolecek
Responsible-Changed-By: jdolecek@NetBSD.org
Responsible-Changed-When: Sun, 04 Mar 2018 23:07:20 +0000
Responsible-Changed-Why:
I noticed some AVX support changes in PHP, noticed that NetBSD is not
on the list of operating systems supporting AVX according to wiki and eventually
found this upon digging. Moving under my name since I want to look
at this after I'll finish PCID support.


State-Changed-From-To: open->feedback
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Sun, 04 Mar 2018 23:15:40 +0000
State-Changed-Why:
Actually, tried the test program under -current under Parallels which
claims to support AVX, seemed to work fine. Can you re-test with 8.0_BETA
and/or the -current kernel such as 8.99.12?

Relevant lines from cpuctl identify 0 on my (working) vm:
cpu0: xsave features 0x7<x87,SSE,AVX>
cpu0: enabled xsave 0x7<x87,SSE,AVX>


State-Changed-From-To: feedback->open
State-Changed-By: maya@NetBSD.org
State-Changed-When: Mon, 05 Mar 2018 03:16:53 +0000
State-Changed-Why:
I provided feedback.


From: coypu@sdf.org
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/50332: AVX instructions don't work but OSXSAVE flag is set
Date: Mon, 5 Mar 2018 03:15:57 +0000

 avx is specifically being disabled for xen

 see x86/identcpu.c

 #ifdef XEN
 	x86_fpu_save = FPU_SAVE_FXSAVE;

 this is left over from PR 49150, which is left over from a xen security
 advisory (XSA-52).

 maxv said this:
 http://mail-index.netbsd.org/port-xen/2017/11/08/msg009123.html

From: =?UTF-8?B?SmFyb23DrXIgRG9sZcSNZWs=?= <jaromir.dolecek@gmail.com>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>, Manuel Bouyer <bouyer@antioche.eu.org>
Cc: 
Subject: Re: kern/50332 (AVX instructions don't work but OSXSAVE flag is set)
Date: Tue, 19 Jun 2018 00:55:27 +0200

 I'm working on this, have some local hacks in xen/cpu.c with which the
 AVX test program passes. It seems we could just test for OSXSAVE being
 already set in Xen case. For Xen, NetBSD kernel does not modify this
 option. On my machine, that flag is set when I boot DOMU with default
 options, and not set when I add no-xsave. So perhaps we could have
 conditional on this only.

 Manuel, you observed that panic after max's commits in 2017-11-07
 which led to fixes in rev 1.26 and 1.27 of arch/x86/x86/fpu.c

 I've checked on releng.netbsd.org, that machine has this flag also
 off, but it's DOMU. Can you please confirm to me whether cpuctl
 identify returns OSXSAVE on the appropriate DOM0, under which you run
 that DOMU anita run which panicced?

 Jaromir

From: Manuel Bouyer <bouyer@antioche.eu.org>
To: =?iso-8859-1?Q?Jarom=EDr?= Dole?ek <jaromir.dolecek@gmail.com>
Cc: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Subject: Re: kern/50332 (AVX instructions don't work but OSXSAVE flag is set)
Date: Tue, 19 Jun 2018 10:20:14 +0200

 On Tue, Jun 19, 2018 at 12:55:27AM +0200, Jaromír Dole?ek wrote:
 > I'm working on this, have some local hacks in xen/cpu.c with which the
 > AVX test program passes. It seems we could just test for OSXSAVE being
 > already set in Xen case. For Xen, NetBSD kernel does not modify this
 > option. On my machine, that flag is set when I boot DOMU with default
 > options, and not set when I add no-xsave. So perhaps we could have
 > conditional on this only.
 > 
 > Manuel, you observed that panic after max's commits in 2017-11-07
 > which led to fixes in rev 1.26 and 1.27 of arch/x86/x86/fpu.c
 > 
 > I've checked on releng.netbsd.org, that machine has this flag also
 > off, but it's DOMU. Can you please confirm to me whether cpuctl
 > identify returns OSXSAVE on the appropriate DOM0, under which you run
 > that DOMU anita run which panicced?

 Here's what cpuctl says for dom0:
 xen1:/usr/home/bouyer#cpuctl identify 0
 cpu0: highest basic info 00000006
 cpu0: highest extended info 80000008
 cpu0: "Intel(R) Xeon(TM) CPU 3.00GHz"
 cpu0: Intel (686-class)
 cpu0: family 0xf model 0x6 stepping 0x4 (id 0xf64)
 cpu0: features 0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE>
 cpu0: features 0xbfebfbff<MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2>
 cpu0: features 0xbfebfbff<SS,HTT,TM,SBF>
 cpu0: features1 0xe4bd<SSE3,DTES64,MONITOR,DS-CPL,VMX,EST,CID,CX16,xTPR,PDCM>
 cpu0: features2 0x20100800<SYSCALL/SYSRET,XD,EM64T>
 cpu0: features3 0x1<LAHF>
 cpu0: I-cache 12K uOp cache 8-way, D-cache 16KB 64B/line 8-way
 cpu0: L2 cache 2MB 64B/line 8-way
 cpu0: ITLB 4K/4M: 64 entries
 cpu0: DTLB 4K/4M: 64 entries
 cpu0: Initial APIC ID 5
 cpu0: Cluster/Package ID 1
 cpu0: Core ID 0
 cpu0: SMT ID 1
 cpu0: microcode version 0x2, platform ID 0

 The CPU is quite old. Also the hypervisor version changed ...

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

From: tg@gmplib.org (=?utf-8?Q?Torbj=C3=B6rn?= Granlund)
To: Manuel Bouyer <bouyer@antioche.eu.org>
Cc: jdolecek@NetBSD.org,  gnats-admin@netbsd.org,  netbsd-bugs@netbsd.org,  gnats-bugs@NetBSD.org
Subject: Re: kern/50332 (AVX instructions don't work but OSXSAVE flag is set)
Date: Tue, 19 Jun 2018 10:55:48 +0200

 Is there a way I could avoid getting these notifications?

 (This bug report is 4 years old.  Also, I long ago swapped out the last
 NetBSD/Xen install after critical stability issues for production
 systems .)

 --=20
 Torbj=C3=B6rn
 Please encrypt, key id 0xC8601622

State-Changed-From-To: open->pending-pullups
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Tue, 19 Jun 2018 19:55:17 +0000
State-Changed-Why:
Committed a fix, will request pullups if no fallout is observed.


From: "Jaromir Dolecek" <jdolecek@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/50332 CVS commit: src/sys/arch
Date: Tue, 19 Jun 2018 19:50:20 +0000

 Module Name:	src
 Committed By:	jdolecek
 Date:		Tue Jun 19 19:50:19 UTC 2018

 Modified Files:
 	src/sys/arch/x86/include: fpu.h
 	src/sys/arch/x86/x86: cpu.c fpu.c identcpu.c
 	src/sys/arch/xen/x86: cpu.c

 Log Message:
 fix FPU initialization on Xen to allow e.g. AVX when supported by hardware;
 only use XSAVE when the the CPUID OSXSAVE bit is set, as this seems to be
 reliable indication

 tested with Xen 4.2.6 DOM0/DOMU on Intel CPU, without and with no-xsave flag,
 so should work also on those AMD CPUs, which have XSAVE disabled by default;
 also tested with Xen DOM0 4.8.3

 fixes PR kern/50332 by Torbjorn Granlund; sorry it took three years to address

 XXX pullup netbsd-8


 To generate a diff of this commit:
 cvs rdiff -u -r1.9 -r1.10 src/sys/arch/x86/include/fpu.h
 cvs rdiff -u -r1.155 -r1.156 src/sys/arch/x86/x86/cpu.c
 cvs rdiff -u -r1.39 -r1.40 src/sys/arch/x86/x86/fpu.c
 cvs rdiff -u -r1.72 -r1.73 src/sys/arch/x86/x86/identcpu.c
 cvs rdiff -u -r1.117 -r1.118 src/sys/arch/xen/x86/cpu.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: pending-pullups->open
State-Changed-By: maxv@NetBSD.org
State-Changed-When: Sat, 23 Jun 2018 10:11:31 +0000
State-Changed-Why:
Reverted. By the way, the correct tag should have been "needs-pullups",
not "pending-pullups".


From: "Jaromir Dolecek" <jdolecek@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/50332 CVS commit: src/sys/arch
Date: Sat, 23 Jun 2018 10:30:22 +0000

 Module Name:	src
 Committed By:	jdolecek
 Date:		Sat Jun 23 10:30:22 UTC 2018

 Modified Files:
 	src/sys/arch/x86/x86: identcpu.c
 	src/sys/arch/xen/x86: cpu.c

 Log Message:
 re-do the XEN XSAVE support, this time to leave all probe code in
 cpu_probe_fpu(), and have XEN cpu_init() just act

 the xen probe is now guarded by XEN_USE_XSAVE option and XSAVE
 support is thus still disabled by default (same as before), so it
 wouldn't interfere with maxv's eager fpu rototil, while still
 allowing testing for others

 PR kern/50332


 To generate a diff of this commit:
 cvs rdiff -u -r1.76 -r1.77 src/sys/arch/x86/x86/identcpu.c
 cvs rdiff -u -r1.121 -r1.122 src/sys/arch/xen/x86/cpu.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: tg@gmplib.org (=?utf-8?Q?Torbj=C3=B6rn?= Granlund)
To: maxv@NetBSD.org
Cc: jdolecek@NetBSD.org,  netbsd-bugs@netbsd.org,  gnats-admin@netbsd.org,  gnats-bugs@NetBSD.org
Subject: Re: port-xen/50332 (AVX instructions don't work but OSXSAVE flag is set)
Date: Sat, 23 Jun 2018 14:28:13 +0200

 I REALLY don't want these notifications any longer.  Seriously!

 (If the traffic doesn't stop, my last resort will be blacklist the
 culprit netbsd mail server.)

 --=20
 Torbj=C3=B6rn
 Please encrypt, key id 0xC8601622

State-Changed-From-To: open->analyzed
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Sun, 08 Jul 2018 17:28:06 +0000
State-Changed-Why:
There is code to enable this hidden curently behind option, need to enable
it by default once FPU rototill is done.


State-Changed-From-To: analyzed->closed
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Mon, 19 Nov 2018 22:22:08 +0000
State-Changed-Why:
Finally enabled by default. Original committed no longer interested, so closing
right away.


From: "Jaromir Dolecek" <jdolecek@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/50332 CVS commit: src/sys/arch/x86/x86
Date: Mon, 19 Nov 2018 22:21:33 +0000

 Module Name:	src
 Committed By:	jdolecek
 Date:		Mon Nov 19 22:21:33 UTC 2018

 Modified Files:
 	src/sys/arch/x86/x86: identcpu.c

 Log Message:
 enable XSAVE (and hence AVX) under XEN by default, fixes PR kern/50332

 okayed by maxv@


 To generate a diff of this commit:
 cvs rdiff -u -r1.82 -r1.83 src/sys/arch/x86/x86/identcpu.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

>Unformatted:

 Removed email address for original submitter from 'From' upon submitter
 request.

Home	PR Database Search