NetBSD Problem Report #42441

From www@NetBSD.org  Fri Dec 11 09:48:47 2009
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by www.NetBSD.org (Postfix) with ESMTP id E6FD463BB2F
	for <gnats-bugs@gnats.NetBSD.org>; Fri, 11 Dec 2009 09:48:47 +0000 (UTC)
Message-Id: <20091211094847.823E163BAB5@www.NetBSD.org>
Date: Fri, 11 Dec 2009 09:48:47 +0000 (UTC)
From: petea_garstea@fincombank.com
Reply-To: petea_garstea@fincombank.com
To: gnats-bugs@NetBSD.org
Subject: DOM0 and DOMU work with only one cpu
X-Send-Pr-Version: www-1.0

>Number:         42441
>Notify-List:    gson@gson.org
>Category:       port-xen
>Synopsis:       DOM0 and DOMU work with only one cpu
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    port-xen-maintainer
>State:          closed
>Class:          support
>Submitter-Id:   net
>Arrival-Date:   Fri Dec 11 09:50:00 +0000 2009
>Closed-Date:    Mon Mar 15 13:55:55 +0000 2021
>Last-Modified:  Mon Mar 15 13:55:55 +0000 2021
>Originator:     petea gasrstea
>Release:        netbsd 5-0-1 stable updated
>Organization:
fincombank
>Environment:
DOM-0:
NetBSD NetBSD 5.0.1 NetBSD 5.0.1 (XEN3_DOM0) #0: Thu Dec  3 17:56:47 EET 2009  peter@NetBSD:/usr/src/sys/arch/i386/compile/XEN3_DOM0 i386

DOMU
NetBSD  5.0.1 NetBSD 5.0.1 (XEN3_DOMU) #0: Tue Dec  8 21:21:30 EET 2009  peter@NetBSD:/usr/objdir/sys/arch/i386/compile/XEN3_DOMU i386
>Description:
hello!
i updated the system and install the:
xentools3-3.1.4nb4
xenkernel3-3.1.4nb3

i have the system with 4 cpu and configured DOM0 to use only one cpu
(dom0-cpus 1) - in xend config file, and DOMU with the 1,2,3 cpus, it doesnt work, then i tried to do it manualy with the following commands:

DOM0:
xm vcpu-set 0 1
xm vcpu-pin 0 0 0

DOMU:
xm vcpu-set 2 3
xm vcpu-pin 2 0 1
xm vcpu-pin 2 1 2
xm vcpu-pin 2 2 3

xm vcpu-list
Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
Domain-0                             0     0     0   r--     741.2 0
Domain-0                             0     1     -   --p       0.0 0
Domain-0                             0     2     -   --p       0.0 0
Domain-0                             0     3     -   --p       0.0 0
NetBSD-STABLE-DOMU                   2     0     1   -b-      12.4 1
NetBSD-STABLE-DOMU                   2     1     -   --p       0.0 2
NetBSD-STABLE-DOMU                   2     2     -   --p       0.0 3

So i see that dom0 has 4 vcpu but i gave to it only one vcpu, The domu has been created with the 3 vcpu and useses cpu 1,2,3 but second and third vcpu have pause state that means domu doesnt use it

xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   256     1     r-----    744.0
NetBSD-STABLE-DOMU                           2   512     1     -b----     14.2

My conclusion is:
Im doing something wrong
or the hyperviosion is not working correctly. 
>How-To-Repeat:

>Fix:

>Release-Note:

>Audit-Trail:
From: Manuel Bouyer <bouyer@antioche.eu.org>
To: gnats-bugs@NetBSD.org
Cc: port-xen-maintainer@NetBSD.org, gnats-admin@NetBSD.org,
        netbsd-bugs@NetBSD.org
Subject: Re: port-xen/42441: DOM0 and DOMU work with only one cpu
Date: Fri, 11 Dec 2009 12:43:15 +0100

 On Fri, Dec 11, 2009 at 09:50:00AM +0000, petea_garstea@fincombank.com wrote:
 > DOM-0:
 > NetBSD NetBSD 5.0.1 NetBSD 5.0.1 (XEN3_DOM0) #0: Thu Dec  3 17:56:47 EET 2009  peter@NetBSD:/usr/src/sys/arch/i386/compile/XEN3_DOM0 i386
 > 
 > DOMU
 > NetBSD  5.0.1 NetBSD 5.0.1 (XEN3_DOMU) #0: Tue Dec  8 21:21:30 EET 2009  peter@NetBSD:/usr/objdir/sys/arch/i386/compile/XEN3_DOMU i386
 > >Description:
 > hello!
 > i updated the system and install the:
 > xentools3-3.1.4nb4
 > xenkernel3-3.1.4nb3
 > 
 > i have the system with 4 cpu and configured DOM0 to use only one cpu
 > (dom0-cpus 1) - in xend config file, and DOMU with the 1,2,3 cpus, it doesnt work, then i tried to do it manualy with the following commands:
 > 
 > DOM0:
 > xm vcpu-set 0 1
 > xm vcpu-pin 0 0 0
 > 
 > DOMU:
 > xm vcpu-set 2 3
 > xm vcpu-pin 2 0 1
 > xm vcpu-pin 2 1 2
 > xm vcpu-pin 2 2 3
 > 
 > xm vcpu-list
 > Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
 > Domain-0                             0     0     0   r--     741.2 0
 > Domain-0                             0     1     -   --p       0.0 0
 > Domain-0                             0     2     -   --p       0.0 0
 > Domain-0                             0     3     -   --p       0.0 0
 > NetBSD-STABLE-DOMU                   2     0     1   -b-      12.4 1
 > NetBSD-STABLE-DOMU                   2     1     -   --p       0.0 2
 > NetBSD-STABLE-DOMU                   2     2     -   --p       0.0 3
 > 
 > So i see that dom0 has 4 vcpu but i gave to it only one vcpu, The domu has been created with the 3 vcpu and useses cpu 1,2,3 but second and third vcpu have pause state that means domu doesnt use it
 > 
 > xm list
 > Name                                        ID   Mem VCPUs      State   Time(s)
 > Domain-0                                     0   256     1     r-----    744.0
 > NetBSD-STABLE-DOMU                           2   512     1     -b----     14.2
 > 
 > My conclusion is:
 > Im doing something wrong
 > or the hyperviosion is not working correctly. 

 NetBSD/Xen doesn't support SMP yet at the domains level (SMP will be used by
 the hypervisor to schedule different domains to different CPUs)

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

From: Jeff Rizzo <riz@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-xen/42441
Date: Sat, 04 Feb 2012 14:31:22 -0800

 SMP support has been enabled for Xen DOMU in -current, and will be in 
 NetBSD 6.0.  DOM0 support is there, but not enabled yet, as there are 
 still some issues with it.

State-Changed-From-To: open->closed
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Tue, 19 Jun 2018 21:30:09 +0000
State-Changed-Why:
DOMU supports SMP since 2012. Probably good enough for now.


State-Changed-From-To: closed->open
State-Changed-By: gson@NetBSD.org
State-Changed-When: Wed, 20 Jun 2018 07:27:50 +0000
State-Changed-Why:
DOM0 SMP support may not be important to jdolecek, but it's important to me.


From: Andreas Gustafsson <gson@gson.org>
To: jdolecek@NetBSD.org
Cc: gnats-bugs@NetBSD.org,
    port-xen-maintainer@netbsd.org,
    petea_garstea@fincombank.com
Subject: Re: port-xen/42441 (DOM0 and DOMU work with only one cpu)
Date: Wed, 20 Jun 2018 10:25:30 +0300

 jdolecek@NetBSD.org wrote:
 > DOMU supports SMP since 2012. Probably good enough for now.

 It's not good enough for me.  DOM0 SMP support has been at the top of
 my wish list of NetBSD features for years, and still is.

 For one thing, it would allow us to switch the i386 and amd64 test
 runs on the TNF testbed from qemu to Xen, which would reduce the time
 it takes to run the tests from more than 12 hours to a couple of
 hours, and also make us less exposed to qemu's numerous bugs.
 -- 
 Andreas Gustafsson, gson@gson.org

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: port-xen-maintainer@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org, petea_garstea@fincombank.com, gson@gson.org
Subject: Re: port-xen/42441 (DOM0 and DOMU work with only one cpu)
Date: Wed, 20 Jun 2018 12:18:57 +0200

 On Wed, Jun 20, 2018 at 07:30:01AM +0000, Andreas Gustafsson wrote:
 >  It's not good enough for me.  DOM0 SMP support has been at the top of
 >  my wish list of NetBSD features for years, and still is.

 So what is wrong with DOM0 and SMP kernels? Can you reproduce failure?

 Martin

From: Manuel Bouyer <bouyer@antioche.eu.org>
To: Martin Husemann <martin@duskware.de>
Cc: gnats-bugs@NetBSD.org, port-xen-maintainer@netbsd.org,
        gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
        petea_garstea@fincombank.com, gson@gson.org
Subject: Re: port-xen/42441 (DOM0 and DOMU work with only one cpu)
Date: Wed, 20 Jun 2018 12:23:23 +0200

 On Wed, Jun 20, 2018 at 12:18:57PM +0200, Martin Husemann wrote:
 > On Wed, Jun 20, 2018 at 07:30:01AM +0000, Andreas Gustafsson wrote:
 > >  It's not good enough for me.  DOM0 SMP support has been at the top of
 > >  my wish list of NetBSD features for years, and still is.
 > 
 > So what is wrong with DOM0 and SMP kernels? Can you reproduce failure?

 serveral dom0-only drivers (backends, some parts of xenbus) are not MP-safe.

 Just grep for 'splx' here ...

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

From: Martin Husemann <martin@duskware.de>
To: Manuel Bouyer <bouyer@antioche.eu.org>
Cc: gnats-bugs@NetBSD.org
Subject: Re: port-xen/42441 (DOM0 and DOMU work with only one cpu)
Date: Wed, 20 Jun 2018 12:37:31 +0200

 On Wed, Jun 20, 2018 at 12:23:23PM +0200, Manuel Bouyer wrote:
 > serveral dom0-only drivers (backends, some parts of xenbus) are not MP-safe.
 > 
 > Just grep for 'splx' here ...

 That does not tell me a lot without deeper code review and xen knowledge,
 I fear - or I am missing your point.

 Martin

From: Manuel Bouyer <bouyer@antioche.eu.org>
To: Martin Husemann <martin@duskware.de>
Cc: gnats-bugs@NetBSD.org
Subject: Re: port-xen/42441 (DOM0 and DOMU work with only one cpu)
Date: Wed, 20 Jun 2018 13:13:58 +0200

 On Wed, Jun 20, 2018 at 12:37:31PM +0200, Martin Husemann wrote:
 > On Wed, Jun 20, 2018 at 12:23:23PM +0200, Manuel Bouyer wrote:
 > > serveral dom0-only drivers (backends, some parts of xenbus) are not MP-safe.
 > > 
 > > Just grep for 'splx' here ...
 > 
 > That does not tell me a lot without deeper code review and xen knowledge,
 > I fear - or I am missing your point.

 You're right, that's not enough. There is at last one place which
 is not MP-safe and runs without the big lock (AFAIK), it is the
 dom0-specific event code. 

 There may also be issues with the grant tables and xenbus, but I don't remmber
 the details and I may be wrong.

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

State-Changed-From-To: open->closed
State-Changed-By: gdt@NetBSD.org
State-Changed-When: Mon, 15 Mar 2021 13:55:55 +0000
State-Changed-Why:
The consensus on the list is that this works now.  I'm running a dom0 with two cpus; xen
provided two and the XEN3_DOM0 build from the checked-in config (current) detected
and used both.   As always, feel free to open a new PR if there is some failure mode.


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.46 2020/01/03 16:35:01 leot Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2020 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.