NetBSD Problem Report #57121

From brad@anduin.eldar.org  Mon Dec 19 15:19:08 2022
Return-Path: <brad@anduin.eldar.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 4F5441A921F
	for <gnats-bugs@gnats.NetBSD.org>; Mon, 19 Dec 2022 15:19:08 +0000 (UTC)
Message-Id: <202212191519.2BJFJ3a1004998@anduin.eldar.org>
Date: Mon, 19 Dec 2022 10:19:03 -0500 (EST)
From: brad@anduin.eldar.org
Reply-To: brad@anduin.eldar.org
To: gnats-bugs@NetBSD.org
Subject: Pure NetBSD PVH guests do not appear to be able to use more than 4GB of memory
X-Send-Pr-Version: 3.95

>Number:         57121
>Category:       port-xen
>Synopsis:       Pure NetBSD PVH guests do not appear to be able to use more than 4GB of memory
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    bouyer
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Dec 19 15:20:00 +0000 2022
>Closed-Date:    Sat Dec 24 09:41:07 +0000 2022
>Last-Modified:  Sat Dec 24 09:41:07 +0000 2022
>Originator:     brad@anduin.eldar.org
>Release:        NetBSD 10.0_BETA
>Organization:
	eldar.org
>Environment:
System: NetBSD nbsd10test 10.0_BETA NetBSD 10.0_BETA (GENERIC) #0: Sat Dec 17 10:31:04 EST 2022  brad@samwise.nat.eldar.org:/lhome/NetBSD_10_branch_20221216/amd64/OBJ/sys/arch/amd64/compile/GENERIC amd64
Architecture: x86_64
Machine: amd64
>Description:

With a pure PVH 64bit NetBSD guest using the GENERIC kernel, the guest
does not appear to be able to use more than 4GB of memory.  That is,
no matter how much memory you give the guest the guest caps it at 4GB.
This does not happen with ArchLinux guests on the same DOM0.  The same
guest configured as a PV system with pvshim sees all of the memory.

>How-To-Repeat:

Simple config for the PVH guest with redacted MAC address:

kernel = "/lhome/xen/kernels/NetBSD_10.x/NetBSD_10_branch_20221216/amd64/netbsd-GENERIC"
memory = 8192
cpu_weight = 32
type="pvh"
name = "nbsd10test"
vcpus = 1
vif = [ 'mac=WH:AT:TE:VE:R0:00, bridge=bridge4' ]
disk = [ 'phy:/dev/mapper/rustvg0-nbsd10testlv0,0x01,w','phy:/dev/mapper/rustvg0-nbsd10testlv1,0x02,w','phy:/dev/mapper/rustvg0-nbsd10testlv2,0x03,w','phy:/dev/mapper/rustvg0-nbsd10testlv3,0x04,w' ]
root = "xbd0"

From the DOM0:

DOM0# xl list nbsd10test -v
Name                                        ID   Mem VCPUs      State   Time(s)   UUID                            Reason-Code                                                     Security Label
nbsd10test                                  33  8191     1     -b----       1.4 e7c5bab3-2bd9-4994-9ef9-b80af5e0b927        -                -


From the guest:

nbsd10test# dmesg|grep memory
[     1.000000] total memory = 4096 MB
[     1.000000] avail memory = 3936 M
nbsd10test# cat /proc/meminfo
        total:    used:    free:  shared: buffers: cached:
Mem:  4137267200 150634496 3986632704        0 11395072 56442880
Swap: 10737414144        0 10737414144
MemTotal:   4040300 kB
MemFree:    3893196 kB
MemShared:        0 kB
Buffers:    3893196 kB
Cached:       55120 kB
SwapTotal: 10485756 kB
SwapFree:  10485756 kB


The DOM0 is a NetBSD/amd64 system running 9.3_STABLE with
xenkernel415-4.15.3 from pkgsrc 2022Q3.  As mentioned ArchLinux PVH
guests see all of the memory given to them.

>Fix:

You can work around this by running the guest as a PV guest with or
without pvshim.  However, the desire, or need, to run the guest in
pure PVH mode will likely become more desirable or required as time
goes on.

>Release-Note:

>Audit-Trail:

Responsible-Changed-From-To: port-xen-maintainer->bouyer
Responsible-Changed-By: bouyer@NetBSD.org
Responsible-Changed-When: Fri, 23 Dec 2022 16:09:12 +0000
Responsible-Changed-Why:
.


State-Changed-From-To: open->feedback
State-Changed-By: bouyer@NetBSD.org
State-Changed-When: Fri, 23 Dec 2022 16:09:12 +0000
State-Changed-Why:
Candidate fix commited to HEAD.


From: "Manuel Bouyer" <bouyer@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/57121 CVS commit: src/sys/arch/x86/x86
Date: Fri, 23 Dec 2022 16:05:44 +0000

 Module Name:	src
 Committed By:	bouyer
 Date:		Fri Dec 23 16:05:44 UTC 2022

 Modified Files:
 	src/sys/arch/x86/x86: x86_machdep.c

 Log Message:
 x86_add_cluster() takes the end of the segment, not the size.
 Should fix PR port-xen/57121


 To generate a diff of this commit:
 cvs rdiff -u -r1.152 -r1.153 src/sys/arch/x86/x86/x86_machdep.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: Manuel Bouyer <bouyer@antioche.eu.org>
To: gnats-bugs@netbsd.org
Cc: port-xen-maintainer@netbsd.org, gnats-admin@netbsd.org,
        netbsd-bugs@netbsd.org
Subject: Re: port-xen/57121: Pure NetBSD PVH guests do not appear to be able
 to use more than 4GB of memory
Date: Fri, 23 Dec 2022 17:08:19 +0100

 --cKygguVIx5P7Hdxx
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline

 On Mon, Dec 19, 2022 at 03:20:00PM +0000, brad@anduin.eldar.org wrote:
 > With a pure PVH 64bit NetBSD guest using the GENERIC kernel, the guest
 > does not appear to be able to use more than 4GB of memory.  That is,
 > no matter how much memory you give the guest the guest caps it at 4GB.
 > This does not happen with ArchLinux guests on the same DOM0.  The same
 > guest configured as a PV system with pvshim sees all of the memory.

 Hello
 I just commited the attached patch to HEAD; it should apply cleanly to
 netbsd-10 too. Can you test ?
 I can't test with much more than 4GB on my devel box (not enough RAM) ...

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

 --cKygguVIx5P7Hdxx
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: attachment; filename=diff

 Index: x86/x86/x86_machdep.c
 ===================================================================
 RCS file: /cvsroot/src/sys/arch/x86/x86/x86_machdep.c,v
 retrieving revision 1.152
 diff -u -p -r1.152 x86_machdep.c
 --- x86/x86/x86_machdep.c	20 Aug 2022 23:48:51 -0000	1.152
 +++ x86/x86/x86_machdep.c	23 Dec 2022 16:02:54 -0000
 @@ -870,11 +870,13 @@ x86_add_xen_clusters(void)
  			switch (map_entry[i].type) {
  			case XEN_HVM_MEMMAP_TYPE_RAM:
  				x86_add_cluster(map_entry[i].addr,
 -				    map_entry[i].size, BIM_Memory);
 +				    map_entry[i].addr + map_entry[i].size,
 +				    BIM_Memory);
  				break;
  			case XEN_HVM_MEMMAP_TYPE_ACPI:
  				x86_add_cluster(map_entry[i].addr,
 -				    map_entry[i].size, BIM_ACPI);
 +				    map_entry[i].addr + map_entry[i].size,
 +				    BIM_ACPI);
  				break;
  			}
  		}

 --cKygguVIx5P7Hdxx--

From: Brad Spencer <brad@anduin.eldar.org>
To: Manuel Bouyer <bouyer@antioche.eu.org>
Cc: gnats-bugs@netbsd.org, port-xen-maintainer@netbsd.org,
        gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: port-xen/57121: Pure NetBSD PVH guests do not appear to be able
 to use more than 4GB of memory
Date: Fri, 23 Dec 2022 11:14:43 -0500

 Manuel Bouyer <bouyer@antioche.eu.org> writes:

 > [1:text/plain Hide]
 >
 > On Mon, Dec 19, 2022 at 03:20:00PM +0000, brad@anduin.eldar.org wrote:
 >> With a pure PVH 64bit NetBSD guest using the GENERIC kernel, the guest
 >> does not appear to be able to use more than 4GB of memory.  That is,
 >> no matter how much memory you give the guest the guest caps it at 4GB.
 >> This does not happen with ArchLinux guests on the same DOM0.  The same
 >> guest configured as a PV system with pvshim sees all of the memory.
 >
 > Hello
 > I just commited the attached patch to HEAD; it should apply cleanly to
 > netbsd-10 too. Can you test ?
 > I can't test with much more than 4GB on my devel box (not enough RAM) ...


 Yes, no problem in testing this.  I should have the results sometime
 today.  Thanks...




 -- 
 Brad Spencer - brad@anduin.eldar.org - KC8VKS - http://anduin.eldar.org

From: Brad Spencer <brad@anduin.eldar.org>
To: Manuel Bouyer <bouyer@antioche.eu.org>
Cc: gnats-bugs@netbsd.org, port-xen-maintainer@netbsd.org,
        gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: port-xen/57121: Pure NetBSD PVH guests do not appear to be able
 to use more than 4GB of memory
Date: Fri, 23 Dec 2022 12:41:50 -0500

 Manuel Bouyer <bouyer@antioche.eu.org> writes:

 > On Mon, Dec 19, 2022 at 03:20:00PM +0000, brad@anduin.eldar.org wrote:
 >> With a pure PVH 64bit NetBSD guest using the GENERIC kernel, the guest
 >> does not appear to be able to use more than 4GB of memory.  That is,
 >> no matter how much memory you give the guest the guest caps it at 4GB.
 >> This does not happen with ArchLinux guests on the same DOM0.  The same
 >> guest configured as a PV system with pvshim sees all of the memory.
 >
 > Hello
 > I just commited the attached patch to HEAD; it should apply cleanly to
 > netbsd-10 too. Can you test ?
 > I can't test with much more than 4GB on my devel box (not enough RAM) ...


 Thanks again.. this appears to have fixed the problem.  I tested with
 8GB and it reported just fine, as did 1GB on a netbsd-10 PVH DOMU using
 the GENERIC kernel.  The patch applied just fine to 10, as you
 mentioned.




 -- 
 Brad Spencer - brad@anduin.eldar.org - KC8VKS - http://anduin.eldar.org

From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/57121 CVS commit: [netbsd-10] src/sys/arch/x86/x86
Date: Sat, 24 Dec 2022 09:26:56 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Sat Dec 24 09:26:56 UTC 2022

 Modified Files:
 	src/sys/arch/x86/x86 [netbsd-10]: x86_machdep.c

 Log Message:
 Pull up following revision(s) (requested by bouyer in ticket #21):

 	sys/arch/x86/x86/x86_machdep.c: revision 1.153

 x86_add_cluster() takes the end of the segment, not the size.

 Should fix PR port-xen/57121


 To generate a diff of this commit:
 cvs rdiff -u -r1.152 -r1.152.4.1 src/sys/arch/x86/x86/x86_machdep.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: feedback->closed
State-Changed-By: bouyer@NetBSD.org
State-Changed-When: Sat, 24 Dec 2022 09:41:07 +0000
State-Changed-Why:
Fix pulled up to netbsd-10.


>Unformatted:

Home	PR Database Search