NetBSD Problem Report #57641

From gson@gson.org  Tue Oct  3 13:16:07 2023
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 269471A923A
	for <gnats-bugs@gnats.NetBSD.org>; Tue,  3 Oct 2023 13:16:07 +0000 (UTC)
Message-Id: <20231003131603.AE130253F95@guava.gson.org>
Date: Tue,  3 Oct 2023 16:16:03 +0300 (EEST)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@NetBSD.org
Subject: -current/i386 no longer installs in 32 MB
X-Send-Pr-Version: 3.95

>Number:         57641
>Category:       port-i386
>Synopsis:       -current/i386 no longer installs in 32 MB
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    port-i386-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Oct 03 13:20:00 +0000 2023
>Last-Modified:  Sat Oct 07 12:00:05 +0000 2023
>Originator:     Andreas Gustafsson
>Release:        NetBSD-current, source date >= 2023.09.23.18.21.43
>Organization:
>Environment:
System: NetBSD
Architecture: i386
Machine: i386
>Description:

In the ongoing discussion re "i386 minimum space requirements" on
port-i386, it was claimed that NetBSD/i386 will successfully install
with 32 MB of RAM.  I figured I'd test that in qemu using -current,
but it didn't work - sysinst hung at the point where it runs
"/usr/sbin/certctl rehash".

This appears to be a recent problem, even more recent than the
introduction of "/usr/sbin/certctl rehash" in sysinst.  A bisection
pointed at this commit:

  2023.09.23.18.21.11 ad src/common/lib/libc/gen/radixtree.c 1.32
  2023.09.23.18.21.11 ad src/sys/kern/init_main.c 1.546
  2023.09.23.18.21.11 ad src/sys/kern/kern_descrip.c 1.261
  2023.09.23.18.21.11 ad src/sys/kern/kern_lwp.c 1.255
  2023.09.23.18.21.11 ad src/sys/kern/kern_mutex_obj.c 1.14
  2023.09.23.18.21.11 ad src/sys/kern/kern_resource.c 1.194
  2023.09.23.18.21.11 ad src/sys/kern/kern_rwlock_obj.c 1.12
  2023.09.23.18.21.11 ad src/sys/kern/kern_turnstile.c 1.49
  2023.09.23.18.21.11 ad src/sys/kern/subr_kcpuset.c 1.20
  2023.09.23.18.21.11 ad src/sys/kern/vfs_cwd.c 1.11
  2023.09.23.18.21.11 ad src/sys/kern/vfs_init.c 1.64
  2023.09.23.18.21.11 ad src/sys/kern/vfs_lockf.c 1.81
  2023.09.23.18.21.11 ad src/sys/rump/librump/rumpkern/rump.c 1.360
  2023.09.23.18.21.12 ad src/sys/rump/librump/rumpvfs/rump_vfs.c 1.97
  2023.09.23.18.21.12 ad src/sys/sys/namei.src 1.64
  2023.09.23.18.21.12 ad src/sys/uvm/uvm_init.c 1.59
  2023.09.23.18.21.12 ad src/sys/uvm/uvm_map.c 1.410
  2023.09.23.18.21.12 ad src/sys/uvm/uvm_readahead.c 1.16
  2023.09.23.18.21.43 ad src/sys/rump/include/rump/rump_namei.h 1.52
  2023.09.23.18.21.43 ad src/sys/sys/namei.h 1.119

For logs, see:

  https://www.gson.org/netbsd/bugs/build/i386-lomem/commits-2023.09.html#2023.09.23.18.21.43

>How-To-Repeat:

  anita --sets base,kern-GENERIC,etc --memory-size 32M install http://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/latest/i386/

>Fix:

>Audit-Trail:
From: Andrew Doran <ad@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: releng@netbsd.org
Subject: Re: port-i386/57641: -current/i386 no longer installs in 32 MB
Date: Wed, 4 Oct 2023 20:17:51 +0000

 Hi,

 It's probably not an satisfying answer but I don't think it's a valid test
 because DIAGNOSTIC uses a lot more memory and we shouldn't be shipping
 actual release kernels with it enabled.

 Looking back over the history for the amd64 GENERIC kernel it seems like
 nobody has disabled it for any release since NetBSD 5 though which sucks.

 Cc: releng people, do we have a procedure for releases somewhere that I
 could update?

 Thanks,
 Andrew

From: Taylor R Campbell <riastradh@NetBSD.org>
To: gnats-bugs@netbsd.org
Cc: port-i386-maintainer@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org, gson@gson.org (Andreas Gustafsson)
Subject: Re: port-i386/57641: -current/i386 no longer installs in 32 MB
Date: Wed, 4 Oct 2023 22:50:25 +0000

 > Date: Wed,  4 Oct 2023 20:20:02 +0000 (UTC)
 > From: Andrew Doran <ad@netbsd.org>
 >=20
 > It's probably not an satisfying answer but I don't think it's a valid test
 > because DIAGNOSTIC uses a lot more memory and we shouldn't be shipping
 > actual release kernels with it enabled.

 How much more memory?  DIAGNOSTIC is supposed to be reasonably cheap,
 mostly cheap predicted-not-taken branches in assertions.  Should we
 move more things to DEBUG instead for expensive checks and higher
 memory usage?

 > Looking back over the history for the amd64 GENERIC kernel it seems like
 > nobody has disabled it for any release since NetBSD 5 though which sucks.

 What history did you check?  It has always been disabled in every
 release branch.  See, e.g., this commit on the netbsd-9 branch:

 http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/amd64/conf/GENERIC?rev=3D1=
 .531.2.4&content-type=3Dtext/x-cvsweb-markup&only_with_tag=3Dnetbsd-9

 I checked every branch back to netbsd-2 and found it disabled.

 > Cc: releng people, do we have a procedure for releases somewhere that I
 > could update?

 I assume there must be a procedure somewhere that already has this!

From: matthew green <mrg@eterna.com.au>
To: Taylor R Campbell <riastradh@NetBSD.org>
Cc: port-i386-maintainer@netbsd.org, gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org, gson@gson.org (Andreas Gustafsson),
    gnats-bugs@netbsd.org, ad@netbsd.org
Subject: re: port-i386/57641: -current/i386 no longer installs in 32 MB
Date: Thu, 05 Oct 2023 10:58:47 +1100

 > > Looking back over the history for the amd64 GENERIC kernel it seems like
 > > nobody has disabled it for any release since NetBSD 5 though which sucks.
 >
 > What history did you check?  It has always been disabled in every
 > release branch.  See, e.g., this commit on the netbsd-9 branch:

 netbsd-10 branch hasn't had it disabled yet, but we normally don't
 until near the release anyway.  netbsd-5 had it disabled in current
 when the branch was made, netbsd-6 disabled it shortly before the
 release (see rev 1.348.2.6), etc.

 my netbsd-5 to netbsd-9 trees all have it disabled for amd64, and
 also i386..


 .mrg.

From: Martin Husemann <martin@duskware.de>
To: Andrew Doran <ad@netbsd.org>
Cc: gnats-bugs@netbsd.org, releng@netbsd.org
Subject: Re: port-i386/57641: -current/i386 no longer installs in 32 MB
Date: Thu, 5 Oct 2023 08:28:23 +0200

 On Wed, Oct 04, 2023 at 08:17:51PM +0000, Andrew Doran wrote:
 > Looking back over the history for the amd64 GENERIC kernel it seems like
 > nobody has disabled it for any release since NetBSD 5 though which sucks.

 Nah, it is disabled as part of the release procedure, but only on the
 release branch before we enter RC1 state.
 It will happen for -10 soon too.

 Martin

From: Andrew Doran <ad@netbsd.org>
To: Taylor R Campbell <riastradh@NetBSD.org>
Cc: gnats-bugs@netbsd.org, port-i386-maintainer@netbsd.org,
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
	Andreas Gustafsson <gson@gson.org>
Subject: Re: port-i386/57641: -current/i386 no longer installs in 32 MB
Date: Fri, 6 Oct 2023 22:26:06 +0000

 On Wed, Oct 04, 2023 at 10:50:25PM +0000, Taylor R Campbell wrote:

 > > From: Andrew Doran <ad@netbsd.org>
 > > 
 > > It's probably not an satisfying answer but I don't think it's a valid test
 > > because DIAGNOSTIC uses a lot more memory and we shouldn't be shipping
 > > actual release kernels with it enabled.
 > 
 > How much more memory?  DIAGNOSTIC is supposed to be reasonably cheap,
 > mostly cheap predicted-not-taken branches in assertions.

 There's not an easy formula for that.  kmem adds sizeof(size_t) to every
 allocation to track that size allocated is the same as size freed, which
 eats a lot more space than expected because alignment requirements (for
 cache friendliness) still need to be met.

 > Should we move more things to DEBUG instead for expensive checks and
 > higher memory usage?

 Yeah I think so.  I'll cover it with DEBUG in a week or so if nobody has a
 serious objection.

 > > Looking back over the history for the amd64 GENERIC kernel it seems like
 > > nobody has disabled it for any release since NetBSD 5 though which sucks.
 > 
 > What history did you check?  It has always been disabled in every
 > release branch.  See, e.g., this commit on the netbsd-9 branch:
 > 
 > http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/amd64/conf/GENERIC?rev=1.531.2.4&content-type=text/x-cvsweb-markup&only_with_tag=netbsd-9

 I looked at the history for amd64 GENERIC on cvsweb.netbsd.org, searched for
 DIAGNOSTIC and it didn't find anything like that.  Maybe the page hadn't
 finished loading or something.  I also had a hazy memory of installing a
 release years ago and being sure it had DIAGNOSTIC enabled.  In any case it
 seems I was wrong, cheers.

 Andrew

From: matthew green <mrg@eterna.com.au>
To: Andrew Doran <ad@netbsd.org>
Cc: gnats-bugs@netbsd.org, port-i386-maintainer@netbsd.org,
    gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
    Andreas Gustafsson <gson@gson.org>,
    Taylor R Campbell <riastradh@NetBSD.org>
Subject: re: port-i386/57641: -current/i386 no longer installs in 32 MB
Date: Sat, 07 Oct 2023 15:51:34 +1100

 Andrew Doran writes:
 > On Wed, Oct 04, 2023 at 10:50:25PM +0000, Taylor R Campbell wrote:
 >
 > > > From: Andrew Doran <ad@netbsd.org>
 > > > =

 > > > It's probably not an satisfying answer but I don't think it's a vali=
 d test
 > > > because DIAGNOSTIC uses a lot more memory and we shouldn't be shippi=
 ng
 > > > actual release kernels with it enabled.
 > > =

 > > How much more memory?  DIAGNOSTIC is supposed to be reasonably cheap,
 > > mostly cheap predicted-not-taken branches in assertions.
 >
 > There's not an easy formula for that.  kmem adds sizeof(size_t) to every
 > allocation to track that size allocated is the same as size freed, which
 > eats a lot more space than expected because alignment requirements (for
 > cache friendliness) still need to be met.

 while i agree it may not be true to the "reasonably cheap" rule,
 i think that the damage caused by wrong kmem_free() size is far
 too dangerous to push this to only DEBUG.

 (i am suddenly thinking about turning on KMEM_SIZE in all my
 kernels anyway, cuz this cost enables something i'd rather see
 crash than corrupt.)


 .mrg.

From: Taylor R Campbell <riastradh@NetBSD.org>
To: matthew green <mrg@eterna.com.au>
Cc: Andrew Doran <ad@netbsd.org>, gnats-bugs@netbsd.org,
	port-i386-maintainer@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org, Andreas Gustafsson <gson@gson.org>
Subject: Re: port-i386/57641: -current/i386 no longer installs in 32 MB
Date: Sat, 7 Oct 2023 11:57:17 +0000

 > Date: Sat, 07 Oct 2023 15:51:34 +1100
 > from: matthew green <mrg@eterna.com.au>
 >=20
 > Andrew Doran writes:
 > > On Wed, Oct 04, 2023 at 10:50:25PM +0000, Taylor R Campbell wrote:
 > >
 > > > > From: Andrew Doran <ad@netbsd.org>
 > > > >=20
 > > > > It's probably not an satisfying answer but I don't think it's a val=
 id test
 > > > > because DIAGNOSTIC uses a lot more memory and we shouldn't be shipp=
 ing
 > > > > actual release kernels with it enabled.
 > > >=20
 > > > How much more memory?  DIAGNOSTIC is supposed to be reasonably cheap,
 > > > mostly cheap predicted-not-taken branches in assertions.
 > >
 > > There's not an easy formula for that.  kmem adds sizeof(size_t) to every
 > > allocation to track that size allocated is the same as size freed, which
 > > eats a lot more space than expected because alignment requirements (for
 > > cache friendliness) still need to be met.
 >=20
 > while i agree it may not be true to the "reasonably cheap" rule,
 > i think that the damage caused by wrong kmem_free() size is far
 > too dangerous to push this to only DEBUG.

 Is this 32MB i386 failure a new regression?  Did it work with
 DIAGNOSTIC and non-DIAGNOSTIC kernels alike before?

Home	PR Database Search