NetBSD Problem Report #60478
From www@netbsd.org Wed Jul 22 20:17:24 2026
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits)
client-signature RSA-PSS (2048 bits))
(Client CN "mail.netbsd.org", Issuer "YR2" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id A86931A923C
for <gnats-bugs@gnats.NetBSD.org>; Wed, 22 Jul 2026 20:17:24 +0000 (UTC)
Message-Id: <20260722201723.895741A923D@mollari.NetBSD.org>
Date: Wed, 22 Jul 2026 20:17:23 +0000 (UTC)
From: campbell+netbsd@mumble.net
Reply-To: campbell+netbsd@mumble.net
To: gnats-bugs@NetBSD.org
Subject: releng testbed is busted on VAX
X-Send-Pr-Version: www-1.0
X-From4GNATS: "campbell+netbsd@mumble.net via gnats" <gnats-admin@NetBSD.org>
>Number: 60478
>Category: port-vax
>Synopsis: releng testbed is busted on VAX
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: port-vax-maintainer
>State: analyzed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Wed Jul 22 20:20:00 +0000 2026
>Closed-Date:
>Last-Modified: Sat Aug 15 13:35:01 +0000 2026
>Originator: Taylor R Campbell
>Release: current
>Organization:
The TestBSD Vaxxination, Inc.
>Environment:
>Description:
Something seems to have broken the VAX releng testbed on around
July 19th:
build: OK with 425516 lines of log, install: OK
commit 2026.07.19.16.03.58 kre src/bin/sh/jobs.c 1.128
commit 2026.07.19.16.03.58 kre src/bin/sh/sh.1 1.279
build: OK with 425289 lines of log, install: failed
commit 2026.07.19.16.16.04 kre src/bin/sh/jobs.c 1.129
build: OK with 424685 lines of log, install: failed
...
commit 2026.07.21.04.05.06 riastradh src/libexec/ld.elf_so/search.c 1.30
build: OK with 425375 lines of log, install: failed
...
commit 2026.07.22.08.39.21 mrg src/external/mit/xorg/bin/xcutsel/Makefile 1.3
build: OK with 425121 lines of log, install: failed
First couple failures looked like this:
[ 1.1200030] kern.module.path=/stand/vax/11.99.7/modules
[ 1.1200030] WARNING: preposterous TOD clock time
[ 1.1200030] WARNING: using default initial time
init: kernel security level ch/install.sh: Cannot set tty process group (Bad file descriptor) at 1462
/install.sh: Cannot set tty process group (Bad file descriptor) at 1462
Terminal type (just hit ENTER for 'vt220'): xterm
/install.sh: Cannot set tty process group (Bad file descriptor) at 1462
To return to the installer, quit this shell by typing 'exit' or ^D.
# [ 6.2600030] entropy: best effort
*anita timeout*
Then they looked like:
[ 1.0700030] kern.module.path=/stand/vax/11.99.7/modules
[ 1.0800030] WARNING: preposterous TOD clock time
[ 1.0800030] WARNING: using default initial time
Fatal error
Enter pathname of shell or RETURN for /bin/sh: [ 6.2700030] entropy: best effort
*anita timeout*
>How-To-Repeat:
anita test https://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/20260722083921Z/vax/
>Fix:
Yes, please!
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: port-vax-maintainer->kre
Responsible-Changed-By: riastradh@NetBSD.org
Responsible-Changed-When: Wed, 22 Jul 2026 20:34:24 +0000
Responsible-Changed-Why:
Looks like your recent changes to sh(1) may be related to this, can
you take a look?
https://releng.netbsd.org/b5reports/vax/commits-2026.07.html#build-2026.07.19.15.37.56
From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: port-vax/60478: releng testbed is busted on VAX
Date: Thu, 23 Jul 2026 08:06:31 +0700
Date: Wed, 22 Jul 2026 20:20:00 +0000 (UTC)
From: "campbell+netbsd@mumble.net via gnats" <gnats-admin@NetBSD.org>
Message-ID: <20260722202000.C49AD1A923D@mollari.NetBSD.org>
| Something seems to have broken the VAX releng testbed on around
| July 19th:
That was a sh problem, which was since reverted. It should be fixed
in a new build now. What's the build date of the /bin/sh (presumably the
one on the vax install media) ? (Using /rescue/sh from the same build
wouldn't help, not that /rescue is likely to exist at that point).
| anita test https://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/20260722083921Z/vax/
It should certainly have been fixed before then. What anita is doing
differently on some ports than others would be interesting to know.
The amd64 install failed the same way (and was fixed after the change
was reverted), but the i386 install wasn't bothered by the change at
all (and I am still running it as my /bin/sh).
There's no question but that the alteration (since reverted) wasn't
complete, and could lead to this issue (the "bad file descriptor" would
be -1, as nothing would have attempted to open the controlling tty to use).
The question is why the different path through the shell for amd64 and vax
(and perhaps others) than for i386?
kre
From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@netbsd.org
Cc: kre@netbsd.org, campbell+netbsd@mumble.net
Subject: Re: port-vax/60478: releng testbed is busted on VAX
Date: Fri, 24 Jul 2026 13:01:06 +0300
Robert Elz via gnats wrote:
> | anita test https://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/20260722083921Z/vax/
>
> It should certainly have been fixed before then. What anita is doing
> differently on some ports than others would be interesting to know.
That is probably the wrong question to ask. When tests fail on the
testbed, in the vast majority of cases, it means the testbed has found
a bug in NetBSD (or in this case, more than one), not that NetBSD has
found a bug in the testbed. And when different ports behave
differently, it's usually because of actual differences between the
ports that will also manifest outside the testbed environment.
> The amd64 install failed the same way (and was fixed after the change
> was reverted), but the i386 install wasn't bothered by the change at
> all (and I am still running it as my /bin/sh).
This, too, probably means NetBSD/i386 and NetBSD/amd64 behave
differently, not that anita tests them differently.
> There's no question but that the alteration (since reverted) wasn't
> complete, and could lead to this issue (the "bad file descriptor" would
> be -1, as nothing would have attempted to open the controlling tty to use).
> The question is why the different path through the shell for amd64 and vax
> (and perhaps others) than for i386?
I believe we are dealing with two different bugs. If your sh change
was fully reverted, it can't be the cause of the ongoing vax breakage.
The install console logs show different error messages before and after
the reversion:
https://releng.netbsd.org/b5reports/vax/2026/2026.07.19.16.16.04/install.log
init: kernel security level /install.sh: Cannot set tty process group (Bad file descriptor) at 1462
/install.sh: Cannot set tty process group (Bad file descriptor) at 1462
Terminal type (just hit ENTER for 'vt220'): xterm
/install.sh: Cannot set tty process group (Bad file descriptor) at 1462
https://releng.netbsd.org/b5reports/vax/2026/2026.07.21.04.05.06/install.log
[ 1.0700030] kern.module.path=/stand/vax/11.99.7/modules
[ 1.0800030] WARNING: preposterous TOD clock time
[ 1.0800030] WARNING: using default initial time
Fatal error
Enter pathname of shell or RETURN for /bin/sh:
I have kicked off a couple of builds to narrow down when the new error appeared.
--
Andreas Gustafsson, gson@gson.org
From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@netbsd.org
Cc: kre@netbsd.org, campbell+netbsd@mumble.net
Subject: Re: port-vax/60478: releng testbed is busted on VAX
Date: Fri, 24 Jul 2026 18:01:05 +0300
> I have kicked off a couple of builds to narrow down when the new error appeared.
They have now finished and show that the ongoing issue started with this commit:
2026.07.19.20.06.42 riastradh src/libexec/ld.elf_so/Makefile 1.156
Logs:
https://releng.netbsd.org/b5reports/vax/commits-2026.07.html#2026.07.19.20.06.42
--
Andreas Gustafsson, gson@gson.org
From: Robert Elz <kre@munnari.OZ.AU>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@netbsd.org, campbell+netbsd@mumble.net
Subject: Re: port-vax/60478: releng testbed is busted on VAX
Date: Sat, 25 Jul 2026 08:43:01 +0700
Date: Fri, 24 Jul 2026 13:01:06 +0300
From: Andreas Gustafsson <gson@gson.org>
Message-ID: <27235.14306.517467.218498@guava.gson.org>
| That is probably the wrong question to ask. When tests fail on the
| testbed, in the vast majority of cases, it means the testbed has found
| a bug in NetBSD (or in this case, more than one), not that NetBSD has
| found a bug in the testbed.
Agreed, though in this case it wasn't tests failing, but the install.
I also didn't believe there was any bug in the testbed, I just want to
(if I can) try and work out what the differences are between the i386
(and perhaps others) versions and the amd64 (and clearly vax) version,
so I can understand better why the latter hit the bug I inserted into
sh, and the former did not. Something different is happening there,
but I am not suggesting that difference is a bug - I'd like to know what
it is so I can make sure that sh (which has one bug, or a different one,
at the minute) can get correctly fixed.
[Aside to anyone reading this for fun: the sh bugs I mention aren't ones
you are going to encounter accidentally, you have to work fairly hard to get
things into a state where anything will go wrong, and right now, the
"wrong" tends to be just an inconvenience in most cases ... it was slightly
worse during the brief period my "fix" was installed.]
| And when different ports behave differently, it's usually because
| of actual differences between the ports that will also manifest
| outside the testbed environment.
It could be that, but this all failed very early in the install process
(as in, nothing really ever got installed).
| This, too, probably means NetBSD/i386 and NetBSD/amd64 behave
| differently, not that anita tests them differently.
It could be indeed. The issue (for me) is that there are 4 different
ways that sh can run processes depending upon its state at the time.
For non-interactive non-job control (just running a simple script) it
is all quite simple (and that part works fine). With both interactive
and job control enabled (a typical sh terminal session) it also works.
An interactive shell without job control (something very rare these
days, but which was, long ago, the normal state) setup is also quite simple,
and probably still works as well as it ever did.
Non-interactive shells using job control are where the issue arises, there
is currently an easily demonstrable bug in the way that works (easy to show,
but it takes a weird script to cause it), which is what I was attempting to
fix (but the "fix" was clearly a little too easy ... it involved a minor
change to the condition on exactly 1 "if" statement).
The change could only have affected a non-interactive shell using job
control - so what I am trying to understand is the mechanism by which
the amd64 and vax installs get into that state (it isn't something most
normal shells ever achieve) whereas the i386 one does not. Generally
job control is just on by default in an interactive shell (where it all
works) and off in non-interactive ones (and none of this would happen).
| I believe we are dealing with two different bugs.
I agree.
| If your sh change was fully reverted,
It was, as far as code was concerned - there was also a semi-related
man page update, "semi-related" just because I noticed the wording in
one section (related to the "-i" (interactive) option) could be improved,
and altered that at the same time, nothing really related to the code change,
and that (man page change) I did not revert. That could not sanely
be related (and I have seen your later mail, and know it wasn't).
| it can't be the cause of the ongoing vax breakage.
No, I didn't really think it would be. The PR kind of suggested it was,
but that made no sense to me, and so thanks for taking the time to find
out what it was. The current issue was from enabling DEBUG in ld.elf_so
which turned on all the assert()s - some of which were broken. It will be
interesting to discover if it is still broken now, after (it is at least
believed) that all of the broken ones of those have been fixed.
| The install console logs show different error messages before and after
| the reversion:
|
| https://releng.netbsd.org/b5reports/vax/2026/2026.07.19.16.16.04/install.log
|
| init: kernel security level /install.sh: Cannot set tty process group (Bad file descriptor) at 1462
Yes, that one was mine change. The "bad file descriptor" is that for
the controlling tty, which will be -1, as this is a non-interactive shell
which should have no (relevant) controlling tty (if it were interactive it
would have one, and all would be OK, but non-interactive shells shouldn't be
doing anything to any controlling tty, whether the process has one or not).
Clearly the shell shouldn't be attempting to set process groups on ttys it
doesn't possess, and that I will fix, but I'd really like to find whatever
it is that is causing job control to be enabled -- if it wasn't on, nothing
would be doing anything with process groups - everything just remains in
whichever one the kernel (or its parent) assigned to the shell itself.
| https://releng.netbsd.org/b5reports/vax/2026/2026.07.21.04.05.06/install.log
|
| [ 1.0700030] kern.module.path=/stand/vax/11.99.7/modules
| [ 1.0800030] WARNING: preposterous TOD clock time
| [ 1.0800030] WARNING: using default initial time
| Fatal error
| Enter pathname of shell or RETURN for /bin/sh:
And yes, ld.elf_so calling abort() because of an assert failure (in init
or sh) is a very good candidate for what happened there. (The time related
warning messages I suspect are "normal", and unrelated.)
| I have kicked off a couple of builds to narrow down when the new
| error appeared.
Thanks again for that (and as I said above, I did see the result).
I'd still appreciate knowing if there is any difference at all between
the way anita is run on amd64 (and vax perhaps) and on i386 ? I'm not
suggesting anything is wrong, or needs changing, I'm just looking to
understand the code paths inside sh. If one perhaps uses ptys and the
other pipes or something, that might explain things - at least it would
give me somewhere to look. On the other hand, if the two use identical
configs, then it must be something about the way init or /etc/rc works,
or how the kernel sets up the state of the world when a single user shell
is to be invoked. But I'd prefer to work from how it all starts, and that
is from anita.
kre
From: Andreas Gustafsson <gson@gson.org>
To: Robert Elz <kre@NetBSD.org>
Cc: gnats-bugs@netbsd.org,
campbell+netbsd@mumble.net
Subject: Re: port-vax/60478: releng testbed is busted on VAX
Date: Sat, 25 Jul 2026 10:29:58 +0300
Robert Elz wrote:
> Agreed, though in this case it wasn't tests failing, but the install.
As I see it, it was a failed test of the install :)
> | And when different ports behave differently, it's usually because
> | of actual differences between the ports that will also manifest
> | outside the testbed environment.
>
> It could be that, but this all failed very early in the install process
> (as in, nothing really ever got installed).
These are not really even install failures, they are failures of the
boot process that happen when booting the install media (but may or
may not be limited to install media).
> | I have kicked off a couple of builds to narrow down when the new
> | error appeared.
>
> Thanks again for that (and as I said above, I did see the result).
Thank you for reverting the shell change quickly so that the two
issues did not overlap. :)
> I'd still appreciate knowing if there is any difference at all between
> the way anita is run on amd64 (and vax perhaps) and on i386 ? I'm not
> suggesting anything is wrong, or needs changing, I'm just looking to
> understand the code paths inside sh. If one perhaps uses ptys and the
> other pipes or something, that might explain things - at least it would
> give me somewhere to look. On the other hand, if the two use identical
> configs, then it must be something about the way init or /etc/rc works,
> or how the kernel sets up the state of the world when a single user shell
> is to be invoked. But I'd prefer to work from how it all starts, and that
> is from anita.
Anita does not run "on" the amd64/i386/vax system being tested; it
runs on a host system and boots an unmodified NetBSD system in a
virtual (or sometimes physical) guest machine. Apart from the obvious
differences in architecture, the guest machines will also differ in
other ways such as the amount of memory and disk, but they are all
executing the standard NetBSD boot process of their respective port.
Anita does not itself run any shell commands on the guest machine as
part of the boot or install.
PS. I am replying to your NetBSD.org address because my mail server
185.55.84.130 is unable to reach the munnari.oz.au mail server 202.29.151.3.
--
Andreas Gustafsson, gson@gson.org
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@netbsd.org
Cc: kre@netbsd.org, campbell+netbsd@mumble.net
Subject: Re: port-vax/60478: releng testbed is busted on VAX
Date: Sat, 25 Jul 2026 09:34:12 +0200
On Sat, Jul 25, 2026 at 01:50:02AM +0000, Robert Elz via gnats wrote:
> I'd still appreciate knowing if there is any difference at all between
> the way anita is run on amd64 (and vax perhaps) and on i386 ?
I think this is less an anita related question but more about the install
medium used.
AFAICT there are three ways an install medium gets into sysinst:
- boot from the final medium (maybe after ffs resize) and run sysinst
from installed userland (I think evbarm USB images are in this
category)
- boot a ramdisk kernel ( / on memory disk ) and run sysinst from the
(crunched) binary inside that
- boot a "normal" kernel with / on the ISO filesystem read-only, with
a few tmpfs overlaying and some bizare scripts faking init and
sysinst runing as the login shell. VAX and sparc64 CD/DVD images
are in this category.
There is also a variant (not available as stock install medium) where
a special file system images is prepared to mount as / and the kernel
is loaded from the qemu command line with some args pointing it at
the boot device - but this should not make a difference to the ramdisk
kernel from a process/shell point of view.
So I guess the interesting part for you is to either look at a VAX install
CD directly or check what is going on in src/distrib/vax/cdroms/installcd.
Martin
From: Robert Elz <kre@munnari.OZ.AU>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@netbsd.org, campbell+netbsd@mumble.net
Subject: Re: port-vax/60478: releng testbed is busted on VAX
Date: Sat, 25 Jul 2026 20:20:25 +0700
Date: Sat, 25 Jul 2026 10:29:58 +0300
From: Andreas Gustafsson <gson@gson.org>
Message-ID: <27236.26102.669249.622087@guava.gson.org>
| As I see it, it was a failed test of the install :)
OK!
| These are not really even install failures, they are failures of the
| boot process that happen when booting the install media (but may or
| may not be limited to install media).
Yes, very early indeed.
| Anita does not run "on" the amd64/i386/vax system being tested;
Yes, I understand that, I was just looking to see if any of the way
it configs the guest systems vary between architectures.
| the guest machines will also differ in
| other ways such as the amount of memory and disk,
Neither of those should matter, different kinds of consoles
might though, do they (at least amd64 and i386) use the same
generic type of console, say vga, or com, or do they differ
in that area? That kind of thing might explain some differences.
| but they are all
| executing the standard NetBSD boot process of their respective port.
Unfortunately, x86 systems don't really have one "standard" boot process,
it all depends upon how things are set up.
| Anita does not itself run any shell commands on the guest machine as
| part of the boot or install.
That's fine, I didn't expect it would, but it does set up the environment
in which the guest finds itself, and if there are material differences there,
it might explain things - but if both i386 and amd64 are (say) booting and
then just using the bios to access a VGA console, I'm finding it hard to
see why one would act differently than the other. Perhaps the difference
is even between EFI and BIOS booting - but somewhere, there must be a
difference (that would in some way affect what the shell when it starts
sees as stdin/stderr). I am fairly sure that nothing is passing odd args
to the boot shell to make a difference that way.
| PS. I am replying to your NetBSD.org address because my mail server
| 185.55.84.130 is unable to reach the munnari.oz.au mail server 202.29.151.3.
You apparently live in a bad (network) neighbourhood. The whole of 185/8
was blocked (for everything) due to some (must have been fairly massive)
bad behaviour. I have whitelisted your specific addr now, so that should
work, at least for now -- I need to make sure my manually added whitelists
survive reboots (the blocks certainly do). Fortunately munnari doesn't
reboot very frequently.
But kre@netbsd.org works just fine for avoiding that kind of problem, for
e-mail.
kre
From: Andreas Gustafsson <gson@gson.org>
To: Robert Elz <kre@munnari.OZ.AU>
Cc: gnats-bugs@netbsd.org,
campbell+netbsd@mumble.net
Subject: Re: port-vax/60478: releng testbed is busted on VAX
Date: Sat, 25 Jul 2026 19:59:22 +0300
Robert Elz wrote:
> Neither of those should matter, different kinds of consoles
> might though, do they (at least amd64 and i386) use the same
> generic type of console, say vga, or com, or do they differ
> in that area? That kind of thing might explain some differences.
All three use emulated serial consoles.
> That's fine, I didn't expect it would, but it does set up the environment
> in which the guest finds itself, and if there are material differences there,
> it might explain things - but if both i386 and amd64 are (say) booting and
> then just using the bios to access a VGA console, I'm finding it hard to
> see why one would act differently than the other. Perhaps the difference
> is even between EFI and BIOS booting - but somewhere, there must be a
> difference (that would in some way affect what the shell when it starts
> sees as stdin/stderr). I am fairly sure that nothing is passing odd args
> to the boot shell to make a difference that way.
For what it's worth, i386 boots from emulated floppies and amd64 from
an emulated CD-ROM, but I don't see how that could make a difference,
either.
> | PS. I am replying to your NetBSD.org address because my mail server
> | 185.55.84.130 is unable to reach the munnari.oz.au mail server 202.29.151.3.
>
> You apparently live in a bad (network) neighbourhood. The whole of 185/8
> was blocked (for everything) due to some (must have been fairly massive)
> bad behaviour. I have whitelisted your specific addr now, so that should
> work, at least for now -- I need to make sure my manually added whitelists
> survive reboots (the blocks certainly do). Fortunately munnari doesn't
> reboot very frequently.
Thanks. I use a small local hosting provider who apparently have a /22,
so most of 185/8 is not them.
--
Andreas Gustafsson, gson@gson.org
Responsible-Changed-From-To: kre->port-vax-maintainer
Responsible-Changed-By: kre@NetBSD.org
Responsible-Changed-When: Wed, 05 Aug 2026 16:16:16 +0000
Responsible-Changed-Why:
the shell issue is long reverted.
If there are remaining vax testbed issued, they have some other cause, and the chance
it is anything I can do anything about is negligible
returning responsibility for this to port-vax-maintainer
State-Changed-From-To: open->analyzed
State-Changed-By: rin@NetBSD.org
State-Changed-When: Sat, 15 Aug 2026 13:33:53 +0000
State-Changed-Why:
This was because DEBUG has been enabled for ld.elf_so.
Workaround committed, but still need more investigation.
From: "Rin Okuyama" <rin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/60478 CVS commit: src/libexec/ld.elf_so/arch/vax
Date: Sat, 15 Aug 2026 13:31:02 +0000
Module Name: src
Committed By: rin
Date: Sat Aug 15 13:31:02 UTC 2026
Modified Files:
src/libexec/ld.elf_so/arch/vax: rtld_start.S
Log Message:
ld.elf_so: vax: XXX Temporarily disable DEBUG code
which has been introduced by rtld_start.S,v 1.21:
https://cvsweb.netbsd.org/bsdweb.cgi/src/libexec/ld.elf_so/arch/vax/rtld_start.S#rev1.21
Due to this code, /bin/sh dies by `Fatal error`, which results in:
PR port-vax/60478 releng testbed is busted on VAX
To generate a diff of this commit:
cvs rdiff -u -r1.25 -r1.26 src/libexec/ld.elf_so/arch/vax/rtld_start.S
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home