NetBSD Problem Report #58283
From tsutsui@ceres.dti.ne.jp Sat May 25 21:16:26 2024
Return-Path: <tsutsui@ceres.dti.ne.jp>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 66AB21A923A
for <gnats-bugs@gnats.NetBSD.org>; Sat, 25 May 2024 21:16:26 +0000 (UTC)
Message-Id: <202405252116.44PLGKLf025374@ceres.dti.ne.jp>
Date: Sun, 26 May 2024 06:16:20 +0900 (JST)
From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
Reply-To: tsutsui@ceres.dti.ne.jp
To: gnats-bugs@NetBSD.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: NetBSD/macppc 10.0 reproducible kernel PGM trap panic
X-Send-Pr-Version: 3.95
>Number: 58283
>Category: port-macppc
>Synopsis: NetBSD/macppc 10.0 reproducible kernel PGM trap panic
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: port-macppc-maintainer
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sat May 25 21:20:00 +0000 2024
>Closed-Date: Thu Jul 04 14:23:55 +0000 2024
>Last-Modified: Thu Jul 04 14:23:55 +0000 2024
>Originator: Izumi Tsutsui
>Release: NetBSD 10.0
>Organization:
>Environment:
System: NetBSD mirage 10.0 NetBSD 10.0 (GENERIC) #0: Thu Mar 28 08:33:33 UTC 2024 mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/macppc/compile/GENERIC macppc
Architecture: powerpc
Machine: macppc
>Description:
After I upgraded NetBSD/macppc 9.2 to 10.0 on my Apus2000/200 (603e)
I get reproducible panic on multiuser when tcsh binary built on 9.0
is executed.
>How-To-Repeat:
---
Open Firmware, 2.0
To continue booting the MacOS type:
BYE<return>
To continue booting from the default boot device type:
BOOT<return>
ok
0 > boot
OF_open bootpath=ata/ATA-Disk@0
read stage 2 blocks: 0123. done!
starting stage 2...
>> NetBSD/macppc OpenFirmware Boot, Revision 1.13 (Sun Oct 18 19:24:30 UTC 2020)
Cannot use bootpath
Using boot-device instead
11800060+167172 [463008+446635]=0xc481e4
start=0x100000
[ 1.0000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003,
[ 1.0000000] 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013,
[ 1.0000000] 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023,
[ 1.0000000] 2024
[ 1.0000000] The NetBSD Foundation, Inc. All rights reserved.
[ 1.0000000] Copyright (c) 1982, 1986, 1989, 1991, 1993
[ 1.0000000] The Regents of the University of California. All rights reserved.
[ 1.0000000] NetBSD 10.0 (GENERIC) #0: Thu Mar 28 08:33:33 UTC 2024
[ 1.0000000] mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/macppc/compile/GENERIC
[ 1.0000000] total memory = 144 MB
[ 1.0000000] avail memory = 125 MB
[ 1.0000000] found ohare PIC at f3000000
[ 1.0000000] bootpath: ata/ATA-Disk@0/netbsd
[ 1.0000000] mainbus0 (root)
[ 1.0000000] valkyriefb0 at mainbus0 address 0xf1000000
[ 1.0000000] memory0 at mainbus0
[ 1.0000000] bandit0 at mainbus0
[ 1.0000000] pci0 at bandit0 bus 0
[ 1.0000000] pchb0 at pci0 dev 11 function 0
[ 1.0000000] pchb0: Apple Computer Bandit Host-PCI Bridge (rev. 0x02)
[ 1.0000000] ex0 at pci0 dev 13 function 0: 3Com 3c905B-TX 10/100 Ethernet (rev. 0x30)
[ 1.0000000] ex0: interrupting at irq 23
[ 1.0000000] ex0: MAC address 00:50:04:65:51:1a
[ 1.0000000] exphy0 at ex0 phy 24: 3Com internal media interface
[ 1.0000000] exphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
[ 1.0000000] obio0 at pci0 dev 16 function 0: addr 0xf3000000
[ 1.0000000] mesh0 at obio0 offset 0x10000 irq 12: 50MHz, SCSI ID 7
[ 1.0000000] scsibus0 at mesh0: 8 targets, 8 luns per target
[ 1.0000000] zsc0 at obio0 irq 15,16
[ 1.0000000] zstty0 at zsc0 channel 0 (console i/o)
[ 1.0000000] zstty1 at zsc0 channel 1
[ 1.0000000] awacs0 at obio0 offset 0x14000: irq 17,8,9
[ 1.0000000] awacs0: speaker
[ 1.0000000] audio0 at awacs0: playback, capture, full duplex
[ 1.0000000] audio0: slinear_be:16 2ch 44100Hz, blk 4096 bytes (23.2ms) for playback
[ 1.0000000] audio0: slinear_be:16 2ch 44100Hz, blk 4096 bytes (23.2ms) for recording
[ 1.0000000] spkr0 at audio0: PC Speaker (synthesized)
[ 1.0000000] wsbell at spkr0 not configured
[ 1.0000000] swim3 at obio0 offset 0x15000 not configured
[ 1.0000000] nvram0 at obio0 offset 0x60000
[ 1.0000000] cuda0 at obio0 offset 0x16000 irq 18
[ 1.0000000] nadb0 at cuda0: Apple Desktop Bus
[ 1.0000000] iic0 at cuda0: I2C bus
[ 1.0000000] videopll0 at iic0 addr 0x50: Apple onboard video PLL
[ 1.0000000] wdc0 at obio0 offset 0x20000 irq property not found; using 13: DMA transfer
[ 1.0000000] atabus0 at wdc0 channel 0
[ 1.0000000] vmmask 7cc00000 schedmask 7fc00000 highmask 7fc00000
[ 1.0001103] scsibus0: waiting 2 seconds for devices to settle...
[ 1.0299895] WARNING: system needs entropy for security; see entropy(7)
[ 1.3699144] adbkbd0 at nadb0 addr 2: extended keyboard
[ 1.4099181] wskbd0 at adbkbd0 mux 1
[ 1.4099181] wsmouse0 at adbkbd0 mux 0
[ 1.4099181] adbms0 at nadb0 addr 3: 1-button, 100 dpi mouse
[ 1.4699212] wsmouse1 at adbms0 mux 0
[ 4.3400673] wd0 at atabus0 drive 0
[ 4.3601329] wd0: <ST320011A>
[ 4.3601329] wd0: 19092 MB, 38792 cyl, 16 head, 63 sec, 512 bytes/sect x 39102336 sectors
[ 4.7301068] cd0 at scsibus0 target 3 lun 0: <MATSHITA, CD-ROM CR-8024, 2.0e> cdrom removable
[ 6.4900938] swwdog0: software watchdog initialized
[ 6.5301256] valkyriefb0: switched to 800 x 600 in 8 bit colour
[ 6.5801017] wsdisplay1 at valkyriefb0 kbdmux 1
[ 6.5801017] valkyriefb0: switched to 800 x 600 in 8 bit colour
[ 6.6601104] wsdisplay2 at valkyriefb0 kbdmux 1
[ 6.6601104] boot device: wd0
[ 6.6601104] root on wd0a dumps on wd0b
[ 6.7101503] root file system type: ffs
[ 6.7101503] kern.module.path=/stand/macppc/10.0/modules
[ 7.1701851] entropy: best effort
Sun May 26 05:59:18 JST 2024
Starting root file system check:
/dev/rwd0a: file system is clean; not checking
[..snip..]
NetBSD/macppc (lancer) (ttyZ0)
login: root
Password:
May 26 06:00:55 lancer login: ROOT LOGIN (root) on tty ttyZ0
Last login: Sun May 26 05:21:42 2024 on ttyZ0
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003,
2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013,
2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023,
2024
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
NetBSD 10.0 (GENERIC) #0: Thu Mar 28 08:33:33 UTC 2024
Terminal type is vt100.
We recommend creating a non-root account and using su(1) for root access.
lancer# sysctl -w ddb.onpanic=1
ddb.onpanic: 0 -> 1
lancer# file /usr/local/bin/tcsh
/usr/local/bin/tcsh: ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1 (SYSV), dynamically linked, interpreter /usr/libexec/ld.elf_so, for NetBSD 9.0, with debug_info, not stripped
lancer# /usr/local/bin/tcsh
[ 136.9671671] trap: pid 1981.1981 (tcsh): kernel PGM trap @ 0x12cbc8 (SRR1=0x89032)
[ 136.9671671] panic: trap
[ 136.9671671] cpu0: Begin traceback...
[ 136.9671671] 0x00c44cc0: at vpanic+0x158
[ 136.9671671] 0x00c44cf0: at panic+0x50
[ 136.9671671] 0x00c44d30: at trap+0x3a8
[ 136.9671671] 0x00c44de0: kernel PGM trap by vec_state_release+0: srr1=0x89032
[ 136.9671671] r1=0x16fcd80 cr=0x44222424 xer=0 ctr=0x12cbc8
[ 136.9671671] 0x016fcd80: at l3cr_config+0x34
[ 136.9671671] 0x016fcda0: at mi_switch+0x178
[ 136.9671671] 0x016fcde0: at sleepq_block+0xe0
[ 136.9671671] 0x016fce00: at cv_wait_sig+0xc0
[ 136.9671671] 0x016fce30: at pipe_read+0x398
[ 136.9671671] 0x016fce70: at dofileread+0x88
[ 136.9671671] 0x016fcec0: at syscall+0x294
[ 136.9671671] 0x016fcf20: user SC trap #3 by 0xfdc4c128: srr1=0xd032
[ 136.9671671] r1=0xffff7300 cr=0x24224222 xer=0 ctr=0xfdc4c120
[ 136.9671671] cpu0: End traceback...
Stopped in pid 1981.1981 (tcsh) at netbsd:vpanic+0x15c: or r3, r26,
r26
db>
---
Note executing /usr/local/bin/tcsh on single user doesn't trigger
the panic.
BTW, on NetBSD/macppc 9.1 "cpu0" is probed just after mainbus is attached,
but it's no longer probed/attached on 10.0?
---
NetBSD 9.1 (GENERIC) #0: Sun Oct 18 19:24:30 UTC 2020
mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/macppc/compile/GENERIC
total memory = 144 MB
avail memory = 126 MB
timecounter: Timecounters tick every 10.000 msec
Kernelized RAIDframe activated
found ohare PIC at f3000000
bootpath: ata/ATA-Disk@0/netbsd
mainbus0 (root)
cpu0 at mainbus0: 603ev (Revision 2.1), ID 0 (primary)
cpu0: HID0 0x90c001<DOZE,DPM,ICE,DCE,NOPTI>, powersave: 1
cpu0: ohare L2 cache enabled
valkyriefb0 at mainbus0 address 0xf1000000
valkyriefb0: waiting for videopll...
---
https://dmesgd.nycbug.org/index.cgi?do=view&id=5855
>Fix:
No idea.
---
Izumi Tsutsui
>Release-Note:
>Audit-Trail:
From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@netbsd.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: Re: port-macppc/58283: NetBSD/macppc 10.0 reproducible kernel PGM trap
panic
Date: Sun, 26 May 2024 12:01:53 +0900
> BTW, on NetBSD/macppc 9.1 "cpu0" is probed just after mainbus is attached,
> but it's no longer probed/attached on 10.0?
>
> ---
> NetBSD 9.1 (GENERIC) #0: Sun Oct 18 19:24:30 UTC 2020
> mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/macppc/compile/GENERIC
> total memory = 144 MB
> avail memory = 126 MB
> timecounter: Timecounters tick every 10.000 msec
> Kernelized RAIDframe activated
> found ohare PIC at f3000000
> bootpath: ata/ATA-Disk@0/netbsd
> mainbus0 (root)
> cpu0 at mainbus0: 603ev (Revision 2.1), ID 0 (primary)
> cpu0: HID0 0x90c001<DOZE,DPM,ICE,DCE,NOPTI>, powersave: 1
> cpu0: ohare L2 cache enabled
> valkyriefb0 at mainbus0 address 0xf1000000
> valkyriefb0: waiting for videopll...
> ---
> https://dmesgd.nycbug.org/index.cgi?do=view&id=5855
naru@ said this problem was already reported as PR/57394
https://gnats.netbsd.org/57394
It was fixed in HEAD but not pulled up to netbsd-10
(and 10.0 release).
---
Izumi Tsutsui
From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@netbsd.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: Re: port-macppc/58283: NetBSD/macppc 10.0 reproducible kernel PGM trappanic
Date: Sun, 26 May 2024 12:29:36 +0900
> naru@ said this problem was already reported as PR/57394
> https://gnats.netbsd.org/57394
>
> It was fixed in HEAD but not pulled up to netbsd-10
> (and 10.0 release).
Even with this cpu fix, the kernel PGM trap (on exec tcsh) still occurs.
---
[ 80.8829533] trap: pid 2174.2174 (tcsh): kernel PGM trap @ 0x12cbcc (SRR1=0x89032)
[ 80.8829533] panic: trap
[ 80.8829533] cpu0: Begin traceback...
[ 80.8829533] 0x00c44e80: at vpanic+0x158
[ 80.8829533] 0x00c44eb0: at panic+0x50
[ 80.8829533] 0x00c44ef0: at trap+0x3a8
[ 80.8829533] 0x00c44fa0: kernel PGM trap by vec_state_release+0: srr1=0x89032
[ 80.8829533] r1=0x16f4990 cr=0x44242484 xer=0 ctr=0x12cbcc
[ 80.8829533] 0x016f4990: at l3cr_config+0x34
[ 80.8829533] 0x016f49b0: at mi_switch+0x178
---
l3cr_config may be NULL on 603ev per powerpc/oea/cpu_subr.c,
so there are some NULL pointer deference?
---
Izumi Tsutsui
From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@netbsd.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: Re: port-macppc/58283: NetBSD/macppc 10.0 reproducible kernel PGM trap
panic
Date: Tue, 11 Jun 2024 02:00:48 +0900
> lancer# /usr/local/bin/tcsh
> [ 136.9671671] trap: pid 1981.1981 (tcsh): kernel PGM trap @ 0x12cbc8 (SRR1=0x89032)
> [ 136.9671671] panic: trap
> [ 136.9671671] cpu0: Begin traceback...
> [ 136.9671671] 0x00c44cc0: at vpanic+0x158
> [ 136.9671671] 0x00c44cf0: at panic+0x50
> [ 136.9671671] 0x00c44d30: at trap+0x3a8
> [ 136.9671671] 0x00c44de0: kernel PGM trap by vec_state_release+0: srr1=0x89032
> [ 136.9671671] r1=0x16fcd80 cr=0x44222424 xer=0 ctr=0x12cbc8
> [ 136.9671671] 0x016fcd80: at l3cr_config+0x34
0x12cbc8 is 'dssall' in vec_state_release:
---
0012cbc8 <vec_state_release>:
12cbc8: 7e 00 06 6c dssall
12cbcc: 7c 00 04 ac hwsync
12cbd0: 81 43 00 2c lwz r10,44(r3)
12cbd4: 81 2a 00 94 lwz r9,148(r10)
12cbd8: 55 29 01 ca rlwinm r9,r9,0,7,5
12cbdc: 91 2a 00 94 stw r9,148(r10)
12cbe0: 81 23 00 24 lwz r9,36(r3)
12cbe4: 55 29 01 ca rlwinm r9,r9,0,7,5
12cbe8: 91 23 00 24 stw r9,36(r3)
12cbec: 4e 80 00 20 blr
---
But it looks this function is for ALTIVEC, so this should not be
called on 603ev machines?
---
Izumi Tsutsui
From: matthew green <mrg@eterna23.net>
To: gnats-bugs@netbsd.org
Cc: port-macppc-maintainer@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, tsutsui@ceres.dti.ne.jp
Subject: re: port-macppc/58283: NetBSD/macppc 10.0 reproducible kernel PGM trap panic
Date: Tue, 11 Jun 2024 03:19:13 +1000
> 0x12cbc8 is 'dssall' in vec_state_release:
i think the bug here is that vec_restore_from_mcontext() is missing
this check that is in vec_save_to_mcontext():
/* Save AltiVec context, if any. */
if (!vec_used_p(l))
return false;
can you try adding this near the top of vec_restore_from_mcontext()?
(with just return; void function.)
.mrg.
From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: mrg@eterna23.net
Cc: gnats-bugs@netbsd.org, tsutsui@ceres.dti.ne.jp
Subject: Re: port-macppc/58283: NetBSD/macppc 10.0 reproducible kernel PGM trap
panic
Date: Tue, 11 Jun 2024 03:06:27 +0900
> > 0x12cbc8 is 'dssall' in vec_state_release:
>
> i think the bug here is that vec_restore_from_mcontext() is missing
> this check that is in vec_save_to_mcontext():
>
> /* Save AltiVec context, if any. */
> if (!vec_used_p(l))
> return false;
>
> can you try adding this near the top of vec_restore_from_mcontext()?
> (with just return; void function.)
Hmm, indeed the panic no longer happens with the following patch.
(I'm not sure why netbsd-9 worked though)
---
Index: altivec.c
===================================================================
RCS file: /cvsroot/src/sys/arch/powerpc/oea/altivec.c,v
retrieving revision 1.34
diff -u -p -d -r1.34 altivec.c
--- altivec.c 30 Oct 2021 19:44:56 -0000 1.34
+++ altivec.c 10 Jun 2024 18:01:52 -0000
@@ -163,6 +163,10 @@ vec_restore_from_mcontext(struct lwp *l,
KASSERT(l == curlwp);
+ /* Nothing to do if no AltiVec context. */
+ if (!vec_used_p(l))
+ return;
+
/* we don't need to save the state, just drop it */
pcu_discard(&vec_ops, l, true);
---
Thanks,
---
Izumi Tsutsui
From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@netbsd.org
Cc: mrg@eterna23.net, tsutsui@ceres.dti.ne.jp
Subject: Re: port-macppc/58283: NetBSD/macppc 10.0 reproducible kernel PGM trappanic
Date: Tue, 11 Jun 2024 23:15:39 +0900
> Hmm, indeed the panic no longer happens with the following patch.
> (I'm not sure why netbsd-9 worked though)
>
> ---
> Index: altivec.c
> ===================================================================
> RCS file: /cvsroot/src/sys/arch/powerpc/oea/altivec.c,v
> retrieving revision 1.34
> diff -u -p -d -r1.34 altivec.c
> --- altivec.c 30 Oct 2021 19:44:56 -0000 1.34
> +++ altivec.c 10 Jun 2024 18:01:52 -0000
> @@ -163,6 +163,10 @@ vec_restore_from_mcontext(struct lwp *l,
>
> KASSERT(l == curlwp);
>
> + /* Nothing to do if no AltiVec context. */
> + if (!vec_used_p(l))
> + return;
> +
> /* we don't need to save the state, just drop it */
> pcu_discard(&vec_ops, l, true);
This is triggered by the following commit for PR/56471:
https://github.com/NetBSD/src/commit/abf148505b7dc713c84cfb0a31b3849c863f4500
>> - In vec_restore_from_mcontext() and vec_save_to_mcontext(), allows the
>> mcontext argument to be NULL.
>> - In sendsig_sigcontext(), don't set PSL_VEC in the saved MSR; we can't
>> actually round-trip the AltiVec registers. At least get them saved
>> into the PCB by calling vec_save_to_mcontext() (with a NULL mcontext
>> argument).
>> - In compat_16_sys___sigreturn14(), call vec_restore_from_mcontext()
>> with a NULL mcontext argument, which will force any subsequent use
>> of AltiVec to re-load the AltiVec registers from the PCB.
After this change vec_restore_from_mcontext() is unconditionally called
from sendsig_sigcontext() in sys/arch/powerpc/powerpc/compat_16_machdep.c
and it doesn't have the vec_used_p() check.
I'm not sure if the caller should check it or not, but
it's more consistent and less scary to have a vec_used_p() check
in vec_restore_from_mcontext() as vec_save_to_mcontext() does,
even current all fpu_restore_from_mcontext() calls in
sys/arch/powerpc/powerpc/sig_machdep.c have checks
against _UC_POWERPC_VEC or _UC_POWERPC_SPE.
Thanks,
---
Izumi Tsutsui
From: "matthew green" <mrg@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/58283 CVS commit: src/sys/arch/powerpc/oea
Date: Sat, 15 Jun 2024 19:48:13 +0000
Module Name: src
Committed By: mrg
Date: Sat Jun 15 19:48:13 UTC 2024
Modified Files:
src/sys/arch/powerpc/oea: altivec.c
Log Message:
don't do anything in vec_restore_from_mcontext() if no altivec.
fixes a crash seen on netbsd-10 in PR#58283.
XXX: pullup-10.
To generate a diff of this commit:
cvs rdiff -u -r1.34 -r1.35 src/sys/arch/powerpc/oea/altivec.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->pending-pullups
State-Changed-By: tsutsui@NetBSD.org
State-Changed-When: Fri, 28 Jun 2024 17:19:26 +0000
State-Changed-Why:
[pullup-10 #732]
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/58283 CVS commit: [netbsd-10] src/sys/arch/powerpc/oea
Date: Wed, 3 Jul 2024 18:12:35 +0000
Module Name: src
Committed By: martin
Date: Wed Jul 3 18:12:35 UTC 2024
Modified Files:
src/sys/arch/powerpc/oea [netbsd-10]: altivec.c
Log Message:
Pull up following revision(s) (requested by tsutsui in ticket #732):
sys/arch/powerpc/oea/altivec.c: revision 1.35
don't do anything in vec_restore_from_mcontext() if no altivec.
fixes a crash seen on netbsd-10 in PR#58283.
To generate a diff of this commit:
cvs rdiff -u -r1.34 -r1.34.4.1 src/sys/arch/powerpc/oea/altivec.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: pending-pullups->closed
State-Changed-By: tsutsui@NetBSD.org
State-Changed-When: Thu, 04 Jul 2024 14:23:55 +0000
State-Changed-Why:
Pulled up.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home