NetBSD Problem Report #59017
From www@netbsd.org Tue Jan 21 10:05:37 2025
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits)
client-signature RSA-PSS (2048 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 05FB81A923A
for <gnats-bugs@gnats.NetBSD.org>; Tue, 21 Jan 2025 10:05:37 +0000 (UTC)
Message-Id: <20250121100535.712741A923B@mollari.NetBSD.org>
Date: Tue, 21 Jan 2025 10:05:35 +0000 (UTC)
From: campbell+netbsd@mumble.net
Reply-To: campbell+netbsd@mumble.net
To: gnats-bugs@NetBSD.org
Subject: ruby32 crashes in asciidoctor
X-Send-Pr-Version: www-1.0
>Number: 59017
>Category: pkg
>Synopsis: ruby32 crashes in asciidoctor
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: taca
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Tue Jan 21 10:10:00 +0000 2025
>Last-Modified: Sat Mar 22 08:20:02 +0000 2025
>Originator: Taylor R Campbell
>Release: netbsd-9 userland on netbsd-10 kernel, pkgsrc-current, ruby32-base-3.2.5
>Organization:
The NetBSigaltstackD Gemdation
>Environment:
>Description:
While trying to build a package that uses asciidoctor, ruby crashes with SIGSEGV:
asciidoctor --require=./doc/man.rb --backend=manpage doc/manpage-newsboat.asciidoc
/home/riastradh/pkgsrc/git/cross/pkg/lib/ruby/3.2.0/x86_64-netbsd/rbconfig.rb: [BUG] Segmentation fault
ruby 3.2.6 (2024-10-30 revision 63aeb018eb) [x86_64-netbsd]
-- Control frame information -----------------------------------------------
c:0006 p:---- s:0027 e:000026 DUMMY [FINISH]
c:0005 p:---- s:0024 e:000023 CFUNC :require
c:0004 p:0005 s:0019 e:000018 TOP /home/riastradh/pkgsrc/git/cross/pkg/lib/ruby/3.2.0/rubygems.rb:9 [FINISH]
c:0003 p:---- s:0012 e:000011 CFUNC :require
c:0002 p:0012 s:0007 e:000006 TOP <internal:gem_prelude>:2 [FINISH]
c:0001 p:0000 s:0003 E:000a40 DUMMY [FINISH]
-- Ruby level backtrace information ----------------------------------------
<internal:gem_prelude>:2:in `<internal:gem_prelude>'
<internal:gem_prelude>:2:in `require'
/home/riastradh/pkgsrc/git/cross/pkg/lib/ruby/3.2.0/rubygems.rb:9:in `<top (required)>'
/home/riastradh/pkgsrc/git/cross/pkg/lib/ruby/3.2.0/rubygems.rb:9:in `require'
The crash happened with and without the ruby-yjit option. I've seen this several times over the past few years (with earlier versions of ruby) so I went digging this time now that I have debug data.
The stack trace from gdb is not helpful -- it looks like the error-reporting logic in the SIGSEGV handler itself crashed with SIGSEGV:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007e845f041dd1 in uleb128 (p=0x7f7fff8b5ac0) at addr2line.c:206
206 unsigned char b = (unsigned char)*(*p)++;
(gdb) bt
#0 0x00007e845f041dd1 in uleb128 (p=0x7f7fff8b5ac0) at addr2line.c:206
#1 di_read_debug_abbrev_cu (reader=0x7f7fff8b5b00) at addr2line.c:1048
#2 di_read_cu (reader=0x7f7fff8b5b00) at addr2line.c:1737
#3 fill_lines (num_traces=num_traces@entry=5,
traces=traces@entry=0x7e845f394780 <trace>,
check_debuglink=check_debuglink@entry=1, objp=objp@entry=0x7f7fff8b6478,
lines=lines@entry=0x7e845f504f80, offset=offset@entry=0)
at addr2line.c:2167
#4 0x00007e845f043bc1 in rb_dump_backtrace_with_lines (
num_traces=<optimized out>, traces=traces@entry=0x7e845f394780 <trace>)
at addr2line.c:2611
#5 0x00007e845f039670 in rb_print_backtrace () at vm_dump.c:787
#6 0x00007e845f03976c in rb_vm_bugreport (ctx=ctx@entry=0x0) at vm_dump.c:1080
#7 0x00007e845eeb2598 in rb_bug_for_fatal_signal (default_sighandler=0x0,
sig=sig@entry=11, ctx=ctx@entry=0x0,
fmt=fmt@entry=0x7e845f08a6ea "Segmentation fault") at error.c:813
#8 0x00007e845efb83b0 in sigsegv (sig=11) at signal.c:964
#9 0x00007e845d0a1da0 in _opendir (name=<optimized out>)
at /usr/src/9/lib/libc/gen/opendir.c:72
#10 0x000000010000000b in ?? ()
#11 0x0000000000000000 in ?? ()
Digging into ruby32-base, I discovered that on NetBSD, and only on NetBSD, Ruby avoids sigaltstack for the SIGSEGV handler, and also avoids getting at the siginfo_t. The ChangeLog suggests sigaltstack was disabled on NetBSD back in 2010, probably for the benefit of the ancient netbsd<=5 ABI where the pthread id was based on the stack pointer rather than stored in a thread-local storage register as it has been for a long time:
Mon Feb 15 17:42:20 2010 NARUSE, Yui <naruse@ruby-lang.org>
* signal.c (USE_SIGALTSTACK): NetBSD can't use sigaltstack(2)
with pthread.
http://netbsd.gw.com/cgi-bin/man-cgi?sigaltstack++NetBSD-current
The note about incompatibility of sigaltstack(2) with pthread was taken out of NetBSD's man page in 2012, some time after it became irrelevant with the introduction of thread-local storage support in netbsd-6.
I tried patching vm_core.h in ruby32-base to re-enable sigaltstack in order to get better diagnostics with the following patch:
$NetBSD$
Enable sigaltstack(2) again on NetBSD. This has not been relevant for
a very long time, probably since NetBSD 6.
--- vm_core.h.orig 2024-10-30 09:47:11.000000000 +0000
+++ vm_core.h
@@ -148,7 +148,7 @@ extern int ruby_assert_critical_section_
/* define to 0 to test old code path */
#define WAITPID_USE_SIGCHLD (RUBY_SIGCHLD || SIGCHLD_LOSSY)
-#if defined(SIGSEGV) && defined(HAVE_SIGALTSTACK) && defined(SA_SIGINFO) && !defined(__NetBSD__)
+#if defined(SIGSEGV) && defined(HAVE_SIGALTSTACK) && defined(SA_SIGINFO)
# define USE_SIGALTSTACK
void *rb_allocate_sigaltstack(void);
void *rb_register_sigaltstack(void *);
When I did this, I was not able to get better diagnostics out of gdb...because asciidoctor just worked and didn't crash any more! Tried both with and without ruby-yjit, worked either way.
I still don't know what provoked the original SIGSEGV. But we should maybe just allow Ruby to use sigaltstack on NetBSD like it does on all the other platforms.
>How-To-Repeat:
try to use textproc/ruby-asciidoctor on netbsd-9, probably
>Fix:
1. patch away the `&& !defined(__NetBSD__)' condition on the use of sigaltstack in lang/ruby32-base
2. maybe do the same for lang/ruby31-base, lang/ruby33, lang/ruby34, and anything in wip
3. submit the patch upstream (maybe conditionalized on __NetBSD_Version__ being at least netbsd-6 if they care about really old NetBSD)
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: pkg-manager->taca
Responsible-Changed-By: riastradh@NetBSD.org
Responsible-Changed-When: Tue, 21 Jan 2025 11:52:22 +0000
Responsible-Changed-Why:
Could I trouble you to take a look, and if you think this is a good
idea, submit it upstream?
From: Takahiro Kambe <taca@NetBSD.org>
To: gnats-bugs@netbsd.org, campbell+netbsd@mumble.net
Cc:
Subject: Re: pkg/59017: ruby32 crashes in asciidoctor
Date: Tue, 21 Jan 2025 23:32:27 +0900 (JST)
Hi,
In message <20250121101000.819DC1A923C@mollari.NetBSD.org>
on Tue, 21 Jan 2025 10:10:00 +0000 (UTC),
campbell+netbsd@mumble.net wrote:
>>Number: 59017
>>Category: pkg
>>Synopsis: ruby32 crashes in asciidoctor
>>Confidential: no
>>Severity: serious
>>Priority: medium
>>Responsible: pkg-manager
>>State: open
>>Class: sw-bug
>>Submitter-Id: net
>>Arrival-Date: Tue Jan 21 10:10:00 +0000 2025
>>Originator: Taylor R Campbell
>>Release: netbsd-9 userland on netbsd-10 kernel, pkgsrc-current, ruby32-base-3.2.5
I beleive it is ruby32-base-3.2.6.
And I could not reproduce the problem on my NetBSD 10.0_STABLE environment.
>>Organization:
> The NetBSigaltstackD Gemdation
>>Environment:
>>Description:
> While trying to build a package that uses asciidoctor, ruby crashes with SIGSEGV:
>
> asciidoctor --require=./doc/man.rb --backend=manpage doc/manpage-newsboat.asciidoc
Is there any example file to reproduce this problem?
--
Takahiro Kambe <taca@NetBSD.org>
From: Taylor R Campbell <riastradh@NetBSD.org>
To: Takahiro Kambe <taca@NetBSD.org>
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: pkg/59017: ruby32 crashes in asciidoctor
Date: Tue, 21 Jan 2025 16:51:07 +0000
> Date: Tue, 21 Jan 2025 23:32:27 +0900 (JST)
> From: Takahiro Kambe <taca@NetBSD.org>
>=20
> >>Release: netbsd-9 userland on netbsd-10 kernel, pkgsrc-current, =
ruby32-base-3.2.5
> I beleive it is ruby32-base-3.2.6.
Yes, you're right, copypasta error.
$ pkg_info -I ruby32-base
ruby32-base-3.2.6 Ruby 3.2.6 release minimum base package
> > While trying to build a package that uses asciidoctor, ruby crashes wit=
h SIGSEGV:
> >=20
> > asciidoctor --require=3D./doc/man.rb --backend=3Dmanpage doc/manpage-n=
ewsboat.asciidoc
> Is there any example file to reproduce this problem?
I pushed the package to wip/newsboat237; maybe you can use that to
reproduce it. If that's not convenient I can try pulling out the
asciidoc input to see if it reproduces outside the build environment.
That said, while I think it's worthwhile to diagnose how this crashed,
what do you think of removing the special case of avoiding sigaltstack
on NetBSD?
From: Taylor R Campbell <riastradh@NetBSD.org>
To: Takahiro Kambe <taca@NetBSD.org>
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: pkg/59017: ruby32 crashes in asciidoctor
Date: Tue, 21 Jan 2025 19:16:43 +0000
OK, I spoke too soon: the crash still happens sometimes with the
patch, but it's stochastic; I guess I just got lucky with the first
few trials.
However, with the patch to enable sigaltstack -- and, more
importantly, to enable SA_SIGINFO -- I have more information now.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007a007667b701 in uleb128 (p=3D0x7a007623dca0) at addr2line.c:206
206 addr2line.c: No such file or directory.
(gdb) bt
#0 0x00007a007667b701 in uleb128 (p=3D0x7a007623dca0) at addr2line.c:206
#1 di_read_debug_abbrev_cu (reader=3D0x7a007623dce0) at addr2line.c:1048
#2 di_read_cu (reader=3D0x7a007623dce0) at addr2line.c:1737
#3 fill_lines (num_traces=3Dnum_traces@entry=3D5,
traces=3Dtraces@entry=3D0x7a0076b26680 <trace>,
check_debuglink=3Dcheck_debuglink@entry=3D1, objp=3Dobjp@entry=3D0x7a00=
7623e658,
lines=3Dlines@entry=3D0x7a0075eb6a00, offset=3Doffset@entry=3D0)
at addr2line.c:2167
#4 0x00007a007667d4f1 in rb_dump_backtrace_with_lines (
num_traces=3D<optimized out>, traces=3Dtraces@entry=3D0x7a0076b26680 <t=
race>)
at addr2line.c:2611
#5 0x00007a00766716d0 in rb_print_backtrace () at vm_dump.c:787
#6 0x00007a00766717cc in rb_vm_bugreport (ctx=3Dctx@entry=3D0x7a007623ea20)
at vm_dump.c:1080
#7 0x00007a00764e9f98 in rb_bug_for_fatal_signal (default_sighandler=3D0x0,
sig=3Dsig@entry=3D11, ctx=3Dctx@entry=3D0x7a007623ea20,
fmt=3Dfmt@entry=3D0x7a00767bb52e "Segmentation fault at %p") at error.c=
:813
#8 0x00007a00765f0084 in sigsegv (sig=3D11, info=3D0x7a007623e9a0,
ctx=3D0x7a007623ea20) at signal.c:964
#9 0x00007a00746a1da0 in _opendir (name=3D<optimized out>)
at /usr/src/9/lib/libc/gen/opendir.c:72
#10 0x000000010000000b in ?? ()
#11 0x0000000000000000 in ?? ()
(gdb) fr 8
#8 0x00007a00765f0084 in sigsegv (sig=3D11, info=3D0x7a007623e9a0,
ctx=3D0x7a007623ea20) at signal.c:964
964 rb_bug_for_fatal_signal(default_sigsegv_handler, sig, SIGINFO_C=
TX, "Segmentation fault" MESSAGE_FAULT_ADDRESS);
(gdb) print *info
$1 =3D {
si_pad =3D "\v\000\000\000\001", '\000' <repeats 12 times>, "\240\000\377=
\177\177\000\000\006", '\000' <repeats 102 times>, _info =3D {_signo =3D 11=
, _code =3D 1,
_errno =3D 0, _pad =3D 0, _reason =3D {_rt =3D {_pid =3D -16736256, _ui=
d =3D 32639,
_value =3D {sival_int =3D 6, sival_ptr =3D 0x6}}, _child =3D {
_pid =3D -16736256, _uid =3D 32639, _status =3D 6, _utime =3D 0, _s=
time =3D 0},
_fault =3D {_addr =3D 0x7f7fff00a000, _trap =3D 6, _trap2 =3D 0, _tra=
p3 =3D 0},
_poll =3D {_band =3D 140187715805184, _fd =3D 6}, _syscall =3D {
_sysnum =3D -16736256, _retval =3D {32639, 6}, _error =3D 0, _args =
=3D {0, 0,
0, 0, 0, 0, 0, 0}}, _ptrace_state =3D {_pe_report_event =3D -1673=
6256,
_option =3D {_pe_other_pid =3D 32639, _pe_lwp =3D 32639}}}}}
(gdb) print *(ucontext_t *)ctx
$2 =3D {uc_flags =3D 655373, uc_link =3D 0x0, uc_sigmask =3D {__bits =3D {0=
, 0, 0, 0}},
uc_stack =3D {ss_sp =3D 0x0, ss_size =3D 0, ss_flags =3D 0}, uc_mcontext =
=3D {
__gregs =3D {134142411075584, 118, 3353560262451, 22, 134142403527024, =
32,
1, 514, 134142411075584, 140187715785200, 134142403577003,
134142411075624, 349634, 2498, 134142410498040, 0, 0, 35, 35, 6, 4,
134142403527117, 71, 66054, 140187715785152, 63},
_mc_tlsbase =3D 134142411307248,
__fpregs =3D "\177\003", '\000' <repeats 22 times>, "\240\037\000\000\3=
77\377", '\000' <repeats 130 times>, "\\\217\302\365(\\oA", '\000' <repeats=
14 times>, "\340C", '\000' <repeats 327 times>}}
Instruction pointer (rip) of the signal ucontext:
(gdb) print (void *)((ucontext_t *)ctx)->uc_mcontext.__gregs[21]
$4 =3D (void *) 0x7a00764fc1cd <each_location+27>
(gdb) info line *(each_location+27)
Line 6512 of "gc.c" starts at address 0x7a00764fc1cd <each_location+27>
and ends at 0x7a00764fc1dc <each_location+42>.
Stack pointer (rsp) of the signal ucontext:
(gdb) print (void *)((ucontext_t *)ctx)->uc_mcontext.__gregs[24]
$5 =3D (void *) 0x7f7fff0051c0
If I have understood this correctly, it might correspond to this
frame;
(gdb) frame view 0x7f7fff0051c0 0x7a00764fc1cd
#0 each_location (objspace=3D0x7a006e052000, x=3D0x44a0427a1af34, n=3D-123=
985769,
cb=3D0x3) at gc.c:6512
6512 cb(objspace, v);
(gdb) info frame
Stack level 0, frame at 0x7f7fff0051c0:
rip =3D 0x7a00764fc1cd in each_location (gc.c:6512); saved rip =3D <not sa=
ved>
Outermost frame: previous frame identical to this frame (corrupt stack?)
source language c.
Arglist at 0x7a007623dae8, args: objspace=3D0x7a006e052000, x=3D0x44a0427a=
1af34,
n=3D-123985769, cb=3D0x3
Locals at 0x7a007623dae8, Previous frame's sp is 0x7a007623db20
Saved registers:
rbx at 0x7a007623daf0, rbp at 0x7a007623daf8, r12 at 0x7a007623db00,
r13 at 0x7a007623db08, r14 at 0x7a007623db10, rip at 0x7a007623db18
However, the frame pointer rbp looks wrong:
(gdb) print (void *)((ucontext_t *)ctx)->uc_mcontext.__gregs[12]
$11 =3D (void *) 0x555c2
And the candidate value for objspace shown above isn't mapped:
(gdb) print *objspace
Cannot access memory at address 0x7a006e052000
Plus the value of n looks wrong, so maybe I misunderstood the mcontext
and led gdb astray with `frame view'.
Disassembly of each_location:
(gdb) disas each_location+27
Dump of assembler code for function each_location:
0x00007a00764fc1b2 <+0>: test %rdx,%rdx
0x00007a00764fc1b5 <+3>: je 0x7a00764fc1ea <each_location+56>
0x00007a00764fc1b7 <+5>: push %r14
0x00007a00764fc1b9 <+7>: push %r13
0x00007a00764fc1bb <+9>: push %r12
0x00007a00764fc1bd <+11>: push %rbp
0x00007a00764fc1be <+12>: push %rbx
0x00007a00764fc1bf <+13>: mov %rcx,%r14
0x00007a00764fc1c2 <+16>: mov %rdx,%rbp
0x00007a00764fc1c5 <+19>: mov %rsi,%r13
0x00007a00764fc1c8 <+22>: mov %rdi,%r12
0x00007a00764fc1cb <+25>: xor %ebx,%ebx
=3D> 0x00007a00764fc1cd <+27>: mov 0x0(%r13,%rbx,8),%rsi
0x00007a00764fc1d2 <+32>: mov %r12,%rdi
0x00007a00764fc1d5 <+35>: callq *%r14
0x00007a00764fc1d8 <+38>: add $0x1,%rbx
0x00007a00764fc1dc <+42>: cmp %rbx,%rbp
0x00007a00764fc1df <+45>: jne 0x7a00764fc1cd <each_location+27>
0x00007a00764fc1e1 <+47>: pop %rbx
0x00007a00764fc1e2 <+48>: pop %rbp
0x00007a00764fc1e3 <+49>: pop %r12
0x00007a00764fc1e5 <+51>: pop %r13
0x00007a00764fc1e7 <+53>: pop %r14
0x00007a00764fc1e9 <+55>: retq =20
0x00007a00764fc1ea <+56>: retq =20
End of assembler dump.
r13 (__gregs[9]) and rbx (__gregs[13]):
(gdb) fr 8
#8 0x00007a00765f0084 in sigsegv (sig=3D11, info=3D0x7a007623e9a0,=20
ctx=3D0x7a007623ea20) at signal.c:964
964 rb_bug_for_fatal_signal(default_sigsegv_handler, sig, SIGINFO_C=
TX, "Segmentation fault" MESSAGE_FAULT_ADDRESS);
(gdb) print (void *)((ucontext_t *)ctx)->uc_mcontext.__gregs[9]
$9 =3D (void *) 0x7f7fff0051f0
(gdb) print (void *)((ucontext_t *)ctx)->uc_mcontext.__gregs[13]
$10 =3D (void *) 0x9c2
The base pointer (r13=3D0x7f7fff0051f0) is OK but adding the index
(rbx*8=3D0x9c2*8) is not -- this obviously ran just past the end of a
mapped region:
(gdb) x/xg 0x7f7fff0051f0
0x7f7fff0051f0: 0x00007a0076c2f000
(gdb) x/xg 0x7f7fff0051f0 + 0x9c2*8
0x7f7fff00a000: Cannot access memory at address 0x7f7fff00a000
(gdb) x/xg 0x7f7fff0051f0 + 0x9c2*8 - 8
0x7f7fff009ff8: 0x0000000000000076
From: Takahiro Kambe <taca@NetBSD.org>
To: riastradh@NetBSD.org
Cc: gnats-bugs@NetBSD.org
Subject: Re: pkg/59017: ruby32 crashes in asciidoctor
Date: Tue, 11 Feb 2025 15:30:34 +0900 (JST)
In message <20250121165111.957D084E7D@mail.netbsd.org>
on Tue, 21 Jan 2025 16:51:07 +0000,
Taylor R Campbell <riastradh@NetBSD.org> wrote:
> I pushed the package to wip/newsboat237; maybe you can use that to
> reproduce it. If that's not convenient I can try pulling out the
> asciidoc input to see if it reproduces outside the build environment.
I tried to test with wip/newsboat237 but I could not fetch
curl-sys-0.4.75+curl-8.10.0.crate.
--
Takahiro Kambe <taca@NetBSD.org>
From: Takahiro Kambe <taca@NetBSD.org>
To: riastradh@NetBSD.org
Cc: gnats-bugs@NetBSD.org
Subject: Re: pkg/59017: ruby32 crashes in asciidoctor
Date: Tue, 11 Feb 2025 16:08:50 +0900 (JST)
In message <20250211.153034.652185085790951569.taca@back-street.net>
on Tue, 11 Feb 2025 15:30:34 +0900 (JST),
Takahiro Kambe <taca@NetBSD.org> wrote:
> I tried to test with wip/newsboat237 but I could not fetch
> curl-sys-0.4.75+curl-8.10.0.crate.
I found out that devel/git-remote-hg fails, too.
--
Takahiro Kambe <taca@NetBSD.org>
From: Takahiro Kambe <taca@NetBSD.org>
To: riastradh@NetBSD.org
Cc: gnats-bugs@NetBSD.org
Subject: Re: pkg/59017: ruby32 crashes in asciidoctor
Date: Tue, 11 Feb 2025 16:17:44 +0900 (JST)
In message <20250211.160850.1952963749944404041.taca@back-street.net>
on Tue, 11 Feb 2025 16:08:50 +0900 (JST),
Takahiro Kambe <taca@NetBSD.org> wrote:
> In message <20250211.153034.652185085790951569.taca@back-street.net>
> on Tue, 11 Feb 2025 15:30:34 +0900 (JST),
> Takahiro Kambe <taca@NetBSD.org> wrote:
>> I tried to test with wip/newsboat237 but I could not fetch
>> curl-sys-0.4.75+curl-8.10.0.crate.
> I found out that devel/git-remote-hg fails, too.
And it was first time only to crash...
--
Takahiro Kambe <taca@NetBSD.org>
From: Thomas Klausner <wiz@gatalith.at>
To: NetBSD bugtracking <gnats-bugs@NetBSD.org>
Cc:
Subject: Re: pkg/59017: ruby32 crashes in asciidoctor
Date: Sat, 22 Mar 2025 09:17:37 +0100
I think this has enough information now to report it upstream - taca@
would only be an intermediary.
Can you please report this upstream directly?
Thomas
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.47 2022/09/11 19:34:41 kim Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2025
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home