NetBSD Problem Report #48782
From www@NetBSD.org Sun May 4 06:18:12 2014
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
(using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits))
(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
by mollari.NetBSD.org (Postfix) with ESMTPS id 1EA2EA5813
for <gnats-bugs@gnats.NetBSD.org>; Sun, 4 May 2014 06:18:12 +0000 (UTC)
Message-Id: <20140504061810.B83F4A583D@mollari.NetBSD.org>
Date: Sun, 4 May 2014 06:18:10 +0000 (UTC)
From: nullnilaki@gmail.com
Reply-To: nullnilaki@gmail.com
To: gnats-bugs@NetBSD.org
Subject: # top [1] Floating point exception (core dumped) top
X-Send-Pr-Version: www-1.0
>Number: 48782
>Category: port-alpha
>Synopsis: # top [1] Floating point exception (core dumped) top
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: martin
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sun May 04 06:20:00 +0000 2014
>Closed-Date: Sun Jul 20 07:46:12 +0000 2014
>Last-Modified: Sun Jul 20 07:46:12 +0000 2014
>Originator: nullnilaki
>Release: NetBSD 6.99.40
>Organization:
Japan
>Environment:
NetBSD 6.99.40 NetBSD 6.99.40 (GENERIC-$Revision: 1.358 $) #1: Sun May 4 05:40:04 UTC 2014 naruaki@:/usr/obj.alpha/sys/arch/alpha/compile/GENERIC alpha
>Description:
# top [1] Floating point exception (core dumped) top
>How-To-Repeat:
top
>Fix:
Please revert this change.
http://cvsweb.netbsd.org/cgi-bin/cvsweb.cgi/src/lib/libm/Makefile.diff?r1=1.156&r2=1.157&f=h
http://cvsweb.netbsd.org/cgi-bin/cvsweb.cgi/src/tests/lib/libm/Makefile.diff?r1=1.21&r2=1.22&f=h
---
load averages: 0.00, 0.00, 0.00; up 0+00:00:54 00:48:09
8 processes: 7 sleeping, 1 on CPU
CPU states: 0.2% user, 0.0% nice, 0.0% system, 0.2% interrupt, 99.6% idle
Memory: 18M Act, 2688K Inact, 5656K Exec, 11M File, 460M Free
Swap: 1025M Total, 1025M Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
420 root 43 0 3960K 1912K CPU 0:00 0.00% 0.00% top
0 root 96 0 0K 10M atath 0:00 0.00% 0.00% [system]
396 root 85 0 9840K 4056K wait 0:00 0.00% 0.00% login
158 root 85 0 7536K 2592K kqueue 0:00 0.00% 0.00% syslogd
410 root 85 0 3888K 2272K wait 0:00 0.00% 0.00% sh
1 root 85 0 3624K 1704K wait 0:00 0.00% 0.00% init
377 root 85 0 3776K 1576K kqueue 0:00 0.00% 0.00% inetd
341 root 85 0 3568K 1536K nanoslp 0:00 0.00% 0.00% cron
>Release-Note:
>Audit-Trail:
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: port-alpha/48782: # top [1] Floating point exception (core dumped) top
Date: Sun, 4 May 2014 09:42:47 +0200
This would only paper over the issue, as the code does a division by zero in
hash_lookup_uint():
412 result = NULL;
413 if ((bucket = &(ht->buckets[(key % ht->num_buckets)])) != NULL)
414 {
415 ll = &(bucket->list);
and ht->num_buckets is 0 (and an integer type). This is undefined
behaviour - but I am actually not sure why libm changes interfere
here, though I remember similar effects on sun3 due to bugs in fp emulation
of TME.
Martin
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: port-alpha/48782: # top [1] Floating point exception (core dumped) top
Date: Sun, 4 May 2014 09:51:55 +0200
Ok, libm comes into play as the code calculates the next prime for a given
number to get the number of hash buckets.
And, unbelievable but true: it does that using double values.
Martin
From: "Naruaki.Etomi" <nullnilaki@gmail.com>
To: gnats-bugs@netbsd.org
Cc: martin@duskware.de
Subject: Re: port-alpha/48782: # top [1] Floating point exception (core
dumped) top
Date: Mon, 5 May 2014 10:47:03 +0900
> Ok, libm comes into play as the code calculates the next prime for a given
> number to get the number of hash buckets.
> And, unbelievable but true: it does that using double values.
I'm sorry. Could you explain it again?
I think that this case is -mieee-with-inexact option bug.
Please see.
http://gcc.gnu.org/ml/gcc-bugs/2001-04/msg00408.html
---
# more test.c
#include <stdio.h>
int main (int argc, char *argv[])
{
double a, b, c;
a = 1.;
b = 3.;
c = a/b;
fprintf (stderr, "%g / %g = %g\n", a, b, c);
}
# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
Target: alpha--netbsd
Configured with:
/usr/src6/tools/gcc/../../external/gpl3/gcc/dist/configure
--target=alpha--netbsd --enable-long-long --enable-threads
--with-bugurl=http://www.NetBSD.org/Misc/send-pr.html
--with-pkgversion='NetBSD nb1 20120916' --with-system-zlib
--enable-__cxa_atexit
--with-mpc-lib=/var/obj/mknative/alpha/usr/src6/external/lgpl3/mpc/lib/libmpc
--with-mpfr-lib=/var/obj/mknative/alpha/usr/src6/external/lgpl3/mpfr/lib/libmpfr
--with-gmp-lib=/var/obj/mknative/alpha/usr/src6/external/lgpl3/gmp/lib/libgmp
--with-mpc-include=/usr/src6/external/lgpl3/mpc/dist/src
--with-mpfr-include=/usr/src6/external/lgpl3/mpfr/dist/src
--with-gmp-include=/usr/src6/external/lgpl3/gmp/lib/libgmp/arch/alpha
--enable-tls --disable-multilib --disable-symvers
--disable-libstdcxx-pch --build=x86_64-unknown-netbsd6.0.
--host=alpha--netbsd
--with-sysroot=/var/obj/mknative/alpha/usr/src6/destdir.alpha
Thread model: posix
gcc version 4.8.3 (NetBSD nb2 20140304)
# gcc -mieee-with-inexact test.c
# ./a.out 1 / 3 = nan
# cc -V Compaq C V6.5-011 on Compaq Tru64 UNIX V5.1B (Rev. 2650)
Compiler Driver V6.5-003 (sys) cc Driver
# cc -ieee_with_inexact test.c
# ./a.out 1 / 3 = 0.3333
---
Please revert this change immediately.
http://cvsweb.netbsd.org/cgi-bin/cvsweb.cgi/src/lib/libm/Makefile.diff?r1=1.156&r2=1.157&f=h
http://cvsweb.netbsd.org/cgi-bin/cvsweb.cgi/src/tests/lib/libm/Makefile.diff?r1=1.21&r2=1.22&f=h
Alpha architecture is miner target about gcc.
Nothing needs to be done.
Can you fix gcc's bug?
---
That's very nice of you!
Naruaki Etomi / nullnilaki@gmail.com
From: Martin Husemann <martin@duskware.de>
To: "Naruaki.Etomi" <nullnilaki@gmail.com>
Cc: gnats-bugs@netbsd.org
Subject: Re: port-alpha/48782: # top [1] Floating point exception (core dumped) top
Date: Mon, 5 May 2014 10:40:44 +0200
On Mon, May 05, 2014 at 10:47:03AM +0900, Naruaki.Etomi wrote:
> I'm sorry. Could you explain it again?
I just tried to understand why that change caused this regression.
> I think that this case is -mieee-with-inexact option bug.
It is way worse than only this flag, it seems to me that floating point
handling is completely broken on alpha/current. This may be either
kernel or gcc 4.8 fault, but it is hard to tell (if you single step
through the code, it all works).
Martin
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/48782 CVS commit: src
Date: Mon, 5 May 2014 18:08:33 +0000
Module Name: src
Committed By: martin
Date: Mon May 5 18:08:33 UTC 2014
Modified Files:
src/lib/libm: Makefile
src/tests/lib/libm: Makefile
Log Message:
Comment out the ieee fp flags for alpha for now - something (tm) is not
working right, see PR port-alpha/48782.
To generate a diff of this commit:
cvs rdiff -u -r1.157 -r1.158 src/lib/libm/Makefile
cvs rdiff -u -r1.22 -r1.23 src/tests/lib/libm/Makefile
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Naruaki.Etomi" <nullnilaki@gmail.com>
To: gnats-bugs@netbsd.org
Cc: martin@duskware.de
Subject: Re: port-alpha/48782: # top [1] Floating point exception (core
dumped) top
Date: Tue, 6 May 2014 13:29:47 +0900
> I think that this case is -mieee-with-inexact option bug.
Sorry, I guess I was mistaken.
I was installed gcc-4.2.3 for Tru64 UNIX.
Please see.(thanks Tobias!)
http://mail-index.netbsd.org/pkgsrc-users/2008/02/19/msg000417.html
---------------
ds15> more test.c
#include <stdio.h>
int main (int argc, char *argv[])
{
double a, b, c;
a = 1.;
b = 3.;
c = a/b;
fprintf (stderr, "%g / %g = %g\n", a, b, c);
}
ds15> /usr/local/gcc4/bin/gcc -v
Using built-in specs.
Target: alpha-dec-osf5.1b
Configured with: ../gcc-4.2.3/configure --prefix=/usr/local/gcc4
--enable-languages=c,c++ --enable-threads=posix --disable-nls
--host=alpha-dec-osf5.1b --without-gnu-ld --with-ld=/usr/ccs/bin/ld
--without-gnu-as --with-as=/usr/bin/as --disable-libssp
Thread model: posix
gcc version 4.2.3
ds15> uname -a
OSF1 ds15 V5.1 2650 alpha
ds15> /usr/local/gcc4/bin/gcc -mieee-with-inexact test.c
ds15> ./a.out
1 / 3 = 0.333333
ds15> more test.s
.file 1 "test.c"
.verstamp 3 11
.set noreorder
.set volatile
.set noat
.arch ev4
.rdata
$LC2:
.ascii "%g / %g = %g\12\0"
.align 3
$LC0:
.long 0
.long 1072693248
.align 3
$LC1:
.long 0
.long 1074266112
.text
.align 2
.globl main
.ent main
main:
.eflag 48
.frame $15,64,$26,0
.mask 0x4008000,-64
ldgp $29,0($27)
$main..ng:
lda $30,-64($30)
stq $26,0($30)
stq $15,8($30)
bis $31,$30,$15
.prologue 1
bis $31,$16,$1
stq $17,56($15)
stl $1,48($15)
lda $1,$LC0
ldt $f10,0($1)
stt $f10,32($15)
lda $1,$LC1
ldt $f10,0($1)
stt $f10,24($15)
ldt $f11,32($15)
ldt $f10,24($15)
divtsui $f11,$f10,$f12
trapb
cpys $f12,$f12,$f10
stt $f10,16($15)
lda $1,_iob
lda $1,112($1)
bis $31,$1,$16
lda $17,$LC2
ldt $f18,32($15)
ldt $f19,24($15)
ldt $f20,16($15)
jsr $26,fprintf
ldgp $29,0($26)
bis $31,$15,$30
ldq $26,0($30)
ldq $15,8($30)
lda $30,64($30)
ret $31,($26),1
.end main
---------------
It seems to me that gcc is correct.
Maybe alpha/fp_complete.c is worng.
---
That's very nice of you!
Naruaki Etomi / nullnilaki@gmail.com
From: Martin Husemann <martin@duskware.de>
To: "Naruaki.Etomi" <nullnilaki@gmail.com>
Cc: gnats-bugs@netbsd.org
Subject: Re: port-alpha/48782: # top [1] Floating point exception (core dumped) top
Date: Tue, 6 May 2014 09:18:45 +0200
On Tue, May 06, 2014 at 01:29:47PM +0900, Naruaki.Etomi wrote:
> It seems to me that gcc is correct.
> Maybe alpha/fp_complete.c is worng.
That is my guess as well.
Martin
From: "Naruaki.Etomi" <nullnilaki@gmail.com>
To: gnats-bugs@netbsd.org
Cc: port-alpha-maintainer@netbsd.org, martin@duskware.de
Subject: Re: port-alpha/48782: # top [1] Floating point exception (core
dumped) top
Date: Sat, 10 May 2014 17:55:11 +0900
I think that this case is conflict between pcu_load() and fpu_state_load().
---------------------------
-mieee-with-inexact option use:
/usr/src/sys/arch/alpha/alpha/fp_complete.c
static int
alpha_fp_complete_at(alpha_instruction *trigger_pc, struct lwp *l,
uint64_t *ucode)
{
int needsig;
...
if (__predict_false(copyin(trigger_pc, &inst, sizeof inst))) {
this_cannot_happen(6, -1);
return SIGSEGV;
}
fpu_load(); <---------------------------------------------------- point
alpha_pal_wrfen(1);
/usr/src/sys/kern/subr_pcu.c
pcu_load(const pcu_ops_t *pcu)
{
const u_int id = pcu->pcu_id;
...
/* Does this CPU already have our PCU state loaded? */
if (ci == curci) {
KASSERT(curci->ci_pcu_curlwp[id] == l);
KASSERT(pcu_used_p(pcu));
panic("Re-enable"); <----------------------------------- point
/* Re-enable */
pcu->pcu_state_load(l, PCU_LOADED | PCU_ENABLE);
<------- point
panic: Re-enable
Stopped in pid 411.1 (a.out) at netbsd:cpu_Debugger+0x4: ret zero,(ra
)
db> tr
cpu_Debugger() at netbsd:cpu_Debugger+0x4
db_panic() at netbsd:db_panic+0x118
vpanic() at netbsd:vpanic+0x1a8
panic() at netbsd:panic+0x58
pcu_load() at netbsd:pcu_load+0x288
alpha_fp_complete_at() at netbsd:alpha_fp_complete_at+0x60
alpha_fp_complete() at netbsd:alpha_fp_complete+0x78
trap() at netbsd:trap+0x114
XentArith() at netbsd:XentArith+0x20
--- arithmetic trap (from ipl 0) ---
--- user mode ---
---------------------------
I think that trapb instruction incompatible with fpu_state_load().
---------------------------
# diff -u fp_complete.c.orig fp_complete.c
--- fp_complete.c.orig 2014-05-10 08:02:31.000000000 +0000
+++ fp_complete.c 2014-05-10 07:48:32.000000000 +0000
@@ -721,6 +721,10 @@
void
fpu_state_load(struct lwp *l, u_int flags)
{
+ if ((PCU_LOADED | PCU_ENABLE) == flags) {
+ return;
+ }
+
struct pcb * const pcb = lwp_getpcb(l);
---------------------------
Is it right?
Please review.
---------------------------
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
0 root 96 0 0K 13M atath 0:01 0.00% 0.00% [system]
466 root 43 0 3992K 2240K CPU 0:00 0.00% 0.00% top
404 root 85 0 9840K 4088K wait 0:00 0.00% 0.00% login
158 root 85 0 7536K 2592K kqueue 0:00 0.00% 0.00% syslogd
396 root 85 0 3888K 2280K wait 0:00 0.00% 0.00% sh
325 root 85 0 3776K 1736K kqueue 0:00 0.00% 0.00% inetd
1 root 85 0 3624K 1704K wait 0:00 0.00% 0.00% init
397 root 85 0 3568K 1536K nanoslp 0:00 0.00% 0.00% cron
412 root 43 0 32M 18M parked 0:00 0.00% 0.00% dhclient
# vi test.c
#include <stdio.h>
int main (int argc, char *argv[])
{
double a, b, c;
a = 1.;
b = 3.;
c = a/b;
fprintf (stderr, "%g / %g = %g\n", a, b, c);
}
# gcc -mieee-with-inexact test.c
# ./a.out
1 / 3 = 0.333333
---------------------------
---
That's very nice of you!
Naruaki Etomi / nullnilaki@gmail.com
From: Martin Husemann <martin@duskware.de>
To: "Naruaki.Etomi" <nullnilaki@gmail.com>
Cc: gnats-bugs@netbsd.org
Subject: Re: port-alpha/48782: # top [1] Floating point exception (core dumped) top
Date: Thu, 15 May 2014 14:37:39 +0200
Your patch did not make a difference for me (and I guess it shouldn't),
but backing out all of the PCU FPU handling changes made the test failures
drop from 122 to 15.
See runs 4 and 5: http://www.netbsd.org/~martin/alpha-atf/
(same userland, just kernel replaced)
top also works fine (even with libm compiled for ieee mode).
So something in the PCU changes is broken - will have a closer look.
Martin
From: "Naruaki.Etomi" <nullnilaki@gmail.com>
To: gnats-bugs@netbsd.org
Cc: port-alpha-maintainer@netbsd.org, martin@duskware.de
Subject: Re: port-alpha/48782: # top [1] Floating point exception (core
dumped) top
Date: Thu, 15 May 2014 23:52:21 +0900
> Your patch did not make a difference for me (and I guess it shouldn't),
I'm sorry, I couldn't help you.
> So something in the PCU changes is broken - will have a closer look.
I didn't understand that "lazy FPU context switch" very well.
I'm rooting for you, and I will study more.
From: Martin Husemann <martin@duskware.de>
To: "Naruaki.Etomi" <nullnilaki@gmail.com>
Cc: gnats-bugs@netbsd.org
Subject: Re: port-alpha/48782: # top [1] Floating point exception (core dumped) top
Date: Thu, 15 May 2014 16:57:31 +0200
On Thu, May 15, 2014 at 11:52:21PM +0900, Naruaki.Etomi wrote:
> > Your patch did not make a difference for me (and I guess it shouldn't),
> I'm sorry, I couldn't help you.
Well, you pointed at the right spot, so that *did* help.
Martin
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/48782 CVS commit: src/sys/arch/alpha/alpha
Date: Sun, 18 May 2014 10:43:54 +0000
Module Name: src
Committed By: martin
Date: Sun May 18 10:43:54 UTC 2014
Modified Files:
src/sys/arch/alpha/alpha: fp_complete.c
Log Message:
Properly handle PCU_REENABLE in fpu_state_load. Hint from Nick Hudson.
This seems to help PR port-alpha/48782 a lot.
To generate a diff of this commit:
cvs rdiff -u -r1.17 -r1.18 src/sys/arch/alpha/alpha/fp_complete.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
Responsible-Changed-From-To: port-alpha-maintainer->martin
Responsible-Changed-By: martin@NetBSD.org
Responsible-Changed-When: Sun, 18 May 2014 10:49:41 +0000
Responsible-Changed-Why:
I think I fixed it
State-Changed-From-To: open->feedback
State-Changed-By: martin@NetBSD.org
State-Changed-When: Sun, 18 May 2014 10:49:41 +0000
State-Changed-Why:
Could you re-try with -current?
From: "Naruaki.Etomi" <nullnilaki@gmail.com>
To: gnats-bugs@netbsd.org
Cc: martin@netbsd.org, port-alpha-maintainer@netbsd.org
Subject: Re: port-alpha/48782 (# top [1] Floating point exception (core
dumped) top)
Date: Wed, 21 May 2014 02:14:22 +0900
Dear martin,
I carry out a test on current version.
(__KERNEL_RCSID(0, "$NetBSD: fp_complete.c,v 1.21 2014/05/19 07:09:10
matt Exp $");)
$ uname -a
NetBSD 6.99.42 NetBSD 6.99.42 (GENERIC-$Revision: 1.358 $) #0: Tue
May 20 12:29:46 UTC 2014
naruaki@:/usr/obj.alpha/sys/arch/alpha/compile/GENERIC alpha
NaN
--------------------
#include<stdio.h>
#include <math.h>
int main(void){
double a, b;
a = 0.0 / 0.0;
printf("a=%lf isnan=%d\n", a, isnan(a));
b = 1.0 + a;
printf("b=%lf\n", b);
return 0;
}
NetBSD
----------
$ gcc -mieee-with-inexact -lm test1.c
$ ./a.out
a=nan isnan=1
b=nan
Tru64 UNIX
----------
ds15> cc -ieee_with_inexact -lm test1.c
cc: Warning: test1.c, line 6: In this statement, floating-point
overflow occurs in evaluating the expression "0.0/0.0". (floatoverfl)
a = 0.0 / 0.0;
----^
ds15> ./a.out
a=NaNQ isnan=1
b=NaNQ
--------------------
Infinity
--------------------
#include <stdio.h>
#include <math.h>
int main(void){
double a, b;
a = 1.0 / 0.0;
printf("a=%lf isinf=%d\n", a, isinf(a));
b = 1.0 / a;
printf("b=%lf\n", b);
return 0;
}
NetBSD
----------
$ gcc -mieee-with-inexact -lm test2.c
$ ./a.out
a=inf isinf=1
b=0.000000
Tru64 UNIX
----------
Tru64 UNIX is not implemented isinf()
--------------------
Signed zero
--------------------
#include <stdio.h>
#include <math.h>
int main(void){
double a, b, c, d;
a = 1.0 / 0.0;
b = -1.0 / 0.0;
printf("a=%lf b=%lf\n", a, b);
c = 1.0 / a;
d = 1.0 / b;
printf("c = %lf d = %lf\n", c, d);
printf("ans1=%lf, and2=%lf\n", 1.0 / c, 1.0 / d);
return 0;
}
NetBSD
----------
$ gcc -mieee-with-inexact -lm test3.c
$ ./a.out
a=inf b=-inf
c = 0.000000 d = -0.000000
ans1=inf, and2=-inf
Tru64 UNIX
----------
ds15> cc -ieee_with_inexact -lm test3.c
cc: Warning: test3.c, line 6: In this statement, division by zero
occurs in evaluating the expression "1.0/0.0". (zerodiv)
a = 1.0 / 0.0;
----^
cc: Warning: test3.c, line 7: In this statement, division by zero
occurs in evaluating the expression "-1.0/0.0". (zerodiv)
b = -1.0 / 0.0;
----^
ds15> ./a.out
a=INF b=-INF
c = 0.000000 d = -0.000000
ans1=INF, and2=-INF
--------------------
I think that the problem was solved.
Your work is wonderful!
Of course, Userland application(e.g., /pkgsrc/net/mikutter) work fine!
https://twitter.com/nullnilaki/status/468744417319206912/photo/1
That's very nice of you!
--
Naruaki Etomi / nullnilaki@gmail.com
From: Martin Husemann <martin@duskware.de>
To: "Naruaki.Etomi" <nullnilaki@gmail.com>
Cc: gnats-bugs@netbsd.org, martin@netbsd.org,
port-alpha-maintainer@netbsd.org
Subject: Re: port-alpha/48782 (# top [1] Floating point exception (core dumped) top)
Date: Tue, 20 May 2014 20:02:11 +0200
On Wed, May 21, 2014 at 02:14:22AM +0900, Naruaki.Etomi wrote:
> I think that the problem was solved.
> Your work is wonderful!
It was mostly rmind and matt after all, thanks to both of them!
Martin
From: "Naruaki.Etomi" <nullnilaki@gmail.com>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: port-alpha/48782 (# top [1] Floating point exception (core
dumped) top)
Date: Sun, 20 Jul 2014 15:17:14 +0900
This problem is fixed, And please close PR.
Sorry for my late reply...
--
naruaki.etomi
nullnilaki@gmail.com
State-Changed-From-To: feedback->closed
State-Changed-By: wiz@NetBSD.org
State-Changed-When: Sun, 20 Jul 2014 07:46:12 +0000
State-Changed-Why:
Confirmed fixed, thanks for the feedback.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home