NetBSD Problem Report #42656
From www@NetBSD.org Thu Jan 21 15:48:59 2010
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by www.NetBSD.org (Postfix) with ESMTP id 3B5DD63C54F
for <gnats-bugs@gnats.NetBSD.org>; Thu, 21 Jan 2010 15:48:59 +0000 (UTC)
Message-Id: <20100121154859.06D1D63C2E4@www.NetBSD.org>
Date: Thu, 21 Jan 2010 15:48:59 +0000 (UTC)
From: hubertf@NetBSD.org
Reply-To: hubertf@NetBSD.org
To: gnats-bugs@NetBSD.org
Subject: netbsd-5: panic LOCKDEBUG in opencrypto(9) w/ hifn(4)
X-Send-Pr-Version: www-1.0
>Number: 42656
>Category: kern
>Synopsis: netbsd-5: panic LOCKDEBUG in opencrypto(9)
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: kern-bug-people
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Thu Jan 21 15:50:00 +0000 2010
>Closed-Date: Sat Dec 05 18:52:03 +0000 2020
>Last-Modified: Sat Dec 05 18:52:03 +0000 2020
>Originator: Hubert Feyrer
>Release: netbsd-5 branch sources from 2010-01-19
>Organization:
n/a
>Environment:
NetBSD/i386 with hifn(4) PCI card
>Description:
Trying to establish performance of hifn(4), I get:
# uname -a
NetBSD 5.0_STABLE NetBSD 5.0_STABLE (GENERIC) #22: Tue Jan 19 14:10:05 CET 2010
feyrer@host:/home/feyrer/repository/Image_Creation/trunk/obj.build/sys/arch/i3
86/compile/GENERIC i386
#
# dmesg | grep -i hifn
hifn0 at pci0 dev 20 function 0: Hifn 7955, rev. 0
hifn0: 3DES/AES, 32KB dram, interrupting at irq 5
#
# openssl speed -evp aes-128-cbc -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 1 aes-128-cbcīs in 27.89s
Doing aes-128-cbc for 3s on 64 size blocks:
1 aes-128-cbcīs Mutex error: lockdebug_barrier: spin lock held
lock address : 0x00000000c04a2774 type : spin
initialized : 0x00000000c0113204
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 0
current lwp : 0x00000000cb9dbac0 last held: 0x00000000cb9dbac0
last locked : 0x00000000c0117984 unlocked : 0x00000000c0118b69
owner field : 0x0000000000010600 wait/spin: 0/1
panic: LOCKDEBUG
fatal breakpoint trap in supervisor mode
trap type 1 code 0 eip c0349efc cs 8 eflags 246 cr2 cb079000 ilevel 6
Stopped in pid 349.1 (openssl) at netbsd:breakpoint+0x4: popl %ebp
db{0}> bt
breakpoint(c0431c16,cb8cf7f8,c045db80,c02a5a0f,6,1,0,0,cb8cf7f8,c0100e1a) at net
bsd:breakpoint+0x4
panic(c0431c18,c042e02a,c0400028,c0431c31,ebc0,1000001,6,0,0,0) at netbsd:panic+
0x1b0
lockdebug_abort1(c0431c31,1,0,0,0,cb9dbac0,0,0,cb9dbac0,cb8cf934) at netbsd:lock
debug_abort1+0xbb
mutex_vector_enter(c04a8e8c,cb9c9d40,cb8cf8ec,c029e49f,cba0ebcc,cb9c9d40,cb8cf8e
c,c029e87a,8,20) at netbsd:mutex_vector_enter+0x3b7
rnd_extract_data(cb8cf8f0,20,1,4,4,4,293,c027351e,0,0) at netbsd:rnd_extract_dat
a+0x13
arc4_randrekey(cacfa17c,10,c04a2776,c1191401,cacfa17c,0,0,4,0,0) at netbsd:arc4_
randrekey+0x3d
hifn_newsession(cacfa000,cb8cfa5c,cb8cfb0c,c02c19c2,0,0,4,c03d8d06,cb9e6940,0) a
t netbsd:hifn_newsession+0xeb
crypto_newsession(cb8cfb54,cb8cfb0c,1,c04139cc,62d,cb9c343c,0,c02e5577,0,cb8cfb0
c) at netbsd:crypto_newsession+0xac
cryptodev_session(c11a8ec0,cb8cfc50,297,c027351e,7,0,cb8cfbac,c03d8d06,c06f9cc0,
c11a8ec0) at netbsd:cryptodev_session+0x432
cryptof_ioctl(cb9c70c0,c0206370,cb8cfc50,0,c028ffd4,cb9dbac0,7,c06f4e14,3,0) at n
etbsd:cryptof_ioctl+0x78
sys_ioctl(cb9dbac0,cb8cfd00,cb8cfd28,bb9bd000,ca479350,ca479350,293,3,c0206370,b
b8120d0) at netbsd:sys_ioctl+0x13f
syscall(cb8cfd48,b3,ab,1f,1f,bb8120f0,b,bfbfb738,bbb99640,bb8120d0) at netbsd:sy
scall+0xc8
db{0}>
db{0}> sh reg
ds 0x10
es 0x10
fs 0x30
gs 0x10
edi 0xc0400028 __func__.7317
esi 0xc0431c18 copyright+0x1ecd8
ebp 0xcb8cf7ac
ebx 0x104
edx 0xc0000011
ecx 0x6
eax 0x1
eip 0xc0349efc breakpoint+0x4
cs 0x8
eflags 0x246
esp 0xcb8cf7ac
ss 0x10
netbsd:breakpoint+0x4: popl %ebp
db{0}>
db{0}> show lock crypto_mtx
lock address : 0x00000000c04a2774 type : spin
initialized : 0x00000000c0113204
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 0
current lwp : 0x00000000cb9d9d40 last held: 0x00000000cb9d9d40
last locked : 0x00000000c0117984 unlocked : 0x00000000c0118b69
owner field : 0x0000000000010600 wait/spin: 0/1
db{0}>
db{0}> show lock rndpool_mtx
lock address : 0x00000000c04a8e8c type : sleep/adaptive
initialized : 0x00000000c024b673
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 1
current cpu : 0 last held: 0
current lwp : 0x00000000cb9d9d40 last held: 000000000000000000
last locked : 0x00000000c024b0df unlocked : 0x00000000c0380f5c
owner field : 000000000000000000 wait/spin: 0/0
Turnstile chain at 0xc06f5060.
=> No active turnstile for this lock.
db{0}>
In GDB (gdb netbsd.gdb):
1) īlast lockedī from output.
(gdb) list *(0x00000000c0117984)
0xc0117984 is in cryptodev_session (/usr/nbcvs/src-5/sys/opencrypto/cryptodev.c:1615).
1610 }
1611 }
1612
1613 /* crypto_newsession requires that we hold the mutex. */
1614 mutex_spin_enter(&crypto_mtx);
1615 error = crypto_newsession(&sid, crihead, crypto_devallowsoft);
1616 if (!error) {
1617 DPRINTF(("cyrptodev_session: got session %d\n", (uint32_t)sid));
1618 cse = csecreate(fcr, sid, crie.cri_key, crie.cri_klen,
1619 cria.cri_key, cria.cri_klen, (txform ? sop->cipher : 0), sop->mac,
(gdb)
2) line from panic message:
(gdb) list *(cryptodev_session+0x432)
0xc01179a2 is in cryptodev_session (/usr/nbcvs/src-5/sys/opencrypto/cryptodev.c:1616).
1611 }
1612
1613 /* crypto_newsession requires that we hold the mutex. */
1614 mutex_spin_enter(&crypto_mtx);
1615 error = crypto_newsession(&sid, crihead, crypto_devallowsoft);
1616 if (!error) {
1617 DPRINTF(("cyrptodev_session: got session %d\n", (uint32_t)sid));
1618 cse = csecreate(fcr, sid, crie.cri_key, crie.cri_klen,
1619 cria.cri_key, cria.cri_klen, (txform ? sop->cipher : 0), sop->mac,
1620 (tcomp ? sop->comp_alg : 0), txform, thash, tcomp);
>How-To-Repeat:
- Plug in hifn(4) card into NetBSD/i386 machine with netbsd-5 kernel sources from 20100119 with options LOCKDEBUG in kernel
- Run above openssl command
- See panic after some time
>Fix:
Unknown, probably some locking problem.
Some suggestions made while getting the above data were
(thanks to martin, pooka & phone):
* we may try to sleep on a mutex_enter() on rndpool_mtx while holding the crypto_mtx spin lock
* The comment near the CRIOGET case may be related here:
/* XXX deprecated, remove after 5.0 */
* the same may be in cryptoopen()
* that type of locking misuse seems common
* one of those was removed recently from bpf
>Release-Note:
>Audit-Trail:
From: Hubert Feyrer <hubert@feyrer.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/42656: netbsd-5: panic LOCKDEBUG in opencrypto(9) w/
hifn(4)
Date: Fri, 22 Jan 2010 11:43:12 +0100 (CET)
On Thu, 21 Jan 2010, gnats-admin@netbsd.org wrote:
>> Synopsis: netbsd-5: panic LOCKDEBUG in opencrypto(9) w/ hifn(4)
A data point: when using the same hifn(4) card in another i386 machine, I
don't get the panic. Yet, performance of hifn(4) is only a bit above
software performance, in contrast to glxsb(4). For AES128:
sw aes-128-cbc 3603.05k 3963.26k 4072.27k 8183.50k 8276.37k
hifn aes-128-cbc 118.20k 374.71k 1021.79k 3611.90k 9482.04k
glxsb aes-128-cbc 278.07k 1098.74k 3379.97k 10894.20k 23729.58k
More numbers are available at
http://www.feyrer.de/Misc/crypto-comparison2.txt
- Hubert
From: Hubert Feyrer <hubert@feyrer.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/42656: netbsd-5: panic LOCKDEBUG in opencrypto(9) w/
hifn(4)
Date: Fri, 22 Jan 2010 12:06:02 +0100 (CET)
On Fri, 22 Jan 2010, Hubert Feyrer wrote:
> A data point: when using the same hifn(4) card in another i386 machine, I
> don't get the panic.
The machine had LOCKDEBUG, but not DIAGNOSTIC.
Adding DIAGNOSTICS to the kernel, the panics happen, too
(with a somewhat funky traceback, though):
Mutex error: lockdebug_barrier: spin lock held
lock address : 0x00000000c0477af4 type : spin
initialized : 0x00000000c0113304
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 0
current lwp : 0x00000000cb5c6780 last held: 0x00000000cb5c6780
last locked : 0x00000000c0117945 unlocked : 0x00000000c011874f
owner field : 0x0000000000010600 wait/spin: 0/1
panic: LOCKDEBUG
Begin traceback...
copyright(c06cda20,0,0,c0427090,ca016d80,ca016d18,c0477af4,c04240e4,cb5c6780,c0117945)
at netbsd:__qdivrem+0x4f1db
ld_prime(ca016d41,0,0,c0427090,ca016e00,ca016d98,c0477974,c04240e4,0,c029f654)
at 0xc06cd9a0
ld_prime(ca016d41,0,0,c0427090,ca016e00,ca016d98,c0477974,c04240e4,0,c029f654)
at 0xc06cd9a0
_prop_array_pool(ca016dc0,0,cb060bc0,c042503c,cb060bc0,cb060b58,c0477400,c04240f0,0,0)
at 0xca016ec0
Bad frame pointer: 0xc06ccae0
End traceback...
dump to dev 0,1 not possible
rebooting...
- Hubert
From: Hubert Feyrer <hubert@feyrer.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/42656: netbsd-5: panic LOCKDEBUG in opencrypto(9) w/
hifn(4)
Date: Fri, 22 Jan 2010 12:28:31 +0100 (CET)
On Fri, 22 Jan 2010, Hubert Feyrer wrote:
> Adding DIAGNOSTICS to the kernel, the panics happen, too
> (with a somewhat funky traceback, though):
Here's an updated panic, which shows the same place, but that it happens
in glxsb - so it's not specific to hifn, but a general bug in the
opencrypto system.
Full data:
Fr 22 Jan 2010 12:08:01 CET
NetBSD 5.0/i386 with sources from netbsd-5 branch as of 20090121
sh -c 'while true ; do openssl speed -evp aes-128-cbc -elapsed ; done'
# Mutex error: lockdebug_barrier: spin lock held
lock address : 0x00000000c0477af4 type : spin
initialized : 0x00000000c0113304
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 0
current lwp : 0x00000000cb579860 last held: 0x00000000cb579860
last locked : 0x00000000c0117945 unlocked : 0x00000000c0118aa9
owner field : 0x0000000000010600 wait/spin: 0/1
panic: LOCKDEBUG
fatal breakpoint trap in supervisor mode
trap type 1 code 0 eip c033879c cs 8 eflags 246 cr2 bb9bd600 ilevel 6
Stopped in pid 875.1 (openssl) at netbsd:breakpoint+0x4: popl
%ebp
db{0}>
db{0}> bt
breakpoint(c04063c0,cb5b5838,c0426f80,c02a1f4f,6,1,0,0,cb5b5838,c0100e11)
at net
bsd:breakpoint+0x4
panic(c04063c2,c0403e73,c03daf74,c04063db,4,1000001,6,0,20,cb5b58f0) at
netbsd:panic+0x1b0
lockdebug_abort1(c04063db,1,0,0,cb5b59c4,cb5b5dc0,cb579860,0,cb5b5cbc,cb5b58f4)
at netbsd:lockdebug_abort1+0xbb
mutex_vector_enter(c047fb2c,cb5b5d48,80879d8,4,cb56d768,cb579860,cb5b58dc,c03b5b56,cb763240,20)
at netbsd:mutex_vec
tor_enter+0x3cd
rnd_extract_data(cb5b58f0,20,1,c03b5b56,cb763240,cb4f8b00,cb5b590c,c03b5b56,cb763240,cb4f8b00)
at netbsd:rnd_extrac
t_data+0x13
arc4_randrekey(c0e40a10,10,cb5b5a3c,c029bb08,ca016d4c,c0e40a00,0,c0117945,cb579860,6)
at netbsd:arc4_randrekey+0x3d
glxsb_crypto_newsession(ca67ce74,cb5b5a6c,cb5b5b1c,0,caf60340,c0477af6,cb5b5a6c,c06ca040,4,0)
at netbsd:glxsb_crypt
o_newsession+0x131
crypto_newsession(cb5b5b64,cb5b5b1c,1,c03ee300,62d,c044b6d4,cb5b5adc,0,c029f797,cb579860)
at netbsd:crypto_newsessi
on+0xac
cryptodev_session(c0d66f00,cb5b5c60,cb5b5bac,c03b5b56,c06d09a0,c04719c0,cb5b5bac,c0427098,c04719c0,c0d66f00)
at net
bsd:cryptodev_session+0x3f3
cryptof_ioctl(cb4feec0,c0206370,cb5b5c60,0,c028f794,cb579860,7,c06cba14,3,0)
at netbsd:cryptof_ioctl+0x78
sys_ioctl(cb579860,cb5b5d00,cb5b5d28,3,c0206370,bb8120d0,b,c03759f6,0,0)
at netbsd:sys_ioctl+0x13f
syscall(cb5b5d48,b3,ab,1f,1f,bb8120f0,b,bfbfb6b8,bbb99420,bb8120d0) at
netbsd:syscall+0xbd
db{0}>
db{0}> show reg
ds 0x10
es 0x10
fs 0x30
gs 0x10
edi 0xc03daf74 __func__.7295
esi 0xc04063c2 copyright+0x18c22
ebp 0xcb5b57ec
ebx 0x104
edx 0x8
ecx 0x6
eax 0x1
eip 0xc033879c breakpoint+0x4
cs 0x8
eflags 0x246
esp 0xcb5b57ec
ss 0x10
netbsd:breakpoint+0x4: popl %ebp
db{0}>
db{0}> show lock crypto_mtx
lock address : 0x00000000c0477af4 type : spin
initialized : 0x00000000c0113304
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 0
current lwp : 0x00000000cb579860 last held: 0x00000000cb579860
last locked : 0x00000000c0117945 unlocked : 0x00000000c0118aa9
owner field : 0x0000000000010600 wait/spin: 0/1
db{0}>
db{0}> show lock rndpool_mtx
lock address : 0x00000000c047fb2c type : sleep/adaptive
initialized : 0x00000000c0258354
shared holds : 0 exclusive: 0
shares wanted: 0 exclusive: 1
current cpu : 0 last held: 0
current lwp : 0x00000000cb579860 last held: 000000000000000000
last locked : 0x00000000c0257dff unlocked : 0x00000000c03610bc
owner field : 000000000000000000 wait/spin: 0/0
Turnstile chain at 0xc06cbee0.
=> No active turnstile for this lock.
db{0}>
gdb netbsd.gdb:
(gdb) # address from "last locked"
(gdb) list *(0x00000000c0117945)
0xc0117945 is in cryptodev_session
(/usr/cvs/src-5/sys/opencrypto/cryptodev.c:1615).
1610 }
1611 }
1612
1613 /* crypto_newsession requires that we hold the mutex. */
1614 mutex_spin_enter(&crypto_mtx);
1615 error = crypto_newsession(&sid, crihead, crypto_devallowsoft);
1616 if (!error) {
1617 DPRINTF(("cyrptodev_session: got session %d\n", (uint32_t)sid));
1618 cse = csecreate(fcr, sid, crie.cri_key, crie.cri_klen,
1619 cria.cri_key, cria.cri_klen, (txform ? sop->cipher : 0), sop->mac,
(gdb)
(gdb) # line from panic message
(gdb) list *(cryptodev_session+0x3f3)
0xc0117963 is in cryptodev_session
(/usr/cvs/src-5/sys/opencrypto/cryptodev.c:1616).
1611 }
1612
1613 /* crypto_newsession requires that we hold the mutex. */
1614 mutex_spin_enter(&crypto_mtx);
1615 error = crypto_newsession(&sid, crihead, crypto_devallowsoft);
1616 if (!error) {
1617 DPRINTF(("cyrptodev_session: got session %d\n", (uint32_t)sid));
1618 cse = csecreate(fcr, sid, crie.cri_key, crie.cri_klen,
1619 cria.cri_key, cria.cri_klen, (txform ? sop->cipher : 0), sop->mac,
1620 (tcomp ? sop->comp_alg : 0), txform, thash, tcomp);
(gdb)
From: Hubert Feyrer <hubert@feyrer.de>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/42656: netbsd-5: panic LOCKDEBUG in opencrypto(9)
Date: Wed, 27 Jan 2010 21:42:12 +0100 (CET)
Looking more into the interaction of opencrypto and rnd,
points:
1/3: disabling rnd collection from hifn0 => still panics
2/3: removing pseudo-device rnd from the kernel => needed for hifn(4)
3/3: patching arc4_randrekey() to not use rnd_extract_data() =>
Details:
1/2:
The panic still happens if the hifn0 device is not used for collecting
random data.
# rndctl -l
Source Bits Type Flags
wd0 2367 disk estimate, collect
hifn0 522989 rng collect
# rndctl -C -d hifn0
# rndctl -l
Source Bits Type Flags
wd0 2367 disk estimate, collect
hifn0 526426 rng
# while true ; do openssl speed -evp aes-128-cbc -elapsed ; done
...
Mutex error: lockdebug_barrier: spin lock held
lock address : 0x00000000c04a9954 type : spin
initialized : 0x00000000c0113204
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 0
current cpu : 0 last held: 0
current lwp : 0x00000000cb9f85c0 last held: 0x00000000cb9f85c0
last locked : 0x00000000c0117984 unlocked : 0x00000000c0118b69
owner field : 0x0000000000010600 wait/spin: 0/1
panic: LOCKDEBUG
fatal breakpoint trap in supervisor mode
trap type 1 code 0 eip c034f34c cs 8 eflags 246 cr2 bb9bd770 ilevel 6
Stopped in pid 475.1 (openssl) at netbsd:breakpoint+0x4: popl
%ebp
db{0}>
db{0}> bt
breakpoint(c0438322,cb91e7f8,c0464cc0,c02aba4f,6,1,0,0,cb91e7f8,c0100e1a) at netbsd:breakpoint+0x4
panic(c0438324,c0434736,c04066e8,c043833d,5b40,1000001,6,0,0,0) at netbsd:panic+0x1b0
lockdebug_abort1(c043833d,1,0,0,0,cb9f85c0,0,0,cb9f85c0,cb91e934) at netbsd:lockdebug_abort1+0xbb
mutex_vector_enter(c04b006c,cb9e8d40,cb91e8ec,c02a44df,cba65b4c,cb9e8d40,cb91e8ec,c02a48ba,8,20) at netbsd:mutex_vector_enter+0x3b7
rnd_extract_data(cb91e8f0,20,1,4,4,4,293,c027955e,0,0) at netbsd:rnd_extract_data+0x13
arc4_randrekey(cad0117c,10,c04a9956,c1198101,cad0117c,0,0,4,0,0) at netbsd:arc4_randrekey+0x3d
hifn_newsession(cad01000,cb91ea5c,cb91eb0c,c027955e,0,0,4,c03df1b6,cb642880,0) at netbsd:hifn_newsession+0xeb
crypto_newsession(cb91eb54,cb91eb0c,1,c0419cbc,62d,cb6267d8,0,c02eb5b7,0,cb91eb0c) at netbsd:crypto_newsession+0xac
cryptodev_session(c11a8f00,cb91ec50,297,c027955e,7,0,cb91ebac,c03df1b6,c0700ec0,c11a8f00) at netbsd:cryptodev_session+0x432
cryptof_ioctl(cb9e6100,c0206370,cb91ec50,0,c0296014,cb9f85c0,7,c06fc014,3,0) at netbsd:cryptof_ioctl+0x78
sys_ioctl(cb9f85c0,cb91ed00,cb91ed28,cb91ed40,c0343532,ca47ae40,293,3,c0206370,bb8120d0) at netbsd:sys_ioctl+0x13f
syscall(cb91ed48,b3,ab,1f,1f,bb8120f0,b,bfbfb748,bbb99640,bb8120d0) at etbsd:syscall+0xc8
db{0}>
2/2:
Disabling "pseudo-device rnd" in the kernel is a no-go:
/usr/src-5/sys/dev/pci/hifn7751.c:56:2: error: #error hifn7751 requires rnd pseudo-devices
3/3:
patching arc4_randrekey() to not use rnd_extract_data()
gives makes the above "openssl speed -evp aes-128-cbc -elapsed" run for
several hours without panic.
Unfortunately, performance still bad:
aes-128-cbc 137.54k 541.53k 1677.18k 4404.90k 14008.05k
aes-128-cbc 135.32k 536.01k 1816.32k 4451.51k 13926.40k
aes-128-cbc 137.91k 531.14k 1689.77k 4468.86k 14024.38k
- Hubert
State-Changed-From-To: open->feedback
State-Changed-By: maya@NetBSD.org
State-Changed-When: Mon, 27 May 2019 03:10:02 +0000
State-Changed-Why:
Relevant code got a lot of changes. Is this still an issue?
State-Changed-From-To: feedback->closed
State-Changed-By: hubertf@NetBSD.org
State-Changed-When: Sat, 05 Dec 2020 18:52:03 +0000
State-Changed-Why:
I cannot check if the new code helps.
Let's hope it does, I cannot test it.
Closing this PR.
>Unformatted:
(Contact us)
$NetBSD: gnats-precook-prs,v 1.4 2018/12/21 14:20:20 maya Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2017
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home