NetBSD Problem Report #38141
From yamt@mwd.biglobe.ne.jp Mon Mar 3 00:18:57 2008
Return-Path: <yamt@mwd.biglobe.ne.jp>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by narn.NetBSD.org (Postfix) with ESMTP id 66F5463B90D
for <gnats-bugs@gnats.NetBSD.org>; Mon, 3 Mar 2008 00:18:57 +0000 (UTC)
Message-Id: <20080303001854.DF91F11702@yamt.dyndns.org>
Date: Mon, 3 Mar 2008 09:18:54 +0900 (JST)
From: yamt@mwd.biglobe.ne.jp
Reply-To: yamt@mwd.biglobe.ne.jp
To: gnats-bugs@gnats.NetBSD.org
Subject: lookup/vfs_busy acquire rwlock recursively
X-Send-Pr-Version: 3.95
>Number: 38141
>Category: kern
>Synopsis: lookup/vfs_busy acquire rwlock recursively
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: kern-bug-people
>State: analyzed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Mon Mar 03 00:20:00 +0000 2008
>Closed-Date:
>Last-Modified: Mon Apr 09 06:00:48 +0000 2012
>Originator: YAMAMOTO Takashi <yamt@mwd.biglobe.ne.jp>
>Release: NetBSD 4.99.55
>Organization:
>Environment:
>Description:
lookup can acquire rwlock with RW_READER recursively.
eg.
lookup (holding the lock via vfs_busy) -> VFS_ROOT
-> tmpfs_root -> tmpfs_alloc_vp -> getnewvnode
-> vfs_busy -> rw_enter
>How-To-Repeat:
code inspection.
>Fix:
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: kern-bug-people->ad
Responsible-Changed-By: ad@NetBSD.org
Responsible-Changed-When: Thu, 01 May 2008 12:08:04 +0000
Responsible-Changed-Why:
take
Responsible-Changed-From-To: ad->kern-bug-people
Responsible-Changed-By: ad@NetBSD.org
Responsible-Changed-When: Fri, 02 May 2008 17:44:22 +0000
Responsible-Changed-Why:
I have put a workaround in place but don't have time to fix the problem
completely right now. I think there could also be an issue with
allocation/deallocation of sync vnodes, I can't remember exactly.
State-Changed-From-To: open->analyzed
State-Changed-By: ad@NetBSD.org
State-Changed-When: Fri, 02 May 2008 17:44:22 +0000
State-Changed-Why:
Workaround implemented.
From: Andrew Doran <ad@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/38141 CVS commit: src/sys
Date: Fri, 2 May 2008 17:40:30 +0000 (UTC)
Module Name: src
Committed By: ad
Date: Fri May 2 17:40:30 UTC 2008
Modified Files:
src/sys/kern: vfs_subr.c
src/sys/sys: lwp.h
Log Message:
PR kern/38141 lookup/vfs_busy acquire rwlock recursively
Until the code paths are fixed properly, put in place an ugly workaround
to make it safe to recursively acquire a read lock on a mount.
To generate a diff of this commit:
cvs rdiff -r1.339 -r1.340 src/sys/kern/vfs_subr.c
cvs rdiff -r1.92 -r1.93 src/sys/sys/lwp.h
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: yamt@mwd.biglobe.ne.jp (YAMAMOTO Takashi)
To: gnats-bugs@NetBSD.org
Cc: ad@netbsd.org, kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org
Subject: Re: PR/38141 CVS commit: src/sys
Date: Sat, 3 May 2008 23:05:16 +0900 (JST)
> Log Message:
> PR kern/38141 lookup/vfs_busy acquire rwlock recursively
>
> Until the code paths are fixed properly, put in place an ugly workaround
> to make it safe to recursively acquire a read lock on a mount.
unfortunately, recursive lock was not necessary to triger
a deadlock. i saw the following while running "build.sh -j128".
(well, the actual deadlock i saw had 10 or more vnodes involved.
the following is a simplified version for explanation.)
LWP1 in vn_open()
1. lock a directory vnode which is VV_ROOT.
LWP2 in lookup()
1. vfs_busy(RW_READER)
2. call VFS_ROOT, which tries to lock the VV_ROOT vnode which
is already locked by LWP1. => block
LWP3 (syncer) in sync_fsync()
1. vfs_trybusy(RW_WRITER) => block
now, LWP1 again:
2. call VOP_CREATE -> getnewvnode -> vfs_busy(RW_READER) => block
YAMAMOTO Takashi
From: Andrew Doran <ad@netbsd.org>
To: YAMAMOTO Takashi <yamt@mwd.biglobe.ne.jp>
Cc: gnats-bugs@NetBSD.org
Subject: Re: PR/38141 CVS commit: src/sys
Date: Sun, 4 May 2008 00:57:01 +0100
On Sat, May 03, 2008 at 11:05:16PM +0900, YAMAMOTO Takashi wrote:
> > Log Message:
> > PR kern/38141 lookup/vfs_busy acquire rwlock recursively
> >
> > Until the code paths are fixed properly, put in place an ugly workaround
> > to make it safe to recursively acquire a read lock on a mount.
>
> unfortunately, recursive lock was not necessary to triger
> a deadlock. i saw the following while running "build.sh -j128".
> (well, the actual deadlock i saw had 10 or more vnodes involved.
> the following is a simplified version for explanation.)
>
> LWP1 in vn_open()
> 1. lock a directory vnode which is VV_ROOT.
> LWP2 in lookup()
> 1. vfs_busy(RW_READER)
> 2. call VFS_ROOT, which tries to lock the VV_ROOT vnode which
> is already locked by LWP1. => block
> LWP3 (syncer) in sync_fsync()
> 1. vfs_trybusy(RW_WRITER) => block
> now, LWP1 again:
> 2. call VOP_CREATE -> getnewvnode -> vfs_busy(RW_READER) => block
Heh, lockmgr() strikes from beyond the grave. I guess we could make it safe
in the short term by making vfs_busy() an even dumber kind of rwlock based
on atomics, with no preference for writers.
sched_fsync() could be a problem becase of VOP_LOCK -> vfs_busy(RW_WRITER).
I suppose there is no need for it to be represented as a vnode, and we could
make the syncer look over the list of file systems for ones that need a
periodic VFS_SYNC().
What do you think?
Andrew
From: Andrew Doran <ad@netbsd.org>
To: gnats-bugs@netbsd.org, yamt@mwd.biglobe.ne.jp
Cc:
Subject: Re: PR/38141 CVS commit: src/sys
Date: Sun, 4 May 2008 01:21:58 +0100
On Sun, May 04, 2008 at 12:00:12AM +0000, Andrew Doran wrote:
> The following reply was made to PR kern/38141; it has been noted by GNATS.
>
> From: Andrew Doran <ad@netbsd.org>
> To: YAMAMOTO Takashi <yamt@mwd.biglobe.ne.jp>
> Cc: gnats-bugs@NetBSD.org
> Subject: Re: PR/38141 CVS commit: src/sys
> Date: Sun, 4 May 2008 00:57:01 +0100
>
> On Sat, May 03, 2008 at 11:05:16PM +0900, YAMAMOTO Takashi wrote:
> > > Log Message:
> > > PR kern/38141 lookup/vfs_busy acquire rwlock recursively
> > >
> > > Until the code paths are fixed properly, put in place an ugly workaround
> > > to make it safe to recursively acquire a read lock on a mount.
> >
> > unfortunately, recursive lock was not necessary to triger
> > a deadlock. i saw the following while running "build.sh -j128".
> > (well, the actual deadlock i saw had 10 or more vnodes involved.
> > the following is a simplified version for explanation.)
> >
> > LWP1 in vn_open()
> > 1. lock a directory vnode which is VV_ROOT.
> > LWP2 in lookup()
> > 1. vfs_busy(RW_READER)
> > 2. call VFS_ROOT, which tries to lock the VV_ROOT vnode which
> > is already locked by LWP1. => block
> > LWP3 (syncer) in sync_fsync()
> > 1. vfs_trybusy(RW_WRITER) => block
> > now, LWP1 again:
> > 2. call VOP_CREATE -> getnewvnode -> vfs_busy(RW_READER) => block
>
> Heh, lockmgr() strikes from beyond the grave. I guess we could make it safe
> in the short term by making vfs_busy() an even dumber kind of rwlock based
> on atomics, with no preference for writers.
>
> sched_fsync() could be a problem becase of VOP_LOCK -> vfs_busy(RW_WRITER).
> I suppose there is no need for it to be represented as a vnode, and we could
> make the syncer look over the list of file systems for ones that need a
> periodic VFS_SYNC().
>
> What do you think?
Actually, it's still not enough. Many of the existing users of vfs_busy()
would also need to be converted to vfs_trybusy(), for example in sys_fchdir:
chdir: vn_lock -> vfs_busy
unmount vfs_busy (mnt_unmounter != NULL) -> vn_lock
Andrew
Responsible-Changed-From-To: kern-bug-people->ad
Responsible-Changed-By: ad@NetBSD.org
Responsible-Changed-When: Tue, 06 May 2008 12:35:18 +0000
Responsible-Changed-Why:
More needs to be done on this.
From: Andrew Doran <ad@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/38141 CVS commit: src/sys/kern
Date: Tue, 6 May 2008 12:37:04 +0000 (UTC)
Module Name: src
Committed By: ad
Date: Tue May 6 12:37:04 UTC 2008
Modified Files:
src/sys/kern: vfs_subr.c
Log Message:
PR kern/38141 lookup/vfs_busy acquire rwlock recursively
getvnode: Use vfs_trybusy, not vfs_busy. If unmount is in progress we
could deadlock, because vnode locks can be held during getnewvnode().
dounmount() locks in the reverse order (vfs_busy -> vnode).
To generate a diff of this commit:
cvs rdiff -r1.341 -r1.342 src/sys/kern/vfs_subr.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Andrew Doran <ad@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/38141 CVS commit: src/sys/kern
Date: Tue, 6 May 2008 12:39:32 +0000 (UTC)
Module Name: src
Committed By: ad
Date: Tue May 6 12:39:32 UTC 2008
Modified Files:
src/sys/kern: vfs_subr.c
Log Message:
PR kern/38141 lookup/vfs_busy acquire rwlock recursively
vfs_busy: don't deadlock if curlwp is unmounting.
To generate a diff of this commit:
cvs rdiff -r1.342 -r1.343 src/sys/kern/vfs_subr.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Andrew Doran <ad@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/38141 CVS commit: src/sys/kern
Date: Tue, 6 May 2008 12:54:25 +0000 (UTC)
Module Name: src
Committed By: ad
Date: Tue May 6 12:54:25 UTC 2008
Modified Files:
src/sys/kern: vfs_syscalls.c
Log Message:
PR kern/38141 lookup/vfs_busy acquire rwlock recursively
- sys_sync: acquire a write lock on the mount since the operation modifies
the mount structure.
- sys_fchdir: use vfs_trybusy(). If an unmount is in progress, just fail it.
To generate a diff of this commit:
cvs rdiff -r1.355 -r1.356 src/sys/kern/vfs_syscalls.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: analyzed->feedback
State-Changed-By: ad@NetBSD.org
State-Changed-When: Tue, 06 May 2008 18:45:48 +0000
State-Changed-Why:
I'm hoping that this should be well and truly fixed now.
Can you please confirm?
From: Andrew Doran <ad@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/38141 CVS commit: src/sys
Date: Tue, 6 May 2008 18:43:45 +0000 (UTC)
Module Name: src
Committed By: ad
Date: Tue May 6 18:43:45 UTC 2008
Modified Files:
src/sys/coda: coda_psdev.c
src/sys/compat/common: vfs_syscalls_20.c
src/sys/compat/netbsd32: netbsd32_compat_20.c
src/sys/compat/osf1: osf1_mount.c
src/sys/compat/ultrix: ultrix_fs.c
src/sys/fs/cd9660: cd9660_vfsops.c
src/sys/fs/filecorefs: filecore_vfsops.c
src/sys/fs/msdosfs: msdosfs_vfsops.c
src/sys/fs/ntfs: ntfs_vfsops.c
src/sys/fs/puffs: puffs_msgif.c
src/sys/fs/union: union_vnops.c
src/sys/kern: vfs_lookup.c vfs_subr.c vfs_subr2.c vfs_syscalls.c
src/sys/miscfs/procfs: procfs_linux.c
src/sys/miscfs/syncfs: sync_vnops.c
src/sys/nfs: nfs_export.c nfs_vfsops.c
src/sys/rump/librump/rumpkern: rump.c
src/sys/sys: fstypes.h mount.h
src/sys/ufs/ext2fs: ext2fs_vfsops.c
src/sys/ufs/ffs: ffs_vfsops.c
src/sys/ufs/lfs: lfs_bio.c lfs_syscalls.c lfs_vfsops.c
src/sys/ufs/mfs: mfs_extern.h mfs_vfsops.c mfs_vnops.c
src/sys/ufs/ufs: ufs_vfsops.c
Log Message:
PR kern/38141 lookup/vfs_busy acquire rwlock recursively
Simplify the mount locking. Remove all the crud to deal with recursion on
the mount lock, and crud to deal with unmount as another weirdo lock.
Hopefully this will once and for all fix the deadlocks with this. With this
commit there are two locks on each mount:
- krwlock_t mnt_unmounting. This is used to prevent unmount across critical
sections like getnewvnode(). It's only ever read locked with rw_tryenter(),
and is only ever write locked in dounmount(). A write hold can't be taken
on this lock if the current LWP could hold a vnode lock.
- kmutex_t mnt_updating. This is taken by threads updating the mount, for
example when going r/o -> r/w, and is only present to serialize updates.
In order to take this lock, a read hold must first be taken on
mnt_unmounting, and the two need to be held across the operation.
One effect of this change: previously if an unmount failed, we would make a
half hearted attempt to back out of it gracefully, but that was unlikely to
work in a lot of cases. Now while an unmount that will be aborted is in
progress, new file operations within the mount will fail instead of being
delayed. That is unlikely to be a problem though, because if the admin
requests unmount of a file system then s(he) has made a decision to deny
access to the resource.
To generate a diff of this commit:
cvs rdiff -r1.44 -r1.45 src/sys/coda/coda_psdev.c
cvs rdiff -r1.29 -r1.30 src/sys/compat/common/vfs_syscalls_20.c
cvs rdiff -r1.23 -r1.24 src/sys/compat/netbsd32/netbsd32_compat_20.c
cvs rdiff -r1.43 -r1.44 src/sys/compat/osf1/osf1_mount.c
cvs rdiff -r1.48 -r1.49 src/sys/compat/ultrix/ultrix_fs.c
cvs rdiff -r1.60 -r1.61 src/sys/fs/cd9660/cd9660_vfsops.c
cvs rdiff -r1.51 -r1.52 src/sys/fs/filecorefs/filecore_vfsops.c
cvs rdiff -r1.64 -r1.65 src/sys/fs/msdosfs/msdosfs_vfsops.c
cvs rdiff -r1.68 -r1.69 src/sys/fs/ntfs/ntfs_vfsops.c
cvs rdiff -r1.70 -r1.71 src/sys/fs/puffs/puffs_msgif.c
cvs rdiff -r1.32 -r1.33 src/sys/fs/union/union_vnops.c
cvs rdiff -r1.107 -r1.108 src/sys/kern/vfs_lookup.c
cvs rdiff -r1.343 -r1.344 src/sys/kern/vfs_subr.c
cvs rdiff -r1.23 -r1.24 src/sys/kern/vfs_subr2.c
cvs rdiff -r1.357 -r1.358 src/sys/kern/vfs_syscalls.c
cvs rdiff -r1.52 -r1.53 src/sys/miscfs/procfs/procfs_linux.c
cvs rdiff -r1.24 -r1.25 src/sys/miscfs/syncfs/sync_vnops.c
cvs rdiff -r1.36 -r1.37 src/sys/nfs/nfs_export.c
cvs rdiff -r1.198 -r1.199 src/sys/nfs/nfs_vfsops.c
cvs rdiff -r1.44 -r1.45 src/sys/rump/librump/rumpkern/rump.c
cvs rdiff -r1.22 -r1.23 src/sys/sys/fstypes.h
cvs rdiff -r1.177 -r1.178 src/sys/sys/mount.h
cvs rdiff -r1.133 -r1.134 src/sys/ufs/ext2fs/ext2fs_vfsops.c
cvs rdiff -r1.225 -r1.226 src/sys/ufs/ffs/ffs_vfsops.c
cvs rdiff -r1.113 -r1.114 src/sys/ufs/lfs/lfs_bio.c
cvs rdiff -r1.131 -r1.132 src/sys/ufs/lfs/lfs_syscalls.c
cvs rdiff -r1.259 -r1.260 src/sys/ufs/lfs/lfs_vfsops.c
cvs rdiff -r1.28 -r1.29 src/sys/ufs/mfs/mfs_extern.h
cvs rdiff -r1.95 -r1.96 src/sys/ufs/mfs/mfs_vfsops.c
cvs rdiff -r1.49 -r1.50 src/sys/ufs/mfs/mfs_vnops.c
cvs rdiff -r1.38 -r1.39 src/sys/ufs/ufs/ufs_vfsops.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: yamt@mwd.biglobe.ne.jp (YAMAMOTO Takashi)
To: gnats-bugs@NetBSD.org
Cc: ad@NetBSD.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: PR/38141 CVS commit: src/sys
Date: Wed, 7 May 2008 09:36:37 +0900 (JST)
> One effect of this change: previously if an unmount failed, we would make a
> half hearted attempt to back out of it gracefully, but that was unlikely to
> work in a lot of cases. Now while an unmount that will be aborted is in
> progress, new file operations within the mount will fail instead of being
> delayed. That is unlikely to be a problem though, because if the admin
> requests unmount of a file system then s(he) has made a decision to deny
> access to the resource.
doesn't it break, say, amd(8)?
YAMAMOTO Takashi
From: Andrew Doran <ad@NetBSD.org>
To: YAMAMOTO Takashi <yamt@mwd.biglobe.ne.jp>
Cc: gnats-bugs@NetBSD.org
Subject: Re: PR/38141 CVS commit: src/sys
Date: Wed, 7 May 2008 02:21:24 +0100
On Wed, May 07, 2008 at 09:36:37AM +0900, YAMAMOTO Takashi wrote:
> > One effect of this change: previously if an unmount failed, we would make a
> > half hearted attempt to back out of it gracefully, but that was unlikely to
> > work in a lot of cases. Now while an unmount that will be aborted is in
> > progress, new file operations within the mount will fail instead of being
> > delayed. That is unlikely to be a problem though, because if the admin
> > requests unmount of a file system then s(he) has made a decision to deny
> > access to the resource.
>
> doesn't it break, say, amd(8)?
It seems like it would be problematic for the idle mount stuff. It might be
possible to use VFS_ROOT() as a single point that can be safely gated. I'll
investigate.
Andrew
From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, ad@NetBSD.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, yamt@mwd.biglobe.ne.jp
Cc:
Subject: Re: PR/38141 CVS commit: src/sys
Date: Wed, 7 May 2008 08:59:19 -0400
On May 7, 12:40am, yamt@mwd.biglobe.ne.jp (YAMAMOTO Takashi) wrote:
-- Subject: Re: PR/38141 CVS commit: src/sys
| The following reply was made to PR kern/38141; it has been noted by GNATS.
|
| From: yamt@mwd.biglobe.ne.jp (YAMAMOTO Takashi)
| To: gnats-bugs@NetBSD.org
| Cc: ad@NetBSD.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
| Subject: Re: PR/38141 CVS commit: src/sys
| Date: Wed, 7 May 2008 09:36:37 +0900 (JST)
|
| > One effect of this change: previously if an unmount failed, we would make a
| > half hearted attempt to back out of it gracefully, but that was unlikely to
| > work in a lot of cases. Now while an unmount that will be aborted is in
| > progress, new file operations within the mount will fail instead of being
| > delayed. That is unlikely to be a problem though, because if the admin
| > requests unmount of a file system then s(he) has made a decision to deny
| > access to the resource.
|
| doesn't it break, say, amd(8)?
amd will fail if umount on a busy filesystem succeeds.
christos
From: Andrew Doran <ad@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: PR/38141 CVS commit: src/sys
Date: Wed, 7 May 2008 14:20:37 +0100
On Wed, May 07, 2008 at 01:00:04PM +0000, Christos Zoulas wrote:
> The following reply was made to PR kern/38141; it has been noted by GNATS.
>
> From: christos@zoulas.com (Christos Zoulas)
> To: gnats-bugs@NetBSD.org, ad@NetBSD.org, gnats-admin@netbsd.org,
> netbsd-bugs@netbsd.org, yamt@mwd.biglobe.ne.jp
> Cc:
> Subject: Re: PR/38141 CVS commit: src/sys
> Date: Wed, 7 May 2008 08:59:19 -0400
>
> On May 7, 12:40am, yamt@mwd.biglobe.ne.jp (YAMAMOTO Takashi) wrote:
> -- Subject: Re: PR/38141 CVS commit: src/sys
>
> | The following reply was made to PR kern/38141; it has been noted by GNATS.
> |
> | From: yamt@mwd.biglobe.ne.jp (YAMAMOTO Takashi)
> | To: gnats-bugs@NetBSD.org
> | Cc: ad@NetBSD.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
> | Subject: Re: PR/38141 CVS commit: src/sys
> | Date: Wed, 7 May 2008 09:36:37 +0900 (JST)
> |
> | > One effect of this change: previously if an unmount failed, we would make a
> | > half hearted attempt to back out of it gracefully, but that was unlikely to
> | > work in a lot of cases. Now while an unmount that will be aborted is in
> | > progress, new file operations within the mount will fail instead of being
> | > delayed. That is unlikely to be a problem though, because if the admin
> | > requests unmount of a file system then s(he) has made a decision to deny
> | > access to the resource.
> |
> | doesn't it break, say, amd(8)?
>
> amd will fail if umount on a busy filesystem succeeds.
I'm guessing how amd works - haven't researched it yet - but I think the
main problem will be new calls to VFS_ROOT() from lookup(). While an idle
file system is being garbage collected by amd, those will fail with EBUSY.
So there's a small window across the unmount where operations would fail
instead of causing an automount to occur.
For an unmount that's not forced, those should be easy enough to gate
because it's OK to wait there. The deadlocks (some of which have been there
for a long time) start happening when we cause threads already in the guts
of the file system code to wait, because there is a tangled mess of locks.
Andrew
State-Changed-From-To: feedback->analyzed
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Sun, 15 Nov 2009 00:00:27 +0000
State-Changed-Why:
this is not fully fixed but does not need to be in feedback.
Responsible-Changed-From-To: ad->kern-bug-people
Responsible-Changed-By: dholland@NetBSD.org
Responsible-Changed-When: Mon, 09 Apr 2012 06:00:48 +0000
Responsible-Changed-Why:
ad resigned, should not own PRs any more
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.