NetBSD Problem Report #40728
From root@nagler-company.com Mon Feb 23 12:46:30 2009
Return-Path: <root@nagler-company.com>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by www.NetBSD.org (Postfix) with ESMTP id 3710263B8C3
for <gnats-bugs@gnats.NetBSD.org>; Mon, 23 Feb 2009 12:46:30 +0000 (UTC)
Message-Id: <20090223124608.F32634EA9F9@s012.nagler-company.com>
Date: Mon, 23 Feb 2009 13:46:08 +0100 (CET)
From: Wolfgang.Stukenbrock@nagler-company.com
Reply-To: Wolfgang.Stukenbrock@nagler-company.com
To: gnats-bugs@gnats.NetBSD.org
Subject: expanding NIS-Netgroups for users and groups is very slow
X-Send-Pr-Version: 3.95
>Number: 40728
>Category: lib
>Synopsis: expanding NIS-Netgroups for users and groups is very slow
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: lib-bug-people
>State: open
>Class: change-request
>Submitter-Id: net
>Arrival-Date: Mon Feb 23 12:50:00 +0000 2009
>Last-Modified: Thu Mar 29 09:25:00 +0000 2012
>Originator: W. Stukenbrock
>Release: NetBSD 4.0
>Organization:
Dr. Nagler & Company GmbH
>Environment:
System: NetBSD s012 4.0 NetBSD 4.0 (NSW-S012) #1: Thu Sep 11 12:21:03 CEST 2008 root@s012:/usr/src/sys/arch/amd64/compile/NSW-S012 amd64
Architecture: x86_64
Machine: amd64
>Description:
The problem has been recognized while running "ls -l" as root and as non-root-user.
When running as root the statement takes significant longer (something around 20 seconds) as running as non-root-user on
the same directory. The call for the non-root-user takes less than 2 seconds.
The system is a YP-client with no local ypserv and setup in passwd and group compat mode with compat-nis.
I've analysed the problem by tracing the network and found a lot of repeated YP-request to check the existens of
passwd.adjunct and passwd.master.byname.
A look into the ls-command sources shows, that it uses the pwcache-module - user_from_uid().
The pwcache-module tries to minimize the lookup-effort of the getpwent-routines by setting stayopen to 1.
Dooing this should keep all possible fd's etc. active in order to avoid redundant requests etc.
In the getpwent (or getgrent) routines the ns-switch is used.
The pwcache calls now getpwuid() to solve the user_from_uid() call.
This call will go though _compat_getpwuid() in getpwent.c, will find the included netgroup (e.g. "+@NU_USER_FOR_HOST") in the
password file and starts netgroup processing (setnetgrent(), getnetgrent(), endnetgrent()).
It will call the ns-switch for passwd_compat for every member in the netgroup with the function getpwnam_r.
This will call _nis_getpwnam_r() - we have compat set to NIS - and is here the problem.
Even if we have set stayopen - _nis_getpwnam_r() uses a local state information and ignores the global stayopen condition.
So it will start over with determining the domainname and for the user root check if there are passwd.adjunct or passwd.master.byname
maps on the server.
The same happens to the group lookup stuff.
The problem gets gets bigger and bigger with the member count of the included netgroup. Each lookup will take up to 3 times the required
time for each member of the group. And if you are out of luck, the pwcache module is not able to cache a lot due to the simple
hash mechanism there.
A look the implementation of _nis_getpwnam_r() and _nis_getpwuid_r() shows, that they are just callint ypmatch(...) and don't touch
any other state information than the domainname and the cached information for passwd.adjunct or passwd.master.byname maps..
So it is easy to avoid the repeated lookups by useing the global state buffer for NIS if stayopen is set there.
No additional synchronisation is needed for the current implementation, because all calls enter a module-monitor prior the first call
to the ns-switch stuff. If this will be changed in the future, the current state information should be copied to the local state info
in theese routines with an adequate mutex aquired, but this is a possbile todo for the future.
With the following patch the run time of ls -l" as user root is much faster as before and reaches the time of a non-root-user,
if there are no passwd.adjunct and passwd.master.byname maps.
An analog problem may be in the HESIOD code part, but we are not useing HESIOD-maps here, so I cannot test anything and therefore
I haven't changed the code there.
>How-To-Repeat:
Setup a large netgroup and add it to /etc/passwd and/or /etc/group.
Run agains a server without passwd.master (and passwd.adjunct) maps.
Try "ls -l" as root and trace the network trafic. You will see two request for passwd.adjunct and passwd.master.byname
for each member in the netgroup and they report all the time, that the maps are not present.
>Fix:
The following patches will use the global state buffer in getpwent.c and getgrent.c if stayopen has been set.
--- getgrent.c 2009/02/23 10:50:32 1.1
+++ getgrent.c 2009/02/23 12:45:45
@@ -1194,9 +1194,17 @@
_DIAGASSERT(result != NULL);
*result = NULL;
- memset(&state, 0, sizeof(state));
- rv = __grscan_nis(retval, grp, buffer, buflen, &state, 1, NULL, gid);
- __grend_nis(&state);
+/* remark: we run under a global mutex inside of this module ... */
+ if (_nis_state.stayopen)
+ { /* use global state only if stayopen is set - otherwiese we would blow up getgrent_r() ... */
+ rv = __grscan_nis(retval, grp, buffer, buflen, &_nis_state, 1, NULL, gid);
+ }
+ else
+ {
+ memset(&state, 0, sizeof(state));
+ rv = __grscan_nis(retval, grp, buffer, buflen, &state, 1, NULL, gid);
+ __grend_nis(&state);
+ }
if (rv == NS_SUCCESS)
*result = grp;
return rv;
@@ -1246,9 +1254,17 @@
_DIAGASSERT(result != NULL);
*result = NULL;
- memset(&state, 0, sizeof(state));
- rv = __grscan_nis(retval, grp, buffer, buflen, &state, 1, name, 0);
- __grend_nis(&state);
+/* remark: we run under a global mutex inside of this module ... */
+ if (_nis_state.stayopen)
+ { /* use global state only if stayopen is set - otherwiese we would blow up getgrent_r() ... */
+ rv = __grscan_nis(retval, grp, buffer, buflen, &_nis_state, 1, name, 0);
+ }
+ else
+ {
+ memset(&state, 0, sizeof(state));
+ rv = __grscan_nis(retval, grp, buffer, buflen, &state, 1, name, 0);
+ __grend_nis(&state);
+ }
if (rv == NS_SUCCESS)
*result = grp;
return rv;
--- getpwent.c 2009/02/23 10:50:32 1.1
+++ getpwent.c 2009/02/23 11:09:31
@@ -1529,14 +1529,21 @@
_DIAGASSERT(result != NULL);
*result = NULL;
- memset(&state, 0, sizeof(state));
- rv = _nis_start(&state);
- if (rv != NS_SUCCESS)
- return rv;
snprintf(buffer, buflen, "%u", (unsigned int)uid);
- rv = _nis_pwscan(retval, pw, buffer, buflen,
- &state, PASSWD_BYUID(&state));
- _nis_end(&state);
+/* remark: we run under a global mutex inside of this module ... */
+ if (_nis_state.stayopen)
+ { /* use global state only if stayopen is set - otherwiese we would blow up getpwent_r() ... */
+ rv = _nis_pwscan(retval, pw, buffer, buflen,
+ &_nis_state, PASSWD_BYUID(&_nis_state));
+ }
+ else
+ { /* keep old semantic if no stayopen set - no need to call _nis_start() here - _nis_pwscan() will do it for us ... */
+ /* use same way as in getgrent.c ... */
+ memset(&state, 0, sizeof(state));
+ rv = _nis_pwscan(retval, pw, buffer, buflen,
+ &state, PASSWD_BYUID(&state));
+ _nis_end(&state);
+ }
if (rv != NS_SUCCESS)
return rv;
if (uid == pw->pw_uid) {
@@ -1593,13 +1600,20 @@
*result = NULL;
snprintf(buffer, buflen, "%s", name);
- memset(&state, 0, sizeof(state));
- rv = _nis_start(&state);
- if (rv != NS_SUCCESS)
- return rv;
- rv = _nis_pwscan(retval, pw, buffer, buflen,
- &state, PASSWD_BYNAME(&state));
- _nis_end(&state);
+/* remark: we run under a global mutex inside of this module ... */
+ if (_nis_state.stayopen)
+ { /* use global state only if stayopen is set - otherwiese we would blow up getpwent_r() ... */
+ rv = _nis_pwscan(retval, pw, buffer, buflen,
+ &_nis_state, PASSWD_BYNAME(&_nis_state));
+ }
+ else
+ { /* keep old semantic if no stayopen set - no need to call _nis_start() here - _nis_pwscan() will do it for us ... */
+ /* use same way as in getgrent.c ... */
+ memset(&state, 0, sizeof(state));
+ rv = _nis_pwscan(retval, pw, buffer, buflen,
+ &state, PASSWD_BYNAME(&state));
+ _nis_end(&state);
+ }
if (rv != NS_SUCCESS)
return rv;
if (strcmp(name, pw->pw_name) == 0) {
>Audit-Trail:
From: Wolfgang Stukenbrock <Wolfgang.Stukenbrock@nagler-company.com>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: lib/40728: expanding NIS-Netgroups for users and groups is very slow
Date: Mon, 23 Feb 2009 16:49:07 +0100
Hi, again,
additional testing has chown, that I missed to enter the folloing part
of the patch to getpwent.c.
Without this change, "+" lines will drop the stayopen state and loop
with requesting the "master.passwd" map again
Sorry my fault.
here the missing patch part.
@@ -2054,7 +2068,7 @@
state->mode = COMPAT_FULL;
/* reset passwd_compat
search */
/* XXXREENTRANT: setpassent is not thread safe ? */
- (void) _passwdcompat_setpassent(0);
+ (void)
_passwdcompat_setpassent(_compat_state.stayopen);
break;
case '@': /* `+@netgroup' */
state->mode = COMPAT_NETGROUP;
gnats-admin@NetBSD.org wrote:
>Thank you very much for your problem report.
>It has the internal identification `lib/40728'.
>The individual assigned to look at your
>report is: lib-bug-people.
>
>>Category: lib
>>Responsible: lib-bug-people
>>Synopsis: expanding NIS-Netgroups for users and groups is very slow
>>Arrival-Date: Mon Feb 23 12:50:00 +0000 2009
>>
>
From: Wolfgang Stukenbrock <wolfgang.stukenbrock@nagler-company.com>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: lib/40728: expanding NIS-Netgroups for users and groups is very slow
Date: Thu, 29 Mar 2012 11:24:31 +0200
This is a multi-part message in MIME format.
--------------040207020604010308030301
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Hi again,
I've now switched my passwd database setup from passwd to master.passwd
YP-maps and recognised a bug in my patch.
Due to the fact that _nis_start() is no longer called prior calling
_nis_pwscan in any case, we may fail to select the master.passwd map
correctly.
Due to the fact that we should not call _nis_start() if we reuse the
state, there is no easy way to add this call again.
The better sollution - from my point of view - is to delay the
map-selection in _nis_pwscan() after the (optional) build-in call to
_nis_start.
The following patch will solve this problem by passing an array instead
of the map name and take the correct map from there after the optional
_nis_start() call in _nis_pwscan:
--- getpwent.c 2010/05/03 12:54:18 1.2
+++ getpwent.c 2012/03/28 11:29:48
@@ -1099,7 +1099,7 @@
char *current; /* current first/next match */
int currentlen; /* length of _nis_current */
enum { /* shadow map type */
- NISMAP_UNKNOWN, /* unknown ... */
+ NISMAP_UNKNOWN = 0, /* unknown ... */
NISMAP_NONE, /* none: use "passwd.by*" */
NISMAP_ADJUNCT, /* pw_passwd from
"passwd.adjunct.*" */
NISMAP_MASTER /* all from
"master.passwd.by*" */
@@ -1111,11 +1111,17 @@
static struct passwd _nis_passwd;
static char _nis_passwdbuf[_GETPW_R_SIZE_MAX];
+static const char __nis_pw_n_1[] = "master.passwd.byname";
+static const char __nis_pw_n_2[] = "passwd.byname";
+static const char __nis_pw_u_1[] = "master.passwd.byuid";
+static const char __nis_pw_u_2[] = "passwd.byuid";
+
+static const char * const __nis_pw_n_map[4] = { __nis_pw_n_2,
__nis_pw_n_2, __nis_pw_n_2, __nis_pw_n_1 };
+static const char * const __nis_pw_u_map[4] = { __nis_pw_u_2,
__nis_pw_u_2, __nis_pw_u_2, __nis_pw_u_1 };
+
/* macros for deciding which NIS maps to use. */
-#define PASSWD_BYNAME(x) ((x)->maptype == NISMAP_MASTER \
- ? "master.passwd.byname" :
"passwd.byname")
-#define PASSWD_BYUID(x) ((x)->maptype == NISMAP_MASTER \
- ? "master.passwd.byuid" :
"passwd.byuid")
+#define PASSWD_BYNAME(x) ((x)->maptype == NISMAP_MASTER ?
__nis_pw_n_1 : __nis_pw_n_2)
+#define PASSWD_BYUID(x) ((x)->maptype == NISMAP_MASTER ?
__nis_pw_u_1 : __nis_pw_u_2)
static int
_nis_start(struct nis_state *state)
@@ -1236,7 +1242,7 @@
*/
static int
_nis_pwscan(int *retval, struct passwd *pw, char *buffer, size_t buflen,
- struct nis_state *state, const char *map)
+ struct nis_state *state, const char * const *map_arr)
{
char *data;
int nisr, rv, datalen;
@@ -1245,7 +1251,7 @@
_DIAGASSERT(pw != NULL);
_DIAGASSERT(buffer != NULL);
_DIAGASSERT(state != NULL);
- _DIAGASSERT(map != NULL);
+ _DIAGASSERT(map_arr != NULL);
*retval = 0;
@@ -1257,9 +1263,10 @@
data = NULL;
rv = NS_NOTFOUND;
+ _DIAGASSERT(state->maptype > 0 && state->maptype <
sizeof(map_arr)/sizeof(const char*));
/* search map */
- nisr = yp_match(state->domain, map, buffer, (int)strlen(buffer),
+ nisr = yp_match(state->domain, map_arr[state->maptype], buffer,
(int)strlen(buffer),
&data, &datalen);
switch (nisr) {
case 0:
@@ -1494,7 +1501,7 @@
snprintf(_nis_passwdbuf, sizeof(_nis_passwdbuf), "%u",
(unsigned int)uid);
rv = _nis_pwscan(&rerror, &_nis_passwd,
_nis_passwdbuf, sizeof(_nis_passwdbuf),
- &_nis_state, PASSWD_BYUID(&_nis_state));
+ &_nis_state, __nis_pw_u_map);
if (!_nis_state.stayopen)
_nis_end(&_nis_state);
if (rv == NS_SUCCESS && uid == _nis_passwd.pw_uid)
@@ -1527,14 +1534,14 @@
if (_nis_state.stayopen)
{ /* use global state only if stayopen is set - otherwiese we
would blow up getpwent_r() ... */
rv = _nis_pwscan(retval, pw, buffer, buflen,
- &_nis_state, PASSWD_BYUID(&_nis_state));
+ &_nis_state, __nis_pw_u_map);
}
else
{ /* keep old semantic if no stayopen set - no need to call
_nis_start() here - _nis_pwscan() will do it for us ... */
/* use same way as in getgrent.c ... */
memset(&state, 0, sizeof(state));
rv = _nis_pwscan(retval, pw, buffer, buflen,
- &state, PASSWD_BYUID(&state));
+ &state, __nis_pw_u_map);
_nis_end(&state);
}
if (rv != NS_SUCCESS)
@@ -1564,7 +1571,7 @@
snprintf(_nis_passwdbuf, sizeof(_nis_passwdbuf), "%s", name);
rv = _nis_pwscan(&rerror, &_nis_passwd,
_nis_passwdbuf, sizeof(_nis_passwdbuf),
- &_nis_state, PASSWD_BYNAME(&_nis_state));
+ &_nis_state, __nis_pw_n_map);
if (!_nis_state.stayopen)
_nis_end(&_nis_state);
if (rv == NS_SUCCESS && strcmp(name, _nis_passwd.pw_name) == 0)
@@ -1597,14 +1604,14 @@
if (_nis_state.stayopen)
{ /* use global state only if stayopen is set - otherwiese we
would blow up getpwent_r() ... */
rv = _nis_pwscan(retval, pw, buffer, buflen,
- &_nis_state, PASSWD_BYNAME(&_nis_state));
+ &_nis_state, __nis_pw_n_map);
}
else
{ /* keep old semantic if no stayopen set - no need to call
_nis_start() here - _nis_pwscan() will do it for us ... */
/* use same way as in getgrent.c ... */
memset(&state, 0, sizeof(state));
rv = _nis_pwscan(retval, pw, buffer, buflen,
- &state, PASSWD_BYNAME(&state));
+ &state, __nis_pw_n_map);
_nis_end(&state);
}
if (rv != NS_SUCCESS)
###### end of additional patch ##########
And for more easy integration the complete patch for getpwent.c against
5.1-release again as attachment ....
--------------040207020604010308030301
Content-Type: text/plain;
name="getpwent-1.c-patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="getpwent-1.c-patch"
--- getpwent.c 2010/05/03 12:50:13 1.1
+++ getpwent.c 2012/03/28 11:29:48
@@ -1099,7 +1099,7 @@
char *current; /* current first/next match */
int currentlen; /* length of _nis_current */
enum { /* shadow map type */
- NISMAP_UNKNOWN, /* unknown ... */
+ NISMAP_UNKNOWN = 0, /* unknown ... */
NISMAP_NONE, /* none: use "passwd.by*" */
NISMAP_ADJUNCT, /* pw_passwd from "passwd.adjunct.*" */
NISMAP_MASTER /* all from "master.passwd.by*" */
@@ -1111,11 +1111,17 @@
static struct passwd _nis_passwd;
static char _nis_passwdbuf[_GETPW_R_SIZE_MAX];
+static const char __nis_pw_n_1[] = "master.passwd.byname";
+static const char __nis_pw_n_2[] = "passwd.byname";
+static const char __nis_pw_u_1[] = "master.passwd.byuid";
+static const char __nis_pw_u_2[] = "passwd.byuid";
+
+static const char * const __nis_pw_n_map[4] = { __nis_pw_n_2, __nis_pw_n_2, __nis_pw_n_2, __nis_pw_n_1 };
+static const char * const __nis_pw_u_map[4] = { __nis_pw_u_2, __nis_pw_u_2, __nis_pw_u_2, __nis_pw_u_1 };
+
/* macros for deciding which NIS maps to use. */
-#define PASSWD_BYNAME(x) ((x)->maptype == NISMAP_MASTER \
- ? "master.passwd.byname" : "passwd.byname")
-#define PASSWD_BYUID(x) ((x)->maptype == NISMAP_MASTER \
- ? "master.passwd.byuid" : "passwd.byuid")
+#define PASSWD_BYNAME(x) ((x)->maptype == NISMAP_MASTER ? __nis_pw_n_1 : __nis_pw_n_2)
+#define PASSWD_BYUID(x) ((x)->maptype == NISMAP_MASTER ? __nis_pw_u_1 : __nis_pw_u_2)
static int
_nis_start(struct nis_state *state)
@@ -1236,7 +1242,7 @@
*/
static int
_nis_pwscan(int *retval, struct passwd *pw, char *buffer, size_t buflen,
- struct nis_state *state, const char *map)
+ struct nis_state *state, const char * const *map_arr)
{
char *data;
int nisr, rv, datalen;
@@ -1245,7 +1251,7 @@
_DIAGASSERT(pw != NULL);
_DIAGASSERT(buffer != NULL);
_DIAGASSERT(state != NULL);
- _DIAGASSERT(map != NULL);
+ _DIAGASSERT(map_arr != NULL);
*retval = 0;
@@ -1257,9 +1263,10 @@
data = NULL;
rv = NS_NOTFOUND;
+ _DIAGASSERT(state->maptype > 0 && state->maptype < sizeof(map_arr)/sizeof(const char*));
/* search map */
- nisr = yp_match(state->domain, map, buffer, (int)strlen(buffer),
+ nisr = yp_match(state->domain, map_arr[state->maptype], buffer, (int)strlen(buffer),
&data, &datalen);
switch (nisr) {
case 0:
@@ -1494,7 +1501,7 @@
snprintf(_nis_passwdbuf, sizeof(_nis_passwdbuf), "%u", (unsigned int)uid);
rv = _nis_pwscan(&rerror, &_nis_passwd,
_nis_passwdbuf, sizeof(_nis_passwdbuf),
- &_nis_state, PASSWD_BYUID(&_nis_state));
+ &_nis_state, __nis_pw_u_map);
if (!_nis_state.stayopen)
_nis_end(&_nis_state);
if (rv == NS_SUCCESS && uid == _nis_passwd.pw_uid)
@@ -1522,14 +1529,21 @@
_DIAGASSERT(result != NULL);
*result = NULL;
- memset(&state, 0, sizeof(state));
- rv = _nis_start(&state);
- if (rv != NS_SUCCESS)
- return rv;
snprintf(buffer, buflen, "%u", (unsigned int)uid);
- rv = _nis_pwscan(retval, pw, buffer, buflen,
- &state, PASSWD_BYUID(&state));
- _nis_end(&state);
+/* remark: we run under a global mutex inside of this module ... */
+ if (_nis_state.stayopen)
+ { /* use global state only if stayopen is set - otherwiese we would blow up getpwent_r() ... */
+ rv = _nis_pwscan(retval, pw, buffer, buflen,
+ &_nis_state, __nis_pw_u_map);
+ }
+ else
+ { /* keep old semantic if no stayopen set - no need to call _nis_start() here - _nis_pwscan() will do it for us ... */
+ /* use same way as in getgrent.c ... */
+ memset(&state, 0, sizeof(state));
+ rv = _nis_pwscan(retval, pw, buffer, buflen,
+ &state, __nis_pw_u_map);
+ _nis_end(&state);
+ }
if (rv != NS_SUCCESS)
return rv;
if (uid == pw->pw_uid) {
@@ -1557,7 +1571,7 @@
snprintf(_nis_passwdbuf, sizeof(_nis_passwdbuf), "%s", name);
rv = _nis_pwscan(&rerror, &_nis_passwd,
_nis_passwdbuf, sizeof(_nis_passwdbuf),
- &_nis_state, PASSWD_BYNAME(&_nis_state));
+ &_nis_state, __nis_pw_n_map);
if (!_nis_state.stayopen)
_nis_end(&_nis_state);
if (rv == NS_SUCCESS && strcmp(name, _nis_passwd.pw_name) == 0)
@@ -1586,13 +1600,20 @@
*result = NULL;
snprintf(buffer, buflen, "%s", name);
- memset(&state, 0, sizeof(state));
- rv = _nis_start(&state);
- if (rv != NS_SUCCESS)
- return rv;
- rv = _nis_pwscan(retval, pw, buffer, buflen,
- &state, PASSWD_BYNAME(&state));
- _nis_end(&state);
+/* remark: we run under a global mutex inside of this module ... */
+ if (_nis_state.stayopen)
+ { /* use global state only if stayopen is set - otherwiese we would blow up getpwent_r() ... */
+ rv = _nis_pwscan(retval, pw, buffer, buflen,
+ &_nis_state, __nis_pw_n_map);
+ }
+ else
+ { /* keep old semantic if no stayopen set - no need to call _nis_start() here - _nis_pwscan() will do it for us ... */
+ /* use same way as in getgrent.c ... */
+ memset(&state, 0, sizeof(state));
+ rv = _nis_pwscan(retval, pw, buffer, buflen,
+ &state, __nis_pw_n_map);
+ _nis_end(&state);
+ }
if (rv != NS_SUCCESS)
return rv;
if (strcmp(name, pw->pw_name) == 0) {
@@ -2048,7 +2069,7 @@
state->mode = COMPAT_FULL;
/* reset passwd_compat search */
/* XXXREENTRANT: setpassent is not thread safe ? */
- (void) _passwdcompat_setpassent(0);
+ (void) _passwdcompat_setpassent(_compat_state.stayopen);
break;
case '@': /* `+@netgroup' */
state->mode = COMPAT_NETGROUP;
--------------040207020604010308030301--
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home