NetBSD Problem Report #50602
From mlelstv@tazz.1st.de Wed Dec 30 15:22:06 2015
Return-Path: <mlelstv@tazz.1st.de>
Received: from mail.netbsd.org (mail.NetBSD.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
by mollari.NetBSD.org (Postfix) with ESMTPS id 562BC7ACC8
for <gnats-bugs@gnats.NetBSD.org>; Wed, 30 Dec 2015 15:22:06 +0000 (UTC)
Message-Id: <20151230135115.B4AF9269FA@tazz.1st.de>
Date: Wed, 30 Dec 2015 14:51:15 +0100 (CET)
From: mlelstv@serpens.de
Reply-To: mlelstv@serpens.de
To: gnats-bugs@NetBSD.org
Subject: removing ethernet cable panics the kernel
X-Send-Pr-Version: 3.95
>Number: 50602
>Category: kern
>Synopsis: removing ethernet cable panics the kernel
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: kern-bug-people
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Wed Dec 30 15:25:00 +0000 2015
>Closed-Date: Wed Apr 06 05:46:17 +0000 2016
>Last-Modified: Wed Apr 06 05:46:17 +0000 2016
>Originator: Michael van Elst
>Release: NetBSD 7.99.25
>Organization:
>Environment:
System: NetBSD tazz 7.99.25 NetBSD 7.99.25 (TAZZ) #1: Wed Dec 30 13:05:01 CET 2015 mlelstv@gossam:/home/netbsd-current/obj.amd64/home/netbsd-current/src/sys/arch/amd64/compile/TAZZ amd64
Architecture: x86_64
Machine: amd64
>Description:
The recent changes to the arp/llinfo code causes a kernel panic
because llinfo code is protected by rwlocks but can be called
from interrupt.
Unplugging the ethernet cable (on a wm device) calls in6_if_link_down()
in the driver interrupt which calls rt_newaddrmsg() which ends in
nd6_rtrequest()
...
IF_AFDATA_RLOCK(ifp);
ln = lla_lookup(LLTABLE6(ifp), flags, rt_getkey(rt));
IF_AFDATA_RUNLOCK(ifp);
which panics with a lockdebug message that rw_enter is called
from interrupt.
>How-To-Repeat:
Boot -current, unplug ethernet cable.
>Fix:
>Release-Note:
>Audit-Trail:
From: Michael van Elst <mlelstv@serpens.de>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/50602: removing ethernet cable panics the kernel
Date: Sun, 3 Jan 2016 09:41:52 +0100
Here are the functions from the backtrace that lead to the panic:
rw_enter
nd6_rtrequest
rtrequest1
rtrequest
rtrequest_newmsg
nd6_prefix_offlink
pfxlist_onlink_check
in6_if_link_down
if_link_state_change
mii_phy_update
inphy_service
mii_pollstat
wm_linkintr
wm_intr_legacy
From: Michael van Elst <mlelstv@serpens.de>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/50602: removing ethernet cable panics the kernel
Date: Sun, 3 Jan 2016 15:01:54 +0100
Maybe this.
The patch
- defers link state handling to a softint
- fixes the destruction of the afdata lock
- makes the lock a _KERNEL only field to avoid
conflicts with kvm(3) users.
Index: sys/net/if.c
===================================================================
RCS file: /cvsroot/src/sys/net/if.c,v
retrieving revision 1.319
diff -p -u -r1.319 if.c
--- sys/net/if.c 20 Nov 2015 08:10:36 -0000 1.319
+++ sys/net/if.c 3 Jan 2016 13:55:38 -0000
@@ -198,6 +198,7 @@ static void if_attachdomain1(struct ifne
static int ifconf(u_long, void *);
static int if_clone_create(const char *);
static int if_clone_destroy(const char *);
+static void if_link_state_change_si(void *);
#if defined(INET) || defined(INET6)
static void sysctl_net_pktq_setup(struct sysctllog **, int);
@@ -592,6 +593,7 @@ if_initialize(ifnet_t *ifp)
ifp->if_broadcastaddr = 0; /* reliably crash if used uninitialized */
ifp->if_link_state = LINK_STATE_UNKNOWN;
+ ifp->if_old_link_state = LINK_STATE_UNKNOWN;
ifp->if_capenable = 0;
ifp->if_csum_flags_tx = 0;
@@ -617,6 +619,10 @@ if_initialize(ifnet_t *ifp)
IF_AFDATA_LOCK_INIT(ifp);
+ ifp->if_link_si = softint_establish(SOFTINT_NET, if_link_state_change_si, ifp);
+ if (ifp->if_link_si == NULL)
+ panic("%s: softint_establish() failed", __func__);
+
if_getindex(ifp);
}
@@ -895,6 +901,11 @@ again:
ifioctl_detach(ifp);
+ IF_AFDATA_LOCK_DESTROY(ifp);
+
+ softint_disestablish(ifp->if_link_si);
+ ifp->if_link_si = NULL;
+
/*
* remove packets that came from ifp, from software interrupt queues.
*/
@@ -1407,8 +1418,6 @@ void
if_link_state_change(struct ifnet *ifp, int link_state)
{
int s;
- int old_link_state;
- struct domain *dp;
s = splnet();
if (ifp->if_link_state == link_state) {
@@ -1416,8 +1425,26 @@ if_link_state_change(struct ifnet *ifp,
return;
}
- old_link_state = ifp->if_link_state;
ifp->if_link_state = link_state;
+ softint_schedule(ifp->if_link_si);
+
+ splx(s);
+}
+
+
+void
+if_link_state_change_si(void *arg)
+{
+ struct ifnet *ifp = arg;
+ int s;
+ int link_state, old_link_state;
+ struct domain *dp;
+
+ s = splnet();
+ link_state = ifp->if_link_state;
+ old_link_state = ifp->if_old_link_state;
+ ifp->if_old_link_state = ifp->if_link_state;
+
#ifdef DEBUG
log(LOG_DEBUG, "%s: link state %s (was %s)\n", ifp->if_xname,
link_state == LINK_STATE_UP ? "UP" :
Index: sys/net/if.h
===================================================================
RCS file: /cvsroot/src/sys/net/if.h,v
retrieving revision 1.193
diff -p -u -r1.193 if.h
--- sys/net/if.h 2 Oct 2015 03:08:26 -0000 1.193
+++ sys/net/if.h 3 Jan 2016 13:55:39 -0000
@@ -351,12 +351,14 @@ typedef struct ifnet {
struct ifnet_lock *if_ioctl_lock;
#ifdef _KERNEL /* XXX kvm(3) */
struct callout *if_slowtimo_ch;
-#endif
#ifdef GATEWAY
struct kmutex *if_afdata_lock;
#else
struct krwlock *if_afdata_lock;
#endif
+ void *if_link_si; /* softint to handle link state changes */
+ int if_old_link_state; /* previous link state */
+#endif
} ifnet_t;
#define if_mtu if_data.ifi_mtu
@@ -452,6 +454,8 @@ typedef struct ifnet {
(ifp)->if_afdata_lock = mutex_obj_alloc(MUTEX_DEFAULT, IPL_NET); \
} while (0)
+#define IF_AFDATA_LOCK_DESTROY(ifp) mutex_obj_free((ifp)->if_afdata_lock)
+
#define IF_AFDATA_WLOCK(ifp) mutex_enter((ifp)->if_afdata_lock)
#define IF_AFDATA_RLOCK(ifp) mutex_enter((ifp)->if_afdata_lock)
#define IF_AFDATA_WUNLOCK(ifp) mutex_exit((ifp)->if_afdata_lock)
@@ -459,7 +463,6 @@ typedef struct ifnet {
#define IF_AFDATA_LOCK(ifp) IF_AFDATA_WLOCK(ifp)
#define IF_AFDATA_UNLOCK(ifp) IF_AFDATA_WUNLOCK(ifp)
#define IF_AFDATA_TRYLOCK(ifp) mutex_tryenter((ifp)->if_afdata_lock)
-#define IF_AFDATA_DESTROY(ifp) mutex_destroy((ifp)->if_afdata_lock)
#define IF_AFDATA_LOCK_ASSERT(ifp) \
KASSERT(mutex_owned((ifp)->if_afdata_lock))
@@ -474,6 +477,8 @@ typedef struct ifnet {
#define IF_AFDATA_LOCK_INIT(ifp) \
do {(ifp)->if_afdata_lock = rw_obj_alloc();} while (0)
+#define IF_AFDATA_LOCK_DESTROY(ifp) rw_obj_free((ifp)->if_afdata_lock)
+
#define IF_AFDATA_WLOCK(ifp) rw_enter((ifp)->if_afdata_lock, RW_WRITER)
#define IF_AFDATA_RLOCK(ifp) rw_enter((ifp)->if_afdata_lock, RW_READER)
#define IF_AFDATA_WUNLOCK(ifp) rw_exit((ifp)->if_afdata_lock)
@@ -481,7 +486,6 @@ typedef struct ifnet {
#define IF_AFDATA_LOCK(ifp) IF_AFDATA_WLOCK(ifp)
#define IF_AFDATA_UNLOCK(ifp) IF_AFDATA_WUNLOCK(ifp)
#define IF_AFDATA_TRYLOCK(ifp) rw_tryenter((ifp)->if_afdata_lock, RW_WRITER)
-#define IF_AFDATA_DESTROY(ifp) rw_destroy((ifp)->if_afdata_lock)
#define IF_AFDATA_LOCK_ASSERT(ifp) \
KASSERT(rw_lock_held((ifp)->if_afdata_lock))
From: Ryota Ozaki <ozaki-r@netbsd.org>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
Michael van Elst <mlelstv@serpens.de>
Subject: Re: kern/50602: removing ethernet cable panics the kernel
Date: Mon, 4 Jan 2016 12:47:58 +0900
On Sun, Jan 3, 2016 at 11:05 PM, Michael van Elst <mlelstv@serpens.de> wrote:
> The following reply was made to PR kern/50602; it has been noted by GNATS.
>
> From: Michael van Elst <mlelstv@serpens.de>
> To: gnats-bugs@netbsd.org
> Cc:
> Subject: Re: kern/50602: removing ethernet cable panics the kernel
> Date: Sun, 3 Jan 2016 15:01:54 +0100
>
> Maybe this.
>
> The patch
>
> - defers link state handling to a softint
I concur the change.
> - fixes the destruction of the afdata lock
Heh... I'll commit this separately.
> - makes the lock a _KERNEL only field to avoid
> conflicts with kvm(3) users.
I remember that riastradh suggested me that we don't need the
ifdef for kvm(3). IIUC, the added variables aren't accessed
by kvm(3) so that we don't need to hide them to userland.
I'd remove it if nobody objects.
ozaki-r
>
> Index: sys/net/if.c
> ===================================================================
> RCS file: /cvsroot/src/sys/net/if.c,v
> retrieving revision 1.319
> diff -p -u -r1.319 if.c
> --- sys/net/if.c 20 Nov 2015 08:10:36 -0000 1.319
> +++ sys/net/if.c 3 Jan 2016 13:55:38 -0000
> @@ -198,6 +198,7 @@ static void if_attachdomain1(struct ifne
> static int ifconf(u_long, void *);
> static int if_clone_create(const char *);
> static int if_clone_destroy(const char *);
> +static void if_link_state_change_si(void *);
>
> #if defined(INET) || defined(INET6)
> static void sysctl_net_pktq_setup(struct sysctllog **, int);
> @@ -592,6 +593,7 @@ if_initialize(ifnet_t *ifp)
> ifp->if_broadcastaddr = 0; /* reliably crash if used uninitialized */
>
> ifp->if_link_state = LINK_STATE_UNKNOWN;
> + ifp->if_old_link_state = LINK_STATE_UNKNOWN;
>
> ifp->if_capenable = 0;
> ifp->if_csum_flags_tx = 0;
> @@ -617,6 +619,10 @@ if_initialize(ifnet_t *ifp)
>
> IF_AFDATA_LOCK_INIT(ifp);
>
> + ifp->if_link_si = softint_establish(SOFTINT_NET, if_link_state_change_si, ifp);
> + if (ifp->if_link_si == NULL)
> + panic("%s: softint_establish() failed", __func__);
> +
> if_getindex(ifp);
> }
>
> @@ -895,6 +901,11 @@ again:
>
> ifioctl_detach(ifp);
>
> + IF_AFDATA_LOCK_DESTROY(ifp);
> +
> + softint_disestablish(ifp->if_link_si);
> + ifp->if_link_si = NULL;
> +
> /*
> * remove packets that came from ifp, from software interrupt queues.
> */
> @@ -1407,8 +1418,6 @@ void
> if_link_state_change(struct ifnet *ifp, int link_state)
> {
> int s;
> - int old_link_state;
> - struct domain *dp;
>
> s = splnet();
> if (ifp->if_link_state == link_state) {
> @@ -1416,8 +1425,26 @@ if_link_state_change(struct ifnet *ifp,
> return;
> }
>
> - old_link_state = ifp->if_link_state;
> ifp->if_link_state = link_state;
> + softint_schedule(ifp->if_link_si);
> +
> + splx(s);
> +}
> +
> +
> +void
> +if_link_state_change_si(void *arg)
> +{
> + struct ifnet *ifp = arg;
> + int s;
> + int link_state, old_link_state;
> + struct domain *dp;
> +
> + s = splnet();
> + link_state = ifp->if_link_state;
> + old_link_state = ifp->if_old_link_state;
> + ifp->if_old_link_state = ifp->if_link_state;
> +
> #ifdef DEBUG
> log(LOG_DEBUG, "%s: link state %s (was %s)\n", ifp->if_xname,
> link_state == LINK_STATE_UP ? "UP" :
> Index: sys/net/if.h
> ===================================================================
> RCS file: /cvsroot/src/sys/net/if.h,v
> retrieving revision 1.193
> diff -p -u -r1.193 if.h
> --- sys/net/if.h 2 Oct 2015 03:08:26 -0000 1.193
> +++ sys/net/if.h 3 Jan 2016 13:55:39 -0000
> @@ -351,12 +351,14 @@ typedef struct ifnet {
> struct ifnet_lock *if_ioctl_lock;
> #ifdef _KERNEL /* XXX kvm(3) */
> struct callout *if_slowtimo_ch;
> -#endif
> #ifdef GATEWAY
> struct kmutex *if_afdata_lock;
> #else
> struct krwlock *if_afdata_lock;
> #endif
> + void *if_link_si; /* softint to handle link state changes */
> + int if_old_link_state; /* previous link state */
> +#endif
> } ifnet_t;
>
> #define if_mtu if_data.ifi_mtu
> @@ -452,6 +454,8 @@ typedef struct ifnet {
> (ifp)->if_afdata_lock = mutex_obj_alloc(MUTEX_DEFAULT, IPL_NET); \
> } while (0)
>
> +#define IF_AFDATA_LOCK_DESTROY(ifp) mutex_obj_free((ifp)->if_afdata_lock)
> +
> #define IF_AFDATA_WLOCK(ifp) mutex_enter((ifp)->if_afdata_lock)
> #define IF_AFDATA_RLOCK(ifp) mutex_enter((ifp)->if_afdata_lock)
> #define IF_AFDATA_WUNLOCK(ifp) mutex_exit((ifp)->if_afdata_lock)
> @@ -459,7 +463,6 @@ typedef struct ifnet {
> #define IF_AFDATA_LOCK(ifp) IF_AFDATA_WLOCK(ifp)
> #define IF_AFDATA_UNLOCK(ifp) IF_AFDATA_WUNLOCK(ifp)
> #define IF_AFDATA_TRYLOCK(ifp) mutex_tryenter((ifp)->if_afdata_lock)
> -#define IF_AFDATA_DESTROY(ifp) mutex_destroy((ifp)->if_afdata_lock)
>
> #define IF_AFDATA_LOCK_ASSERT(ifp) \
> KASSERT(mutex_owned((ifp)->if_afdata_lock))
> @@ -474,6 +477,8 @@ typedef struct ifnet {
> #define IF_AFDATA_LOCK_INIT(ifp) \
> do {(ifp)->if_afdata_lock = rw_obj_alloc();} while (0)
>
> +#define IF_AFDATA_LOCK_DESTROY(ifp) rw_obj_free((ifp)->if_afdata_lock)
> +
> #define IF_AFDATA_WLOCK(ifp) rw_enter((ifp)->if_afdata_lock, RW_WRITER)
> #define IF_AFDATA_RLOCK(ifp) rw_enter((ifp)->if_afdata_lock, RW_READER)
> #define IF_AFDATA_WUNLOCK(ifp) rw_exit((ifp)->if_afdata_lock)
> @@ -481,7 +486,6 @@ typedef struct ifnet {
> #define IF_AFDATA_LOCK(ifp) IF_AFDATA_WLOCK(ifp)
> #define IF_AFDATA_UNLOCK(ifp) IF_AFDATA_WUNLOCK(ifp)
> #define IF_AFDATA_TRYLOCK(ifp) rw_tryenter((ifp)->if_afdata_lock, RW_WRITER)
> -#define IF_AFDATA_DESTROY(ifp) rw_destroy((ifp)->if_afdata_lock)
>
> #define IF_AFDATA_LOCK_ASSERT(ifp) \
> KASSERT(rw_lock_held((ifp)->if_afdata_lock))
>
>
From: mlelstv@serpens.de (Michael van Elst)
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/50602: removing ethernet cable panics the kernel
Date: Mon, 4 Jan 2016 07:09:51 +0000 (UTC)
ozaki-r@netbsd.org (Ryota Ozaki) writes:
>I remember that riastradh suggested me that we don't need the
>ifdef for kvm(3). IIUC, the added variables aren't accessed
>by kvm(3) so that we don't need to hide them to userland.
>I'd remove it if nobody objects.
The point is that the structure to userland must not change
without adding appropriate compatibility code.
But since the added variables are only accessed by the kernel
the simple solution is to put them inside #ifdef _KERNEL
as it was already done for the if_slowtimo_ch variable.
--
--
Michael van Elst
Internet: mlelstv@serpens.de
"A potential Snark may lurk in every tree."
From: Ryota Ozaki <ozaki-r@netbsd.org>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
Michael van Elst <mlelstv@serpens.de>
Subject: Re: kern/50602: removing ethernet cable panics the kernel
Date: Mon, 4 Jan 2016 18:37:21 +0900
On Mon, Jan 4, 2016 at 4:15 PM, Michael van Elst <mlelstv@serpens.de> wrote:
> The following reply was made to PR kern/50602; it has been noted by GNATS.
>
> From: mlelstv@serpens.de (Michael van Elst)
> To: gnats-bugs@netbsd.org
> Cc:
> Subject: Re: kern/50602: removing ethernet cable panics the kernel
> Date: Mon, 4 Jan 2016 07:09:51 +0000 (UTC)
>
> ozaki-r@netbsd.org (Ryota Ozaki) writes:
>
> >I remember that riastradh suggested me that we don't need the
> >ifdef for kvm(3). IIUC, the added variables aren't accessed
> >by kvm(3) so that we don't need to hide them to userland.
>
> >I'd remove it if nobody objects.
>
> The point is that the structure to userland must not change
> without adding appropriate compatibility code.
>
> But since the added variables are only accessed by the kernel
> the simple solution is to put them inside #ifdef _KERNEL
> as it was already done for the if_slowtimo_ch variable.
Hm. My understanding is that we can keep compatibility by
appending new variables to a data structure while inserting
variables to middle of a data structure or removing variables
breaks compatibility. Basically struct ifnet is accessed from
userland via struct ifreq and one exception is kvm(3) that is
used by netstat(1) and vmstat(1). The former guarantees
compatibility on changing ifnet and the compatibility for
the latter is guaranteed by always appending new variables
to ifnet when we change ifnet.
Correct me if I'm wrong (probably I am!).
ozaki-r
>
> --
> --
> Michael van Elst
> Internet: mlelstv@serpens.de
> "A potential Snark may lurk in every tree."
>
From: Ryota Ozaki <ozaki-r@netbsd.org>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
Michael van Elst <mlelstv@serpens.de>
Subject: Re: kern/50602: removing ethernet cable panics the kernel
Date: Wed, 10 Feb 2016 13:24:14 +0900
So I've revisited this change.
I rebased mlelstv's patch:
http://www.netbsd.org/~ozaki-r/softint-if_link_state_change.diff
It works for me. I tested plugging/unplugging a cable repeatedly
under load or no load. I didn't see any panic and inconsistency
of the link state.
I think we can commit this change. Any thoughts?
ozaki-r
On Mon, Jan 4, 2016 at 6:37 PM, Ryota Ozaki <ozaki-r@netbsd.org> wrote:
> On Mon, Jan 4, 2016 at 4:15 PM, Michael van Elst <mlelstv@serpens.de> wrote:
>> The following reply was made to PR kern/50602; it has been noted by GNATS.
>>
>> From: mlelstv@serpens.de (Michael van Elst)
>> To: gnats-bugs@netbsd.org
>> Cc:
>> Subject: Re: kern/50602: removing ethernet cable panics the kernel
>> Date: Mon, 4 Jan 2016 07:09:51 +0000 (UTC)
>>
>> ozaki-r@netbsd.org (Ryota Ozaki) writes:
>>
>> >I remember that riastradh suggested me that we don't need the
>> >ifdef for kvm(3). IIUC, the added variables aren't accessed
>> >by kvm(3) so that we don't need to hide them to userland.
>>
>> >I'd remove it if nobody objects.
>>
>> The point is that the structure to userland must not change
>> without adding appropriate compatibility code.
>>
>> But since the added variables are only accessed by the kernel
>> the simple solution is to put them inside #ifdef _KERNEL
>> as it was already done for the if_slowtimo_ch variable.
>
> Hm. My understanding is that we can keep compatibility by
> appending new variables to a data structure while inserting
> variables to middle of a data structure or removing variables
> breaks compatibility. Basically struct ifnet is accessed from
> userland via struct ifreq and one exception is kvm(3) that is
> used by netstat(1) and vmstat(1). The former guarantees
> compatibility on changing ifnet and the compatibility for
> the latter is guaranteed by always appending new variables
> to ifnet when we change ifnet.
>
> Correct me if I'm wrong (probably I am!).
>
> ozaki-r
>
>>
>> --
>> --
>> Michael van Elst
>> Internet: mlelstv@serpens.de
>> "A potential Snark may lurk in every tree."
>>
From: "Ryota Ozaki" <ozaki-r@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/50602 CVS commit: src/sys/net
Date: Mon, 15 Feb 2016 08:08:05 +0000
Module Name: src
Committed By: ozaki-r
Date: Mon Feb 15 08:08:04 UTC 2016
Modified Files:
src/sys/net: if.c if.h
Log Message:
Run if_link_state_change in softint
if_link_state_change can execute the network stack that is expected to
not run in hardware interrupt (at least now), however network drivers
may call it in hardware interrupt. Avoid that by introducing a new
softint for if_link_state_change.
The original patch is provided by mlelstv@ and tweaked a bit by me.
Should fix PR kern/50602.
To generate a diff of this commit:
cvs rdiff -u -r1.323 -r1.324 src/sys/net/if.c
cvs rdiff -u -r1.195 -r1.196 src/sys/net/if.h
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->feedback
State-Changed-By: ozaki-r@NetBSD.org
State-Changed-When: Tue, 22 Mar 2016 08:19:53 +0000
State-Changed-Why:
Please close if the problem is fixed.
State-Changed-From-To: feedback->closed
State-Changed-By: mlelstv@NetBSD.org
State-Changed-When: Wed, 06 Apr 2016 05:46:17 +0000
State-Changed-Why:
problem is fixed.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home