NetBSD Problem Report #53059
From kardel@pip.kardel.name Tue Feb 27 11:47:39 2018
Return-Path: <kardel@pip.kardel.name>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id CAACC7A1B3
for <gnats-bugs@gnats.NetBSD.org>; Tue, 27 Feb 2018 11:47:39 +0000 (UTC)
Message-Id: <20180227114732.38682DA0D8B@pip.kardel.name>
Date: Tue, 27 Feb 2018 12:47:32 +0100 (CET)
From: kardel@netbsd.org
Reply-To: kardel@netbsd.org
To: gnats-bugs@NetBSD.org
Subject: "nvmectl identify nvme0" locks up system
X-Send-Pr-Version: 3.95
>Number: 53059
>Category: kern
>Synopsis: repeated nvmectl identify nvme0 locks up sytem
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: jdolecek
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Tue Feb 27 11:50:00 +0000 2018
>Closed-Date: Sun Mar 18 12:29:47 +0000 2018
>Last-Modified: Sun Mar 18 12:29:47 +0000 2018
>Originator: Frank Kardel
>Release: NetBSD 8.99.12-20180224180000
>Organization:
>Environment:
System: NetBSD pip.kardel.name 8.99.12 NetBSD 8.99.12 (PIPGEN) #10: Sat Feb 24 19:26:44 CET 2018 kardel@xxx:/src/NetBSD/act/src/obj.amd64/sys/arch/amd64/compile/PIPGEN amd64
Architecture: x86_64
Machine: amd64
>Description:
repeated invocation of "nvmectl identify nvme0" (about 45-50 times) will
lock up the system hard. for the more relaxed people - smartd takes about
22:30h to lock up the system.
If you are lucky yousee the panic: nvme0: invalid ccb detected triggerted
via softint_dispatch()->nvme_poll().
needless to say: breaking into DDB via USB attached keyboards does not work here.
>How-To-Repeat:
csh(as root): repeat 64 nvmectl identify nvme0
>Fix:
no fix - check for admin q depletion - it is close to the limit of admin q entries.
>Release-Note:
>Audit-Trail:
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/53059: "nvmectl identify nvme0" locks up system
Date: Tue, 27 Feb 2018 14:35:32 +0100
I can reproduce it but luckily have a serial console.
This is with Christos's latest change:
nvme_poll() at netbsd:nvme_poll+0x12d
nvmeioctl() at netbsd:nvmeioctl+0x19a
cdev_ioctl() at netbsd:cdev_ioctl+0x98
VOP_IOCTL() at netbsd:VOP_IOCTL+0x3b
vn_ioctl() at netbsd:vn_ioctl+0xa1
sys_ioctl() at netbsd:sys_ioctl+0x103
syscall() at netbsd:syscall+0x1d8
db{0}> sh reg
ds 2b73
es 8a78
fs 55a6
gs 6
rdi ffffffff81401260 x86_io
rsi 3f8
rbp ffff800144b78a68
rbx ffff80001da4505c
rdx 1
rcx 8
rax 7f
r8 ffff800143841000
r9 ffffffff806441cb nvme_pt_fill
r10 ffff800144b78ba0
r11 10
r12 800
r13 f9
r14 c6
r15 ffffe4011daa0408
rip ffffffff8021db05 breakpoint+0x5
cs 8
rflags 202
rsp ffff800144b78a68
ss 10
netbsd:breakpoint+0x5: leave
(gdb) list *nvmeioctl+0x19a
0xffffffff80644e65 is in nvmeioctl (../../../../dev/ic/nvme.c:971).
[.. nvme_command_passthrough ..]
(gdb) list *nvme_poll+0x12d
0xffffffff80643fb2 is in nvme_poll (../../../../dev/ic/nvme.c:1043).
1038 ccb->ccb_cookie = &state;
1039
1040 nvme_q_submit(sc, q, ccb, nvme_poll_fill);
1041 while (!ISSET(state.c.flags, htole16(NVME_CQE_PHASE))) {
1042 if (nvme_q_complete(sc, q) == 0)
1043 delay(step);
1044
Martin
From: Frank Kardel <kardel@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/53059: "nvmectl identify nvme0" locks up system
Date: Tue, 27 Feb 2018 14:52:32 +0100
That identifies the tight loop - good.
Now why can we run ~45 nvmectl's successfully from user level and then
get stuck - is the admin queue exhausted?
Frank
On 02/27/18 14:40, Martin Husemann wrote:
> The following reply was made to PR kern/53059; it has been noted by GNATS.
>
> From: Martin Husemann <martin@duskware.de>
> To: gnats-bugs@NetBSD.org
> Cc:
> Subject: Re: kern/53059: "nvmectl identify nvme0" locks up system
> Date: Tue, 27 Feb 2018 14:35:32 +0100
>
> I can reproduce it but luckily have a serial console.
>
> This is with Christos's latest change:
>
> nvme_poll() at netbsd:nvme_poll+0x12d
> nvmeioctl() at netbsd:nvmeioctl+0x19a
> cdev_ioctl() at netbsd:cdev_ioctl+0x98
> VOP_IOCTL() at netbsd:VOP_IOCTL+0x3b
> vn_ioctl() at netbsd:vn_ioctl+0xa1
> sys_ioctl() at netbsd:sys_ioctl+0x103
> syscall() at netbsd:syscall+0x1d8
>
> db{0}> sh reg
> ds 2b73
> es 8a78
> fs 55a6
> gs 6
> rdi ffffffff81401260 x86_io
> rsi 3f8
> rbp ffff800144b78a68
> rbx ffff80001da4505c
> rdx 1
> rcx 8
> rax 7f
> r8 ffff800143841000
> r9 ffffffff806441cb nvme_pt_fill
> r10 ffff800144b78ba0
> r11 10
> r12 800
> r13 f9
> r14 c6
> r15 ffffe4011daa0408
> rip ffffffff8021db05 breakpoint+0x5
> cs 8
> rflags 202
> rsp ffff800144b78a68
> ss 10
> netbsd:breakpoint+0x5: leave
>
>
> (gdb) list *nvmeioctl+0x19a
> 0xffffffff80644e65 is in nvmeioctl (../../../../dev/ic/nvme.c:971).
> [.. nvme_command_passthrough ..]
>
> (gdb) list *nvme_poll+0x12d
> 0xffffffff80643fb2 is in nvme_poll (../../../../dev/ic/nvme.c:1043).
> 1038 ccb->ccb_cookie = &state;
> 1039
> 1040 nvme_q_submit(sc, q, ccb, nvme_poll_fill);
> 1041 while (!ISSET(state.c.flags, htole16(NVME_CQE_PHASE))) {
> 1042 if (nvme_q_complete(sc, q) == 0)
> 1043 delay(step);
> 1044
>
>
> Martin
>
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/53059: "nvmectl identify nvme0" locks up system
Date: Tue, 27 Feb 2018 16:10:49 +0100
I called nvme_dumpregs after it hangs:
nvme0: cap 0x00f000203c013fff
nvme0: mpsmax 27 (134217728)
nvme0: mpsmin 12 (4096)
nvme0: css 1
nvme0: nssrs 0
nvme0: dstrd 4
nvme0: to 30000 msec
nvme0: ams 0
nvme0: cqr 65536
nvme0: mqes 16384
nvme0: vs 0x10100
nvme0: cc 0x460001
nvme0: iocqes 4 (16)
nvme0: iosqes 6 (64)
nvme0: shn 0
nvme0: ams 0
nvme0: mps 12 (4096)
nvme0: css 0
nvme0: en 1
nvme0: csts 0x00000001
nvme0: rdy 1
nvme0: cfs 0
nvme0: shst 0
nvme0: aqa 0x001f001f
nvme0: acqs 31
nvme0: asqs 31
nvme0: asq 0x000000011d722000
nvme0: acq 0x000000011d72a000
Martin
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/53059: "nvmectl identify nvme0" locks up system
Date: Tue, 27 Feb 2018 16:31:48 +0100
On Tue, Feb 27, 2018 at 03:15:01PM +0000, Martin Husemann wrote:
> I called nvme_dumpregs after it hangs:
>
> nvme0: cap 0x00f000203c013fff
> nvme0: mpsmax 27 (134217728)
I also have a core dump, but unfortunately gdb can't backtrace through
the trapframe currently, it seems:
#8 0xffffffff8021db05 in breakpoint ()
#9 0xffffffff805d1a4e in comintr (arg=0xffffe4011daa0408)
at ../../../../dev/ic/com.c:2113
#10 0xffffffff8020aeb6 in handle_ioapic_edge6 ()
#11 0x0000000000000000 in ?? ()
Martin
Responsible-Changed-From-To: kern-bug-people->jdolecek
Responsible-Changed-By: jdolecek@NetBSD.org
Responsible-Changed-When: Thu, 15 Mar 2018 10:56:06 +0000
Responsible-Changed-Why:
I'm looking into this.
State-Changed-From-To: open->feedback
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Thu, 15 Mar 2018 10:56:06 +0000
State-Changed-Why:
Possible fix was sent to PR kern/52769, can you try it out? It identified
ccb leak on command errors, which I believe is the case here.
Patch: http://www.netbsd.org/~jdolecek/nvme_avail_put.diff
State-Changed-From-To: feedback->analyzed
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Fri, 16 Mar 2018 19:05:37 +0000
State-Changed-Why:
That patch is unlikely to help here. I'm going to refactor the code
to avoid polled command for identify, which should fix the issue.
From: Frank Kardel <kardel@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/53059 (repeated nvmectl identify nvme0 locks up sytem)
Date: Fri, 16 Mar 2018 20:10:19 +0100
It seems to be with all ioctl triggered admin-q interactions that are
processed via polled command. E. g, smartd will lead to a system lockup
after 22:30h.
Thanks for looking into it.
On 03/16/18 20:05, jdolecek@NetBSD.org wrote:
> Synopsis: repeated nvmectl identify nvme0 locks up sytem
>
> State-Changed-From-To: feedback->analyzed
> State-Changed-By: jdolecek@NetBSD.org
> State-Changed-When: Fri, 16 Mar 2018 19:05:37 +0000
> State-Changed-Why:
> That patch is unlikely to help here. I'm going to refactor the code
> to avoid polled command for identify, which should fix the issue.
>
>
>
State-Changed-From-To: analyzed->feedback
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Sat, 17 Mar 2018 00:29:17 +0000
State-Changed-Why:
Committed possible fix on HEAD, rev 1.35 of src/sys/dev/ic/nvme.c
Can you please check if it fixes your problem?
From: "Jaromir Dolecek" <jdolecek@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/53059 CVS commit: src/sys/dev/ic
Date: Sat, 17 Mar 2018 00:28:03 +0000
Module Name: src
Committed By: jdolecek
Date: Sat Mar 17 00:28:03 UTC 2018
Modified Files:
src/sys/dev/ic: nvme.c
Log Message:
switch handling of passthrough commands to use queue, instead of polling
should fix PR kern/53059 by Frank Kardel
To generate a diff of this commit:
cvs rdiff -u -r1.34 -r1.35 src/sys/dev/ic/nvme.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Frank Kardel <kardel@netbsd.org>
To: gnats-bugs@NetBSD.org, jdolecek@NetBSD.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org
Cc:
Subject: Re: PR/53059 CVS commit: src/sys/dev/ic
Date: Sat, 17 Mar 2018 09:10:53 +0100
Well, we are not quite there yet.
I get a page fault trap, code=0.
hand copied stack trace (no dump device available at this point, photo
available on request).
nvme_pt_fill() at nvme_pt_fill+0x1e
nvme_poll() at nvme_poll+0xa0
nvme_attach() atnvme_attch+0x9a4
nvme_pci_attch() at name_pci_attach+0x530
config_attach_loc()
config_found_sm_loc()
pci_probe_device()
pci_enumerate_bus()
pcirescan()
pciattach()
....
Frank
On 03/17/18 01:30, Jaromir Dolecek wrote:
> The following reply was made to PR kern/53059; it has been noted by GNATS.
>
> From: "Jaromir Dolecek" <jdolecek@netbsd.org>
> To: gnats-bugs@gnats.NetBSD.org
> Cc:
> Subject: PR/53059 CVS commit: src/sys/dev/ic
> Date: Sat, 17 Mar 2018 00:28:03 +0000
>
> Module Name: src
> Committed By: jdolecek
> Date: Sat Mar 17 00:28:03 UTC 2018
>
> Modified Files:
> src/sys/dev/ic: nvme.c
>
> Log Message:
> switch handling of passthrough commands to use queue, instead of polling
>
> should fix PR kern/53059 by Frank Kardel
>
>
> To generate a diff of this commit:
> cvs rdiff -u -r1.34 -r1.35 src/sys/dev/ic/nvme.c
>
> Please note that diffs are not public domain; they are subject to the
> copyright notices on the relevant files.
>
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: jdolecek@NetBSD.org, netbsd-bugs@netbsd.org, gnats-admin@netbsd.org,
kardel@netbsd.org
Subject: Re: kern/53059 (repeated nvmectl identify nvme0 locks up sytem)
Date: Sat, 17 Mar 2018 09:11:00 +0100
I have just done 128
nvmectl identify nvme0
in a row and the system still is happy.
Martin
From: "Jaromir Dolecek" <jdolecek@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/53059 CVS commit: src/sys/dev/ic
Date: Sat, 17 Mar 2018 09:36:32 +0000
Module Name: src
Committed By: jdolecek
Date: Sat Mar 17 09:36:32 UTC 2018
Modified Files:
src/sys/dev/ic: nvme.c
Log Message:
fix passthrough command usage also in nvme_get_number_of_queues(), fixes
memory corruption and possible panic on boot
PR kern/53059
To generate a diff of this commit:
cvs rdiff -u -r1.35 -r1.36 src/sys/dev/ic/nvme.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: =?UTF-8?B?SmFyb23DrXIgRG9sZcSNZWs=?= <jaromir.dolecek@gmail.com>
To: Frank Kardel <kardel@netbsd.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: PR/53059 CVS commit: src/sys/dev/ic
Date: Sat, 17 Mar 2018 10:38:29 +0100
--94eb2c05f1f6c84ca80567987c43
Content-Type: text/plain; charset="UTF-8"
Try now with rev 1.36 please, should be fixed.
Jaromir
2018-03-17 9:10 GMT+01:00 Frank Kardel <kardel@netbsd.org>:
>
> Well, we are not quite there yet.
>
> I get a page fault trap, code=0.
>
> hand copied stack trace (no dump device available at this point, photo
available on request).
>
> nvme_pt_fill() at nvme_pt_fill+0x1e
>
> nvme_poll() at nvme_poll+0xa0
>
> nvme_attach() atnvme_attch+0x9a4
>
> nvme_pci_attch() at name_pci_attach+0x530
>
> config_attach_loc()
>
> config_found_sm_loc()
>
> pci_probe_device()
>
> pci_enumerate_bus()
>
> pcirescan()
>
> pciattach()
>
> ....
>
> Frank
>
>
>
> On 03/17/18 01:30, Jaromir Dolecek wrote:
>>
>> The following reply was made to PR kern/53059; it has been noted by
GNATS.
>>
>> From: "Jaromir Dolecek" <jdolecek@netbsd.org>
>> To: gnats-bugs@gnats.NetBSD.org
>> Cc:
>> Subject: PR/53059 CVS commit: src/sys/dev/ic
>> Date: Sat, 17 Mar 2018 00:28:03 +0000
>>
>> Module Name: src
>> Committed By: jdolecek
>> Date: Sat Mar 17 00:28:03 UTC 2018
>> Modified Files:
>> src/sys/dev/ic: nvme.c
>> Log Message:
>> switch handling of passthrough commands to use queue, instead of
polling
>> should fix PR kern/53059 by Frank Kardel
>> To generate a diff of this commit:
>> cvs rdiff -u -r1.34 -r1.35 src/sys/dev/ic/nvme.c
>> Please note that diffs are not public domain; they are subject to the
>> copyright notices on the relevant files.
>>
>
>
--94eb2c05f1f6c84ca80567987c43
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Try now with rev 1.36 please, should be fixed.<br><br>Jaro=
mir<br><br>2018-03-17 9:10 GMT+01:00 Frank Kardel <<a href=3D"mailto:kar=
del@netbsd.org">kardel@netbsd.org</a>>:<br>><br>> Well, we are not=
quite there yet.<br>><br>> I get a page fault trap, code=3D0.<br>>=
;<br>> hand copied stack trace (no dump device available at this point, =
photo available on request).<br>><br>> nvme_pt_fill() at nvme_pt_fill=
+0x1e<br>><br>> nvme_poll() at nvme_poll+0xa0<br>><br>> nvme_at=
tach() atnvme_attch+0x9a4<br>><br>> nvme_pci_attch() at name_pci_atta=
ch+0x530<br>><br>> config_attach_loc()<br>><br>> config_found_s=
m_loc()<br>><br>> pci_probe_device()<br>><br>> pci_enumerate_bu=
s()<br>><br>> pcirescan()<br>><br>> pciattach()<br>><br>>=
....<br>><br>> Frank<br>><br>><br>><br>> On 03/17/18 01:=
30, Jaromir Dolecek wrote:<br>>><br>>> The following reply was =
made to PR kern/53059; it has been noted by GNATS.<br>>><br>>> =
From: "Jaromir Dolecek" <<a href=3D"mailto:jdolecek@netbsd.org=
">jdolecek@netbsd.org</a>><br>>> To: <a href=3D"mailto:gnats-bugs@=
gnats.NetBSD.org">gnats-bugs@gnats.NetBSD.org</a><br>>> Cc:<br>>&g=
t; Subject: PR/53059 CVS commit: src/sys/dev/ic<br>>> Date: Sat, 17 M=
ar 2018 00:28:03 +0000<br>>><br>>> =C2=A0 Module Name: =C2=A0sr=
c<br>>> =C2=A0 Committed By: jdolecek<br>>> =C2=A0 Date: =C2=A0=
=C2=A0 =C2=A0 =C2=A0 Sat Mar 17 00:28:03 UTC 2018<br>>> =C2=A0 =C2=
=A0 Modified Files:<br>>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 src/sys/dev/ic:=
nvme.c<br>>> =C2=A0 =C2=A0 Log Message:<br>>> =C2=A0 switch ha=
ndling of passthrough commands to use queue, instead of polling<br>>>=
=C2=A0 =C2=A0 should fix PR kern/53059 by Frank Kardel<br>>> =C2=A0 =
=C2=A0 =C2=A0 To generate a diff of this commit:<br>>> =C2=A0 cvs rdi=
ff -u -r1.34 -r1.35 src/sys/dev/ic/nvme.c<br>>> =C2=A0 =C2=A0 Please =
note that diffs are not public domain; they are subject to the<br>>> =
=C2=A0 copyright notices on the relevant files.<br>>> =C2=A0<br>><=
br>><br></div>
--94eb2c05f1f6c84ca80567987c43--
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: jdolecek@NetBSD.org, netbsd-bugs@netbsd.org, gnats-admin@netbsd.org,
kardel@netbsd.org
Subject: Re: kern/53059 (repeated nvmectl identify nvme0 locks up sytem)
Date: Sat, 17 Mar 2018 11:05:41 +0100
Still works fine for me with the latest version.
Martin
From: Frank Kardel <kardel@netbsd.org>
To: gnats-bugs@NetBSD.org, jdolecek@NetBSD.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org
Cc:
Subject: Re: PR/53059 CVS commit: src/sys/dev/ic
Date: Sat, 17 Mar 2018 18:58:58 +0100
Much better !
it boots, it survives > 1024 identifies...
Great - looks fixed now.
Frank
On 03/17/18 10:40, Jaromir Dolecek wrote:
> The following reply was made to PR kern/53059; it has been noted by GNATS.
>
> From: "Jaromir Dolecek" <jdolecek@netbsd.org>
> To: gnats-bugs@gnats.NetBSD.org
> Cc:
> Subject: PR/53059 CVS commit: src/sys/dev/ic
> Date: Sat, 17 Mar 2018 09:36:32 +0000
>
> Module Name: src
> Committed By: jdolecek
> Date: Sat Mar 17 09:36:32 UTC 2018
>
> Modified Files:
> src/sys/dev/ic: nvme.c
>
> Log Message:
> fix passthrough command usage also in nvme_get_number_of_queues(), fixes
> memory corruption and possible panic on boot
>
> PR kern/53059
>
>
> To generate a diff of this commit:
> cvs rdiff -u -r1.35 -r1.36 src/sys/dev/ic/nvme.c
>
> Please note that diffs are not public domain; they are subject to the
> copyright notices on the relevant files.
>
State-Changed-From-To: feedback->pending-pullups
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Sat, 17 Mar 2018 19:30:23 +0000
State-Changed-Why:
Fix confirmed. Requested pullup to netbsd-8, it's ticket #641.
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/53059 CVS commit: [netbsd-8] src/sys/dev/ic
Date: Sun, 18 Mar 2018 11:05:27 +0000
Module Name: src
Committed By: martin
Date: Sun Mar 18 11:05:27 UTC 2018
Modified Files:
src/sys/dev/ic [netbsd-8]: ld_nvme.c nvme.c nvmevar.h
Log Message:
Pull up following revision(s) (requested by jdolecek in ticket #641):
sys/dev/ic/nvme.c: revision 1.34
sys/dev/ic/nvme.c: revision 1.35
sys/dev/ic/nvme.c: revision 1.36
sys/dev/ic/nvme.c: revision 1.37
sys/dev/ic/ld_nvme.c: revision 1.19
sys/dev/ic/nvmevar.h: revision 1.15
refactor the locking code around DIOCGCACHE handling to be reusable
for other infrequent commands,it uses single condvar for simplicity,
and uses it both when waiting for ccb or command completion - this
is fine, since usually there will be just one such command qeueued anyway
use this to finally properly implement DIOCCACHESYNC - return only after
the command is confirmed as completed by the controller.
switch handling of passthrough commands to use queue, instead of polling
should fix PR kern/53059 by Frank Kardel
fix passthrough command usage also in nvme_get_number_of_queues(), fixes
memory corruption and possible panic on boot
also remove now duplicate nvme_ccb_put() call from
nvme_get_number_of_queues()
To generate a diff of this commit:
cvs rdiff -u -r1.16.2.1 -r1.16.2.2 src/sys/dev/ic/ld_nvme.c
cvs rdiff -u -r1.30.2.1 -r1.30.2.2 src/sys/dev/ic/nvme.c
cvs rdiff -u -r1.13.6.1 -r1.13.6.2 src/sys/dev/ic/nvmevar.h
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: pending-pullups->closed
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Sun, 18 Mar 2018 12:29:47 +0000
State-Changed-Why:
Pullup to netbsd-8 done. Thanks for report and testing!
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.43 2018/01/16 07:36:43 maya Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2017
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home