NetBSD Problem Report #52501
From www@NetBSD.org Wed Aug 23 08:59:05 2017
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (verified OK))
by mollari.NetBSD.org (Postfix) with ESMTPS id 807017A28C
for <gnats-bugs@gnats.NetBSD.org>; Wed, 23 Aug 2017 08:59:05 +0000 (UTC)
Message-Id: <20170823085903.DEB567A28F@mollari.NetBSD.org>
Date: Wed, 23 Aug 2017 08:59:03 +0000 (UTC)
From: coypu@sdf.org
Reply-To: coypu@sdf.org
To: gnats-bugs@NetBSD.org
Subject: erlite quickly fails to allocate memory and processes wedge
X-Send-Pr-Version: www-1.0
>Number: 52501
>Category: port-evbmips
>Synopsis: erlite quickly fails to allocate memory and processes wedge
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: port-evbmips-maintainer
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Wed Aug 23 09:00:00 +0000 2017
>Closed-Date: Mon Sep 11 07:27:51 +0000 2017
>Last-Modified: Sat Apr 14 11:30:03 +0000 2018
>Originator: coypu
>Release: NetBSD 8.99.1
>Organization:
>Environment:
NetBSD localhost 8.99.1 NetBSD 8.99.1 (ERLITE.MP) #10: Wed Aug 23 09:38:46 IDT 2017 fly@loggy:/home/fly/mips64eb/sys/arch/evbmips/compile/ERLITE.MP evbmips
>Description:
Unpacking sets, I get processes that are stuck in needbuf (needbuffers_cv):
Looking at vmstat -m, noteworthy line:
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
buf16k 16384 572 1454 166 103 1 102 102 1 1 0
Full output:
# vmstat -m
pid 979(sh): ABI set to N64 (e_flags=0x20000007)
Memory resource pool statistics
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
amappl 80 414 0 44 4 0 4 4 0 inf 0
anonpl 32 1739 0 364 7 0 7 7 0 inf 0
biopl 288 299 0 270 10 0 10 10 0 inf 8
buf16k 16384 572 1454 166 103 1 102 102 1 1 0
buf2k 2048 1796 0 289 450 60 390 448 1 1 0
buf4k 4096 16 0 6 9 3 6 8 1 1 0
buf64k 65536 4 0 0 5 0 5 5 1 1 1
buf8k 8192 9 0 2 10 2 8 10 1 1 1
bufpl 288 2189 0 19 78 0 78 78 0 inf 0
cwdi 64 24 0 3 1 0 1 1 0 inf 0
dwc2qh 176 5 0 0 1 0 1 1 0 inf 0
dwc2qtd 56 2 0 1 1 0 1 1 0 inf 0
dwc2xfer 352 12 0 1 1 0 1 1 0 inf 0
execargs 262144 2010 0 2010 3 2 1 2 0 16 1
fdfile 64 78 0 33 1 0 1 1 0 inf 0
ffsdino2 256 12765 0 0 412 0 412 412 0 inf 0
ffsino 256 12765 0 0 412 0 412 412 0 inf 0
file 128 76 0 7 2 0 2 2 0 inf 0
filedesc 800 24 0 3 3 0 3 3 0 inf 0
icmp 24 68 0 68 1 1 0 1 0 inf 0
icmp6 24 80 0 80 2 1 1 1 0 inf 1
in6pcbpl 288 16 0 15 1 0 1 1 0 inf 0
inmltpl 48 2 0 0 1 0 1 1 0 inf 0
inpcbpl 240 38 0 36 1 0 1 1 0 inf 0
kcpuset 32 83 0 8 1 0 1 1 0 inf 0
kcredpl 160 39 0 5 1 0 1 1 0 inf 0
kmem-1024 1024 122 0 7 15 0 15 15 0 inf 0
kmem-112 112 40 0 3 1 0 1 1 0 inf 0
kmem-128 128 43 0 5 1 0 1 1 0 inf 0
kmem-16 16 6 0 2 1 0 1 1 0 inf 0
kmem-160 160 93 0 10 2 0 2 2 0 inf 0
kmem-192 192 107 0 0 3 0 3 3 0 inf 0
kmem-2048 2048 67 0 5 16 0 16 16 0 inf 0
kmem-224 224 11 0 2 1 0 1 1 0 inf 0
kmem-24 24 494 0 48 2 0 2 2 0 inf 0
kmem-256 256 8 0 0 1 0 1 1 0 inf 0
kmem-32 32 390 0 36 2 0 2 2 0 inf 0
kmem-320 320 106 0 2 5 0 5 5 0 inf 0
kmem-384 384 31 0 10 2 0 2 2 0 inf 0
kmem-40 40 87 0 8 1 0 1 1 0 inf 0
kmem-4096 4096 22 0 2 10 0 10 10 0 inf 0
kmem-448 448 48 0 11 3 0 3 3 0 inf 0
kmem-48 48 240 0 9 2 0 2 2 0 inf 0
kmem-512 512 18 0 6 1 0 1 1 0 inf 0
kmem-56 56 4061 0 6 28 0 28 28 0 inf 0
kmem-64 64 87 0 8 1 0 1 1 0 inf 0
kmem-768 768 101 0 7 10 0 10 10 0 inf 0
kmem-80 80 99 0 3 1 0 1 1 0 inf 0
kmem-8192 8192 8 0 1 8 0 8 8 0 inf 1
kmem-96 96 79 0 5 1 0 1 1 0 inf 0
ksiginfo 72 3 0 2 1 0 1 1 0 inf 0
kva-16384 16384 2 0 0 1 0 1 1 0 inf 0
kva-24576 24576 1 0 0 1 0 1 1 0 inf 0
kva-32768 32768 2 0 0 1 0 1 1 0 inf 0
kva-40960 40960 1 0 0 1 0 1 1 0 inf 0
kva-57344 57344 1 0 0 1 0 1 1 0 inf 0
kva-65536 65536 142 0 0 18 0 18 18 0 inf 0
kva-98304 98304 1 0 0 1 0 1 1 0 inf 0
lockf 112 9 0 0 1 0 1 1 0 inf 0
lwppl 1088 52 0 0 8 0 8 8 0 inf 0
mbpl 512 23 0 0 3 0 3 3 1 inf 1
mutex 32 13506 0 0 54 0 54 54 0 inf 0
ncache 192 14304 0 0 341 0 341 341 0 inf 0
pcache 896 85 0 4 9 0 9 9 0 inf 0
pcglarge 1024 16 0 12 2 0 2 2 0 inf 1
pcgnormal 256 201 0 83 5 0 5 5 0 inf 1
pdict16 72 60 0 44 1 0 1 1 0 inf 0
pdict32 88 4 0 0 1 0 1 1 0 inf 0
phpool-0 56 676 0 66 5 0 5 5 0 inf 0
phpool-256 80 33 0 0 1 0 1 1 0 inf 0
phpool-512 112 3 0 0 1 0 1 1 0 inf 0
phpool-64 56 80 0 2 1 0 1 1 0 inf 0
piperd 320 8 0 0 1 0 1 1 0 inf 0
pipewr 320 7 0 0 1 0 1 1 0 inf 0
plimitpl 240 10 0 0 1 0 1 1 0 inf 0
pmappl 840 2405 0 2387 3 0 3 3 0 inf 0
pnbufpl 1024 9 0 0 2 0 2 2 0 inf 0
procpl 744 22 0 0 2 0 2 2 0 inf 0
proparay 48 25 0 0 1 0 1 1 0 inf 0
propdata 40 3 0 0 1 0 1 1 0 inf 0
propdict 48 87 0 24 1 0 1 1 0 inf 0
propnmbr 56 13 0 0 1 0 1 1 0 inf 0
propstng 40 147 0 44 1 0 1 1 0 inf 0
pstatspl 448 22 0 0 2 0 2 2 0 inf 0
ptimerpl 264 1 0 0 1 0 1 1 0 inf 0
ptimerspl 304 1 0 0 1 0 1 1 0 inf 0
pvpl 24 365160 0 361570 13 0 13 13 1 inf 1
ractx 32 432 0 0 2 0 2 2 0 inf 0
rndctx 24 1 0 0 1 0 1 1 0 inf 0
rndsample 536 6 0 0 1 0 1 1 0 274 0
rndtemp 512 1 0 0 1 0 1 1 0 inf 0
rtentpl 312 28 0 2 1 0 1 1 0 inf 0
rwlock 32 4 0 0 1 0 1 1 0 inf 0
scxspl 256 33926 0 33926 2 0 2 2 2 inf 2
sigacts 3096 21 0 0 11 0 11 11 0 inf 0
socket 584 21 0 0 2 0 2 2 0 inf 0
tcpcbpl 832 20 0 20 1 0 1 1 0 inf 1
tmpfs_node 216 1 0 0 1 0 1 1 0 inf 0
tstilepl 96 52 0 0 1 0 1 1 0 inf 0
uarea 8192 22 0 0 22 0 22 22 0 inf 0
uareasys 8192 31 0 0 31 0 31 31 0 inf 0
ufsdir 264 1 0 0 1 0 1 1 0 inf 0
vcachepl 336 12776 0 0 533 0 533 533 0 inf 0
vmembt 56 13 0 0 1 0 1 1 0 inf 0
vmmpepl 144 1062 0 0 19 0 19 19 0 inf 0
vmsppl 376 21 0 0 1 0 1 1 0 inf 0
wapbldealloc 32 237 0 237 1 0 1 1 0 inf 1
wapblentrypl 40 320 0 320 1 0 1 1 0 inf 1
wapblinopl 32 25135 0 25134 1 0 1 1 0 inf 0
Totals 512749 1454 427503 2765 70 2695
In use 26155K, total allocated 30496K; utilization 85.8%
dmesg:
Loaded initial symtab at 0xffffffff805db138, strtab at 0xffffffff80625d90, # entries 12738
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
NetBSD 8.99.1 (ERLITE.MP) #10: Wed Aug 23 09:38:46 IDT 2017
fly@loggy:/home/fly/mips64eb/sys/arch/evbmips/compile/ERLITE.MP
Cavium Octeon CN50XX
total memory = 512 MB
avail memory = 498 MB
timecounter: Timecounters tick every 10.000 msec
mainbus0 (root)
cpunode0 at mainbus0: 2 cores, crypto+kasumi, 64bit-mul, unaligned-access ok, booted 0
cpu0 at cpunode0 core 0: 500.00MHz (hz cycles = 5000000, delay divisor = 500)
cpu0: Cavium CN50xx (0xd0601) Rev. 1 with software emulated floating point
cpu0: 64 TLB entries, 512TB (49-bit) VAs, 512TB (49-bit) PAs, 256MB max page size
cpu0: 32KB/128B 4-way set-associative L1 instruction cache
cpu0: 16KB/128B 64-way set-associative write-through coherent L1 data cache
cpu0: 128KB/128B 8-way set-associative write-back L2 unified cache
cpu1 at cpunode0 core 1 disabled (unresponsive)
wdog0 at cpunode0: default period is 4 seconds
iobus0 at mainbus0
iobus0: initializing POW
iobus0: initializing FPA
com0 at iobus0 address 0x0001180000000800: ns16650, no ERS, working fifo
com0: console
com at iobus0 address 0x0001180000000c00 not configured
octeon_rnm0 at iobus0 address 0x0001180040000000
octeon_rnm0: random number generator enabled: 1hz
octeon_twsi at iobus0 address 0x0001180000001000 not configured
octeon_mpi at iobus0 address 0x0001070000001000 not configured
octeon_gmx0 at iobus0 address 0x0001180008000000
cnmac0 at octeon_gmx0: address=0x0001180008000000: RGMII
cnmac0: Ethernet address dc:9f:db:16:22:4d
atphy0 at cnmac0 phy 7: Atheros AR8035 10/100/1000 PHY, rev. 2
atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseSX-FDX, 1000baseT-FDX, auto
cnmac1 at octeon_gmx0: address=0x0001180008000000: RGMII
cnmac1: Ethernet address dc:9f:db:16:22:4e
atphy1 at cnmac1 phy 6: Atheros AR8035 10/100/1000 PHY, rev. 2
atphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseSX-FDX, 1000baseT-FDX, auto
cnmac2 at octeon_gmx0: address=0x0001180008000000: RGMII
cnmac2: Ethernet address dc:9f:db:16:22:4f
atphy2 at cnmac2 phy 5: Atheros AR8035 10/100/1000 PHY, rev. 2
atphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseSX-FDX, 1000baseT-FDX, auto
dwctwo0 at iobus0 address 0x0001180068000000
usb0 at dwctwo0: USB revision 2.0
bootbus0 at mainbus0
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
uhub0 at usb0: vendor 0000 (0000) DWC2 root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub0: 1 port with 1 removable, self powered
umass0 at uhub0 port 1 configuration 1 interface 0
umass0: vendor 13fe (0x13fe) USB DISK 2.0 (0x4100), rev 2.00/1.00, addr 2
umass0: using SCSI over Bulk-Only
scsibus0 at umass0: 2 targets, 1 lun per target
uhub0: illegal enable change, port 1
sd0 at scsibus0 target 0 lun 0: <, USB DISK 2.0, PMAP> disk removable
sd0: 3824 MB, 959 cyl, 255 head, 32 sec, 512 bytes/sect x 7831552 sectors
WARNING: 1 error while detecting hardware; check system log.
boot device: sd0
root on sd0a dumps on sd0b
mountroot: trying ffs...
root file system type: ffs
kern.module.path=/stand/evbmips/8.99.1/modules
WARNING: no TOD clock present
WARNING: using filesystem time
WARNING: CHECK AND RESET THE DATE!
init: copying out path `/sbin/init' 11
Backtrace of stuck process:
load: 1.01 cmd: tar 550 [needbuf] 1.51u 59.90s 0% 1840k
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
4908 1 3 0 80 980000041c5711e0 gzip pipe_wr
550 1 3 0 0 980000041f9f11a0 tar needbuf
..
db{0}> bt/a 980000041f9f11a0
pid 550.1 at 0x9800000410976000
0x9800000410977740: kernel_text+d8 (0,0,0,0) ra ffffffff803e614c sz 0
0x9800000410977740: mi_switch+1ec (0,0,0,0) ra ffffffff803e2840 sz 96
0x98000004109777a0: sleepq_block+140 (0,0,0,0) ra ffffffff803ac0e0 sz 64
0x98000004109777e0: cv_timedwait+158 (0,0,0,0) ra ffffffff8045c624 sz 80
0x9800000410977830: allocbuf+354 (0,0,0,0) ra ffffffff8045cdcc sz 128
0x98000004109778b0: getblk+1cc (0,0,0,0) ra ffffffff80363f30 sz 96
0x9800000410977910: ffs_getblk+48 (0,0,0,0) ra ffffffff8035fb44 sz 80
0x9800000410977960: ffs_balloc+1384 (0,0,0,980000041ff21ae0) ra ffffffff803756d8 sz 320
0x9800000410977aa0: ufs_gop_alloc+c0 (0,0,0,980000041ff21ae0) ra ffffffff8036cf6c sz 96
0x9800000410977b00: ufs_balloc_range+144 (0,0,0,980000041ff21ae0) ra ffffffff80368aec sz 160
0x9800000410977ba0: ffs_write+5fc (0,0,0,980000041ff21ae0) ra ffffffff80483aac sz 192
0x9800000410977c60: VOP_WRITE+44 (980000000e2f9660,9800000410977d00,10,980000041ff21ae0) ra ffffffff8047a93c sz 80
0x9800000410977cb0: vn_write+144 (980000000e2f9660,9800000410977d00,10,980000041ff21ae0) ra ffffffff80420da0 sz 80
0x9800000410977d00: dofilewrite+b0 (9800000410977d30,9800000410977d00,50540,400) ra ffffffff8025a10c sz 128
0x9800000410977d80: netbsd32_write+2c (9800000410977d30,9800000410977d00,50540,8) ra ffffffff80248a14 sz 48
0x9800000410977db0: syscall+114 (9800000410977d30,9800000410977d00,50540,784f9a74) ra ffffffff80239254 sz 240
0x9800000410977ea0: mips64r2_systemcall+d4 (9800000410977d30,9800000410977d00,50540,784f9a74) ra 784f9a74 sz 0
PC 0x784f9a74: not in kernel space
>How-To-Repeat:
>Fix:
>Release-Note:
>Audit-Trail:
From: coypu@sdf.org
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: port-evbmips/52501: erlite quickly fails to allocate memory and
processes wedge
Date: Wed, 23 Aug 2017 09:47:03 +0000
Important absent info, from another boot:
ELF file is 64 bit
Allocating memory for ELF segment: addr: 0xffffffff80200000 (adjusted to: 0x200000), size 0x4a1310
Allocated memory for ELF segment: addr: 0xffffffff80200000, size 0x4a1310
Processing PHDR 1
Loading 46a010 bytes at ffffffff80200000
Clearing 37300 bytes at ffffffff8066a010
## Loading Linux kernel with entry point: 0xffffffff80200000 ...
Bootloader: Done loading app on coremask: 0x3
phys segment: 0x10000000 @ 0
adding 0x1fe000 @ 0x2000 to freelist 3
adding 0xf95e000 @ 0x6a2000 to freelist 3
phys segment: 0x10000000 @ 0x410000000
adding 0x10000000 @ 0x410000000 to freelist 0
Loaded initial symtab at 0xffffffff805daa28, strtab at 0xffffffff80625638, # entries 12735
From: coypu@sdf.org
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: port-evbmips/52501: erlite quickly fails to allocate memory and
processes wedge
Date: Wed, 23 Aug 2017 21:56:19 +0000
Deleting this definition (suggested by nick) allows me to use all
memory.
this definition make uvm_km_alloc try to allocate only from one freelist
(0) while I have two.
--- sys/arch/mips/include/vmparam.h 2017-08-24 00:50:16.988961601 +0300
+++ /home/fly/vmparam.h 2017-08-24 00:50:06.125860018 +0300
@@ -208,8 +208,6 @@
#endif
#ifdef _KERNEL
-#define UVM_KM_VMFREELIST mips_poolpage_vmfreelist
-extern int mips_poolpage_vmfreelist;
#ifdef ENABLE_MIPS_16KB_PAGE
extern vaddr_t mips_vm_maxuser_address;
#endif
From: Nick Hudson <skrll@netbsd.org>
To: gnats-bugs@NetBSD.org, port-evbmips-maintainer@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, coypu@sdf.org
Cc:
Subject: Re: port-evbmips/52501: erlite quickly fails to allocate memory and
processes wedge
Date: Thu, 24 Aug 2017 21:48:54 +0100
This is a multi-part message in MIME format.
--------------9794A98B3DAA9AD1C6CAE85F
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
I guess this helps with being able to use all free memory as well?
If so this change is preferable because it at least tries to allocate
pages (for DMA) from the best range, but falls back if there aren't any.
Nick
--------------9794A98B3DAA9AD1C6CAE85F
Content-Type: text/x-patch;
name="uvm_km.c.diff"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="uvm_km.c.diff"
Index: sys/uvm/uvm_km.c
===================================================================
RCS file: /cvsroot/src/sys/uvm/uvm_km.c,v
retrieving revision 1.143
diff -u -p -r1.143 uvm_km.c
--- sys/uvm/uvm_km.c 1 Jun 2017 02:45:15 -0000 1.143
+++ sys/uvm/uvm_km.c 24 Aug 2017 20:45:18 -0000
@@ -663,7 +663,7 @@ uvm_km_alloc(struct vm_map *map, vsize_t
pg = uvm_pagealloc_strat(NULL, offset, NULL, pgaflags,
#ifdef UVM_KM_VMFREELIST
- UVM_PGA_STRAT_ONLY, UVM_KM_VMFREELIST
+ UVM_PGA_STRAT_FALLBACK, UVM_KM_VMFREELIST
#else
UVM_PGA_STRAT_NORMAL, 0
#endif
--------------9794A98B3DAA9AD1C6CAE85F--
From: coypu@sdf.org
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: port-evbmips/52501: erlite quickly fails to allocate memory and
processes wedge
Date: Mon, 28 Aug 2017 17:33:44 +0000
This works - I can use all RAM.
State-Changed-From-To: open->pending-pullups
State-Changed-By: skrll@NetBSD.org
State-Changed-When: Thu, 07 Sep 2017 07:00:49 +0000
State-Changed-Why:
[pullup-8 #267]
From: "Soren Jacobsen" <snj@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/52501 CVS commit: [netbsd-8] src/sys/arch/mips/include
Date: Mon, 11 Sep 2017 05:20:26 +0000
Module Name: src
Committed By: snj
Date: Mon Sep 11 05:20:26 UTC 2017
Modified Files:
src/sys/arch/mips/include [netbsd-8]: vmparam.h
Log Message:
Pull up following revision(s) (requested by skrll in ticket #267):
sys/arch/mips/include/vmparam.h: revision 1.60
Don't define UVM_KM_VMFREELIST on mips as it excludes some memory
ranges unnecessarily.
PR/52501 - erlite quickly fails to allocate memory and processes wedge
To generate a diff of this commit:
cvs rdiff -u -r1.57.8.1 -r1.57.8.2 src/sys/arch/mips/include/vmparam.h
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: pending-pullups->feedback
State-Changed-By: skrll@NetBSD.org
State-Changed-When: Mon, 11 Sep 2017 06:56:22 +0000
State-Changed-Why:
ok to close?
State-Changed-From-To: feedback->closed
State-Changed-By: maya@NetBSD.org
State-Changed-When: Mon, 11 Sep 2017 07:27:51 +0000
State-Changed-Why:
fixed & pulled up, thanks.
From: coypu@sdf.org
To: gnats-bugs@NetBSD.org
Cc: skrll@netbsd.org
Subject: Re: port-evbmips/52501 (erlite quickly fails to allocate memory and
processes wedge)
Date: Mon, 11 Sep 2017 07:27:04 +0000
Sure, but I still like your original change better :)
From: "Nick Hudson" <skrll@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/52501 CVS commit: src/sys/arch/mips/include
Date: Thu, 7 Sep 2017 06:36:24 +0000
Module Name: src
Committed By: skrll
Date: Thu Sep 7 06:36:24 UTC 2017
Modified Files:
src/sys/arch/mips/include: vmparam.h
Log Message:
Don't define UVM_KM_VMFREELIST on mips as it excludes some memory
ranges unnecessarily.
PR/52501 - erlite quickly fails to allocate memory and processes wedge
To generate a diff of this commit:
cvs rdiff -u -r1.59 -r1.60 src/sys/arch/mips/include/vmparam.h
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.43 2018/01/16 07:36:43 maya Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2017
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home