NetBSD Problem Report #44625
From hash@abox3.so-net.ne.jp Wed Feb 23 04:34:51 2011
Return-Path: <hash@abox3.so-net.ne.jp>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by www.NetBSD.org (Postfix) with ESMTP id 39B5A63B842
for <gnats-bugs@gnats.NetBSD.org>; Wed, 23 Feb 2011 04:34:51 +0000 (UTC)
Message-Id: <201102230434.p1N4Yj9A002564@mx62.ms.so-net.ne.jp>
Date: Wed, 23 Feb 2011 13:34:46 +0900
From: Takahiro HAYASHI <hash@abox3.so-net.ne.jp>
To: gnats-bugs@gnats.NetBSD.org
Subject: memory exhaustion on NFS server while writing many files
>Number: 44625
>Category: kern
>Synopsis: memory exhaustion on NFS server while writing many files
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: dholland
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Wed Feb 23 04:35:00 +0000 2011
>Closed-Date: Mon Aug 08 16:08:54 +0000 2011
>Last-Modified: Mon Aug 08 16:08:54 +0000 2011
>Originator: Takahiro HAYASHI
>Release: NetBSD-current (5.99.40) 2010-11-30 10:50:00 UTC or later
>Organization:
>Environment:
NFS-client:
System: NetBSD halt 5.99.46 NetBSD 5.99.46 (HALT) #0: Tue Feb 22 12:34:31 JST 2011 root@halt:/usr/build2/obj.i386/sys/arch/i386/compile/HALT i386
PhysicalMemory: 4096MB
mount-opts: ruin:/usr/nfs on /usr/nfs type nfs (fsid: 0xb02/0x70b, reads: sync 0 async 0, writes: sync 0 async 0, [nfs: addr=fc00::6c, port=2049, addrlen=28, sotype=1, proto=0, fhsize=0, flags=0x8240<intr,nfsv3,resvport>, wsize=32768, rsize=32768, readdirsize=8192, timeo=300, retrans=10, maxgrouplist=16, readahead=2, leaseterm=0, deadthresh=9])
NFS-server:
System: NetBSD ruin 5.99.46 NetBSD 5.99.46 (MONOLITHIC) #0: Tue Feb 22 12:15:45 JST 2011 root@halt:/usr/build2/obj.i386/sys/arch/i386/compile/MONOLITHIC i386
PhysicalMemory: 512MB
>Description:
The utilization of kernel memory resource pool on NFS
server shown at last line from "vmstat -m" increases up to
most of available memory while writing many files via NFS.
The memory utilization does not decrease even if
copy is stopped and copied files are removed.
mbuf does not seem to leak as far as i see netstat -m.
At last, console of NFS server does not respond,
however, the server responds to ping.
This does not happen when writing a single huge file.
The exported filesystem on server is FFSv2 with WAPBL.
The following is diff between outputs from vmstat -m
on NFS server at a certain time and 22 minutes later.
--- 0 2011-02-22 13:25:45.000000000 +0900
+++ 3 2011-02-22 13:47:13.000000000 +0900
@@ -1,18 +1,18 @@
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
- 16 175 81 33796 1280 0
- 32 735 161 2119895 640 0
- 64 988 36 6963 320 0
- 128 760 8 4969 160 0
- 256 47 17 7607 80 0
+ 16 175 81 71073 1280 0
+ 32 736 160 8101764 640 0
+ 64 988 36 11138 320 0
+ 128 761 7 8265 160 0
+ 256 47 17 14669 80 0
512 34 6 45 40 0
1024 115 1 385 20 0
2048 32 2 1798 10 0
- 4096 154 3 577 5 0
+ 4096 154 3 721 5 0
8192 4 3 9 5 0
16384 3 0 4 5 0
- 32768 1 0 1 5 0
- 65536 11 0 11 5 0
+ 32768 0 0 1 5 0
+ 65536 12 0 12 5 0
131072 1 0 1 5 0
Memory usage type by bucket size
@@ -38,12 +38,12 @@
8192 wapbl, vmem, sysctlnode, temp
16384 vmem, sysctlnode, temp
32768 vmem
- 65536 devbuf
+ 65536 vmem, devbuf
131072 blist
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
- wapbl 5 14K 14K 78644K 674 0 0 32:0,256:1,512:2,4096:1,8192:1
+ wapbl 5 14K 14K 78644K 1522 0 0 32:0,256:1,512:2,4096:1,8192:1
NTFS mount 0 0K 1K 78644K 1 0 0 256:0
MSDOSFS mount 0 0K 1K 78644K 1 0 0 256:0
prop dictionary 67 9K 9K 78644K 84 0 0 128:67
@@ -53,11 +53,11 @@
VM map 2 1K 1K 78644K 2 0 0 512:2
USB 80 11K 11K 78644K 90 0 0 16:10,32:10,64:34,128:15,256:10,1024:0,4096:1
USB device 30 31K 31K 78644K 30 0 0 16:10,128:4,256:1,2048:15
- soname 3 1K 1K 78644K 111 0 0 16:1,32:1,128:1
- packet tags 1 1K 4K 78644K 2109809 0 0 32:1
- sockaddr 62 2K 2K 78644K 201 0 0 16:9,32:53
+ soname 3 1K 1K 78644K 186 0 0 16:1,32:1,128:1
+ packet tags 0 0K 4K 78644K 8084258 0 0 32:0
+ sockaddr 64 2K 2K 78644K 376 0 0 16:9,32:55
tcpcongctl 2 1K 1K 78644K 2 0 0 32:2
- vmem 6 35K 51K 78644K 12 0 0 16:1,64:1,512:0,2048:3,4096:0,8192:0,16384:0,32768:1
+ vmem 6 59K 87K 78644K 13 0 0 16:1,64:1,512:0,2048:3,4096:0,8192:0,16384:0,32768:0,65536:1
blist 2 73K 73K 78644K 2 0 0 32:1,131072:1
NETSMBDEV 1 1K 1K 78644K 1 0 0 16:1
RAIDframe 10 35K 35K 78644K 10 0 0 32:1,2048:1,4096:8
@@ -66,9 +66,9 @@
sysctldata 97 11K 11K 78644K 138 0 0 16:48,64:23,128:16,256:8,512:0,1024:0,2048:2
devbuf 121 750K 750K 78644K 129 0 0 16:13,32:19,64:5,128:29,256:11,512:25,1024:2,4096:6,65536:11
DMA map 305 524K 524K 78644K 305 0 0 64:176,256:1,4096:128
- pcb 9 1K 2K 78644K 128 0 0 16:2,32:0,128:7
- temp 5 38K 62K 78644K 58079 0 0 16:0,32:0,64:0,128:0,256:1,512:0,1024:1,2048:0,4096:1,8192:0,16384:2
- routetbl 37 4K 4K 78644K 69 0 0 16:8,32:6,128:22,256:1
+ pcb 9 1K 2K 78644K 221 0 0 16:2,32:0,128:7
+ temp 5 38K 62K 78644K 114574 0 0 16:0,32:0,64:0,128:0,256:1,512:0,1024:1,2048:0,4096:1,8192:0,16384:2
+ routetbl 38 4K 4K 78644K 70 0 0 16:8,32:6,128:23,256:1
UFS mount 3 7K 7K 78644K 3 0 0 256:1,2048:1,4096:1
Export Host 2 1K 1K 78644K 2 0 0 128:2
ip_moptions 1 1K 1K 78644K 1 0 0 16:1
@@ -87,115 +87,116 @@
acpi 1976 140K 142K 78644K 7594 0 0 16:44,32:616,64:735,128:578,256:2,512:1,1024:0
Memory totals: In Use Free Requests
- 1887K 58K 2177748
+ 1911K 58K 8209885
Memory resource pool statistics
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
-amappl 48 252 0 28 3 0 3 3 0 inf 0
-anonpl 16 2839 0 102 11 0 11 11 0 inf 0
-ataspl 56 126338 0 126338 1 0 1 1 0 inf 1
-biopl 176 67 0 33 2 0 2 2 0 inf 0
-buf16k 16384 1616 0 884 259 65 194 206 1 1 1
+amappl 48 309 0 28 4 0 4 4 0 inf 0
+anonpl 16 3046 0 102 12 0 12 12 0 inf 0
+ataspl 56 268759 0 268759 1 0 1 1 0 inf 1
+biopl 176 70 0 33 2 0 2 2 0 inf 0
+buf16k 16384 2927 0 2448 309 150 159 234 1 1 0
buf1k 1024 2 0 2 1 0 1 1 1 1 1
-buf2k 2048 10797 2 2953 253 7 246 246 1 1 0
-buf4k 4096 67 0 45 2 0 2 2 1 1 0
+buf2k 2048 20897 2 17308 288 90 198 270 1 1 0
+buf4k 4096 122 0 98 2 0 2 2 1 1 0
buf512b 512 1 0 1 1 0 1 1 1 1 1
-buf8k 8192 32 0 20 4 2 2 3 1 1 0
-bufpl 176 8848 0 238 375 0 375 375 0 inf 0
-cwdi 64 40 0 23 1 0 1 1 0 inf 0
-execargs 262144 141 0 141 2 1 1 1 0 16 1
+buf8k 8192 44 0 39 5 3 2 3 1 1 1
+bufpl 176 9709 0 238 412 0 412 412 0 inf 0
+cwdi 64 64 0 23 1 0 1 1 0 inf 0
+execargs 262144 221 0 221 2 1 1 1 0 16 1
extent 20 133 0 119 1 0 1 1 0 inf 0
-fdfile 64 189 0 88 3 0 3 3 0 inf 0
-ffsdino2 256 32721 0 14 2045 0 2045 2045 0 inf 0
-ffsino 172 32709 0 5 1422 0 1422 1422 0 inf 0
-file 64 97 0 9 2 0 2 2 0 inf 0
-filedesc 704 38 0 24 8 0 8 8 0 inf 2
-fstrans 20 25 0 3 1 0 1 1 0 inf 0
-in6pcbpl 144 42 0 23 1 0 1 1 0 inf 0
+fdfile 64 287 0 88 4 0 4 4 0 inf 0
+ffsdino2 256 32746 0 14 2046 0 2046 2046 0 inf 0
+ffsino 172 32735 0 5 1424 0 1424 1424 0 inf 0
+file 64 111 0 9 2 0 2 2 0 inf 0
+filedesc 704 64 0 24 8 0 8 8 0 inf 0
+fstrans 20 50 0 3 1 0 1 1 0 inf 0
+in6pcbpl 144 48 0 29 1 0 1 1 0 inf 0
inmltpl 32 2 0 0 1 0 1 1 0 inf 0
-inpcbpl 112 38 0 23 1 0 1 1 0 inf 0
-kcredpl 192 87 0 0 5 0 5 5 0 inf 0
-kmem-100 100 102 0 0 21 0 21 21 0 inf 0
-kmem-1024 1024 77 0 0 77 0 77 77 0 inf 0
-kmem-12 12 196 0 0 5 0 5 5 0 inf 0
-kmem-128 128 44 0 0 11 0 11 11 0 inf 0
-kmem-1280 1280 23 0 0 23 0 23 23 0 inf 0
-kmem-1536 1536 4 0 0 4 0 4 4 0 inf 0
-kmem-16 16 137 0 0 5 0 5 5 0 inf 0
+inpcbpl 112 50 0 35 1 0 1 1 0 inf 0
+kcredpl 192 111 0 17 5 0 5 5 0 inf 0
+kmem-100 100 129 0 9 24 0 24 24 0 inf 0
+kmem-1024 1024 116 0 40 77 0 77 77 0 inf 1
+kmem-12 12 301 0 28 7 0 7 7 0 inf 0
+kmem-128 128 56 0 6 13 0 13 13 0 inf 0
+kmem-1280 1280 31 0 7 24 0 24 24 0 inf 0
+kmem-1536 1536 16 0 0 16 0 16 16 0 inf 0
+kmem-16 16 163 0 22 5 0 5 5 0 inf 0
kmem-1792 1792 2 0 0 2 0 2 2 0 inf 0
-kmem-20 20 169107 0 0 6765 0 6765 6765 0 inf 0
+kmem-20 20 387780 0 2 15512 0 15512 15512 0 inf 0
kmem-2048 2048 5 0 0 5 0 5 5 0 inf 0
-kmem-2304 2304 6 0 0 6 0 6 6 0 inf 0
-kmem-24 24 193 0 0 10 0 10 10 0 inf 0
+kmem-2304 2304 8 0 2 6 0 6 6 0 inf 0
+kmem-24 24 217 0 17 10 0 10 10 0 inf 0
kmem-2560 2560 4 0 0 4 0 4 4 0 inf 0
-kmem-28 28 20 0 0 2 0 2 2 0 inf 0
+kmem-28 28 29 0 4 2 0 2 2 0 inf 0
kmem-2816 2816 2 0 0 2 0 2 2 0 inf 0
kmem-3072 3072 1 0 0 1 0 1 1 0 inf 0
-kmem-32 32 2112 0 0 132 0 132 132 0 inf 0
+kmem-32 32 2116 0 2072 132 0 132 132 0 inf 124
kmem-3328 3328 1 0 0 1 0 1 1 0 inf 0
kmem-3584 3584 4 0 0 4 0 4 4 0 inf 0
-kmem-36 36 2094 0 0 150 0 150 150 0 inf 0
+kmem-36 36 2104 0 2050 150 0 150 150 0 inf 139
kmem-3840 3840 1 0 0 1 0 1 1 0 inf 0
-kmem-40 40 129 0 0 11 0 11 11 0 inf 0
-kmem-44 44 6 0 0 1 0 1 1 0 inf 0
-kmem-48 48 31 0 0 4 0 4 4 0 inf 0
-kmem-512 512 107 0 0 107 0 107 107 0 inf 0
-kmem-56 56 3 0 0 1 0 1 1 0 inf 0
+kmem-40 40 164 0 41 11 0 11 11 0 inf 0
+kmem-44 44 12 0 0 2 0 2 2 0 inf 0
+kmem-48 48 35 0 1 4 0 4 4 0 inf 0
+kmem-512 512 112 0 24 110 0 110 110 0 inf 22
+kmem-56 56 4 0 1 1 0 1 1 0 inf 0
kmem-64 64 17 0 0 3 0 3 3 0 inf 0
-kmem-72 72 206 0 0 30 0 30 30 0 inf 0
-kmem-768 768 79 0 0 79 0 79 79 0 inf 0
-kmem-8 8 2 0 0 1 0 1 1 0 inf 0
-kmem-84 84 86 0 0 15 0 15 15 0 inf 0
-ksiginfo 48 29 0 0 1 0 1 1 0 inf 0
-kvakernel 4096 48151 0 0 1505 0 1505 1505 0 inf 0
-kvakmem 4096 600 0 0 19 0 19 19 0 inf 0
-lockf 64 5 0 0 1 0 1 1 0 inf 0
-lwppl 672 63 0 0 11 0 11 11 0 inf 0
-mbpl 256 258 0 0 18 0 18 18 1 inf 1
-mclpl 2048 206 0 0 107 0 107 107 4 8030 4
-mutex 64 384 0 0 7 0 7 7 0 inf 0
-ncache 128 25468 0 0 822 0 822 822 0 inf 0
+kmem-72 72 232 0 72 30 0 30 30 0 inf 0
+kmem-768 768 110 0 34 79 0 79 79 0 inf 3
+kmem-8 8 2 0 2 1 0 1 1 0 inf 1
+kmem-84 84 93 0 4 15 0 15 15 0 inf 0
+ksiginfo 48 58 0 29 1 0 1 1 0 inf 0
+kvakernel 4096 102837 0 21 3213 0 3213 3213 0 inf 0
+kvakmem 4096 707 0 6 22 0 22 22 0 inf 0
+lockf 64 5 0 4 1 0 1 1 0 inf 0
+lwppl 672 70 0 9 11 0 11 11 0 inf 0
+mbpl 256 355 0 111 18 1 17 18 1 inf 0
+mclpl 2048 310 0 125 107 4 103 107 4 8030 10
+mutex 64 400 0 31 7 0 7 7 0 inf 0
+ncache 128 26672 0 15446 822 0 822 822 0 inf 62
nfsreqcachepl 56 64 0 0 1 0 1 1 0 inf 0
-nfsrvdescpl 184 1494911 0 1494910 1 0 1 1 0 inf 0
+nfsrvdescpl 184 5997280 0 5997280 1 0 1 1 0 inf 1
pcache 576 86 0 14 11 0 11 11 0 inf 0
pcachecpu 64 81 0 0 2 0 2 2 0 inf 0
-pcglarge 512 43 0 9 5 0 5 5 0 inf 0
-pcgnormal 128 925 0 68 29 0 29 29 0 inf 0
+pcglarge 512 108 0 43 9 0 9 9 0 inf 0
+pcgnormal 128 2391 0 1741 52 14 38 52 0 inf 1
pdict16 44 19 0 4 1 0 1 1 0 inf 0
pdict32 60 9 0 1 1 0 1 1 0 inf 0
-pdppl 4096 45 0 31 37 0 37 37 0 inf 23
-phpool-0 32 44996 0 74 354 0 354 354 0 inf 0
-phpool-64 36 9072 0 1 81 0 81 81 0 inf 0
-piperd 192 7 0 0 1 0 1 1 0 inf 0
-pipewr 192 6 0 0 1 0 1 1 0 inf 0
+pdppl 4096 103 0 87 39 21 18 37 0 inf 2
+phpool-0 32 99755 0 269 784 0 784 784 0 inf 0
+phpool-64 36 19556 0 1 175 0 175 175 0 inf 0
+piperd 192 19 0 12 1 0 1 1 0 inf 0
+pipewr 192 23 0 14 1 0 1 1 0 inf 0
plimitpl 196 3 0 0 1 0 1 1 0 inf 0
-pmappl 96 37 0 0 1 0 1 1 0 inf 0
-pnbufpl 1024 169046 0 0 42262 0 42262 42262 0 inf 0
-procpl 448 21 0 0 3 0 3 3 0 inf 0
+pmappl 96 46 0 27 1 0 1 1 0 inf 0
+pnbufpl 1024 387713 0 2 96928 0 96928 96928 0 inf 0
+procpl 448 28 0 9 3 0 3 3 0 inf 0
propdict 32 85 0 17 1 0 1 1 0 inf 0
propnmbr 32 17 0 4 1 0 1 1 0 inf 0
propstng 20 80 0 8 1 0 1 1 0 inf 0
-pstatspl 268 21 0 0 2 0 2 2 0 inf 0
-ptimerpl 164 5 0 2 1 0 1 1 0 inf 0
-ptimerspl 140 5 0 2 1 0 1 1 0 inf 0
-pvpl 20 3881 0 0 20 0 20 20 0 inf 0
-ractx 24 521 0 0 4 0 4 4 0 inf 0
-rndsample 528 1953 0 1952 1 0 1 1 0 inf 0
-rtentpl 184 29 0 0 2 0 2 2 0 inf 0
-sigacts 2056 36 0 0 36 0 36 36 0 inf 0
-socket 332 57 0 0 5 0 5 5 0 inf 0
-synpl 188 2 0 2 1 0 1 1 0 inf 1
-tcpcbpl 500 20 0 7 2 0 2 2 0 inf 0
-tcpipqepl 44 3061 0 3061 1 0 1 1 0 inf 1
+pstatspl 268 28 0 9 2 0 2 2 0 inf 0
+ptimerpl 164 11 0 8 1 0 1 1 0 inf 0
+ptimerspl 140 11 0 8 1 0 1 1 0 inf 0
+pvpl 20 5202 0 20 26 0 26 26 0 inf 0
+ractx 24 521 0 434 4 0 4 4 0 inf 0
+rndsample 528 4152 0 4151 1 0 1 1 0 inf 0
+rtentpl 184 30 0 0 2 0 2 2 0 inf 0
+sigacts 2056 50 0 29 41 0 41 41 0 inf 20
+socket 332 61 0 20 6 0 6 6 0 inf 1
+synpl 188 5 0 5 1 1 0 1 0 inf 0
+tcpcbpl 500 26 0 13 2 0 2 2 0 inf 0
+tcpipqepl 44 7993 0 7993 1 0 1 1 0 inf 1
tmpfs_dirent_0xcbebc004 140 1 0 0 1 0 1 1 0 inf 0
-tstilepl 52 63 0 0 1 0 1 1 0 inf 0
-uarea 12288 63 0 0 63 0 63 63 0 inf 0
-ufsdir 264 26 0 0 2 0 2 2 0 inf 0
-vmembt 28 9401 0 0 65 0 65 65 0 inf 0
-vmmpepl 84 475 0 0 10 0 10 10 0 inf 0
-vmsppl 212 37 0 0 2 0 2 2 0 inf 0
-vnodepl 172 32713 0 0 1423 0 1423 1423 0 inf 0
-wapblinopl 20 57933 0 57933 1 0 1 1 0 inf 1
+tstilepl 52 68 0 11 1 0 1 1 0 inf 0
+uaoeltpl 84 2 0 0 1 0 1 1 0 inf 0
+uarea 12288 68 0 11 65 0 65 65 0 inf 8
+ufsdir 264 31 0 31 3 0 3 3 0 inf 3
+vmembt 28 19279 0 0 133 0 133 133 0 inf 0
+vmmpepl 84 577 0 0 13 0 13 13 0 inf 0
+vmsppl 212 41 0 0 3 0 3 3 0 inf 0
+vnodepl 172 32731 0 0 1424 0 1424 1424 0 inf 0
+wapblinopl 20 118147 0 118147 1 0 1 1 0 inf 1
-In use 228132K, total allocated 230402K; utilization 99.0%
+In use 438950K, total allocated 450688K; utilization 97.4%
>How-To-Repeat:
1. prepare NFS server and NFS client
2. mount_nfs exported directory on client with "-i,-T"
3. copy NetBSD src tree on NFS client
% rsync -a /local/src /nfs/1/ &
% rsync -a /local/src /nfs/2/ &
% rsync -a /local/src /nfs/3/ &
% rsync -a /local/src /nfs/4/ &
4. watch "vmstat -m -w30" on NFS server
>Fix:
No idea.
--
Takahiro HAYASHI
>Release-Note:
>Audit-Trail:
From: hash@abox3.so-net.ne.jp
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/44625: memory exhaustion on NFS server while writing many files
Date: Tue, 15 Mar 2011 01:53:43 +0900
I posted incorrect data.
The outputs from vmstat are unlikely; The statistics other than
Memory resource pool ones are not gathered in MONOLITHIC kernel.
Sorry for confusing.
I tried to vmstat -mW on MONOLITHIC kernel (see below), however,
output of vmstat is not much different from the one in orinal post.
The Npage of pnbufpl grows extremely.
I also tried some kernels checked out on other date. It seems that
this leakage is seen after struct componentname changes on Nov 30 2010.
regards,
--
Takahiro HAYASHI
Memory resource pool statistics
Name Size Requests Fail Releases InUse Avail Pgreq Pgrel Npage PageSz Hiwat Minpg Maxpg Idle Flags Util
amappl 48 262 0 47 215 37 3 0 3 4096 3 0 inf 0 0x040 84.0%
anonpl 16 2766 0 108 2658 136 11 0 11 4096 11 0 inf 0 0x1040 94.4%
aobj 72 2 0 0 2 54 1 0 1 4096 1 0 inf 0 0x040 3.5%
ataspl 56 191105 0 191105 0 72 1 0 1 4096 1 0 inf 1 0x040 0.0%
biopl 176 66 0 32 34 12 2 0 2 4096 2 0 inf 0 0x040 73.0%
buf16k 16384 1832 0 1265 567 5 209 66 143 65536 200 1 1 0 0x000 99.1%
buf1k 1024 2 0 2 0 64 1 0 1 65536 1 1 1 1 0x000 0.0%
buf2k 2048 13196 0 8037 5159 1081 221 26 195 65536 218 1 1 0 0x000 82.7%
buf4k 4096 66 0 53 13 19 3 1 2 65536 2 1 1 1 0x000 40.6%
buf512b 512 1 0 1 0 128 1 0 1 65536 1 1 1 1 0x000 0.0%
buf8k 8192 26 0 25 1 7 2 1 1 65536 2 1 1 0 0x000 12.5%
bufpl 176 7698 0 154 7544 0 328 0 328 4096 328 0 inf 0 0x040 98.8%
cwdi 64 34 0 21 13 50 1 0 1 4096 1 0 inf 0 0x040 20.3%
execargs 262144 136 0 136 0 1 1 0 1 262144 1 0 16 1 0xc00 0.0%
extent 20 140 0 126 14 189 1 0 1 4096 1 0 inf 0 0x040 6.8%
fdfile 64 164 0 92 72 117 3 0 3 4096 3 0 inf 0 0x1040 37.5%
ffsdino2 256 35522 0 8 35514 6 2220 0 2220 4096 2220 0 inf 0 0x000 100.0%
ffsino 172 35511 0 0 35511 1 1544 0 1544 4096 1544 0 inf 0 0x040 96.6%
file 64 88 0 16 72 54 2 0 2 4096 2 0 inf 0 0x040 56.2%
filedesc 704 34 0 21 13 22 7 0 7 4096 7 0 inf 1 0x040 31.9%
fstrans 20 23 0 4 19 184 1 0 1 4096 1 0 inf 0 0x040 9.3%
in6pcbpl 144 41 0 22 19 9 1 0 1 4096 1 0 inf 0 0x040 66.8%
inmltpl 32 2 0 0 2 125 1 0 1 4096 1 0 inf 0 0x040 1.6%
inpcbpl 112 37 0 22 15 21 1 0 1 4096 1 0 inf 0 0x040 41.0%
kcredpl 192 89 0 8 81 24 5 0 5 4096 5 0 inf 0 0x040 75.9%
kmem-100 100 74 0 9 65 10 15 0 15 512 15 0 inf 1 0xc00 84.6%
kmem-1024 1024 102 0 48 54 47 101 0 101 1024 101 0 inf 47 0xc00 53.5%
kmem-12 12 248989 0 3 248986 32 5929 0 5929 512 5929 0 inf 0 0xc00 98.4%
kmem-128 128 74 0 9 65 11 19 0 19 512 19 0 inf 1 0xc00 85.5%
kmem-1280 1280 6 0 4 2 2 4 0 4 1280 4 0 inf 2 0xc00 50.0%
kmem-1536 1536 4 0 0 4 0 4 0 4 1536 4 0 inf 0 0xc00 100.0%
kmem-16 16 186 0 8 178 14 6 0 6 512 6 0 inf 0 0xc00 92.7%
kmem-1792 1792 2 0 0 2 0 2 0 2 1792 2 0 inf 0 0xc00 100.0%
kmem-20 20 20 0 3 17 8 1 0 1 512 1 0 inf 0 0xc00 66.4%
kmem-2048 2048 9 0 0 9 0 9 0 9 2048 9 0 inf 0 0xc00 100.0%
kmem-2304 2304 3 0 3 0 3 3 0 3 2304 3 0 inf 3 0xc00 0.0%
kmem-24 24 2116 0 2076 40 2081 101 0 101 512 101 0 inf 96 0xc00 1.9%
kmem-256 256 5 0 3 2 3 5 0 5 256 5 0 inf 3 0xc00 40.0%
kmem-2560 2560 4 0 0 4 0 4 0 4 2560 4 0 inf 0 0xc00 100.0%
kmem-28 28 2095 0 2064 31 2075 117 0 117 512 117 0 inf 111 0xc00 1.4%
kmem-2816 2816 2 0 0 2 0 2 0 2 2816 2 0 inf 0 0xc00 100.0%
kmem-3072 3072 1 0 0 1 0 1 0 1 3072 1 0 inf 0 0xc00 100.0%
kmem-32 32 126 0 31 95 33 8 0 8 512 8 0 inf 1 0xc00 74.2%
kmem-3328 3328 1 0 0 1 0 1 0 1 3328 1 0 inf 0 0xc00 100.0%
kmem-3584 3584 4 0 0 4 0 4 0 4 3584 4 0 inf 0 0xc00 100.0%
kmem-36 36 6 0 0 6 8 1 0 1 512 1 0 inf 0 0xc00 42.2%
kmem-3840 3840 1 0 0 1 0 1 0 1 3840 1 0 inf 0 0xc00 100.0%
kmem-4 4 188 0 6 182 74 2 0 2 512 2 0 inf 0 0xc00 71.1%
kmem-40 40 30 0 0 30 6 3 0 3 512 3 0 inf 0 0xc00 78.1%
kmem-4096 4096 3 0 0 3 0 3 0 3 4096 3 0 inf 0 0xc00 100.0%
kmem-44 44 1 0 0 1 10 1 0 1 512 1 0 inf 0 0xc00 8.6%
kmem-48 48 3 0 0 3 7 1 0 1 512 1 0 inf 0 0xc00 28.1%
kmem-512 512 101 0 0 101 0 101 0 101 512 101 0 inf 0 0xc00 100.0%
kmem-64 64 215 0 0 215 1 27 0 27 512 27 0 inf 0 0xc00 99.5%
kmem-72 72 5 0 0 5 2 1 0 1 512 1 0 inf 0 0xc00 70.3%
kmem-768 768 65 0 0 65 0 65 0 65 768 65 0 inf 0 0xc00 100.0%
kmem-8 8 118 0 0 118 10 2 0 2 512 2 0 inf 0 0xc00 92.2%
kmem-84 84 15 0 0 15 3 3 0 3 512 3 0 inf 0 0xc00 82.0%
ksiginfo 48 29 0 0 29 55 1 0 1 4096 1 0 inf 0 0x040 34.0%
kvakernel 4096 68027 0 0 68027 5 2126 0 2126 131072 2126 0 inf 0 0x600 100.0%
kvakmem 4096 535 0 0 535 9 17 0 17 131072 17 0 inf 0 0x600 98.3%
lockf 64 4 0 0 4 59 1 0 1 4096 1 0 inf 0 0x040 6.2%
lwppl 672 64 0 0 64 2 11 0 11 4096 11 0 inf 0 0x040 95.5%
mbpl 256 257 0 0 257 31 18 0 18 4096 18 1 inf 1 0x000 89.2%
mclpl 2048 196 0 0 196 8 102 0 102 4096 102 4 8030 4 0x000 96.1%
mutex 64 367 0 0 367 11 6 0 6 4096 6 0 inf 0 0x040 95.6%
ncache 128 9875 0 0 9875 14 319 0 319 4096 319 0 inf 0 0x040 96.7%
nfsreqcachepl 56 64 0 0 64 8 1 0 1 4096 1 0 inf 0 0x040 87.5%
nfsrvdescpl 184 1095125 0 1095125 0 22 1 0 1 4096 1 0 inf 1 0x040 0.0%
pcache 576 86 0 14 72 5 11 0 11 4096 11 0 inf 0 0x040 92.0%
pcachecpu 64 82 0 0 82 44 2 0 2 4096 2 0 inf 0 0x040 64.1%
pcglarge 512 44 0 9 35 5 5 0 5 4096 5 0 inf 0 0x000 87.5%
pcgnormal 128 1061 0 391 670 260 30 0 30 4096 30 0 inf 3 0x040 69.8%
pdict16 44 19 0 4 15 77 1 0 1 4096 1 0 inf 0 0x040 16.1%
pdict32 60 7 0 1 6 61 1 0 1 4096 1 0 inf 0 0x040 8.8%
pdppl 4096 55 0 41 14 19 33 0 33 4096 33 0 inf 19 0x000 42.4%
phpool-0 32 65046 0 94 64952 72 512 0 512 4096 512 0 inf 0 0x040 99.1%
phpool-128 44 2 0 0 2 90 1 0 1 4096 1 0 inf 0 0x040 2.1%
phpool-64 36 8753 0 0 8753 95 79 0 79 4096 79 0 inf 0 0x040 97.4%
piperd 192 8 0 0 8 13 1 0 1 4096 1 0 inf 0 0x040 37.5%
pipewr 192 6 0 0 6 15 1 0 1 4096 1 0 inf 0 0x040 28.1%
plimitpl 196 3 0 0 3 17 1 0 1 4096 1 0 inf 0 0x040 14.4%
pmappl 96 33 0 0 33 9 1 0 1 4096 1 0 inf 0 0x040 77.3%
pnbufpl 1024 248919 0 0 248919 1 62230 0 62230 4096 62230 0 inf 0 0x000 100.0%
procpl 448 22 0 0 22 5 3 0 3 4096 3 0 inf 0 0x040 80.2%
propdict 32 85 0 17 68 59 1 0 1 4096 1 0 inf 0 0x040 53.1%
propnmbr 32 15 0 4 11 116 1 0 1 4096 1 0 inf 0 0x040 8.6%
propstng 20 80 0 8 72 131 1 0 1 4096 1 0 inf 0 0x040 35.2%
pstatspl 268 22 0 0 22 8 2 0 2 4096 2 0 inf 0 0x040 72.0%
ptimerpl 164 5 0 2 3 21 1 0 1 4096 1 0 inf 0 0x040 12.0%
ptimerspl 140 5 0 2 3 26 1 0 1 4096 1 0 inf 0 0x040 10.3%
pvpl 20 4139 0 0 4139 124 21 0 21 4096 21 0 inf 0 0x1040 96.2%
ractx 24 517 0 0 517 159 4 0 4 4096 4 0 inf 0 0x040 75.7%
rndsample 528 2958 0 2957 1 6 1 0 1 4096 1 0 inf 0 0x040 12.9%
rtentpl 148 29 0 0 29 25 2 0 2 4096 2 0 inf 0 0x040 52.4%
sigacts 2056 33 0 0 33 0 33 0 33 4096 33 0 inf 0 0x040 50.2%
socket 332 42 0 0 42 6 4 0 4 4096 4 0 inf 0 0x040 85.1%
synpl 188 1 0 1 0 21 1 0 1 4096 1 0 inf 1 0x040 0.0%
tcpcbpl 500 19 0 6 13 3 2 0 2 4096 2 0 inf 0 0x040 79.3%
tcpipqepl 44 4154 0 4154 0 92 1 0 1 4096 1 0 inf 1 0x040 0.0%
tmpfs_dirent_0xcbe4d000 140 1 0 0 1 28 1 0 1 4096 1 0 inf 0 0x040 3.4%
tstilepl 52 64 0 0 64 14 1 0 1 4096 1 0 inf 0 0x040 81.2%
uarea 8192 64 0 0 64 0 64 0 64 8192 64 0 inf 0 0xc00 100.0%
ufsdir 264 10 0 0 10 5 1 0 1 4096 1 0 inf 0 0x040 64.5%
vmembt 28 7574 0 0 7574 111 53 0 53 4096 53 0 inf 0 0x040 97.7%
vmmpepl 84 510 0 0 510 18 11 0 11 4096 11 0 inf 0 0x040 95.1%
vmsppl 212 33 0 0 33 5 2 0 2 4096 2 0 inf 0 0x040 85.4%
vnodepl 172 35519 0 0 35519 16 1545 0 1545 4096 1545 0 inf 0 0x040 96.5%
wapblinopl 20 92489 0 92489 0 203 1 0 1 4096 1 0 inf 1 0x040 0.0%
In use 299051K, total allocated 302970K; utilization 98.7%
From: Mindaugas Rasiukevicius <rmind@netbsd.org>
To: hash@abox3.so-net.ne.jp, dholland@netbsd.org
Cc: gnats-bugs@NetBSD.org, kern-bug-people@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/44625: memory exhaustion on NFS server while writing many
files
Date: Thu, 17 Mar 2011 05:31:36 +0000
hash@abox3.so-net.ne.jp wrote:
> <...>
> The Npage of pnbufpl grows extremely.
>
> I also tried some kernels checked out on other date. It seems that
> this leakage is seen after struct componentname changes on Nov 30 2010.
http://nxr.netbsd.org/xref/src/sys/nfs/nfs_serv.c?r=1.156#465
A leak in this error path (at least)?
> --
> Takahiro HAYASHI
--
Mindaugas
From: hash@abox3.so-net.ne.jp
To: Mindaugas Rasiukevicius <rmind@netbsd.org>, dholland@netbsd.org
Cc: gnats-bugs@NetBSD.org, kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org
Subject: Re: kern/44625: memory exhaustion on NFS server while writing many files
Date: Thu, 17 Mar 2011 16:19:22 +0900
hi,
On Thu, 17 Mar 2011 05:31:36 +0000
Mindaugas Rasiukevicius <rmind@netbsd.org> wrote:
> hash@abox3.so-net.ne.jp wrote:
> > <...>
> > The Npage of pnbufpl grows extremely.
> >
> > I also tried some kernels checked out on other date. It seems that
> > this leakage is seen after struct componentname changes on Nov 30 2010.
>
> http://nxr.netbsd.org/xref/src/sys/nfs/nfs_serv.c?r=1.156#465
>
> A leak in this error path (at least)?
Good catch! I confirm that lookup'ing various nonexistent names
(like below script) on NFS client causes leak.
Thank you.
#!/bin/sh -
i=1
while : ; do
# /usr/nfs/ is empty
cat /usr/nfs/$i > /dev/null 2>&1
i=$((i + 1))
done
--
Takahiro HAYASHI
Responsible-Changed-From-To: kern-bug-people->dholland
Responsible-Changed-By: dholland@NetBSD.org
Responsible-Changed-When: Thu, 17 Mar 2011 09:15:50 +0000
Responsible-Changed-Why:
my fault
From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org, Mindaugas Rasiukevicius <rmind@netbsd.org>
Cc:
Subject: Re: kern/44625: memory exhaustion on NFS server while writing many
files
Date: Thu, 17 Mar 2011 09:18:53 +0000
On Thu, Mar 17, 2011 at 05:35:01AM +0000, Mindaugas Rasiukevicius wrote:
> > The Npage of pnbufpl grows extremely.
> >
> > I also tried some kernels checked out on other date. It seems that
> > this leakage is seen after struct componentname changes on Nov 30 2010.
>
> http://nxr.netbsd.org/xref/src/sys/nfs/nfs_serv.c?r=1.156#465
>
> A leak in this error path (at least)?
good catch and boy don't I feel stupid. :-/
This may help (not tested yet though):
Index: nfs/nfs_serv.c
===================================================================
RCS file: /cvsroot/src/sys/nfs/nfs_serv.c,v
retrieving revision 1.156
diff -u -p -r1.156 nfs_serv.c
--- nfs/nfs_serv.c 5 Feb 2011 13:35:11 -0000 1.156
+++ nfs/nfs_serv.c 17 Mar 2011 09:14:46 -0000
@@ -463,6 +463,12 @@ nfsrv_lookup(struct nfsrv_descript *nfsd
}
if (error) {
+ if (nd.ni_pathbuf != NULL) {
+ pathbuf_destroy(nd.ni_pathbuf);
+ }
+ if (ipb != NULL) {
+ pathbuf_destroy(ipb);
+ }
nfsm_reply(NFSX_POSTOPATTR(v3));
nfsm_srvpostop_attr(dirattr_ret, &dirattr);
return (0);
Index: nfs/nfs_srvsubs.c
===================================================================
RCS file: /cvsroot/src/sys/nfs/nfs_srvsubs.c,v
retrieving revision 1.8
diff -u -p -r1.8 nfs_srvsubs.c
--- nfs/nfs_srvsubs.c 30 Nov 2010 10:30:03 -0000 1.8
+++ nfs/nfs_srvsubs.c 17 Mar 2011 09:14:46 -0000
@@ -276,6 +276,7 @@ nfs_namei(struct nameidata *ndp, nfsrvfh
out:
if (ndp->ni_pathbuf != NULL) {
pathbuf_destroy(ndp->ni_pathbuf);
+ ndp->ni_pathbuf = NULL;
} else {
PNBUF_PUT(path);
}
--
David A. Holland
dholland@netbsd.org
From: hash@abox3.so-net.ne.jp
To: dholland@NetBSD.org
Cc: gnats-bugs@NetBSD.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/44625: memory exhaustion on NFS server while writing many files
Date: Thu, 17 Mar 2011 19:49:40 +0900
On Thu, 17 Mar 2011 09:20:05 +0000 (UTC)
David Holland <dholland-bugs@netbsd.org> wrote:
> The following reply was made to PR kern/44625; it has been noted by GNATS.
>
> From: David Holland <dholland-bugs@netbsd.org>
> To: gnats-bugs@NetBSD.org, Mindaugas Rasiukevicius <rmind@netbsd.org>
> Cc:
> Subject: Re: kern/44625: memory exhaustion on NFS server while writing many
> files
> Date: Thu, 17 Mar 2011 09:18:53 +0000
>
> On Thu, Mar 17, 2011 at 05:35:01AM +0000, Mindaugas Rasiukevicius wrote:
> > > The Npage of pnbufpl grows extremely.
> > >
> > > I also tried some kernels checked out on other date. It seems that
> > > this leakage is seen after struct componentname changes on Nov 30 2010.
> >
> > http://nxr.netbsd.org/xref/src/sys/nfs/nfs_serv.c?r=1.156#465
> >
> > A leak in this error path (at least)?
>
> good catch and boy don't I feel stupid. :-/
>
> This may help (not tested yet though):
I try to rsync x4 and currently it does not seem to leak.
BTW does it need to check whether nd.ni_pathbuf != NULL
at 6 lines below?
Thanks.
> Index: nfs/nfs_serv.c
> ===================================================================
> RCS file: /cvsroot/src/sys/nfs/nfs_serv.c,v
> retrieving revision 1.156
> diff -u -p -r1.156 nfs_serv.c
> --- nfs/nfs_serv.c 5 Feb 2011 13:35:11 -0000 1.156
> +++ nfs/nfs_serv.c 17 Mar 2011 09:14:46 -0000
> @@ -463,6 +463,12 @@ nfsrv_lookup(struct nfsrv_descript *nfsd
> }
>
> if (error) {
> + if (nd.ni_pathbuf != NULL) {
> + pathbuf_destroy(nd.ni_pathbuf);
> + }
> + if (ipb != NULL) {
> + pathbuf_destroy(ipb);
> + }
> nfsm_reply(NFSX_POSTOPATTR(v3));
> nfsm_srvpostop_attr(dirattr_ret, &dirattr);
> return (0);
> Index: nfs/nfs_srvsubs.c
> ===================================================================
> RCS file: /cvsroot/src/sys/nfs/nfs_srvsubs.c,v
> retrieving revision 1.8
> diff -u -p -r1.8 nfs_srvsubs.c
> --- nfs/nfs_srvsubs.c 30 Nov 2010 10:30:03 -0000 1.8
> +++ nfs/nfs_srvsubs.c 17 Mar 2011 09:14:46 -0000
> @@ -276,6 +276,7 @@ nfs_namei(struct nameidata *ndp, nfsrvfh
> out:
> if (ndp->ni_pathbuf != NULL) {
> pathbuf_destroy(ndp->ni_pathbuf);
> + ndp->ni_pathbuf = NULL;
> } else {
> PNBUF_PUT(path);
> }
>
>
> --
> David A. Holland
> dholland@netbsd.org
>
--
Takahiro HAYASHI
From: "David A. Holland" <dholland@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/44625 CVS commit: src/sys/nfs
Date: Sat, 19 Mar 2011 01:34:25 +0000
Module Name: src
Committed By: dholland
Date: Sat Mar 19 01:34:25 UTC 2011
Modified Files:
src/sys/nfs: nfs_serv.c nfs_srvsubs.c
Log Message:
Fix memory leak introduced with the struct pathbuf changes. Hi, me.
Closes PR 44625.
To generate a diff of this commit:
cvs rdiff -u -r1.156 -r1.157 src/sys/nfs/nfs_serv.c
cvs rdiff -u -r1.8 -r1.9 src/sys/nfs/nfs_srvsubs.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/44625: memory exhaustion on NFS server while writing many
files
Date: Sat, 19 Mar 2011 01:43:00 +0000
On Thu, Mar 17, 2011 at 10:50:07AM +0000, hash@abox3.so-net.ne.jp wrote:
> > This may help (not tested yet though):
>
> I try to rsync x4 and currently it does not seem to leak.
Good...
> BTW does it need to check whether nd.ni_pathbuf != NULL
> at 6 lines below?
Yes - it's an error branch for more than one error case. If nfs_namei
succeeds and then the webnfs goop fails, the pathbuf will need to be
destroyed. However, if nfs_namei fails the pathbuf should be null.
There's another bug though, which is that nfs_namei shouldn't ever be
sending back a non-null path buffer if it fails, which it apparently
does in some circumstances since I assume you weren't using webnfs.
Will attend to that tomorrow.
--
David A. Holland
dholland@netbsd.org
From: "David A. Holland" <dholland@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/44625 CVS commit: src/sys/nfs
Date: Mon, 8 Aug 2011 16:04:07 +0000
Module Name: src
Committed By: dholland
Date: Mon Aug 8 16:04:07 UTC 2011
Modified Files:
src/sys/nfs: nfs_serv.c nfs_srvsubs.c
Log Message:
nfs_namei() should not return a non-null path buffer except on success,
even though the callers are apparently prepared to cope.
Fixes last tidyup part of PR 44625.
To generate a diff of this commit:
cvs rdiff -u -r1.159 -r1.160 src/sys/nfs/nfs_serv.c
cvs rdiff -u -r1.10 -r1.11 src/sys/nfs/nfs_srvsubs.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->closed
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Mon, 08 Aug 2011 16:08:54 +0000
State-Changed-Why:
Fixed the last bit (for a five-months' value of "tomorrow", sigh).
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home