NetBSD Problem Report #55707
From kardel@kardel.name Fri Oct 9 08:11:14 2020
Return-Path: <kardel@kardel.name>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 4E1421A921F
for <gnats-bugs@gnats.NetBSD.org>; Fri, 9 Oct 2020 08:11:14 +0000 (UTC)
Message-Id: <20201009081107.43578AAAC81A@pip.kardel.name>
Date: Fri, 9 Oct 2020 10:11:07 +0200 (CEST)
From: kardel@kardel.name
Reply-To: kardel@kardel.name
To: gnats-bugs@NetBSD.org
Subject: ZFS seems to trigger a lot of xcalls
X-Send-Pr-Version: 3.95
>Number: 55707
>Category: kern
>Synopsis: ZFS seems to trigger a lot of xcalls
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: chs
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Fri Oct 09 08:15:00 +0000 2020
>Closed-Date: Thu Jan 05 09:06:42 +0000 2023
>Last-Modified: Thu Jan 05 09:06:42 +0000 2023
>Originator: Frank Kardel
>Release: NetBSD 9.99.73
>Organization:
>Environment:
System: NetBSD pip 9.99.73 NetBSD 9.99.73 (PIPGEN) #0: Tue Oct 6 22:46:19 CEST 2020 kardel@pip:/src/NetBSD/cur/src/obj.amd64/sys/arch/amd64/compile/PIPGEN amd64
Architecture: x86_64
Machine: amd64
>Description:
ZFS seems to trigger massive xcall work. How a single thread can can
take more that 100% escapes me but could be a top/sampling bug.
This is on a AMD Ryzen 9 X3950X (16 cores/32 threads).
The workload is a single threaded bareos(bacula derivitive) base
backup of ZFS file systems with ls4 compressen and deduplication.
I wonder whether the xcall load could be reduced. It is clear that
ZFS does a lot more kernel work (dedup, compression, raidz etc),
but does it need so any xcalls?
With that many xcalls I have seem CPU % to crep up above 1200%
which seems to be a high workload.
top:
PID LID USERNAME PRI STATE TIME WCPU CPU NAME COMMAND
0 210 root 127 xcall/10 286:11 337.06% 337.06% xcall/10 [system]
0 216 root 127 xcall/11 269:04 315.33% 315.33% xcall/11 [system]
0 164 root 127 xcall/13 257:14 235.69% 235.69% xcall/13 [system]
0 120 root 127 xcall/6 308:27 203.71% 203.71% xcall/6 [system]
0 198 root 127 xcall/8 291:53 193.16% 193.16% xcall/8 [system]
0 384 root 127 xcall/17 240:15 155.71% 155.71% xcall/17 [system]
0 414 root 127 xcall/22 194:37 129.35% 129.35% xcall/22 [system]
0 96 root 127 RUN/2 301:26 114.45% 114.45% xcall/2 [system]
0 304 root 127 xcall/25 170:53 88.62% 88.62% xcall/25 [system]
0 204 root 127 xcall/9 290:37 88.57% 88.57% xcall/9 [system]
0 222 root 127 xcall/12 263:09 80.81% 80.81% xcall/12 [system]
0 390 root 127 xcall/18 226:49 75.98% 75.98% xcall/18 [system]
0 396 root 127 xcall/19 204:52 63.62% 63.62% xcall/19 [system]
0 114 root 127 xcall/5 311:53 61.04% 61.04% xcall/5 [system]
0 402 root 127 xcall/20 226:08 56.98% 56.98% xcall/20 [system]
0 170 root 127 xcall/14 243:54 55.91% 55.91% xcall/14 [system]
0 298 root 127 xcall/24 176:27 54.20% 54.20% xcall/24 [system]
0 310 root 127 xcall/26 172:31 52.54% 52.54% xcall/26 [system]
0 328 root 127 xcall/29 140:24 52.39% 52.39% xcall/29 [system]
0 408 root 127 xcall/21 198:44 49.37% 49.37% xcall/21 [system]
0 102 root 127 xcall/3 313:49 46.97% 46.97% xcall/3 [system]
0 292 root 127 xcall/23 201:36 42.19% 42.19% xcall/23 [system]
0 182 root 127 xcall/16 238:50 39.26% 39.26% xcall/16 [system]
0 25 root 127 xcall/1 260:37 33.98% 33.98% xcall/1 [system]
9248 29770 root 32 CPU/1 309:46 29.88% 29.88% - bareos-fd
0 761 root 96 xclocv/1 246:05 24.80% 24.80% zfs [system]
0 176 root 127 xcall/15 235:02 23.05% 23.05% xcall/15 [system]
0 322 root 127 xcall/28 152:48 8.54% 8.54% xcall/28 [system]
0 7 root 127 xcall/0 224:40 7.71% 7.71% xcall/0 [system]
0 334 root 127 xcall/30 103:55 7.42% 7.42% xcall/30 [system]
0 340 root 127 xcall/31 106:06 5.08% 5.08% xcall/31 [system]
9900 915 bareossd 84 poll/3 70:33 4.35% 4.35% - bareos-sd
24461 24461 pgsql 85 poll/15 0:06 0.20% 0.20% - postgres
0 108 root 127 xcall/4 311:07 0.00% 0.00% xcall/4 [system]
0 192 root 127 xcall/7 297:59 0.00% 0.00% xcall/7 [system]
0 444 root 127 xcall/27 158:40 0.00% 0.00% xcall/27 [system]
0 978 root 126 pgdaem/1 6:34 0.00% 0.00% pgdaemon [system]
0 9 root 125 vdrain/1 3:11 0.00% 0.00% vdrain [system]
0 979 root 124 syncer/2 2:59 0.00% 0.00% ioflush [system]
0 755 root 123 physio/1 2:12 0.00% 0.00% physiod [system]
0 2791 root 221 raidio/1 0:39 0.00% 0.00% raidio2 [system]
0 763 root 96 zfscv/2 0:23 0.00% 0.00% zfs [system]
9370 9370 kdm 85 select/3 0:20 0.00% 0.00% - kdm_greet
0 2790 root 221 rfnode/1 0:18 0.00% 0.00% raid2 [system]
8341 8341 root 191 pause/0 0:16 0.00% 0.00% - ntpd
vmstat -e:
interrupt total rate type
bus_dma bounces 76055857 1148 misc
bus_dma loads 159639431 2411 misc
bus_dma nbouncebufs 151 0 misc
pdpolicy reactanon 1461551 22 misc
pdpolicy reactexec 86592 1 misc
vmcmd kills 22146 0 misc
vmcmd extends 9683 0 misc
vmcmd calls 104605 1 misc
entropy notify 6 0 misc
entropy request 1 0 misc
entropy consolidate 3 0 misc
entropy immediate 3 0 misc
pserialize exclusive access 6313 0 misc
vmem static_bt_inuse 200 0 misc
vmem static_bt_count 200 0 misc
TLB shootdown 185232513 2798 intr
cpu0 cprng_strong reseed 4 0 misc
cpu0 cprng_strong reseed 4 0 misc
softint net/0 40492228 611 misc
softint net block/0 81126 1 misc
softint bio/0 2221870 33 misc
softint bio block/0 42 0 misc
softint clk/0 6643944 100 misc
softint clk block/0 19602 0 misc
softint ser/0 3528946 53 misc
callout late/0 32 0 misc
callout wait/0 35 0 misc
crosscall unicast 1010 0 misc
crosscall broadcast 609278581 9203 misc
nfs timer 10 0 misc
nfs timer start 9 0 misc
nfs timer stop 9 0 misc
cpu0 timer 6619323 99 intr
cpu0 AST IPI 637001317 9622 misc
cpu0 generic IPI 37331730 563 misc
cpu0 xcall IPI 75133 1 misc
cpu0 kpreempt IPI 1154985 17 misc
cpu1 cprng_strong reseed 4 0 misc
cpu1 cprng_strong reseed 3 0 misc
softint net/1 349428 5 misc
softint net block/1 14 0 misc
softint bio/1 73326994 1107 misc
softint bio block/1 32255 0 misc
softint clk/1 3213706 48 misc
softint clk block/1 51 0 misc
softint ser/1 4751098 71 misc
softint ser block/1 1 0 misc
callout late/1 11 0 misc
callout wait/1 92 0 misc
cpu1 timer 6619113 99 misc
cpu1 AST IPI 626040633 9457 misc
cpu1 xcall IPI 87675 1 misc
cpu1 kpreempt IPI 1370822 20 misc
cpu2 cprng_strong reseed 3 0 misc
cpu2 cprng_strong reseed 2 0 misc
softint net/2 193838 2 misc
softint net block/2 2 0 misc
softint bio/2 22 0 misc
softint clk/2 2887556 43 misc
softint ser/2 9648 0 misc
cpu2 timer 6619600 99 misc
cpu2 AST IPI 625282278 9445 misc
cpu2 xcall IPI 91278 1 misc
cpu2 kpreempt IPI 503810 7 misc
cpu3 cprng_strong reseed 3 0 misc
cpu3 cprng_strong reseed 2 0 misc
softint net/3 105424 1 misc
softint bio/3 6 0 misc
softint clk/3 2318488 35 misc
softint ser/3 9648 0 misc
cpu3 timer 6619594 99 misc
cpu3 AST IPI 621167253 9383 misc
cpu3 xcall IPI 97750 1 misc
cpu3 kpreempt IPI 211776 3 misc
cpu4 cprng_strong reseed 3 0 misc
cpu4 cprng_strong reseed 2 0 misc
softint net/4 126084 1 misc
softint bio/4 4 0 misc
softint clk/4 1826916 27 misc
softint ser/4 9648 0 misc
cpu4 timer 6619591 99 misc
cpu4 AST IPI 618809723 9347 misc
cpu4 xcall IPI 98803 1 misc
cpu4 kpreempt IPI 148634 2 misc
cpu5 cprng_strong reseed 3 0 misc
cpu5 cprng_strong reseed 2 0 misc
softint net/5 91489 1 misc
softint clk/5 1450466 21 misc
softint ser/5 9648 0 misc
cpu5 timer 6619567 99 misc
cpu5 AST IPI 617105876 9322 misc
cpu5 xcall IPI 99623 1 misc
cpu5 kpreempt IPI 117579 1 misc
cpu6 cprng_strong reseed 2 0 misc
cpu6 cprng_strong reseed 2 0 misc
softint net/6 91482 1 misc
softint bio/6 4 0 misc
softint clk/6 1235827 18 misc
softint ser/6 9648 0 misc
cpu6 timer 6619599 99 misc
cpu6 AST IPI 616226396 9308 misc
cpu6 xcall IPI 99597 1 misc
cpu6 kpreempt IPI 128607 1 misc
cpu7 cprng_strong reseed 2 0 misc
cpu7 cprng_strong reseed 2 0 misc
softint net/7 91467 1 misc
softint clk/7 1144559 17 misc
softint ser/7 9648 0 misc
softint ser block/7 1 0 misc
cpu7 timer 6619579 99 misc
cpu7 AST IPI 616026692 9305 misc
cpu7 xcall IPI 99445 1 misc
cpu7 kpreempt IPI 107853 1 misc
cpu8 cprng_strong reseed 2 0 misc
cpu8 cprng_strong reseed 2 0 misc
softint net/8 91496 1 misc
softint net block/8 1 0 misc
softint clk/8 1271321 19 misc
softint ser/8 9648 0 misc
callout wait/8 1 0 misc
cpu8 timer 6619598 99 misc
cpu8 AST IPI 615761683 9301 misc
cpu8 xcall IPI 98898 1 misc
cpu8 kpreempt IPI 125666 1 misc
cpu9 cprng_strong reseed 2 0 misc
cpu9 cprng_strong reseed 2 0 misc
softint net/9 91468 1 misc
softint clk/9 1141547 17 misc
softint ser/9 9648 0 misc
cpu9 timer 6619601 99 misc
cpu9 AST IPI 614915934 9289 misc
cpu9 xcall IPI 99506 1 misc
cpu9 kpreempt IPI 120030 1 misc
cpu10 cprng_strong reseed 2 0 misc
cpu10 cprng_strong reseed 2 0 misc
softint net/10 91491 1 misc
softint clk/10 1019351 15 misc
softint ser/10 9648 0 misc
softint ser block/10 1 0 misc
cpu10 timer 6619585 99 misc
cpu10 AST IPI 614458607 9282 misc
cpu10 xcall IPI 99821 1 misc
cpu10 kpreempt IPI 105079 1 misc
cpu11 cprng_strong reseed 2 0 misc
cpu11 cprng_strong reseed 2 0 misc
softint net/11 91475 1 misc
softint clk/11 1068134 16 misc
softint ser/11 9648 0 misc
cpu11 timer 6619598 99 misc
cpu11 AST IPI 614540535 9283 misc
cpu11 xcall IPI 99234 1 misc
cpu11 kpreempt IPI 102304 1 misc
cpu12 cprng_strong reseed 2 0 misc
cpu12 cprng_strong reseed 2 0 misc
softint net/12 91473 1 misc
softint clk/12 1205121 18 misc
softint ser/12 9648 0 misc
cpu12 timer 6619596 99 misc
cpu12 AST IPI 614820677 9287 misc
cpu12 xcall IPI 98779 1 misc
cpu12 kpreempt IPI 119165 1 misc
cpu13 cprng_strong reseed 2 0 misc
cpu13 cprng_strong reseed 2 0 misc
softint net/13 91476 1 misc
softint clk/13 1344225 20 misc
softint ser/13 9648 0 misc
cpu13 timer 6619599 99 misc
cpu13 AST IPI 615010259 9290 misc
cpu13 xcall IPI 99011 1 misc
cpu13 kpreempt IPI 117963 1 misc
cpu14 cprng_strong reseed 2 0 misc
cpu14 cprng_strong reseed 2 0 misc
softint net/14 91507 1 misc
softint net block/14 3 0 misc
softint clk/14 2674465 40 misc
softint ser/14 9648 0 misc
callout wait/14 1 0 misc
cpu14 timer 6619601 99 misc
cpu14 AST IPI 617522880 9328 misc
cpu14 xcall IPI 98392 1 misc
cpu14 kpreempt IPI 106741 1 misc
cpu15 cprng_strong reseed 2 0 misc
cpu15 cprng_strong reseed 2 0 misc
softint net/15 91559 1 misc
softint net block/15 7 0 misc
softint clk/15 4036380 60 misc
softint clk block/15 1 0 misc
softint ser/15 9648 0 misc
softint ser block/15 1 0 misc
callout wait/15 4 0 misc
cpu15 timer 6619584 99 misc
cpu15 AST IPI 625956050 9455 misc
cpu15 xcall IPI 88385 1 misc
cpu15 kpreempt IPI 169277 2 misc
cpu16 cprng_strong reseed 2 0 misc
cpu16 cprng_strong reseed 2 0 misc
softint net/16 91462 1 misc
softint clk/16 533008 8 misc
softint ser/16 9648 0 misc
cpu16 timer 6619582 99 misc
cpu16 AST IPI 610648348 9224 misc
cpu16 xcall IPI 100268 1 misc
cpu16 kpreempt IPI 3601 0 misc
cpu17 cprng_strong reseed 2 0 misc
cpu17 cprng_strong reseed 2 0 misc
softint net/17 91463 1 misc
softint clk/17 395148 5 misc
softint ser/17 9648 0 misc
cpu17 timer 6619582 99 misc
cpu17 AST IPI 610524154 9222 misc
cpu17 xcall IPI 100285 1 misc
cpu17 kpreempt IPI 4052 0 misc
cpu18 cprng_strong reseed 2 0 misc
cpu18 cprng_strong reseed 2 0 misc
softint net/18 91463 1 misc
softint clk/18 191111 2 misc
softint ser/18 9648 0 misc
cpu18 timer 6619601 99 misc
cpu18 AST IPI 610323258 9219 misc
cpu18 xcall IPI 100544 1 misc
cpu18 kpreempt IPI 782 0 misc
cpu19 cprng_strong reseed 2 0 misc
cpu19 cprng_strong reseed 2 0 misc
softint net/19 91462 1 misc
softint clk/19 197297 2 misc
softint ser/19 9648 0 misc
cpu19 timer 6619601 99 misc
cpu19 AST IPI 610303077 9219 misc
cpu19 xcall IPI 100550 1 misc
cpu19 kpreempt IPI 430 0 misc
cpu20 cprng_strong reseed 1 0 misc
cpu20 cprng_strong reseed 2 0 misc
softint net/20 91463 1 misc
softint clk/20 71946 1 misc
softint ser/20 9648 0 misc
callout wait/20 1 0 misc
cpu20 timer 6619601 99 misc
cpu20 AST IPI 609850761 9212 misc
cpu20 xcall IPI 100592 1 misc
cpu20 kpreempt IPI 217 0 misc
cpu21 cprng_strong reseed 2 0 misc
cpu21 cprng_strong reseed 2 0 misc
softint net/21 91467 1 misc
softint net block/21 1 0 misc
softint clk/21 80410 1 misc
softint ser/21 9648 0 misc
cpu21 timer 6619600 99 misc
cpu21 AST IPI 609847715 9212 misc
cpu21 xcall IPI 100686 1 misc
cpu21 kpreempt IPI 319 0 misc
cpu22 cprng_strong reseed 2 0 misc
softint net/22 91463 1 misc
softint clk/22 83375 1 misc
softint ser/22 9648 0 misc
cpu22 timer 6619600 99 misc
cpu22 AST IPI 609793064 9211 misc
cpu22 xcall IPI 100658 1 misc
cpu22 kpreempt IPI 136 0 misc
cpu23 cprng_strong reseed 1 0 misc
cpu23 cprng_strong reseed 2 0 misc
softint net/23 91463 1 misc
softint clk/23 51690 0 misc
softint ser/23 9648 0 misc
cpu23 timer 6619600 99 misc
cpu23 AST IPI 609684437 9210 misc
cpu23 xcall IPI 100662 1 misc
cpu23 kpreempt IPI 181 0 misc
cpu24 cprng_strong reseed 1 0 misc
cpu24 cprng_strong reseed 2 0 misc
softint net/24 91463 1 misc
softint clk/24 81201 1 misc
softint ser/24 9648 0 misc
softint ser block/24 1 0 misc
cpu24 timer 6619600 99 misc
cpu24 AST IPI 609699934 9210 misc
cpu24 xcall IPI 100594 1 misc
cpu24 kpreempt IPI 520 0 misc
cpu25 cprng_strong reseed 1 0 misc
cpu25 cprng_strong reseed 2 0 misc
softint net/25 91463 1 misc
softint clk/25 70230 1 misc
softint ser/25 9648 0 misc
softint ser block/25 1 0 misc
cpu25 timer 6619595 99 misc
cpu25 AST IPI 609653962 9209 misc
cpu25 xcall IPI 100660 1 misc
cpu25 kpreempt IPI 82 0 misc
cpu26 cprng_strong reseed 1 0 misc
cpu26 cprng_strong reseed 2 0 misc
softint net/26 91463 1 misc
softint clk/26 54454 0 misc
softint ser/26 9648 0 misc
cpu26 timer 6619600 99 misc
cpu26 AST IPI 609614354 9208 misc
cpu26 xcall IPI 100606 1 misc
cpu26 kpreempt IPI 105 0 misc
cpu27 cprng_strong reseed 1 0 misc
cpu27 cprng_strong reseed 2 0 misc
softint net/27 91463 1 misc
softint clk/27 102875 1 misc
softint ser/27 9648 0 misc
cpu27 timer 6619600 99 misc
cpu27 AST IPI 609650842 9209 misc
cpu27 xcall IPI 100614 1 misc
cpu27 kpreempt IPI 282 0 misc
cpu28 cprng_strong reseed 1 0 misc
cpu28 cprng_strong reseed 2 0 misc
softint net/28 91463 1 misc
softint clk/28 95412 1 misc
softint ser/28 9648 0 misc
softint ser block/28 1 0 misc
cpu28 timer 6619600 99 misc
cpu28 AST IPI 609626733 9209 misc
cpu28 xcall IPI 100660 1 misc
cpu28 kpreempt IPI 267 0 misc
cpu29 cprng_strong reseed 2 0 misc
cpu29 cprng_strong reseed 2 0 misc
softint net/29 91467 1 misc
softint clk/29 218743 3 misc
softint ser/29 9648 0 misc
cpu29 timer 6619600 99 misc
cpu29 AST IPI 609824167 9212 misc
cpu29 xcall IPI 100440 1 misc
cpu29 kpreempt IPI 243 0 misc
cpu30 cprng_strong reseed 1 0 misc
cpu30 cprng_strong reseed 2 0 misc
softint net/30 91466 1 misc
softint clk/30 1108859 16 misc
softint ser/30 9648 0 misc
softint ser block/30 1 0 misc
cpu30 timer 6619595 99 misc
cpu30 AST IPI 610991451 9229 misc
cpu30 xcall IPI 100249 1 misc
cpu30 kpreempt IPI 111 0 misc
cpu31 cprng_strong reseed 2 0 misc
cpu31 cprng_strong reseed 2 0 misc
softint net/31 91464 1 misc
softint clk/31 2861956 43 misc
softint ser/31 9648 0 misc
callout wait/31 1 0 misc
cpu31 timer 6619589 99 misc
cpu31 AST IPI 614368932 9280 misc
cpu31 xcall IPI 95012 1 misc
cpu31 kpreempt IPI 334 0 misc
ioapic0 pin 1 6 0 intr
msix0 vec 0 38 0 intr
msix0 vec 1 83 0 intr
msix0 vec 2 653 0 intr
msix0 vec 3 16 0 intr
msix0 vec 5 4 0 intr
msix0 vec 7 4 0 intr
msix1 vec 1 30 0 intr
msix1 vec 2 56 0 intr
msix1 vec 3 6 0 intr
msix1 vec 4 6 0 intr
ioapic1 pin 6 161 0 intr
ioapic1 pin 4 717347 10 intr
ioapic1 pin 5 62320 0 intr
ioapic1 pin 7 397656 6 intr
msix2 vec 0 94877 1 intr
msix2 vec 1 83604 1 intr
msix2 vec 2 1 0 intr
msix3 vec 0 141155 2 intr
msix3 vec 1 13478 0 intr
msix3 vec 2 13612 0 intr
msix3 vec 3 33366 0 intr
msix3 vec 4 1 0 intr
msi4 vec 0 52 0 intr
msix5 vec 0 35 0 intr
msi6 vec 0 3877494 58 intr
msi7 vec 0 8539625 129 intr
msix9 vec 0 67827989 1024 intr
msix10 vec 0 1 0 intr
acpicpu0 C1 (HALT) 699740559 10570 misc
acpicpu0 P0 (3700 MHz) 352 0 misc
acpicpu0 P1 (2800 MHz) 331 0 misc
acpicpu0 P2 (2200 MHz) 257 0 misc
acpicpu1 C1 (HALT) 683204576 10320 misc
acpicpu1 P0 (3700 MHz) 352 0 misc
acpicpu1 P1 (2800 MHz) 331 0 misc
acpicpu1 P2 (2200 MHz) 257 0 misc
acpicpu2 C1 (HALT) 697384260 10534 misc
acpicpu2 P0 (3700 MHz) 352 0 misc
acpicpu2 P1 (2800 MHz) 331 0 misc
acpicpu2 P2 (2200 MHz) 257 0 misc
acpicpu3 C1 (HALT) 703514504 10627 misc
acpicpu3 P0 (3700 MHz) 352 0 misc
acpicpu3 P1 (2800 MHz) 331 0 misc
acpicpu3 P2 (2200 MHz) 257 0 misc
acpicpu4 C1 (HALT) 718806902 10858 misc
acpicpu4 P0 (3700 MHz) 352 0 misc
acpicpu4 P1 (2800 MHz) 331 0 misc
acpicpu4 P2 (2200 MHz) 257 0 misc
acpicpu5 C1 (HALT) 727584762 10991 misc
acpicpu5 P0 (3700 MHz) 352 0 misc
acpicpu5 P1 (2800 MHz) 331 0 misc
acpicpu5 P2 (2200 MHz) 257 0 misc
acpicpu6 C1 (HALT) 730667330 11037 misc
acpicpu6 P0 (3700 MHz) 352 0 misc
acpicpu6 P1 (2800 MHz) 331 0 misc
acpicpu6 P2 (2200 MHz) 257 0 misc
acpicpu7 C1 (HALT) 724562504 10945 misc
acpicpu7 P0 (3700 MHz) 352 0 misc
acpicpu7 P1 (2800 MHz) 331 0 misc
acpicpu7 P2 (2200 MHz) 257 0 misc
acpicpu8 C1 (HALT) 725187463 10954 misc
acpicpu8 P0 (3700 MHz) 352 0 misc
acpicpu8 P1 (2800 MHz) 331 0 misc
acpicpu8 P2 (2200 MHz) 257 0 misc
acpicpu9 C1 (HALT) 732968857 11072 misc
acpicpu9 P0 (3700 MHz) 352 0 misc
acpicpu9 P1 (2800 MHz) 331 0 misc
acpicpu9 P2 (2200 MHz) 257 0 misc
acpicpu10 C1 (HALT) 729234552 11015 misc
acpicpu10 P0 (3700 MHz) 352 0 misc
acpicpu10 P1 (2800 MHz) 331 0 misc
acpicpu10 P2 (2200 MHz) 257 0 misc
acpicpu11 C1 (HALT) 727013297 10982 misc
acpicpu11 P0 (3700 MHz) 352 0 misc
acpicpu11 P1 (2800 MHz) 331 0 misc
acpicpu11 P2 (2200 MHz) 257 0 misc
acpicpu12 C1 (HALT) 725551288 10960 misc
acpicpu12 P0 (3700 MHz) 352 0 misc
acpicpu12 P1 (2800 MHz) 331 0 misc
acpicpu12 P2 (2200 MHz) 257 0 misc
acpicpu13 C1 (HALT) 729637309 11022 misc
acpicpu13 P0 (3700 MHz) 352 0 misc
acpicpu13 P1 (2800 MHz) 331 0 misc
acpicpu13 P2 (2200 MHz) 257 0 misc
acpicpu14 C1 (HALT) 725378000 10957 misc
acpicpu14 P0 (3700 MHz) 352 0 misc
acpicpu14 P1 (2800 MHz) 331 0 misc
acpicpu14 P2 (2200 MHz) 257 0 misc
acpicpu15 C1 (HALT) 714850333 10798 misc
acpicpu15 P0 (3700 MHz) 352 0 misc
acpicpu15 P1 (2800 MHz) 331 0 misc
acpicpu15 P2 (2200 MHz) 257 0 misc
acpicpu16 C1 (HALT) 780913140 11796 misc
acpicpu16 P0 (3700 MHz) 352 0 misc
acpicpu16 P1 (2800 MHz) 331 0 misc
acpicpu16 P2 (2200 MHz) 257 0 misc
acpicpu17 C1 (HALT) 781392945 11803 misc
acpicpu17 P0 (3700 MHz) 352 0 misc
acpicpu17 P1 (2800 MHz) 331 0 misc
acpicpu17 P2 (2200 MHz) 257 0 misc
acpicpu18 C1 (HALT) 782578306 11821 misc
acpicpu18 P0 (3700 MHz) 352 0 misc
acpicpu18 P1 (2800 MHz) 331 0 misc
acpicpu18 P2 (2200 MHz) 257 0 misc
acpicpu19 C1 (HALT) 784627138 11852 misc
acpicpu19 P0 (3700 MHz) 352 0 misc
acpicpu19 P1 (2800 MHz) 331 0 misc
acpicpu19 P2 (2200 MHz) 257 0 misc
acpicpu20 C1 (HALT) 781765018 11809 misc
acpicpu20 P0 (3700 MHz) 352 0 misc
acpicpu20 P1 (2800 MHz) 331 0 misc
acpicpu20 P2 (2200 MHz) 257 0 misc
acpicpu21 C1 (HALT) 784666574 11853 misc
acpicpu21 P0 (3700 MHz) 352 0 misc
acpicpu21 P1 (2800 MHz) 331 0 misc
acpicpu21 P2 (2200 MHz) 257 0 misc
acpicpu22 C1 (HALT) 785147476 11860 misc
acpicpu22 P0 (3700 MHz) 352 0 misc
acpicpu22 P1 (2800 MHz) 331 0 misc
acpicpu22 P2 (2200 MHz) 257 0 misc
acpicpu23 C1 (HALT) 783972537 11842 misc
acpicpu23 P0 (3700 MHz) 352 0 misc
acpicpu23 P1 (2800 MHz) 331 0 misc
acpicpu23 P2 (2200 MHz) 257 0 misc
acpicpu24 C1 (HALT) 786051926 11874 misc
acpicpu24 P0 (3700 MHz) 352 0 misc
acpicpu24 P1 (2800 MHz) 331 0 misc
acpicpu24 P2 (2200 MHz) 257 0 misc
acpicpu25 C1 (HALT) 786580436 11882 misc
acpicpu25 P0 (3700 MHz) 352 0 misc
acpicpu25 P1 (2800 MHz) 331 0 misc
acpicpu25 P2 (2200 MHz) 257 0 misc
acpicpu26 C1 (HALT) 785940984 11872 misc
acpicpu26 P0 (3700 MHz) 352 0 misc
acpicpu26 P1 (2800 MHz) 331 0 misc
acpicpu26 P2 (2200 MHz) 257 0 misc
acpicpu27 C1 (HALT) 786748593 11884 misc
acpicpu27 P0 (3700 MHz) 352 0 misc
acpicpu27 P1 (2800 MHz) 331 0 misc
acpicpu27 P2 (2200 MHz) 257 0 misc
acpicpu28 C1 (HALT) 787221354 11891 misc
acpicpu28 P0 (3700 MHz) 352 0 misc
acpicpu28 P1 (2800 MHz) 331 0 misc
acpicpu28 P2 (2200 MHz) 257 0 misc
acpicpu29 C1 (HALT) 787905293 11902 misc
acpicpu29 P0 (3700 MHz) 352 0 misc
acpicpu29 P1 (2800 MHz) 331 0 misc
acpicpu29 P2 (2200 MHz) 257 0 misc
acpicpu30 C1 (HALT) 791359974 11954 misc
acpicpu30 P0 (3700 MHz) 352 0 misc
acpicpu30 P1 (2800 MHz) 331 0 misc
acpicpu30 P2 (2200 MHz) 257 0 misc
acpicpu31 C1 (HALT) 788347794 11908 misc
acpicpu31 P0 (3700 MHz) 352 0 misc
acpicpu31 P1 (2800 MHz) 331 0 misc
acpicpu31 P2 (2200 MHz) 257 0 misc
cpu0 cprng_fast reseed 3 0 misc
cpu1 cprng_fast reseed 3 0 misc
cpu2 cprng_fast reseed 2 0 misc
cpu3 cprng_fast reseed 2 0 misc
cpu4 cprng_fast reseed 2 0 misc
cpu5 cprng_fast reseed 2 0 misc
cpu6 cprng_fast reseed 2 0 misc
cpu7 cprng_fast reseed 2 0 misc
cpu8 cprng_fast reseed 2 0 misc
cpu9 cprng_fast reseed 2 0 misc
cpu10 cprng_fast reseed 2 0 misc
cpu11 cprng_fast reseed 2 0 misc
cpu12 cprng_fast reseed 2 0 misc
cpu13 cprng_fast reseed 2 0 misc
cpu14 cprng_fast reseed 2 0 misc
cpu15 cprng_fast reseed 2 0 misc
cpu16 cprng_fast reseed 2 0 misc
cpu17 cprng_fast reseed 2 0 misc
cpu18 cprng_fast reseed 2 0 misc
cpu19 cprng_fast reseed 2 0 misc
cpu20 cprng_fast reseed 2 0 misc
cpu21 cprng_fast reseed 2 0 misc
cpu22 cprng_fast reseed 2 0 misc
cpu23 cprng_fast reseed 2 0 misc
cpu24 cprng_fast reseed 2 0 misc
cpu25 cprng_fast reseed 2 0 misc
cpu26 cprng_fast reseed 2 0 misc
cpu27 cprng_fast reseed 2 0 misc
cpu28 cprng_fast reseed 2 0 misc
cpu29 cprng_fast reseed 2 0 misc
cpu30 cprng_fast reseed 2 0 misc
cpu31 cprng_fast reseed 2 0 misc
kpreempt defer: critical section 1047682 15 misc
kpreempt defer: kernel_lock 19898 0 misc
kpreempt immediate 9176145 138 misc
cpu0 entropy softint 131588 1 misc
cpu1 entropy softint 4966079 75 misc
msi12 vec 0 301 0 intr
wapbl fsid 0xa822/0x78b commit 6625 0 misc
wapbl fsid 0xa822/0x78b journal write total 312230 4 misc
wapbl fsid 0xa822/0x78b journal write finished async 16890 0 misc
wapbl fsid 0xa822/0x78b metadata async write 193102 2 misc
wapbl fsid 0xa822/0x78b cache flush 13250 0 misc
wapbl fsid 0xa821/0x78b commit 99 0 misc
wapbl fsid 0xa821/0x78b journal write total 51189 0 misc
wapbl fsid 0xa821/0x78b journal write finished async 1384 0 misc
wapbl fsid 0xa821/0x78b metadata async write 33412 0 misc
wapbl fsid 0xa821/0x78b cache flush 198 0 misc
wapbl fsid 0xa806/0x78b commit 19 0 misc
wapbl fsid 0xa806/0x78b journal write total 9090 0 misc
wapbl fsid 0xa806/0x78b journal write finished async 67 0 misc
wapbl fsid 0xa806/0x78b metadata async write 5937 0 misc
wapbl fsid 0xa806/0x78b cache flush 38 0 misc
>How-To-Repeat:
Use ZFS with dedup, lz4 compression on a 3 vdev raidz1 pool and
run a plain bareos base backup for some TBytes.
>Fix:
?
>Release-Note:
>Audit-Trail:
From: matthew green <mrg@eterna.com.au>
To: gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
gnats-bugs@netbsd.org
Cc:
Subject: re: kern/55707: ZFS seems to trigger a lot of xcalls
Date: Sun, 03 Oct 2021 09:59:54 +1100
while debugging an instance of this on an 80-cpu thread system, i was
asked to colllected some backtraces and threadlist of this, and i've
copied them here (they're large):
https://www.netbsd.org/~mrg/pr55707/
.mrg.
From: "Chuck Silvers" <chs@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/55707 CVS commit: src/external/cddl/osnet/dist/uts/common/fs/zfs
Date: Wed, 4 May 2022 15:49:55 +0000
Module Name: src
Committed By: chs
Date: Wed May 4 15:49:55 UTC 2022
Modified Files:
src/external/cddl/osnet/dist/uts/common/fs/zfs: arc.c
Log Message:
fix ARC checks for available memory:
there's an extra check that we inherited from FreeBSD that tries to
detect KVA exhaustion on platforms with limited KVA, but the condition
that decided whether to use the extra check was using a FreeBSDism
that doesn't exist on NetBSD, resulting in this check being used on
all platforms. on amd64 systems with lots of memory, this extra check
would result in the ARC thinking that it constantly needed to reclaim memory,
resulting in all the xcall threads running all the time but not doing
anything useful. change this condition so that this extra check for
KVA exhaustion is only used on 32-bit platforms. fixes PR 55707.
To generate a diff of this commit:
cvs rdiff -u -r1.20 -r1.21 \
src/external/cddl/osnet/dist/uts/common/fs/zfs/arc.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
Responsible-Changed-From-To: kern-bug-people->chs
Responsible-Changed-By: chs@NetBSD.org
Responsible-Changed-When: Wed, 04 May 2022 15:52:04 +0000
Responsible-Changed-Why:
take
State-Changed-From-To: open->feedback
State-Changed-By: chs@NetBSD.org
State-Changed-When: Wed, 04 May 2022 15:52:04 +0000
State-Changed-Why:
this should be fixed by src/external/cddl/osnet/dist/uts/common/fs/zfs/arc.c rev 1.21,
can you confirm?
From: Frank Kardel <kardel@netbsd.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/55707 (ZFS seems to trigger a lot of xcalls)
Date: Wed, 4 May 2022 17:55:11 +0200
Good catch.
Will test soon.
Frank
On 05/04/22 17:52, chs@NetBSD.org wrote:
> Synopsis: ZFS seems to trigger a lot of xcalls
>
> Responsible-Changed-From-To: kern-bug-people->chs
> Responsible-Changed-By: chs@NetBSD.org
> Responsible-Changed-When: Wed, 04 May 2022 15:52:04 +0000
> Responsible-Changed-Why:
> take
>
>
> State-Changed-From-To: open->feedback
> State-Changed-By: chs@NetBSD.org
> State-Changed-When: Wed, 04 May 2022 15:52:04 +0000
> State-Changed-Why:
> this should be fixed by src/external/cddl/osnet/dist/uts/common/fs/zfs/arc.c rev 1.21,
> can you confirm?
>
>
>
From: Frank Kardel <kardel@kardel.name>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/55707 (ZFS seems to trigger a lot of xcalls)
Date: Wed, 4 May 2022 17:56:49 +0200
Good catch.
Will test probably on the weekend.
Frank
On 05/04/22 17:52, chs@NetBSD.org wrote:
> Synopsis: ZFS seems to trigger a lot of xcalls
>
> Responsible-Changed-From-To: kern-bug-people->chs
> Responsible-Changed-By: chs@NetBSD.org
> Responsible-Changed-When: Wed, 04 May 2022 15:52:04 +0000
> Responsible-Changed-Why:
> take
>
>
> State-Changed-From-To: open->feedback
> State-Changed-By: chs@NetBSD.org
> State-Changed-When: Wed, 04 May 2022 15:52:04 +0000
> State-Changed-Why:
> this should be fixed by src/external/cddl/osnet/dist/uts/common/fs/zfs/arc.c rev 1.21,
> can you confirm?
>
>
>
From: Frank Kardel <kardel@netbsd.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/55707 (ZFS seems to trigger a lot of xcalls)
Date: Thu, 5 May 2022 08:09:44 +0200
My testcase is a bareos (fork of bacula) incremental backup.
This run has a database (postgres) ohase and then a file system scan phase.
Previously there where high xcall rates.
I see significant improvements.
- xcalls load vanished
- IO seems to have improved
- the incremental backup reduced the run time from 48 minutes to 37
minutes
So far very good. But another peculiarity showed up.
pgdaemon gets very busy but does not start to scan.
top
load averages: 2.15, 1.00, 0.44; up 0+00:06:50 07:28:26
898 threads: 160 idle, 3 runnable, 725 sleeping, 8 zombie, 2 on CPU
CPU states: 0.9% user, 0.0% nice, 5.1% system, 0.2% interrupt, 93.6%
idle
Memory: 3732M Act, 1863M Inact, 34M Wired, 140M Exec, 2766M File, 41G Free
Swap: 544G Total, 544G Free
PID LID USERNAME PRI STATE TIME WCPU CPU NAME COMMAND
0 984 root 126 CPU/4 2:28 95.21% 95.21% pgdaemon [system]
2795 2795 pgsql 77 zfscv/0 1:09 37.30% 37.30% - postgres
0 8005 root 127 zfscv/0 0:01 4.98% 4.98% zio_write [system]
0 1733 root 96 zfscv/0 0:02 2.10% 2.10% zfs [system]
0 7047 root 127 zfscv/1 0:01 1.71% 1.71% zio_write [system]
0 1638 root 127 zfscv/3 0:01 0.98% 0.98% zio_write [system]
0 6601 root 126 zfscv/16 0:01 0.88% 0.88% zio_write [system]
0 6596 root 126 zfscv/18 0:01 0.83% 0.83% zio_write [system]
0 1632 root 127 zfscv/0 0:02 0.78% 0.78% zio_write [system]
0 6595 root 126 zfscv/1 0:01 0.73% 0.73% zio_write [system]
0 1736 root 126 zfscv/0 0:01 0.54% 0.54% zio_write [system]
0 29247 root 127 zfscv/6 0:00 0.34% 0.34% zio_rea*0 [system]
3824 3824 clamav 85 poll/3 0:14 0.00% 0.00% - clamd
vmstat -s
...
0 pages freed by daemon
0 pages scanned by daemon
0 anonymous pages scanned by daemon
0 object pages scanned by daemon
0 pages reactivated
0 pages found busy by daemon
0 total pending pageouts
949303 pages deactivated
...
Looks like pgdaemon gets woken up just to find out that there is nothing
to do.
So this is already very goog progress.
Thanks,
Frank
On 05/04/22 18:00, Frank Kardel wrote:
> The following reply was made to PR kern/55707; it has been noted by GNATS.
>
> From: Frank Kardel <kardel@kardel.name>
> To: gnats-bugs@netbsd.org
> Cc:
> Subject: Re: kern/55707 (ZFS seems to trigger a lot of xcalls)
> Date: Wed, 4 May 2022 17:56:49 +0200
>
> Good catch.
>
> Will test probably on the weekend.
>
> Frank
>
>
> On 05/04/22 17:52, chs@NetBSD.org wrote:
> > Synopsis: ZFS seems to trigger a lot of xcalls
> >
> > Responsible-Changed-From-To: kern-bug-people->chs
> > Responsible-Changed-By: chs@NetBSD.org
> > Responsible-Changed-When: Wed, 04 May 2022 15:52:04 +0000
> > Responsible-Changed-Why:
> > take
> >
> >
> > State-Changed-From-To: open->feedback
> > State-Changed-By: chs@NetBSD.org
> > State-Changed-When: Wed, 04 May 2022 15:52:04 +0000
> > State-Changed-Why:
> > this should be fixed by src/external/cddl/osnet/dist/uts/common/fs/zfs/arc.c rev 1.21,
> > can you confirm?
> >
> >
> >
>
From: Frank Kardel <kardel@netbsd.org>
To: gnats-bugs@netbsd.org, kern-bug-people <kern-bug-people@netbsd.org>
Cc:
Subject: Re: PR/55707 CVS commit:
src/external/cddl/osnet/dist/uts/common/fs/zfs
Date: Sat, 7 May 2022 20:54:13 +0200
Further testing gives:
- xcalls do not appear in significant counts
- pgdaemon runs all the time after a while
- zfs kernel pools (pool(9)) require a lot of memory (> 50% on a
64G machine) - somewhat expected ZFS behavior but seems not be limited
- processes using ZFS pools slow down and wait on vmem and
zio_buf... and other zfs waitchannels
- zfs pools (zfs abstraction) are unusable at this stage
So, while this fix looked good at start it brings the ZFS system down
after a while.
Looks like this needs more investigation.
Maybe this fix should be backed out until then.
Best regards,
Frank
vmstat -m
Memory resource pool statistics
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg
Maxpg Idle
amappl 88 348824 0 315902 865 132 733 735 0
inf 0
anonpl 40 8704909 0 4079863 52358 6398 45960 48565 0
inf 0
arc_buf_hdr_t_f 208 6907754 0 4497447 130397 3538 126859 126861 0
inf 0
arc_buf_t 40 2504702 0 2499075 3956 3815 141 656 0
inf 0
ataspl 160 161409436 0 161409436 1139 1138 1 3 0 inf 1
biopl 280 2144448 21 2144438 23004 23003 1 1136 0
inf 0
buf16k 16896 13776 0 13203 1692 1494 198 281 0
inf 0
buf1k 1536 168 0 168 83 82 1 83 0
inf 1
buf2k 2560 1668 0 1668 1650 1649 1 957 0
inf 1
buf32k 33280 643163 442 448649 414253 219738 194515 201563 0
inf 1
buf4k 4096 777170 0 665295 527831 415955 111876 140565 0
inf 1
buf512b 1024 16428 0 16427 1108 1107 1 930 0
inf 0
buf64k 65536 12 0 0 13 0 13 13 0
inf 1
buf8k 8704 26445 0 26122 925 876 49 131 0
inf 0
bufpl 280 355070 0 47772 23653 103 23550 23551 0
inf 0
cgdcpl 88 3076 0 3076 53 53 0 2 0 inf 0
cwdi 192 11265 0 10887 23 4 19 19 0
inf 0
dmu_buf_impl_t 216 8421890 0 8399719 130037 127513 2524 17241 0
inf 0
dnode_t 640 6484618 0 6467510 425627 421251 4376 49015 0
inf 0
execargs 262144 14923 0 14923 1126 1125 1 16 0
16 1
extent 48 68 0 27 1 0 1 1 0 inf 0
fcrpl 184 422 0 410 4 0 4 4 4 inf 3
fdfile 64 225465 0 213580 334 112 222 296 0
inf 0
ffsdino1 136 1458285 0 1458285 16421 16421 0 4393 0
inf 0
ffsdino2 264 20330927 0 20182146 75494 65542 9952 14695 0
inf 0
ffsino 280 21285904 0 21137123 63225 52565 10660 15745 0
inf 0
file 128 136397 0 133647 92 0 92 92 0
inf 0
filedesc 832 11245 0 10867 182 87 95 97 0
inf 0
fstlwp 128 27298 0 25867 48 1 47 47 0
inf 0
icmp 32 1693 0 1693 525 525 0 1 0 inf 0
icmp6 32 2609 0 2609 857 857 0 1 0 inf 0
igmppl 40 1 0 0 1 0 1 1 0 inf 0
in6pcbpl 296 2985970 0 2985919 97 93 4 5 0 inf 0
inmltpl 56 256 0 244 1 0 1 1 0 inf 0
inpcbpl 256 3136768 0 3136472 46 24 22 30 0
inf 0
kcpuset 64 21769 0 20881 15 0 15 15 0
inf 0
kcredpl 192 33317 0 32207 69 3 66 69 0
inf 0
kmem-00016 16 4031533 0 4001321 1519 1093 426 696 0
inf 0
kmem-00032 32 2975151 0 2957902 3766 3256 510 1189 0
inf 0
kmem-00064 128 6945586 0 6922587 27679 26212 1467 5814 0
inf 0
kmem-00128 192 4411922 0 4378652 23929 21347 2582 7658 0
inf 0
kmem-00192 256 3737248 0 3729453 40482 39292 1190 9243 0
inf 0
kmem-00256 320 2644706 0 2641925 21832 21462 370 2495 0
inf 0
kmem-00320 384 75860 0 71438 507 5 502 507 0
inf 0
kmem-00384 448 4936259 0 4919108 176119 174123 1996 9637 0
inf 0
kmem-00448 512 31893 0 30949 240 70 170 230 0
inf 0
kmem-00512 576 17429 0 16819 134 6 128 134 0
inf 0
kmem-00768 832 56191 0 44784 3372 515 2857 2865 0
inf 0
kmem-01024 1088 231485 0 195622 15212 3216 11996 12993 0
inf 0
kmem-02048 2112 510788 0 493001 225514 207725 17789 29465 0
inf 2
kmem-04096 4096 47753 0 47020 7331 6595 736 1483 0
inf 3
ksiginfo 136 62868 0 62865 165 164 1 14 0
inf 0
ktrace 128 1165 0 1165 38 38 0 37 0
inf 0
kva-12288 12288 1918 0 1901 4 1 3 4 0 inf 0
kva-16384 16384 630073 2404 564606 6162 2067 4095 4095 0
inf 0
kva-20480 20480 766 0 730 16 12 4 7 0 inf 0
kva-24576 24576 37 0 2 4 0 4 4 0 inf 0
kva-28672 28672 519 0 496 24 21 3 7 0 inf 0
kva-32768 32768 332243 130 292975 18232 13314 4918 4919 0
inf 0
kva-36864 36864 269 0 3 39 1 38 39 0
inf 0
kva-49152 49152 4 0 2 1 0 1 1 0 inf 0
kva-65536 65536 450540 38 413939 56288 47120 9168 14358 0
inf 0
kva-8192 8192 365573 9 246580 4797 1076 3721 3722 0
inf 0
llentrypl 280 1052 0 1039 2 1 1 2 0 inf 0
lockf 112 12985 0 12920 8 2 6 8 0 inf 0
lwppl 1088 18321 0 17038 585 127 458 567 0
inf 0
mbpl 520 1066408 0 1064010 2051 1635 416 580 0
inf 0
mclpl 2112 661475 0 659893 92254 90656 1598 2471 0
2094276 16
metaslab_alloc_ 72 1416870 0 1416870 6925 6925 0 23 0
inf 0
mutex 64 23895026 0 23744290 7072 4672 2400 3522 0
inf 0
namecache 192 20339689 0 20189973 10680 3541 7139 7161 0
inf 0
nfsnodepl 280 4 0 0 1 0 1 1 0 inf 0
nfsvapl 184 4 0 0 1 0 1 1 0 inf 0
npfcn4pl 144 14500 0 14493 238 237 1 100 0
inf 0
npfcn6pl 192 735 0 735 384 384 0 4 0 inf 0
npftblpl 48 2 0 0 1 0 1 1 0 inf 0
pcache 2752 208 0 4 204 0 204 204 0
inf 0
pcachecpu 128 6541 0 0 211 0 211 211 0
inf 0
pcglarge 1088 1517560 0 1517544 120376 120367 9 2593 0
inf 1
pcgnormal 320 48193604 0 48193514 367200 367189 11 5892 0
inf 0
pdict16 80 630 0 566 2 0 2 2 0 inf 0
pdict32 96 15 0 2 1 0 1 1 0 inf 0
pdppl 4096 11260 0 10902 5538 5178 360 374 0
inf 2
phpool-128 72 3766 0 3256 26 4 22 22 0
inf 0
tmpfs_node 232 38115 0 37964 57 46 11 57 0
inf 0
tstile 128 18321 0 17038 55 0 55 55 0
inf 0
uaoeltpl 104 1106 0 76 29 0 29 29 0
inf 0
uarea 24576 11079 0 10338 5043 4300 743 860 0
inf 2
uareasys 24576 8912 0 8370 6692 6150 542 1122 0
inf 0
ufsdir 272 1705 0 1704 629 628 1 2 0 inf 0
ufsdq 88 58 0 28 1 0 1 1 0 inf 0
vcachepl 640 23829477 0 23679536 139899 114898 25001 36871 0
inf 0
vmembt 64 3048084 0 2701999 6422 0 6422 6422 0
inf 0
vmmpepl 192 4391031 0 4332352 4595 1781 2814 2903 0
inf 9
vmsppl 352 11179 0 10822 41 7 34 34 0
inf 0
vndxpl 288 3081 0 3081 50 50 0 1 0 inf 0
wapbldealloc 40 4494 0 4400 366 365 1 2 0 inf 0
wapblentrypl 48 13636 0 13636 197 197 0 1 0 inf 0
wapblinopl 40 14138 0 14135 1 0 1 1 0 inf 0
xhcixfer 320 7 0 3 1 0 1 1 0 inf 0
xhcixfer 320 15 0 3 2 0 2 2 0 inf 0
xhcixfer 320 5 0 3 1 0 1 1 0 inf 0
zfs_znode_cache 248 4064949 0 4064639 82052 81955 97 8726 0
inf 0
zil_lwb_cache 208 14905 0 14874 259 257 2 45 0
inf 0
zio_buf_1024 1536 647135 1 464116 157658 66120 91538 98121 0
inf 0
zio_buf_10240 12288 92448 7 87488 32867 27907 4960 12109 0
inf 0
zio_buf_114688 118784 15195 1 15195 6536 6536 0 330 0
inf 0
zio_buf_12288 14336 87236 12 86034 28051 26849 1202 5257 0
inf 0
zio_buf_131072 131072 570625 14 570368 69254 68997 257 1072 0
inf 0
zio_buf_14336 16384 68628 10 67743 27309 26424 885 10181 0
inf 0
zio_buf_1536 2048 496380 2 371712 86587 24202 62385 63619 0
inf 0
zio_buf_16384 16384 989387 1571 986806 358824 356243 2581 13831 0
inf 0
zio_buf_2048 2560 576027 2 479449 245068 148490 96578 97788 0
inf 0
zio_buf_20480 24576 83274 33 82470 24472 23668 804 3309 0
inf 0
zio_buf_24576 28672 83183 39 82399 20161 19377 784 1065 0
inf 0
zio_buf_2560 3072 372835 3 286967 167567 81699 85868 109667 0
inf 0
zio_buf_28672 32768 61214 22 60860 18234 17880 354 615 0
inf 0
zio_buf_3072 3584 310332 9 155587 193755 39010 154745 155714 0
inf 0
zio_buf_32768 32768 57565 51 57206 17773 17414 359 660 0
inf 0
zio_buf_3584 4096 419924 3 189075 275481 44632 230849 232440 0
inf 0
zio_buf_4096 4096 570671 108 498494 200689 128512 72177 75547 0
inf 0
zio_buf_40960 45056 70018 18 69620 18892 18494 398 645 0
inf 0
zio_buf_49152 53248 63271 35 53681 23738 14148 9590 9749 0
inf 0
zio_buf_512 1024 6873852 0 6191358 434084 243580 190504 222715 0
inf 0
zio_buf_5120 6144 179047 20 170924 55403 47280 8123 12460 0
inf 0
zio_buf_57344 61440 50685 7 50391 16366 16072 294 585 0
inf 0
zio_buf_6144 7168 100309 11 95804 27781 23276 4505 6375 0
inf 0
zio_buf_512 1024 6873852 0 6191358 434084 243580 190504 222715 0
inf 0
zio_buf_5120 6144 179047 20 170924 55403 47280 8123 12460 0
inf 0
zio_buf_57344 61440 50685 7 50391 16366 16072 294 585 0
inf 0
zio_buf_6144 7168 100309 11 95804 27781 23276 4505 6375 0
inf 0
zio_buf_65536 65536 1639850 7 1639850 84010 84010 0 741 0
inf 0
zio_buf_7168 8192 100042 13 89334 32608 21900 10708 11389 0
inf 0
zio_buf_8192 8192 100777 12 91716 37259 28198 9061 9568 0
inf 0
zio_buf_81920 86016 24692 0 24692 8580 8580 0 270 0
inf 0
zio_buf_98304 102400 15669 0 15669 6950 6950 0 320 0
inf 0
zio_cache 992 3890686 0 3890594 90299 90247 52 1247 0
inf 0
zio_data_buf_10 1536 320653 0 279563 128878 108271 20607 20664 0
inf 0
zio_data_buf_10 12288 109129 0 88203 64760 43834 20926 20938 0
inf 0
zio_data_buf_11 118784 65455 0 64648 38245 37438 807 4384 0
inf 0
zio_data_buf_12 14336 89334 0 74479 51705 36850 14855 14866 0
inf 0
zio_data_buf_13 131072 2460355 0 2394583 667611 601839 65772 118796 0
inf 0
zio_data_buf_14 16384 73563 0 62210 42336 30983 11353 11365 0
inf 0
zio_data_buf_15 2048 199308 0 151679 76082 52267 23815 23815 0
inf 0
zio_data_buf_16 16384 62453 0 53757 33598 24902 8696 8713 0
inf 0
zio_data_buf_20 2560 172828 0 117171 133365 77708 55657 55661 0
inf 0
zio_data_buf_20 24576 115344 0 101825 74640 61121 13519 13530 0
inf 0
zio_data_buf_24 28672 120288 0 109807 85262 74781 10481 10517 0
inf 0
zio_data_buf_25 3072 135280 0 91557 100274 56551 43723 43726 0
inf 0
zio_data_buf_28 32768 84945 0 77602 55752 48409 7343 7353 0
inf 0
zio_data_buf_30 3584 106880 0 73745 75474 42339 33135 33139 0
inf 0
zio_data_buf_32 32768 82947 0 77325 55995 50373 5622 19870 0
inf 0
zio_data_buf_35 4096 91374 0 65481 64707 38814 25893 25896 0
inf 0
zio_data_buf_40 4096 77072 0 56626 52575 32129 20446 20446 0
inf 0
zio_data_buf_40 45056 152006 0 143329 110362 101685 8677 32285 0
inf 0
zio_data_buf_49 53248 114182 0 106732 75836 68386 7450 10019 0
inf 0
zio_data_buf_51 1024 807255 0 730101 169098 149800 19298 19298 0
inf 0
zio_data_buf_51 6144 118021 0 86095 81130 49204 31926 31928 0
inf 0
zio_data_buf_57 61440 101030 0 94940 71166 65076 6090 12156 0
inf 0
zio_data_buf_61 7168 97202 0 73880 63247 39925 23322 23330 0
inf 0
zio_data_buf_65 65536 116590 0 112490 92774 88674 4100 24211 0
inf 0
zio_data_buf_71 8192 79791 0 62525 47807 30541 17266 17276 0
inf 0
zio_data_buf_81 8192 72922 0 58911 41948 27937 14011 14017 0
inf 0
zio_data_buf_81 86016 104931 0 101590 67907 64566 3341 12569 0
inf 0
zio_data_buf_98 102400 73099 0 70443 41308 38652 2656 8877 0
inf 0
zio_link_cache 56 4359198 0 4359180 4159 4153 6 73 0
inf 0
Totals 2611133762 5065 2597562150 9387208 7128911 2258297
In use 26266313K, total allocated 34255628K; utilization 76.7%
On 05/04/22 17:50, Chuck Silvers wrote:
> The following reply was made to PR kern/55707; it has been noted by GNATS.
>
> From: "Chuck Silvers" <chs@netbsd.org>
> To: gnats-bugs@gnats.NetBSD.org
> Cc:
> Subject: PR/55707 CVS commit: src/external/cddl/osnet/dist/uts/common/fs/zfs
> Date: Wed, 4 May 2022 15:49:55 +0000
>
> Module Name: src
> Committed By: chs
> Date: Wed May 4 15:49:55 UTC 2022
>
> Modified Files:
> src/external/cddl/osnet/dist/uts/common/fs/zfs: arc.c
>
> Log Message:
> fix ARC checks for available memory:
> there's an extra check that we inherited from FreeBSD that tries to
> detect KVA exhaustion on platforms with limited KVA, but the condition
> that decided whether to use the extra check was using a FreeBSDism
> that doesn't exist on NetBSD, resulting in this check being used on
> all platforms. on amd64 systems with lots of memory, this extra check
> would result in the ARC thinking that it constantly needed to reclaim memory,
> resulting in all the xcall threads running all the time but not doing
> anything useful. change this condition so that this extra check for
> KVA exhaustion is only used on 32-bit platforms. fixes PR 55707.
>
>
> To generate a diff of this commit:
> cvs rdiff -u -r1.20 -r1.21 \
> src/external/cddl/osnet/dist/uts/common/fs/zfs/arc.c
>
> Please note that diffs are not public domain; they are subject to the
> copyright notices on the relevant files.
>
State-Changed-From-To: feedback->closed
State-Changed-By: kardel@NetBSD.org
State-Changed-When: Thu, 05 Jan 2023 09:06:42 +0000
State-Changed-Why:
usable
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.47 2022/09/11 19:34:41 kim Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2023
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home