NetBSD Problem Report #58666

From he@smistad.uninett.no  Sun Sep  8 07:43:00 2024
Return-Path: <he@smistad.uninett.no>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 1152E1A923B
	for <gnats-bugs@gnats.NetBSD.org>; Sun,  8 Sep 2024 07:43:00 +0000 (UTC)
Message-Id: <20240908074251.9B29D43F25F@smistad.uninett.no>
Date: Sun,  8 Sep 2024 09:42:51 +0200 (CEST)
From: he@NetBSD.org
Reply-To: he@NetBSD.org
To: gnats-bugs@NetBSD.org
Subject: panic: lock error: Reader / writer lock: rw_vector_enter,357: locking against myself
X-Send-Pr-Version: 3.95

>Number:         58666
>Category:       kern
>Synopsis:       panic: lock error: Reader / writer lock: rw_vector_enter,357: locking against myself
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    kern-bug-people
>State:          analyzed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Sep 08 07:45:00 +0000 2024
>Closed-Date:    
>Last-Modified:  Thu Sep 12 15:10:01 +0000 2024
>Originator:     Havard Eidnes
>Release:        NetBSD 10.0_STABLE
>Organization:
 I try...
>Environment:
System: NetBSD fxxxx.uninett.no 10.0_STABLE NetBSD 10.0_STABLE (GENERIC_PAE) #0: Sat Sep  7 21:01:01 CEST 2024  he@fxxxx.uninett.no:/usr/obj/sys/arch/i386/compile/GENERIC_PAE i386
Architecture: i386
Machine: i386
>Description:

	I recently upgraded this host from 9.3 to 10.0_STABLE, but sadly,
	this appears to have introduced instability.

	From the most recent rc.log:

savecore: reboot after panic: [ 24097.1118660] panic: lock error: Reader / writer lock: rw_vector_enter,357: locking against myself: lock 0xce5820ec cpu 1 lwp 0xcf091cc0

	and looking at the saved core dump with crash shows:

crash> bt
_KERNEL_OPT_MEMORY_RBFLAGS(104,0,cf091cc0,c120993c,c1282d38,f4023a10,c0cc9f38,104,0,0) at 0
kern_reboot(104,0,0,0,c159c020,c14ca138,ce5820ec,f4023a20,c0cc9ff1,c1282d38) at sys_reboot
vpanic(c1282d38,f4023a2c,f4023adc,c0cc282f,c1282d38,c1280716,c117fb30,165,c127f433,ce5820ec) at vpanic+0x1a6
panic(c1282d38,c1280716,c117fb30,165,c127f433,ce5820ec,1,cf091cc0,c117fb30,38356563) at vprintf
lockdebug_abort(c117fb30,165,ce5820ec,c14ca138,c127f433,f4023b68,c0c986a5,0,0,0) at lockdebug_abort+0x11f
rw_abort.constprop.0(0,0,0,0,0,0,0,0,0,0) at rw_abort.constprop.0+0x38
rw_vector_enter(ce5820ec,0,f4023ca4,c0c467c8,ce5820e8,80000001,0,0,c1a045ec,ffffffff) at rw_vector_enter+0x1e8
vm_map_lock_read(ce5820e8,80000001,0,0,c1a045ec,ffffffff,ce3eb9c0,c1a045b0,2ad90467,80000001) at vm_map_lock_read+0x1c
uvm_fault_internal(ce5820e8,0,2,0,ffffffff,ffffffff,4000,1723,f4023cd4,ce5820e8) at uvm_fault_internal+0xcf
trap() at trap+0x472
--- trap (number 6) ---
pmap_ctor(0,e02a0600,1,0,f0,0,c1507dc0,0,f4023e24,c0cc7b3a) at pmap_ctor+0xb7
pool_cache_get_slow(f4023e14,0,1,e02a0600,d0b9b4cc,41,d0b9b590,f4023e40,c049d8c3,c1507dc0) at pool_cache_get_slow+0xa3
pool_cache_get_paddr(c1507dc0,1,0,1,d0b9b4cc,f4023e60,c0c4e8b8,d0b9b4d8,3,0) at pool_cache_get_paddr+0xa3
pmap_create(d0b9b4d8,3,0,e0bbb7c0,cfa07840,0,f4023e8c,c0c4f386,1000,bf7ff000) at pmap_create+0x23
uvmspace_alloc(1000,bf7ff000,1,e0bbc364,cecbe994,41,e0bbb7c0,cfa07840,0,f4023ea4) at uvmspace_alloc+0x106
uvmspace_fork(ce5820e8,e0bbb7c0,e0bbb7c0,cfa07840,f4023f10,c0c7bbf8,cfa07840,e0bbb7c0,0,cfa07840) at uvmspace_fork+0x3f
uvm_proc_fork(cfa07840,e0bbb7c0,0,cfa07840,6a,0,0,106,cf091cc0,2) at uvm_proc_fork+0x43
fork1(cf091cc0,0,14,0,0,0,0,f4023f60,f4023f9c,c04a2116) at fork1+0x3ca
sys_fork(cf091cc0,f4023f68,f4023f60,e53f9000,28,2,f4023f60,f4023f68,0,0) at sys_fork+0x48
syscall() at syscall+0x1d6
--- syscall (number 2) ---
badc8f97:
crash> 

	Hmm, this kenrel is, as far as I can see, not compiled with LOCKDEBUG,
	it's a straight 10.0_STABLE GENERIC_PAE kernel.  The boot messages are:

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003,
    2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013,
    2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023,
    2024
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 10.0_STABLE (GENERIC_PAE) #0: Sat Sep  7 21:01:01 CEST 2024
        he@fxxxx.uninett.no:/usr/obj/sys/arch/i386/compile/GENERIC_PAE
total memory = 10239 MB
avail memory = 10019 MB
timecounter: Timecounters tick every 10.000 msec
Kernelized RAIDframe activated
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
mainbus0 (root)
ACPI: RSDP 0x00000000000FDFD0 000024 (v02 IBM   )
ACPI: XSDT 0x00000000BFFCED80 00004C (v01 IBM    SERDEFNT 00001000 IBM  45444F43)
ACPI: FACP 0x00000000BFFCECC0 000084 (v02 IBM    SERDEFNT 00001000 IBM  45444F43)
ACPI: DSDT 0x00000000BFFCB980 00250D (v02 IBM    SERDEFNT 00001000 INTL 20041203)
ACPI: FACS 0x00000000BFFCEA00 000040
ACPI: APIC 0x00000000BFFCEC00 000090 (v01 IBM    SERDEFNT 00001000 IBM  45444F43)
ACPI: SRAT 0x00000000BFFCEAC0 0000E8 (v01 IBM    SERDEFNT 00001000 IBM  45444F43)
ACPI: HPET 0x00000000BFFCEA80 000038 (v01 IBM    SERDEFNT 00001000 IBM  45444F43)
ACPI: MCFG 0x00000000BFFCEA40 00003C (v01 IBM    SERDEFNT 00001000 IBM  45444F43)
ACPI: 1 ACPI AML tables successfully acquired and loaded
ioapic0 at mainbus0 apid 14: pa 0xfec00000, version 0x20, 24 pins
ioapic1 at mainbus0 apid 13: pa 0xfec80000, version 0x20, 24 pins
cpu0 at mainbus0 apid 0
cpu0: Use lfence to serialize rdtsc
cpu0: Intel(R) Xeon(R) CPU           E5335  @ 2.00GHz, id 0x6fb
cpu0: node 0, package 0, core 0, smt 0
cpu1 at mainbus0 apid 1
cpu1: Intel(R) Xeon(R) CPU           E5335  @ 2.00GHz, id 0x6fb
cpu1: node 0, package 0, core 1, smt 0
cpu2 at mainbus0 apid 2
cpu2: Intel(R) Xeon(R) CPU           E5335  @ 2.00GHz, id 0x6fb
cpu2: node 0, package 0, core 2, smt 0
cpu3 at mainbus0 apid 3
cpu3: Intel(R) Xeon(R) CPU           E5335  @ 2.00GHz, id 0x6fb
cpu3: node 0, package 0, core 3, smt 0
acpi0 at mainbus0: Intel ACPICA 20221020
acpi0: X/RSDT: OemId <IBM   ,SERDEFNT,00001000>, AslId <IBM ,45444f43>
acpi0: MCFG: segment 0, bus 0-20, address 0x00000000e0000000
acpi0: SCI interrupting at int 9
acpi0: fixed power button present
timecounter: Timecounter "ACPI-Fast" frequency 3579545 Hz quality 1000
hpet0 at acpi0: high precision event timer (mem 0xfed00000-0xfed00400)
timecounter: Timecounter "hpet0" frequency 14318180 Hz quality 2000
COM1 (PNP0501) at acpi0 not configured
PIC2 (PNP0003) at acpi0 not configured
PIC3 (PNP0003) at acpi0 not configured
attimer1 at acpi0 (TMR, PNP0100): io 0x40-0x43 irq 0
pcppi1 at acpi0 (SPKR, PNP0800): io 0x61
midi0 at pcppi1: PC speaker
sysbeep0 at pcppi1
COPR (PNP0C04) at acpi0 not configured
apm0 at acpi0: Power Management spec V1.2
attimer1: attached to pcppi1
ipmi0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0: Intel 5000X Memory Controller Hub (rev. 0x31)
ppb0 at pci0 dev 2 function 0: Intel 5000 Series Chipset PCI Express x8 Port 2-3 (rev. 0x31)
ppb0: PCI Express capability version 1 <Root Port of PCI-E Root Complex> x8 @ 2.5GT/s
pci1 at ppb0 bus 16
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
ppb1 at pci1 dev 0 function 0: Intel 63xxESB PCI Express Upstream Port (rev. 0x01)
ppb1: PCI Express capability version 1 <Upstream Port of PCI-E Switch>
pci2 at ppb1 bus 17
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
ppb2 at pci2 dev 0 function 0: Intel 63xxESB PCI Express Downstream Port #1 (rev. 0x01)
ppb2: PCI Express capability version 1 <Downstream Port of PCI-E Switch> x8 @ 2.5GT/s
pci3 at ppb2 bus 19
pci3: i/o space, memory space enabled, rd/line, wr/inv ok
ppb3 at pci2 dev 1 function 0: Intel 63xxESB PCI Express Downstream Port #2 (rev. 0x01)
ppb3: PCI Express capability version 1 <Downstream Port of PCI-E Switch> x4 @ 2.5GT/s
pci4 at ppb3 bus 18
pci4: i/o space, memory space enabled
ppb4 at pci1 dev 0 function 3: Intel 63xxESB PCI Express to PCI-X Bridge (rev. 0x01)
ppb4: PCI Express capability version 1 <PCI-E to PCI/PCI-X Bridge>
pci5 at ppb4 bus 20
pci5: i/o space, memory space enabled, rd/line, wr/inv ok
ppb5 at pci0 dev 3 function 0: Intel 5000 Series Chipset PCI Express x4 Port 3 (rev. 0x31)
ppb5: PCI Express capability version 1 <Root Port of PCI-E Root Complex> x4 @ 2.5GT/s
pci6 at ppb5 bus 35
pci6: i/o space, memory space enabled
ppb6 at pci0 dev 4 function 0: Intel 5000 Series Chipset PCI Express x8 Port 4-5 (rev. 0x31)
ppb6: PCI Express capability version 1 <Root Port of PCI-E Root Complex> x8 @ 2.5GT/s
pci7 at ppb6 bus 7
pci7: i/o space, memory space enabled, rd/line, wr/inv ok
ppb7 at pci0 dev 5 function 0: Intel 5000 Series Chipset PCI Express x4 Port 5 (rev. 0x31)
ppb7: PCI Express capability version 1 <Root Port of PCI-E Root Complex> x4 @ 2.5GT/s
pci8 at ppb7 bus 34
pci8: i/o space, memory space enabled
ppb8 at pci0 dev 6 function 0: Intel 5000 Series Chipset PCI Express x4 Port 6 (rev. 0x31)
ppb8: PCI Express capability version 1 <Root Port of PCI-E Root Complex> x4 @ 2.5GT/s
pci9 at ppb8 bus 3
pci9: i/o space, memory space enabled, rd/line, wr/inv ok
ppb9 at pci9 dev 0 function 0: ServerWorks BCM5714/BCM5715 Integral PCI-E to PCI-X Bridge (rev. 0xc3)
ppb9: PCI Express capability version 1 <PCI-E to PCI/PCI-X Bridge>
pci10 at ppb9 bus 4
pci10: i/o space, memory space enabled, rd/line, wr/inv ok
bnx0 at pci10 dev 0 function 0: Broadcom NetXtreme II BCM5708 1000Base-T
bnx0: /usr/src/sys/dev/pci/if_bnx.c(722): Management firmware enabled but not running!
bnx0: Ethernet address 00:1a:64:94:71:cc
bnx0: ASIC BCM5708 B2 (0x57081020)
bnx0: PCI-X 64bit 133MHz
bnx0: B/C (3.4.4); Bufs (RX:2;TX:2); Flags (MFW); MFW (NOT RUNNING!)
bnx0: Coal (RX:6,6,18,18; TX:20,20,80,80)
brgphy0 at bnx0 phy 1: BCM5708C 1000BASE-T media interface, rev. 6
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bnx0: interrupting at ioapic0 pin 18
ppb10 at pci0 dev 7 function 0: Intel 5000 Series Chipset PCI Express x4 Port 7 (rev. 0x31)
ppb10: PCI Express capability version 1 <Root Port of PCI-E Root Complex> x4 @ 2.5GT/s
pci11 at ppb10 bus 2
pci11: i/o space, memory space enabled, rd/line, wr/inv ok
aac0 at pci11 dev 0 function 0: IBM ServeRAID 8k
aac0: interrupting at ioapic0 pin 17
aac0: Enabling 64-bit address support
aac0: Enable 64-bit array support
aac0: New comm. interface enabled
aac0: MIPS 5KC at 250MHz, 32MB mem (16MB cache), optional battery not installed
aac0: Kernel 5.2-0 [Build 11844], Monitor 5.2-0 [Build 11844], S/N 812619
aac0: Controller supports: 0x17055d<SNAPSHOT,WCACHE,DATA64,HOSTTIME,WINDOW4GB,SOFTERR,SGMAP64>
ld0 at aac0 unit 0: RAID 1 (Mirror)
ld0: 232 GB, 30377 cyl, 255 head, 63 sec, 512 bytes/sect x 488017920 sectors
Intel 5000 Series Chipset DMA Engine (miscellaneous system, revision 0x31) at pci0 dev 8 function 0 not configured
pchb1 at pci0 dev 16 function 0: Intel 5000 Series Chipset FSB Registers (rev. 0x31)
pchb2 at pci0 dev 16 function 1: Intel 5000 Series Chipset FSB Registers (rev. 0x31)
pchb3 at pci0 dev 16 function 2: Intel 5000 Series Chipset FSB Registers (rev. 0x31)
pchb4 at pci0 dev 17 function 0: Intel 5000 Series Chipset Reserved Registers (rev. 0x31)
pchb5 at pci0 dev 19 function 0: Intel 5000 Series Chipset Reserved Registers (rev. 0x31)
pchb6 at pci0 dev 21 function 0: Intel 5000 Series Chipset FBD Registers (rev. 0x31)
pchb7 at pci0 dev 22 function 0: Intel 5000 Series Chipset FBD Registers (rev. 0x31)
ppb11 at pci0 dev 28 function 0: Intel 63xxESB PCI Express Port #1 (rev. 0x09)
ppb11: PCI Express capability version 1 <Root Port of PCI-E Root Complex> x1 @ 2.5GT/s
pci12 at ppb11 bus 5
pci12: i/o space, memory space enabled, rd/line, wr/inv ok
ppb12 at pci12 dev 0 function 0: ServerWorks BCM5714/BCM5715 Integral PCI-E to PCI-X Bridge (rev. 0xc3)
ppb12: PCI Express capability version 1 <PCI-E to PCI/PCI-X Bridge>
pci13 at ppb12 bus 6
pci13: i/o space, memory space enabled, rd/line, wr/inv ok
bnx1 at pci13 dev 0 function 0: Broadcom NetXtreme II BCM5708 1000Base-T
bnx1: /usr/src/sys/dev/pci/if_bnx.c(722): Management firmware enabled but not running!
bnx1: Ethernet address 00:1a:64:94:71:ce
bnx1: ASIC BCM5708 B2 (0x57081020)
bnx1: PCI-X 64bit 133MHz
bnx1: B/C (3.4.4); Bufs (RX:2;TX:2); Flags (MFW); MFW (NOT RUNNING!)
bnx1: Coal (RX:6,6,18,18; TX:20,20,80,80)
brgphy1 at bnx1 phy 1: BCM5708C 1000BASE-T media interface, rev. 6
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bnx1: interrupting at ioapic0 pin 16
uhci0 at pci0 dev 29 function 0: Intel 63xxESB USB UHCI Controller (rev. 0x09)
uhci0: interrupting at ioapic0 pin 23
usb0 at uhci0: USB revision 1.0
uhci1 at pci0 dev 29 function 1: Intel 63xxESB USB UHCI Controller (rev. 0x09)
uhci1: interrupting at ioapic0 pin 22
usb1 at uhci1: USB revision 1.0
uhci2 at pci0 dev 29 function 2: Intel 63xxESB USB UHCI Controller (rev. 0x09)
uhci2: interrupting at ioapic0 pin 23
usb2 at uhci2: USB revision 1.0
ehci0 at pci0 dev 29 function 7: Intel 63xxESB USB EHCI Controller (rev. 0x09)
ehci0: 64-bit DMA - limited
ehci0: interrupting at ioapic0 pin 23
ehci0: EHCI version 1.0
ehci0: 3 companion controllers, 2 ports each: uhci0 uhci1 uhci2
ehci0: Using DMA subregion for control data structures
usb3 at ehci0: USB revision 2.0
ppb13 at pci0 dev 30 function 0: Intel 82801BA Hub-PCI Bridge (rev. 0xd9)
pci14 at ppb13 bus 1
pci14: i/o space, memory space enabled
vga0 at pci14 dev 1 function 0: ATI Technologies ES1000 (rev. 0x02)
wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
drm at vga0 not configured
ichlpcib0 at pci0 dev 31 function 0: Intel 63xxESB LPC Interface Bridge (rev. 0x09)
timecounter: Timecounter "ichlpcib0" frequency 3579545 Hz quality 1000
ichlpcib0: 24-bit timer
tco0 at ichlpcib0: TCO (watchdog) timer configured.
tco0: Min/Max interval 1/367 seconds
piixide0 at pci0 dev 31 function 1: Intel 631xESB/632xESB IDE Controller (rev. 0x09)
piixide0: bus-master DMA support present
piixide0: primary channel configured to compatibility mode
piixide0: primary channel interrupting at ioapic0 pin 14
atabus0 at piixide0 channel 0
piixide0: secondary channel configured to compatibility mode
piixide0: secondary channel ignored (disabled)
ichsmb0 at pci0 dev 31 function 3: Intel 63xxESB SMBus Controller (rev. 0x09)
ichsmb0: interrupting at ioapic0 pin 20
iic0 at ichsmb0: I2C bus
isa0 at ichlpcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, 16-byte FIFO
pckbc0 at isa0 port 0x60-0x64
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
acpicpu0 at cpu0: ACPI CPU
acpicpu0: C1: HLT, lat   0 us, pow     0 mW
acpicpu0: T0: I/O, lat   1 us, pow     0 mW, 100 %
acpicpu0: T1: I/O, lat   1 us, pow     0 mW,  88 %
acpicpu0: T2: I/O, lat   1 us, pow     0 mW,  76 %
acpicpu0: T3: I/O, lat   1 us, pow     0 mW,  64 %
acpicpu0: T4: I/O, lat   1 us, pow     0 mW,  52 %
acpicpu0: T5: I/O, lat   1 us, pow     0 mW,  40 %
acpicpu0: T6: I/O, lat   1 us, pow     0 mW,  28 %
acpicpu0: T7: I/O, lat   1 us, pow     0 mW,  16 %
coretemp0 at cpu0: thermal sensor, 1 C resolution
coretemp0: autoconfiguration error: WARNING: Tjmax(0) retrieved was below expected range, using default (100).
acpicpu1 at cpu1: ACPI CPU
coretemp1 at cpu1: thermal sensor, 1 C resolution
coretemp1: autoconfiguration error: WARNING: Tjmax(0) retrieved was below expected range, using default (100).
acpicpu2 at cpu2: ACPI CPU
coretemp2 at cpu2: thermal sensor, 1 C resolution
coretemp2: autoconfiguration error: WARNING: Tjmax(0) retrieved was below expected range, using default (100).
acpicpu3 at cpu3: ACPI CPU
coretemp3 at cpu3: thermal sensor, 1 C resolution
coretemp3: autoconfiguration error: WARNING: Tjmax(0) retrieved was below expected range, using default (100).
WARNING: system needs entropy for security; see entropy(7)
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
uhub0 at usb1: NetBSD (0x0000) UHCI root hub (0x0000), class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhub1 at usb0: NetBSD (0x0000) UHCI root hub (0x0000), class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
atapibus0 at atabus0: 2 targets
uhub2 at usb2: NetBSD (0x0000) UHCI root hub (0x0000), class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhub3 at usb3: NetBSD (0x0000) EHCI root hub (0x0000), class 9/0, rev 2.00/1.00, addr 1
uhub3: 6 ports with 6 removable, self powered
cd0 at atapibus0 drive 0: <UJDA770 DVD/CDRW, , 1.24> cdrom removable
IPsec: Initialized Security Association Processing.
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33)
cd0(piixide0:0:0): using PIO mode 4, DMA mode 2 (using DMA)
uhub4 at uhub3 port 3: vendor 05e3 (0x05e3) USB2.0 Hub (0x0607), class 9/0, rev 2.00/8.05, addr 2
uhub4: multiple transaction translators
uhub4: 4 ports with 4 removable, self powered
uhidev0 at uhub1 port 2 configuration 1 interface 0
uhidev0: Generic (0x0a81) USB (0x0103), rev 1.10/1.10, addr 2, iclass 3/1
ukbd0 at uhidev0
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub1 port 2 configuration 1 interface 1
uhidev1: Generic (0x0a81) USB (0x0103), rev 1.10/1.10, addr 2, iclass 3/1
uhidev1: 3 report ids
ums0 at uhidev1 reportid 1: 5 buttons and Z dir
wsmouse0 at ums0 mux 0
uhid0 at uhidev1 reportid 2: input=2, output=0, feature=0
uhid1 at uhidev1 reportid 3: input=1, output=0, feature=0
uhub5 at uhub4 port 1: vendor 05e3 (0x05e3) USB2.0 Hub (0x0607), class 9/0, rev 2.00/8.05, addr 3
uhub5: multiple transaction translators
uhub5: 4 ports with 4 removable, self powered
uftdi0 at uhub5 port 1 configuration 1 interface 0
uftdi0: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 4
ucom0 at uftdi0 portno 1
uftdi1 at uhub5 port 1 configuration 1 interface 1
uftdi1: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 4
ucom1 at uftdi1 portno 2
uftdi2 at uhub5 port 1 configuration 1 interface 2
uftdi2: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 4
ucom2 at uftdi2 portno 3
uftdi3 at uhub5 port 1 configuration 1 interface 3
uftdi3: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 4
ucom3 at uftdi3 portno 4
uftdi4 at uhub5 port 2 configuration 1 interface 0
uftdi4: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 5
ucom4 at uftdi4 portno 1
uftdi5 at uhub5 port 2 configuration 1 interface 1
uftdi5: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 5
ucom5 at uftdi5 portno 2
uftdi6 at uhub5 port 2 configuration 1 interface 2
uftdi6: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 5
ucom6 at uftdi6 portno 3
uftdi7 at uhub5 port 2 configuration 1 interface 3
uftdi7: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 5
ucom7 at uftdi7 portno 4
uftdi8 at uhub5 port 3 configuration 1 interface 0
uftdi8: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 6
ucom8 at uftdi8 portno 1
uftdi9 at uhub5 port 3 configuration 1 interface 1
uftdi9: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 6
ucom9 at uftdi9 portno 2
uftdi10 at uhub5 port 3 configuration 1 interface 2
uftdi10: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 6
ucom10 at uftdi10 portno 3
uftdi11 at uhub5 port 3 configuration 1 interface 3
uftdi11: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 6
ucom11 at uftdi11 portno 4
uftdi12 at uhub5 port 4 configuration 1 interface 0
uftdi12: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 7
ucom12 at uftdi12 portno 1
uftdi13 at uhub5 port 4 configuration 1 interface 1
uftdi13: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 7
ucom13 at uftdi13 portno 2
uftdi14 at uhub5 port 4 configuration 1 interface 2
uftdi14: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 7
ucom14 at uftdi14 portno 3
uftdi15 at uhub5 port 4 configuration 1 interface 3
uftdi15: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 7
ucom15 at uftdi15 portno 4
uhub6 at uhub4 port 2: vendor 05e3 (0x05e3) USB2.0 Hub (0x0607), class 9/0, rev 2.00/8.05, addr 8
uhub6: multiple transaction translators
uhub6: 4 ports with 4 removable, self powered
uftdi16 at uhub6 port 1 configuration 1 interface 0
uftdi16: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 9
ucom16 at uftdi16 portno 1
uftdi17 at uhub6 port 1 configuration 1 interface 1
uftdi17: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 9
ucom17 at uftdi17 portno 2
uftdi18 at uhub6 port 1 configuration 1 interface 2
uftdi18: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 9
ucom18 at uftdi18 portno 3
uftdi19 at uhub6 port 1 configuration 1 interface 3
uftdi19: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 9
ucom19 at uftdi19 portno 4
uftdi20 at uhub6 port 2 configuration 1 interface 0
uftdi20: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 10
ucom20 at uftdi20 portno 1
uftdi21 at uhub6 port 2 configuration 1 interface 1
uftdi21: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 10
ucom21 at uftdi21 portno 2
uftdi22 at uhub6 port 2 configuration 1 interface 2
uftdi22: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 10
ucom22 at uftdi22 portno 3
uftdi23 at uhub6 port 2 configuration 1 interface 3
uftdi23: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 10
ucom23 at uftdi23 portno 4
uftdi24 at uhub6 port 3 configuration 1 interface 0
uftdi24: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 11
ucom24 at uftdi24 portno 1
uftdi25 at uhub6 port 3 configuration 1 interface 1
uftdi25: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 11
ucom25 at uftdi25 portno 2
uftdi26 at uhub6 port 3 configuration 1 interface 2
uftdi26: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 11
ucom26 at uftdi26 portno 3
uftdi27 at uhub6 port 3 configuration 1 interface 3
uftdi27: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 11
ucom27 at uftdi27 portno 4
uftdi28 at uhub6 port 4 configuration 1 interface 0
uftdi28: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 12
ucom28 at uftdi28 portno 1
uftdi29 at uhub6 port 4 configuration 1 interface 1
uftdi29: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 12
ucom29 at uftdi29 portno 2
uftdi30 at uhub6 port 4 configuration 1 interface 2
uftdi30: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 12
ucom30 at uftdi30 portno 3
uftdi31 at uhub6 port 4 configuration 1 interface 3
uftdi31: FTDI (0x0403) USB Fast Serial Adapter (0x6011), rev 2.00/8.00, addr 12
ucom31 at uftdi31 portno 4
waiting for devices: ipmi0
ipmi0: version 2.0 interface KCS iobase 0xca8/0x8 spacing 4
WARNING: system needs entropy for security; see entropy(7)
swwdog0: software watchdog initialized
WARNING: 4 errors while detecting hardware; check system log.
boot device: ld0
root on ld0a dumps on ld0b
/: replaying log to memory
root file system type: ffs
kern.module.path=/stand/i386/10.0/modules
WARNING: NVRAM century is 1 but RTC year is 2024
entropy: best effort
/: replaying log to disk
/var: replaying log to disk
entropy: ready
/usr: replaying log to disk
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)

	Hmm, looking at the "ps" output in crash reveals that quite
	a number of cron processes are (stuck?) in "tstile" state:

# cd /var/crash
# crash -N netbsd.2 -M netbsd.2.core
Crash version 10.0, image version 10.0_STABLE.
WARNING: versions differ, you may not be able to examine this image.
crash: _kvm_kvatop(0)
Kernel compiled without options LOCKDEBUG.
System panicked: lock error: Reader / writer lock: rw_vector_enter,357: locking against myself: lock 0xce5820ec cpu 1 lwp 0xcf091cc0
Backtrace from time of crash is available.
crash> 
crash> show lock 0xce5820ec
Sorry, no record of a lock with address 0xce5820ec found.
crash> 
crash> ps
PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
16815 16815 3   3         0           d0b86c80               cron tstile
5947  5947 3   3         0           de46f4c0               cron vfork
16462 16462 2   2        40           de459180               cron
12011 12011 2   0        40           d0717080               cron
16059 16059 3   3         0           ddc3bd00               cron tstile
16562 16562 3   3         0           de45dcc0               cron vfork
12873 12873 2   2        40           de458c40               cron
17462 17462 2   0        40           de481a80               cron
15584 15584 3   3         0           ddc3ba40               cron tstile
16707 16707 3   3         0           dde23040               cron vfork
14368 14368 3   1         0           d06bd5c0               cron vm_map
16615 16615 2   0        40           ddc3b200               cron
15669 15669 3   3         0           deee0180               cron tstile
16450 16450 3   2         0           deee2cc0               cron tstile
18738 18738 3   3         0           de458400               cron vfork
10938 10938 3   2         0           d0b86440               cron vfork
13544 13544 3   3         0           d06bdb40               cron tstile
12772 12772 3   2         0           deee2480               cron tstile
16093 16093 3   1         0           deee2a00               cron tstile
10735 10735 3   1         0           cf0911c0               cron vfork
26718 26718 3   3         0           de459440               cron vfork
17447 17447 3   2         0           de45d480               cron vfork
12476 12476 3   2         0           d06bd300               cron tstile
13804 13804 3   3         0           d0592a80               cron tstile
8503  8503 3   3         0           d0ae8400               cron vfork
15954 15954 3   2         0           deede940               cron vfork
14157 14157 3   2         0           d088f100               cron tstile
15956 15956 3   3         0           dde7e340               cron tstile
16626 16626 3   3         0           d0b86180               cron vfork
16682 16682 3   2         0           de457c00               cron vfork
15706 15706 3   2         0           de431380               cron tstile
15946 15946 3   3         0           d0ae8c40               cron tstile
14384 14384 3   3         0           de541040               cron vfork
14419 14419 3   2         0           ddcc8240               cron vfork
15334 15334 3   3         0           d0856bc0               cron tstile
14756 14756 3   2         0           de46f780               cron tstile
14483 14483 3   3         0           de45d740               cron vfork
16028 16028 3   2         0           deedfc40               cron vfork
13545 13545 3   3         0           d0856900               cron tstile
14371 14371 3   2         0           deedf6c0               cron tstile
15064 15064 3   3         0           ddc1e480               cron vfork
17909 17909 3   2         0           de45da00               cron vfork
15341 15341 3   2         0           deee2740               cron tstile
16417 16417 3   3         0           d0856640               cron tstile
15750 15750 3   1         0           dde235c0               cron tstile
14076 14076 3   1         0           ddc1ecc0               cron vfork
15160 15160 3   3         0           dde23880               cron vfork
16813 16813 3   2         0           ce4ef300               cron vfork
14116 14116 3   3         0           deedf400               cron tstile
836    836 3   2         0           deee0440               cron tstile
838    838 3   3         0           ddc1e740               cron vfork
12350 12350 3   2         0           deee09c0               cron vfork
17270 17270 3   2         0           dde7e8c0               cron tstile
16961 16961 3   3         0           d0856380               cron tstile
15936 15936 3   3         0           deee0c80               cron vfork
17401 17401 3   2         0           deedf140               cron vfork
14230 14230 2   0        40           ddcc8500                 sh
16628 16628 3   2         0           deedec00         python3.11 tstile
18358 18358 3   3       180           de457680               cron pipe_rd
14769 14769 3   3       180           de431bc0               cron pipe_rd
16795 16795 2   2        40           de458140               find
18606 18606 3   0       180           deede680           postdrop netio
15091 15091 3   3       180           ddc3b4c0           sendmail pipe_rd
17022 17022 3   2       180           de5415c0                tee pipe_rd
15162 15162 3   1       180           dde7eb80                 sh wait
16775 16775 3   1       180           deee21c0                 sh wait
16375 16375 3   0       180           deedf980               cron pipe_rd
6775  6775 3   1       180           d088f3c0               tcsh ttyraw
925    925 2   2       140           d088fc00             systat
325    325 3   0       180           cea33c00               tail kqueue
313    313 3   0       180           d08560c0               tcsh pause
2227  2227 3   1       180           cebad140               tcsh pause
2093  2093 3   0       180           d0717b80               tcsh pause
1836  1836 3   2       180           d07178c0              xterm select
1705  1705 3   2       180           ce4efb40               tcsh pause
296    296 2   1       100           ce4ef040               sshd
806    806 3   0       180           ce4ef880               sshd poll
2085  2085 3   0       180           ce572080              xterm select
2210  2210 3   1       180           ce474d40               tcsh pause
1953  1953 3   3       180           d0717600               sshd poll
415    415 3   3       180           ce572340               sshd poll
414    414 3   1       1c0           cde460c0              xterm select
401    401 3   0       180           cea33940               tcsh pause
400    400 2   1       100           cec949c0               sshd
396    396 3   3       180           d05927c0              getty ttyraw
395    395 3   0       180           d0592500              getty ttyraw
394    394 3   2       180           d0592240              getty ttyraw
393    393 3   1       180           ce4747c0              getty ttyraw
1708   387 2   0  10000140           ce5728c0           collectd
1708   386 2   2  10000140           cde46640           collectd
1708   385 2   1       100           ce572600           collectd
1708   384 2   1       100           cde46bc0           collectd
1708 >1982 7   3  10000140           cea33100           collectd
1708  1706 3   1       180           cea333c0           collectd parked
1708  2038 3   2       180           cea33680           collectd parked
1708  2165 3   3       180           cebad980           collectd parked
1708  1624 3   0       180           cebad400           collectd parked
1708  2168 3   2       180           cec94c80           collectd parked
1708  1708 2   2       140           d02f7d00           collectd
2164  2164 2   1       100           d02f7780          conserver
1790  1790 3   0       180           cf091a00               sshd poll
1321  1321 2   2       100           d02f7200          conserver
1973  1973 3   2       180           cf091480          conserver select
1906 >1906 7   1       140           cf091cc0               cron
1841  1841 3   0       180           cf091740              inetd kqueue
1826  1826 2   2   1000040           ce3538c0               qmgr
1843  1843 2   2   1000040           cec94180             master
1327  1327 3   3       180           cec94440               sshd poll
1367  1367 3   1       180           cebadc40             powerd kqueue
1184  1184 3   1       180           cebad6c0               ntpd pause
683    683 3   1       180           cde46900            unbound kqueue
678    678 2   0       140           ce572b80            syslogd
1        1 3   0       180           ce3e44c0               init wait
0      176 3   1       200           ce3e4780            physiod physiod
0      220 3   2       200           ce474500          pooldrain pooldrain
0      219 2   3       240           ce474240            ioflush
0      218 3   0       200           ce3e4d00           pgdaemon pgdaemon
0      217 2   0       200           ce3e4a40              ipmi0
0      214 3   0       200           ce353600          swwreboot swwreboot
0      213 2   2       240           ce353b80             npfgc0
0      212 3   1       200           ce3e4200            rt_free rt_free
0      211 3   1       200           ce390cc0              unpgc unpgc
0      210 3   1       200           ce2ff5c0    key_timehandler key_timehandler
0      209 3   3       200           ce390a00    icmp6_wqinput/3 icmp6_wqinput
0      208 3   2       200           ce390740    icmp6_wqinput/2 icmp6_wqinput
0      207 3   1       200           ce390480    icmp6_wqinput/1 icmp6_wqinput
0      206 2   0       200           ce3901c0    icmp6_wqinput/0
0      205 2   2       200           ce38ac80          nd6_timer
0      204 3   3       200           ce38a9c0    carp6_wqinput/3 carp6_wqinput
0      203 3   2       200           ce38a700    carp6_wqinput/2 carp6_wqinput
0      202 3   1       200           ce38a440    carp6_wqinput/1 carp6_wqinput
0      201 3   0       200           ce38a180    carp6_wqinput/0 carp6_wqinput
0      200 3   3       200           ce381c40     carp_wqinput/3 carp_wqinput
0      199 3   2       200           ce381980     carp_wqinput/2 carp_wqinput
0      198 3   1       200           ce3816c0     carp_wqinput/1 carp_wqinput
0      197 3   0       200           ce381400     carp_wqinput/0 carp_wqinput
0      196 3   3       200           ce381140     icmp_wqinput/3 icmp_wqinput
0      195 3   2       200           ce36ac00     icmp_wqinput/2 icmp_wqinput
0      194 3   1       200           ce36a940     icmp_wqinput/1 icmp_wqinput
0      193 3   0       200           ce36a680     icmp_wqinput/0 icmp_wqinput
0      192 3   1       200           ce36a3c0           rt_timer rt_timer
0      180 3   1       200           ce353340        vmem_rehash vmem_rehash
0      185 2   0       240           ce2ff880               usb3
0      183 2   3       240           ce353080               usb1
0      184 2   2       240           ce36a100               usb2
0       31 3   3       200           ce2ffb40          atapibus0 sccomp
0       63 3   1       200           ce358bc0               usb0 usbevt
0      125 2   2       200           ce358900          coretemp3
0      124 2   1       200           ce358640          coretemp2
0      123 2   3       200           ce358380          coretemp1
0      122 2   2       200           ce3580c0          coretemp0
0      113 3   1       200           ce2ff300          entbutler entropy
0      112 3   0       200           ce2ff040            atabus0 atath
0      111 3   0       200           cdf3ed40         usbtask-dr usbtsk
0      110 3   0       200           cdf3ea80         usbtask-hc usbtsk
0      109 3   3       200           cdf3e7c0               bnx1 bnx1
0      108 3   0       200           cdf3e500               bnx0 bnx0
0    > 107 7   2       240           cdf3e240              ipmi0
0    > 106 7   0       240           cdf35d00               apm0
0      105 3   3       200           cdf35a40            xcall/3 xcall
0      104 1   3       200           cdf35780          softser/3
0      103 1   3       200           cdf354c0          softclk/3
0      102 1   3       200           cdf35200          softbio/3
0      101 1   3       200           cdf32cc0          softnet/3
0      100 1   3       201           cdf32a00             idle/3
0       99 3   2       200           cdf32740            xcall/2 xcall
0       98 1   2       200           cdf32480          softser/2
0       97 1   2       200           cdf321c0          softclk/2
0       96 1   2       200           cdf2ec80          softbio/2
0       30 1   2       200           cdf2e9c0          softnet/2
0       29 1   2       201           cdf2e700             idle/2
0       28 3   1       200           cdf2e440            xcall/1 xcall
0       27 1   1       200           cdf2e180          softser/1
0       26 1   1       200           cdf0ac40          softclk/1
0       25 1   1       200           cdf0a980          softbio/1
0       24 1   1       200           cdf0a6c0          softnet/1
0       23 1   1       201           cdf0a400             idle/1
0       22 3   0       200           cdf0a140           lnxsyswq lnxsyswq
0       21 3   0       200           cdf09c00           lnxubdwq lnxubdwq
0       20 3   0       200           cdf09940           lnxpwrwq lnxpwrwq
0       19 3   0       200           cdf09680           lnxlngwq lnxlngwq
0       18 3   1       200           cdf093c0           lnxhipwq lnxhipwq
0       17 3   2       200           cdf09100           lnxrcugc lnxrcugc
0       16 3   0       240           cdf04bc0             sysmon smtaskq
0       15 3   0       200           cdf04900         pmfsuspend pmfsuspend
0       14 3   3       200           cdf04640           pmfevent pmfevent
0       13 3   0       200           cdf04380         sopendfree sopendfr
0       12 3   2       200           cdf040c0             ifwdog ifwdog
0       11 3   1       200           cdc38b80            iflnkst iflnkst
0       10 3   0       200           cdc388c0           nfssilly nfssilly
0        9 3   3       240           cdc38600             vdrain vdrain
0        8 3   0       200           cdc38340          modunload mod_unld
0        7 3   0       200           cdc38080            xcall/0 xcall
0        6 1   0       200           cdc36b40          softser/0
0        5 3   0       200           cdc36880          softclk/0 tstile
0        4 1   0       200           cdc365c0          softbio/0
0        3 3   0       200           cdc36300          softnet/0 tstile
0        2 1   0       201           cdc36040             idle/0
0        0 3   1       200           c14c9900            swapper uvm
crash> 
crash> show tstiles
  PID   LID          COMMAND      WAITING-FOR     WAIT-CHANNEL
16815 16815             cron         ddcc8500         d85d0a40
16059 16059             cron         ddcc8500         d85d0a40
15584 15584             cron         ddcc8500         d85d0a40
15669 15669             cron         ddcc8500         d85d0a40
16450 16450             cron         ddcc8500         d85d0a40
13544 13544             cron         ddcc8500         d85d0a40
12772 12772             cron         ddcc8500         d85d0a40
16093 16093             cron         ddcc8500         d85d0a40
12476 12476             cron         ddcc8500         d85d0a40
13804 13804             cron         ddcc8500         d85d0a40
14157 14157             cron         ddcc8500         d85d0a40
15956 15956             cron         ddcc8500         d85d0a40
15706 15706             cron         ddcc8500         d85d0a40
15946 15946             cron         ddcc8500         d85d0a40
15334 15334             cron         ddcc8500         d85d0a40
14756 14756             cron         ddcc8500         d85d0a40
13545 13545             cron         ddcc8500         d85d0a40
14371 14371             cron         ddcc8500         d85d0a40
15341 15341             cron         ddcc8500         d85d0a40
16417 16417             cron         ddcc8500         d85d0a40
15750 15750             cron         ddcc8500         d85d0a40
14116 14116             cron         ddcc8500         d85d0a40
  836   836             cron         ddcc8500         d85d0a40
17270 17270             cron         ddcc8500         d85d0a40
16961 16961             cron         ddcc8500         d85d0a40
16628 16628       python3.11         ddcc8500         d85d0a40
    0     5           system         cdf321c0         cdc21080
    0     3           system         cdf321c0         cdc21080
crash> 

	And some more decoding (I don't know if this is useful...):

crash> x/x d85d0a40
d85d0a40:       ddcc8505
crash> 
d85d0a44:       0
crash> 
d85d0a48:       0
crash> 
d85d0a4c:       0
crash> 
d85d0a50:       0
crash> 
d85d0a54:       0
crash> 
d85d0a58:       0
crash> 
d85d0a5c:       0
crash> 
d85d0a60:       0
crash> 
d85d0a64:       0
crash> 
d85d0a68:       0
crash> 
d85d0a6c:       0
crash> 
d85d0a70:       0
crash> 
d85d0a74:       0
crash> 
d85d0a78:       0
crash> 
d85d0a7c:       0
crash> 
d85d0a80:       0
crash> x/x ddcc8505
ddcc8505:       40cdc211
crash> 
ddcc8509:       2d03175
crash> 
ddcc850d:       0
crash> 
ddcc8511:       0
crash> 
ddcc8515:       a8000000
crash> 
ddcc8519:       cdc35b
crash> 
ddcc851d:       0
crash> 
ddcc8521:       f36260
crash> 
ddcc8525:       0
crash> 
ddcc8529:       a8000000
crash> 
ddcc852d:       f3627f
crash> 
ddcc8531:       1000000
crash>

	Sadly, "show all locks" cannot be used in crash>

crash> show all locks
db_show_all_locks: can only be used in-kernel.
crash> 

	The python3.11 process is most probably the borgmatic
	backup process, and was started 03:30, while the crash
	appears to have happened 05:55.

>How-To-Repeat:
	I thought that this was caused by "pkgsrc stress", but my attempt
	at building firefox with -j 3 was finished (unsuccessfully) a
	number of hours befre this crash.

fxxxx# uptime
 9:09AM  up  3:14, 2 users, load averages: 0.00, 0.00, 0.00
fxxxx# 

	While this host has many USB-connected serial ports
	for serving a number of remote serial console ports,
	it doesn't itself have a serial console, and I'm about
	to spend a week away from this host, so hands-on "live"
	debugging with ddb is for the immediate future off the
	table, sadly.

>Fix:
	Would be nice.

>Release-Note:

>Audit-Trail:

State-Changed-From-To: open->feedback
State-Changed-By: riastradh@NetBSD.org
State-Changed-When: Sun, 08 Sep 2024 10:53:07 +0000
State-Changed-Why:
feedback requested


From: Taylor R Campbell <riastradh@NetBSD.org>
To: Havard Eidnes <he@NetBSD.org>
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: kern/58666: panic: lock error: Reader / writer lock: rw_vector_enter,357: locking against myself
Date: Sun, 8 Sep 2024 10:51:29 +0000

 > Date: Sun,  8 Sep 2024 09:42:51 +0200 (CEST)
 > From: he@NetBSD.org
 >=20
 > uvm_fault_internal(ce5820e8,0,2,0,ffffffff,ffffffff,4000,1723,f4023cd4,ce=
 5820e8) at uvm_fault_internal+0xcf
 > trap() at trap+0x472
 > --- trap (number 6) ---
 > pmap_ctor(0,e02a0600,1,0,f0,0,c1507dc0,0,f4023e24,c0cc7b3a) at pmap_ctor+=
 0xb7

 Can you get the line number of pmap_ctor+0xb7 in your kernel, and
 disassemble it?

 $TOOLDIR/bin/i486--netbsdelf-addr2line -e ./netbsd.gdb -i pmap_ctor+0xb7
 $TOOLDIR/bin/i486--netbsdelf-gdb -ex 'disas pmap_ctor+0xb7' ./netbsd.gdb

From: Taylor R Campbell <riastradh@NetBSD.org>
To: Havard Eidnes <he@NetBSD.org>
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org,
	Chuck Silvers <chs@NetBSD.org>
Subject: Re: kern/58666: panic: lock error: Reader / writer lock: rw_vector_enter,357: locking against myself
Date: Sun, 8 Sep 2024 11:14:31 +0000

 This is a multi-part message in MIME format.
 --=_IfcsmmonH+FC4mEL4QHaEfWN6ZIGlhUw

 Can you please try the attached patch and see if you can still
 reproduce the issue?

 I don't know why this might have changed from 9.3 to 10.0_STABLE, but
 the bug the patch fixes may explain your symptom.

 --=_IfcsmmonH+FC4mEL4QHaEfWN6ZIGlhUw
 Content-Type: text/plain; charset="ISO-8859-1"; name="pr58666-i386pmapcrash"
 Content-Transfer-Encoding: quoted-printable
 Content-Disposition: attachment; filename="pr58666-i386pmapcrash.patch"

 # HG changeset patch
 # User Taylor R Campbell <riastradh@NetBSD.org>
 # Date 1725793843 0
 #      Sun Sep 08 11:10:43 2024 +0000
 # Branch trunk
 # Node ID 136d545c35b390c461167f41873831ea1109c668
 # Parent  38029f50cf37ce95668214c8d1a5a30aa712814a
 # EXP-Topic riastradh-pr58666-i386pmapcrash
 x86/pmap: Use UVM_KMF_WAITVA to ensure pmap_pdp_alloc never fails.

 This is used as the backing page allocator for pmap_pdp_pool, and
 pmap_ctor assumes that PR_WAITOK allocations from it don't fail and
 unconditionally writes to the resulting kva, which if null leads
 nowhere good.

 It is unclear to me why uvm_km_alloc can accept any combination of
 the options UVM_KMF_NOWAIT and UVM_KMF_WAITVA.  It seems to me that
 at least one should be required (and they should be exclusive), and
 any other use should trip an assertion.

 PR kern/58666: panic: lock error: Reader / writer lock:
 rw_vector_enter,357: locking against myself

 diff -r 38029f50cf37 -r 136d545c35b3 sys/arch/x86/x86/pmap.c
 --- a/sys/arch/x86/x86/pmap.c	Sat Aug 31 15:49:02 2024 +0000
 +++ b/sys/arch/x86/x86/pmap.c	Sun Sep 08 11:10:43 2024 +0000
 @@ -2844,7 +2844,8 @@ pmap_pdp_alloc(struct pool *pp, int flag
  {
  	return (void *)uvm_km_alloc(kernel_map,
  	    PAGE_SIZE * PDP_SIZE, PAGE_SIZE * PDP_SIZE,
 -	    ((flags & PR_WAITOK) ? 0 : UVM_KMF_NOWAIT | UVM_KMF_TRYLOCK) |
 +	    ((flags & PR_WAITOK) ? UVM_KMF_WAITVA
 +		: UVM_KMF_NOWAIT | UVM_KMF_TRYLOCK) |
  	    UVM_KMF_WIRED);
  }
 =20

 --=_IfcsmmonH+FC4mEL4QHaEfWN6ZIGlhUw--

From: Havard Eidnes <he@NetBSD.org>
To: riastradh@NetBSD.org
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: kern/58666: panic: lock error: Reader / writer lock:
 rw_vector_enter,357: locking against myself
Date: Sun, 08 Sep 2024 13:42:24 +0200 (CEST)

 >> Date: Sun,  8 Sep 2024 09:42:51 +0200 (CEST)
 >> From: he@NetBSD.org
 >> 
 >> uvm_fault_internal(ce5820e8,0,2,0,ffffffff,ffffffff,4000,1723,f4023cd4,ce5820e8) at uvm_fault_internal+0xcf
 >> trap() at trap+0x472
 >> --- trap (number 6) ---
 >> pmap_ctor(0,e02a0600,1,0,f0,0,c1507dc0,0,f4023e24,c0cc7b3a) at pmap_ctor+0xb7
 >
 > Can you get the line number of pmap_ctor+0xb7 in your kernel, and
 > disassemble it?
 >
 > $TOOLDIR/bin/i486--netbsdelf-addr2line -e ./netbsd.gdb -i pmap_ctor+0xb7
 > $TOOLDIR/bin/i486--netbsdelf-gdb -ex 'disas pmap_ctor+0xb7' ./netbsd.gdb

 Even though I see you have a suggested fix, for the record here
 is the equivalent of the above, using a rebuilt netbsd.gdb:

 Doesn't look like addr2line wants to play ball:

 fxxxx# /usr/tools/bin/i486--netbsdelf-addr2line -e ./netbsd.2.gdb -i pmap_ctor+0xb7
 ??:0
 fxxxx# 

 fxxxx# gdb -q netbsd.2.gdb
 ...
 (gdb) info line *(pmap_ctor+0xb7)
 Line 2702 of "/usr/src/sys/arch/x86/x86/pmap.c"
    starts at address 0xc049e659 <pmap_ctor+174>
    and ends at 0xc049e664 <pmap_ctor+185>.

 OK, this isn't disass from pmap_ctor+0xb7, but I thought this
 would be easier to read (marked +0xb7 with <--):

 (gdb) disass pmap_ctor
 Dump of assembler code for function pmap_ctor:
    0xc049e5ab <+0>:       push   %ebp
    0xc049e5ac <+1>:       mov    %esp,%ebp
    0xc049e5ae <+3>:       push   %edi
    0xc049e5af <+4>:       push   %esi
    0xc049e5b0 <+5>:       push   %ebx
    0xc049e5b1 <+6>:       sub    $0x14,%esp
    0xc049e5b4 <+9>:       mov    0xc(%ebp),%ebx
    0xc049e5b7 <+12>:      movl   $0x0,0x8(%esp)
    0xc049e5bf <+20>:      movl   $0x2,0x4(%esp)
    0xc049e5c7 <+28>:      lea    0xc0(%ebx),%eax
    0xc049e5cd <+34>:      mov    %eax,(%esp)
    0xc049e5d0 <+37>:      call   0xc0c877dd <mutex_init>
    0xc049e5d5 <+42>:      lea    0xc4(%ebx),%eax
    0xc049e5db <+48>:      mov    %eax,(%esp)
    0xc049e5de <+51>:      call   0xc0c984ae <rw_init>
    0xc049e5e3 <+56>:      movl   $0x1,0x4(%esp)
    0xc049e5eb <+64>:      lea    0x74(%ebx),%eax
    0xc049e5ee <+67>:      mov    %eax,(%esp)
    0xc049e5f1 <+70>:      call   0xc0cbf506 <kcpuset_create>
    0xc049e5f6 <+75>:      movl   $0x1,0x4(%esp)
    0xc049e5fe <+83>:      lea    0x78(%ebx),%eax
    0xc049e601 <+86>:      mov    %eax,(%esp)
    0xc049e604 <+89>:      call   0xc0cbf506 <kcpuset_create>
    0xc049e609 <+94>:      movl   $0x0,0x88(%ebx)
    0xc049e613 <+104>:     movl   $0x0,0x54(%ebx)
    0xc049e61a <+111>:     movl   $0x0,0x60(%ebx)
    0xc049e621 <+118>:     movl   $0x0,0x58(%ebx)
    0xc049e628 <+125>:     movl   $0x0,0x5c(%ebx)
    0xc049e62f <+132>:     movl   $0x1,0x4(%esp)
    0xc049e637 <+140>:     movl   $0xc1508060,(%esp)
    0xc049e63e <+147>:     call   0xc0cc6295 <pool_get>
    0xc049e643 <+152>:     mov    %eax,0x24(%ebx)
    0xc049e646 <+155>:     mov    %ebx,0xc(%ebp)
    0xc049e649 <+158>:     mov    %eax,%ebx
    0xc049e64b <+160>:     movl   $0x0,-0x14(%ebp)
    0xc049e652 <+167>:     movl   $0x0,-0x10(%ebp)
    0xc049e659 <+174>:     mov    $0x1000,%ecx   <-- +0xb7
    0xc049e65e <+179>:     mov    %ebx,%edi
    0xc049e660 <+181>:     xor    %eax,%eax
    0xc049e662 <+183>:     rep stos %eax,%es:(%edi)
    0xc049e664 <+185>:     xor    %esi,%esi
    0xc049e666 <+187>:     lea    -0x14(%ebp),%eax
    0xc049e669 <+190>:     mov    %eax,0x8(%esp)
    0xc049e66d <+194>:     mov    %esi,%eax
    0xc049e66f <+196>:     shl    $0xc,%eax
    0xc049e672 <+199>:     add    %ebx,%eax
    0xc049e674 <+201>:     mov    %eax,0x4(%esp)
    0xc049e678 <+205>:     movl   $0xc14dfd40,(%esp)
    0xc049e67f <+212>:     call   0xc049e3f0 <pmap_extract>
    0xc049e684 <+217>:     mov    -0x14(%ebp),%eax
    0xc049e687 <+220>:     or     0xc14eb048,%eax
    0xc049e68d <+226>:     mov    -0x10(%ebp),%edx
    0xc049e690 <+229>:     or     0xc14eb04c,%edx
    0xc049e696 <+235>:     or     $0x3,%eax
    0xc049e699 <+238>:     mov    %eax,0x2fe0(%ebx,%esi,8)
    0xc049e6a0 <+245>:     mov    %edx,0x2fe4(%ebx,%esi,8)
    0xc049e6a7 <+252>:     add    $0x1,%esi
    0xc049e6aa <+255>:     cmp    $0x4,%esi
    0xc049e6ad <+258>:     jne    0xc049e666 <pmap_ctor+187>
    0xc049e6af <+260>:     lea    0x3000(%ebx),%edx
    0xc049e6b5 <+266>:     mov    0xc14955a8,%eax
    0xc049e6ba <+271>:     shl    $0x3,%eax
    0xc049e6bd <+274>:     mov    %edx,%edi
    0xc049e6bf <+276>:     mov    $0xbfdff000,%esi
    0xc049e6c4 <+281>:     cmp    $0x4,%eax
    0xc049e6c7 <+284>:     jae    0xc049e763 <pmap_ctor+440>
    0xc049e6cd <+290>:     test   $0x2,%al
    0xc049e6cf <+292>:     je     0xc049e6d3 <pmap_ctor+296>
    0xc049e6d1 <+294>:     movsw  %ds:(%esi),%es:(%edi)
    0xc049e6d3 <+296>:     test   $0x1,%al
    0xc049e6d5 <+298>:     je     0xc049e6d8 <pmap_ctor+301>
    0xc049e6d7 <+300>:     movsb  %ds:(%esi),%es:(%edi)
    0xc049e6d8 <+301>:     movl   $0xc14dfe40,(%esp)
    0xc049e6df <+308>:     call   0xc01253c0 <mutex_enter>
    0xc049e6e4 <+313>:     mov    0xc(%ebp),%eax
 --Type <RET> for more, q to quit, c to continue without paging--
    0xc049e6e7 <+316>:     mov    0x24(%eax),%esi
    0xc049e6ea <+319>:     mov    0xc14955a8,%eax
    0xc049e6ef <+324>:     mov    0x2ffc(%esi,%eax,8),%edx
    0xc049e6f6 <+331>:     or     0x2ff8(%esi,%eax,8),%edx
    0xc049e6fd <+338>:     je     0xc049e77c <pmap_ctor+465>
    0xc049e6ff <+340>:     mov    0xc(%ebp),%ebx
    0xc049e702 <+343>:     xor    %eax,%eax
    0xc049e704 <+345>:     mov    0x2fe0(%esi,%eax,8),%ecx
    0xc049e70b <+352>:     and    $0xfffff000,%ecx
    0xc049e711 <+358>:     mov    0x2fe4(%esi,%eax,8),%edx
    0xc049e718 <+365>:     and    $0xfffff,%edx
    0xc049e71e <+371>:     mov    %ecx,0x28(%ebx,%eax,8)
    0xc049e722 <+375>:     mov    %edx,0x2c(%ebx,%eax,8)
    0xc049e726 <+379>:     add    $0x1,%eax
    0xc049e729 <+382>:     cmp    $0x4,%eax
    0xc049e72c <+385>:     jne    0xc049e704 <pmap_ctor+345>
    0xc049e72e <+387>:     mov    0xc1508040,%eax
    0xc049e733 <+392>:     mov    %eax,0x1c(%ebx)
    0xc049e736 <+395>:     test   %eax,%eax
    0xc049e738 <+397>:     je     0xc049e740 <pmap_ctor+405>
    0xc049e73a <+399>:     lea    0x1c(%ebx),%edx
    0xc049e73d <+402>:     mov    %edx,0x20(%eax)
    0xc049e740 <+405>:     mov    %ebx,0xc1508040
    0xc049e746 <+411>:     movl   $0xc1508040,0x20(%ebx)
    0xc049e74d <+418>:     movl   $0xc14dfe40,(%esp)
    0xc049e754 <+425>:     call   0xc01253e0 <mutex_exit>
    0xc049e759 <+430>:     xor    %eax,%eax
    0xc049e75b <+432>:     add    $0x14,%esp
    0xc049e75e <+435>:     pop    %ebx
    0xc049e75f <+436>:     pop    %esi
    0xc049e760 <+437>:     pop    %edi
    0xc049e761 <+438>:     pop    %ebp
    0xc049e762 <+439>:     ret    
    0xc049e763 <+440>:     test   $0x1,%dl
    0xc049e766 <+443>:     jne    0xc049e7a4 <pmap_ctor+505>
    0xc049e768 <+445>:     test   $0x2,%edi
    0xc049e76e <+451>:     jne    0xc049e793 <pmap_ctor+488>
    0xc049e770 <+453>:     mov    %eax,%ecx
    0xc049e772 <+455>:     shr    $0x2,%ecx
    0xc049e775 <+458>:     rep movsl %ds:(%esi),%es:(%edi)
    0xc049e777 <+460>:     jmp    0xc049e6cd <pmap_ctor+290>
    0xc049e77c <+465>:     movl   $0xc14dfe40,(%esp)
    0xc049e783 <+472>:     call   0xc01253e0 <mutex_exit>
    0xc049e788 <+477>:     mov    0xc(%ebp),%eax
    0xc049e78b <+480>:     mov    0x24(%eax),%ebx
    0xc049e78e <+483>:     jmp    0xc049e64b <pmap_ctor+160>
    0xc049e793 <+488>:     movzwl (%esi),%edx
    0xc049e796 <+491>:     mov    %dx,(%edi)
    0xc049e799 <+494>:     add    $0x2,%edi
    0xc049e79c <+497>:     add    $0x2,%esi
    0xc049e79f <+500>:     sub    $0x2,%eax
    0xc049e7a2 <+503>:     jmp    0xc049e770 <pmap_ctor+453>
    0xc049e7a4 <+505>:     movzbl 0xbfdff000,%ecx
    0xc049e7ab <+512>:     mov    %cl,0x3000(%ebx)
    0xc049e7b1 <+518>:     lea    0x3001(%ebx),%edi
    0xc049e7b7 <+524>:     mov    $0xbfdff001,%esi
    0xc049e7bc <+529>:     sub    $0x1,%eax
    0xc049e7bf <+532>:     jmp    0xc049e768 <pmap_ctor+445>
 End of assembler dump.
 (gdb) 

From: Taylor R Campbell <riastradh@NetBSD.org>
To: Havard Eidnes <he@NetBSD.org>
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org,
	Chuck Silvers <chs@NetBSD.org>
Subject: Re: kern/58666: panic: lock error: Reader / writer lock:
	rw_vector_enter,357: locking against myself
Date: Sun, 8 Sep 2024 12:04:36 +0000

 > Date: Sun, 08 Sep 2024 13:42:24 +0200 (CEST)
 > From: Havard Eidnes <he@NetBSD.org>
 >=20
 > Doesn't look like addr2line wants to play ball:
 >=20
 > fxxxx# /usr/tools/bin/i486--netbsdelf-addr2line -e ./netbsd.2.gdb -i pmap=
 _ctor+0xb7
 > ??:0

 Huh, I guess something was fixed in binutiles between netbsd-10 and
 HEAD.  Anyway, gdb gives this to us.

 > fxxxx# gdb -q netbsd.2.gdb
 > ...
 > (gdb) info line *(pmap_ctor+0xb7)
 > Line 2702 of "/usr/src/sys/arch/x86/x86/pmap.c"
 >    starts at address 0xc049e659 <pmap_ctor+174>
 >    and ends at 0xc049e664 <pmap_ctor+185>.

 This is:

    2688 static void
    2689 pmap_pdp_init(pd_entry_t *pdir)
    2690 {
 ...
    2702 	memset(PAGE_ALIGNED(pdir), 0, PDP_SIZE * PAGE_SIZE);

 https://nxr.netbsd.org/xref/src/sys/arch/x86/x86/pmap.c?r=3D1.423#2702

 Together with the uvm_fault_internal(map, va, flags) line in the stack
 trace, we see this is a null pointer dereference:

 uvm_fault_internal(ce5820e8,0,2,0,ffffffff,ffffffff,4000,1723,f4023cd4,ce58=
 20e8) at uvm_fault_internal+0xcf

 > OK, this isn't disass from pmap_ctor+0xb7, but I thought this
 > would be easier to read (marked +0xb7 with <--):
 >=20
 > (gdb) disass pmap_ctor
 > Dump of assembler code for function pmap_ctor:
 > [...]
 >    0xc049e659 <+174>:     mov    $0x1000,%ecx   <-- +0xb7
 >    0xc049e65e <+179>:     mov    %ebx,%edi
 >    0xc049e660 <+181>:     xor    %eax,%eax
 >    0xc049e662 <+183>:     rep stos %eax,%es:(%edi)

 What you marked is 0xae=3D174, not 0xb7=3D183.  0xb7=3D183 is the REP STOS
 instruction, which is to say, memset.

 In other words, this is consistent with my hypothesis that:

 1. the fault is memset of null

 2. the pointer which is null came from pool_get(&pmap_pdp_pool,
    PR_WAITOK) which should never return null, unless...

 3. the backing allocator for pmap_pdp_pool fails and returns null
    despite being passed PR_WAITOK, which...

 4. can happen if this is a PAE kernel (as it is) so the backing
    allocator is pmap_pdp_alloc which calls uvm_km_alloc without
    UVM_KMF_WAITVA, and...

 5. RAM is currently short so uvm_km_alloc would have to sleep and wait
    for the pagedaemon to free pages before it can return, in which
    case it returns null instead of sleeping because pmap_pdp_alloc
    didn't pass UVM_KMF_WAITVA.

 The patch attempts to fix this by passing UVM_KMF_WAITVA in
 pmap_pdp_alloc so it sleeps instead of failing in this case.

 I also started to review the tree for other cases of uvm_km_alloc that
 don't pass either UVM_KMF_WAITVA or UVM_KMF_NOWAIt, and, hoo boy,
 there's a lot of potential issues here in obscure corners like acorn32
 which I have no hope of testing myself.

State-Changed-From-To: feedback->analyzed
State-Changed-By: riastradh@NetBSD.org
State-Changed-When: Sun, 08 Sep 2024 17:02:45 +0000
State-Changed-Why:
problem analyzed, patch drafted


From: Taylor R Campbell <riastradh@NetBSD.org>
To: Havard Eidnes <he@NetBSD.org>
Cc: gnats-bugs@NetBSD.org,
	netbsd-bugs@NetBSD.org, Chuck Silvers <chs@NetBSD.org>
Subject: Re: kern/58666: panic: lock error: Reader / writer lock:
	rw_vector_enter,357: locking against myself
Date: Sun, 8 Sep 2024 17:12:18 +0000

 > Date: Sun, 8 Sep 2024 12:04:36 +0000
 > From: Taylor R Campbell <riastradh@NetBSD.org>
 > 
 > 5. RAM is currently short so uvm_km_alloc would have to sleep and wait
 >    for the pagedaemon to free pages before it can return, in which
 >    case it returns null instead of sleeping because pmap_pdp_alloc
 >    didn't pass UVM_KMF_WAITVA.

 Correction: kernel virtual address space, not RAM, is short, so
 uvm_km_alloc would have to sleep and wait for the pagedaemon to find
 some kva pages to free.  By passing UVM_KMF_WAITVA, pmap_pdp_alloc
 would wait for kva instead of returning null.

From: Havard Eidnes <he@NetBSD.org>
To: riastradh@NetBSD.org
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org, chs@NetBSD.org
Subject: Re: kern/58666: panic: lock error: Reader / writer lock:
 rw_vector_enter,357: locking against myself
Date: Sun, 08 Sep 2024 19:48:41 +0200 (CEST)

 >> 5. RAM is currently short so uvm_km_alloc would have to sleep and wa=
 it
 >>    for the pagedaemon to free pages before it can return, in which
 >>    case it returns null instead of sleeping because pmap_pdp_alloc
 >>    didn't pass UVM_KMF_WAITVA.
 >
 > Correction: kernel virtual address space, not RAM, is short, so
 > uvm_km_alloc would have to sleep and wait for the pagedaemon to find
 > some kva pages to free.  By passing UVM_KMF_WAITVA, pmap_pdp_alloc
 > would wait for kva instead of returning null.

 I was going to say something like "probably not short of RAM".

 So this turns into: What can I as a system manager (and we as a
 project) do to

 a) monitor the state of KVA
 b) reduce the KVA pressure on the i386 port (or better manage KVA
    pressure situations)

 For a) I suspect we don't really have anything which can be
 readily used(?)

 For b) I recall having reduced maxvnodes on my trusty old
 i386-running T60, but I'm not sure I've done that on this
 particular host, so that's at least something to try.  With 9.3
 that has never been necessary, and it's seen its fair bit of
 abuse by doing "-j 3" rust builds as part of testing /
 validation, and also firefox builds with the same -j 3 setting.
 However, it would be nice if the default parameter choices were
 not so prone to causing a wedge.

 Hmm, the maxvnodes choice, is that solely based on "available
 RAM"?  As you may recall, this host has 10G physical memory, but
 has much smaller kernel virtual address space.

 It also seems to me that the pagedaemon isn't managing to
 actually do anything to remedy the situation when KVA either
 becomes fragmented or comes under pressure.  Is there something
 we can do (to the code / KVA usage / KVA management) to prevent
 this from turning into a wedge / hang situation, too easily
 triggered with the default parameter choices?

 Best regards,

 - H=E5vard

From: Havard Eidnes <he@NetBSD.org>
To: riastradh@NetBSD.org
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org, chs@NetBSD.org
Subject: Re: kern/58666: panic: lock error: Reader / writer lock:
 rw_vector_enter,357: locking against myself
Date: Mon, 09 Sep 2024 10:03:13 +0200 (CEST)

 > For b) I recall having reduced maxvnodes on my trusty old
 > i386-running T60, but I'm not sure I've done that on this
 > particular host, so that's at least something to try.  With 9.3
 > that has never been necessary, and it's seen its fair bit of
 > abuse by doing "-j 3" rust builds as part of testing /
 > validation, and also firefox builds with the same -j 3 setting.
 > However, it would be nice if the default parameter choices were
 > not so prone to causing a wedge.

 The host which prompted me to report this problem is now back up,
 and it turns out that I have *already* tweaked maxvnodes:

 # Reduce KVA pressure
 kern.maxvnodes=40000

 Regards,

 - Havard

From: Havard Eidnes <he@NetBSD.org>
To: riastradh@NetBSD.org
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org, chs@NetBSD.org
Subject: Re: kern/58666: panic: lock error: Reader / writer lock:
 rw_vector_enter,357: locking against myself
Date: Thu, 12 Sep 2024 17:09:45 +0200 (CEST)

 Hm, just a brief status report.

 The machine wedged in "normal operation" which doesn't really
 involve anything particularly "havy".  I think the trigger might
 have been the daily run; we monitor with collectd, and the load
 average increased to 5 and stayed at that (or higher) shortly
 after daily started.  Later the load increased to 15, and at
 the end CPU load increased significantly, but I suspect those
 processes were stuck in "tstile".

 Insight into what this might be and what to look for (e.g. via
 ddb, but also via "normal" logged-in methods) would be much
 appreciated.

 (I'm still 500km away, so it's been rebooted now.)

 Regards,

 - H=E5vard

>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.47 2022/09/11 19:34:41 kim Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2024 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.