NetBSD Problem Report #53066
From martin@aprisoft.de Fri Mar 2 15:55:33 2018
Return-Path: <martin@aprisoft.de>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 825C37A167
for <gnats-bugs@gnats.NetBSD.org>; Fri, 2 Mar 2018 15:55:33 +0000 (UTC)
Message-Id: <20180302155522.1B79C5CC770@emmas.aprisoft.de>
Date: Fri, 2 Mar 2018 16:55:22 +0100 (CET)
From: martin@NetBSD.org
Reply-To: martin@NetBSD.org
To: gnats-bugs@NetBSD.org
Subject: nearly reproducable softint panic on shutdown
X-Send-Pr-Version: 3.95
>Number: 53066
>Category: kern
>Synopsis: nearly reproducable softint panic on shutdown
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: jdolecek
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Fri Mar 02 16:00:01 +0000 2018
>Closed-Date: Fri Jun 01 18:44:03 +0000 2018
>Last-Modified: Fri Jun 01 18:44:03 +0000 2018
>Originator: Martin Husemann
>Release: NetBSD 8.99.12
>Organization:
The NetBSD Foundation, Inc.
>Environment:
System: NetBSD seven-days-to-the-wolves.aprisoft.de 8.99.12 NetBSD 8.99.12 (GENERIC) #196: Tue Feb 27 14:22:55 CET 2018 martin@seven-days-to-the-wolves.aprisoft.de:/work/src/sys/arch/amd64/compile/GENERIC amd64
Architecture: x86_64
Machine: amd64
>Description:
I get a pretty reliable panic on shutdown on this machine:
syncing disks... done
unmounted procfs on /proc type procfs
unmounted ptyfs on /dev/pts type ptyfs
unmounted kernfs on /kern type kernfs
uhub1: detached
uhub0: detached
atabus0: detached
cd0: detached
usb1: detached
usb0: detached
atabus1: detached
atapibus0: detached
uhub8: detached
uhub7: detached
uhub6: detached
uhub5: detached
uhub4: detached
uhub2: detached
spkr1: detached
Skipping crash dump on recursive panic
panic: kernel diagnostic assertion "offset != 0 && offset < softint_bytes" faile
d: file "../../../../kern/kern_softint.c", line 487 0 32768
cpu0: Begin traceback...
vpanic() at netbsd:vpanic+0x140
ch_voltag_convert_in() at netbsd:ch_voltag_convert_in
softint_schedule() at netbsd:softint_schedule+0x7e
usb_schedsoftintr() at netbsd:usb_schedsoftintr+0x26
xhci_intr() at netbsd:xhci_intr+0x7c
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
handle_ioapic_level1() at netbsd:handle_ioapic_level1+0x6a
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80013d2cf0f0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
cpu0: End traceback...
Full dmesg:
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017,
2018 The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
NetBSD 8.99.12 (GENERIC) #196: Tue Feb 27 14:22:55 CET 2018
martin@seven-days-to-the-wolves.aprisoft.de:/work/src/sys/arch/amd64/compile/GENERIC
total memory = 16382 MB
avail memory = 15883 MB
timecounter: Timecounters tick every 10.000 msec
Kernelized RAIDframe activated
running cgd selftest aes-xts-256 aes-xts-512 done
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
System manufacturer System Product Name (System Version)
mainbus0 (root)
ACPI: RSDP 0x00000000000FBAD0 000024 (v02 ACPIAM)
ACPI: XSDT 0x00000000CFE90100 00005C (v01 030811 XSDT1042 20110308 MSFT 00000097)
ACPI: FACP 0x00000000CFE90290 0000F4 (v03 030811 FACP1042 20110308 MSFT 00000097)
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe0Block: 64/32 (20171110/tbfadt-642)
ACPI: DSDT 0x00000000CFE90450 00E040 (v01 A1585 A1585000 00000000 INTL 20060113)
ACPI: FACS 0x00000000CFEA8000 000040
ACPI: APIC 0x00000000CFE90390 00007C (v01 030811 APIC1042 20110308 MSFT 00000097)
ACPI: MCFG 0x00000000CFE90410 00003C (v01 030811 OEMMCFG 20110308 MSFT 00000097)
ACPI: OEMB 0x00000000CFEA8040 000072 (v01 030811 OEMB1042 20110308 MSFT 00000097)
ACPI: SRAT 0x00000000CFE9F8A0 000108 (v01 AMD FAM_F_10 00000002 AMD 00000001)
ACPI: HPET 0x00000000CFE9F9B0 000038 (v01 030811 OEMHPET 20110308 MSFT 00000097)
ACPI: SSDT 0x00000000CFE9F9F0 000DA4 (v01 A M I POWERNOW 00000001 AMD 00000001)
ACPI: Executed 3 blocks of module-level executable AML code
ACPI: 2 ACPI AML tables successfully acquired and loaded
ioapic0 at mainbus0 apid 6: pa 0xfec00000, version 0x21, 24 pins
cpu0 at mainbus0 apid 0
cpu0: AMD Phenom(tm) II X6 1075T Processor, id 0x100fa0
cpu0: package 0, core 0, smt 0
cpu1 at mainbus0 apid 1
cpu1: AMD Phenom(tm) II X6 1075T Processor, id 0x100fa0
cpu1: package 0, core 1, smt 0
cpu2 at mainbus0 apid 2
cpu2: AMD Phenom(tm) II X6 1075T Processor, id 0x100fa0
cpu2: package 0, core 2, smt 0
cpu3 at mainbus0 apid 3
cpu3: AMD Phenom(tm) II X6 1075T Processor, id 0x100fa0
cpu3: package 0, core 3, smt 0
cpu4 at mainbus0 apid 4
cpu4: AMD Phenom(tm) II X6 1075T Processor, id 0x100fa0
cpu4: package 0, core 4, smt 0
cpu5 at mainbus0 apid 5
cpu5: AMD Phenom(tm) II X6 1075T Processor, id 0x100fa0
cpu5: package 0, core 5, smt 0
acpi0 at mainbus0: Intel ACPICA 20171110
acpi0: X/RSDT: OemId <030811,XSDT1042,20110308>, AslId <MSFT,00000097>
acpi0: MCFG: segment 0, bus 0-255, address 0x00000000e0000000
acpi0: SCI interrupting at int 9
timecounter: Timecounter "ACPI-Safe" frequency 3579545 Hz quality 900
hpet0 at acpi0: high precision event timer (mem 0xfed00000-0xfed00400)
timecounter: Timecounter "hpet0" frequency 14318180 Hz quality 2000
acpiec0 at acpi0 (EC0, PNP0C09): io 0x62,0x66
attimer1 at acpi0 (TMR, PNP0100): io 0x40-0x43 irq 0
pcppi1 at acpi0 (SPKR, PNP0800): io 0x61
spkr0 at pcppi1: PC Speaker
wsbell at spkr0 not configured
midi0 at pcppi1: PC speaker
sysbeep0 at pcppi1
UAR1 (PNP0501) at acpi0 not configured
aibs0 at acpi0 (ASOC, ATK0110-16843024): ASUSTeK AI Booster
OMSC (PNP0C02) at acpi0 not configured
RMSC (PNP0C02) at acpi0 not configured
SIOR (PNP0C02) at acpi0 not configured
PCIE (PNP0C02) at acpi0 not configured
RMEM (PNP0C01) at acpi0 not configured
acpibut0 at acpi0 (PWRB, PNP0C0C-170): ACPI Power Button
acpiwmi0 at acpi0 (AOD, PNP0C14-0): ACPI WMI Interface
acpiwmibus at acpiwmi0 not configured
attimer1: attached to pcppi1
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0: vendor 1002 product 5957 (rev. 0x00)
ppb0 at pci0 dev 2 function 0: vendor 1002 product 5978 (rev. 0x00)
ppb0: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x16 @ 5.0GT/s
ppb0: link is x2 @ 2.5GT/s
pci1 at ppb0 bus 6
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
nouveau0 at pci1 dev 0 function 0: vendor 10de product 0e22 (rev. 0xa1)
nouveau0: interrupting at ioapic0 pin 18 (nouveau)
Zone kernel: Available graphics memory: 5488278 kiB
Zone dma32: Available graphics memory: 2097152 kiB
00001030
00010130
00002261
drm: Supports vblank timestamp caching Rev 2 (21.10.2013).
drm: Driver supports precise vblank timestamp query.
nouveau0: info: No connectors reported connected with modes
drm: Cannot find any crtc or sizes - going 1024x768
nouveaufb0 at nouveau0
nouveau0: info: registered panic notifier
nouveaufb0: framebuffer at 0xffff800143524000, size 1024x768, depth 32, stride 4096
wsdisplay0 at nouveaufb0 kbdmux 1
wsmux1: connecting to wsdisplay0
hdaudio0 at pci1 dev 0 function 1: HD Audio Controller
hdaudio0: interrupting at ioapic0 pin 19
hdafg0 at hdaudio0: vendor 10de product 0012
hdafg0: DP00 8ch: Digital Out [Jack]
hdafg0: 8ch/0ch 48000Hz PCM16*
hdafg1 at hdaudio0: vendor 10de product 0012
hdafg1: DP00 8ch: Digital Out [Jack]
hdafg1: 8ch/0ch 48000Hz PCM16*
hdafg2 at hdaudio0: vendor 10de product 0012
hdafg2: DP00 8ch: Digital Out [Jack]
hdafg2: 8ch/0ch 48000Hz PCM16*
hdafg3 at hdaudio0: vendor 10de product 0012
hdafg3: DP00 8ch: Digital Out [Jack]
hdafg3: 8ch/0ch 48000Hz PCM16*
ppb1 at pci0 dev 4 function 0: vendor 1002 product 597a (rev. 0x00)
ppb1: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x4 @ 5.0GT/s
pci2 at ppb1 bus 5
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
nvme0 at pci2 dev 0 function 0: vendor 144d product a802 (rev. 0x01)
nvme0: NVMe 1.1
nvme0: interrupting at ioapic0 pin 16
nvme0: SAMSUNG MZVPV256HDGL-00000, firmware BXW7300Q, serial S1XWNYAH408386
ld0 at nvme0 nsid 1
ld0: 238 GB, 31130 cyl, 255 head, 63 sec, 512 bytes/sect x 500118192 sectors
ppb2 at pci0 dev 9 function 0: vendor 1002 product 597e (rev. 0x00)
ppb2: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x2 @ 5.0GT/s
ppb2: link is x1 @ 2.5GT/s
pci3 at ppb2 bus 4
pci3: i/o space, memory space enabled, rd/line, wr/inv ok
jmide0 at pci3 dev 0 function 0: vendor 197b product 2361 (rev. 0x10)
jmide0: 1 PATA port, 1 SATA port
jmide0: interrupting at ioapic0 pin 17
ahcisata0 at jmide0
ahcisata0: AHCI revision 1.10, 1 port, 32 slots, CAP 0xc722ff00<PSC,SSC,PMD,SPM,ISS=0x2=Gen2,SCLO,SAL,SALP,SNCQ,S64A>
atabus0 at ahcisata0 channel 0
jmide0: PCI IDE interface used
jmide0: bus-master DMA support present
jmide0: primary channel wired to native-PCI mode
jmide0: primary channel is unused
jmide0: secondary channel wired to native-PCI mode
jmide0: secondary channel is PATA
atabus1 at jmide0 channel 1
ppb3 at pci0 dev 10 function 0: vendor 1002 product 597f (rev. 0x00)
ppb3: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x2 @ 5.0GT/s
ppb3: link is x1 @ 5.0GT/s
pci4 at ppb3 bus 3
pci4: i/o space, memory space enabled, rd/line, wr/inv ok
xhci0 at pci4 dev 0 function 0: vendor 1033 product 0194 (rev. 0x03)
xhci0: interrupting at ioapic0 pin 18
xhci0: xHCI version 0.96
usb0 at xhci0: USB revision 3.0
usb1 at xhci0: USB revision 2.0
ahcisata1 at pci0 dev 17 function 0: vendor 1002 product 4391 (rev. 0x40)
ahcisata1: interrupting at ioapic0 pin 19
ahcisata1: 64-bit DMA
ahcisata1: AHCI revision 1.20, 6 ports, 32 slots, CAP 0xf732ff05<PSC,SSC,PMD,SPM,ISS=0x3=Gen3,SCLO,SAL,SALP,SMPS,SSNTF,SNCQ,S64A>
atabus2 at ahcisata1 channel 0
atabus3 at ahcisata1 channel 1
atabus4 at ahcisata1 channel 2
atabus5 at ahcisata1 channel 3
atabus6 at ahcisata1 channel 4
atabus7 at ahcisata1 channel 5
ohci0 at pci0 dev 18 function 0: vendor 1002 product 4397 (rev. 0x00)
ohci0: interrupting at ioapic0 pin 18
ohci0: OHCI version 1.0, legacy support
usb2 at ohci0: USB revision 1.0
ehci0 at pci0 dev 18 function 2: vendor 1002 product 4396 (rev. 0x00)
ehci0: interrupting at ioapic0 pin 17
ehci0: dropped intr workaround enabled
ehci0: EHCI version 1.0
ehci0: 1 companion controller, 5 ports: ohci0
usb3 at ehci0: USB revision 2.0
ohci1 at pci0 dev 19 function 0: vendor 1002 product 4397 (rev. 0x00)
ohci1: interrupting at ioapic0 pin 18
ohci1: OHCI version 1.0, legacy support
usb4 at ohci1: USB revision 1.0
ehci1 at pci0 dev 19 function 2: vendor 1002 product 4396 (rev. 0x00)
ehci1: interrupting at ioapic0 pin 17
ehci1: dropped intr workaround enabled
ehci1: EHCI version 1.0
ehci1: 1 companion controller, 5 ports: ohci1
usb5 at ehci1: USB revision 2.0
piixpm0 at pci0 dev 20 function 0: vendor 1002 product 4385 (rev. 0x42)
piixpm0: polling (SB800)
iic0 at piixpm0: I2C bus
iic1 at piixpm0: I2C bus
iic2 at piixpm0: I2C bus
iic3 at piixpm0: I2C bus
hdaudio1 at pci0 dev 20 function 2: HD Audio Controller
hdaudio1: interrupting at ioapic0 pin 16
hdafg4 at hdaudio1: vendor 1106 product 0440
hdafg4: DAC00 8ch: Speaker [Jack], HP Out [Jack]
hdafg4: ADC01 2ch: Line In [Jack], Mic In [Jack]
hdafg4: HDMI02 2ch: Digital Out [Jack]
hdafg4: DIG03 2ch: SPDIF Out [Jack]
hdafg4: 8ch/2ch 48000Hz PCM16*
audio0 at hdafg4: full duplex, playback, capture, mmap, independent
hdafg4: Virtual format configured - Format SLINEAR, precision 16, channels 2, frequency 48000
hdafg4: Latency: 128 milliseconds
spkr1 at audio0: PC Speaker (synthesized)
wsbell at spkr1 not configured
pcib0 at pci0 dev 20 function 3: vendor 1002 product 439d (rev. 0x40)
ppb4 at pci0 dev 20 function 4: vendor 1002 product 4384 (rev. 0x40)
pci5 at ppb4 bus 2
pci5: i/o space, memory space enabled
fwohci0 at pci5 dev 8 function 0: vendor 1106 product 3044 (rev. 0xc0)
fwohci0: interrupting at ioapic0 pin 20
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:1f:c6:00:00:0a:ca:1a
fwohci0: Phy 1394a available S400, 2 ports.
fwohci0: Link S400, max_rec 2048 bytes.
ieee1394if0 at fwohci0: IEEE1394 bus
fwip0 at ieee1394if0: IP over IEEE1394
fwohci0: Initiate bus reset
ohci2 at pci0 dev 20 function 5: vendor 1002 product 4399 (rev. 0x00)
ohci2: interrupting at ioapic0 pin 18
ohci2: OHCI version 1.0, legacy support
usb6 at ohci2: USB revision 1.0
ppb5 at pci0 dev 21 function 0: vendor 1002 product 43a0 (rev. 0x00)
ppb5: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x1 @ 5.0GT/s
ppb5: link is x1 @ 2.5GT/s
pci6 at ppb5 bus 1
pci6: i/o space, memory space enabled, rd/line, wr/inv ok
re0 at pci6 dev 0 function 0: RealTek 8168/8111 PCIe Gigabit Ethernet (rev. 0x06)
re0: interrupting at ioapic0 pin 16
re0: Ethernet address bc:ae:c5:46:16:58
re0: using 256 tx descriptors
rgephy0 at re0 phy 7: RTL8169S/8110S/8211 1000BASE-T media interface, rev. 4
rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
ohci3 at pci0 dev 22 function 0: vendor 1002 product 4397 (rev. 0x00)
ohci3: interrupting at ioapic0 pin 18
ohci3: OHCI version 1.0, legacy support
usb7 at ohci3: USB revision 1.0
ehci2 at pci0 dev 22 function 2: vendor 1002 product 4396 (rev. 0x00)
ehci2: interrupting at ioapic0 pin 17
ehci2: dropped intr workaround enabled
ehci2: EHCI version 1.0
ehci2: 1 companion controller, 4 ports: ohci3
usb8 at ehci2: USB revision 2.0
pchb1 at pci0 dev 24 function 0: vendor 1022 product 1200 (rev. 0x00)
pchb2 at pci0 dev 24 function 1: vendor 1022 product 1201 (rev. 0x00)
pchb3 at pci0 dev 24 function 2: vendor 1022 product 1202 (rev. 0x00)
amdnb_misc0 at pci0 dev 24 function 3: AMD NB Misc Configuration
amdtemp0 at amdnb_misc0: AMD CPU Temperature Sensors (Family10h)
pchb4 at pci0 dev 24 function 4: vendor 1022 product 1204 (rev. 0x00)
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com0: console
pckbc0 at isa0 port 0x60-0x64
acpicpu0 at cpu0: ACPI CPU
acpicpu0: C1: HLT, lat 0 us, pow 0 mW
acpicpu0: C2: I/O, lat 75 us, pow 1 mW
acpicpu0: P0: FFH, lat 4 us, pow 19507 mW, 3000 MHz
acpicpu0: P1: FFH, lat 4 us, pow 14500 mW, 2300 MHz
acpicpu0: P2: FFH, lat 4 us, pow 10535 mW, 1600 MHz
acpicpu0: P3: FFH, lat 4 us, pow 6210 mW, 800 MHz
acpicpu0: T0: I/O, lat 1 us, pow 0 mW, 100 %
acpicpu0: T1: I/O, lat 1 us, pow 0 mW, 88 %
acpicpu0: T2: I/O, lat 1 us, pow 0 mW, 76 %
acpicpu0: T3: I/O, lat 1 us, pow 0 mW, 64 %
acpicpu0: T4: I/O, lat 1 us, pow 0 mW, 52 %
acpicpu0: T5: I/O, lat 1 us, pow 0 mW, 40 %
acpicpu0: T6: I/O, lat 1 us, pow 0 mW, 28 %
acpicpu0: T7: I/O, lat 1 us, pow 0 mW, 16 %
acpicpu1 at cpu1: ACPI CPU
acpicpu2 at cpu2: ACPI CPU
acpicpu3 at cpu3: ACPI CPU
acpicpu4 at cpu4: ACPI CPU
acpicpu5 at cpu5: ACPI CPU
fwohci0: BUS reset
fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode
ieee1394if0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me)
ieee1394if0: bus manager 0
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
timecounter: Timecounter "TSC" frequency 3010296450 Hz quality 3000
uhub0 at usb0: vendor 1033 (0x1033) xHCI Root Hub (0000), class 9/0, rev 3.00/1.00, addr 0
uhub0: 2 ports with 2 removable, self powered
uhub1 at usb1: vendor 1033 (0x1033) xHCI Root Hub (0000), class 9/0, rev 2.00/1.00, addr 0
uhub1: 2 ports with 2 removable, self powered
uhub2 at usb3: vendor 1002 (0x1002) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub2: 5 ports with 5 removable, self powered
uhub3 at usb5: vendor 1002 (0x1002) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub3: 5 ports with 5 removable, self powered
uhub4 at usb7: vendor 1002 (0x1002) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub4: 4 ports with 4 removable, self powered
uhub5 at usb4: vendor 1002 (0x1002) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub5: 5 ports with 5 removable, self powered
uhub6 at usb6: vendor 1002 (0x1002) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub6: 2 ports with 2 removable, self powered
uhub7 at usb2: vendor 1002 (0x1002) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub7: 5 ports with 5 removable, self powered
ahcisata1 port 0: device present, speed: 6.0Gb/s
ahcisata1 port 2: device present, speed: 1.5Gb/s
IPsec: Initialized Security Association Processing.
uhub8 at usb8: vendor 1002 (0x1002) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub8: 4 ports with 4 removable, self powered
ehci0: handing over full speed device on port 1 to ohci0
wd0 at atabus2 drive 0
wd0: <ST2000DL003-9VT166>
wd0: drive supports 16-sector PIO transfers, LBA48 addressing
wd0: 1863 GB, 3876021 cyl, 16 head, 63 sec, 512 bytes/sect x 3907029168 sectors
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133), WRITE DMA FUA, NCQ (32 tags)
wd0(ahcisata1:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA), NCQ (31 tags)
atapibus0 at atabus4: 1 targets
cd0 at atapibus0 drive 0: <DRW-24D5MT, K28H2EA3236, 1.10> cdrom removable
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
cd0(ahcisata1:2:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA)
ehci0: handing over full speed device on port 2 to ohci0
ehci0: handing over full speed device on port 3 to ohci0
uplcom0 at uhub7 port 1
uplcom0: Prolific Technology Inc. (0x67b) USB-Serial Controller (0x2303), rev 1.10/3.00, addr 2
ucom0 at uplcom0
ehci0: handing over full speed device on port 5 to ohci0
boot device: wd0
root on wd0a dumps on wd0b
/: replaying log to memory
root file system type: ffs
kern.module.path=/stand/amd64/8.99.12/modules
/: replaying log to disk
cd0(ahcisata1:2:0): DEFERRED ERROR, key = 0x2
uplcom1 at uhub7 port 2
uplcom1: Prolific Technology Inc. (0x67b) USB 2.0 To COM Device (0x2303), rev 1.10/3.00, addr 3
ucom1 at uplcom1
uplcom2 at uhub7 port 3
uplcom2: Prolific Technology Inc. (0x67b) USB-Serial Controller (0x2303), rev 1.10/3.00, addr 4
ucom2 at uplcom2
uplcom3 at uhub7 port 5
uplcom3: Prolific Technology (0x67b) PL2303 Serial adapter (ATEN/IOGEAR UC232A) (0x2303), rev 1.10/2.02, addr 5
ucom3 at uplcom3
aibs0: warning over limit on 'CPU Temperature'
aibs0: warning under limit on 'Chassis Fan Speed'
aibs0: warning under limit on 'Power Fan Speed'
wsdisplay0: screen 0 added (default, vt100 emulation)
wsdisplay0: screen 1 added (default, vt100 emulation)
wsdisplay0: screen 2 added (default, vt100 emulation)
wsdisplay0: screen 3 added (default, vt100 emulation)
wsdisplay0: screen 4 added (default, vt100 emulation)
>How-To-Repeat:
On this machine: shutdown -p now
>Fix:
n/a
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: kern-bug-people->skrll
Responsible-Changed-By: skrll@NetBSD.org
Responsible-Changed-When: Sat, 03 Mar 2018 13:15:58 +0000
Responsible-Changed-Why:
Take
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/53066: nearly reproducable softint panic on shutdown
Date: Sat, 21 Apr 2018 14:56:01 +0200
This patch seems to fix it for me. It looks like a good enough compromise
from a layering point of view.
Maybe xhci should mask interrupts earlier (or the intr handler should not
return 1 in this case), but that is something for someone with better
hardware knowledge to decide.
Martin
Index: usb.c
===================================================================
RCS file: /cvsroot/src/sys/dev/usb/usb.c,v
retrieving revision 1.168
diff -u -p -r1.168 usb.c
--- usb.c 28 Oct 2017 00:37:12 -0000 1.168
+++ usb.c 21 Apr 2018 12:27:31 -0000
@@ -1081,7 +1081,7 @@ usb_schedsoftintr(struct usbd_bus *bus)
if (bus->ub_usepolling) {
bus->ub_methods->ubm_softint(bus);
- } else {
+ } else if (bus->ub_soft != NULL) { /* bus may have been detached */
kpreempt_disable();
softint_schedule(bus->ub_soft);
kpreempt_enable();
From: =?UTF-8?B?SmFyb23DrXIgRG9sZcSNZWs=?= <jaromir.dolecek@gmail.com>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc:
Subject: Re: kern/53066: nearly reproducable softint panic on shutdown
Date: Sat, 21 Apr 2018 18:07:16 +0200
I have the same issue, for me it happens on every reboot, if I boot
from USB 2.0 stick. Started to look at it more closely after seeing
your ub_soft patch.
The interrupt in my case was triggered from scsipi_prevent() via
sd_lastclose(). At this moment I don't expect the bus to be detached
yet, so I started to investigated more.
Eventually turned out this patch on xhci.c side resolves my issue -
use the usb2 bus for scheduling the softint. Of course, this probably
breaks USB 3.0 devices. But it shows the root problem, in that
xhci_intr() triggers the softint handling on wrong bus, which is
already detached by the time.
I may figure what is the right thing to do here, but help would be
appreciated, as I'm not familiar with how XHCI works. One possible
solution would be to simply config_detach() in xhci_detach() sc_bus2
first - seems the underlying usb code seems to not care on which the
callback is triggered.
Jaromir
Index: xhci.c
===================================================================
RCS file: /cvsroot/src/sys/dev/usb/xhci.c,v
retrieving revision 1.88
diff -u -p -r1.88 xhci.c
--- xhci.c 21 Apr 2018 15:53:24 -0000 1.88
+++ xhci.c 21 Apr 2018 15:53:50 -0000
@@ -1236,7 +1236,7 @@ xhci_intr(void *v)
ret = xhci_intr1(sc);
if (ret) {
- usb_schedsoftintr(&sc->sc_bus);
+ usb_schedsoftintr(&sc->sc_bus2);
}
done:
mutex_spin_exit(&sc->sc_intr_lock);
2018-04-21 15:00 GMT+02:00 Martin Husemann <martin@duskware.de>:
> The following reply was made to PR kern/53066; it has been noted by GNATS.
>
> From: Martin Husemann <martin@duskware.de>
> To: gnats-bugs@NetBSD.org
> Cc:
> Subject: Re: kern/53066: nearly reproducable softint panic on shutdown
> Date: Sat, 21 Apr 2018 14:56:01 +0200
>
> This patch seems to fix it for me. It looks like a good enough compromise
> from a layering point of view.
>
> Maybe xhci should mask interrupts earlier (or the intr handler should not
> return 1 in this case), but that is something for someone with better
> hardware knowledge to decide.
>
> Martin
Responsible-Changed-From-To: skrll->jdolecek
Responsible-Changed-By: jdolecek@NetBSD.org
Responsible-Changed-When: Sun, 22 Apr 2018 07:49:10 +0000
Responsible-Changed-Why:
Looking at this.
State-Changed-From-To: open->feedback
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Sun, 22 Apr 2018 07:49:10 +0000
State-Changed-Why:
I've committed change which fixes this for me. Can you test?
sys/dev/usb/xhci.c rev 1.89
From: "Jaromir Dolecek" <jdolecek@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/53066 CVS commit: src/sys/dev/usb
Date: Sun, 22 Apr 2018 07:47:15 +0000
Module Name: src
Committed By: jdolecek
Date: Sun Apr 22 07:47:15 UTC 2018
Modified Files:
src/sys/dev/usb: xhci.c
Log Message:
trigger the softint processing on that child bus which is not detached yet
fixes PR kern/53066 by Martin Husemann
To generate a diff of this commit:
cvs rdiff -u -r1.88 -r1.89 src/sys/dev/usb/xhci.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/53066: nearly reproducable softint panic on shutdown
Date: Mon, 23 Apr 2018 09:55:01 +0200
The code in -current does NOT work for me:
[ 42.9429700] panic: kernel diagnostic assertion "sc->sc_child || sc->sc_child2" failed: file "../../../../dev/usb/xhci.c", line 1239
[ 42.9429700] cpu0: Begin traceback...
[ 42.9429700] vpanic() at netbsd:vpanic+0x16f
[ 42.9429700] ch_voltag_convert_in() at netbsd:ch_voltag_convert_in
[ 42.9429700] xhci_intr() at netbsd:xhci_intr+0xd4
[ 42.9429700] intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x36
[ 42.9429700] Xhandle_ioapic_level1() at netbsd:Xhandle_ioapic_level1+0x6a
[ 42.9429700] --- interrupt ---
[ 42.9429700] Xspllower() at netbsd:Xspllower+0xe
[ 42.9429700] aprint_normal_dev() at netbsd:aprint_normal_dev+0x47
[ 42.9429700] config_detach() at netbsd:config_detach+0x410
[ 42.9429700] config_detach_all() at netbsd:config_detach_all+0x97
[ 42.9429700] cpu_reboot() at netbsd:cpu_reboot+0x19a
[ 42.9429700] sys_reboot() at netbsd:sys_reboot+0x75
[ 42.9429700] syscall() at netbsd:syscall+0x208
I also tested NetBSD 8.0 RC1 and the problem does not happen there for me.
Martin
From: =?UTF-8?B?SmFyb23DrXIgRG9sZcSNZWs=?= <jaromir.dolecek@gmail.com>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc:
Subject: Re: kern/53066: nearly reproducable softint panic on shutdown
Date: Mon, 23 Apr 2018 13:30:26 +0200
Can you please check backtrace for reboot command, to see what in
particular triggered the interrupt? It was waiting on xscmd for my
sd(4) case, might be something else for yours.
It's very strange that interrupt for xhci() would be triggered when
both child busses are already detached. Or does this happen during
boot, when the children are not attached yet?
Jaromir
2018-04-23 10:00 GMT+02:00 Martin Husemann <martin@duskware.de>:
> The following reply was made to PR kern/53066; it has been noted by GNATS.
>
> From: Martin Husemann <martin@duskware.de>
> To: gnats-bugs@NetBSD.org
> Cc:
> Subject: Re: kern/53066: nearly reproducable softint panic on shutdown
> Date: Mon, 23 Apr 2018 09:55:01 +0200
>
> The code in -current does NOT work for me:
>
> [ 42.9429700] panic: kernel diagnostic assertion "sc->sc_child || sc->sc_child2" failed: file "../../../../dev/usb/xhci.c", line 1239
> [ 42.9429700] cpu0: Begin traceback...
> [ 42.9429700] vpanic() at netbsd:vpanic+0x16f
> [ 42.9429700] ch_voltag_convert_in() at netbsd:ch_voltag_convert_in
> [ 42.9429700] xhci_intr() at netbsd:xhci_intr+0xd4
> [ 42.9429700] intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x36
> [ 42.9429700] Xhandle_ioapic_level1() at netbsd:Xhandle_ioapic_level1+0x6a
> [ 42.9429700] --- interrupt ---
> [ 42.9429700] Xspllower() at netbsd:Xspllower+0xe
> [ 42.9429700] aprint_normal_dev() at netbsd:aprint_normal_dev+0x47
> [ 42.9429700] config_detach() at netbsd:config_detach+0x410
> [ 42.9429700] config_detach_all() at netbsd:config_detach_all+0x97
> [ 42.9429700] cpu_reboot() at netbsd:cpu_reboot+0x19a
> [ 42.9429700] sys_reboot() at netbsd:sys_reboot+0x75
> [ 42.9429700] syscall() at netbsd:syscall+0x208
>
>
> I also tested NetBSD 8.0 RC1 and the problem does not happen there for me.
>
> Martin
>
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: jdolecek@NetBSD.org
Subject: Re: kern/53066: nearly reproducable softint panic on shutdown
Date: Mon, 23 Apr 2018 13:53:00 +0200
On Mon, Apr 23, 2018 at 11:35:00AM +0000, Jaromír Dole?ek wrote:
> Can you please check backtrace for reboot command, to see what in
> particular triggered the interrupt? It was waiting on xscmd for my
> sd(4) case, might be something else for yours.
Sure:
db{0}> trace/a ffffe4042bba04e0
trace: pid 37 lid 1 at 0xffff800144becd70
sleepq_block() at netbsd:sleepq_block+0x91
cv_wait() at netbsd:cv_wait+0xfb
kthread_join() at netbsd:kthread_join+0x89
audiodetach() at netbsd:audiodetach+0xcd
config_detach() at netbsd:config_detach+0x121
config_detach_all() at netbsd:config_detach_all+0x97
cpu_reboot() at netbsd:cpu_reboot+0x19a
sys_reboot() at netbsd:sys_reboot+0x75
syscall() at netbsd:syscall+0x208
Nothing related to USB or umass.
> It's very strange that interrupt for xhci() would be triggered when
> both child busses are already detached.
Yes, but that is what is happening:
[ 1.1321399] xhci0 at pci4 dev 0 function 0: vendor 1033 product 0194 (rev. 0x03)
[ 1.1321399] xhci0: interrupting at ioapic0 pin 18
[ 1.1321399] usb0 at xhci0: USB revision 3.0
[ 1.1321399] usb1 at xhci0: USB revision 2.0
..
[ 1.1321399] fwohci0: BUS reset
[ 1.1321399] fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode
[ 1.1321399] ieee1394if0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me)
[ 1.1321399] ieee1394if0: bus manager 0
[ 2.0103103] uhub0 at usb0: NetBSD (0000) xHCI Root Hub (0000), class 9/0, rev 3.00/1.00, addr 0
[ 2.1127363] uhub1 at usb1: NetBSD (0000) xHCI Root Hub (0000), class 9/0, rev 2.00/1.00, addr 0
[ 2.8278330] uhub2 at usb3: NetBSD (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
..
# reboot
reboot: rebooted by root
[ 44.5996539] syncing disks... done
[ 44.6396593] uhub1: detached
[ 44.6696670] uhub0: detached
[ 44.6996751] atabus0: detached
[ 44.7396859] cd0: detached
[ 44.7696940] usb1: detached
[ 44.7997124] usb0: detached
[ 44.8297205] atabus1: detached
[ 44.8697210] atapibus0: detached
[ 44.9097317] wd0: detached
[ 44.9397399] uhub6: detached
[ 44.9697480] uhub5: detached
[ 44.9997560] uhub4: detached
[ 45.0397668] uhub3: detached
[ 45.0697749] uhub2: detached
[ 45.0997830] spkr1: detached
[ 45.1397938] Skipping crash dump on recursive panic
[ 45.1397938] panic: kernel diagnostic assertion "sc->sc_child || sc->sc_child2" failed: file "../../../../dev/usb/xhci.c", line 1239
[ 45.1397938] cpu0: Begin traceback...
[ 45.1397938] vpanic() at netbsd:vpanic+0x16f
[ 45.1397938] ch_voltag_convert_in() at netbsd:ch_voltag_convert_in
[ 45.1397938] xhci_intr() at netbsd:xhci_intr+0xd4
[ 45.1397938] intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x36
[ 45.1397938] Xhandle_ioapic_level1() at netbsd:Xhandle_ioapic_level1+0x6a
[ 45.1397938] --- interrupt ---
[ 45.1397938] x86_pause() at netbsd:x86_pause
[ 45.1397938] callout_softclock() at netbsd:callout_softclock+0x41a
[ 45.1397938] softint_dispatch() at netbsd:softint_dispatch+0xee
[ 45.1397938] DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80013d2db0f0
[ 45.1397938] Xsoftintr() at netbsd:Xsoftintr+0x4f
Martin
From: =?UTF-8?B?SmFyb23DrXIgRG9sZcSNZWs=?= <jaromir.dolecek@gmail.com>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc:
Subject: Re: kern/53066: nearly reproducable softint panic on shutdown
Date: Mon, 23 Apr 2018 20:23:11 +0200
So it's apparently mishandled shared interrupt in your case. Checked
xhci_intr1(), it simply does not check if it should handle the
interrupt, and always triggers the usb callback.
Looking at dmesg, I see pin 18 intr is shared with ohci0-3, and
nouveau0. From the detach list before panic, I see strikingly missing
e.g. usb7, to which uplcom0 is attached. So I guess something is still
triggering interrupts for uplcom0/ohci3 during detach, which xhci0
misinterprets as something for itself.
Quick fix would be to check sc_child/sc_child2 and skip the
usb_schedsoftintr() call. Correct fix would be to fix xhci_intr() to
check if it should handle the interrupt.
I'll see what can be done to improve this.
Jaromir
2018-04-23 13:53 GMT+02:00 Martin Husemann <martin@duskware.de>:
> On Mon, Apr 23, 2018 at 11:35:00AM +0000, Jarom=C3=ADr Dole?ek wrote:
>> Can you please check backtrace for reboot command, to see what in
>> particular triggered the interrupt? It was waiting on xscmd for my
>> sd(4) case, might be something else for yours.
>
> Sure:
>
> db{0}> trace/a ffffe4042bba04e0
> trace: pid 37 lid 1 at 0xffff800144becd70
> sleepq_block() at netbsd:sleepq_block+0x91
> cv_wait() at netbsd:cv_wait+0xfb
> kthread_join() at netbsd:kthread_join+0x89
> audiodetach() at netbsd:audiodetach+0xcd
> config_detach() at netbsd:config_detach+0x121
> config_detach_all() at netbsd:config_detach_all+0x97
> cpu_reboot() at netbsd:cpu_reboot+0x19a
> sys_reboot() at netbsd:sys_reboot+0x75
> syscall() at netbsd:syscall+0x208
>
> Nothing related to USB or umass.
>
>> It's very strange that interrupt for xhci() would be triggered when
>> both child busses are already detached.
>
> Yes, but that is what is happening:
>
> [ 1.1321399] xhci0 at pci4 dev 0 function 0: vendor 1033 product 0194 (=
rev. 0x03)
> [ 1.1321399] xhci0: interrupting at ioapic0 pin 18
> [ 1.1321399] usb0 at xhci0: USB revision 3.0
> [ 1.1321399] usb1 at xhci0: USB revision 2.0
> ..
> [ 1.1321399] fwohci0: BUS reset
> [ 1.1321399] fwohci0: node_id=3D0xc800ffc0, gen=3D1, CYCLEMASTER mode
> [ 1.1321399] ieee1394if0: 1 nodes, maxhop <=3D 0 cable IRM irm(0) (me)
> [ 1.1321399] ieee1394if0: bus manager 0
> [ 2.0103103] uhub0 at usb0: NetBSD (0000) xHCI Root Hub (0000), class 9=
/0, rev 3.00/1.00, addr 0
> [ 2.1127363] uhub1 at usb1: NetBSD (0000) xHCI Root Hub (0000), class 9=
/0, rev 2.00/1.00, addr 0
> [ 2.8278330] uhub2 at usb3: NetBSD (0000) EHCI root hub (0000), class 9=
/0, rev 2.00/1.00, addr 1
> ..
> # reboot
> reboot: rebooted by root
> [ 44.5996539] syncing disks... done
> [ 44.6396593] uhub1: detached
> [ 44.6696670] uhub0: detached
> [ 44.6996751] atabus0: detached
> [ 44.7396859] cd0: detached
> [ 44.7696940] usb1: detached
> [ 44.7997124] usb0: detached
> [ 44.8297205] atabus1: detached
> [ 44.8697210] atapibus0: detached
> [ 44.9097317] wd0: detached
> [ 44.9397399] uhub6: detached
> [ 44.9697480] uhub5: detached
> [ 44.9997560] uhub4: detached
> [ 45.0397668] uhub3: detached
> [ 45.0697749] uhub2: detached
> [ 45.0997830] spkr1: detached
> [ 45.1397938] Skipping crash dump on recursive panic
> [ 45.1397938] panic: kernel diagnostic assertion "sc->sc_child || sc->sc=
_child2" failed: file "../../../../dev/usb/xhci.c", line 1239
> [ 45.1397938] cpu0: Begin traceback...
> [ 45.1397938] vpanic() at netbsd:vpanic+0x16f
> [ 45.1397938] ch_voltag_convert_in() at netbsd:ch_voltag_convert_in
> [ 45.1397938] xhci_intr() at netbsd:xhci_intr+0xd4
> [ 45.1397938] intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x36
> [ 45.1397938] Xhandle_ioapic_level1() at netbsd:Xhandle_ioapic_level1+0x=
6a
> [ 45.1397938] --- interrupt ---
> [ 45.1397938] x86_pause() at netbsd:x86_pause
> [ 45.1397938] callout_softclock() at netbsd:callout_softclock+0x41a
> [ 45.1397938] softint_dispatch() at netbsd:softint_dispatch+0xee
> [ 45.1397938] DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff800=
13d2db0f0
> [ 45.1397938] Xsoftintr() at netbsd:Xsoftintr+0x4f
>
>
>
> Martin
From: "Jaromir Dolecek" <jdolecek@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/53066 CVS commit: src/sys/dev/usb
Date: Mon, 23 Apr 2018 19:55:00 +0000
Module Name: src
Committed By: jdolecek
Date: Mon Apr 23 19:55:00 UTC 2018
Modified Files:
src/sys/dev/usb: xhci.c xhcireg.h
Log Message:
enable code to only trigger usb processing when EINT is set, to
avoid misinterpreting shared interrupt for another device
when clearing USBSTS, actually preserve the bits which spec requires to
preserve, and actually clear bit 1, which should be actually always
cleared to zero by spec
also #ifdef XHCI_DEBUG some unnecessary register reads
this should finally resolve PR kern/53066 also for Martin
To generate a diff of this commit:
cvs rdiff -u -r1.89 -r1.90 src/sys/dev/usb/xhci.c
cvs rdiff -u -r1.9 -r1.10 src/sys/dev/usb/xhcireg.h
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: =?UTF-8?B?SmFyb23DrXIgRG9sZcSNZWs=?= <jaromir.dolecek@gmail.com>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc:
Subject: Re: kern/53066: nearly reproducable softint panic on shutdown
Date: Mon, 23 Apr 2018 22:01:16 +0200
Can you try with src/sys/dev/usb/xhci.c rev. 1.90? Added code which
should help with the shared interrupt, if that is indeed the problem.
On my system the xHCI uses MSI and doesn't share interrupt with
anything. I don't see any panic on reboot even there are no attached
devices.
Jaromir
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/53066: nearly reproducable softint panic on shutdown
Date: Tue, 24 Apr 2018 10:19:13 +0200
The latest version works fine for me!
Martin
State-Changed-From-To: feedback->pending-pullups
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Tue, 24 Apr 2018 15:21:14 +0000
State-Changed-Why:
Requested pullup to -8, #787
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/53066 CVS commit: [netbsd-8] src/sys/dev/usb
Date: Sat, 5 May 2018 15:05:40 +0000
Module Name: src
Committed By: martin
Date: Sat May 5 15:05:39 UTC 2018
Modified Files:
src/sys/dev/usb [netbsd-8]: xhci.c xhcireg.h
Log Message:
Pull up following revision(s) (requested by jdolecek in ticket #787):
sys/dev/usb/xhci.c: revision 1.88-1.90
sys/dev/usb/xhcireg.h: revision 1.10
add KASSERT() that sc_child* is set to NULL after child detach; just for
readability, it's not immediatelly obvious this is done in xhci_childdet()
no functional changes
trigger the softint processing on that child bus which is not detached yet
fixes PR kern/53066 by Martin Husemann
enable code to only trigger usb processing when EINT is set, to
avoid misinterpreting shared interrupt for another device
when clearing USBSTS, actually preserve the bits which spec requires to
preserve, and actually clear bit 1, which should be actually always
cleared to zero by spec
also #ifdef XHCI_DEBUG some unnecessary register reads
this should finally resolve PR kern/53066 also for Martin
To generate a diff of this commit:
cvs rdiff -u -r1.72.2.5 -r1.72.2.6 src/sys/dev/usb/xhci.c
cvs rdiff -u -r1.9 -r1.9.6.1 src/sys/dev/usb/xhcireg.h
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: pending-pullups->closed
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Fri, 01 Jun 2018 18:44:03 +0000
State-Changed-Why:
Pullups done.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.43 2018/01/16 07:36:43 maya Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2017
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home