NetBSD Problem Report #53894
From www@NetBSD.org Sat Jan 19 22:29:44 2019
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 0B2227A154
for <gnats-bugs@gnats.NetBSD.org>; Sat, 19 Jan 2019 22:29:44 +0000 (UTC)
Message-Id: <20190119222942.0377F7A214@mollari.NetBSD.org>
Date: Sat, 19 Jan 2019 22:29:41 +0000 (UTC)
From: vezhlys@gmail.com
Reply-To: vezhlys@gmail.com
To: gnats-bugs@NetBSD.org
Subject: "ehci_sync_hc: timed out" with ACPI/SMP enabled on eBox 3352DX3-AP
X-Send-Pr-Version: www-1.0
>Number: 53894
>Category: port-i386
>Synopsis: "ehci_sync_hc: timed out" with ACPI/SMP enabled on eBox 3352DX3-AP
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: port-i386-maintainer
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sat Jan 19 22:30:00 +0000 2019
>Last-Modified: Tue Mar 05 08:00:02 +0000 2024
>Originator: Andrius V
>Release: 8.99.30
>Organization:
>Environment:
NetBSD vertexpc 8.99.30 NetBSD 8.99.30 (GENERIC) #0: Tue Jan 15 14:23:56 UTC 2019 mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/i386/compile/GENERIC i386
>Description:
Recent testing showed that enabling IDE legacy mode in BIOS allows to boot into NetBSD on DM&P eBox 3352DX3-AP VortexDX3 based system with ACPI/SMP enabled (without legacy mode, IDE controller timeouts). However, there is one major caveat that USB devices fail to initialize in this case with repetitive "ehci_sync_hc: timed out" message in boot log, which eventually leads to next lines "uhub1: autoconfiguration error: device problem, disabling port 1", "uhub1: port 2, device disappeared after reset". However, USB devices initialize and work properly if system is booted without ACPI/SMP support (boot -12). Please see dmesg messages for comparison below. Only occasional disconnect/reconnect may happen while typing on keyboard in this case.
dmesg with ACPI/SMP:
NetBSD 8.99.30 (GENERIC) #0: Tue Jan 15 14:23:56 UTC 2019
mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/i386/compile/GENERIC
total memory = 1983 MB
avail memory = 1929 MB
timecounter: Timecounters tick every 10.000 msec
Kernelized RAIDframe activated
running cgd selftest aes-xts-256 aes-xts-512 done
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
efi: missing or invalid systbl
SMBIOS rev. 2.7 @ 0xfaf20 (43 entries)
RDC Semiconductor Co., Ltd. EMKORE (1.0 )
mainbus0 (root)
ACPI: RSDP 0x00000000000F7920 000024 (v03 ACPIAM)
ACPI: XSDT 0x000000007BFD0100 00004C (v01 R D C QA09 20151227 MSFT 00000097)
ACPI: FACP 0x000000007BFD0290 0000F4 (v04 R D C OEMFACP 20151227 MSFT 00000097)
ACPI: DSDT 0x000000007BFD05D0 005329 (v02 EMKORE EMKORE00 00000000 INTL 20060707)
ACPI: FACS 0x000000007BFDE000 000040
ACPI: APIC 0x000000007BFD0390 00005C (v02 R D C APIC1706 20151227 MSFT 00000097)
ACPI: MSDM 0x000000007BFD03F0 000055 (v03 R D C QA09 20151227 MSFT 00000097)
ACPI: SLIC 0x000000007BFD0450 000176 (v01 R D C QA09 20151227 MSFT 00000097)
ACPI: OEMB 0x000000007BFDE040 000082 (v01 R D C AMI_OEM 20151227 MSFT 00000097)
ACPI: 1 ACPI AML tables successfully acquired and loaded
ioapic0 at mainbus0 apid 2: pa 0xfec00000, version 0x20, 24 pins
cpu0 at mainbus0 apid 0
cpu0: 2 page colors
cpu0: calibrating local timer
cpu0: apic clock running at 666 MHz
cpu0: Vortex86DX3, id 0x611
cpu0: package 0, core 0, smt 0
cpu1 at mainbus0 apid 1
cpu1: 2 page colors
cpu1: Vortex86DX3, id 0x611
cpu1: package 1, core 0, smt 0
acpi0 at mainbus0: Intel ACPICA 20181213
acpi0: X/RSDT: OemId <R D C , QA09 ,20151227>, AslId <MSFT,00000097>
acpi0: SCI interrupting at int 9
acpi0: fixed power button present
timecounter: Timecounter "ACPI-Safe" frequency 3579545 Hz quality 900
acpi0: ACPI-Safe 24-bit timer
acpiec0 at acpi0 (EC0, PNP0C09): io 0x62,0x66
attimer1 at acpi0 (TMR, PNP0100): io 0x40-0x43 irq 0
pcppi1 at acpi0 (SPKR, PNP0800): io 0x61
midi0 at pcppi1: PC speaker
sysbeep0 at pcppi1
COPR (PNP0C04) at acpi0 not configured
RMSC (PNP0C02) at acpi0 not configured
OMSC (PNP0C02) at acpi0 not configured
acpiacad0 at acpi0 (AC0, ACPI0003): ACPI AC Adapter
RMEM (PNP0C01) at acpi0 not configured
acpibut0 at acpi0 (PWRB, PNP0C0C-170): ACPI Power Button
acpibut1 at acpi0 (SLPB, PNP0C0E): ACPI Sleep Button
acpilid0 at acpi0 (LID, PNP0C0D): ACPI Lid Switch
acpitz0 at acpi0 (THRM): cpu0
acpitz0: polling interval 30.0 seconds
acpitz0: levels: critical 92.0 C, passive 78.0 C, passive cooling
apm0 at acpi0: Power Management spec V1.2
attimer1: attached to pcppi1
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0: vendor 17f3 product 6023 (rev. 0x02)
ppb0 at pci0 dev 1 function 0: vendor 17f3 product 1031 (rev. 0x01)
ppb0: PCI Express capability version 1 <PCI/PCI-X to PCI-E Bridge> x1 @ 2.5GT/s
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
ppb1 at pci0 dev 2 function 0: vendor 17f3 product 1031 (rev. 0x01)
ppb1: PCI Express capability version 1 <PCI/PCI-X to PCI-E Bridge> x1 @ 2.5GT/s
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
pcib0 at pci0 dev 7 function 0: vendor 17f3 product 6035 (rev. 0x01)
pcib1 at pci0 dev 7 function 1: vendor 17f3 product 6035 (rev. 0x01)
vte0 at pci0 dev 8 function 0: vendor 17f3 product 6040 (rev. 0x00)
vte0: Ethernet address xx:xx:xx:xx:xx:xx
vte0: interrupting at ioapic0 pin 19
ukphy0 at vte0 phy 1: OUI 0xfcff2f, model 0x0005, rev. 0
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
ohci0 at pci0 dev 10 function 0: vendor 17f3 product 6060 (rev. 0x14)
ohci0: interrupting at ioapic0 pin 18
ohci0: OHCI version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
ehci0 at pci0 dev 10 function 1: vendor 17f3 product 6061 (rev. 0x08)
ehci0: interrupting at ioapic0 pin 21
ehci0: EHCI version 1.0
ehci0: 1 companion controller, 4 ports: ohci0
usb1 at ehci0: USB revision 2.0
rdcide0 at pci0 dev 12 function 0: RDC R1012 IDE controller (rev. 0x02)
rdcide0: bus-master DMA support present
rdcide0: primary channel configured to compatibility mode
rdcide0: primary channel interrupting at ioapic0 pin 14
atabus0 at rdcide0 channel 0
rdcide0: secondary channel configured to compatibility mode
rdcide0: secondary channel ignored (disabled)
vga0 at pci0 dev 13 function 0: vendor 17f3 product 2015 (rev. 0x00)
wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
drm at vga0 not configured
hdaudio0 at pci0 dev 14 function 0: HD Audio Controller
hdaudio0: interrupting at ioapic0 pin 18
hdafg0 at hdaudio0 vendor 0x10EC product 0x0262 nid 0x01: vendor 10ec product 0262
hdafg0: DAC00 2ch: Speaker [Jack]
nid=14 [pin: Speaker (Green Jack)]
nid=0C [audio mixer] [source: dac, record]
nid=02 [audio output] [source: dac]
nid=0B [audio mixer] [source: record]
hdafg0: DIG01 2ch: Digital Out [Jack]
nid=1E [pin: Digital Out (Black Jack)]
nid=06 [audio output] [source: dac]
hdafg0: ADC02 2ch: Mic In [Jack]
nid=19 [pin: Mic In (Pink Jack)]
hdafg0: ADC03 2ch: Line In [Jack]
nid=1A [pin: Line In (Blue Jack)]
hdafg0: 2ch/2ch 44100Hz 48000Hz 96000Hz 192000Hz PCM16 PCM20 PCM24 AC3
audio0 at hdafg0: full duplex, playback, capture, mmap, independent
hdafg0: Virtual format configured - Format SLINEAR, precision 16, channels 2, frequency 48000
hdafg0: Latency: 128 milliseconds
isa0 at pcib0
pckbc0 at isa0 port 0x60-0x64
pckbdprobe: reset error 5
pmsprobe: reset error 5
isa at pcib1 not configured
acpicpu0 at cpu0: ACPI CPU
acpicpu0: HALT not supported?
acpicpu0: C1: HLT, lat 0 us, pow 0 mW
acpicpu0: T0: I/O, lat 1 us, pow 0 mW, 100 %
acpicpu0: T1: I/O, lat 1 us, pow 0 mW, 88 %
acpicpu0: T2: I/O, lat 1 us, pow 0 mW, 76 %
acpicpu0: T3: I/O, lat 1 us, pow 0 mW, 64 %
acpicpu0: T4: I/O, lat 1 us, pow 0 mW, 52 %
acpicpu0: T5: I/O, lat 1 us, pow 0 mW, 40 %
acpicpu0: T6: I/O, lat 1 us, pow 0 mW, 28 %
acpicpu0: T7: I/O, lat 1 us, pow 0 mW, 16 %
acpicpu0: id 1, lapic id 0, cap 0x0000, flags 0x00100c25
acpicpu1 at cpu1: ACPI CPU
acpicpu1: HALT not supported?
acpicpu1: id 2, lapic id 1, cap 0x0000, flags 0x00000c21
Initializing SSP: ac3b5c83 62fd8be 75262144 3cdb1f14 80cf06f8 cb42734a 48c375bf 1af02484
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
cpu1: TSC skew=3 drift=-5
cpu1: running
TSC not known invariant on this CPU
acpiacad0: AC adapter online.
acpiacad0: AC adapter connected
acpicpu1: ACPI CPUs started
uhub0 at usb0: NetBSD (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub0: 4 ports with 4 removable, self powered
uhub1 at usb1: NetBSD (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub1: 4 ports with 4 removable, self powered
IPsec: Initialized Security Association Processing.
wd0 at atabus0 drive 0
wd0: <SS16G D0 RDC SD-IDE HOST CONTROLLER>
wd0: drive supports 1-sector PIO transfers, LBA addressing
wd0: 15193 MB, 30869 cyl, 16 head, 63 sec, 512 bytes/sect x 31116288 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
wd0(rdcide0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100) (using DMA)
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
uhub1: autoconfiguration error: device problem, disabling port 1
ehci0: handing over low speed device on port 2 to ohci0
uhub1: port 2, device disappeared after reset
ehci0: handing over full speed device on port 3 to ohci0
uhub1: port 3, device disappeared after reset
Searching for RAID components...
WARNING: 1 error while detecting hardware; check system log.
boot device: wd0
root on wd0a dumps on wd0b
root file system type: ffs
kern.module.path=/stand/i386/8.99.30/modules
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)
dmesg with ACPI/SMP disabled:
NetBSD 8.99.30 (GENERIC) #0: Tue Jan 15 14:23:56 UTC 2019
mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/i386/compile/GENERIC
total memory = 1983 MB
avail memory = 1929 MB
rnd: seeded with 128 bits
timecounter: Timecounters tick every 10.000 msec
Kernelized RAIDframe activated
running cgd selftest aes-xts-256 aes-xts-512 done
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
RDC Semiconductor Co., Ltd. EMKORE (1.0 )
mainbus0 (root)
cpu0 at mainbus0
cpu0: Vortex86DX3, id 0x611
cpu0: package 0, core 0, smt 0
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0: vendor 17f3 product 6023 (rev. 0x02)
ppb0 at pci0 dev 1 function 0: vendor 17f3 product 1031 (rev. 0x01)
ppb0: PCI Express capability version 1 <PCI/PCI-X to PCI-E Bridge> x1 @ 2.5GT/s
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
ppb1 at pci0 dev 2 function 0: vendor 17f3 product 1031 (rev. 0x01)
ppb1: PCI Express capability version 1 <PCI/PCI-X to PCI-E Bridge> x1 @ 2.5GT/s
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
pcib0 at pci0 dev 7 function 0: vendor 17f3 product 6035 (rev. 0x01)
pcib1 at pci0 dev 7 function 1: vendor 17f3 product 6035 (rev. 0x01)
vte0 at pci0 dev 8 function 0: vendor 17f3 product 6040 (rev. 0x00)
vte0: Ethernet address xx:xx:xx:xx:xx:xx
vte0: interrupting at irq 5
ukphy0 at vte0 phy 1: OUI 0xfcff2f, model 0x0005, rev. 0
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
ohci0 at pci0 dev 10 function 0: vendor 17f3 product 6060 (rev. 0x14)
ohci0: interrupting at irq 15
ohci0: OHCI version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
ehci0 at pci0 dev 10 function 1: vendor 17f3 product 6061 (rev. 0x08)
ehci0: interrupting at irq 7
ehci0: BIOS has given up ownership
ehci0: EHCI version 1.0
ehci0: 1 companion controller, 4 ports: ohci0
usb1 at ehci0: USB revision 2.0
rdcide0 at pci0 dev 12 function 0: RDC R1012 IDE controller (rev. 0x02)
rdcide0: bus-master DMA support present
rdcide0: primary channel configured to compatibility mode
rdcide0: primary channel interrupting at irq 14
atabus0 at rdcide0 channel 0
rdcide0: secondary channel configured to compatibility mode
rdcide0: secondary channel ignored (disabled)
vga0 at pci0 dev 13 function 0: vendor 17f3 product 2015 (rev. 0x00)
wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
drm at vga0 not configured
hdaudio0 at pci0 dev 14 function 0: HD Audio Controller
hdaudio0: interrupting at irq 15
hdafg0 at hdaudio0: vendor 10ec product 0262
hdafg0: DAC00 2ch: Speaker [Jack]
hdafg0: DIG01 2ch: Digital Out [Jack]
hdafg0: ADC02 2ch: Mic In [Jack]
hdafg0: ADC03 2ch: Line In [Jack]
hdafg0: 2ch/2ch 44100Hz 48000Hz 96000Hz 192000Hz PCM16 PCM20 PCM24 AC3
audio0 at hdafg0: full duplex, playback, capture, mmap, independent
hdafg0: Virtual format configured - Format SLINEAR, precision 16, channels 2, frequency 48000
hdafg0: Latency: 128 milliseconds
isa0 at pcib0
pckbc0 at isa0 port 0x60-0x64
attimer0 at isa0 port 0x40-0x43
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker
sysbeep0 at pcppi0
attimer0: attached to pcppi0
isa at pcib1 not configured
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
uhub0 at usb0: NetBSD (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub0: 4 ports with 4 removable, self powered
uhub1 at usb1: NetBSD (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub1: 4 ports with 4 removable, self powered
IPsec: Initialized Security Association Processing.
axen0 at uhub1 port 1
axen0: D-Link Elec. Corp. (0x2001) D-Link DUB-1312 (0x4a00), rev 2.10/1.00, addr 2
axen0: AX88179
axen0: Ethernet address xx:xx:xx:xx:xx:xx
rgephy0 at axen0 phy 3: RTL8169S/8110S/8211 1000BASE-T media interface, rev. 5
rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, auto
ehci0: handing over low speed device on port 2 to ohci0
wd0 at atabus0 drive 0
wd0: <SS16G D0 RDC SD-IDE HOST CONTROLLER>
wd0: drive supports 1-sector PIO transfers, LBA addressing
wd0: 15193 MB, 30869 cyl, 16 head, 63 sec, 512 bytes/sect x 31116288 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
wd0(rdcide0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100) (using DMA)
ehci0: handing over full speed device on port 3 to ohci0
boot device: wd0
root on wd0a dumps on wd0b
root file system type: ffs
kern.module.path=/stand/i386/8.99.30/modules
WARNING: clock gained 2 days
uhidev0 at uhub0 port 2 configuration 1 interface 0
uhidev0: USB (0x1c4f) USB Keykoard (0x02), rev 1.10/1.10, addr 2, iclass 3/1
ukbd0 at uhidev0: 8 Variable keys, 6 Array codes
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub0 port 2 configuration 1 interface 1
uhidev1: USB (0x1c4f) USB Keykoard (0x02), rev 1.10/1.10, addr 2, iclass 3/0
uhidev1: 2 report ids
uhid0 at uhidev1 reportid 1: input=2, output=0, feature=0
uhid1 at uhidev1 reportid 2: input=1, output=0, feature=0
uchcom0 at uhub0 port 3
uchcom0: QinHeng Electronics (0x1a86) USB2.0-Ser! (0x7523), rev 1.10/2.54, addr 3
ucom0 at uchcom0
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)
>How-To-Repeat:
Boot machine with ACPI/SMP enabled
>Fix:
N/A. Workaround is to boot without ACPI/SMP enabled.
>Release-Note:
>Audit-Trail:
From: "David H. Gutteridge" <david@gutteridge.ca>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP
enabled on eBox 3352DX3-AP
Date: Sat, 19 Jan 2019 20:09:06 -0500
It would probably be helpful to provide USB_DEBUG output. To do so,
please see http://www.netbsd.org/docs/kernel/#usb-debugging,
specifically the part following "USB debugging code has been changed in
NetBSD 7.99 and later."
Dave
From: Andrius V <vezhlys@gmail.com>
To: gnats-bugs@netbsd.org
Cc: port-i386-maintainer@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org
Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP enabled
on eBox 3352DX3-AP
Date: Sun, 20 Jan 2019 22:56:58 +0200
--000000000000d76b42057fe9fc98
Content-Type: text/plain; charset="UTF-8"
Hi,
Thank you for the prompt reply. I can provide this output but it is a
bit tricky since USB ports are the only available besides Ethernet and
VGA (and integrated vte ethernet controller doesn't work properly as
well unfortunately, so ssh is out of option). Does it matter when to
do this output? I guess non ACPI/SMP boot won't provide the required
data as well? Currently I did a cron job which outputs the vmstat -u
usbhist result to file on boot. Please see an attached report for
results. Theoretically, I can modify cronjob to post report a bit
later if needed.
Regards,
Andrius V
On Sun, Jan 20, 2019 at 3:10 AM David H. Gutteridge <david@gutteridge.ca> wrote:
>
> The following reply was made to PR port-i386/53894; it has been noted by GNATS.
>
> From: "David H. Gutteridge" <david@gutteridge.ca>
> To: gnats-bugs@netbsd.org
> Cc:
> Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP
> enabled on eBox 3352DX3-AP
> Date: Sat, 19 Jan 2019 20:09:06 -0500
>
> It would probably be helpful to provide USB_DEBUG output. To do so,
> please see http://www.netbsd.org/docs/kernel/#usb-debugging,
> specifically the part following "USB debugging code has been changed in
> NetBSD 7.99 and later."
>
> Dave
>
>
--000000000000d76b42057fe9fc98
Content-Type: text/plain; charset="US-ASCII"; name="usbhist.txt"
Content-Disposition: attachment; filename="usbhist.txt"
Content-Transfer-Encoding: base64
Content-ID: <f_jr5dkcck0>
X-Attachment-Id: f_jr5dkcck0
NTAwMDAgZW50cmllcywgbmV4dCBpcyAxNTUKMDAwMDAwLjAwMDAwMCBvaGNpX2luaXQjMUAwOiBj
YWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjMUAwOiBjYWxsZWQhCjAwMDAwMC4w
MDAwMDAgb2hjaV9hbGxvY19zZWQjMUAwOiBhbGxvY2F0aW5nIGNodW5rCjAwMDAwMC4wMDAwMDAg
b2hjaV9hbGxvY19zZWQjMkAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQj
M0AwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjNEAwOiBjYWxsZWQhCjAw
MDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjNUAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hj
aV9hbGxvY19zZWQjNkAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjN0Aw
OiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjOEAwOiBjYWxsZWQhCjAwMDAw
MC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjOUAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9h
bGxvY19zZWQjMTBAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzExQDA6
IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCMxMkAwOiBjYWxsZWQhCjAwMDAw
MC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjMTNAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lf
YWxsb2Nfc2VkIzE0QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCMxNUAw
OiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjMTZAMDogY2FsbGVkIQowMDAw
MDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzE3QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNp
X2FsbG9jX3NlZCMxOEAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjMTlA
MDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzIwQDA6IGNhbGxlZCEKMDAw
MDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCMyMUAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hj
aV9hbGxvY19zZWQjMjJAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzIz
QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCMyNEAwOiBjYWxsZWQhCjAw
MDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjMjVAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9o
Y2lfYWxsb2Nfc2VkIzI2QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCMy
N0AwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjMjhAMDogY2FsbGVkIQow
MDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzI5QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBv
aGNpX2FsbG9jX3NlZCMzMEAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQj
MzFAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzMyQDA6IGNhbGxlZCEK
MDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCMzM0AwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAg
b2hjaV9hbGxvY19zZWQjMzRAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2Vk
IzM1QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCMzNkAwOiBjYWxsZWQh
CjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjMzdAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAw
IG9oY2lfYWxsb2Nfc2VkIzM4QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3Nl
ZCMzOUAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjNDBAMDogY2FsbGVk
IQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzQxQDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAw
MCBvaGNpX2FsbG9jX3NlZCM0MkAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19z
ZWQjNDNAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzQ0QDA6IGNhbGxl
ZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCM0NUAwOiBjYWxsZWQhCjAwMDAwMC4wMDAw
MDAgb2hjaV9hbGxvY19zZWQjNDZAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nf
c2VkIzQ3QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCM0OEAwOiBjYWxs
ZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjNDlAMDogY2FsbGVkIQowMDAwMDAuMDAw
MDAwIG9oY2lfYWxsb2Nfc2VkIzUwQDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9j
X3NlZCM1MUAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjNTJAMDogY2Fs
bGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzUzQDA6IGNhbGxlZCEKMDAwMDAwLjAw
MDAwMCBvaGNpX2FsbG9jX3NlZCM1NEAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxv
Y19zZWQjNTVAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzU2QDA6IGNh
bGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCM1N0AwOiBjYWxsZWQhCjAwMDAwMC4w
MDAwMDAgb2hjaV9hbGxvY19zZWQjNThAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxs
b2Nfc2VkIzU5QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCM2MEAwOiBj
YWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjNjFAMDogY2FsbGVkIQowMDAwMDAu
MDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzYyQDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2Fs
bG9jX3NlZCM2M0AwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjNjRAMDog
Y2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzY1QDA6IGNhbGxlZCEKMDAwMDAw
LjAwMDAwMCBvaGNpX2FsbG9jX3NlZCM2NkAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9p
bml0IzFAMDogY29sZCBzdGFydGVkCjAwMDAwMC4wMDAwMDAgb2hjaV9pbml0IzFAMDogc2MgMHhj
M2Y3MzAwODogcmVzZXR0aW5nCjAwMDAwMC4wMDAwMDAgb2hjaV9kdW1wcmVncyMxQDA6IGNhbGxl
ZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2R1bXByZWdzIzFAMDogcmV2PTB4MDAwMDAxMTAgY29udHJv
bD0weDAwMDAwMGJmIGNvbW1hbmQ9MHgwMDAwMDAwMAowMDAwMDAuMDAwMDAwIG9oY2lfZHVtcHJl
Z3MjMUAwOiAgICAgICAgICAgICAgICBpbnRyc3RhdD0weDAwMDAwMDA0IGludHJlPTB4MDAwMDAw
MDAgaW50cmQ9MHgwMDAwMDAwMAowMDAwMDAuMDAwMDAwIG9oY2lfZHVtcHJlZ3MjMUAwOiAgICAg
ICAgICAgICAgICBoY2NhPTB4NDIwMDUwMDAgcGVyY3VyPTB4MDAwMDAwMDAgY3RybGhkPTB4NDIw
MDc3ZDAKMDAwMDAwLjAwMDAwMCBvaGNpX2R1bXByZWdzIzFAMDogICAgICAgICAgICAgICAgY3Ry
bGN1cj0weDAwMDAwMDAwIGJ1bGtoZD0weDQyMDA3N2EwIGJ1bGtjdXI9MHgwMDAwMDAwMAowMDAw
MDAuMDAwMDAwIG9oY2lfZHVtcHJlZ3MjMUAwOiAgICAgICAgICAgICAgICBkb25lPTB4MDAwMDAw
MDAgZm1pdmFsPTB4YTc3ODJlZGYgZm1yZW09MHg4MDAwMmUxNAowMDAwMDAuMDAwMDAwIG9oY2lf
ZHVtcHJlZ3MjMUAwOiAgICAgICAgICAgICAgICBmbW51bT0weDAwMDAwMDBjIHBlcnN0PTB4MDAw
MDJhMmYgbHN0aHJzPTB4MDAwMDA2MjgKMDAwMDAwLjAwMDAwMCBvaGNpX2R1bXByZWdzIzFAMDog
ICAgICAgICAgICAgICAgZGVzY2E9MHgwZjAwMDkwNCBkZXNjYj0weDAwMWUwMDAwIHN0YXQ9MHgw
MDAwMDAwMAowMDAwMDAuMDAwMDAwIG9oY2lfZHVtcHJlZ3MjMUAwOiAgICAgICAgICAgICAgICBw
b3J0MT0weDAwMDAwMDAwIHBvcnQyPTB4MDAwMDAwMDAKMDAwMDAwLjAwMDAwMCBvaGNpX2R1bXBy
ZWdzIzFAMDogICAgICAgICAgSENDQTogZnJhbWVfbnVtYmVyPTB4MDAwYyBkb25lX2hlYWQ9MHgw
MDAwMDAwMAowMDAwMDAuMDAwMDAwIG9oY2lfaW5pdCMxQDA6IGVuYWJsaW5nIDB4ODAwMDAwNWIK
MDAwMDAxLjk3Nzk4NyBvaGNpX29wZW4jMUAwOiBjYWxsZWQhCjAwMDAwMS45Nzc5OTMgb2hjaV9v
cGVuIzFAMDogcGlwZT0weGMzYWFiOTM4LCBhZGRyPTAsIGVuZHB0PTAgKDApCjAwMDAwMS45Nzgw
NDUgb2hjaV9yb290aHViX2N0cmwjMUAwOiBjYWxsZWQhCjAwMDAwMS45NzgwNTAgb2hjaV9yb290
aHViX2N0cmwjMUAwOiB0eXBlPTB4ODAgcmVxdWVzdD0wNgowMDAwMDEuOTc4MDU3IG9oY2lfcm9v
dGh1Yl9jdHJsIzFAMDogd1ZhbHVlPTB4MDEwMAowMDAwMDEuOTc4MDc1IG9oY2lfb3BlbiMyQDA6
IGNhbGxlZCEKMDAwMDAxLjk3ODA4MCBvaGNpX29wZW4jMkAwOiBwaXBlPTB4YzNhYWI5MzgsIGFk
ZHI9MCwgZW5kcHQ9MCAoMCkKMDAwMDAxLjk3ODA4OSBvaGNpX3Jvb3RodWJfY3RybCMyQDA6IGNh
bGxlZCEKMDAwMDAxLjk3ODA5NCBvaGNpX3Jvb3RodWJfY3RybCMyQDA6IHR5cGU9MHgwMCByZXF1
ZXN0PTA1CjAwMDAwMS45OTI1Mjkgb2hjaV9vcGVuIzNAMDogY2FsbGVkIQowMDAwMDEuOTkyNTMz
IG9oY2lfb3BlbiMzQDA6IHBpcGU9MHhjM2FhYjkzOCwgYWRkcj0xLCBlbmRwdD0wICgxKQowMDAw
MDEuOTkyNTQ1IG9oY2lfcm9vdGh1Yl9jdHJsIzNAMDogY2FsbGVkIQowMDAwMDEuOTkyNTUxIG9o
Y2lfcm9vdGh1Yl9jdHJsIzNAMDogdHlwZT0weDgwIHJlcXVlc3Q9MDYKMDAwMDAxLjk5MjU1NiBv
aGNpX3Jvb3RodWJfY3RybCMzQDA6IHdWYWx1ZT0weDAxMDAKMDAwMDAxLjk5MjU3MiBvaGNpX3Jv
b3RodWJfY3RybCM0QDA6IGNhbGxlZCEKMDAwMDAxLjk5MjU3NiBvaGNpX3Jvb3RodWJfY3RybCM0
QDA6IHR5cGU9MHg4MCByZXF1ZXN0PTA2CjAwMDAwMS45OTI1ODEgb2hjaV9yb290aHViX2N0cmwj
NEAwOiB3VmFsdWU9MHgwMzAwCjAwMDAwMS45OTI1ODkgb2hjaV9yb290aHViX2N0cmwjNUAwOiBj
YWxsZWQhCjAwMDAwMS45OTI1OTMgb2hjaV9yb290aHViX2N0cmwjNUAwOiB0eXBlPTB4ODAgcmVx
dWVzdD0wNgowMDAwMDEuOTkyNjAwIG9oY2lfcm9vdGh1Yl9jdHJsIzVAMDogd1ZhbHVlPTB4MDMw
MAowMDAwMDEuOTkyNjA5IG9oY2lfcm9vdGh1Yl9jdHJsIzZAMDogY2FsbGVkIQowMDAwMDEuOTky
NjE0IG9oY2lfcm9vdGh1Yl9jdHJsIzZAMDogdHlwZT0weDgwIHJlcXVlc3Q9MDYKMDAwMDAxLjk5
MjYxOSBvaGNpX3Jvb3RodWJfY3RybCM2QDA6IHdWYWx1ZT0weDAzMDEKMDAwMDAxLjk5MjYyNyBv
aGNpX3Jvb3RodWJfY3RybCM3QDA6IGNhbGxlZCEKMDAwMDAxLjk5MjYzMSBvaGNpX3Jvb3RodWJf
Y3RybCM3QDA6IHR5cGU9MHg4MCByZXF1ZXN0PTA2CjAwMDAwMS45OTI2MzYgb2hjaV9yb290aHVi
X2N0cmwjN0AwOiB3VmFsdWU9MHgwMzAxCjAwMDAwMS45OTI2NDggb2hjaV9yb290aHViX2N0cmwj
OEAwOiBjYWxsZWQhCjAwMDAwMS45OTI2NTMgb2hjaV9yb290aHViX2N0cmwjOEAwOiB0eXBlPTB4
ODAgcmVxdWVzdD0wNgowMDAwMDEuOTkyNjU4IG9oY2lfcm9vdGh1Yl9jdHJsIzhAMDogd1ZhbHVl
PTB4MDMwMgowMDAwMDEuOTkyNjY2IG9oY2lfcm9vdGh1Yl9jdHJsIzlAMDogY2FsbGVkIQowMDAw
MDEuOTkyNjcxIG9oY2lfcm9vdGh1Yl9jdHJsIzlAMDogdHlwZT0weDgwIHJlcXVlc3Q9MDYKMDAw
MDAxLjk5MjY3NiBvaGNpX3Jvb3RodWJfY3RybCM5QDA6IHdWYWx1ZT0weDAzMDIKMDAwMDAxLjk5
MzIzNyBvaGNpX3Jvb3RodWJfY3RybCMxMEAwOiBjYWxsZWQhCjAwMDAwMS45OTMyNDIgb2hjaV9y
b290aHViX2N0cmwjMTBAMDogdHlwZT0weDgwIHJlcXVlc3Q9MDYKMDAwMDAxLjk5MzI0OSBvaGNp
X3Jvb3RodWJfY3RybCMxMEAwOiB3VmFsdWU9MHgwMjAwCjAwMDAwMS45OTMyNjEgb2hjaV9yb290
aHViX2N0cmwjMTFAMDogY2FsbGVkIQowMDAwMDEuOTkzMjY2IG9oY2lfcm9vdGh1Yl9jdHJsIzEx
QDA6IHR5cGU9MHg4MCByZXF1ZXN0PTA2CjAwMDAwMS45OTMyNzAgb2hjaV9yb290aHViX2N0cmwj
MTFAMDogd1ZhbHVlPTB4MDIwMAowMDAwMDEuOTkzMjgwIG9oY2lfcm9vdGh1Yl9jdHJsIzEyQDA6
IGNhbGxlZCEKMDAwMDAxLjk5MzI4NCBvaGNpX3Jvb3RodWJfY3RybCMxMkAwOiB0eXBlPTB4MDAg
cmVxdWVzdD0wOQowMDAwMDEuOTkzMjk3IG9oY2lfcm9vdGh1Yl9jdHJsIzEzQDA6IGNhbGxlZCEK
MDAwMDAxLjk5MzMwMiBvaGNpX3Jvb3RodWJfY3RybCMxM0AwOiB0eXBlPTB4YTAgcmVxdWVzdD0w
NgowMDAwMDEuOTkzMzM4IG9oY2lfb3BlbiM0QDA6IGNhbGxlZCEKMDAwMDAxLjk5MzM0MyBvaGNp
X29wZW4jNEAwOiBwaXBlPTB4YzNhYWI4NTgsIGFkZHI9MSwgZW5kcHQ9MTI5ICgxKQowMDAwMDIu
MjAyNjU3IG9oY2lfcm9vdGh1Yl9jdHJsIzE0QDA6IGNhbGxlZCEKMDAwMDAyLjIwMjY2MiBvaGNp
X3Jvb3RodWJfY3RybCMxNEAwOiB0eXBlPTB4MjMgcmVxdWVzdD0wMwowMDAwMDIuMjAyNjY4IG9o
Y2lfcm9vdGh1Yl9jdHJsIzE0QDA6IHNldCBwb3J0IHBvd2VyIDEKMDAwMDAyLjIwMjY4MCBvaGNp
X3Jvb3RodWJfY3RybCMxNUAwOiBjYWxsZWQhCjAwMDAwMi4yMDI2ODUgb2hjaV9yb290aHViX2N0
cmwjMTVAMDogdHlwZT0weDIzIHJlcXVlc3Q9MDMKMDAwMDAyLjIwMjY4OSBvaGNpX3Jvb3RodWJf
Y3RybCMxNUAwOiBzZXQgcG9ydCBwb3dlciAyCjAwMDAwMi4yMDI2OTcgb2hjaV9yb290aHViX2N0
cmwjMTZAMDogY2FsbGVkIQowMDAwMDIuMjAyNzAyIG9oY2lfcm9vdGh1Yl9jdHJsIzE2QDA6IHR5
cGU9MHgyMyByZXF1ZXN0PTAzCjAwMDAwMi4yMDI3MDggb2hjaV9yb290aHViX2N0cmwjMTZAMDog
c2V0IHBvcnQgcG93ZXIgMwowMDAwMDIuMjAyNzE1IG9oY2lfcm9vdGh1Yl9jdHJsIzE3QDA6IGNh
bGxlZCEKMDAwMDAyLjIwMjcyMCBvaGNpX3Jvb3RodWJfY3RybCMxN0AwOiB0eXBlPTB4MjMgcmVx
dWVzdD0wMwowMDAwMDIuMjAyNzI2IG9oY2lfcm9vdGh1Yl9jdHJsIzE3QDA6IHNldCBwb3J0IHBv
d2VyIDQKMDAwMDAyLjcxMjgzNyBvaGNpX3Jvb3RodWJfY3RybCMxOEAwOiBjYWxsZWQhCjAwMDAw
Mi43MTI4NDEgb2hjaV9yb290aHViX2N0cmwjMThAMDogdHlwZT0weGEwIHJlcXVlc3Q9MDAKMDAw
MDAyLjcxMjg1MSBvaGNpX3Jvb3RodWJfY3RybCMxOUAwOiBjYWxsZWQhCjAwMDAwMi43MTI4NTYg
b2hjaV9yb290aHViX2N0cmwjMTlAMDogdHlwZT0weGEzIHJlcXVlc3Q9MDAKMDAwMDAyLjcxMjg2
MSBvaGNpX3Jvb3RodWJfY3RybCMxOUAwOiBnZXQgcG9ydCBzdGF0dXMgaT0xCjAwMDAwMi43MTI4
Njggb2hjaV9yb290aHViX2N0cmwjMTlAMDogcG9ydCBzdGF0dXM9MHgwMTAwCjAwMDAwMi43MTI4
NzYgb2hjaV9yb290aHViX2N0cmwjMjBAMDogY2FsbGVkIQowMDAwMDIuNzEyODgyIG9oY2lfcm9v
dGh1Yl9jdHJsIzIwQDA6IHR5cGU9MHhhMyByZXF1ZXN0PTAwCjAwMDAwMi43MTI4ODYgb2hjaV9y
b290aHViX2N0cmwjMjBAMDogZ2V0IHBvcnQgc3RhdHVzIGk9MgowMDAwMDIuNzEyODkxIG9oY2lf
cm9vdGh1Yl9jdHJsIzIwQDA6IHBvcnQgc3RhdHVzPTB4MDEwMAowMDAwMDIuNzEyOTAwIG9oY2lf
cm9vdGh1Yl9jdHJsIzIxQDA6IGNhbGxlZCEKMDAwMDAyLjcxMjkwNCBvaGNpX3Jvb3RodWJfY3Ry
bCMyMUAwOiB0eXBlPTB4YTMgcmVxdWVzdD0wMAowMDAwMDIuNzEyOTEwIG9oY2lfcm9vdGh1Yl9j
dHJsIzIxQDA6IGdldCBwb3J0IHN0YXR1cyBpPTMKMDAwMDAyLjcxMjkxNiBvaGNpX3Jvb3RodWJf
Y3RybCMyMUAwOiBwb3J0IHN0YXR1cz0weDAxMDAKMDAwMDAyLjcxMjkyNCBvaGNpX3Jvb3RodWJf
Y3RybCMyMkAwOiBjYWxsZWQhCjAwMDAwMi43MTI5Mjkgb2hjaV9yb290aHViX2N0cmwjMjJAMDog
dHlwZT0weGEzIHJlcXVlc3Q9MDAKMDAwMDAyLjcxMjkzNCBvaGNpX3Jvb3RodWJfY3RybCMyMkAw
OiBnZXQgcG9ydCBzdGF0dXMgaT00CjAwMDAwMi43MTI5Mzkgb2hjaV9yb290aHViX2N0cmwjMjJA
MDogcG9ydCBzdGF0dXM9MHgwMTAwCgo=
--000000000000d76b42057fe9fc98--
Responsible-Changed-From-To: port-i386-maintainer->skrll
Responsible-Changed-By: gutteridge@NetBSD.org
Responsible-Changed-When: Sun, 20 Jan 2019 23:50:13 +0000
Responsible-Changed-Why:
skrll: Please advise on any further detail needed here?
From: Nick Hudson <nick.hudson@gmx.co.uk>
To: gnats-bugs@NetBSD.org, port-i386-maintainer@netbsd.org,
netbsd-bugs@netbsd.org, gnats-admin@netbsd.org, gutteridge@NetBSD.org,
vezhlys@gmail.com
Cc:
Subject: Re: port-i386/53894 ("ehci_sync_hc: timed out" with ACPI/SMP enabled
on eBox 3352DX3-AP)
Date: Mon, 21 Jan 2019 06:40:28 +0000
On 20/01/2019 23:50, gutteridge@NetBSD.org wrote:
> Synopsis: "ehci_sync_hc: timed out" with ACPI/SMP enabled on eBox 3352DX3-AP
This sounds more like an MP/interrupt problem for your system than usb.
ehcidebug=1 should shows us if we're seeing interrupts.
Nick
From: Andrius V <vezhlys@gmail.com>
To: Nick Hudson <nick.hudson@gmx.co.uk>
Cc: gnats-bugs@netbsd.org, port-i386-maintainer@netbsd.org,
netbsd-bugs@netbsd.org, gnats-admin@netbsd.org, gutteridge@netbsd.org
Subject: Re: port-i386/53894 ("ehci_sync_hc: timed out" with ACPI/SMP enabled
on eBox 3352DX3-AP)
Date: Thu, 24 Jan 2019 00:10:42 +0200
On Mon, Jan 21, 2019 at 8:40 AM Nick Hudson <nick.hudson@gmx.co.uk> wrote:
>
> On 20/01/2019 23:50, gutteridge@NetBSD.org wrote:
> > Synopsis: "ehci_sync_hc: timed out" with ACPI/SMP enabled on eBox 3352DX3-AP
>
> This sounds more like an MP/interrupt problem for your system than usb.
>
> ehcidebug=1 should shows us if we're seeing interrupts.
>
Hi,
Do you mean EHCI_DEBUG option in kernel configuration or something
else? Currently I was testing kernel which included EHCI_DEBUG option
line and I set hw.ehci.debug value as well, however I didn't see any
interrupt messages during boot. There were no vmstat output changes
either (I tried delaying it for few minutes, but it didn't give more
than few additional messages of "usb_event_thread#0@0: sc 0xc3d5d1b8
woke up", "usb_discover#1@0: called!" compared to original one I've
attached). After removing USB Ethernet device and leaving USB keyboard
only, "ehci_sync_hc: timed out" messages disappeared from boot log,
but USB keyboard still didn't power up (no LED lights lit up),
appeared in dmesg or worked nevertheless. Finally, I tried to boot
without SMP but with ACPI enabled (boot -1), the problem stayed the
same. So I would assume it can be ACPI related issue, since disabling
ACPI and SMP makes USB devices work (though with occasional
reconnection cycles). I couldn't test an opposite situation (SMP
enabled/ACPI disabled), because I am getting IDE interrupt errors in
this case and system fails to boot.
Regards,
Andrius V
On Mon, Jan 21, 2019 at 8:40 AM Nick Hudson <nick.hudson@gmx.co.uk> wrote:
>
> On 20/01/2019 23:50, gutteridge@NetBSD.org wrote:
> > Synopsis: "ehci_sync_hc: timed out" with ACPI/SMP enabled on eBox 3352DX3-AP
>
> This sounds more like an MP/interrupt problem for your system than usb.
>
> ehcidebug=1 should shows us if we're seeing interrupts.
>
> Nick
>
Responsible-Changed-From-To: skrll->port-i386-maintainer
Responsible-Changed-By: skrll@NetBSD.org
Responsible-Changed-When: Wed, 30 Jan 2019 08:33:19 +0000
Responsible-Changed-Why:
not a usb bug. the problem is interrupts related.
From: Andrius V <vezhlys@gmail.com>
To: gnats-bugs@netbsd.org
Cc: port-i386-maintainer@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, Taylor R Campbell <riastradh@netbsd.org>
Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP enabled
on eBox 3352DX3-AP
Date: Sun, 14 Nov 2021 11:28:59 +0200
Placed system's acpidump to https://netbsd.org/~andvar/acpidump.txt on
request from Taylor.
On Sun, Jan 20, 2019 at 10:56 PM Andrius V <vezhlys@gmail.com> wrote:
>
> Hi,
>
> Thank you for the prompt reply. I can provide this output but it is a
> bit tricky since USB ports are the only available besides Ethernet and
> VGA (and integrated vte ethernet controller doesn't work properly as
> well unfortunately, so ssh is out of option). Does it matter when to
> do this output? I guess non ACPI/SMP boot won't provide the required
> data as well? Currently I did a cron job which outputs the vmstat -u
> usbhist result to file on boot. Please see an attached report for
> results. Theoretically, I can modify cronjob to post report a bit
> later if needed.
>
> Regards,
> Andrius V
>
> On Sun, Jan 20, 2019 at 3:10 AM David H. Gutteridge <david@gutteridge.ca> wrote:
> >
> > The following reply was made to PR port-i386/53894; it has been noted by GNATS.
> >
> > From: "David H. Gutteridge" <david@gutteridge.ca>
> > To: gnats-bugs@netbsd.org
> > Cc:
> > Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP
> > enabled on eBox 3352DX3-AP
> > Date: Sat, 19 Jan 2019 20:09:06 -0500
> >
> > It would probably be helpful to provide USB_DEBUG output. To do so,
> > please see http://www.netbsd.org/docs/kernel/#usb-debugging,
> > specifically the part following "USB debugging code has been changed in
> > NetBSD 7.99 and later."
> >
> > Dave
> >
> >
From: Andrius V <vezhlys@gmail.com>
To: port-i386-maintainer@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, gnats-bugs@netbsd.org
Cc:
Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP enabled
on eBox 3352DX3-AP
Date: Wed, 16 Mar 2022 10:25:14 +0200
Hi,
I did some investigation on this issue in recent days and here are my findings:
* I can confirm, that it is interrupts issue, more specifically level
type PCI interrupts are not working (thus, USB, network, SCI, probably
audio too (untested)) interrupts doesn't work (checked by intrctl and
vmstat).
* The interrupt establishing flow seems OK though, did some debugging,
which shows that the flow doesn't seem to be different from similar
system which has no issues and visually everything is "established".
* By advice I checked MADT overrides: system reports two overrides,
and indeed, it seems that IRQ 0 NetBSD do not override. Unfortunately,
I have no idea, where to do that in NetBSD code (FreeBSD seems to go
through all MADT table and do overrides, in NetBSD case only SCI
interrupt individually comes to override code). Would that fix the
issue, honestly, I am not certain, I did try silly hacks, like not
masking interrupts by default, which added "i82489_ipi_init: ESR
00000040" to dmesg, but none helped to make PCI interrupts work.
acpi0: [MADT] INTR OVERRIDE : BUS 0, IRQ 0, GSI 2, FLAGS 0x00
acpi0: [MADT] INTR OVERRIDE : BUS 0, IRQ 9, GSI 9, FLAGS 0x0D
* SoC does not support MSI.
* current kernel doesn't report "ehci_sync_hc: timed out" anymore
compared to my original dmesg report, just silently boots without any
uhidev devices attached, likely related to recent USB changes.
* FreeBSD 12.0 (the last release which boots out of the box on the
system), and recent Linux kernels has no problems with these
interrupts. OpenBSD have likely the same issue as NetBSD though, since
USB and network also doesn't work.
* Booting without ACPI/SMP enabled, makes interrupts work (it doesn't
go through apic in this case of course).
Unfortunately, I am out of ideas for now on next steps, but will keep
investigating once I have some, any help would be welcome. Thank you.
Below attaching snippet from dmesg from acpi verbose (acpi dump still
available here https://netbsd.org/~andvar/acpidump.txt):
acpi0: [MADT] LAPIC : CPU ID 1, LAPIC ID 0, FLAGS 0x01
acpi0: [MADT] LAPIC : CPU ID 2, LAPIC ID 1, FLAGS 0x01
acpi0: [MADT] I/O APIC : ID 2, GSI 0, ADDR 0xFEC00000
acpi0: [MADT] INTR OVERRIDE : BUS 0, IRQ 0, GSI 2, FLAGS 0x00
acpi0: [MADT] INTR OVERRIDE : BUS 0, IRQ 9, GSI 9, FLAGS 0x0D
acpi0: [FADT] FACS : 0x000000007BFDE000
acpi0: [FADT] DSDT : 0x000000007BFD05D0
acpi0: [FADT] INT_MODEL : 0x0000000000000001
acpi0: [FADT] PM_PROFILE : 0x0000000000000002
acpi0: [FADT] SCI_INT : 0x0000000000000009
acpi0: [FADT] SMI_CMD : 0x00000000000000B2
acpi0: [FADT] ACPI_ENABLE : 0x00000000000000E1
acpi0: [FADT] ACPI_DISABLE : 0x000000000000001E
acpi0: [FADT] S4BIOS_REQ : 0x0000000000000000
acpi0: [FADT] PSTATE_CNT : 0x00000000000000E2
acpi0: [FADT] PM1a_EVT_BLK : 0x0000000000000800
acpi0: [FADT] PM1b_EVT_BLK : 0x0000000000000000
acpi0: [FADT] PM1a_CNT_BLK : 0x0000000000000804
acpi0: [FADT] PM1b_CNT_BLK : 0x0000000000000000
acpi0: [FADT] PM2_CNT_BLK : 0x0000000000000000
acpi0: [FADT] PM_TMR_BLK : 0x0000000000000808
acpi0: [FADT] GPE0_BLK : 0x0000000000000820
acpi0: [FADT] GPE1_BLK : 0x0000000000000000
acpi0: [FADT] PM1_EVT_LEN : 0x0000000000000004
acpi0: [FADT] PM1_CNT_LEN : 0x0000000000000002
acpi0: [FADT] PM2_CNT_LEN : 0x0000000000000000
acpi0: [FADT] PM_TMR_LEN : 0x0000000000000004
acpi0: [FADT] GPE0_BLK_LEN : 0x0000000000000008
acpi0: [FADT] GPE1_BLK_LEN : 0x0000000000000000
acpi0: [FADT] GPE1_BASE : 0x0000000000000000
acpi0: [FADT] CST_CNT : 0x00000000000000E3
acpi0: [FADT] P_LVL2_LAT : 0x0000000000000065
acpi0: [FADT] P_LVL3_LAT : 0x00000000000003E9
acpi0: [FADT] FLUSH_SIZE : 0x0000000000000400
acpi0: [FADT] FLUSH_STRIDE : 0x0000000000000010
acpi0: [FADT] DUTY_OFFSET : 0x0000000000000001
acpi0: [FADT] DUTY_WIDTH : 0x0000000000000003
acpi0: [FADT] DAY_ALRM : 0x000000000000000D
acpi0: [FADT] MON_ALRM : 0x0000000000000000
acpi0: [FADT] CENTURY : 0x0000000000000032
acpi0: [FADT] IAPC_BOOT_ARCH : 0x0000000000000003
acpi0: [FADT] RESET_VALUE : 0x0000000000000006
acpi0: [FADT] ARM_BOOT_ARCH : 0x0000000000000000
acpi0: [FADT] X_FACS : 0x000000007BFDE000
acpi0: [FADT] X_DSDT : 0x000000007BFD05D0
acpi0: [FADT] RESET_REG : 0x0000000000000CF9, SPACE ID 1, BIT
WIDTH 8, BIT OFFSET 0, ACCESS WIDTH 0
acpi0: [FADT] X_PM1a_EVT_BLK : 0x0000000000000800, SPACE ID 1, BIT
WIDTH 32, BIT OFFSET 0, ACCESS WIDTH 0
acpi0: [FADT] X_PM1b_EVT_BLK : 0x0000000000000000, SPACE ID 1, BIT
WIDTH 32, BIT OFFSET 0, ACCESS WIDTH 0
acpi0: [FADT] X_PM1a_CNT_BLK : 0x0000000000000804, SPACE ID 1, BIT
WIDTH 16, BIT OFFSET 0, ACCESS WIDTH 0
acpi0: [FADT] X_PM1b_CNT_BLK : 0x0000000000000000, SPACE ID 0, BIT
WIDTH 0, BIT OFFSET 0, ACCESS WIDTH 0
acpi0: [FADT] X_PM2_CNT_BLK : 0x0000000000000000, SPACE ID 0, BIT
WIDTH 0, BIT OFFSET 0, ACCESS WIDTH 0
acpi0: [FADT] X_PM_TMR_BLK : 0x0000000000000808, SPACE ID 1, BIT
WIDTH 32, BIT OFFSET 0, ACCESS WIDTH 0
acpi0: [FADT] X_GPE0_BLK : 0x0000000000000820, SPACE ID 1, BIT
WIDTH 64, BIT OFFSET 0, ACCESS WIDTH 0
acpi0: [FADT] X_GPE1_BLK : 0x0000000000000000, SPACE ID 1, BIT
WIDTH 32, BIT OFFSET 0, ACCESS WIDTH 0
acpi0: [FADT] SLEEP_CTRL_REG : 0x0000000000000000, SPACE ID 0, BIT
WIDTH 0, BIT OFFSET 0, ACCESS WIDTH 0
acpi0: [FADT] SLEEP_STAT_REG : 0x0000000000000000, SPACE ID 0, BIT
WIDTH 0, BIT OFFSET 0, ACCESS WIDTH 0
acpi0: [FADT] WBINVD : 0x0000000000000001
acpi0: [FADT] WBINVD_FLUSH : 0x0000000000000000
acpi0: [FADT] PROC_C1 : 0x0000000000000000
acpi0: [FADT] P_LVL2_UP : 0x0000000000000000
acpi0: [FADT] PWR_BUTTON : 0x0000000000000000
acpi0: [FADT] SLP_BUTTON : 0x0000000000000020
acpi0: [FADT] FIX_RTC : 0x0000000000000000
acpi0: [FADT] RTC_S4 : 0x0000000000000080
acpi0: [FADT] TMR_VAL_EXT : 0x0000000000000000
acpi0: [FADT] DCK_CAP : 0x0000000000000000
acpi0: [FADT] RESET_REG_SUP : 0x0000000000000000
acpi0: [FADT] SEALED_CASE : 0x0000000000000000
acpi0: [FADT] HEADLESS : 0x0000000000000000
acpi0: [FADT] CPU_SW_SLP : 0x0000000000000000
acpi0: [FADT] PCI_EXP_WAK : 0x0000000000000000
acpi0: [FADT] PLATFORM_CLK : 0x0000000000000000
acpi0: [FADT] S4_RTC_STS : 0x0000000000000000
acpi0: [FADT] REMOTE_POWER : 0x0000000000000000
acpi0: [FADT] APIC_CLUSTER : 0x0000000000000000
acpi0: [FADT] APIC_PHYSICAL : 0x0000000000000000
acpi0: [FADT] HW_REDUCED : 0x0000000000000000
acpi0: [FADT] LOW_POWER_S0 : 0x0000000000000000
acpi0: [\ ] HID - UID - ADR -
acpi0: [P001] HID - UID - ADR -
acpi0: [P002] HID - UID - ADR -
acpi0: [_SB ] HID - UID - ADR -
acpi0: [PCI0] HID PNP0A03 UID 0 ADR 0x0000000000000000
acpi0: [VGA ] HID - UID - ADR 0x00000000000D0000
acpi0: [CRTD] HID - UID - ADR 0x0000000000000100
acpi0: [P0P1] HID - UID - ADR 0x0000000000010000
acpi0: [PCI1] HID - UID - ADR 0x0000000000000000
acpi0: [P0P2] HID - UID - ADR 0x0000000000020000
acpi0: [PCI2] HID - UID - ADR 0x0000000000000000
acpi0: [P0P3] HID - UID - ADR 0x0000000000030000
acpi0: [PCI3] HID - UID - ADR 0x0000000000000000
acpi0: [SBRG] HID - UID - ADR 0x0000000000070000
acpi0: [PIC ] HID PNP0000 UID - ADR -
acpi0: [DMAD] HID PNP0200 UID - ADR -
acpi0: [TMR ] HID PNP0100 UID - ADR -
acpi0: [RTC0] HID PNP0B00 UID - ADR -
acpi0: [PS2K] HID PNP0303 UID - ADR -
acpi0: [PS2M] HID PNP0F03 UID - ADR -
acpi0: [SPKR] HID PNP0800 UID - ADR -
acpi0: [COPR] HID PNP0C04 UID - ADR -
acpi0: [RMSC] HID PNP0C02 UID 16 ADR -
acpi0: [UARA] HID PNP0501 UID COM1 ADR -
acpi0: [UARB] HID PNP0501 UID COM2 ADR -
acpi0: [UARC] HID PNP0501 UID COM3 ADR -
acpi0: [UARD] HID PNP0501 UID COM4 ADR -
acpi0: [UARE] HID PNP0501 UID COM5 ADR -
acpi0: [UARF] HID PNP0501 UID COM6 ADR -
acpi0: [UARG] HID PNP0501 UID COM7 ADR -
acpi0: [UARH] HID PNP0501 UID COM8 ADR -
acpi0: [UARI] HID PNP0501 UID COM9 ADR -
acpi0: [LPTA] HID PNP0400 UID - ADR -
acpi0: [OMSC] HID PNP0C02 UID 0 ADR -
acpi0: [EC0 ] HID PNP0C09 UID - ADR -
acpi0: [SBF1] HID - UID - ADR 0x0000000000070001
acpi0: [LAN0] HID - UID - ADR 0x0000000000080000
acpi0: [USB1] HID - UID - ADR 0x00000000000A0000
acpi0: [EUSB] HID - UID - ADR 0x00000000000A0001
acpi0: [IDE0] HID - UID - ADR 0x00000000000C0000
acpi0: [CHN0] HID - UID - ADR 0x0000000000000000
acpi0: [DRV0] HID - UID - ADR 0x0000000000000000
acpi0: [DRV1] HID - UID - ADR 0x0000000000000001
acpi0: [CHN1] HID - UID - ADR 0x0000000000000001
acpi0: [DRV0] HID - UID - ADR 0x0000000000000000
acpi0: [DRV1] HID - UID - ADR 0x0000000000000001
acpi0: [HDAC] HID - UID - ADR 0x00000000000E0000
acpi0: [AC0 ] HID ACPI0003 UID - ADR -
acpi0: [BAT0] HID PNP0C0A UID 0 ADR -
acpi0: [BAT1] HID PNP0C0A UID 1 ADR -
acpi0: [RMEM] HID PNP0C01 UID 1 ADR -
acpi0: [LNKA] HID PNP0C0F UID 1 ADR -
acpi0: [LNKB] HID PNP0C0F UID 2 ADR -
acpi0: [LNKC] HID PNP0C0F UID 3 ADR -
acpi0: [LNKD] HID PNP0C0F UID 4 ADR -
acpi0: [LNKE] HID PNP0C0F UID 5 ADR -
acpi0: [LNKF] HID PNP0C0F UID 6 ADR -
acpi0: [LNKG] HID PNP0C0F UID 7 ADR -
acpi0: [LNKH] HID PNP0C0F UID 8 ADR -
acpi0: [PWRB] HID PNP0C0C UID 170 ADR -
acpi0: [SLPB] HID PNP0C0E UID - ADR -
acpi0: [LID ] HID PNP0C0D UID - ADR -
acpi0: [_TZ ] HID - UID - ADR -
acpi0: [THRM] HID - UID - ADR -
\ [06] []
P001 [12] []
P002 [12] []
_SB [06] []
PCI0 [06] [] (PCI) @ 0x00:0x00:0x00:0x00 [R] [B] -> 0x00:0x00
VGA [06] [] (PCI) @ 0x00:0x00:0x0D:0x00
CRTD [06] []
P0P1 [06] [] (PCI) @ 0x00:0x00:0x01:0x00 [B] -> 0x00:0x01
PCI1 [06] [] (PCI) @ 0x00:0x01:0x00:0x00
P0P2 [06] [] (PCI) @ 0x00:0x00:0x02:0x00 [B] -> 0x00:0x02
PCI2 [06] [] (PCI) @ 0x00:0x02:0x00:0x00
P0P3 [06] [] (PCI) @ 0x00:0x00:0x03:0x00
PCI3 [06] []
SBRG [06] [] (PCI) @ 0x00:0x00:0x07:0x00
PIC [06] []
DMAD [06] []
TMR [06] [] <attimer1>
RTC0 [06] []
PS2K [06] []
PS2M [06] []
SPKR [06] [] <pcppi1>
COPR [06] []
RMSC [06] []
UARA [06] []
UARB [06] []
UARC [06] []
UARD [06] []
UARE [06] []
UARF [06] []
UARG [06] []
UARH [06] []
UARI [06] []
LPTA [06] []
OMSC [06] []
EC0 [06] [] <acpiec0>
SBF1 [06] [] (PCI) @ 0x00:0x00:0x07:0x01
LAN0 [06] [] (PCI) @ 0x00:0x00:0x08:0x00
USB1 [06] [] (PCI) @ 0x00:0x00:0x0A:0x00
EUSB [06] [] (PCI) @ 0x00:0x00:0x0A:0x01
IDE0 [06] [] (PCI) @ 0x00:0x00:0x0C:0x00
CHN0 [06] []
DRV0 [06] []
DRV1 [06] []
CHN1 [06] []
DRV0 [06] []
DRV1 [06] []
HDAC [06] [] (PCI) @ 0x00:0x00:0x0E:0x00
AC0 [06] [] <acpiacad0>
BAT0 [06] []
BAT1 [06] []
RMEM [06] []
LNKA [06] []
LNKB [06] []
LNKC [06] []
LNKD [06] []
LNKE [06] []
LNKF [06] []
LNKG [06] []
LNKH [06] []
PWRB [06] [] <acpibut0>
SLPB [06] [] <acpibut1>
LID [06] [] <acpilid0>
_TZ [06] []
THRM [13] [] <acpitz0>
Regards
Andrius V
On Sun, Nov 14, 2021 at 11:35 AM Andrius V <vezhlys@gmail.com> wrote:
>
> The following reply was made to PR port-i386/53894; it has been noted by GNATS.
>
> From: Andrius V <vezhlys@gmail.com>
> To: gnats-bugs@netbsd.org
> Cc: port-i386-maintainer@netbsd.org, gnats-admin@netbsd.org,
> netbsd-bugs@netbsd.org, Taylor R Campbell <riastradh@netbsd.org>
> Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP enabled
> on eBox 3352DX3-AP
> Date: Sun, 14 Nov 2021 11:28:59 +0200
>
> Placed system's acpidump to https://netbsd.org/~andvar/acpidump.txt on
> request from Taylor.
>
> On Sun, Jan 20, 2019 at 10:56 PM Andrius V <vezhlys@gmail.com> wrote:
> >
> > Hi,
> >
> > Thank you for the prompt reply. I can provide this output but it is a
> > bit tricky since USB ports are the only available besides Ethernet and
> > VGA (and integrated vte ethernet controller doesn't work properly as
> > well unfortunately, so ssh is out of option). Does it matter when to
> > do this output? I guess non ACPI/SMP boot won't provide the required
> > data as well? Currently I did a cron job which outputs the vmstat -u
> > usbhist result to file on boot. Please see an attached report for
> > results. Theoretically, I can modify cronjob to post report a bit
> > later if needed.
> >
> > Regards,
> > Andrius V
> >
> > On Sun, Jan 20, 2019 at 3:10 AM David H. Gutteridge <david@gutteridge.ca> wrote:
> > >
> > > The following reply was made to PR port-i386/53894; it has been noted by GNATS.
> > >
> > > From: "David H. Gutteridge" <david@gutteridge.ca>
> > > To: gnats-bugs@netbsd.org
> > > Cc:
> > > Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP
> > > enabled on eBox 3352DX3-AP
> > > Date: Sat, 19 Jan 2019 20:09:06 -0500
> > >
> > > It would probably be helpful to provide USB_DEBUG output. To do so,
> > > please see http://www.netbsd.org/docs/kernel/#usb-debugging,
> > > specifically the part following "USB debugging code has been changed in
> > > NetBSD 7.99 and later."
> > >
> > > Dave
> > >
> > >
>
From: Andrius V <vezhlys@gmail.com>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP enabled
on eBox 3352DX3-AP
Date: Tue, 5 Mar 2024 09:57:31 +0200
On Wed, Mar 16, 2022 at 10:25=E2=80=AFAM Andrius V <vezhlys@gmail.com> wrot=
e:
>
> Hi,
>
> I did some investigation on this issue in recent days and here are my fin=
dings:
> * I can confirm, that it is interrupts issue, more specifically level
> type PCI interrupts are not working (thus, USB, network, SCI, probably
> audio too (untested)) interrupts doesn't work (checked by intrctl and
> vmstat).
> * The interrupt establishing flow seems OK though, did some debugging,
> which shows that the flow doesn't seem to be different from similar
> system which has no issues and visually everything is "established".
> * By advice I checked MADT overrides: system reports two overrides,
> and indeed, it seems that IRQ 0 NetBSD do not override. Unfortunately,
> I have no idea, where to do that in NetBSD code (FreeBSD seems to go
> through all MADT table and do overrides, in NetBSD case only SCI
> interrupt individually comes to override code). Would that fix the
> issue, honestly, I am not certain, I did try silly hacks, like not
> masking interrupts by default, which added "i82489_ipi_init: ESR
> 00000040" to dmesg, but none helped to make PCI interrupts work.
> acpi0: [MADT] INTR OVERRIDE : BUS 0, IRQ 0, GSI 2, FLAGS 0x00
> acpi0: [MADT] INTR OVERRIDE : BUS 0, IRQ 9, GSI 9, FLAGS 0x0D
> * SoC does not support MSI.
> * current kernel doesn't report "ehci_sync_hc: timed out" anymore
> compared to my original dmesg report, just silently boots without any
> uhidev devices attached, likely related to recent USB changes.
> * FreeBSD 12.0 (the last release which boots out of the box on the
> system), and recent Linux kernels has no problems with these
> interrupts. OpenBSD have likely the same issue as NetBSD though, since
> USB and network also doesn't work.
> * Booting without ACPI/SMP enabled, makes interrupts work (it doesn't
> go through apic in this case of course).
>
> Unfortunately, I am out of ideas for now on next steps, but will keep
> investigating once I have some, any help would be welcome. Thank you.
>
Hi,
I had few additional attempts to investigate the issue, unfortunately
without any results which would lead to understanding the issue. I
would be willing to give up my eBox 3360 with RS-485 serial ports if
anybody would be interested to work on this.
From the latest attempts I was still focussing on interrupt
initialization, which increasingly feels like a wrong place to look
at. Likely level type interrupts themselves should be investigated,
but I don't know how to do that effectively.
I did debug the flow from lapic.c, ioapic.c, pci_machdep.c, intrc.c
and recently mpacpi.c, compared to my VIA system which is somewhat
similar. Compared to FreeBSD code, which has differences, but I can't
pinpoint anything of importance, extINT being the most visible
difference, but it doesn't seem to affect level type interrupts in any
way (14 now can boot as well). In general, from side it looks like all
those attempts show "flawless" initialization, without errors or going
to erroneous path. System boots without issues as well, however the
end result is that USB, Ethernet doesn't work since no level type
interrupts are acknowledged from them (or at least it looks like so).
ISA interrupts though work, thus serial can be used and system itself
seems to be stable otherwise.
System BIOS also has "RDC engineer mode" by clicking F4 while in BIOS.
It provides few additional options like disabling APIC bus, which
allows to boot system with ACPI enabled and interrupts working,
however they are wired through PIC IRQs and second core needs to be
disabled to boot.
In recent attempts I enabled MPDEBUG which gives more log messages
about configuring the interrupts, also shows that mpacpi takes care of
both MATD overrides, all logs are expected and matches working
systems:
....
mpacpi: found root PCI bus 0
mpacpi: found subordinate bus 1
mpacpi: found subordinate bus 2
mpacpi: 3 PCI busses
mpacpi: ISA interrupt override 0 -> 2 (0/0)
mpacpi: ISA interrupt override 9 -> 9 (1/3)
mpacpi: configuring PCI bus 0 int routing
pciroute: dev 13 INTA on globint 22
pciroute: done adding entry
pciroute: dev 1 INTA on globint 16
pciroute: done adding entry
pciroute: dev 1 INTB on globint 17
pciroute: done adding entry
pciroute: dev 1 INTC on globint 18
pciroute: done adding entry
pciroute: dev 1 INTD on globint 19
pciroute: done adding entry
pciroute: dev 2 INTA on globint 17
pciroute: done adding entry
pciroute: dev 2 INTB on globint 18
pciroute: done adding entry
pciroute: dev 2 INTC on globint 19
pciroute: done adding entry
pciroute: dev 2 INTD on globint 16
pciroute: done adding entry
pciroute: dev 3 INTA on globint 18
pciroute: done adding entry
pciroute: dev 3 INTB on globint 19
pciroute: done adding entry
pciroute: dev 3 INTC on globint 16
pciroute: done adding entry
pciroute: dev 3 INTD on globint 17
pciroute: done adding entry
pciroute: dev 8 INTA on globint 19
pciroute: done adding entry
pciroute: dev 10 INTA on globint 18
pciroute: done adding entry
pciroute: dev 10 INTB on globint 21
pciroute: done adding entry
pciroute: dev 12 INTA on globint 23
pciroute: done adding entry
pciroute: dev 14 INTA on globint 18
pciroute: done adding entry
pciroute: dev 15 INTA on globint 21
pciroute: done adding entry
pciroute: done
mpacpi: configuring PCI bus 1 int routing
pciroute: dev 0 INTA on globint 16
pciroute: done adding entry
pciroute: dev 0 INTB on globint 17
pciroute: done adding entry
pciroute: dev 0 INTC on globint 18
pciroute: done adding entry
pciroute: dev 0 INTD on globint 19
pciroute: done adding entry
pciroute: done
mpacpi: configuring PCI bus 2 int routing
pciroute: dev 0 INTA on globint 17
pciroute: done adding entry
pciroute: dev 0 INTB on globint 18
pciroute: done adding entry
pciroute: dev 0 INTC on globint 19
pciroute: done adding entry
pciroute: dev 0 INTD on globint 16
pciroute: done adding entry
pciroute: done
ioapic0: pin 2 attached to isa0 irq 0 (type 0<type=3D0> flags 0<pol=3D0,tri=
g=3D0>)
ioapic0: pin 1 attached to isa0 irq 1 (type 0<type=3D0> flags 0<pol=3D0,tri=
g=3D0>)
ioapic0: pin 3 attached to isa0 irq 3 (type 0<type=3D0> flags 0<pol=3D0,tri=
g=3D0>)
ioapic0: pin 4 attached to isa0 irq 4 (type 0<type=3D0> flags 0<pol=3D0,tri=
g=3D0>)
ioapic0: pin 5 attached to isa0 irq 5 (type 0<type=3D0> flags 0<pol=3D0,tri=
g=3D0>)
ioapic0: pin 6 attached to isa0 irq 6 (type 0<type=3D0> flags 0<pol=3D0,tri=
g=3D0>)
ioapic0: pin 7 attached to isa0 irq 7 (type 0<type=3D0> flags 0<pol=3D0,tri=
g=3D0>)
ioapic0: pin 8 attached to isa0 irq 8 (type 0<type=3D0> flags 0<pol=3D0,tri=
g=3D0>)
ioapic0: pin 9 attached to isa0 irq 9 (type 0<type=3D0> flags
0xd<pol=3D0x1=3DAct Hi,trig=3D0x3=3DLevel>)
ioapic0: pin 10 attached to isa0 irq 10 (type 0<type=3D0> flags 0<pol=3D0,t=
rig=3D0>)
ioapic0: pin 11 attached to isa0 irq 11 (type 0<type=3D0> flags 0<pol=3D0,t=
rig=3D0>)
ioapic0: pin 12 attached to isa0 irq 12 (type 0<type=3D0> flags 0<pol=3D0,t=
rig=3D0>)
ioapic0: pin 13 attached to isa0 irq 13 (type 0<type=3D0> flags 0<pol=3D0,t=
rig=3D0>)
ioapic0: pin 14 attached to isa0 irq 14 (type 0<type=3D0> flags 0<pol=3D0,t=
rig=3D0>)
ioapic0: pin 15 attached to isa0 irq 15 (type 0<type=3D0> flags 0<pol=3D0,t=
rig=3D0>)
ioapic0: pin 22 attached to pci0 device 13 INT_A (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 16 attached to pci0 device 1 INT_A (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 17 attached to pci0 device 1 INT_B (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 18 attached to pci0 device 1 INT_C (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 19 attached to pci0 device 1 INT_D (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 17 attached to pci0 device 2 INT_A (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 18 attached to pci0 device 2 INT_B (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 19 attached to pci0 device 2 INT_C (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 16 attached to pci0 device 2 INT_D (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 18 attached to pci0 device 3 INT_A (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 19 attached to pci0 device 3 INT_B (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 16 attached to pci0 device 3 INT_C (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 17 attached to pci0 device 3 INT_D (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 19 attached to pci0 device 8 INT_A (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 18 attached to pci0 device 10 INT_A (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 21 attached to pci0 device 10 INT_B (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 23 attached to pci0 device 12 INT_A (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 18 attached to pci0 device 14 INT_A (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 21 attached to pci0 device 15 INT_A (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 16 attached to pci1 device 0 INT_A (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 17 attached to pci1 device 0 INT_B (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 18 attached to pci1 device 0 INT_C (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 19 attached to pci1 device 0 INT_D (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 17 attached to pci2 device 0 INT_A (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 18 attached to pci2 device 0 INT_B (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 19 attached to pci2 device 0 INT_C (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
ioapic0: pin 16 attached to pci2 device 0 INT_D (type 0<type=3D0> flags
0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
...
Regards,
Andrius V
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.47 2022/09/11 19:34:41 kim Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2024
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home