NetBSD Problem Report #53894

From www@NetBSD.org  Sat Jan 19 22:29:44 2019
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 0B2227A154
	for <gnats-bugs@gnats.NetBSD.org>; Sat, 19 Jan 2019 22:29:44 +0000 (UTC)
Message-Id: <20190119222942.0377F7A214@mollari.NetBSD.org>
Date: Sat, 19 Jan 2019 22:29:41 +0000 (UTC)
From: vezhlys@gmail.com
Reply-To: vezhlys@gmail.com
To: gnats-bugs@NetBSD.org
Subject: "ehci_sync_hc: timed out" with ACPI/SMP enabled on eBox 3352DX3-AP
X-Send-Pr-Version: www-1.0

>Number:         53894
>Category:       port-i386
>Synopsis:       "ehci_sync_hc: timed out" with ACPI/SMP enabled on eBox 3352DX3-AP
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    port-i386-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Jan 19 22:30:00 +0000 2019
>Last-Modified:  Tue Mar 05 08:00:02 +0000 2024
>Originator:     Andrius V
>Release:        8.99.30
>Organization:
>Environment:
NetBSD vertexpc 8.99.30 NetBSD 8.99.30 (GENERIC) #0: Tue Jan 15 14:23:56 UTC 2019  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/i386/compile/GENERIC i386
>Description:
Recent testing showed that enabling IDE legacy mode in BIOS allows to boot into NetBSD on DM&P eBox 3352DX3-AP VortexDX3 based system with ACPI/SMP enabled (without legacy mode, IDE controller timeouts). However, there is one major caveat that USB devices fail to initialize in this case with repetitive "ehci_sync_hc: timed out" message in boot log, which eventually leads to next lines "uhub1: autoconfiguration error: device problem, disabling port 1", "uhub1: port 2, device disappeared after reset". However, USB devices initialize and work properly if system is booted without ACPI/SMP support (boot -12). Please see dmesg messages for comparison below. Only occasional disconnect/reconnect may happen while typing on keyboard in this case.

dmesg with ACPI/SMP:
NetBSD 8.99.30 (GENERIC) #0: Tue Jan 15 14:23:56 UTC 2019
	mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/i386/compile/GENERIC
total memory = 1983 MB
avail memory = 1929 MB
timecounter: Timecounters tick every 10.000 msec
Kernelized RAIDframe activated
running cgd selftest aes-xts-256 aes-xts-512 done
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
efi: missing or invalid systbl
SMBIOS rev. 2.7 @ 0xfaf20 (43 entries)
RDC Semiconductor Co., Ltd. EMKORE                 (1.0                   )
mainbus0 (root)
ACPI: RSDP 0x00000000000F7920 000024 (v03 ACPIAM)
ACPI: XSDT 0x000000007BFD0100 00004C (v01 R D C  QA09     20151227 MSFT 00000097)
ACPI: FACP 0x000000007BFD0290 0000F4 (v04 R D C  OEMFACP  20151227 MSFT 00000097)
ACPI: DSDT 0x000000007BFD05D0 005329 (v02 EMKORE EMKORE00 00000000 INTL 20060707)
ACPI: FACS 0x000000007BFDE000 000040
ACPI: APIC 0x000000007BFD0390 00005C (v02 R D C  APIC1706 20151227 MSFT 00000097)
ACPI: MSDM 0x000000007BFD03F0 000055 (v03 R D C  QA09     20151227 MSFT 00000097)
ACPI: SLIC 0x000000007BFD0450 000176 (v01 R D C  QA09     20151227 MSFT 00000097)
ACPI: OEMB 0x000000007BFDE040 000082 (v01 R D C  AMI_OEM  20151227 MSFT 00000097)
ACPI: 1 ACPI AML tables successfully acquired and loaded
ioapic0 at mainbus0 apid 2: pa 0xfec00000, version 0x20, 24 pins
cpu0 at mainbus0 apid 0
cpu0: 2 page colors
cpu0: calibrating local timer
cpu0: apic clock running at 666 MHz
cpu0: Vortex86DX3, id 0x611
cpu0: package 0, core 0, smt 0
cpu1 at mainbus0 apid 1
cpu1: 2 page colors
cpu1: Vortex86DX3, id 0x611
cpu1: package 1, core 0, smt 0
acpi0 at mainbus0: Intel ACPICA 20181213
acpi0: X/RSDT: OemId <R D C , QA09   ,20151227>, AslId <MSFT,00000097>
acpi0: SCI interrupting at int 9
acpi0: fixed power button present
timecounter: Timecounter "ACPI-Safe" frequency 3579545 Hz quality 900
acpi0: ACPI-Safe 24-bit timer
acpiec0 at acpi0 (EC0, PNP0C09): io 0x62,0x66
attimer1 at acpi0 (TMR, PNP0100): io 0x40-0x43 irq 0
pcppi1 at acpi0 (SPKR, PNP0800): io 0x61
midi0 at pcppi1: PC speaker
sysbeep0 at pcppi1
COPR (PNP0C04) at acpi0 not configured
RMSC (PNP0C02) at acpi0 not configured
OMSC (PNP0C02) at acpi0 not configured
acpiacad0 at acpi0 (AC0, ACPI0003): ACPI AC Adapter
RMEM (PNP0C01) at acpi0 not configured
acpibut0 at acpi0 (PWRB, PNP0C0C-170): ACPI Power Button
acpibut1 at acpi0 (SLPB, PNP0C0E): ACPI Sleep Button
acpilid0 at acpi0 (LID, PNP0C0D): ACPI Lid Switch
acpitz0 at acpi0 (THRM): cpu0
acpitz0: polling interval 30.0 seconds
acpitz0: levels: critical 92.0 C, passive 78.0 C, passive cooling
apm0 at acpi0: Power Management spec V1.2
attimer1: attached to pcppi1
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0: vendor 17f3 product 6023 (rev. 0x02)
ppb0 at pci0 dev 1 function 0: vendor 17f3 product 1031 (rev. 0x01)
ppb0: PCI Express capability version 1 <PCI/PCI-X to PCI-E Bridge> x1 @ 2.5GT/s
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
ppb1 at pci0 dev 2 function 0: vendor 17f3 product 1031 (rev. 0x01)
ppb1: PCI Express capability version 1 <PCI/PCI-X to PCI-E Bridge> x1 @ 2.5GT/s
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
pcib0 at pci0 dev 7 function 0: vendor 17f3 product 6035 (rev. 0x01)
pcib1 at pci0 dev 7 function 1: vendor 17f3 product 6035 (rev. 0x01)
vte0 at pci0 dev 8 function 0: vendor 17f3 product 6040 (rev. 0x00)
vte0: Ethernet address xx:xx:xx:xx:xx:xx
vte0: interrupting at ioapic0 pin 19
ukphy0 at vte0 phy 1: OUI 0xfcff2f, model 0x0005, rev. 0
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
ohci0 at pci0 dev 10 function 0: vendor 17f3 product 6060 (rev. 0x14)
ohci0: interrupting at ioapic0 pin 18
ohci0: OHCI version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
ehci0 at pci0 dev 10 function 1: vendor 17f3 product 6061 (rev. 0x08)
ehci0: interrupting at ioapic0 pin 21
ehci0: EHCI version 1.0
ehci0: 1 companion controller, 4 ports: ohci0
usb1 at ehci0: USB revision 2.0
rdcide0 at pci0 dev 12 function 0: RDC R1012 IDE controller (rev. 0x02)
rdcide0: bus-master DMA support present
rdcide0: primary channel configured to compatibility mode
rdcide0: primary channel interrupting at ioapic0 pin 14
atabus0 at rdcide0 channel 0
rdcide0: secondary channel configured to compatibility mode
rdcide0: secondary channel ignored (disabled)
vga0 at pci0 dev 13 function 0: vendor 17f3 product 2015 (rev. 0x00)
wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
drm at vga0 not configured
hdaudio0 at pci0 dev 14 function 0: HD Audio Controller
hdaudio0: interrupting at ioapic0 pin 18
hdafg0 at hdaudio0 vendor 0x10EC product 0x0262 nid 0x01: vendor 10ec product 0262
hdafg0: DAC00 2ch: Speaker [Jack]
nid=14 [pin: Speaker (Green Jack)]
nid=0C [audio mixer] [source: dac, record]
nid=02 [audio output] [source: dac]
nid=0B [audio mixer] [source: record]
hdafg0: DIG01 2ch: Digital Out [Jack]
nid=1E [pin: Digital Out (Black Jack)]
nid=06 [audio output] [source: dac]
hdafg0: ADC02 2ch: Mic In [Jack]
nid=19 [pin: Mic In (Pink Jack)]
hdafg0: ADC03 2ch: Line In [Jack]
nid=1A [pin: Line In (Blue Jack)]
hdafg0: 2ch/2ch 44100Hz 48000Hz 96000Hz 192000Hz PCM16 PCM20 PCM24 AC3
audio0 at hdafg0: full duplex, playback, capture, mmap, independent
hdafg0: Virtual format configured - Format SLINEAR, precision 16, channels 2, frequency 48000
hdafg0: Latency: 128 milliseconds
isa0 at pcib0
pckbc0 at isa0 port 0x60-0x64
pckbdprobe: reset error 5
pmsprobe: reset error 5
isa at pcib1 not configured
acpicpu0 at cpu0: ACPI CPU
acpicpu0: HALT not supported?
acpicpu0: C1: HLT, lat   0 us, pow     0 mW
acpicpu0: T0: I/O, lat   1 us, pow     0 mW, 100 %
acpicpu0: T1: I/O, lat   1 us, pow     0 mW,  88 %
acpicpu0: T2: I/O, lat   1 us, pow     0 mW,  76 %
acpicpu0: T3: I/O, lat   1 us, pow     0 mW,  64 %
acpicpu0: T4: I/O, lat   1 us, pow     0 mW,  52 %
acpicpu0: T5: I/O, lat   1 us, pow     0 mW,  40 %
acpicpu0: T6: I/O, lat   1 us, pow     0 mW,  28 %
acpicpu0: T7: I/O, lat   1 us, pow     0 mW,  16 %
acpicpu0: id 1, lapic id 0, cap 0x0000, flags 0x00100c25
acpicpu1 at cpu1: ACPI CPU
acpicpu1: HALT not supported?
acpicpu1: id 2, lapic id 1, cap 0x0000, flags 0x00000c21
Initializing SSP: ac3b5c83 62fd8be 75262144 3cdb1f14 80cf06f8 cb42734a 48c375bf 1af02484 
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
cpu1: TSC skew=3 drift=-5
cpu1: running
TSC not known invariant on this CPU
acpiacad0: AC adapter online.
acpiacad0: AC adapter connected
acpicpu1: ACPI CPUs started
uhub0 at usb0: NetBSD (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub0: 4 ports with 4 removable, self powered
uhub1 at usb1: NetBSD (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub1: 4 ports with 4 removable, self powered
IPsec: Initialized Security Association Processing.
wd0 at atabus0 drive 0
wd0: <SS16G D0 RDC SD-IDE HOST CONTROLLER>
wd0: drive supports 1-sector PIO transfers, LBA addressing
wd0: 15193 MB, 30869 cyl, 16 head, 63 sec, 512 bytes/sect x 31116288 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
wd0(rdcide0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100) (using DMA)
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
ehci_sync_hc: timed out
uhub1: autoconfiguration error: device problem, disabling port 1
ehci0: handing over low speed device on port 2 to ohci0
uhub1: port 2, device disappeared after reset
ehci0: handing over full speed device on port 3 to ohci0
uhub1: port 3, device disappeared after reset
Searching for RAID components...
WARNING: 1 error while detecting hardware; check system log.
boot device: wd0
root on wd0a dumps on wd0b
root file system type: ffs
kern.module.path=/stand/i386/8.99.30/modules
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)

dmesg with ACPI/SMP disabled:
NetBSD 8.99.30 (GENERIC) #0: Tue Jan 15 14:23:56 UTC 2019
        mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/i386/compile/GENERIC
total memory = 1983 MB
avail memory = 1929 MB
rnd: seeded with 128 bits
timecounter: Timecounters tick every 10.000 msec
Kernelized RAIDframe activated
running cgd selftest aes-xts-256 aes-xts-512 done
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
RDC Semiconductor Co., Ltd. EMKORE                 (1.0                   )
mainbus0 (root)
cpu0 at mainbus0
cpu0: Vortex86DX3, id 0x611
cpu0: package 0, core 0, smt 0
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0: vendor 17f3 product 6023 (rev. 0x02)
ppb0 at pci0 dev 1 function 0: vendor 17f3 product 1031 (rev. 0x01)
ppb0: PCI Express capability version 1 <PCI/PCI-X to PCI-E Bridge> x1 @ 2.5GT/s
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
ppb1 at pci0 dev 2 function 0: vendor 17f3 product 1031 (rev. 0x01)
ppb1: PCI Express capability version 1 <PCI/PCI-X to PCI-E Bridge> x1 @ 2.5GT/s
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
pcib0 at pci0 dev 7 function 0: vendor 17f3 product 6035 (rev. 0x01)
pcib1 at pci0 dev 7 function 1: vendor 17f3 product 6035 (rev. 0x01)
vte0 at pci0 dev 8 function 0: vendor 17f3 product 6040 (rev. 0x00)
vte0: Ethernet address xx:xx:xx:xx:xx:xx
vte0: interrupting at irq 5
ukphy0 at vte0 phy 1: OUI 0xfcff2f, model 0x0005, rev. 0
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
ohci0 at pci0 dev 10 function 0: vendor 17f3 product 6060 (rev. 0x14)
ohci0: interrupting at irq 15
ohci0: OHCI version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
ehci0 at pci0 dev 10 function 1: vendor 17f3 product 6061 (rev. 0x08)
ehci0: interrupting at irq 7
ehci0: BIOS has given up ownership
ehci0: EHCI version 1.0
ehci0: 1 companion controller, 4 ports: ohci0
usb1 at ehci0: USB revision 2.0
rdcide0 at pci0 dev 12 function 0: RDC R1012 IDE controller (rev. 0x02)
rdcide0: bus-master DMA support present
rdcide0: primary channel configured to compatibility mode
rdcide0: primary channel interrupting at irq 14
atabus0 at rdcide0 channel 0
rdcide0: secondary channel configured to compatibility mode
rdcide0: secondary channel ignored (disabled)
vga0 at pci0 dev 13 function 0: vendor 17f3 product 2015 (rev. 0x00)
wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
drm at vga0 not configured
hdaudio0 at pci0 dev 14 function 0: HD Audio Controller
hdaudio0: interrupting at irq 15
hdafg0 at hdaudio0: vendor 10ec product 0262
hdafg0: DAC00 2ch: Speaker [Jack]
hdafg0: DIG01 2ch: Digital Out [Jack]
hdafg0: ADC02 2ch: Mic In [Jack]
hdafg0: ADC03 2ch: Line In [Jack]
hdafg0: 2ch/2ch 44100Hz 48000Hz 96000Hz 192000Hz PCM16 PCM20 PCM24 AC3
audio0 at hdafg0: full duplex, playback, capture, mmap, independent
hdafg0: Virtual format configured - Format SLINEAR, precision 16, channels 2, frequency 48000
hdafg0: Latency: 128 milliseconds
isa0 at pcib0
pckbc0 at isa0 port 0x60-0x64
attimer0 at isa0 port 0x40-0x43
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker
sysbeep0 at pcppi0
attimer0: attached to pcppi0
isa at pcib1 not configured
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
uhub0 at usb0: NetBSD (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub0: 4 ports with 4 removable, self powered
uhub1 at usb1: NetBSD (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub1: 4 ports with 4 removable, self powered
IPsec: Initialized Security Association Processing.
axen0 at uhub1 port 1
axen0: D-Link Elec. Corp. (0x2001) D-Link DUB-1312 (0x4a00), rev 2.10/1.00, addr 2
axen0: AX88179
axen0: Ethernet address xx:xx:xx:xx:xx:xx
rgephy0 at axen0 phy 3: RTL8169S/8110S/8211 1000BASE-T media interface, rev. 5
rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, auto
ehci0: handing over low speed device on port 2 to ohci0
wd0 at atabus0 drive 0
wd0: <SS16G D0 RDC SD-IDE HOST CONTROLLER>
wd0: drive supports 1-sector PIO transfers, LBA addressing
wd0: 15193 MB, 30869 cyl, 16 head, 63 sec, 512 bytes/sect x 31116288 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
wd0(rdcide0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100) (using DMA)
ehci0: handing over full speed device on port 3 to ohci0
boot device: wd0
root on wd0a dumps on wd0b
root file system type: ffs
kern.module.path=/stand/i386/8.99.30/modules
WARNING: clock gained 2 days
uhidev0 at uhub0 port 2 configuration 1 interface 0
uhidev0: USB (0x1c4f) USB Keykoard (0x02), rev 1.10/1.10, addr 2, iclass 3/1
ukbd0 at uhidev0: 8 Variable keys, 6 Array codes
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub0 port 2 configuration 1 interface 1
uhidev1: USB (0x1c4f) USB Keykoard (0x02), rev 1.10/1.10, addr 2, iclass 3/0
uhidev1: 2 report ids
uhid0 at uhidev1 reportid 1: input=2, output=0, feature=0
uhid1 at uhidev1 reportid 2: input=1, output=0, feature=0
uchcom0 at uhub0 port 3
uchcom0: QinHeng Electronics (0x1a86) USB2.0-Ser! (0x7523), rev 1.10/2.54, addr 3
ucom0 at uchcom0
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)
>How-To-Repeat:
Boot machine with ACPI/SMP enabled
>Fix:
N/A. Workaround is to boot without ACPI/SMP enabled.

>Release-Note:

>Audit-Trail:
From: "David H. Gutteridge" <david@gutteridge.ca>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP
 enabled on eBox 3352DX3-AP
Date: Sat, 19 Jan 2019 20:09:06 -0500

 It would probably be helpful to provide USB_DEBUG output. To do so,
 please see http://www.netbsd.org/docs/kernel/#usb-debugging,
 specifically the part following "USB debugging code has been changed in
 NetBSD 7.99 and later."

 Dave


From: Andrius V <vezhlys@gmail.com>
To: gnats-bugs@netbsd.org
Cc: port-i386-maintainer@netbsd.org, gnats-admin@netbsd.org, 
	netbsd-bugs@netbsd.org
Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP enabled
 on eBox 3352DX3-AP
Date: Sun, 20 Jan 2019 22:56:58 +0200

 --000000000000d76b42057fe9fc98
 Content-Type: text/plain; charset="UTF-8"

 Hi,

 Thank you for the prompt reply. I can provide this output but it is a
 bit tricky since USB ports are the only available besides Ethernet and
 VGA (and integrated vte ethernet controller doesn't work properly as
 well unfortunately, so ssh is out of option). Does it matter when to
 do this output? I guess non ACPI/SMP boot won't provide the required
 data as well? Currently I did a cron job which outputs the vmstat -u
 usbhist result to file on boot. Please see an attached report for
 results. Theoretically, I can modify cronjob to post report a bit
 later if needed.

 Regards,
 Andrius V

 On Sun, Jan 20, 2019 at 3:10 AM David H. Gutteridge <david@gutteridge.ca> wrote:
 >
 > The following reply was made to PR port-i386/53894; it has been noted by GNATS.
 >
 > From: "David H. Gutteridge" <david@gutteridge.ca>
 > To: gnats-bugs@netbsd.org
 > Cc:
 > Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP
 >  enabled on eBox 3352DX3-AP
 > Date: Sat, 19 Jan 2019 20:09:06 -0500
 >
 >  It would probably be helpful to provide USB_DEBUG output. To do so,
 >  please see http://www.netbsd.org/docs/kernel/#usb-debugging,
 >  specifically the part following "USB debugging code has been changed in
 >  NetBSD 7.99 and later."
 >
 >  Dave
 >
 >

 --000000000000d76b42057fe9fc98
 Content-Type: text/plain; charset="US-ASCII"; name="usbhist.txt"
 Content-Disposition: attachment; filename="usbhist.txt"
 Content-Transfer-Encoding: base64
 Content-ID: <f_jr5dkcck0>
 X-Attachment-Id: f_jr5dkcck0

 NTAwMDAgZW50cmllcywgbmV4dCBpcyAxNTUKMDAwMDAwLjAwMDAwMCBvaGNpX2luaXQjMUAwOiBj
 YWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjMUAwOiBjYWxsZWQhCjAwMDAwMC4w
 MDAwMDAgb2hjaV9hbGxvY19zZWQjMUAwOiBhbGxvY2F0aW5nIGNodW5rCjAwMDAwMC4wMDAwMDAg
 b2hjaV9hbGxvY19zZWQjMkAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQj
 M0AwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjNEAwOiBjYWxsZWQhCjAw
 MDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjNUAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hj
 aV9hbGxvY19zZWQjNkAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjN0Aw
 OiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjOEAwOiBjYWxsZWQhCjAwMDAw
 MC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjOUAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9h
 bGxvY19zZWQjMTBAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzExQDA6
 IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCMxMkAwOiBjYWxsZWQhCjAwMDAw
 MC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjMTNAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lf
 YWxsb2Nfc2VkIzE0QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCMxNUAw
 OiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjMTZAMDogY2FsbGVkIQowMDAw
 MDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzE3QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNp
 X2FsbG9jX3NlZCMxOEAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjMTlA
 MDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzIwQDA6IGNhbGxlZCEKMDAw
 MDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCMyMUAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hj
 aV9hbGxvY19zZWQjMjJAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzIz
 QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCMyNEAwOiBjYWxsZWQhCjAw
 MDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjMjVAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9o
 Y2lfYWxsb2Nfc2VkIzI2QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCMy
 N0AwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjMjhAMDogY2FsbGVkIQow
 MDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzI5QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBv
 aGNpX2FsbG9jX3NlZCMzMEAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQj
 MzFAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzMyQDA6IGNhbGxlZCEK
 MDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCMzM0AwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAg
 b2hjaV9hbGxvY19zZWQjMzRAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2Vk
 IzM1QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCMzNkAwOiBjYWxsZWQh
 CjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjMzdAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAw
 IG9oY2lfYWxsb2Nfc2VkIzM4QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3Nl
 ZCMzOUAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjNDBAMDogY2FsbGVk
 IQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzQxQDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAw
 MCBvaGNpX2FsbG9jX3NlZCM0MkAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19z
 ZWQjNDNAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzQ0QDA6IGNhbGxl
 ZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCM0NUAwOiBjYWxsZWQhCjAwMDAwMC4wMDAw
 MDAgb2hjaV9hbGxvY19zZWQjNDZAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nf
 c2VkIzQ3QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCM0OEAwOiBjYWxs
 ZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjNDlAMDogY2FsbGVkIQowMDAwMDAuMDAw
 MDAwIG9oY2lfYWxsb2Nfc2VkIzUwQDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9j
 X3NlZCM1MUAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjNTJAMDogY2Fs
 bGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzUzQDA6IGNhbGxlZCEKMDAwMDAwLjAw
 MDAwMCBvaGNpX2FsbG9jX3NlZCM1NEAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxv
 Y19zZWQjNTVAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzU2QDA6IGNh
 bGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCM1N0AwOiBjYWxsZWQhCjAwMDAwMC4w
 MDAwMDAgb2hjaV9hbGxvY19zZWQjNThAMDogY2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxs
 b2Nfc2VkIzU5QDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2FsbG9jX3NlZCM2MEAwOiBj
 YWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjNjFAMDogY2FsbGVkIQowMDAwMDAu
 MDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzYyQDA6IGNhbGxlZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2Fs
 bG9jX3NlZCM2M0AwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9hbGxvY19zZWQjNjRAMDog
 Y2FsbGVkIQowMDAwMDAuMDAwMDAwIG9oY2lfYWxsb2Nfc2VkIzY1QDA6IGNhbGxlZCEKMDAwMDAw
 LjAwMDAwMCBvaGNpX2FsbG9jX3NlZCM2NkAwOiBjYWxsZWQhCjAwMDAwMC4wMDAwMDAgb2hjaV9p
 bml0IzFAMDogY29sZCBzdGFydGVkCjAwMDAwMC4wMDAwMDAgb2hjaV9pbml0IzFAMDogc2MgMHhj
 M2Y3MzAwODogcmVzZXR0aW5nCjAwMDAwMC4wMDAwMDAgb2hjaV9kdW1wcmVncyMxQDA6IGNhbGxl
 ZCEKMDAwMDAwLjAwMDAwMCBvaGNpX2R1bXByZWdzIzFAMDogcmV2PTB4MDAwMDAxMTAgY29udHJv
 bD0weDAwMDAwMGJmIGNvbW1hbmQ9MHgwMDAwMDAwMAowMDAwMDAuMDAwMDAwIG9oY2lfZHVtcHJl
 Z3MjMUAwOiAgICAgICAgICAgICAgICBpbnRyc3RhdD0weDAwMDAwMDA0IGludHJlPTB4MDAwMDAw
 MDAgaW50cmQ9MHgwMDAwMDAwMAowMDAwMDAuMDAwMDAwIG9oY2lfZHVtcHJlZ3MjMUAwOiAgICAg
 ICAgICAgICAgICBoY2NhPTB4NDIwMDUwMDAgcGVyY3VyPTB4MDAwMDAwMDAgY3RybGhkPTB4NDIw
 MDc3ZDAKMDAwMDAwLjAwMDAwMCBvaGNpX2R1bXByZWdzIzFAMDogICAgICAgICAgICAgICAgY3Ry
 bGN1cj0weDAwMDAwMDAwIGJ1bGtoZD0weDQyMDA3N2EwIGJ1bGtjdXI9MHgwMDAwMDAwMAowMDAw
 MDAuMDAwMDAwIG9oY2lfZHVtcHJlZ3MjMUAwOiAgICAgICAgICAgICAgICBkb25lPTB4MDAwMDAw
 MDAgZm1pdmFsPTB4YTc3ODJlZGYgZm1yZW09MHg4MDAwMmUxNAowMDAwMDAuMDAwMDAwIG9oY2lf
 ZHVtcHJlZ3MjMUAwOiAgICAgICAgICAgICAgICBmbW51bT0weDAwMDAwMDBjIHBlcnN0PTB4MDAw
 MDJhMmYgbHN0aHJzPTB4MDAwMDA2MjgKMDAwMDAwLjAwMDAwMCBvaGNpX2R1bXByZWdzIzFAMDog
 ICAgICAgICAgICAgICAgZGVzY2E9MHgwZjAwMDkwNCBkZXNjYj0weDAwMWUwMDAwIHN0YXQ9MHgw
 MDAwMDAwMAowMDAwMDAuMDAwMDAwIG9oY2lfZHVtcHJlZ3MjMUAwOiAgICAgICAgICAgICAgICBw
 b3J0MT0weDAwMDAwMDAwIHBvcnQyPTB4MDAwMDAwMDAKMDAwMDAwLjAwMDAwMCBvaGNpX2R1bXBy
 ZWdzIzFAMDogICAgICAgICAgSENDQTogZnJhbWVfbnVtYmVyPTB4MDAwYyBkb25lX2hlYWQ9MHgw
 MDAwMDAwMAowMDAwMDAuMDAwMDAwIG9oY2lfaW5pdCMxQDA6IGVuYWJsaW5nIDB4ODAwMDAwNWIK
 MDAwMDAxLjk3Nzk4NyBvaGNpX29wZW4jMUAwOiBjYWxsZWQhCjAwMDAwMS45Nzc5OTMgb2hjaV9v
 cGVuIzFAMDogcGlwZT0weGMzYWFiOTM4LCBhZGRyPTAsIGVuZHB0PTAgKDApCjAwMDAwMS45Nzgw
 NDUgb2hjaV9yb290aHViX2N0cmwjMUAwOiBjYWxsZWQhCjAwMDAwMS45NzgwNTAgb2hjaV9yb290
 aHViX2N0cmwjMUAwOiB0eXBlPTB4ODAgcmVxdWVzdD0wNgowMDAwMDEuOTc4MDU3IG9oY2lfcm9v
 dGh1Yl9jdHJsIzFAMDogd1ZhbHVlPTB4MDEwMAowMDAwMDEuOTc4MDc1IG9oY2lfb3BlbiMyQDA6
 IGNhbGxlZCEKMDAwMDAxLjk3ODA4MCBvaGNpX29wZW4jMkAwOiBwaXBlPTB4YzNhYWI5MzgsIGFk
 ZHI9MCwgZW5kcHQ9MCAoMCkKMDAwMDAxLjk3ODA4OSBvaGNpX3Jvb3RodWJfY3RybCMyQDA6IGNh
 bGxlZCEKMDAwMDAxLjk3ODA5NCBvaGNpX3Jvb3RodWJfY3RybCMyQDA6IHR5cGU9MHgwMCByZXF1
 ZXN0PTA1CjAwMDAwMS45OTI1Mjkgb2hjaV9vcGVuIzNAMDogY2FsbGVkIQowMDAwMDEuOTkyNTMz
 IG9oY2lfb3BlbiMzQDA6IHBpcGU9MHhjM2FhYjkzOCwgYWRkcj0xLCBlbmRwdD0wICgxKQowMDAw
 MDEuOTkyNTQ1IG9oY2lfcm9vdGh1Yl9jdHJsIzNAMDogY2FsbGVkIQowMDAwMDEuOTkyNTUxIG9o
 Y2lfcm9vdGh1Yl9jdHJsIzNAMDogdHlwZT0weDgwIHJlcXVlc3Q9MDYKMDAwMDAxLjk5MjU1NiBv
 aGNpX3Jvb3RodWJfY3RybCMzQDA6IHdWYWx1ZT0weDAxMDAKMDAwMDAxLjk5MjU3MiBvaGNpX3Jv
 b3RodWJfY3RybCM0QDA6IGNhbGxlZCEKMDAwMDAxLjk5MjU3NiBvaGNpX3Jvb3RodWJfY3RybCM0
 QDA6IHR5cGU9MHg4MCByZXF1ZXN0PTA2CjAwMDAwMS45OTI1ODEgb2hjaV9yb290aHViX2N0cmwj
 NEAwOiB3VmFsdWU9MHgwMzAwCjAwMDAwMS45OTI1ODkgb2hjaV9yb290aHViX2N0cmwjNUAwOiBj
 YWxsZWQhCjAwMDAwMS45OTI1OTMgb2hjaV9yb290aHViX2N0cmwjNUAwOiB0eXBlPTB4ODAgcmVx
 dWVzdD0wNgowMDAwMDEuOTkyNjAwIG9oY2lfcm9vdGh1Yl9jdHJsIzVAMDogd1ZhbHVlPTB4MDMw
 MAowMDAwMDEuOTkyNjA5IG9oY2lfcm9vdGh1Yl9jdHJsIzZAMDogY2FsbGVkIQowMDAwMDEuOTky
 NjE0IG9oY2lfcm9vdGh1Yl9jdHJsIzZAMDogdHlwZT0weDgwIHJlcXVlc3Q9MDYKMDAwMDAxLjk5
 MjYxOSBvaGNpX3Jvb3RodWJfY3RybCM2QDA6IHdWYWx1ZT0weDAzMDEKMDAwMDAxLjk5MjYyNyBv
 aGNpX3Jvb3RodWJfY3RybCM3QDA6IGNhbGxlZCEKMDAwMDAxLjk5MjYzMSBvaGNpX3Jvb3RodWJf
 Y3RybCM3QDA6IHR5cGU9MHg4MCByZXF1ZXN0PTA2CjAwMDAwMS45OTI2MzYgb2hjaV9yb290aHVi
 X2N0cmwjN0AwOiB3VmFsdWU9MHgwMzAxCjAwMDAwMS45OTI2NDggb2hjaV9yb290aHViX2N0cmwj
 OEAwOiBjYWxsZWQhCjAwMDAwMS45OTI2NTMgb2hjaV9yb290aHViX2N0cmwjOEAwOiB0eXBlPTB4
 ODAgcmVxdWVzdD0wNgowMDAwMDEuOTkyNjU4IG9oY2lfcm9vdGh1Yl9jdHJsIzhAMDogd1ZhbHVl
 PTB4MDMwMgowMDAwMDEuOTkyNjY2IG9oY2lfcm9vdGh1Yl9jdHJsIzlAMDogY2FsbGVkIQowMDAw
 MDEuOTkyNjcxIG9oY2lfcm9vdGh1Yl9jdHJsIzlAMDogdHlwZT0weDgwIHJlcXVlc3Q9MDYKMDAw
 MDAxLjk5MjY3NiBvaGNpX3Jvb3RodWJfY3RybCM5QDA6IHdWYWx1ZT0weDAzMDIKMDAwMDAxLjk5
 MzIzNyBvaGNpX3Jvb3RodWJfY3RybCMxMEAwOiBjYWxsZWQhCjAwMDAwMS45OTMyNDIgb2hjaV9y
 b290aHViX2N0cmwjMTBAMDogdHlwZT0weDgwIHJlcXVlc3Q9MDYKMDAwMDAxLjk5MzI0OSBvaGNp
 X3Jvb3RodWJfY3RybCMxMEAwOiB3VmFsdWU9MHgwMjAwCjAwMDAwMS45OTMyNjEgb2hjaV9yb290
 aHViX2N0cmwjMTFAMDogY2FsbGVkIQowMDAwMDEuOTkzMjY2IG9oY2lfcm9vdGh1Yl9jdHJsIzEx
 QDA6IHR5cGU9MHg4MCByZXF1ZXN0PTA2CjAwMDAwMS45OTMyNzAgb2hjaV9yb290aHViX2N0cmwj
 MTFAMDogd1ZhbHVlPTB4MDIwMAowMDAwMDEuOTkzMjgwIG9oY2lfcm9vdGh1Yl9jdHJsIzEyQDA6
 IGNhbGxlZCEKMDAwMDAxLjk5MzI4NCBvaGNpX3Jvb3RodWJfY3RybCMxMkAwOiB0eXBlPTB4MDAg
 cmVxdWVzdD0wOQowMDAwMDEuOTkzMjk3IG9oY2lfcm9vdGh1Yl9jdHJsIzEzQDA6IGNhbGxlZCEK
 MDAwMDAxLjk5MzMwMiBvaGNpX3Jvb3RodWJfY3RybCMxM0AwOiB0eXBlPTB4YTAgcmVxdWVzdD0w
 NgowMDAwMDEuOTkzMzM4IG9oY2lfb3BlbiM0QDA6IGNhbGxlZCEKMDAwMDAxLjk5MzM0MyBvaGNp
 X29wZW4jNEAwOiBwaXBlPTB4YzNhYWI4NTgsIGFkZHI9MSwgZW5kcHQ9MTI5ICgxKQowMDAwMDIu
 MjAyNjU3IG9oY2lfcm9vdGh1Yl9jdHJsIzE0QDA6IGNhbGxlZCEKMDAwMDAyLjIwMjY2MiBvaGNp
 X3Jvb3RodWJfY3RybCMxNEAwOiB0eXBlPTB4MjMgcmVxdWVzdD0wMwowMDAwMDIuMjAyNjY4IG9o
 Y2lfcm9vdGh1Yl9jdHJsIzE0QDA6IHNldCBwb3J0IHBvd2VyIDEKMDAwMDAyLjIwMjY4MCBvaGNp
 X3Jvb3RodWJfY3RybCMxNUAwOiBjYWxsZWQhCjAwMDAwMi4yMDI2ODUgb2hjaV9yb290aHViX2N0
 cmwjMTVAMDogdHlwZT0weDIzIHJlcXVlc3Q9MDMKMDAwMDAyLjIwMjY4OSBvaGNpX3Jvb3RodWJf
 Y3RybCMxNUAwOiBzZXQgcG9ydCBwb3dlciAyCjAwMDAwMi4yMDI2OTcgb2hjaV9yb290aHViX2N0
 cmwjMTZAMDogY2FsbGVkIQowMDAwMDIuMjAyNzAyIG9oY2lfcm9vdGh1Yl9jdHJsIzE2QDA6IHR5
 cGU9MHgyMyByZXF1ZXN0PTAzCjAwMDAwMi4yMDI3MDggb2hjaV9yb290aHViX2N0cmwjMTZAMDog
 c2V0IHBvcnQgcG93ZXIgMwowMDAwMDIuMjAyNzE1IG9oY2lfcm9vdGh1Yl9jdHJsIzE3QDA6IGNh
 bGxlZCEKMDAwMDAyLjIwMjcyMCBvaGNpX3Jvb3RodWJfY3RybCMxN0AwOiB0eXBlPTB4MjMgcmVx
 dWVzdD0wMwowMDAwMDIuMjAyNzI2IG9oY2lfcm9vdGh1Yl9jdHJsIzE3QDA6IHNldCBwb3J0IHBv
 d2VyIDQKMDAwMDAyLjcxMjgzNyBvaGNpX3Jvb3RodWJfY3RybCMxOEAwOiBjYWxsZWQhCjAwMDAw
 Mi43MTI4NDEgb2hjaV9yb290aHViX2N0cmwjMThAMDogdHlwZT0weGEwIHJlcXVlc3Q9MDAKMDAw
 MDAyLjcxMjg1MSBvaGNpX3Jvb3RodWJfY3RybCMxOUAwOiBjYWxsZWQhCjAwMDAwMi43MTI4NTYg
 b2hjaV9yb290aHViX2N0cmwjMTlAMDogdHlwZT0weGEzIHJlcXVlc3Q9MDAKMDAwMDAyLjcxMjg2
 MSBvaGNpX3Jvb3RodWJfY3RybCMxOUAwOiBnZXQgcG9ydCBzdGF0dXMgaT0xCjAwMDAwMi43MTI4
 Njggb2hjaV9yb290aHViX2N0cmwjMTlAMDogcG9ydCBzdGF0dXM9MHgwMTAwCjAwMDAwMi43MTI4
 NzYgb2hjaV9yb290aHViX2N0cmwjMjBAMDogY2FsbGVkIQowMDAwMDIuNzEyODgyIG9oY2lfcm9v
 dGh1Yl9jdHJsIzIwQDA6IHR5cGU9MHhhMyByZXF1ZXN0PTAwCjAwMDAwMi43MTI4ODYgb2hjaV9y
 b290aHViX2N0cmwjMjBAMDogZ2V0IHBvcnQgc3RhdHVzIGk9MgowMDAwMDIuNzEyODkxIG9oY2lf
 cm9vdGh1Yl9jdHJsIzIwQDA6IHBvcnQgc3RhdHVzPTB4MDEwMAowMDAwMDIuNzEyOTAwIG9oY2lf
 cm9vdGh1Yl9jdHJsIzIxQDA6IGNhbGxlZCEKMDAwMDAyLjcxMjkwNCBvaGNpX3Jvb3RodWJfY3Ry
 bCMyMUAwOiB0eXBlPTB4YTMgcmVxdWVzdD0wMAowMDAwMDIuNzEyOTEwIG9oY2lfcm9vdGh1Yl9j
 dHJsIzIxQDA6IGdldCBwb3J0IHN0YXR1cyBpPTMKMDAwMDAyLjcxMjkxNiBvaGNpX3Jvb3RodWJf
 Y3RybCMyMUAwOiBwb3J0IHN0YXR1cz0weDAxMDAKMDAwMDAyLjcxMjkyNCBvaGNpX3Jvb3RodWJf
 Y3RybCMyMkAwOiBjYWxsZWQhCjAwMDAwMi43MTI5Mjkgb2hjaV9yb290aHViX2N0cmwjMjJAMDog
 dHlwZT0weGEzIHJlcXVlc3Q9MDAKMDAwMDAyLjcxMjkzNCBvaGNpX3Jvb3RodWJfY3RybCMyMkAw
 OiBnZXQgcG9ydCBzdGF0dXMgaT00CjAwMDAwMi43MTI5Mzkgb2hjaV9yb290aHViX2N0cmwjMjJA
 MDogcG9ydCBzdGF0dXM9MHgwMTAwCgo=
 --000000000000d76b42057fe9fc98--

Responsible-Changed-From-To: port-i386-maintainer->skrll
Responsible-Changed-By: gutteridge@NetBSD.org
Responsible-Changed-When: Sun, 20 Jan 2019 23:50:13 +0000
Responsible-Changed-Why:
skrll: Please advise on any further detail needed here?

From: Nick Hudson <nick.hudson@gmx.co.uk>
To: gnats-bugs@NetBSD.org, port-i386-maintainer@netbsd.org,
 netbsd-bugs@netbsd.org, gnats-admin@netbsd.org, gutteridge@NetBSD.org,
 vezhlys@gmail.com
Cc: 
Subject: Re: port-i386/53894 ("ehci_sync_hc: timed out" with ACPI/SMP enabled
 on eBox 3352DX3-AP)
Date: Mon, 21 Jan 2019 06:40:28 +0000

 On 20/01/2019 23:50, gutteridge@NetBSD.org wrote:
 > Synopsis: "ehci_sync_hc: timed out" with ACPI/SMP enabled on eBox 3352DX3-AP

 This sounds more like an MP/interrupt problem for your system than usb.

 ehcidebug=1 should shows us if we're seeing interrupts.

 Nick

From: Andrius V <vezhlys@gmail.com>
To: Nick Hudson <nick.hudson@gmx.co.uk>
Cc: gnats-bugs@netbsd.org, port-i386-maintainer@netbsd.org, 
	netbsd-bugs@netbsd.org, gnats-admin@netbsd.org, gutteridge@netbsd.org
Subject: Re: port-i386/53894 ("ehci_sync_hc: timed out" with ACPI/SMP enabled
 on eBox 3352DX3-AP)
Date: Thu, 24 Jan 2019 00:10:42 +0200

 On Mon, Jan 21, 2019 at 8:40 AM Nick Hudson <nick.hudson@gmx.co.uk> wrote:
 >
 > On 20/01/2019 23:50, gutteridge@NetBSD.org wrote:
 > > Synopsis: "ehci_sync_hc: timed out" with ACPI/SMP enabled on eBox 3352DX3-AP
 >
 > This sounds more like an MP/interrupt problem for your system than usb.
 >
 > ehcidebug=1 should shows us if we're seeing interrupts.
 >

 Hi,

 Do you mean EHCI_DEBUG option in kernel configuration or something
 else? Currently I was testing kernel which included EHCI_DEBUG option
 line and I set hw.ehci.debug value as well, however I didn't see any
 interrupt messages during boot. There were no vmstat output changes
 either (I tried delaying it for few minutes, but it didn't give more
 than few additional messages of "usb_event_thread#0@0: sc 0xc3d5d1b8
 woke up", "usb_discover#1@0: called!" compared to original one I've
 attached). After removing USB Ethernet device and leaving USB keyboard
 only, "ehci_sync_hc: timed out" messages disappeared from boot log,
 but USB keyboard still didn't power up (no LED lights lit up),
 appeared in dmesg or worked nevertheless. Finally, I tried to boot
 without SMP but with ACPI enabled (boot -1), the problem stayed the
 same. So I would assume it can be ACPI related issue, since disabling
 ACPI and SMP makes USB devices work (though with occasional
 reconnection cycles). I couldn't test an opposite situation (SMP
 enabled/ACPI disabled), because I am getting IDE interrupt errors in
 this case and system fails to boot.

 Regards,
 Andrius V


 On Mon, Jan 21, 2019 at 8:40 AM Nick Hudson <nick.hudson@gmx.co.uk> wrote:
 >
 > On 20/01/2019 23:50, gutteridge@NetBSD.org wrote:
 > > Synopsis: "ehci_sync_hc: timed out" with ACPI/SMP enabled on eBox 3352DX3-AP
 >
 > This sounds more like an MP/interrupt problem for your system than usb.
 >
 > ehcidebug=1 should shows us if we're seeing interrupts.
 >
 > Nick
 >

Responsible-Changed-From-To: skrll->port-i386-maintainer
Responsible-Changed-By: skrll@NetBSD.org
Responsible-Changed-When: Wed, 30 Jan 2019 08:33:19 +0000
Responsible-Changed-Why:
not a usb bug. the problem is interrupts related.


From: Andrius V <vezhlys@gmail.com>
To: gnats-bugs@netbsd.org
Cc: port-i386-maintainer@netbsd.org, gnats-admin@netbsd.org, 
	netbsd-bugs@netbsd.org, Taylor R Campbell <riastradh@netbsd.org>
Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP enabled
 on eBox 3352DX3-AP
Date: Sun, 14 Nov 2021 11:28:59 +0200

 Placed system's acpidump to https://netbsd.org/~andvar/acpidump.txt on
 request from Taylor.

 On Sun, Jan 20, 2019 at 10:56 PM Andrius V <vezhlys@gmail.com> wrote:
 >
 > Hi,
 >
 > Thank you for the prompt reply. I can provide this output but it is a
 > bit tricky since USB ports are the only available besides Ethernet and
 > VGA (and integrated vte ethernet controller doesn't work properly as
 > well unfortunately, so ssh is out of option). Does it matter when to
 > do this output? I guess non ACPI/SMP boot won't provide the required
 > data as well? Currently I did a cron job which outputs the vmstat -u
 > usbhist result to file on boot. Please see an attached report for
 > results. Theoretically, I can modify cronjob to post report a bit
 > later if needed.
 >
 > Regards,
 > Andrius V
 >
 > On Sun, Jan 20, 2019 at 3:10 AM David H. Gutteridge <david@gutteridge.ca> wrote:
 > >
 > > The following reply was made to PR port-i386/53894; it has been noted by GNATS.
 > >
 > > From: "David H. Gutteridge" <david@gutteridge.ca>
 > > To: gnats-bugs@netbsd.org
 > > Cc:
 > > Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP
 > >  enabled on eBox 3352DX3-AP
 > > Date: Sat, 19 Jan 2019 20:09:06 -0500
 > >
 > >  It would probably be helpful to provide USB_DEBUG output. To do so,
 > >  please see http://www.netbsd.org/docs/kernel/#usb-debugging,
 > >  specifically the part following "USB debugging code has been changed in
 > >  NetBSD 7.99 and later."
 > >
 > >  Dave
 > >
 > >

From: Andrius V <vezhlys@gmail.com>
To: port-i386-maintainer@netbsd.org, gnats-admin@netbsd.org, 
	netbsd-bugs@netbsd.org, gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP enabled
 on eBox 3352DX3-AP
Date: Wed, 16 Mar 2022 10:25:14 +0200

 Hi,

 I did some investigation on this issue in recent days and here are my findings:
 * I can confirm, that it is interrupts issue, more specifically level
 type PCI interrupts are not working (thus, USB, network, SCI, probably
 audio too (untested)) interrupts doesn't work (checked by intrctl and
 vmstat).
 * The interrupt establishing flow seems OK though, did some debugging,
 which shows that the flow doesn't seem to be different from similar
 system which has no issues and visually everything is "established".
 * By advice I checked MADT overrides: system reports two overrides,
 and indeed, it seems that IRQ 0 NetBSD do not override. Unfortunately,
 I have no idea, where to do that in NetBSD code (FreeBSD seems to go
 through all MADT table and do overrides, in NetBSD case only SCI
 interrupt individually comes to override code). Would that fix the
 issue, honestly, I am not certain, I did try silly hacks, like not
 masking interrupts by default, which added "i82489_ipi_init: ESR
 00000040" to dmesg, but none helped to make PCI interrupts work.
 acpi0: [MADT] INTR OVERRIDE  : BUS 0, IRQ 0, GSI 2, FLAGS 0x00
 acpi0: [MADT] INTR OVERRIDE  : BUS 0, IRQ 9, GSI 9, FLAGS 0x0D
 * SoC does not support MSI.
 * current kernel doesn't report "ehci_sync_hc: timed out" anymore
 compared to my original dmesg report, just silently boots without any
 uhidev devices attached, likely related to recent USB changes.
 * FreeBSD 12.0 (the last release which boots out of the box on the
 system), and recent Linux kernels has no problems with these
 interrupts. OpenBSD have likely the same issue as NetBSD though, since
 USB and network also doesn't work.
 * Booting without ACPI/SMP enabled, makes interrupts work (it doesn't
 go through apic in this case of course).

 Unfortunately, I am out of ideas for now on next steps, but will keep
 investigating once I have some, any help would be welcome. Thank you.

 Below attaching snippet from dmesg from acpi verbose (acpi dump still
 available here https://netbsd.org/~andvar/acpidump.txt):
 acpi0: [MADT] LAPIC          : CPU ID 1, LAPIC ID 0, FLAGS 0x01
 acpi0: [MADT] LAPIC          : CPU ID 2, LAPIC ID 1, FLAGS 0x01
 acpi0: [MADT] I/O APIC       : ID 2, GSI 0, ADDR 0xFEC00000
 acpi0: [MADT] INTR OVERRIDE  : BUS 0, IRQ 0, GSI 2, FLAGS 0x00
 acpi0: [MADT] INTR OVERRIDE  : BUS 0, IRQ 9, GSI 9, FLAGS 0x0D
 acpi0: [FADT] FACS           : 0x000000007BFDE000
 acpi0: [FADT] DSDT           : 0x000000007BFD05D0
 acpi0: [FADT] INT_MODEL      : 0x0000000000000001
 acpi0: [FADT] PM_PROFILE     : 0x0000000000000002
 acpi0: [FADT] SCI_INT        : 0x0000000000000009
 acpi0: [FADT] SMI_CMD        : 0x00000000000000B2
 acpi0: [FADT] ACPI_ENABLE    : 0x00000000000000E1
 acpi0: [FADT] ACPI_DISABLE   : 0x000000000000001E
 acpi0: [FADT] S4BIOS_REQ     : 0x0000000000000000
 acpi0: [FADT] PSTATE_CNT     : 0x00000000000000E2
 acpi0: [FADT] PM1a_EVT_BLK   : 0x0000000000000800
 acpi0: [FADT] PM1b_EVT_BLK   : 0x0000000000000000
 acpi0: [FADT] PM1a_CNT_BLK   : 0x0000000000000804
 acpi0: [FADT] PM1b_CNT_BLK   : 0x0000000000000000
 acpi0: [FADT] PM2_CNT_BLK    : 0x0000000000000000
 acpi0: [FADT] PM_TMR_BLK     : 0x0000000000000808
 acpi0: [FADT] GPE0_BLK       : 0x0000000000000820
 acpi0: [FADT] GPE1_BLK       : 0x0000000000000000
 acpi0: [FADT] PM1_EVT_LEN    : 0x0000000000000004
 acpi0: [FADT] PM1_CNT_LEN    : 0x0000000000000002
 acpi0: [FADT] PM2_CNT_LEN    : 0x0000000000000000
 acpi0: [FADT] PM_TMR_LEN     : 0x0000000000000004
 acpi0: [FADT] GPE0_BLK_LEN   : 0x0000000000000008
 acpi0: [FADT] GPE1_BLK_LEN   : 0x0000000000000000
 acpi0: [FADT] GPE1_BASE      : 0x0000000000000000
 acpi0: [FADT] CST_CNT        : 0x00000000000000E3
 acpi0: [FADT] P_LVL2_LAT     : 0x0000000000000065
 acpi0: [FADT] P_LVL3_LAT     : 0x00000000000003E9
 acpi0: [FADT] FLUSH_SIZE     : 0x0000000000000400
 acpi0: [FADT] FLUSH_STRIDE   : 0x0000000000000010
 acpi0: [FADT] DUTY_OFFSET    : 0x0000000000000001
 acpi0: [FADT] DUTY_WIDTH     : 0x0000000000000003
 acpi0: [FADT] DAY_ALRM       : 0x000000000000000D
 acpi0: [FADT] MON_ALRM       : 0x0000000000000000
 acpi0: [FADT] CENTURY        : 0x0000000000000032
 acpi0: [FADT] IAPC_BOOT_ARCH : 0x0000000000000003
 acpi0: [FADT] RESET_VALUE    : 0x0000000000000006
 acpi0: [FADT] ARM_BOOT_ARCH  : 0x0000000000000000
 acpi0: [FADT] X_FACS         : 0x000000007BFDE000
 acpi0: [FADT] X_DSDT         : 0x000000007BFD05D0
 acpi0: [FADT] RESET_REG      : 0x0000000000000CF9, SPACE ID 1, BIT
 WIDTH 8, BIT OFFSET 0, ACCESS WIDTH 0
 acpi0: [FADT] X_PM1a_EVT_BLK : 0x0000000000000800, SPACE ID 1, BIT
 WIDTH 32, BIT OFFSET 0, ACCESS WIDTH 0
 acpi0: [FADT] X_PM1b_EVT_BLK : 0x0000000000000000, SPACE ID 1, BIT
 WIDTH 32, BIT OFFSET 0, ACCESS WIDTH 0
 acpi0: [FADT] X_PM1a_CNT_BLK : 0x0000000000000804, SPACE ID 1, BIT
 WIDTH 16, BIT OFFSET 0, ACCESS WIDTH 0
 acpi0: [FADT] X_PM1b_CNT_BLK : 0x0000000000000000, SPACE ID 0, BIT
 WIDTH 0, BIT OFFSET 0, ACCESS WIDTH 0
 acpi0: [FADT] X_PM2_CNT_BLK  : 0x0000000000000000, SPACE ID 0, BIT
 WIDTH 0, BIT OFFSET 0, ACCESS WIDTH 0
 acpi0: [FADT] X_PM_TMR_BLK   : 0x0000000000000808, SPACE ID 1, BIT
 WIDTH 32, BIT OFFSET 0, ACCESS WIDTH 0
 acpi0: [FADT] X_GPE0_BLK     : 0x0000000000000820, SPACE ID 1, BIT
 WIDTH 64, BIT OFFSET 0, ACCESS WIDTH 0
 acpi0: [FADT] X_GPE1_BLK     : 0x0000000000000000, SPACE ID 1, BIT
 WIDTH 32, BIT OFFSET 0, ACCESS WIDTH 0
 acpi0: [FADT] SLEEP_CTRL_REG : 0x0000000000000000, SPACE ID 0, BIT
 WIDTH 0, BIT OFFSET 0, ACCESS WIDTH 0
 acpi0: [FADT] SLEEP_STAT_REG : 0x0000000000000000, SPACE ID 0, BIT
 WIDTH 0, BIT OFFSET 0, ACCESS WIDTH 0
 acpi0: [FADT] WBINVD         : 0x0000000000000001
 acpi0: [FADT] WBINVD_FLUSH   : 0x0000000000000000
 acpi0: [FADT] PROC_C1        : 0x0000000000000000
 acpi0: [FADT] P_LVL2_UP      : 0x0000000000000000
 acpi0: [FADT] PWR_BUTTON     : 0x0000000000000000
 acpi0: [FADT] SLP_BUTTON     : 0x0000000000000020
 acpi0: [FADT] FIX_RTC        : 0x0000000000000000
 acpi0: [FADT] RTC_S4         : 0x0000000000000080
 acpi0: [FADT] TMR_VAL_EXT    : 0x0000000000000000
 acpi0: [FADT] DCK_CAP        : 0x0000000000000000
 acpi0: [FADT] RESET_REG_SUP  : 0x0000000000000000
 acpi0: [FADT] SEALED_CASE    : 0x0000000000000000
 acpi0: [FADT] HEADLESS       : 0x0000000000000000
 acpi0: [FADT] CPU_SW_SLP     : 0x0000000000000000
 acpi0: [FADT] PCI_EXP_WAK    : 0x0000000000000000
 acpi0: [FADT] PLATFORM_CLK   : 0x0000000000000000
 acpi0: [FADT] S4_RTC_STS     : 0x0000000000000000
 acpi0: [FADT] REMOTE_POWER   : 0x0000000000000000
 acpi0: [FADT] APIC_CLUSTER   : 0x0000000000000000
 acpi0: [FADT] APIC_PHYSICAL  : 0x0000000000000000
 acpi0: [FADT] HW_REDUCED     : 0x0000000000000000
 acpi0: [FADT] LOW_POWER_S0   : 0x0000000000000000
 acpi0: [\   ] HID -          UID -    ADR -
 acpi0: [P001] HID -          UID -    ADR -
 acpi0: [P002] HID -          UID -    ADR -
 acpi0: [_SB ] HID -          UID -    ADR -
 acpi0: [PCI0] HID PNP0A03    UID 0    ADR 0x0000000000000000
 acpi0: [VGA ] HID -          UID -    ADR 0x00000000000D0000
 acpi0: [CRTD] HID -          UID -    ADR 0x0000000000000100
 acpi0: [P0P1] HID -          UID -    ADR 0x0000000000010000
 acpi0: [PCI1] HID -          UID -    ADR 0x0000000000000000
 acpi0: [P0P2] HID -          UID -    ADR 0x0000000000020000
 acpi0: [PCI2] HID -          UID -    ADR 0x0000000000000000
 acpi0: [P0P3] HID -          UID -    ADR 0x0000000000030000
 acpi0: [PCI3] HID -          UID -    ADR 0x0000000000000000
 acpi0: [SBRG] HID -          UID -    ADR 0x0000000000070000
 acpi0: [PIC ] HID PNP0000    UID -    ADR -
 acpi0: [DMAD] HID PNP0200    UID -    ADR -
 acpi0: [TMR ] HID PNP0100    UID -    ADR -
 acpi0: [RTC0] HID PNP0B00    UID -    ADR -
 acpi0: [PS2K] HID PNP0303    UID -    ADR -
 acpi0: [PS2M] HID PNP0F03    UID -    ADR -
 acpi0: [SPKR] HID PNP0800    UID -    ADR -
 acpi0: [COPR] HID PNP0C04    UID -    ADR -
 acpi0: [RMSC] HID PNP0C02    UID 16   ADR -
 acpi0: [UARA] HID PNP0501    UID COM1 ADR -
 acpi0: [UARB] HID PNP0501    UID COM2 ADR -
 acpi0: [UARC] HID PNP0501    UID COM3 ADR -
 acpi0: [UARD] HID PNP0501    UID COM4 ADR -
 acpi0: [UARE] HID PNP0501    UID COM5 ADR -
 acpi0: [UARF] HID PNP0501    UID COM6 ADR -
 acpi0: [UARG] HID PNP0501    UID COM7 ADR -
 acpi0: [UARH] HID PNP0501    UID COM8 ADR -
 acpi0: [UARI] HID PNP0501    UID COM9 ADR -
 acpi0: [LPTA] HID PNP0400    UID -    ADR -
 acpi0: [OMSC] HID PNP0C02    UID 0    ADR -
 acpi0: [EC0 ] HID PNP0C09    UID -    ADR -
 acpi0: [SBF1] HID -          UID -    ADR 0x0000000000070001
 acpi0: [LAN0] HID -          UID -    ADR 0x0000000000080000
 acpi0: [USB1] HID -          UID -    ADR 0x00000000000A0000
 acpi0: [EUSB] HID -          UID -    ADR 0x00000000000A0001
 acpi0: [IDE0] HID -          UID -    ADR 0x00000000000C0000
 acpi0: [CHN0] HID -          UID -    ADR 0x0000000000000000
 acpi0: [DRV0] HID -          UID -    ADR 0x0000000000000000
 acpi0: [DRV1] HID -          UID -    ADR 0x0000000000000001
 acpi0: [CHN1] HID -          UID -    ADR 0x0000000000000001
 acpi0: [DRV0] HID -          UID -    ADR 0x0000000000000000
 acpi0: [DRV1] HID -          UID -    ADR 0x0000000000000001
 acpi0: [HDAC] HID -          UID -    ADR 0x00000000000E0000
 acpi0: [AC0 ] HID ACPI0003   UID -    ADR -
 acpi0: [BAT0] HID PNP0C0A    UID 0    ADR -
 acpi0: [BAT1] HID PNP0C0A    UID 1    ADR -
 acpi0: [RMEM] HID PNP0C01    UID 1    ADR -
 acpi0: [LNKA] HID PNP0C0F    UID 1    ADR -
 acpi0: [LNKB] HID PNP0C0F    UID 2    ADR -
 acpi0: [LNKC] HID PNP0C0F    UID 3    ADR -
 acpi0: [LNKD] HID PNP0C0F    UID 4    ADR -
 acpi0: [LNKE] HID PNP0C0F    UID 5    ADR -
 acpi0: [LNKF] HID PNP0C0F    UID 6    ADR -
 acpi0: [LNKG] HID PNP0C0F    UID 7    ADR -
 acpi0: [LNKH] HID PNP0C0F    UID 8    ADR -
 acpi0: [PWRB] HID PNP0C0C    UID 170  ADR -
 acpi0: [SLPB] HID PNP0C0E    UID -    ADR -
 acpi0: [LID ] HID PNP0C0D    UID -    ADR -
 acpi0: [_TZ ] HID -          UID -    ADR -
 acpi0: [THRM] HID -          UID -    ADR -

 \     [06] []
     P001  [12] []
     P002  [12] []
     _SB   [06] []
         PCI0  [06] [] (PCI) @ 0x00:0x00:0x00:0x00 [R] [B] -> 0x00:0x00
             VGA   [06] [] (PCI) @ 0x00:0x00:0x0D:0x00
                 CRTD  [06] []
             P0P1  [06] [] (PCI) @ 0x00:0x00:0x01:0x00 [B] -> 0x00:0x01
                 PCI1  [06] [] (PCI) @ 0x00:0x01:0x00:0x00
             P0P2  [06] [] (PCI) @ 0x00:0x00:0x02:0x00 [B] -> 0x00:0x02
                 PCI2  [06] [] (PCI) @ 0x00:0x02:0x00:0x00
             P0P3  [06] [] (PCI) @ 0x00:0x00:0x03:0x00
                 PCI3  [06] []
             SBRG  [06] [] (PCI) @ 0x00:0x00:0x07:0x00
                 PIC   [06] []
                 DMAD  [06] []
                 TMR   [06] [] <attimer1>
                 RTC0  [06] []
                 PS2K  [06] []
                 PS2M  [06] []
                 SPKR  [06] [] <pcppi1>
                 COPR  [06] []
                 RMSC  [06] []
                 UARA  [06] []
                 UARB  [06] []
                 UARC  [06] []
                 UARD  [06] []
                 UARE  [06] []
                 UARF  [06] []
                 UARG  [06] []
                 UARH  [06] []
                 UARI  [06] []
                 LPTA  [06] []
                 OMSC  [06] []
                 EC0   [06] [] <acpiec0>
             SBF1  [06] [] (PCI) @ 0x00:0x00:0x07:0x01
             LAN0  [06] [] (PCI) @ 0x00:0x00:0x08:0x00
             USB1  [06] [] (PCI) @ 0x00:0x00:0x0A:0x00
             EUSB  [06] [] (PCI) @ 0x00:0x00:0x0A:0x01
             IDE0  [06] [] (PCI) @ 0x00:0x00:0x0C:0x00
                 CHN0  [06] []
                     DRV0  [06] []
                     DRV1  [06] []
                 CHN1  [06] []
                     DRV0  [06] []
                     DRV1  [06] []
             HDAC  [06] [] (PCI) @ 0x00:0x00:0x0E:0x00
             AC0   [06] [] <acpiacad0>
             BAT0  [06] []
             BAT1  [06] []
         RMEM  [06] []
         LNKA  [06] []
         LNKB  [06] []
         LNKC  [06] []
         LNKD  [06] []
         LNKE  [06] []
         LNKF  [06] []
         LNKG  [06] []
         LNKH  [06] []
         PWRB  [06] [] <acpibut0>
         SLPB  [06] [] <acpibut1>
         LID   [06] [] <acpilid0>
     _TZ   [06] []
         THRM  [13] [] <acpitz0>

 Regards
 Andrius V

 On Sun, Nov 14, 2021 at 11:35 AM Andrius V <vezhlys@gmail.com> wrote:
 >
 > The following reply was made to PR port-i386/53894; it has been noted by GNATS.
 >
 > From: Andrius V <vezhlys@gmail.com>
 > To: gnats-bugs@netbsd.org
 > Cc: port-i386-maintainer@netbsd.org, gnats-admin@netbsd.org,
 >         netbsd-bugs@netbsd.org, Taylor R Campbell <riastradh@netbsd.org>
 > Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP enabled
 >  on eBox 3352DX3-AP
 > Date: Sun, 14 Nov 2021 11:28:59 +0200
 >
 >  Placed system's acpidump to https://netbsd.org/~andvar/acpidump.txt on
 >  request from Taylor.
 >
 >  On Sun, Jan 20, 2019 at 10:56 PM Andrius V <vezhlys@gmail.com> wrote:
 >  >
 >  > Hi,
 >  >
 >  > Thank you for the prompt reply. I can provide this output but it is a
 >  > bit tricky since USB ports are the only available besides Ethernet and
 >  > VGA (and integrated vte ethernet controller doesn't work properly as
 >  > well unfortunately, so ssh is out of option). Does it matter when to
 >  > do this output? I guess non ACPI/SMP boot won't provide the required
 >  > data as well? Currently I did a cron job which outputs the vmstat -u
 >  > usbhist result to file on boot. Please see an attached report for
 >  > results. Theoretically, I can modify cronjob to post report a bit
 >  > later if needed.
 >  >
 >  > Regards,
 >  > Andrius V
 >  >
 >  > On Sun, Jan 20, 2019 at 3:10 AM David H. Gutteridge <david@gutteridge.ca> wrote:
 >  > >
 >  > > The following reply was made to PR port-i386/53894; it has been noted by GNATS.
 >  > >
 >  > > From: "David H. Gutteridge" <david@gutteridge.ca>
 >  > > To: gnats-bugs@netbsd.org
 >  > > Cc:
 >  > > Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP
 >  > >  enabled on eBox 3352DX3-AP
 >  > > Date: Sat, 19 Jan 2019 20:09:06 -0500
 >  > >
 >  > >  It would probably be helpful to provide USB_DEBUG output. To do so,
 >  > >  please see http://www.netbsd.org/docs/kernel/#usb-debugging,
 >  > >  specifically the part following "USB debugging code has been changed in
 >  > >  NetBSD 7.99 and later."
 >  > >
 >  > >  Dave
 >  > >
 >  > >
 >

From: Andrius V <vezhlys@gmail.com>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-i386/53894: "ehci_sync_hc: timed out" with ACPI/SMP enabled
 on eBox 3352DX3-AP
Date: Tue, 5 Mar 2024 09:57:31 +0200

 On Wed, Mar 16, 2022 at 10:25=E2=80=AFAM Andrius V <vezhlys@gmail.com> wrot=
 e:
 >
 > Hi,
 >
 > I did some investigation on this issue in recent days and here are my fin=
 dings:
 > * I can confirm, that it is interrupts issue, more specifically level
 > type PCI interrupts are not working (thus, USB, network, SCI, probably
 > audio too (untested)) interrupts doesn't work (checked by intrctl and
 > vmstat).
 > * The interrupt establishing flow seems OK though, did some debugging,
 > which shows that the flow doesn't seem to be different from similar
 > system which has no issues and visually everything is "established".
 > * By advice I checked MADT overrides: system reports two overrides,
 > and indeed, it seems that IRQ 0 NetBSD do not override. Unfortunately,
 > I have no idea, where to do that in NetBSD code (FreeBSD seems to go
 > through all MADT table and do overrides, in NetBSD case only SCI
 > interrupt individually comes to override code). Would that fix the
 > issue, honestly, I am not certain, I did try silly hacks, like not
 > masking interrupts by default, which added "i82489_ipi_init: ESR
 > 00000040" to dmesg, but none helped to make PCI interrupts work.
 > acpi0: [MADT] INTR OVERRIDE  : BUS 0, IRQ 0, GSI 2, FLAGS 0x00
 > acpi0: [MADT] INTR OVERRIDE  : BUS 0, IRQ 9, GSI 9, FLAGS 0x0D
 > * SoC does not support MSI.
 > * current kernel doesn't report "ehci_sync_hc: timed out" anymore
 > compared to my original dmesg report, just silently boots without any
 > uhidev devices attached, likely related to recent USB changes.
 > * FreeBSD 12.0 (the last release which boots out of the box on the
 > system), and recent Linux kernels has no problems with these
 > interrupts. OpenBSD have likely the same issue as NetBSD though, since
 > USB and network also doesn't work.
 > * Booting without ACPI/SMP enabled, makes interrupts work (it doesn't
 > go through apic in this case of course).
 >
 > Unfortunately, I am out of ideas for now on next steps, but will keep
 > investigating once I have some, any help would be welcome. Thank you.
 >

 Hi,

 I had few additional attempts to investigate the issue, unfortunately
 without any results which would lead to understanding the issue. I
 would be willing to give up my eBox 3360 with RS-485 serial ports if
 anybody would be interested to work on this.

 From the latest attempts I was still focussing on interrupt
 initialization, which increasingly feels like a wrong place to look
 at. Likely level type interrupts themselves should be investigated,
 but I don't know how to do that effectively.

 I did debug the flow from lapic.c, ioapic.c, pci_machdep.c, intrc.c
 and recently mpacpi.c, compared to my VIA system which is somewhat
 similar. Compared to FreeBSD code, which has differences, but I can't
 pinpoint anything of importance, extINT being the most visible
 difference, but it doesn't seem to affect level type interrupts in any
 way (14 now can boot as well). In general, from side it looks like all
 those attempts show "flawless" initialization, without errors or going
 to erroneous path. System boots without issues as well, however the
 end result is that USB, Ethernet doesn't work since no level type
 interrupts are acknowledged from them (or at least it looks like so).
 ISA interrupts though work, thus serial can be used and system itself
 seems to be stable otherwise.

 System BIOS also has "RDC engineer mode" by clicking F4 while in BIOS.
 It provides few additional options like disabling APIC bus, which
 allows to boot system with ACPI enabled and interrupts working,
 however they are wired through PIC IRQs and second core needs to be
 disabled to boot.

 In recent  attempts I enabled MPDEBUG which gives more log messages
 about configuring the interrupts, also shows that mpacpi takes care of
 both MATD overrides, all logs are expected and matches working
 systems:
 ....
 mpacpi: found root PCI bus 0
 mpacpi: found subordinate bus 1
 mpacpi: found subordinate bus 2
 mpacpi: 3 PCI busses
 mpacpi: ISA interrupt override 0 -> 2 (0/0)
 mpacpi: ISA interrupt override 9 -> 9 (1/3)
 mpacpi: configuring PCI bus 0 int routing
 pciroute: dev 13 INTA on globint 22
 pciroute: done adding entry
 pciroute: dev 1 INTA on globint 16
 pciroute: done adding entry
 pciroute: dev 1 INTB on globint 17
 pciroute: done adding entry
 pciroute: dev 1 INTC on globint 18
 pciroute: done adding entry
 pciroute: dev 1 INTD on globint 19
 pciroute: done adding entry
 pciroute: dev 2 INTA on globint 17
 pciroute: done adding entry
 pciroute: dev 2 INTB on globint 18
 pciroute: done adding entry
 pciroute: dev 2 INTC on globint 19
 pciroute: done adding entry
 pciroute: dev 2 INTD on globint 16
 pciroute: done adding entry
 pciroute: dev 3 INTA on globint 18
 pciroute: done adding entry
 pciroute: dev 3 INTB on globint 19
 pciroute: done adding entry
 pciroute: dev 3 INTC on globint 16
 pciroute: done adding entry
 pciroute: dev 3 INTD on globint 17
 pciroute: done adding entry
 pciroute: dev 8 INTA on globint 19
 pciroute: done adding entry
 pciroute: dev 10 INTA on globint 18
 pciroute: done adding entry
 pciroute: dev 10 INTB on globint 21
 pciroute: done adding entry
 pciroute: dev 12 INTA on globint 23
 pciroute: done adding entry
 pciroute: dev 14 INTA on globint 18
 pciroute: done adding entry
 pciroute: dev 15 INTA on globint 21
 pciroute: done adding entry
 pciroute: done
 mpacpi: configuring PCI bus 1 int routing
 pciroute: dev 0 INTA on globint 16
 pciroute: done adding entry
 pciroute: dev 0 INTB on globint 17
 pciroute: done adding entry
 pciroute: dev 0 INTC on globint 18
 pciroute: done adding entry
 pciroute: dev 0 INTD on globint 19
 pciroute: done adding entry
 pciroute: done
 mpacpi: configuring PCI bus 2 int routing
 pciroute: dev 0 INTA on globint 17
 pciroute: done adding entry
 pciroute: dev 0 INTB on globint 18
 pciroute: done adding entry
 pciroute: dev 0 INTC on globint 19
 pciroute: done adding entry
 pciroute: dev 0 INTD on globint 16
 pciroute: done adding entry
 pciroute: done
 ioapic0: pin 2 attached to isa0 irq 0 (type 0<type=3D0> flags 0<pol=3D0,tri=
 g=3D0>)
 ioapic0: pin 1 attached to isa0 irq 1 (type 0<type=3D0> flags 0<pol=3D0,tri=
 g=3D0>)
 ioapic0: pin 3 attached to isa0 irq 3 (type 0<type=3D0> flags 0<pol=3D0,tri=
 g=3D0>)
 ioapic0: pin 4 attached to isa0 irq 4 (type 0<type=3D0> flags 0<pol=3D0,tri=
 g=3D0>)
 ioapic0: pin 5 attached to isa0 irq 5 (type 0<type=3D0> flags 0<pol=3D0,tri=
 g=3D0>)
 ioapic0: pin 6 attached to isa0 irq 6 (type 0<type=3D0> flags 0<pol=3D0,tri=
 g=3D0>)
 ioapic0: pin 7 attached to isa0 irq 7 (type 0<type=3D0> flags 0<pol=3D0,tri=
 g=3D0>)
 ioapic0: pin 8 attached to isa0 irq 8 (type 0<type=3D0> flags 0<pol=3D0,tri=
 g=3D0>)
 ioapic0: pin 9 attached to isa0 irq 9 (type 0<type=3D0> flags
 0xd<pol=3D0x1=3DAct Hi,trig=3D0x3=3DLevel>)
 ioapic0: pin 10 attached to isa0 irq 10 (type 0<type=3D0> flags 0<pol=3D0,t=
 rig=3D0>)
 ioapic0: pin 11 attached to isa0 irq 11 (type 0<type=3D0> flags 0<pol=3D0,t=
 rig=3D0>)
 ioapic0: pin 12 attached to isa0 irq 12 (type 0<type=3D0> flags 0<pol=3D0,t=
 rig=3D0>)
 ioapic0: pin 13 attached to isa0 irq 13 (type 0<type=3D0> flags 0<pol=3D0,t=
 rig=3D0>)
 ioapic0: pin 14 attached to isa0 irq 14 (type 0<type=3D0> flags 0<pol=3D0,t=
 rig=3D0>)
 ioapic0: pin 15 attached to isa0 irq 15 (type 0<type=3D0> flags 0<pol=3D0,t=
 rig=3D0>)
 ioapic0: pin 22 attached to pci0 device 13 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 16 attached to pci0 device 1 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 17 attached to pci0 device 1 INT_B (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 18 attached to pci0 device 1 INT_C (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 19 attached to pci0 device 1 INT_D (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 17 attached to pci0 device 2 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 18 attached to pci0 device 2 INT_B (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 19 attached to pci0 device 2 INT_C (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 16 attached to pci0 device 2 INT_D (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 18 attached to pci0 device 3 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 19 attached to pci0 device 3 INT_B (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 16 attached to pci0 device 3 INT_C (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 17 attached to pci0 device 3 INT_D (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 19 attached to pci0 device 8 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 18 attached to pci0 device 10 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 21 attached to pci0 device 10 INT_B (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 23 attached to pci0 device 12 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 18 attached to pci0 device 14 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 21 attached to pci0 device 15 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 16 attached to pci1 device 0 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 17 attached to pci1 device 0 INT_B (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 18 attached to pci1 device 0 INT_C (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 19 attached to pci1 device 0 INT_D (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 17 attached to pci2 device 0 INT_A (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 18 attached to pci2 device 0 INT_B (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 19 attached to pci2 device 0 INT_C (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ioapic0: pin 16 attached to pci2 device 0 INT_D (type 0<type=3D0> flags
 0xf<pol=3D0x3=3DAct Lo,trig=3D0x3=3DLevel>)
 ...

 Regards,
 Andrius V

>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.47 2022/09/11 19:34:41 kim Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2024 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.