NetBSD Problem Report #56677

From www@netbsd.org  Sat Jan 29 23:26:13 2022
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id A80311A9239
	for <gnats-bugs@gnats.NetBSD.org>; Sat, 29 Jan 2022 23:26:13 +0000 (UTC)
Message-Id: <20220129232612.9B95E1A923D@mollari.NetBSD.org>
Date: Sat, 29 Jan 2022 23:26:12 +0000 (UTC)
From: oskar@fessel.org
Reply-To: oskar@fessel.org
To: gnats-bugs@NetBSD.org
Subject: radeon firmware cannot load with root on zfs 
X-Send-Pr-Version: www-1.0

>Number:         56677
>Category:       kern
>Synopsis:       radeon firmware cannot load with root on zfs
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Jan 29 23:30:00 +0000 2022
>Last-Modified:  Sun Jan 30 12:10:01 +0000 2022
>Originator:     Jan-hinrich Fessel
>Release:        -current
>Organization:
>Environment:
NetBSD zippo.fessel.org 9.99.93 NetBSD 9.99.93 (ZAPPA) #0: Sat Jan 22 00:01:59 CET 2022  root@:/var/obj/sys/arch/amd64/compile/ZAPPA amd64

>Description:
Using the setup described at https://wiki.netbsd.org/wiki/RootOnZFS/ the kernel panics during load with an ATI FirePro 2270 x1 as the console device and radeon* enabled.

(handwritten transcript)
kern info: [drm] Loading CEDAR Microcode
r600_cp: Failed to load firmware "radeon/CEDAR_pfp.bin"
{drm:netbsd:evergreen_inot+0x374} *ERROR* Failed to load fi...

radeon0: error: Fatal error during GPU init
radeon0: unable to register drm: 2
panic: chosen: no console device

I also built a custom Ramdisk with the firmware on it at /usr/libdata/firmware/radeon, but that did not change anything.

When not booting to the ifs root, the card works, but without acceleration.  
kern info: [drm] ring test on 0 succeeded in 1 usecs
kern error: [drm:(/hurz/src/sys/external/bsd/drm2/dist/drm/radeon/radeon_r600_dma.c:274)r600_dma_ring_test] *ERROR* radeon: ring 3 test failed (0xCAFEDEAD)
radeon0: autoconfiguration error: error: disabling GPU acceleration

>How-To-Repeat:
follow the instructions at https://wiki.netbsd.org/wiki/RootOnZFS/ with an ATI Radeon installed as console device.  Probably with any card that loads microcode from the filesystem.
While at it, find that the system cannot dump core...

>Fix:

>Audit-Trail:
From: matthew green <mrg@eterna.com.au>
To: gnats-bugs@netbsd.org, oskar@fessel.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org
Subject: re: kern/56677: radeon firmware cannot load with root on zfs 
Date: Sun, 30 Jan 2022 12:27:17 +1100

 this is an entirely separate issue.  can you file a new PR?  thanks.

 > When not booting to the ifs root, the card works, but without accelerati=
 on.  =

 > kern info: [drm] ring test on 0 succeeded in 1 usecs
 > kern error: [drm:(/hurz/src/sys/external/bsd/drm2/dist/drm/radeon/radeon=
 _r600_dma.c:274)r600_dma_ring_test] *ERROR* radeon: ring 3 test failed (0x=
 CAFEDEAD)
 > radeon0: autoconfiguration error: error: disabling GPU acceleration

 i see this on a radeon 5450 on a ryzen 5600G system.  fortunately
 the CPU is fast enough that mostly i don't notice/care but i've
 not figured out what is going wrong -- the test is waiting for a
 different value to come from the write, and it never doesn't, but
 i don't know anything about how this ring is supposed to work.

 what is your CPU/mb/etc?


 .mrg.

From: "oskar@fessel.org" <oskar@fessel.org>
To: gnats-bugs@netbsd.org
Cc: kern-bug-people@netbsd.org,
 gnats-admin@netbsd.org,
 netbsd-bugs@netbsd.org
Subject: Re: kern/56677: radeon firmware cannot load with root on zfs 
Date: Sun, 30 Jan 2022 13:05:35 +0100

 --Apple-Mail=_96D0BB09-3598-401E-8DB8-D4DCCD7701E8
 Content-Transfer-Encoding: quoted-printable
 Content-Type: text/plain;
 	charset=utf-8

 Hej,

 > Am 30.01.2022 um 02:30 schrieb matthew green <mrg@eterna.com.au>:
 >=20
 > The following reply was made to PR kern/56677; it has been noted by =
 GNATS.
 >=20
 > From: matthew green <mrg@eterna.com.au>
 > To: gnats-bugs@netbsd.org, oskar@fessel.org
 > Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
 >    netbsd-bugs@netbsd.org
 > Subject: re: kern/56677: radeon firmware cannot load with root on zfs=20=

 > Date: Sun, 30 Jan 2022 12:27:17 +1100
 >=20
 > this is an entirely separate issue.  can you file a new PR?  thanks.

 You mean a new PR for the missing accel?

 >> When not booting to the ifs root, the card works, but without =
 accelerati=3D
 > on.  =3D
 >=20
 >> kern info: [drm] ring test on 0 succeeded in 1 usecs
 >> kern error: =
 [drm:(/hurz/src/sys/external/bsd/drm2/dist/drm/radeon/radeon=3D
 > _r600_dma.c:274)r600_dma_ring_test] *ERROR* radeon: ring 3 test failed =
 (0x=3D
 > CAFEDEAD)
 >> radeon0: autoconfiguration error: error: disabling GPU acceleration
 >=20
 > i see this on a radeon 5450 on a ryzen 5600G system.  fortunately
 > the CPU is fast enough that mostly i don't notice/care but i've
 > not figured out what is going wrong -- the test is waiting for a
 > different value to come from the write, and it never doesn't, but
 > i don't know anything about how this ring is supposed to work.
 >=20
 > what is your CPU/mb/etc?

 This is a HP Z240 with one Xeon E3-1270v6

 should the complete dmesg pasted into the PR?
 And, should there be a separate PR for the 4 detected connectors when =
 there are actually only two (no DP available=E2=80=A6

 Cheers
 	Oskar

 =E2=80=94 excerpt=E2=80=94
 Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, =
 2005,
     2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, =
 2017,
     2018, 2019, 2020, 2021 The NetBSD Foundation, Inc.  All rights =
 reserved.
 Copyright (c) 1982, 1986, 1989, 1991, 1993
     The Regents of the University of California.  All rights reserved.

 NetBSD 9.99.92 (GENERIC) #0: Sun Dec 12 05:08:10 CET 2021
         root@dweezil.lessef.de:/hurz/obj/sys/arch/amd64/compile/GENERIC
 total memory =3D 32687 MB
 avail memory =3D 31648 MB
 timecounter: Timecounters tick every 10.000 msec
 Kernelized RAIDframe activated
 RTC BIOS diagnostic error =
 0xbd<clock_battery,config_unit,memory_size,fixed_disk,invalid_time>
 timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
 efi: systbl at pa bd7cdf18
 mainbus0 (root)
 ACPI: RSDP 0x00000000BDEFE014 000024 (v02 HPQOEM)
 ACPI: XSDT 0x00000000BDEC2188 0000D4 (v01 HPQOEM SLIC-WKS 00000000      =
 01000013)
 ACPI: FACP 0x00000000BDEED000 0000F4 (v05 HPQOEM SLIC-WKS 00000000 HP   =
 00000001)
 ACPI: DSDT 0x00000000BDEC9000 0209E4 (v02 HPQOEM 802F     00000000 INTL =
 20121018)
 ACPI: FACS 0x00000000BDE5A000 000040
 ACPI: SSDT 0x00000000BDEFC000 000108 (v02 HP     ShmTable 00000001 INTL =
 20121018)
 ACPI: TCPA 0x00000000BDEFB000 000032 (v02 HPQOEM EDK2     00000002      =
 01000013)
 ACPI: SSDT 0x00000000BDEFA000 00038A (v02 HPQOEM Tpm2Tabl 00001000 INTL =
 20121018)
 ACPI: TPM2 0x00000000BDEF9000 000034 (v03 HPQOEM EDK2     00000002      =
 01000013)
 ACPI: UEFI 0x00000000BDE6A000 000042 (v01 HPQOEM EDK2     00000002      =
 01000013)
 ACPI: CELI 0x00000000BDEF8000 000028 (v03 HPQOEM EDK2     00000002      =
 01000013)
 ACPI: SSDT 0x00000000BDEF2000 0051FA (v02 SaSsdt SaSsdt   00003000 INTL =
 20121018)
 ACPI: SSDT 0x00000000BDEF1000 0005B1 (v01 Intel  PerfTune 00001000 INTL =
 20121018)
 ACPI: MSDM 0x00000000BDEF0000 000055 (v03 HPQOEM SLIC-WKS 00000000 HP   =
 00000001)
 ACPI: SLIC 0x00000000BDEEF000 000176 (v01 HPQOEM SLIC-WKS 00000001 HP   =
 00000001)
 ACPI: WSMT 0x00000000BDEEE000 000028 (v01 HPQOEM 802F     00000001 HP   =
 00000001)
 ACPI: HPET 0x00000000BDEEC000 000038 (v01 HPQOEM 802F     00000001 HP   =
 00000001)
 ACPI: APIC 0x00000000BDEEB000 0000BC (v01 HPQOEM 802F     00000001 HP   =
 00000001)
 ACPI: MCFG 0x00000000BDEEA000 00003C (v01 HPQOEM 802F     00000001 HP   =
 00000001)
 ACPI: SSDT 0x00000000BDEC8000 00019A (v02 HPQOEM Sata0Ide 00001000 INTL =
 20121018)
 ACPI: SSDT 0x00000000BDEC7000 000729 (v01 HPQOEM PtidDevc 00001000 INTL =
 20121018)
 ACPI: SSDT 0x00000000BDEC6000 000E73 (v02 CpuRef CpuSsdt  00003000 INTL =
 20121018)
 ACPI: DMAR 0x00000000BDEC5000 000070 (v01 INTEL  SKL      00000001 INTL =
 00000001)
 ACPI: ASF! 0x00000000BDEC4000 0000A5 (v32 HPQOEM  UYA     00000001 TFSM =
 000F4240)
 ACPI: FPDT 0x00000000BDEC3000 000044 (v01 HPQOEM EDK2     00000002      =
 01000013)
 ACPI: BGRT 0x00000000BDEFD000 000038 (v01 HPQOEM EDK2     00000002      =
 01000013)
 ACPI: 8 ACPI AML tables successfully acquired and loaded
 ioapic0 at mainbus0 apid 2: pa 0xfec00000, version 0x20, 120 pins
 x2APIC available but disabled by DMAR table
 cpu0 at mainbus0 apid 0
 cpu0: Use lfence to serialize rdtsc
 cpu0: CPU base freq 3800000000 Hz
 cpu0: CPU max freq 4200000000 Hz
 cpu0: TSC freq CPUID 3792000000 Hz
 cpu0: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz, id 0x906e9
 cpu0: node 0, package 0, core 0, smt 0

 [=E2=80=A6]
 root on dk1 dumps on dk2
 root file system type: ffs
 kern.module.path=3D/stand/amd64/9.99.92/modules
 kern info: [drm] initializing kernel modesetting (CEDAR 0x1002:0x68F2 =
 0x1002:0x0126).
 kern info: [drm] register mmio base: 0xd4100000
 kern info: [drm] register mmio size: 131072
 kern info: ATOM BIOS: CEDAR
 radeon0: info: VRAM: 512M 0x0000000000000000 - 0x000000001FFFFFFF (512M =
 used)
 radeon0: info: GTT: 1024M 0x0000000020000000 - 0x000000005FFFFFFF
 kern info: [drm] Detected VRAM RAM=3D200M, BAR=3D256M
 kern info: [drm] RAM width 64bits DDR
 Zone  kernel: Available graphics memory: 11081336 kiB
 Zone   dma32: Available graphics memory: 2097152 kiB
 kern info: [drm] radeon: 512M of VRAM memory ready
 kern info: [drm] radeon: 1024M of GTT memory ready.
 kern info: [drm] Loading CEDAR Microcode
 kern info: [drm] Internal thermal controller with fan control
 kern info: [drm] radeon: dpm initialized
 kern info: [drm] GART: num cpu pages 262144, num gpu pages 262144
 kern info: [drm] PCIE GART of 1024M enabled (table at =
 0x000000000025E000).
 radeon0: info: WB enabled
 radeon0: info: fence driver on ring 0 use gpu addr 0x0000000020000c00 =
 and cpu addr 0x0xffffb5d400bf9c00
 radeon0: info: fence driver on ring 3 use gpu addr 0x0000000020000c0c =
 and cpu addr 0x0xffffb5d400bf9c0c
 radeon0: info: fence driver on ring 5 use gpu addr 0x000000000005c418 =
 and cpu addr 0x0xffff90827295c418
 kern info: [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
 kern info: [drm] Driver supports precise vblank timestamp query.
 radeon0: info: radeon: MSI limited to 32-bit
 radeon0: info: radeon: using MSI.
 allocated pic msi7 type edge pin 0 level 6 to cpu1 slot 2 idt entry 98
 radeon0: interrupting at msi7 vec 0 (radeon0)
 kern info: [drm] radeon: irq initialized.
 kern info: [drm] ring test on 0 succeeded in 1 usecs
 kern error: =
 [drm:(/hurz/src/sys/external/bsd/drm2/dist/drm/radeon/radeon_r600_dma.c:27=
 4)r600_dma_ring_test] *ERROR* radeon: ring 3 test failed (0xCAFEDEAD)
 radeon0: autoconfiguration error: error: disabling GPU acceleration
 kern info: [drm] Radeon Display Connectors
 kern info: [drm] Connector 0:
 kern info: [drm]   DP-1
 kern info: [drm]   HPD3
 kern info: [drm]   DDC: 0x6460 0x6460 0x6464 0x6464 0x6468 0x6468 0x646c =
 0x646c
 kern info: [drm]   Encoders:
 kern info: [drm]     DFP1: INTERNAL_UNIPHY1
 kern info: [drm] Connector 1:
 kern info: [drm]   DP-2
 kern info: [drm]   HPD1
 kern info: [drm]   DDC: 0x6450 0x6450 0x6454 0x6454 0x6458 0x6458 0x645c =
 0x645c
 kern info: [drm]   Encoders:
 kern info: [drm]     DFP2: INTERNAL_UNIPHY
 kern info: [drm] Connector 2:
 kern info: [drm]   DVI-I-1
 kern info: [drm]   HPD2
 kern info: [drm]   DDC: 0x6440 0x6440 0x6444 0x6444 0x6448 0x6448 0x644c =
 0x644c
 kern info: [drm]   Encoders:
 kern info: [drm]     DFP3: INTERNAL_UNIPHY1
 kern info: [drm]     CRT2: INTERNAL_KLDSCP_DAC2
 kern info: [drm] Connector 3:
 kern info: [drm]   DVI-I-2
 kern info: [drm]   HPD4
 kern info: [drm]   DDC: 0x6430 0x6430 0x6434 0x6434 0x6438 0x6438 0x643c =
 0x643c
 kern info: [drm]   Encoders:
 kern info: [drm]     DFP4: INTERNAL_UNIPHY
 kern info: [drm]     CRT1: INTERNAL_KLDSCP_DAC1
 radeondrmkmsfb0 at radeon0
 radeondrmkmsfb0: framebuffer at 0xc045f000, size 1680x1050, depth 32, =
 stride 6912
 warning: =
 /hurz/src/sys/external/bsd/drm2/dist/drm/radeon/radeon_evergreen.c:4620: =
 Can't enable IRQ/MSI because no handler is installed
 wsdisplay0 at radeondrmkmsfb0 kbdmux 1: console (default, vt100 =
 emulation), using wskbd0


 --Apple-Mail=_96D0BB09-3598-401E-8DB8-D4DCCD7701E8
 Content-Disposition: attachment;
 	filename=smime.p7s
 Content-Type: application/pkcs7-signature;
 	name=smime.p7s
 Content-Transfer-Encoding: base64

 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCBXow
 ggV2MIIDXqADAgECAgMUkGwwDQYJKoZIhvcNAQENBQAweTEQMA4GA1UEChMHUm9vdCBDQTEeMBwG
 A1UECxMVaHR0cDovL3d3dy5jYWNlcnQub3JnMSIwIAYDVQQDExlDQSBDZXJ0IFNpZ25pbmcgQXV0
 aG9yaXR5MSEwHwYJKoZIhvcNAQkBFhJzdXBwb3J0QGNhY2VydC5vcmcwHhcNMjAwNTA1MDU1NjU5
 WhcNMjIwNTA1MDU1NjU5WjA+MRswGQYDVQQDExJKYW4tSGlucmljaCBGZXNzZWwxHzAdBgkqhkiG
 9w0BCQEWEG9za2FyQGZlc3NlbC5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDC
 AsGycihffFv75WcgutRotMOS2qBn+bRjY45BumPMYvguY4OZb8TPxB/BbnJc7SQSlKI2zrPsoAE4
 bnFt7FzPZ3V+9LB+Sa6kDRLWIlBJwpk+ez5JTHFgCoxM4FnTxKoOcCqG2i+ZjAMys+JfVxgrYewR
 3olluQs0YeyMwKkbLt7GVnymrUYcGYEKfNUujI74j8B47oE3f5JFq1iPe31B0Zk6FMjHqQE9LlIU
 d5KRoL1QP5lnSDItpDcxpzxcIG88ed373vgbaftXRv3bzKn+g1fE4Df5fYVsFQSbnj4I6fH5S0Xw
 FPNiY6vOt4mL9mP14g3FrN84U+jvrRAMIsF1AgMBAAGjggFAMIIBPDAMBgNVHRMBAf8EAjAAMFYG
 CWCGSAGG+EIBDQRJFkdUbyBnZXQgeW91ciBvd24gY2VydGlmaWNhdGUgZm9yIEZSRUUgaGVhZCBv
 dmVyIHRvIGh0dHA6Ly93d3cuQ0FjZXJ0Lm9yZzAOBgNVHQ8BAf8EBAMCA6gwQAYDVR0lBDkwNwYI
 KwYBBQUHAwQGCCsGAQUFBwMCBgorBgEEAYI3CgMEBgorBgEEAYI3CgMDBglghkgBhvhCBAEwMgYI
 KwYBBQUHAQEEJjAkMCIGCCsGAQUFBzABhhZodHRwOi8vb2NzcC5jYWNlcnQub3JnMDEGA1UdHwQq
 MCgwJqAkoCKGIGh0dHA6Ly9jcmwuY2FjZXJ0Lm9yZy9yZXZva2UuY3JsMBsGA1UdEQQUMBKBEG9z
 a2FyQGZlc3NlbC5vcmcwDQYJKoZIhvcNAQENBQADggIBAHCwheRsL4bH3Un8jOU401K1YKSBsHTQ
 urj7GfFQzuk1+lUZzUWJPkrGCruiGrRysgCaJ2kOsp/N0xcBx8GsQLgDuUCsUXoJQOo10FnlO51Z
 AoEDS+5pNVWcr6GSPXoWPnZsUgiA24gvI5gSuqW31fWfd4O/L062J/Xs5itpBGuT9oHamywK+8RH
 onzXpspnlyAD1ZtQsXFYpk6rQXWSs9prg7ms7rMAUHE1P+LvRjaKwOIdbAHJSgvbU/JE7whcT0Mg
 AV19IxEAubKK8FCvRAs3RtCQBm7Njs+F1FmRq7UHEKKAgduJGCprc30KuK1f7PX2LXJ01qQmJYuz
 DkDuIydJbFLZEMsx70wtjdDdSyqZYkrumkU+S7mod3scUzkcY8QWWsgQHDLlzRU9OnhxS+iYvUBX
 r31Tg/NkciAqWl84Pc39M+zwgsxSR6qQbcgAv/XQQec2QQzoJM0QWTkkkTC1zMr0ZFGVZsEZcFqG
 LdOxym/4TQoAbFkPD7yjkFts6VGxIj+zxzLDlJxJVqF5ng3ZncdChkxX5DZXjCQUF9aCtFg4tMLL
 Jnv670p4WPqkaJk1ZnnkfLOFpbZs0mBnpHWJtCGyRtSwfL4FZcoFzQ4r7rIwuTQcwoFPCHr07U7B
 Kh7nHYESks/beQdSAFjpF5ywHxd6fTCJ4TdH8eJxhAx2MYIDQzCCAz8CAQEwgYAweTEQMA4GA1UE
 ChMHUm9vdCBDQTEeMBwGA1UECxMVaHR0cDovL3d3dy5jYWNlcnQub3JnMSIwIAYDVQQDExlDQSBD
 ZXJ0IFNpZ25pbmcgQXV0aG9yaXR5MSEwHwYJKoZIhvcNAQkBFhJzdXBwb3J0QGNhY2VydC5vcmcC
 AxSQbDANBglghkgBZQMEAgEFAKCCAZMwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG
 9w0BCQUxDxcNMjIwMTMwMTIwNTM1WjAvBgkqhkiG9w0BCQQxIgQgYjaJwPHTwiSlTxIZpMm2g4xw
 iwYxD5EI9s9Nv3tt3EowgZEGCSsGAQQBgjcQBDGBgzCBgDB5MRAwDgYDVQQKEwdSb290IENBMR4w
 HAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNBIENlcnQgU2lnbmluZyBB
 dXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2FjZXJ0Lm9yZwIDFJBsMIGTBgsqhkiG
 9w0BCRACCzGBg6CBgDB5MRAwDgYDVQQKEwdSb290IENBMR4wHAYDVQQLExVodHRwOi8vd3d3LmNh
 Y2VydC5vcmcxIjAgBgNVBAMTGUNBIENlcnQgU2lnbmluZyBBdXRob3JpdHkxITAfBgkqhkiG9w0B
 CQEWEnN1cHBvcnRAY2FjZXJ0Lm9yZwIDFJBsMA0GCSqGSIb3DQEBAQUABIIBAFSOS6AEnhsnbRWI
 tc/2eDyE1tKYhz8fZkYbnQk8fJgGA9RAuHwbeHbukDq4f/k073DKAiFMyuflpuKldqS5e4JBh2AT
 KKir/lwfXV77nP4UUxoqY9S6+Lggfuh7ShMLhiZxzVyFVTdFW0WS6AXmpUh5HiaGjRI65ue9u1/e
 Nt7OrY1k/hhxasMgKpENWDUyH70+sa0Zz0jum/TSEoXtijG/vmzoMm/rVP2AV1lfkqUs4ruAmQZo
 u/re14pCIDvLvFT2CKDy3m7ta/O8zD2eQJYSOefmXvzODTvE1YXrXnP4xgb2HWvV8qI9Cu88TGbW
 bY/QiY0+tyoxTk9EaiEMyXYAAAAAAAA=
 --Apple-Mail=_96D0BB09-3598-401E-8DB8-D4DCCD7701E8--

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.46 2020/01/03 16:35:01 leot Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2020 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.