NetBSD Problem Report #57104

From www@netbsd.org  Sat Dec 10 16:27:00 2022
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 2AE361A921F
	for <gnats-bugs@gnats.NetBSD.org>; Sat, 10 Dec 2022 16:27:00 +0000 (UTC)
Message-Id: <20221210162658.5533F1A923A@mollari.NetBSD.org>
Date: Sat, 10 Dec 2022 16:26:58 +0000 (UTC)
From: cryintothebluesky@gmail.com
Reply-To: cryintothebluesky@gmail.com
To: gnats-bugs@NetBSD.org
Subject: "intrctl affinity" does not function on Raspberry Pi 4 and other ARM boards
X-Send-Pr-Version: www-1.0

>Number:         57104
>Category:       port-arm
>Synopsis:       "intrctl affinity" does not function on Raspberry Pi 4 and other ARM boards
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    port-arm-maintainer
>State:          feedback
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Dec 10 16:30:00 +0000 2022
>Closed-Date:    
>Last-Modified:  Sun Sep 15 18:30:02 +0000 2024
>Originator:     Sad Clouds
>Release:        9.99.108
>Organization:
>Environment:
NetBSD rp4 9.99.108 NetBSD 9.99.108 (GENERIC64)
>Description:
Both xhci0 and genet0 route their interrupts to CPU0. With high NFS I/O load, CPU0 gets saturated. Would be good to spread interrupts across different CPUs.

rp4# intrctl list
interrupt id       CPU0     CPU1     CPU2     CPU3  device name(s)
armgic irq 0     105940* 2312647* 2122811* 1903849* IPI ast
armgic irq 1        736*     526*     941*    1013* IPI xcall
armgic irq 2          0*       0*       0*       0* IPI nop
armgic irq 3          0*       0*       0*       0* IPI shootdown
armgic irq 4          0*       0*       0*       0* IPI ddb
armgic irq 5         91*      40*      28*      34* IPI generic
armgic irq 27    352801*  352791*  352791*  352791* gtmr0
armgic irq 48         0*       0        0        0  pmu cpu0
armgic irq 49         0        0*       0        0  pmu cpu1
armgic irq 50         0        0        0*       0  pmu cpu2
armgic irq 51         0        0        0        0* pmu cpu3
armgic irq 125        0*       0        0        0  com0
armgic irq 149        0*       0        0        0  bsciic0
armgic irq 153        0*       0        0        0  plcom0
armgic irq 158      579*       0        0        0  sdhc0
armgic irq 175  4530134*       0        0        0  xhci0
armgic irq 189 12573670*       0        0        0  genet0

rp4# intrctl affinity -c 1 -i "armgic irq 189"
intrctl: sysctl kern.intr.affinity: Invalid argument

>How-To-Repeat:

>Fix:

>Release-Note:

>Audit-Trail:
From: Brad Spencer <brad@anduin.eldar.org>
To: gnats-bugs@netbsd.org
Cc: port-arm-maintainer@netbsd.org, gnats-admin@netbsd.org,
        netbsd-bugs@netbsd.org, cryintothebluesky@gmail.com
Subject: Re: port-arm/57104: "intrctl affinity" does not function on
 Raspberry Pi 4 and other ARM boards
Date: Sat, 10 Dec 2022 11:38:20 -0500

 cryintothebluesky@gmail.com writes:

 >>Number:         57104
 >>Category:       port-arm
 >>Synopsis:       "intrctl affinity" does not function on Raspberry Pi 4 and other ARM boards

 [snip]

 > rp4# intrctl affinity -c 1 -i "armgic irq 189"
 > intrctl: sysctl kern.intr.affinity: Invalid argument
 >
 >>How-To-Repeat:
 >
 >>Fix:


 I think that this may be a hardware limitation of the RPI SOC.  If I
 remember correctly the RPI can only deliver hard interrupts to CPU0.




 -- 
 Brad Spencer - brad@anduin.eldar.org - KC8VKS - http://anduin.eldar.org

From: Sad Clouds <cryintothebluesky@gmail.com>
To: Brad Spencer <brad@anduin.eldar.org>
Cc: gnats-bugs@netbsd.org, port-arm-maintainer@netbsd.org,
 gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: port-arm/57104: "intrctl affinity" does not function on
 Raspberry Pi 4 and other ARM boards
Date: Sat, 10 Dec 2022 17:29:48 +0000

 On Sat, 10 Dec 2022 11:38:20 -0500
 Brad Spencer <brad@anduin.eldar.org> wrote:

 > I think that this may be a hardware limitation of the RPI SOC.  If I
 > remember correctly the RPI can only deliver hard interrupts to CPU0.

 This post suggest that at least on Linux, interrupt affinity can be set
 for Raspberry Pi 4. You cannot pin one interrupt to multiple CPUs, but
 you can pin it to any one CPU, as long as it is one CPU per interrupt.

 https://tomcore.io/docs/articles/RaspberryPI/raspberry-pi-4b-network-performance/

State-Changed-From-To: open->feedback
State-Changed-By: jmcneill@NetBSD.org
State-Changed-When: Wed, 14 Dec 2022 11:09:35 +0000
State-Changed-Why:


From: Jared McNeill <jmcneill@invisible.ca>
To: gnats-bugs@netbsd.org
Cc: port-arm-maintainer@netbsd.org, gnats-admin@netbsd.org, 
    netbsd-bugs@netbsd.org
Subject: Re: port-arm/57104: "intrctl affinity" does not function on Raspberry
 Pi 4 and other ARM boards
Date: Wed, 14 Dec 2022 07:07:48 -0400 (AST)

 On Sat, 10 Dec 2022, cryintothebluesky@gmail.com wrote:

 > rp4# intrctl affinity -c 1 -i "armgic irq 189"
 > intrctl: sysctl kern.intr.affinity: Invalid argument

 Setting affinity is only allowed for interrupt handlers marked MPSAFE 
 (see sys/arch/arm/pic/pic.c interrupt_distribute()). This should work for 
 genet in a kernel built with options NET_MPSAFE.

 Take care,
 Jared

From: Nick Hudson <nick.hudson@gmx.co.uk>
To: gnats-bugs@netbsd.org, port-arm-maintainer@netbsd.org,
 gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Cc: 
Subject: Re: port-arm/57104: "intrctl affinity" does not function on Raspberry
 Pi 4 and other ARM boards
Date: Sun, 15 Sep 2024 08:44:46 +0100

 On 10/12/2022 16:30, cryintothebluesky@gmail.com wrote:

 >> Synopsis:       "intrctl affinity" does not function on Raspberry Pi 4 and other ARM boards

 This should be possible after my changes

 	sys/dev/ic/bcmgenet.c:1.20
 	sys/dev/ic/bcmgenetvar.h:1.5

 Nick

From: Nick Hudson <nick.hudson@gmx.co.uk>
To: gnats-bugs@netbsd.org, port-arm-maintainer@netbsd.org,
 gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Cc: 
Subject: Re: port-arm/57104: "intrctl affinity" does not function on Raspberry
 Pi 4 and other ARM boards
Date: Sun, 15 Sep 2024 09:32:17 +0100

 This should be possible after my changes

 	sys/dev/ic/bcmgenet.c:1.20
 	sys/dev/ic/bcmgenetvar.h:1.5
 	sys/dev/acpi/genet_acpi.c:1.6
 	sys/dev/fdt/genet_fdt.c:1.7

 Nick

From: Sad Clouds <cryintothebluesky@gmail.com>
To: gnats-bugs@netbsd.org
Cc: " Nick Hudson  via gnats" <gnats-admin@NetBSD.org>,
 port-arm-maintainer@netbsd.org, gnats-admin@netbsd.org,
 netbsd-bugs@netbsd.org
Subject: Re: port-arm/57104: "intrctl affinity" does not function on
 Raspberry Pi 4 and other ARM boards
Date: Sun, 15 Sep 2024 19:04:48 +0100

 Hi Nick, thanks for doing this, I can try it out in the near future, as
 I'm currently evaluating multiple OSes on RPi3 and RPi4. A few
 questions about your changes:

 1. Is this only available in NetBSD-current?
 2. Do I need to enable NET_MPSAFE kernel option for this to work, or is
 this the default on NetBSD-current?

From: Nick Hudson <nick.hudson@gmx.co.uk>
To: Sad Clouds <cryintothebluesky@gmail.com>, gnats-bugs@netbsd.org
Cc: Nick Hudson via gnats <gnats-admin@NetBSD.org>,
 port-arm-maintainer@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: port-arm/57104: "intrctl affinity" does not function on Raspberry
 Pi 4 and other ARM boards
Date: Sun, 15 Sep 2024 19:29:38 +0100

 On 15/09/2024 19:04, Sad Clouds wrote:
 > Hi Nick, thanks for doing this, I can try it out in the near future, as
 > I'm currently evaluating multiple OSes on RPi3 and RPi4. A few
 > questions about your changes:
 >
 > 1. Is this only available in NetBSD-current?
 atm, yes. -current only.

 > 2. Do I need to enable NET_MPSAFE kernel option for this to work, or is
 > this the default on NetBSD-current?

 NET_MPSAFE is no longer required.

 Nick

>Unformatted:

Home	PR Database Search