NetBSD Problem Report #60457

From www@netbsd.org  Fri Jul 17 19:54:28 2026
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "mail.netbsd.org", Issuer "YR2" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 1D96B1A923B
	for <gnats-bugs@gnats.NetBSD.org>; Fri, 17 Jul 2026 19:54:28 +0000 (UTC)
Message-Id: <20260717195426.A0B1C1A923C@mollari.NetBSD.org>
Date: Fri, 17 Jul 2026 19:54:26 +0000 (UTC)
From: riz@tastylime.net
Reply-To: riz@tastylime.net
To: gnats-bugs@NetBSD.org
Subject: Hang during shutdown/reboot with lagg(4)
X-Send-Pr-Version: www-1.0
X-From4GNATS: "riz@tastylime.net via gnats" <gnats-admin@NetBSD.org>

>Number:         60457
>Category:       kern
>Synopsis:       Hang during shutdown/reboot with lagg(4) interface
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    yamaguchi
>State:          analyzed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Jul 17 19:55:00 +0000 2026
>Closed-Date:    
>Last-Modified:  Sun Jul 19 15:43:26 +0000 2026
>Originator:     Jeff Rizzo
>Release:        11.0_RC6
>Organization:
>Environment:
NetBSD cherimoya.tastylime.net 11.0_RC6 NetBSD 11.0_RC6 (GENERIC) #6: Fri Jul 17 18:46:30 UTC 2026  riz@acai.tastylime.net:/tank/netbsd/netbsd-11/obj/sys/arch/amd64/compile/GENERIC amd64
>Description:
This machine consistently gets stuck sometime _after_ hw.cnmagic is effective (I can't get into ddb) and _before_ everything is undone for shutdown, and it will indefinitely (for at least an hour, anyway) hang:


Jul 17 18:56:59 [ 156.7362082] syncing disks... done
[ 214.3661561] ipmi0: normal state on 'FAN6_OUTLET'
[ 214.4161559] dk11 at sd2 (c3440c02-2fec-11f1-8000-000000000000) deleted
[ 214.4261558] dk10 at sd2 (c3440c01-2fec-11f1-8000-000000000000) deleted
[ 214.4261558] sd2: detached
[ 214.4361555] sd1: detached
[ 214.4361555] dk1 at sd0 (NBImgRoot) deleted
[ 214.4361555] dk0 at sd0 (EFI system) deleted
[ 214.4461557] sd0: detached
[ 214.4461557] scsibus2: detached
[ 214.4461557] scsibus1: detached
[ 214.4561557] scsibus0: detached
[ 214.4561557] dk7 at wd2 (zfs-rpool-wd2) deleted
[ 214.4661550] dk6 at wd2 (rf-component-wd2) deleted
[ 214.4661550] wd2: detached
[ 214.4661550] uhub5: detached
[ 214.4761557] uhub4: detached
[ 214.4761557] uhub3: detached
[ 214.4761557] uhub2: detached
[ 214.4861557] uhub0: detached
[ 214.4861557] igphy1: detached
lagg0: [ 214.4861557] disable distributing on wm1
[ 244.3661277] ipmi0: normal state on 'FAN6_INLET'
[ 244.3761278] ipmi0: normal state on 'FAN5_OUTLET'
[ 244.3761278] ipmi0: normal state on 'FAN5_INLET'
[ 1354.3650987] ipmi0: normal state on 'FAN4_OUTLET'
[ 1354.3750988] ipmi0: normal state on 'FAN3_OUTLET'
[ 1894.3645934] ipmi0: critical under limit on 'FAN4_OUTLET'
[ 1894.3745935] ipmi0: critical under limit on 'FAN3_OUTLET'

...the messages from ipmi0 continue, but no more disks detach.

I have other machines with zfs root which don't exhibit this problem, but this machine currently has a failed component (and working systems don't have failed components):

	NAME                      STATE     READ WRITE CKSUM
	rpool                     DEGRADED     0     0     0
	  raidz2-0                DEGRADED     0     0     0
	    dk3                   ONLINE       0     0     0
	    dk5                   ONLINE       0     0     0
	    13185259948679916708  OFFLINE      0     0     0  was /dev/dk7
	    dk9                   ONLINE       0     0     0

Here's what disks look like after a boot:

[     5.387244] dk0 at sd0: "EFI system", 262144 blocks at 2048, type: msdos
[     5.403522] dk1 at sd0: "NBImgRoot", 976502784 blocks at 264192, type: ffs
[     5.696224] dk2 at wd0: "rf-component-wd0", 4194304 blocks at 128, type: raidframe
[     5.713521] dk3 at wd0: "zfs-rpool-wd0", 31247564632 blocks at 4194432, type: zfs
[     5.835644] dk4 at wd1: "rf-component-wd1", 4194304 blocks at 128, type: raidframe
[     5.844158] dk5 at wd1: "zfs-rpool-wd1", 31247564632 blocks at 4194432, type: zfs
[     5.963521] dk6 at wd2: "rf-component-wd2", 4194304 blocks at 128, type: raidframe
[     5.973521] dk7 at wd2: "zfs-rpool-wd2", 31247564632 blocks at 4194432, type: zfs
[     6.093521] dk8 at wd3: "cherimoya-swap-wd3", 4194304 blocks at 128, type: swap
[     6.103521] dk9 at wd3: "zfs-rpool-wd3", 31247564632 blocks at 4194432, type: zfs
[     7.627989] dk10 at sd2: "c3440c01-2fec-11f1-8000-000000000000", 262144 blocks at 2048, type: msdos
[     7.653664] dk11 at sd2: "c3440c02-2fec-11f1-8000-000000000000", 4853760 blocks at 264192, type: ffs
[     8.203517] raid0: Components: /dev/dk2 /dev/dk4

(wd2 has a ton of errors and is currently awaiting replacement)

I'd love suggestions on more troubleshooting steps - I can't get into DDB at the point of hang; I could sprinkle debugging printfs but so
far they've just told me that x86_reset() isn't reached...
>How-To-Repeat:
boot 11.0_RC6 with zfs root as described in the netbsd wiki (pivot root)

shutdown -r now

get as far as detaching some-but-not-all dk(4) devices

wait indefinitely for IPMI power reset to finish reboot
>Fix:
none given

>Release-Note:

>Audit-Trail:
From: Andrius V <vezhlys@gmail.com>
To: gnats-bugs@netbsd.org
Cc: riz@tastylime.net, yamaguchi@netbsd.org
Subject: Re: kern/60457: Hang during shutdown/reboot (with zfs root with
 failed component)
Date: Sat, 18 Jul 2026 14:18:21 +0300

 On Fri, Jul 17, 2026 at 10:55=E2=80=AFPM riz@tastylime.net via gnats
 <gnats-admin@netbsd.org> wrote:
 >
 > >Number:         60457
 > >Category:       kern
 > >Synopsis:       Hang during shutdown/reboot (with zfs root with failed c=
 omponent)
 > >Confidential:   no
 > >Severity:       serious
 > >Priority:       high
 > >Responsible:    kern-bug-people
 > >State:          open
 > >Class:          sw-bug
 > >Submitter-Id:   net
 > >Arrival-Date:   Fri Jul 17 19:55:00 +0000 2026
 > >Originator:     Jeff Rizzo
 > >Release:        11.0_RC6
 > >Organization:
 > >Environment:
 > NetBSD cherimoya.tastylime.net 11.0_RC6 NetBSD 11.0_RC6 (GENERIC) #6: Fri=
  Jul 17 18:46:30 UTC 2026  riz@acai.tastylime.net:/tank/netbsd/netbsd-11/ob=
 j/sys/arch/amd64/compile/GENERIC amd64
 > >Description:
 > This machine consistently gets stuck sometime _after_ hw.cnmagic is effec=
 tive (I can't get into ddb) and _before_ everything is undone for shutdown,=
  and it will indefinitely (for at least an hour, anyway) hang:
 >
 >
 > Jul 17 18:56:59 [ 156.7362082] syncing disks... done
 > [ 214.3661561] ipmi0: normal state on 'FAN6_OUTLET'
 > [ 214.4161559] dk11 at sd2 (c3440c02-2fec-11f1-8000-000000000000) deleted
 > [ 214.4261558] dk10 at sd2 (c3440c01-2fec-11f1-8000-000000000000) deleted
 > [ 214.4261558] sd2: detached
 > [ 214.4361555] sd1: detached
 > [ 214.4361555] dk1 at sd0 (NBImgRoot) deleted
 > [ 214.4361555] dk0 at sd0 (EFI system) deleted
 > [ 214.4461557] sd0: detached
 > [ 214.4461557] scsibus2: detached
 > [ 214.4461557] scsibus1: detached
 > [ 214.4561557] scsibus0: detached
 > [ 214.4561557] dk7 at wd2 (zfs-rpool-wd2) deleted
 > [ 214.4661550] dk6 at wd2 (rf-component-wd2) deleted
 > [ 214.4661550] wd2: detached
 > [ 214.4661550] uhub5: detached
 > [ 214.4761557] uhub4: detached
 > [ 214.4761557] uhub3: detached
 > [ 214.4761557] uhub2: detached
 > [ 214.4861557] uhub0: detached
 > [ 214.4861557] igphy1: detached
 > lagg0: [ 214.4861557] disable distributing on wm1
 > [ 244.3661277] ipmi0: normal state on 'FAN6_INLET'
 > [ 244.3761278] ipmi0: normal state on 'FAN5_OUTLET'
 > [ 244.3761278] ipmi0: normal state on 'FAN5_INLET'
 > [ 1354.3650987] ipmi0: normal state on 'FAN4_OUTLET'
 > [ 1354.3750988] ipmi0: normal state on 'FAN3_OUTLET'
 > [ 1894.3645934] ipmi0: critical under limit on 'FAN4_OUTLET'
 > [ 1894.3745935] ipmi0: critical under limit on 'FAN3_OUTLET'
 >
 > ...the messages from ipmi0 continue, but no more disks detach.
 >
 > I have other machines with zfs root which don't exhibit this problem, but=
  this machine currently has a failed component (and working systems don't h=
 ave failed components):
 >
 >         NAME                      STATE     READ WRITE CKSUM
 >         rpool                     DEGRADED     0     0     0
 >           raidz2-0                DEGRADED     0     0     0
 >             dk3                   ONLINE       0     0     0
 >             dk5                   ONLINE       0     0     0
 >             13185259948679916708  OFFLINE      0     0     0  was /dev/dk=
 7
 >             dk9                   ONLINE       0     0     0
 >
 > Here's what disks look like after a boot:
 >
 > [     5.387244] dk0 at sd0: "EFI system", 262144 blocks at 2048, type: ms=
 dos
 > [     5.403522] dk1 at sd0: "NBImgRoot", 976502784 blocks at 264192, type=
 : ffs
 > [     5.696224] dk2 at wd0: "rf-component-wd0", 4194304 blocks at 128, ty=
 pe: raidframe
 > [     5.713521] dk3 at wd0: "zfs-rpool-wd0", 31247564632 blocks at 419443=
 2, type: zfs
 > [     5.835644] dk4 at wd1: "rf-component-wd1", 4194304 blocks at 128, ty=
 pe: raidframe
 > [     5.844158] dk5 at wd1: "zfs-rpool-wd1", 31247564632 blocks at 419443=
 2, type: zfs
 > [     5.963521] dk6 at wd2: "rf-component-wd2", 4194304 blocks at 128, ty=
 pe: raidframe
 > [     5.973521] dk7 at wd2: "zfs-rpool-wd2", 31247564632 blocks at 419443=
 2, type: zfs
 > [     6.093521] dk8 at wd3: "cherimoya-swap-wd3", 4194304 blocks at 128, =
 type: swap
 > [     6.103521] dk9 at wd3: "zfs-rpool-wd3", 31247564632 blocks at 419443=
 2, type: zfs
 > [     7.627989] dk10 at sd2: "c3440c01-2fec-11f1-8000-000000000000", 2621=
 44 blocks at 2048, type: msdos
 > [     7.653664] dk11 at sd2: "c3440c02-2fec-11f1-8000-000000000000", 4853=
 760 blocks at 264192, type: ffs
 > [     8.203517] raid0: Components: /dev/dk2 /dev/dk4
 >
 > (wd2 has a ton of errors and is currently awaiting replacement)
 >
 > I'd love suggestions on more troubleshooting steps - I can't get into DDB=
  at the point of hang; I could sprinkle debugging printfs but so
 > far they've just told me that x86_reset() isn't reached...
 > >How-To-Repeat:
 > boot 11.0_RC6 with zfs root as described in the netbsd wiki (pivot root)
 >
 > shutdown -r now
 >
 > get as far as detaching some-but-not-all dk(4) devices
 >
 > wait indefinitely for IPMI power reset to finish reboot
 > >Fix:
 > none given
 >

 Hi,

 Since, I see lagg(4) in the logs, I think it may cause hang, not zfs.
 I have a similar problem on my NAS system. If it is not being brought
 down first, shutdown or reboot hangs, I've just never filed a bug
 report for that.
 If you have this option, try to bring down lagg with ifconfig lagg0
 down and reboot.

 Regards,
 Andrius V

From: Jeff Rizzo <riz@tastylime.net>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/60457: Hang during shutdown/reboot (with zfs root with
 failed component)
Date: Sat, 18 Jul 2026 14:49:49 -0700

 On 7/18/26 04:20, Andrius V via gnats wrote:
 >   If you have this option, try to bring down lagg with ifconfig lagg0
 >   down and reboot.


 I can confirm that `ifconfig lagg0 down` on console does, in fact, allow 
 the shutdown to finish.  I'll edit the PR subject (once I figure out how 
 to) and look for other related PRs.


Responsible-Changed-From-To: kern-bug-people->yamaguchi
Responsible-Changed-By: riz@NetBSD.org
Responsible-Changed-When: Sat, 18 Jul 2026 22:00:12 +0000
Responsible-Changed-Why:
lagg(4) is the underlying culprit


From: Jeff Rizzo <riz@tastylime.net>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/60457 (Hang during shutdown/reboot with lagg(4) interface)
Date: Sat, 18 Jul 2026 17:06:52 -0700

 I was hoping to be able to replicate this in QEMU, but I haven't been 
 able to.  I **have** replicated it on another NetBSD-11 host .

 I'm going to upgrade to a -current kernel on that host and see if I can 
 get into DDB to get more info.


From: Jeff Rizzo <riz@tastylime.net>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/60457 (Hang during shutdown/reboot with lagg(4) interface)
Date: Sat, 18 Jul 2026 18:03:24 -0700

 I managed to get into DDB with a LOCKDEBUG kernel:

 db{9}> ps /l
 PID     LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 2982   2982 3   1         0   ffff81230a576400             reboot lagg0.lacp
 1         1 3   9   8020180   ffff812324b78400               init wait
 0      2106 3   6       200   ffff812309b63400         poolthread pooljob
 0      1658 3   6       200   ffff8123082b6400       bridge_rtage 
 bridge_rtage
 0      1643 3   6       200   ffff8123082b6800         lagg0.lacp mutex
 0      1110 3   0       200   ffff81230b289c00              ipmi0 ipmi0
 0      1561 3   9       200   ffff812309176000     poolthread@127 pooljob
 0      1560 3   4       200   ffff812308702c00     poolthread@127 pooljob
 0      1558 3   1       200   ffff812308702400     poolthread@127 pooljob
 0      1551 3   1       200   ffff812308e8a800     poolthread@127 pooljob
 0      1549 3  11       200   ffff812308e8a000     poolthread@126 pooljob
 0      1548 3   1       200   ffff81230922ec00     poolthread@126 pooljob
 0      1547 3   8       200   ffff81230922e800     poolthread@126 pooljob
 0      1546 3   5       200   ffff81230922e400     poolthread@126 pooljob
 0      1545 3  11       200   ffff81230922e000     poolthread@126 pooljob
 0      1544 3   1       200   ffff812308ecac00     poolthread@126 pooljob
 0      1543 3   2       200   ffff812308eca800     poolthread@126 pooljob
 0      1542 3   3       200   ffff812308eca400     poolthread@126 pooljob
 0      1541 3   4       200   ffff812308eca000     poolthread@126 pooljob
 0      1540 3  11       200   ffff812324fa6c00     poolthread@126 pooljob
 0      1539 3   5       200   ffff812324fa6800     poolthread@126 pooljob
 db{9}> bt /a ffff81230a576400
 trace: pid 2982 lid 2982 at 0xffff8614bd43ba90
 sleepq_block() at netbsd:sleepq_block+0xea
 cv_wait() at netbsd:cv_wait+0xca
 workqueue_q_wait() at netbsd:workqueue_q_wait+0x90
 workqueue_wait() at netbsd:workqueue_wait+0xa9
 lacp_freeport() at netbsd:lacp_freeport+0x61
 lagg_proto_portctrl() at netbsd:lagg_proto_portctrl+0x120
 lagg_port_teardown() at netbsd:lagg_port_teardown+0x18c
 lagg_ifdetach() at netbsd:lagg_ifdetach+0x97
 simplehook_dohooks() at netbsd:simplehook_dohooks+0xa3
 ether_ifdetach() at netbsd:ether_ifdetach+0x23
 wm_detach() at netbsd:wm_detach+0x401
 config_detach_release() at netbsd:config_detach_release+0x1be
 config_detach_all() at netbsd:config_detach_all+0xc4
 cpu_reboot() at netbsd:cpu_reboot+0x1bd
 kern_reboot() at netbsd:kern_reboot+0x5f
 sys_reboot() at netbsd:sys_reboot+0x65
 syscall() at netbsd:syscall+0x9d
 --- syscall (number 208) ---
 netbsd:syscall+0x9d:
 db{9}> bt /a ffff8123082b6800
 trace: pid 0 lid 1643 at 0xffff8614bcdbde80
 sleepq_block() at netbsd:sleepq_block+0xea
 turnstile_block() at netbsd:turnstile_block+0x2d0
 mutex_enter() at netbsd:mutex_enter+0x142
 lacp_linkspeed_work() at netbsd:lacp_linkspeed_work+0x84
 workqueue_worker() at netbsd:workqueue_worker+0xf3
 db{9}> show all locks
 [Locks tracked through LWPs]

 ****** LWP 2982.2982 (reboot) @ 0xffff81230a576400, l_stat=3

 *** Locks held:

 * Lock 0 (initialized at netbsd:if_initialize+0x10e)
 lock address : ffff8123093ef7c0
 type         : sleep/adaptive
 initialized  : netbsd:if_initialize+0x10e
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 relevant cpu :                  1 last held:                  1
 relevant lwp : 0xffff81230a576400 last held: 0xffff81230a576400
 last locked* : netbsd:lagg_ifdetach+0x75
 unlocked     : netbsd:if_link_state_change_work+0x139
 owner field  : 0xffff81230a576400 wait/spin:                0/0
 Turnstile: no active turnstile for this lock.

 * Lock 1 (initialized at netbsd:lagg_clone_create+0x45)
 lock address : ffff8123057eb000
 type         : sleep/adaptive
 initialized  : netbsd:lagg_clone_create+0x45
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  1
 relevant cpu :                  1 last held:                  1
 relevant lwp : 0xffff81230a576400 last held: 0xffff81230a576400
 last locked* : netbsd:lagg_ifdetach+0x7d
 unlocked     : netbsd:lagg_ifdetach+0x69
 owner field  : 0xffff81230a576400 wait/spin:                1/0
 Turnstile:
 => 0 waiting readers:
 => 1 waiting writers: 0xffff8123082b6800

 * Lock 2 (initialized at netbsd:if_initialize+0x10e)
 lock address : ffff812186a45c40
 type         : sleep/adaptive
 initialized  : netbsd:if_initialize+0x10e
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 relevant cpu :                  1 last held:                  1
 relevant lwp : 0xffff81230a576400 last held: 0xffff81230a576400
 last locked* : netbsd:lagg_port_teardown+0xe5
 unlocked     : netbsd:ether_del_vlantag+0x10d
 owner field  : 0xffff81230a576400 wait/spin:                0/0
 Turnstile: no active turnstile for this lock.

 *** Locks wanted: none

 ****** LWP 0.1643 (lagg0.lacp) @ 0xffff8123082b6800, l_stat=3

 *** Locks held: none

 *** Locks wanted:

 * Lock 0 (initialized at netbsd:lagg_clone_create+0x45)
 lock address : ffff8123057eb000
 type         : sleep/adaptive
 initialized  : netbsd:lagg_clone_create+0x45
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  1
 relevant cpu :                  6 last held:                  1
 relevant lwp : 0xffff8123082b6800 last held: 0xffff81230a576400
 last locked* : netbsd:lagg_ifdetach+0x7d
 unlocked     : netbsd:lagg_ifdetach+0x69
 owner field  : 0xffff81230a576400 wait/spin:                1/0
 Turnstile:
 => 0 waiting readers:
 => 1 waiting writers: 0xffff8123082b6800

 [Locks tracked through CPUs]

 db{9}>


State-Changed-From-To: open->analyzed
State-Changed-By: riz@NetBSD.org
State-Changed-When: Sun, 19 Jul 2026 15:43:26 +0000
State-Changed-Why:
deadlock between lacp_freeport() and lacp_linkspeed_work()


>Unformatted:

Home	PR Database Search