NetBSD Problem Report #60732

From www@netbsd.org  Wed Sep 16 00:27:08 2026
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "mail.netbsd.org", Issuer "YR1" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 1B9DE1A923B
	for <gnats-bugs@gnats.NetBSD.org>; Wed, 16 Sep 2026 00:27:08 +0000 (UTC)
Message-Id: <20260916002706.74E4E1A923C@mollari.NetBSD.org>
Date: Wed, 16 Sep 2026 00:27:06 +0000 (UTC)
From: mrg@eterna23.net
Reply-To: mrg@eterna23.net
To: gnats-bugs@NetBSD.org
Subject: amdgpu attempts to take (sleeping) mutex while holding a spin lock
X-Send-Pr-Version: www-1.0
X-From4GNATS: "mrg@eterna23.net via gnats" <gnats-admin@NetBSD.org>

>Number:         60732
>Notify-List:    riastradh
>Category:       kern
>Synopsis:       amdgpu attempts to take (sleeping) mutex while holding a spin lock
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          analyzed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Sep 16 00:30:00 +0000 2026
>Closed-Date:    
>Last-Modified:  Tue Sep 22 04:40:28 +0000 2026
>Originator:     matthew green
>Release:        -current from 2026-09-15
>Organization:
people's front against (bozotic) www (softwar foundation)
>Environment:
NetBSD the-blessing.eterna23.net 11.99.8 NetBSD 11.99.8 (_blessing_) #472: Mon Sep 14 19:19:02 PDT 2026  mrg@yesterday-when-i-was-mad.eterna23.net:/var/obj/amd64-x86_64/usr/src/sys/arch/amd64/compile/_blessing_ amd64

>Description:
while trying to trigger a different locking issue in amdgpu (that annoyingly
disappeared when LOCKDEBUG was enabled), i triggered a locking issue after
restarting X a 4th time.  (i don't think 4 matters.  whatever causes this
problem just is not always happening.)

[ 7579.5137040] Mutex error: mutex_vector_enter,525: spin lock held
[ 7579.5237038] lock address : ffff8694655bb930
[ 7579.5237038] type         : spin
[ 7579.5237038] initialized  : netbsd:amdgpu_gtt_mgr_init+0x71
[ 7579.5337039] shared holds :                  0 exclusive:                  1
[ 7579.5437036] shares wanted:                  0 exclusive:                  0
[ 7579.5437036] relevant cpu :                 14 last held:                 14
[ 7579.5537038] relevant lwp : 0xffff86a34bed5800 last held: 0xffff86a34bed5800
[ 7579.5637033] last locked* : netbsd:amdgpu_gtt_mgr_recover+0x21
[ 7579.5637033] unlocked     : netbsd:amdgpu_gtt_mgr_del+0x41
[ 7579.5737033] owner field  : 0x0000000000010600 wait/spin:                0/1
[ 7579.5837034] panic: LOCKDEBUG: Mutex error: mutex_vector_enter,525: spin lock held

#4  0xffffffff8105112b in lockdebug_abort1 (func=0xffffffff81832640 <__func__.4> "mutex_vector_enter", line=525, ld=0xffff95008810a5c0, s=6, 
    msg=0xffffffff818cfbec "spin lock held", dopanic=true) at /usr/src/sys/kern/subr_lockdebug.c:818
#5  lockdebug_abort1 (func=0xffffffff81832640 <__func__.4> "mutex_vector_enter", line=525, ld=0xffff95008810a5c0, s=6, 
    msg=0xffffffff818cfbec "spin lock held", dopanic=<optimized out>) at /usr/src/sys/kern/subr_lockdebug.c:796
#6  0xffffffff81002b2c in mutex_vector_enter (mtx=0xffff950088143390) at /usr/src/sys/kern/kern_mutex.c:525
#7  0xffffffff809b37a4 in mutex_lock (mutex=0xffff950088143390) at /usr/src/sys/external/bsd/drm2/include/linux/mutex.h:76
#8  gmc_v10_0_flush_gpu_tlb (adev=0xffff95008813e008, vmid=0, vmhub=0, flush_type=<optimized out>)
    at /usr/src/sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_gmc_v10_0.c:353
#9  0xffffffff80972447 in amdgpu_gart_post_update (adev=adev@entry=0xffff95008813e008, gpu_pgstart=gpu_pgstart@entry=1024, gpu_npages=gpu_npages@entry=1)
    at /usr/src/sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_gart.c:299
#10 0xffffffff80972ba7 in amdgpu_gart_bind (adev=0xffff95008813e008, gpu_start=<optimized out>, npages=1, pages=<optimized out>, dmamap=<optimized out>, 
    flags=<optimized out>) at /usr/src/sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_gart.c:523
#11 0xffffffff80a02b5f in amdgpu_ttm_gart_bind (adev=<optimized out>, tbo=<optimized out>, flags=844424930132087)
    at /usr/src/sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_ttm.c:1093
#12 0xffffffff809baf11 in amdgpu_gtt_mgr_recover (man=man@entry=0xffff950088142e10) at /usr/src/sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_gtt_mgr.c:339
#13 0xffffffff8096ae11 in amdgpu_do_asic_reset (hive=<optimized out>, device_list_handle=0xffff95125ce66e80, need_full_reset_arg=0xffff95125ce66e7f)
    at /usr/src/sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_device.c:4121
#14 amdgpu_device_gpu_recover (adev=<optimized out>, job=job@entry=0xffff869466e4b488)
    at /usr/src/sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_device.c:4391
#15 0xffffffff809be765 in amdgpu_job_timedout (s_job=0xffff869466e4b488) at /usr/src/sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_job.c:58
#16 0xffffffff80e18c8a in drm_sched_job_timedout (work=0xffff9500881457e0) at /usr/src/sys/external/bsd/drm2/dist/drm/scheduler/sched_main.c:318
#17 0xffffffff80e0df7b in linux_workqueue_thread (cookie=0xffff86a34bed8000) at /usr/src/sys/external/bsd/common/linux/linux_work.c:455
#18 0xffffffff80210327 in lwp_trampoline ()

the problem is that at the higher level (frame 12 above), a spinlock is taken:

329     int amdgpu_gtt_mgr_recover(struct ttm_mem_type_manager *man)
...
336             spin_lock(&mgr->lock);
337             drm_mm_for_each_node(mm_node, &mgr->mm) {
338                     node = container_of(mm_node, struct amdgpu_gtt_node, node);
339                     r = amdgpu_ttm_recover_gart(node->tbo);
340                     if (r)
341                             break;
342             }
343             spin_unlock(&mgr->lock);

however, downstack at frame 8, this sleeping mutex is taken:

341     static void gmc_v10_0_flush_gpu_tlb(struct amdgpu_device *adev, uint32_t vmid,
342                                             uint32_t vmhub, uint32_t flush_type)
...
353             mutex_lock(&adev->mman.gtt_window_lock);

this violates locking and panic appears.
>How-To-Repeat:
startx multiple times with a AMD RX 5300 GPU installed.
>Fix:
don't take sleep mutex when holding spinlock? :-)

idea i did not look at to see if feasible or try, would be to
make the mutex into an spin mutex.

>Release-Note:

>Audit-Trail:
From: Taylor R Campbell <riastradh@NetBSD.org>
To: matthew green <mrg@eterna23.net>
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: kern/60732: amdgpu attempts to take (sleeping) mutex while holding a spin lock
Date: Wed, 16 Sep 2026 13:04:38 +0000

 This is a multi-part message in MIME format.
 --=_aD0stfJJ+WVg8+9bTfy4bTTT5i9m1xYu
 Content-Transfer-Encoding: quoted-printable

 > Date: Wed, 16 Sep 2026 00:27:06 +0000 (UTC)
 > From: mrg@eterna23.net
 >=20
 > idea i did not look at to see if feasible or try, would be to
 > make the mutex into an spin mutex.

 Won't work -- even if you change gtt_window_lock to a spin lock, the
 function in question sleeps:

     341 static void gmc_v10_0_flush_gpu_tlb(struct amdgpu_device *adev, uin=
 t32_t vmid,
     342 					uint32_t vmhub, uint32_t flush_type)
     343 {
 ...
     377 	r =3D amdgpu_job_alloc_with_ib(adev, 16 * 4, &job);
 ...
     392 	dma_fence_wait(fence, false);
 ...
     403 }

 https://nxr.NetBSD.org/xref/src/sys/external/bsd/drm2/dist/drm/amd/amdgpu/a=
 mdgpu_gmc_v10_0.c?r=3D1.5#333

 Try the attached patch to drop the spin lock, but hold a reference to
 the node so the iteration is safe, while calling
 amdgpu_ttm_recover_gart?

 --=_aD0stfJJ+WVg8+9bTfy4bTTT5i9m1xYu
 Content-Type: text/plain; charset="ISO-8859-1"; name="pr60732-amdgpugartrecoverspinmutex"
 Content-Transfer-Encoding: quoted-printable
 Content-Disposition: attachment; filename="pr60732-amdgpugartrecoverspinmutex.patch"

 # HG changeset patch
 # User Taylor R Campbell <riastradh@NetBSD.org>
 # Date 1789563496 0
 #      Wed Sep 16 12:58:16 2026 +0000
 # Branch trunk
 # Node ID 2c75d691e0dd07e7b4ab40237114314280231942
 # Parent  4fbd288ac1da9e7d9592ddecf04f7513e244152f
 # EXP-Topic riastradh-pr60732-amdgpugartrecoverspinmutex
 WIP: amdgpu: Drop spin lock across possible (non-spin) mutex & wait.

 PR kern/60732: amdgpu attempts to take (sleeping) mutex while holding
 a spin lock

 diff -r 4fbd288ac1da -r 2c75d691e0dd sys/external/bsd/drm2/dist/drm/amd/amd=
 gpu/amdgpu_gtt_mgr.c
 --- a/sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_gtt_mgr.c	Mon Aug 10=
  13:36:10 2026 +0000
 +++ b/sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_gtt_mgr.c	Wed Sep 16=
  12:58:16 2026 +0000
 @@ -336,7 +336,11 @@ int amdgpu_gtt_mgr_recover(struct ttm_me
  	spin_lock(&mgr->lock);
  	drm_mm_for_each_node(mm_node, &mgr->mm) {
  		node =3D container_of(mm_node, struct amdgpu_gtt_node, node);
 +		ttm_bo_get(node->tbo);
 +		spin_unlock(&mgr->lock);
  		r =3D amdgpu_ttm_recover_gart(node->tbo);
 +		spin_lock(&mgr->lock);
 +		ttm_bo_put(node->tbo);
  		if (r)
  			break;
  	}

 --=_aD0stfJJ+WVg8+9bTfy4bTTT5i9m1xYu--

State-Changed-From-To: open->feedback
State-Changed-By: riastradh@NetBSD.org
State-Changed-When: Wed, 16 Sep 2026 13:06:03 +0000
State-Changed-Why:
feedback requested


From: matthew green <mrg@eterna23.net>
To: Taylor R Campbell <riastradh@NetBSD.org>
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: re: kern/60732: amdgpu attempts to take (sleeping) mutex while holding a spin lock
Date: Thu, 17 Sep 2026 10:16:06 +1000

 OK, i think this stops the crash, but reset doesn't work :)

 starting X the first time seems to have triggered the reset that was
 causing this issue, but it has hung X with this in dmesg:

 [  34.7230317] {drm:netbsd:amdgpu_job_timedout+0xad} *ERROR* ring gfx_0.0.=
 0 timeout, signaled seq=3D6, emitted seq=3D7
 [  34.7330320] {drm:netbsd:amdgpu_job_timedout+0xc9} *ERROR* Process infor=
 mation: process X pid 2410 thread X:cs0 pid 2415
 [  34.7530318] amdgpu0: GPU reset begin!
 [  35.2730324] {drm:netbsd:gfx_v10_0_hw_fini+0x133} *ERROR* KGQ disable fa=
 iled
 [  35.5930331] {drm:netbsd:gfx_v10_0_hw_fini+0x93} *ERROR* KCQ disable fai=
 led
 [  35.9030342] {drm:netbsd:gfx_v10_0_hw_fini+0x56} *ERROR* failed to halt =
 cp gfx
 [  35.9330338] amdgpu0: GPU mode1 reset
 [  36.4230349] [drm] psp mode1 reset succeed =

 [  37.7730380] amdgpu0: GPU reset succeeded, trying to resume
 [  37.7830381] [drm] PCIE GART of 512M enabled (table at 0x000000800090000=
 0).
 [  37.7930381] [drm] PSP is resuming...
 [  37.9630384] [drm] reserve 0x900000 from 0x80be400000 for PSP TMR
 [  38.0130385] amdgpu0: warn: RAS: ras ta ucode is not available
 [  38.0630387] [drm] kiq ring mec 2 pipe 1 q 0
 [  38.3630394] amdgpu0 {drm:netbsd:amdgpu_ring_test_helper+0x4a} *ERROR* r=
 ing comp_1.0.0 test failed (-60)
 [  38.3730394] {drm:netbsd:amdgpu_device_ip_resume_phase2+0x9f} *ERROR* re=
 sume of IP block <gfx_v10_0> failed -60
 [  38.3830394] amdgpu0: GPU reset(1) failed
 [  38.3830394] amdgpu0: GPU reset end with ret =3D -60
 [  48.3830624] {drm:netbsd:amdgpu_job_timedout+0xad} *ERROR* ring gfx_0.0.=
 0 timeout, signaled seq=3D7, emitted seq=3D7
 [  48.4030631] {drm:netbsd:amdgpu_job_timedout+0xc9} *ERROR* Process infor=
 mation: process X pid 2410 thread X:cs0 pid 2415
 [  48.4130624] amdgpu0: GPU reset begin!

 ie, this can now debug the real problem... whatever is timing out
 or triggering the reset, probably is why it fails to reset.

 can probably close this PR, and i'll file another one when i have
 enough info about the above.

 thanks!


 .mrg.

State-Changed-From-To: feedback->analyzed
State-Changed-By: mrg@NetBSD.org
State-Changed-When: Tue, 22 Sep 2026 04:40:28 +0000
State-Changed-Why:
feedback was provided


>Unformatted:

Home	PR Database Search