NetBSD Problem Report #59381

From www@netbsd.org  Fri May  2 01:26:06 2025
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits)
	 client-signature RSA-PSS (2048 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 7BD571A9239
	for <gnats-bugs@gnats.NetBSD.org>; Fri,  2 May 2025 01:26:06 +0000 (UTC)
Message-Id: <20250502012605.1F7C41A923C@mollari.NetBSD.org>
Date: Fri,  2 May 2025 01:26:05 +0000 (UTC)
From: campbell+netbsd@mumble.net
Reply-To: campbell+netbsd@mumble.net
To: gnats-bugs@NetBSD.org
Subject: t_ptrace_wait* and t_ptrace_sigchld tests are flaky
X-Send-Pr-Version: www-1.0

>Number:         59381
>Category:       misc
>Synopsis:       t_ptrace_wait* and t_ptrace_sigchld tests are flaky
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    misc-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri May 02 01:30:00 +0000 2025
>Last-Modified:  Fri May 09 01:35:02 +0000 2025
>Originator:     Taylor R Campbell
>Release:        
>Organization:
The NetPtraceD Esperation
>Environment:
>Description:
1. Enabling debug=1 in t_ptrace_wait.c and t_ptrace_sigchld.c seems to generate so much output it breaks the releng testbeds by filling up the disk.  Oops.  (See https://mail-index.netbsd.org/source-changes/2025/04/29/msg156694.html and https://mail-index.netbsd.org/source-changes/2025/05/01/msg156711.html, in which I tried to mitigate it after disasters like https://releng.netbsd.org/b5reports/amd64/2025/2025.04.29.23.32.35/test.log, but, e.g., thousands of tests are still failing on earmv7hf because the disk is full: https://releng.netbsd.org/b5reports/evbarm-earmv7hf/2025/2025.05.01.08.43.02/test.log)

2. t_ptrace_wait4:x86_fpregs_fpu_write sometimes fails with on:

   3327 				FORKEE_ASSERT(vals_fpu.ip.fa_64
   3328 				    == expected_fpu.ip.fa_64);

   https://nxr.netbsd.org/xref/src/tests/lib/libc/sys/t_ptrace_x86_wait.h?r=1.31#3327

   By improving the debug prints, I caught it in the act with vals_fpu.ip.fa_64=0x76543210, expected_fpu.ip.fa_64=0xa9876543210.  No idea what happeened to the upper 0x00000a98!

   Similar tests were failing on amd64 throughout April, on the same assertion: https://releng.netbsd.org/b5reports/amd64/commits-2025.04.html#end

3. t_ptrace_wait4:x86_cve_2018_8897 failed with:

   FAILED: /tmp/build/2025.04.15.22.40.20-i386/src/tests/lib/libc/sys/t_ptrace_wait.h:320: WSTOPSIG(status) != expected: Unexpected stop signal received [Segmentation fault] != [Suspended (signal)]

   https://releng.netbsd.org/b5reports/i386/2025/2025.04.15.22.40.20/test.html#lib_libc_sys_t_ptrace_wait4_x86_cve_2018_8897

4. The testbed seems to be confused by t_ptrace_wait*:access_regs_set_unaligned_pc_0x7, which are not failing but which the b5reports summary thinks are failing:

   https://releng.netbsd.org/b5reports/riscv-riscv64/commits-2025.04.html#end

   I wonder whether this may happen because the riscv64 kernel prints a scary message in the middle of the test which is interpreted as a test failure:

    access_regs_set_unaligned_pc_0x7: [ 4739.4096919] Trapframe @ 0xffffffc029eaeee0 (cause=2 (illegal instruction), status=0x4020, pc=      0x3ff83bd2c7, va=0x8300):
[ 4739.4096919]                         ra =           0x206ba  sp =      0x3ffffee480  gp =           0x3ba70
[ 4739.4096919] tp =      0x3ff825a010  t0 =      0x3ff83bd2b0  t1 =0xffffffffffffffff  t2 =      0x3ff842f170
[ 4739.4096919] s0 =           0x3bc08  s1 =                 0  a0 =                 0  a1 =                 0
[ 4739.4096919] a2 =                 0  a3 =                 0  a4 =        0x33330003  a5 =      0x3ff825a000
[ 4739.4096919] a6 =      0x3ff825a110  a7 =      0x3ff825a120  s2 =           0x304f0  s3 =0xffffffffffffffff
[ 4739.4096919] s4 =      0x3ffffee7b8  s5 =           0x303b8  s6 =      0x3ffffee758  s7 =      0x3ffffee7a0
[ 4739.4096919] s8 =      0x3ff8431eac  s9 =      0x3ff867aa28  s10=      0x3ff84373ec  s11=      0x3ffffee7b8
[ 4739.4096919] t3 =      0x3ffffee2c8  t4 =      0x2300000000  t5 =              0x23  t6 =             0x13e
[0.211157s] Passed.

   https://releng.netbsd.org/b5reports/riscv-riscv64/2025/2025.04.03.14.59.05/test.log

5. Lots of t_ptrace_* tests are failing on pmax (though this might be a qemu mips fpu emulation bug): https://releng.netbsd.org/b5reports/pmax/2025/2025.04.30.15.40.38/test.html#failed-tcs-summary
>How-To-Repeat:
cd /usr/tests/lib/libc/sys
atf-run t_ptrace_* | atf-report

(t_ptrace_kill flakiness is tracked separately, since it is not related to the other t_ptrace_* tests -- see PR misc/59380: t_ptrace_kill is flaky, https://gnats.NetBSD.org/59380)
>Fix:
Yes, please!

>Audit-Trail:
From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/59381 CVS commit: src/tests/lib/libc/sys
Date: Fri, 2 May 2025 02:24:32 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Fri May  2 02:24:32 UTC 2025

 Modified Files:
 	src/tests/lib/libc/sys: t_ptrace.c t_ptrace_amd64_wait.h
 	    t_ptrace_bytetransfer_wait.h t_ptrace_clone_wait.h
 	    t_ptrace_core_wait.h t_ptrace_eventmask_wait.h t_ptrace_fork_wait.h
 	    t_ptrace_kill.c t_ptrace_lwp_wait.h t_ptrace_register_wait.h
 	    t_ptrace_sigchld.c t_ptrace_siginfo_wait.h t_ptrace_signal_wait.h
 	    t_ptrace_syscall_wait.h t_ptrace_threads_wait.h
 	    t_ptrace_topology_wait.h t_ptrace_wait.h t_ptrace_x86_wait.h

 Log Message:
 tests/lib/libc/sys/t_ptrace*: Nix trailing whitespace.

 No functional change intended.

 Preparation for:

 PR misc/59381: t_ptrace_wait* and t_ptrace_sigchld tests are flaky


 To generate a diff of this commit:
 cvs rdiff -u -r1.5 -r1.6 src/tests/lib/libc/sys/t_ptrace.c \
     src/tests/lib/libc/sys/t_ptrace_sigchld.c
 cvs rdiff -u -r1.12 -r1.13 src/tests/lib/libc/sys/t_ptrace_amd64_wait.h
 cvs rdiff -u -r1.1 -r1.2 src/tests/lib/libc/sys/t_ptrace_bytetransfer_wait.h \
     src/tests/lib/libc/sys/t_ptrace_eventmask_wait.h \
     src/tests/lib/libc/sys/t_ptrace_kill.c \
     src/tests/lib/libc/sys/t_ptrace_lwp_wait.h \
     src/tests/lib/libc/sys/t_ptrace_siginfo_wait.h \
     src/tests/lib/libc/sys/t_ptrace_threads_wait.h \
     src/tests/lib/libc/sys/t_ptrace_topology_wait.h
 cvs rdiff -u -r1.3 -r1.4 src/tests/lib/libc/sys/t_ptrace_clone_wait.h \
     src/tests/lib/libc/sys/t_ptrace_syscall_wait.h
 cvs rdiff -u -r1.9 -r1.10 src/tests/lib/libc/sys/t_ptrace_core_wait.h
 cvs rdiff -u -r1.7 -r1.8 src/tests/lib/libc/sys/t_ptrace_fork_wait.h
 cvs rdiff -u -r1.4 -r1.5 src/tests/lib/libc/sys/t_ptrace_register_wait.h
 cvs rdiff -u -r1.6 -r1.7 src/tests/lib/libc/sys/t_ptrace_signal_wait.h
 cvs rdiff -u -r1.36 -r1.37 src/tests/lib/libc/sys/t_ptrace_wait.h
 cvs rdiff -u -r1.31 -r1.32 src/tests/lib/libc/sys/t_ptrace_x86_wait.h

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/59381 CVS commit: src/tests/lib/libc/sys
Date: Fri, 2 May 2025 02:24:44 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Fri May  2 02:24:44 UTC 2025

 Modified Files:
 	src/tests/lib/libc/sys: t_ptrace.c t_ptrace_threads_wait.h
 	    t_ptrace_wait.h t_ptrace_x86_wait.h

 Log Message:
 tests/lib/libc/sys/t_ptrace*: Sprinkle improved diagnostics.

 This should make it easier to look at the test failure output and
 figure out what's going on.

 PR misc/59381: t_ptrace_wait* and t_ptrace_sigchld tests are flaky


 To generate a diff of this commit:
 cvs rdiff -u -r1.6 -r1.7 src/tests/lib/libc/sys/t_ptrace.c
 cvs rdiff -u -r1.2 -r1.3 src/tests/lib/libc/sys/t_ptrace_threads_wait.h
 cvs rdiff -u -r1.37 -r1.38 src/tests/lib/libc/sys/t_ptrace_wait.h
 cvs rdiff -u -r1.32 -r1.33 src/tests/lib/libc/sys/t_ptrace_x86_wait.h

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/59381 CVS commit: src/tests/lib/libc/sys
Date: Fri, 2 May 2025 02:30:21 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Fri May  2 02:30:21 UTC 2025

 Modified Files:
 	src/tests/lib/libc/sys: t_ptrace_wait.h

 Log Message:
 t_ptrace*: Show values in hex for FORKEE_ASSERT_EQ/NEQ.

 Should make the output easier to read for:

 PR misc/59381: t_ptrace_wait* and t_ptrace_sigchld tests are flaky


 To generate a diff of this commit:
 cvs rdiff -u -r1.38 -r1.39 src/tests/lib/libc/sys/t_ptrace_wait.h

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/59381 CVS commit: src/tests/lib/libc/sys
Date: Fri, 2 May 2025 02:37:08 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Fri May  2 02:37:07 UTC 2025

 Modified Files:
 	src/tests/lib/libc/sys: t_ptrace_sigchld.c t_ptrace_wait.c

 Log Message:
 t_ptrace_*: Turn debug output back off.

 Even with the output reduction I recently implemented, this seems to
 be filling up the releng testbed disk with excess output.  Not
 completely obviated by the recent diagnostic improvements -- e.g.,
 the traceme_signalmasked_crash_fpe test still uses DPRINTF for some
 output that isn't available any other way.  But let's get the tests
 running to completion again before improving that.

 PR misc/59381: t_ptrace_wait* and t_ptrace_sigchld tests are flaky


 To generate a diff of this commit:
 cvs rdiff -u -r1.6 -r1.7 src/tests/lib/libc/sys/t_ptrace_sigchld.c
 cvs rdiff -u -r1.194 -r1.195 src/tests/lib/libc/sys/t_ptrace_wait.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/59381 CVS commit: src/tests/lib/libc/sys
Date: Fri, 2 May 2025 02:52:41 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Fri May  2 02:52:40 UTC 2025

 Modified Files:
 	src/tests/lib/libc/sys: t_ptrace_signal_wait.h t_ptrace_wait.h

 Log Message:
 t_ptrace_*: More diagnostic output for assertion failures.

 PR misc/59381: t_ptrace_wait* and t_ptrace_sigchld tests are flaky


 To generate a diff of this commit:
 cvs rdiff -u -r1.7 -r1.8 src/tests/lib/libc/sys/t_ptrace_signal_wait.h
 cvs rdiff -u -r1.39 -r1.40 src/tests/lib/libc/sys/t_ptrace_wait.h

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/59381 CVS commit: src/tests/lib/libc/sys
Date: Fri, 9 May 2025 01:30:11 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Fri May  9 01:30:11 UTC 2025

 Modified Files:
 	src/tests/lib/libc/sys: t_ptrace_wait.h

 Log Message:
 t_ptrace*: Fix FORKEE_ASSERT_MEMEQ output and failureness.

 PR misc/59381: t_ptrace_wait* and t_ptrace_sigchld tests are flaky


 To generate a diff of this commit:
 cvs rdiff -u -r1.40 -r1.41 src/tests/lib/libc/sys/t_ptrace_wait.h

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/59381 CVS commit: src/tests/lib/libc/sys
Date: Fri, 9 May 2025 01:32:25 +0000

 Module Name:	src
 Committed By:	riastradh
 Date:		Fri May  9 01:32:25 UTC 2025

 Modified Files:
 	src/tests/lib/libc/sys: t_ptrace_wait.h

 Log Message:
 t_ptrace_*: Put spaces in hexdump output for legibility.

 PR misc/59381: t_ptrace_wait* and t_ptrace_sigchld tests are flaky


 To generate a diff of this commit:
 cvs rdiff -u -r1.41 -r1.42 src/tests/lib/libc/sys/t_ptrace_wait.h

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

Home	PR Database Search