NetBSD Problem Report #60100

From www@netbsd.org  Wed Mar 18 17:19:14 2026
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "mail.netbsd.org", Issuer "R12" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 923E41A9239
	for <gnats-bugs@gnats.NetBSD.org>; Wed, 18 Mar 2026 17:19:14 +0000 (UTC)
Message-Id: <20260318171913.66E131A923A@mollari.NetBSD.org>
Date: Wed, 18 Mar 2026 17:19:13 +0000 (UTC)
From: pyhirricturtle@gmail.com
Reply-To: pyhirricturtle@gmail.com
To: gnats-bugs@NetBSD.org
Subject: vax libm: ALTENTRY labels land on NOP padding instead of entry mask, causing SIGILL on CALLS
X-Send-Pr-Version: www-1.0
X-From4GNATS: "pyhirricturtle@gmail.com via gnats" <gnats-admin@NetBSD.org>

>Number:         60100
>Category:       port-vax
>Synopsis:       vax libm: ALTENTRY labels land on NOP padding instead of entry mask, causing SIGILL on CALLS
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    port-vax-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Mar 18 17:20:00 +0000 2026
>Last-Modified:  Wed Mar 18 19:40:00 +0000 2026
>Originator:     FireTurtle
>Release:        NetBSD 10 (also present in -current trunk)
>Organization:
>Environment:
System: NetBSD/vax (SIMH MicroVAX 3900)
>Description:
In several vax libm assembly files, ALTENTRY() is placed immediately before ENTRY(). The intent is for the ALTENTRY symbol to share the following ENTRY's .word register-save mask. However, ENTRY() begins with _ALIGN_TEXT, which inserts NOP padding (0x0101...) for alignment. The ALTENTRY label ends up pointing at the NOP padding rather than the .word mask.

When the function is called via the ALTENTRY name using CALLS, the processor reads the NOPs as the entry mask (e.g., 0x0101), saves the wrong registers, and begins execution at the wrong offset. This results in SIGILL or other undefined behavior.

Affected files:
- lib/libm/arch/vax/n_cbrt.S: ALTENTRY(cbrt) before ENTRY(d_cbrt, 0x00c0)
- lib/libm/arch/vax/n_cabs.S: ALTENTRY(cabs) before ENTRY(_hypot, 0x8040)
- lib/libm/arch/vax/n_scalbn.S: ALTENTRY(_scalbnl) before ENTRY(_scalbn, 0)

For example in n_cbrt.S:

    ALTENTRY(cbrt)       /* label here */
    ENTRY(d_cbrt, 0x00c0)  /* _ALIGN_TEXT inserts NOPs, then .word 0x00c0 */

objdump -d shows the cbrt symbol points several bytes before d_cbrt -- into the NOP alignment padding -- rather than at d_cbrt's .word 0x00c0 entry mask.

This may be related to the historical port-vax/18483 (2002), which reported SIGILL in atan() -- the same class of symptom. That PR noted static linking worked but dynamic linking crashed.
>How-To-Repeat:
1. Inspect the generated binary:
   objdump -d libm.so | grep -A5 '<cbrt>'
   The cbrt symbol points into NOP padding, not at the .word entry mask.

2. Or call cbrt(8.0) from a dynamically-linked program on vax:
   #include <math.h>
   #include <stdio.h>
   int main() { printf("%f\n", cbrt(8.0)); return 0; }
   This will crash with SIGILL.
>Fix:
The simplest fix for each affected file is to move ALTENTRY after the alignment and .word mask, making it an alias that points to the same address as the ENTRY name. For example in n_cbrt.S:

    ENTRY(d_cbrt, 0x00c0)
    ALTENTRY(cbrt)        /* now points at same address as d_cbrt */

Alternatively, give each alternate name its own ENTRY() with the correct mask.

The broader question is whether the ALTENTRY macro itself should be redefined to account for _ALIGN_TEXT in ENTRY, but that is a design decision for the port maintainers.

>Audit-Trail:
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-vax/60100: vax libm: ALTENTRY labels land on NOP padding
 instead of entry mask, causing SIGILL on CALLS
Date: Wed, 18 Mar 2026 20:35:10 +0100

 The only usage of ALTENTRY in libm in -current is in

 src/lib/libm/arch/vax/n_sqrt.S:


 /* **************************** internal procedure */

         .hidden __libm_dsqrt_r5
 ALTENTRY(__libm_dsqrt_r5)
         halt
         halt
 __libm_dsqrt_r5_lcl:
                                 /* ENTRY POINT FOR cdabs and cdsqrt     */
                                 /* returns double square root scaled by */
                                 /* 2^%r6        */


 and that is probably proper ALTENTRY usage. All other places have been
 replaced by STRONG_ALIAS.

 Anyway, for both types of usage, this patch should fix the issue.

 Index: asm.h
 ===================================================================
 RCS file: /cvsroot/src/sys/arch/vax/include/asm.h,v
 retrieving revision 1.30
 diff -u -p -r1.30 asm.h
 --- asm.h	6 Jan 2025 10:46:44 -0000	1.30
 +++ asm.h	18 Mar 2026 19:34:26 -0000
 @@ -78,7 +78,7 @@
  #define ASENTRY(x, regs)	_ENTRY(_ASM_LABEL(x), regs); _PROF_PROLOGUE
  #define END(x)			.size _C_LABEL(x),.-_C_LABEL(x)

 -#define ALTENTRY(x)		.globl _C_LABEL(x); _C_LABEL(x):
 +#define ALTENTRY(x)		.text; _ALIGN_TEXT; .globl _C_LABEL(x); _C_LABEL(x):
  #ifdef _NETBSD_REVISIONID
  #define RCSID(x)	.pushsection ".ident","MS",@progbits,1;		\
  			.asciz x;					\


 Martin

Home	PR Database Search