NetBSD Problem Report #54093
From gson@gson.org Mon Apr 1 15:29:04 2019
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id ECF117A189
for <gnats-bugs@gnats.NetBSD.org>; Mon, 1 Apr 2019 15:29:03 +0000 (UTC)
Message-Id: <20190401152857.C320D98940C@guava.gson.org>
Date: Mon, 1 Apr 2019 18:28:57 +0300 (EEST)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@NetBSD.org
Subject: Recent test regressions on sparc
X-Send-Pr-Version: 3.95
>Number: 54093
>Category: port-sparc
>Synopsis: Recent test regressions on sparc
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: martin
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Mon Apr 01 15:30:00 +0000 2019
>Closed-Date: Fri Dec 06 18:35:43 +0000 2019
>Last-Modified: Fri Dec 06 18:35:43 +0000 2019
>Originator: Andreas Gustafsson
>Release: NetBSD-current, source date >= 2019.03.29.20.18.09
>Organization:
>Environment:
System: NetBSD
Architecture: sparc
Machine: sparc
>Description:
The following test cases have all recently started failing on sparc:
lib/csu/t_crt0:initfini2
lib/csu/t_ifunc_static:ifunc_static
usr.bin/c++/t_call_once:call_once_static
usr.bin/c++/t_cxxruntime:cxxruntime_static
usr.bin/c++/t_hello:hello_static
usr.bin/c++/t_pthread_once:pthread_once_static
usr.bin/c++/t_static_destructor:static_destructor_static
usr.bin/cc/t_hello:hello_profile
They were broken some time during the period of build breakage that
started with the commit titled "back to jemalloc for the masses":
2019.03.27.18.41.18 christos src/share/mk/bsd.own.mk 1.1117
and ended with
2019.03.29.20.18.09 christos src/sys/arch/sparc/include/vmparam.h 1.48
Logs at:
http://www.gson.org/netbsd/bugs/build/sparc/commits-2019.03.html#2019.03.29.20.18.09
http://releng.netbsd.org/b5reports/sparc/commits-2019.03.html#2019.03.29.20.27.45
>How-To-Repeat:
>Fix:
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: port-sparc-maintainer->martin
Responsible-Changed-By: martin@NetBSD.org
Responsible-Changed-When: Mon, 01 Apr 2019 15:37:01 +0000
Responsible-Changed-Why:
Take - looks similar to the sparc64 -m32 -static jemalloc lossage
From: Takeshi Nakayama <nakayama@NetBSD.org>
To: gnats-bugs@netbsd.org, martin@NetBSD.org
Cc: port-sparc-maintainer@netbsd.org, netbsd-bugs@netbsd.org,
gnats-admin@netbsd.org, gson@gson.org
Subject: Re: port-sparc/54093 (Recent test regressions on sparc)
Date: Sun, 03 Nov 2019 04:01:30 +0900 (JST)
Hello,
I found that the cause of this probrem is that TLS data area is not
properly aligned. jemalloc uses TLS, so the problem has become
apparent.
netbsd-8% readelf -l /lib/libc.so | egrep 'TLS|Align'
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
TLS 0x140000 0x00140000 0x00140000 0x00000 0x00004 R 0x4
netbsd-9% readelf -l /lib/libc.so | egrep 'TLS|Align'
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
TLS 0x1d2000 0x001e2000 0x001e2000 0x00b10 0x00b14 R 0x8
When I upgraded to netbsd-9, statically linked binaries and some
pkg binaries compiled with netbsd-8 now cause bus errors. So I
take a look in details and found the cause by referring to the
following document.
https://docs.oracle.com/cd/E19683-01/817-3677/6mj8mbtcb/index.html
The attached patch fixes the problem (and also fixes PR/54074).
lib/libc/tls/tls.c is a fix for statically linked binaries and
libexec/ld.elf_so/tls.c is a fix for dynamically linked binaries.
I think there is a similar problem in the case of __HAVE_TLS_VARIANT_I,
but I couldn't confirm it because I don't have such a machine.
-- Takeshi Nakayama
Index: lib/libc/tls/tls.c
===================================================================
RCS file: /cvsroot/src/lib/libc/tls/tls.c,v
retrieving revision 1.9
diff -u -d -r1.9 tls.c
--- lib/libc/tls/tls.c 13 Jul 2018 19:50:21 -0000 1.9
+++ lib/libc/tls/tls.c 1 Nov 2019 13:01:25 -0000
@@ -57,6 +57,9 @@
static const void *tls_initaddr;
static size_t tls_initsize;
static size_t tls_size;
+#ifdef __HAVE_TLS_VARIANT_II
+static size_t tls_align;
+#endif
static size_t tls_allocation;
static void *initial_thread_tcb;
@@ -85,7 +88,7 @@
if (initial_thread_tcb == NULL) {
#ifdef __HAVE_TLS_VARIANT_II
- tls_size = roundup2(tls_size, sizeof(void *));
+ tls_size = roundup2(tls_size, tls_align);
#endif
tls_allocation = tls_size + sizeof(*tcb);
@@ -149,6 +152,9 @@
tls_initaddr = (void *)(phdr->p_vaddr + data->dlpi_addr);
tls_initsize = phdr->p_filesz;
tls_size = phdr->p_memsz;
+#ifdef __HAVE_TLS_VARIANT_II
+ tls_align = phdr->p_align;
+#endif
}
return 0;
}
Index: libexec/ld.elf_so/tls.c
===================================================================
RCS file: /cvsroot/src/libexec/ld.elf_so/tls.c,v
retrieving revision 1.12
diff -u -d -r1.12 tls.c
--- libexec/ld.elf_so/tls.c 13 Apr 2019 00:23:32 -0000 1.12
+++ libexec/ld.elf_so/tls.c 1 Nov 2019 13:01:25 -0000
@@ -48,6 +48,9 @@
static size_t _rtld_tls_static_space; /* Static TLS space allocated */
static size_t _rtld_tls_static_offset; /* Next offset for static TLS to use */
+#ifndef __HAVE_TLS_VARIANT_I
+static size_t _rtld_tls_static_align; /* Static TLS space alignment */
+#endif
size_t _rtld_tls_dtv_generation = 1;
size_t _rtld_tls_max_index = 1;
@@ -99,7 +102,7 @@
#ifndef __HAVE_TLS_VARIANT_I
_rtld_tls_static_space = roundup2(_rtld_tls_static_space,
- sizeof(void *));
+ _rtld_tls_static_align);
#endif
dbg(("_rtld_tls_static_space %zu", _rtld_tls_static_space));
@@ -118,11 +121,13 @@
struct tls_tcb *tcb;
uint8_t *p, *q;
- p = xcalloc(_rtld_tls_static_space + sizeof(struct tls_tcb));
#ifdef __HAVE_TLS_VARIANT_I
+ p = xcalloc(_rtld_tls_static_space + sizeof(struct tls_tcb));
tcb = (struct tls_tcb *)p;
p += sizeof(struct tls_tcb);
#else
+ p = xcalloc(roundup2(_rtld_tls_static_space + sizeof(struct tls_tcb),
+ _rtld_tls_static_align));
p += _rtld_tls_static_space;
tcb = (struct tls_tcb *)p;
tcb->tcb_self = tcb;
@@ -207,7 +212,11 @@
_rtld_die();
}
+#ifdef __HAVE_TLS_VARIANT_I
p = xmalloc(obj->tlssize);
+#else
+ p = xmalloc(roundup2(obj->tlssize, obj->tlsalign));
+#endif
memcpy(p, obj->tlsinit, obj->tlsinitsize);
memset(p + obj->tlsinitsize, 0, obj->tlssize - obj->tlsinitsize);
@@ -260,6 +269,10 @@
}
obj->tlsoffset = offset;
_rtld_tls_static_offset = next_offset;
+#ifndef __HAVE_TLS_VARIANT_I
+ if (_rtld_tls_static_align < obj->tlsalign)
+ _rtld_tls_static_align = obj->tlsalign;
+#endif
obj->tls_done = 1;
return 0;
From: "Joerg Sonnenberger" <joerg@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/54093 CVS commit: src
Date: Mon, 4 Nov 2019 12:45:10 +0000
Module Name: src
Committed By: joerg
Date: Mon Nov 4 12:45:10 UTC 2019
Modified Files:
src/lib/libc/tls: Makefile.inc tls.c
src/libexec/ld.elf_so: Makefile tls.c
src/usr.bin/ldd: Makefile.elf
Log Message:
PR 54093: Align static TLS area to max_align_t.
To generate a diff of this commit:
cvs rdiff -u -r1.2 -r1.3 src/lib/libc/tls/Makefile.inc
cvs rdiff -u -r1.9 -r1.10 src/lib/libc/tls/tls.c
cvs rdiff -u -r1.141 -r1.142 src/libexec/ld.elf_so/Makefile
cvs rdiff -u -r1.12 -r1.13 src/libexec/ld.elf_so/tls.c
cvs rdiff -u -r1.5 -r1.6 src/usr.bin/ldd/Makefile.elf
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Takeshi Nakayama <nakayama@NetBSD.org>
To: gnats-bugs@netbsd.org, joerg@netbsd.org
Cc: martin@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
gson@gson.org
Subject: Re: PR/54093 CVS commit: src
Date: Wed, 06 Nov 2019 05:24:24 +0900 (JST)
>>> "Joerg Sonnenberger" <joerg@netbsd.org> wrote
> The following reply was made to PR port-sparc/54093; it has been noted by GNATS.
>
> From: "Joerg Sonnenberger" <joerg@netbsd.org>
> To: gnats-bugs@gnats.NetBSD.org
> Cc:
> Subject: PR/54093 CVS commit: src
> Date: Mon, 4 Nov 2019 12:45:10 +0000
>
> Module Name: src
> Committed By: joerg
> Date: Mon Nov 4 12:45:10 UTC 2019
>
> Modified Files:
> src/lib/libc/tls: Makefile.inc tls.c
> src/libexec/ld.elf_so: Makefile tls.c
> src/usr.bin/ldd: Makefile.elf
>
> Log Message:
> PR 54093: Align static TLS area to max_align_t.
I think static TLS area should align with p_align in elf program
header, not with sizeof(max_align_t).
sizeof(max_align_t) is 12 on i386, it is not valid on roundup2.
This seems to cause some atf tests to fail.
http://releng.netbsd.org/b5reports/i386/commits-2019.11.html#2019.11.04.15.12.10
usr.bin/c++/t_static_destructor/static_destructor_static
lib/csu/t_crt0/initfini2
usr.bin/c++/t_cxxruntime/cxxruntime_static
In addition, when the alignment of 16 bytes or more is required as
in the attached test program, strange results occur. Especially
when statically linking, variable initialization fails.
So please revert this change and apply the patch I proposed in this PR?
-- Takeshi Nakayama
#include <stdio.h>
#include <lwp.h>
#include <inttypes.h>
__thread uint32_t a __aligned(32) = 32;
__thread uint64_t b __aligned(16) = 16;
int main(int argc, char *argv[])
{
printf("_lwp_getprivate() = %p\n", _lwp_getprivate());
printf("&a = %p, a = %"PRIu32", misalign = %"PRIuPTR"\n",
&a, a, (uintptr_t)&a & (a-1));
printf("&b = %p, b = %"PRIu64", misalign = %"PRIu64"\n",
&b, b, (uintptr_t)&b & (b-1));
}
From: Joerg Sonnenberger <joerg@bec.de>
To: gnats-bugs@netbsd.org
Cc: martin@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
Andreas Gustafsson <gson@gson.org>
Subject: Re: PR/54093 CVS commit: src
Date: Tue, 5 Nov 2019 23:09:42 +0100
On Tue, Nov 05, 2019 at 08:25:01PM +0000, Takeshi Nakayama wrote:
> The following reply was made to PR port-sparc/54093; it has been noted by GNATS.
>
> From: Takeshi Nakayama <nakayama@NetBSD.org>
> To: gnats-bugs@netbsd.org, joerg@netbsd.org
> Cc: martin@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
> gson@gson.org
> Subject: Re: PR/54093 CVS commit: src
> Date: Wed, 06 Nov 2019 05:24:24 +0900 (JST)
>
> >>> "Joerg Sonnenberger" <joerg@netbsd.org> wrote
>
> > The following reply was made to PR port-sparc/54093; it has been noted by GNATS.
> >
> > From: "Joerg Sonnenberger" <joerg@netbsd.org>
> > To: gnats-bugs@gnats.NetBSD.org
> > Cc:
> > Subject: PR/54093 CVS commit: src
> > Date: Mon, 4 Nov 2019 12:45:10 +0000
> >
> > Module Name: src
> > Committed By: joerg
> > Date: Mon Nov 4 12:45:10 UTC 2019
> >
> > Modified Files:
> > src/lib/libc/tls: Makefile.inc tls.c
> > src/libexec/ld.elf_so: Makefile tls.c
> > src/usr.bin/ldd: Makefile.elf
> >
> > Log Message:
> > PR 54093: Align static TLS area to max_align_t.
>
> I think static TLS area should align with p_align in elf program
> header, not with sizeof(max_align_t).
We don't support non-standard alignments of TLS. Fixing that is
intrusive in the dynamic linker and I have no plan to do that ATM.
> sizeof(max_align_t) is 12 on i386, it is not valid on roundup2.
> This seems to cause some atf tests to fail.
That would be a separate bug on i386.
Joerg
From: Takeshi Nakayama <nakayama@NetBSD.org>
To: joerg@bec.de
Cc: gnats-bugs@netbsd.org, martin@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, gson@gson.org
Subject: Re: PR/54093 CVS commit: src
Date: Wed, 06 Nov 2019 14:34:20 +0900 (JST)
>>> Joerg Sonnenberger <joerg@bec.de> wrote
> > I think static TLS area should align with p_align in elf program
> > header, not with sizeof(max_align_t).
>
> We don't support non-standard alignments of TLS. Fixing that is
> intrusive in the dynamic linker and I have no plan to do that ATM.
Should we follow p_align for static linking? Since it is now
aligned with a value different from p_align, some atf tests fails
on amd64 like sparc before the change. Previously, p_align and
sizeof(void *) had the same value on amd64 by chance.
http://releng.netbsd.org/b5reports/amd64/commits-2019.11.html#2019.11.04.15.12.10
lib/csu/t_ifunc_static/ifunc_static
usr.bin/c++/t_call_once/call_once_static
usr.bin/c++/t_call_once2/call_once2_static
usr.bin/c++/t_cxxruntime/cxxruntime_static
usr.bin/c++/t_hello/hello_static
usr.bin/c++/t_pthread_once/pthread_once_static
usr.bin/c++/t_static_destructor/static_destructor_static
> > sizeof(max_align_t) is 12 on i386, it is not valid on roundup2.
> > This seems to cause some atf tests to fail.
>
> That would be a separate bug on i386.
For dynamic linking, I understood the change since you fixed to use
roundup2 with alignof(max_align_t) instead of sizeof(max_align_t).
-- Takeshi Nakayama
From: Takeshi Nakayama <nakayama@NetBSD.org>
To: joerg@bec.de
Cc: gnats-bugs@netbsd.org, martin@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, gson@gson.org
Subject: Re: PR/54093 CVS commit: src
Date: Wed, 06 Nov 2019 16:46:27 +0900 (JST)
>>> Takeshi Nakayama <nakayama@NetBSD.org> wrote
> >>> Joerg Sonnenberger <joerg@bec.de> wrote
>
> > > I think static TLS area should align with p_align in elf program
> > > header, not with sizeof(max_align_t).
> >
> > We don't support non-standard alignments of TLS. Fixing that is
> > intrusive in the dynamic linker and I have no plan to do that ATM.
>
> Should we follow p_align for static linking? Since it is now
> aligned with a value different from p_align, some atf tests fails
> on amd64 like sparc before the change. Previously, p_align and
> sizeof(void *) had the same value on amd64 by chance.
I prepared a patch that simplified my previous patch to fix for
static linking. Is it okay to commit?
The rounding logic should be correct because it does the same thing
as _rtld_tls_offset_allocate in libexec/ld.elf_so/tls.c.
https://nxr.netbsd.org/xref/src/libexec/ld.elf_so/tls.c#234
-- Takeshi Nakayama
Index: lib/libc/tls/Makefile.inc
===================================================================
RCS file: /cvsroot/src/lib/libc/tls/Makefile.inc,v
retrieving revision 1.3
diff -u -d -r1.3 Makefile.inc
--- lib/libc/tls/Makefile.inc 4 Nov 2019 12:45:10 -0000 1.3
+++ lib/libc/tls/Makefile.inc 6 Nov 2019 06:29:53 -0000
@@ -6,4 +6,4 @@
.PATH: ${.PARSEDIR} ${ARCHDIR}/tls
SRCS+= tls.c
-CPPFLAGS.tls.c+= -D_LIBC_SOURCE -std=gnu11
+CPPFLAGS.tls.c+= -D_LIBC_SOURCE
Index: lib/libc/tls/tls.c
===================================================================
RCS file: /cvsroot/src/lib/libc/tls/tls.c,v
retrieving revision 1.11
diff -u -d -r1.11 tls.c
--- lib/libc/tls/tls.c 5 Nov 2019 22:22:42 -0000 1.11
+++ lib/libc/tls/tls.c 6 Nov 2019 06:29:53 -0000
@@ -46,7 +46,6 @@
#include <link_elf.h>
#include <lwp.h>
#include <stdbool.h>
-#include <stdalign.h>
#include <stddef.h>
#include <stdlib.h>
#include <string.h>
@@ -85,9 +84,6 @@
uint8_t *p;
if (initial_thread_tcb == NULL) {
-#ifdef __HAVE_TLS_VARIANT_II
- tls_size = roundup2(tls_size, alignof(max_align_t));
-#endif
tls_allocation = tls_size + sizeof(*tcb);
initial_thread_tcb = p = mmap(NULL, tls_allocation,
@@ -105,7 +101,7 @@
tcb = (struct tls_tcb *)p;
p += sizeof(struct tls_tcb);
#else
- /* LINTED tls_size is rounded above */
+ /* LINTED tls_size is rounded in __libc_static_tls_setup_cb */
tcb = (struct tls_tcb *)(p + tls_size);
tcb->tcb_self = tcb;
#endif
@@ -149,7 +145,11 @@
continue;
tls_initaddr = (void *)(phdr->p_vaddr + data->dlpi_addr);
tls_initsize = phdr->p_filesz;
+#ifdef __HAVE_TLS_VARIANT_I
tls_size = phdr->p_memsz;
+#else
+ tls_size = roundup2(phdr->p_memsz, phdr->p_align);
+#endif
}
return 0;
}
From: "Takeshi Nakayama" <nakayama@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/54093 CVS commit: src/lib/libc/tls
Date: Thu, 21 Nov 2019 23:06:16 +0000
Module Name: src
Committed By: nakayama
Date: Thu Nov 21 23:06:16 UTC 2019
Modified Files:
src/lib/libc/tls: tls.c
Log Message:
Fix PR/54074 and PR/54093 completely.
More similar to the ld.elf_so logic, it is necessary to align with
p_align first. Also, invert the #ifdef condition for consistency.
Should fix regression for static linking binaries:
http://releng.netbsd.org/b5reports/sparc/commits-2019.11.html#2019.11.10.23.39.03
http://releng.netbsd.org/b5reports/sparc64/commits-2019.11.html#2019.11.16.04.10.33
To generate a diff of this commit:
cvs rdiff -u -r1.12 -r1.13 src/lib/libc/tls/tls.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/54093 CVS commit: [netbsd-9] src
Date: Tue, 26 Nov 2019 08:12:27 +0000
Module Name: src
Committed By: martin
Date: Tue Nov 26 08:12:27 UTC 2019
Modified Files:
src/lib/libc/tls [netbsd-9]: Makefile.inc tls.c
src/libexec/ld.elf_so [netbsd-9]: Makefile tls.c
src/usr.bin/ldd [netbsd-9]: Makefile.elf
Log Message:
Pull up following revision(s) (requested by nakayama in ticket #469):
lib/libc/tls/tls.c: revision 1.10
lib/libc/tls/tls.c: revision 1.11
lib/libc/tls/tls.c: revision 1.12
lib/libc/tls/tls.c: revision 1.13
libexec/ld.elf_so/tls.c: revision 1.13
libexec/ld.elf_so/tls.c: revision 1.14
libexec/ld.elf_so/Makefile: revision 1.142
lib/libc/tls/Makefile.inc: revision 1.3
usr.bin/ldd/Makefile.elf: revision 1.6
PR 54093: Align static TLS area to max_align_t.
Use alignof and not size_t for platforms with non-natural base
alignments.
Mirror the ld.elf_so logic for handling aligning the TLS size.
Most noticable, recompute the start of the TLS area for variant I
relative to the TCB. This makes a difference when the segment size and
base alignment don't agree.
Fix PR/54074 and PR/54093 completely.
More similar to the ld.elf_so logic, it is necessary to align with
p_align first. Also, invert the #ifdef condition for consistency.
Should fix regression for static linking binaries:
http://releng.netbsd.org/b5reports/sparc/commits-2019.11.html#2019.11.10.23.39.03
http://releng.netbsd.org/b5reports/sparc64/commits-2019.11.html#2019.11.16.04.10.33
To generate a diff of this commit:
cvs rdiff -u -r1.2 -r1.2.46.1 src/lib/libc/tls/Makefile.inc
cvs rdiff -u -r1.9 -r1.9.2.1 src/lib/libc/tls/tls.c
cvs rdiff -u -r1.141 -r1.141.2.1 src/libexec/ld.elf_so/Makefile
cvs rdiff -u -r1.12 -r1.12.2.1 src/libexec/ld.elf_so/tls.c
cvs rdiff -u -r1.5 -r1.5.30.1 src/usr.bin/ldd/Makefile.elf
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->closed
State-Changed-By: martin@NetBSD.org
State-Changed-When: Fri, 06 Dec 2019 18:35:43 +0000
State-Changed-Why:
Tests are working again
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.45 2018/12/21 14:23:33 maya Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2017
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home