NetBSD Problem Report #58154
From www@netbsd.org Mon Apr 15 08:33:08 2024
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 151891A9238
for <gnats-bugs@gnats.NetBSD.org>; Mon, 15 Apr 2024 08:33:08 +0000 (UTC)
Message-Id: <20240415083306.3C9D71A923A@mollari.NetBSD.org>
Date: Mon, 15 Apr 2024 08:33:06 +0000 (UTC)
From: pho@cielonegro.org
Reply-To: pho@cielonegro.org
To: gnats-bugs@NetBSD.org
Subject: aarch64: firefox-124.0.2 crashes very frequently on NetBSD/aarch64 10.0, likely related to thread-local storage
X-Send-Pr-Version: www-1.0
>Number: 58154
>Notify-List: riastradh
>Category: lib
>Synopsis: aarch64: firefox-124.0.2 crashes very frequently on NetBSD/aarch64 10.0, likely related to thread-local storage
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: lib-bug-people
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Mon Apr 15 08:35:00 +0000 2024
>Closed-Date: Wed Aug 07 11:46:34 +0000 2024
>Last-Modified: Wed Aug 07 11:46:34 +0000 2024
>Originator: PHO
>Release: 10.0
>Organization:
TNF
>Environment:
NetBSD yukari.cielonegro.org 10.0 NetBSD 10.0 (GENERIC64) #0: Thu Mar 28 08:33:33 UTC 2024 mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/evbarm/compile/GENERIC64 evbarm
>Description:
https://mail-index.netbsd.org/tech-userlevel/2024/04/14/msg014295.html
Tab processes of Firefox crash very frequently on NetBSD/aarch64 10.0. Building it with PKG_OPTIONS.firefox+=debug-info revealed that when it crashes it segfaults at one of these two places non-deterministically:
third_party/rlbox/include/rlbox_noop_sandbox.hpp:
> rlbox_noop_sandbox_thread_data* get_rlbox_noop_sandbox_thread_data();
> # define RLBOX_NOOP_SANDBOX_STATIC_VARIABLES() \
> thread_local rlbox::rlbox_noop_sandbox_thread_data \
> rlbox_noop_sandbox_thread_info{ 0, 0 }; \
> namespace rlbox { \
> rlbox_noop_sandbox_thread_data* get_rlbox_noop_sandbox_thread_data() \
> { \
> return &rlbox_noop_sandbox_thread_info; \
> } \
> } \
> static_assert(true, "Enforce semi-colon")
> ...
> template<typename T, typename T_Converted, typename... T_Args>
> auto impl_invoke_with_func_ptr(T_Converted* func_ptr, T_Args&&... params)
> {
> #ifdef RLBOX_EMBEDDER_PROVIDES_TLS_STATIC_VARIABLES
> auto& thread_data = *get_rlbox_noop_sandbox_thread_data();
> #endif
> auto old_sandbox = thread_data.sandbox; // <-- CRASHES HERE!
> thread_data.sandbox = this;
> auto on_exit =
> detail::make_scope_exit([&] { thread_data.sandbox = old_sandbox; });
> return (*func_ptr)(params...);
> }
media/libjpeg/simd/arm/aarch64/jsimd.c:
> static THREAD_LOCAL unsigned int simd_support = ~0;
> ...
> LOCAL(void)
> init_simd(void)
> {
> #ifndef NO_GETENV
> char env[2] = { 0 };
> #endif
> #if defined(__linux__) || defined(ANDROID) || defined(__ANDROID__)
> int bufsize = 1024; /* an initial guess for the line buffer size limit */
> #endif
>
> if (simd_support != ~0U) // <-- CRASHES HERE!
> return;
>
> simd_support = 0;
So both of these cases involve TLS, that is, tab processes segfault while attempting to access thread-local variables. At run-time these functions reside in libxul.so, which is dlopen'ed by the main process. I recall there were a few issues in TLS handling in the past but riastradh@ fixed them before we branched 10.0, right?
"readelf -r libxul.so" shows no R_AARCH64_TLS_TPR in its relocation table but only shows R_AARCH64_TLSDESC, so I believe these variables use local-dynamic model. I tried to create a minimal reproducer but it didn't crash:
https://gist.github.com/depressed-pho/b6894fdaef94a1b9aa5459b1a2f65590
This is the preprocessor output for rlbox_noop_sandbox.hpp, which is instantiated by config/external/rlbox/rlbox_thread_locals.cpp. The thread_local isn't replaced so it's the C++ reserved word:
> thread_local rlbox::rlbox_noop_sandbox_thread_data rlbox_noop_sandbox_thread_info{ 0, 0 }; namespace rlbox { rlbox_noop_sandbox_thread_data* get_rlbox_noop_sandbox_thread_data() { return &rlbox_noop_sandbox_thread_info; } } static_assert(true, "Enforce semi-colon");
This is the command for compiling rlbox_thread_locals.cpp. It's compiled with -fPIC with no explicit -ftls-model, so it's defaulted to global-dynamic:
> /usr/pkgsrc/www/firefox/work/.cwrapper/bin/c++ -std=gnu++17 -o rlbox_thread_locals.o -c -I/usr/pkgsrc/www/firefox/work/build/dist/stl_wrappers -I/usr/pkgsrc/www/firefox/work/build/dist/system_wrappers -include /usr/pkgsrc/www/firefox/work/firefox-124.0.2/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -DNDEBUG=1 -DTRIMMED=1 -DBUILDING_SOUNDTOUCH=1 -DST_NO_EXCEPTION_HANDLING=1 -DMOZ_HAS_MOZGLUE -I/usr/pkgsrc/www/firefox/work/firefox-124.0.2/media/libsoundtouch/src -I/usr/pkgsrc/www/firefox/work/build/media/libsoundtouch/src -I/usr/pkgsrc/www/firefox/work/build/security/rlbox -I/usr/pkgsrc/www/firefox/work/firefox-124.0.2/third_party/simde -I/usr/pkgsrc/www/firefox/work/firefox-124.0.2/third_party/wasm2c/wasm2c -I/usr/pkgsrc/www/firefox/work/build/dist/include -I/usr/pkg/include/nspr -I/usr/pkg/include/nss -I/usr/pkg/include/nspr -I/usr/pkgsrc/www/firefox/work/build/dist/include/nss -I/usr/pkg/include/pixman-1 -I/usr/pkg/include -DMOZILLA_CLIENT -i
nclude /usr/pkgsrc/www/firefox/work/build/mozilla-config.h -DPNG_NO_ASSEMBLER_CODE -I/usr/include -I/usr/pkg/include -I/usr/pkg/include/nspr -I/usr/pkg/include/libdrm -I/usr/pkg/include/glib-2.0 -I/usr/pkg/include/gio-unix-2.0 -I/usr/pkg/lib/glib-2.0/include -I/usr/pkg/include/ffmpeg6 -I/usr/pkg/include/freetype2 -I/usr/pkg/include/harfbuzz -fno-sized-deallocation -fno-aligned-new -O2 -D_GLIBCXX_INCLUDE_NEXT_C_HEADERS -D__LOCALE_C_ONLY -I/usr/include -I/usr/pkg/include -I/usr/pkg/include/nspr -I/usr/pkg/include/libdrm -I/usr/pkg/include/glib-2.0 -I/usr/pkg/include/gio-unix-2.0 -I/usr/pkg/lib/glib-2.0/include -I/usr/pkg/include/ffmpeg6 -I/usr/pkg/include/freetype2 -I/usr/pkg/include/harfbuzz -fno-exceptions -Dunix -fPIC -DPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -gdwarf-4 -O2 -fomit-frame-pointer -funwind-tables -Wall -Wempty-body -Wignored-qualifiers -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wno-invalid-offseto
f -Wc++2a-compat -Wcomma-subscript -Wvolatile -Wno-error=deprecated -Wduplicated-cond -Wimplicit-fallthrough -Wlogical-op -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=free-nonheap-object -Wno-multistatement-macros -Wno-error=class-memaccess -Wformat -Wformat-overflow=2 -Wno-psabi -Wno-error=builtin-macro-redefined -include /usr/pkgsrc/www/firefox/work/firefox-124.0.2/media/libsoundtouch/src/soundtouch_perms.h -fno-strict-aliasing -ffp-contract=off -MD -MP -MF .deps/rlbox_thread_locals.o.pp /usr/pkgsrc/www/firefox/work/firefox-124.0.2/config/external/rlbox/rlbox_thread_locals.cpp
And this is the preprocessor output for jsimd.c. The THREAD_LOCAL is defined as __thread:
> # 33 "/usr/pkgsrc/www/firefox/work/firefox-124.0.2/media/libjpeg/simd/arm/aarch64/jsimd.c"
> static __thread unsigned int simd_support = ~0;
This is the command for compiling jsimd.c. It's compiled with -fPIC with no explicit -ftls-model, so it's also defaulted to global-dynamic:
> /usr/pkgsrc/www/firefox/work/.cwrapper/bin/cc -std=gnu99 -o jsimd.o -c -I/usr/pkgsrc/www/firefox/work/build/dist/system_wrappers -include /usr/pkgsrc/www/firefox/work/firefox-124.0.2/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -DNDEBUG=1 -DTRIMMED=1 -DHAVE_VLD1_S16_X3 -DHAVE_VLD1_U16_X2 -DHAVE_VLD1Q_U8_X4 -DNEON_INTRINSICS -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -DSTATIC_EXPORTABLE_JS_API -I/usr/pkgsrc/www/firefox/work/firefox-124.0.2/media/libjpeg -I/usr/pkgsrc/www/firefox/work/build/media/libjpeg -I/usr/pkgsrc/www/firefox/work/firefox-124.0.2/media/libjpeg/simd/arm -I/usr/pkgsrc/www/firefox/work/firefox-124.0.2/media/libjpeg/simd/arm/aarch64 -I/usr/pkgsrc/www/firefox/work/build/dist/include -I/usr/pkg/include/nspr -I/usr/pkg/include/nss -I/usr/pkg/include/nspr -I/usr/pkgsrc/www/firefox/work/build/dist/include/nss -I/usr/pkg/include/pixman-1 -I/usr/pkg/include -include /usr/pkgsrc/www/firefox/work/build/mozilla-config.h -DMOZI
LLA_CLIENT -DPNG_NO_ASSEMBLER_CODE -I/usr/include -I/usr/pkg/include -I/usr/pkg/include/nspr -I/usr/pkg/include/libdrm -I/usr/pkg/include/glib-2.0 -I/usr/pkg/include/gio-unix-2.0 -I/usr/pkg/lib/glib-2.0/include -I/usr/pkg/include/ffmpeg6 -I/usr/pkg/include/freetype2 -I/usr/pkg/include/harfbuzz -O2 -D_GLIBCXX_INCLUDE_NEXT_C_HEADERS -D__LOCALE_C_ONLY -I/usr/include -I/usr/pkg/include -I/usr/pkg/include/nspr -I/usr/pkg/include/libdrm -I/usr/pkg/include/glib-2.0 -I/usr/pkg/include/gio-unix-2.0 -I/usr/pkg/lib/glib-2.0/include -I/usr/pkg/include/ffmpeg6 -I/usr/pkg/include/freetype2 -I/usr/pkg/include/harfbuzz -Dunix -fPIC -DPIC -ffunction-sections -fdata-sections -fno-math-errno -pthread -gdwarf-4 -O2 -fomit-frame-pointer -funwind-tables -Wall -Wempty-body -Wignored-qualifiers -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wduplicated-cond -Wlogical-op -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=free-nonheap-object
-Wno-multistatement-macros -Wno-error=class-memaccess -Wformat -Wformat-overflow=2 -Werror=implicit-function-declaration -Wno-psabi -Wno-error=builtin-macro-redefined -fno-strict-aliasing -ffp-contract=off -MD -MP -MF .deps/jsimd.o.pp /usr/pkgsrc/www/firefox/work/firefox-124.0.2/media/libjpeg/simd/arm/aarch64/jsimd.c
I put my set of binary packages for Firefox and its transitive dependencies here. Since it's huge I will probably delete these when the cause of the problem is tracked down and the PR is closed:
https://akari.cielonegro.org/tmp/firefox-bin-pkgs/
This is one of coredumps I got:
https://akari.cielonegro.org/tmp/firefox-bin-pkgs/firefox.core.gz
And this is the output of "LD_DEBUG=1 firefox". I launched firefox, created a new profile, and ran it until a tab crashed. I couldn't find anything obviously wrong though:
https://akari.cielonegro.org/tmp/firefox-bin-pkgs/firefox.ld-debug.log
These are libraries that Firefox dlopen'ed, either directly or indirectly:
> dlopen of (null) 0x1
> dlopen of (null) 0x101
> dlopen of /usr/lib/i18n/libEUC.so.5.0 0x1
> dlopen of /usr/lib/i18n/libUTF8.so.5.0 0x1
> dlopen of /usr/lib/i18n/libiconv_std.so.5.0 0x1
> dlopen of /usr/lib/i18n/libmapper_646.so.5.0 0x1
> dlopen of /usr/lib/i18n/libmapper_parallel.so.5.0 0x1
> dlopen of /usr/lib/i18n/libmapper_serial.so.5.0 0x1
> dlopen of /usr/lib/i18n/libmapper_std.so.5.0 0x1
> dlopen of /usr/lib/i18n/libmapper_zone.so.5.0 0x1
> dlopen of /usr/pkg/lib/dri/swrast_dri.so 0x102
> dlopen of /usr/pkg/lib/firefox/libgkcodecs.so 0x101
> dlopen of /usr/pkg/lib/firefox/libipcclientcerts.so 0x1
> dlopen of /usr/pkg/lib/firefox/liblgpllibs.so 0x101
> dlopen of /usr/pkg/lib/firefox/libmozgtk.so 0x101
> dlopen of /usr/pkg/lib/firefox/libmozsqlite3.so 0x101
> dlopen of /usr/pkg/lib/firefox/libmozwayland.so 0x101
> dlopen of /usr/pkg/lib/firefox/libosclientcerts.so 0x1
> dlopen of /usr/pkg/lib/firefox/libxul.so 0x101
> dlopen of /usr/pkg/lib/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-svg.so 0x1
> dlopen of /usr/pkg/lib/gio/modules/libdconfsettings.so 0x1
> dlopen of /usr/pkg/lib/gtk-3.0/3.0.0/immodules/im-uim.so 0x1
> dlopen of /usr/pkg/lib/nss/libfreebl3.so 0x202
> dlopen of /usr/pkg/lib/nss/libnssckbi.so 0x1
> dlopen of /usr/pkg/lib/nss/libsoftokn3.so 0x202
> dlopen of /usr/pkg/lib/uim/plugin/libuim-skk.so 0x2
> dlopen of /usr/pkg/lib/uim/plugin/libuim-sqlite3.so 0x2
> dlopen of /usr/pkg/lib/uim/plugin/libuim-xkb.so 0x2
> dlopen of libEGL.so 0x1
> dlopen of libGL.so 0x1
> dlopen of libGL.so.1 0x102
> dlopen of libXcursor.so.1 0x1
> dlopen of libXss.so.1 0x1
> dlopen of libdrm.so.2 0x201
> dlopen of libgbm.so.1 0x201
> dlopen of libgio-2.0.so.0 0x1
> dlopen of libpulse.so.0 0x1
> dlopen of nss_compat.so.0 0x201
> dlopen of nss_dns.so.0 0x201
> dlopen of nss_files.so.0 0x201
> dlopen of nss_multicast_dns.so.0 0x201
> dlopen of nss_nis.so.0 0x201
>How-To-Repeat:
Launch Firefox, open any website, and see the tab crashes. For the rare occasion it doesn't crash. In that case restart Firefox and try again.
>Fix:
So I speculated that there were some kind of limit in the size of TLS blocks that dlopen(3) could sanely handle, and libxul.so exceeded it. I modified /usr/pkg/bin/firefox based on this speculation:
> #!/bin/sh
> LD_PRELOAD=/usr/pkg/lib/firefox/libxul.so /usr/pkg/lib/firefox/firefox "$@"
To my surprise this actually worked! Firefox hasn't crashed even once since this modification! But of course this is only a workaround. We haven't found the root cause.
>Release-Note:
>Audit-Trail:
From: PHO <pho@cielonegro.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: lib/58154: aarch64: firefox-124.0.2 crashes very frequently on
NetBSD/aarch64 10.0, likely related to thread-local storage
Date: Mon, 15 Apr 2024 17:39:35 +0900
This is what `readelf -d /usr/pkg/lib/firefox/libxul.so` says:
> Dynamic section at offset 0x7b6cff0 contains 83 entries:
> Tag Type Name/Value
> 0x0000000000000001 (NEEDED) Shared library: [libpthread.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libjemalloc.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libgkcodecs.so]
> 0x0000000000000001 (NEEDED) Shared library: [liblgpllibs.so]
> 0x0000000000000001 (NEEDED) Shared library: [libmozsqlite3.so]
> 0x0000000000000001 (NEEDED) Shared library: [libmozgtk.so]
> 0x0000000000000001 (NEEDED) Shared library: [libmozwayland.so]
> 0x0000000000000001 (NEEDED) Shared library: [libicui18n.so.74]
> 0x0000000000000001 (NEEDED) Shared library: [libicuuc.so.74]
> 0x0000000000000001 (NEEDED) Shared library: [libicudata.so.74]
> 0x0000000000000001 (NEEDED) Shared library: [librt.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libX11.so.6]
> 0x0000000000000001 (NEEDED) Shared library: [libXcomposite.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libXdamage.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libXext.so.6]
> 0x0000000000000001 (NEEDED) Shared library: [libXfixes.so.3]
> 0x0000000000000001 (NEEDED) Shared library: [libXrandr.so.2]
> 0x0000000000000001 (NEEDED) Shared library: [libXrender.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libffi.so.8]
> 0x0000000000000001 (NEEDED) Shared library: [libplds4.so]
> 0x0000000000000001 (NEEDED) Shared library: [libplc4.so]
> 0x0000000000000001 (NEEDED) Shared library: [libnspr4.so]
> 0x0000000000000001 (NEEDED) Shared library: [libz.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libssl3.so]
> 0x0000000000000001 (NEEDED) Shared library: [libsmime3.so]
> 0x0000000000000001 (NEEDED) Shared library: [libnss3.so]
> 0x0000000000000001 (NEEDED) Shared library: [libnssutil3.so]
> 0x0000000000000001 (NEEDED) Shared library: [libfreetype.so.6]
> 0x0000000000000001 (NEEDED) Shared library: [libfontconfig.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libgdk-3.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libharfbuzz.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libpangocairo-1.0.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libpango-1.0.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libgtk-3.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libatk-1.0.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libcairo.so.2]
> 0x0000000000000001 (NEEDED) Shared library: [libcairo-gobject.so.2]
> 0x0000000000000001 (NEEDED) Shared library: [libgdk_pixbuf-2.0.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libglib-2.0.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libintl.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libgobject-2.0.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libgio-2.0.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libwebp.so.7]
> 0x0000000000000001 (NEEDED) Shared library: [libwebpdemux.so.2]
> 0x0000000000000001 (NEEDED) Shared library: [libevent.so.5]
> 0x0000000000000001 (NEEDED) Shared library: [libvpx.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libpixman-1.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libxcb-shm.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libX11-xcb.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libxcb.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libXcursor.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libXi.so.6]
> 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.9]
> 0x0000000000000001 (NEEDED) Shared library: [libm.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libc.so.12]
> 0x000000000000000e (SONAME) Library soname: [libxul.so]
> 0x000000000000000f (RPATH) Library rpath: [/usr/pkg/lib/firefox:/usr/pkg/lib:/usr/pkg/lib/nspr:/usr/pkg/lib/nss:/usr/pkg/lib/ffmpeg6]
> 0x0000000000000019 (INIT_ARRAY) 0x76d3ff0
> 0x000000000000001b (INIT_ARRAYSZ) 1400 (bytes)
> 0x000000000000001a (FINI_ARRAY) 0x76d4568
> 0x000000000000001c (FINI_ARRAYSZ) 8 (bytes)
> 0x0000000000000004 (HASH) 0x288
> 0x0000000000000005 (STRTAB) 0x15e50
> 0x0000000000000006 (SYMTAB) 0x4fc0
> 0x000000000000000a (STRSZ) 66459 (bytes)
> 0x000000000000000b (SYMENT) 24 (bytes)
> 0x0000000000000003 (PLTGOT) 0x7b7d560
> 0x0000000000000002 (PLTRELSZ) 67704 (bytes)
> 0x0000000000000014 (PLTREL) RELA
> 0x0000000000000017 (JMPREL) 0xa206b8
> 0x0000000000000007 (RELA) 0x27df0
> 0x0000000000000008 (RELASZ) 10455240 (bytes)
> 0x0000000000000009 (RELAENT) 24 (bytes)
> 0x000000006ffffffc (VERDEF) 0x27878
> 0x000000006ffffffd (VERDEFNUM) 2
> 0x0000000000000018 (BIND_NOW)
> 0x000000006ffffffb (FLAGS_1) Flags: NOW
> 0x000000006ffffffe (VERNEED) 0x278b0
> 0x000000006fffffff (VERNEEDNUM) 9
> 0x000000006ffffff0 (VERSYM) 0x261ec
> 0x000000006ffffff9 (RELACOUNT) 435507
> 0x0000000000000000 (NULL) 0x0
From: PHO <pho@cielonegro.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: lib/58154: aarch64: firefox-124.0.2 crashes very frequently on
NetBSD/aarch64 10.0, likely related to thread-local storage
Date: Tue, 14 May 2024 21:00:26 +0900
I did it. I have managed to track down the cause.
https://github.com/NetBSD/src/commit/e50ac3613df8e5365faebd39e0d7983f31446204
https://github.com/NetBSD/src/commit/ee895c8f3880c6542c1c7db4c7f77235cfac88d1
https://github.com/NetBSD/src/commit/b00640999a436897bc5be86efac4943056d255a6
Can anybody review these? The second one is the bugfix. With these
patches Firefox completely stopped crashing even without my workaround.
From: Taylor R Campbell <riastradh@NetBSD.org>
To: PHO <pho@cielonegro.org>
Cc: gnats-bugs@netbsd.org,
lib-bug-people@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org
Subject: Re: lib/58154: aarch64: firefox-124.0.2 crashes very frequently on
NetBSD/aarch64 10.0, likely related to thread-local storage
Date: Tue, 14 May 2024 13:34:10 +0000
> Date: Tue, 14 May 2024 21:00:26 +0900
> From: PHO <pho@cielonegro.org>
>=20
> I did it. I have managed to track down the cause.
> =20
> https://github.com/NetBSD/src/commit/e50ac3613df8e5365faebd39e0d7983f3144=
6204
> https://github.com/NetBSD/src/commit/ee895c8f3880c6542c1c7db4c7f77235cfac=
88d1
> https://github.com/NetBSD/src/commit/b00640999a436897bc5be86efac4943056d2=
55a6
>=20
> Can anybody review these? The second one is the bugfix. With these=20
> patches Firefox completely stopped crashing even without my workaround.
Awesome, thanks! Sorry I didn't get around to digging into this one.
We should get a reproducer committed into atf. Have you drafted a
small reproducer that we could adapt?
From: PHO <pho@cielonegro.org>
To: Taylor R Campbell <riastradh@NetBSD.org>
Cc: gnats-bugs@netbsd.org, lib-bug-people@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org
Subject: Re: lib/58154: aarch64: firefox-124.0.2 crashes very frequently on
NetBSD/aarch64 10.0, likely related to thread-local storage
Date: Tue, 14 May 2024 23:23:17 +0900
> Awesome, thanks! Sorry I didn't get around to digging into this one.
> We should get a reproducer committed into atf. Have you drafted a
> small reproducer that we could adapt?
I tried hard, but no I couldn't. The buggy code conditionally branched
based on a value read from an uninitialized part of memory, and most of
the time the value it read was zero, which prevented the bug from rising
up to the surface. The reason why it hit Firefox so frequently was
apparently that Firefox allocated unusually many TLS blocks.
If only we had MSAN...
From: Tobias Nygren <tnn@NetBSD.org>
To: gnats-bugs@netbsd.org
Cc: PHO <pho@cielonegro.org>, riastradh@netbsd.org
Subject: Re: lib/58154: aarch64: firefox-124.0.2 crashes very frequently on
NetBSD/aarch64 10.0, likely related to thread-local storage
Date: Mon, 1 Jul 2024 17:55:27 +0200
On Tue, 14 May 2024 12:05:01 +0000 (UTC)
PHO <pho@cielonegro.org> wrote:
> https://github.com/NetBSD/src/commit/e50ac3613df8e5365faebd39e0d7983f31446204
> https://github.com/NetBSD/src/commit/ee895c8f3880c6542c1c7db4c7f77235cfac88d1
> https://github.com/NetBSD/src/commit/b00640999a436897bc5be86efac4943056d255a6
>
> Can anybody review these? The second one is the bugfix. With these
> patches Firefox completely stopped crashing even without my workaround.
These patches also stop kicad from segfaulting.
I think we should commit them, with a netbsd-10 pullup.
Changes look good to me.
From: Taylor R Campbell <riastradh@NetBSD.org>
To: Tobias Nygren <tnn@NetBSD.org>
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@NetBSD.org, PHO <pho@cielonegro.org>
Subject: Re: lib/58154: aarch64: firefox-124.0.2 crashes very frequently on
NetBSD/aarch64 10.0, likely related to thread-local storage
Date: Mon, 1 Jul 2024 16:32:01 +0000
> Date: Mon, 1 Jul 2024 17:55:27 +0200
> From: Tobias Nygren <tnn@NetBSD.org>
>=20
> On Tue, 14 May 2024 12:05:01 +0000 (UTC)
> PHO <pho@cielonegro.org> wrote:
>=20
> > https://github.com/NetBSD/src/commit/e50ac3613df8e5365faebd39e0d7983f3=
1446204
> > https://github.com/NetBSD/src/commit/ee895c8f3880c6542c1c7db4c7f77235c=
fac88d1
> > https://github.com/NetBSD/src/commit/b00640999a436897bc5be86efac494305=
6d255a6
> > =20
> > Can anybody review these? The second one is the bugfix. With these=20
> > patches Firefox completely stopped crashing even without my workaround.
>=20
> These patches also stop kicad from segfaulting.
> I think we should commit them, with a netbsd-10 pullup.
> Changes look good to me.
The changes look good to me, but do you have an example that will at
least exercise the branch? I.e., a program that exercises the case
tlsdesc->td_tlsindex > DTV_MAX_INDEX(dtv)?
I would like to commit an automatic test so that, even if it doesn't
deterministically crash while we have the bug, we keep the code path
exercised in-tree.
The first commit
https://github.com/NetBSD/src/commit/e50ac3613df8e5365faebd39e0d7983f314462=
04
is fine without any tests, go ahead and commit that now.
The third commit
https://github.com/NetBSD/src/commit/b00640999a436897bc5be86efac4943056d255=
a6
sounds fine, but is there a reference for the interpretation of the
DTV_MAX_INDEX quantity? Like a page number that you can cite in the
Drepper ELF TLS doc https://akkadia.org/drepper/tls.pdf? If so, can
you add a citation as a comment here?
From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/58154 CVS commit: src/libexec/ld.elf_so
Date: Mon, 22 Jul 2024 23:14:25 +0000
Module Name: src
Committed By: riastradh
Date: Mon Jul 22 23:14:25 UTC 2024
Modified Files:
src/libexec/ld.elf_so: README.TLS tls.c
Log Message:
ld.elf_so: Sprinkle comments and references for thread-local storage.
Maybe this will help the TLS business to be less mysterious to the
next traveller to pass by here.
Prompted by PR lib/58154.
To generate a diff of this commit:
cvs rdiff -u -r1.6 -r1.7 src/libexec/ld.elf_so/README.TLS
cvs rdiff -u -r1.19 -r1.20 src/libexec/ld.elf_so/tls.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/58154 CVS commit: src/libexec/ld.elf_so
Date: Mon, 22 Jul 2024 23:15:57 +0000
Module Name: src
Committed By: riastradh
Date: Mon Jul 22 23:15:57 UTC 2024
Modified Files:
src/libexec/ld.elf_so: tls.c
Log Message:
ld.elf_so: Add comments explaining DTV allocation size.
Patch by pho@ for PR lib/58154.
To generate a diff of this commit:
cvs rdiff -u -r1.20 -r1.21 src/libexec/ld.elf_so/tls.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/58154 CVS commit: src/tests/libexec/ld.elf_so
Date: Mon, 22 Jul 2024 23:18:30 +0000
Module Name: src
Committed By: riastradh
Date: Mon Jul 22 23:18:30 UTC 2024
Modified Files:
src/tests/libexec/ld.elf_so: t_tls_extern.c
Log Message:
tests/libexec/ld.elf_so/t_tls_extern: Test PR lib/58154.
To generate a diff of this commit:
cvs rdiff -u -r1.14 -r1.15 src/tests/libexec/ld.elf_so/t_tls_extern.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/58154 CVS commit: src/libexec/ld.elf_so/arch/aarch64
Date: Mon, 22 Jul 2024 23:18:50 +0000
Module Name: src
Committed By: riastradh
Date: Mon Jul 22 23:18:50 UTC 2024
Modified Files:
src/libexec/ld.elf_so/arch/aarch64: rtld_start.S
Log Message:
ld.elf_so aarch64/rtld_start.S: Sprinkle comments.
No functional change intended.
Prompted by PR lib/58154.
To generate a diff of this commit:
cvs rdiff -u -r1.5 -r1.6 src/libexec/ld.elf_so/arch/aarch64/rtld_start.S
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/58154 CVS commit: src
Date: Tue, 23 Jul 2024 18:11:53 +0000
Module Name: src
Committed By: riastradh
Date: Tue Jul 23 18:11:53 UTC 2024
Modified Files:
src/libexec/ld.elf_so/arch/aarch64: rtld_start.S
src/tests/libexec/ld.elf_so: t_tls_extern.c
Log Message:
ld.elf_so aarch64/rtld_start.S: Fix dynamic TLS fast path branch.
Bug found and patch prepared by pho@.
PR lib/58154
To generate a diff of this commit:
cvs rdiff -u -r1.6 -r1.7 src/libexec/ld.elf_so/arch/aarch64/rtld_start.S
cvs rdiff -u -r1.15 -r1.16 src/tests/libexec/ld.elf_so/t_tls_extern.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/58154 CVS commit: src/libexec/ld.elf_so
Date: Tue, 23 Jul 2024 22:00:00 +0000
Module Name: src
Committed By: riastradh
Date: Tue Jul 23 22:00:00 UTC 2024
Modified Files:
src/libexec/ld.elf_so: tls.c
Log Message:
ld.elf_so: Fix off-by-one error in common __tls_get_addr.
DTV_MAX_INDEX(dtv) is an _inclusive_ upper bound, i.e., the actual
maximum value of the allowed indices. It is not an exclusive upper
bound -- it is not the number of allowed indices.
This off-by-one doesn't hurt correctness, but it may hurt performance
by using the slow path when the fast path would be safe.
Found by pho@ in PR lib/58154.
To generate a diff of this commit:
cvs rdiff -u -r1.21 -r1.22 src/libexec/ld.elf_so/tls.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->needs-pullups
State-Changed-By: riastradh@NetBSD.org
State-Changed-When: Tue, 23 Jul 2024 22:10:27 +0000
State-Changed-Why:
fixed in HEAD, needs pullup-10 and probably pullup-9
From: "Havard Eidnes" <he@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/58154 CVS commit: pkgsrc/lang/rust
Date: Wed, 31 Jul 2024 08:05:51 +0000
Module Name: pkgsrc
Committed By: he
Date: Wed Jul 31 08:05:51 UTC 2024
Modified Files:
pkgsrc/lang/rust: Makefile distinfo
Added Files:
pkgsrc/lang/rust/patches:
patch-compiler_rustc__target_src_spec_targets_aarch64__be__unknown__netbsd.rs
patch-compiler_rustc__target_src_spec_targets_aarch64__unknown__netbsd.rs
patch-compiler_rustc__target_src_spec_targets_sparc64__unknown__netbsd.rs
patch-vendor_openssl-src-111.28.1+1.1.1w_openssl_Configurations_10-main.conf
patch-vendor_openssl-src-111.28.1+1.1.1w_src_lib.rs
patch-vendor_openssl-sys-0.9.102_build_find__normal.rs
patch-vendor_openssl-sys-0.9.102_build_main.rs
patch-vendor_openssl-sys-0.9.92_build_find__normal.rs
patch-vendor_openssl-sys-0.9.92_build_main.rs
Log Message:
lang/rust: pull over fixes to rust 1.79 from pkgsrc-wip:
1) Turn off the use ot thread local storage on NetBSD/aarch64,
due to NetBSD PR#58154 and our desire to support systems
built before the fix was applied.
2) Use the fixed 1.78.0 bootstrap for NetBSD/aarch64 (which
is now built with the workaround from #1 applied).
3) Reinstate the patches to the openssl-sys and openssl-src
vendor crates, and their checksum fixes, taken from pkgsrc-wip
rust 1.78.0 and adapted.
4) Make the NetBSD/sparc64 target spec more similar to the other
sparc64 targets. Does not resolve the cross-build problem, sadly.
To generate a diff of this commit:
cvs rdiff -u -r1.314 -r1.315 pkgsrc/lang/rust/Makefile
cvs rdiff -u -r1.173 -r1.174 pkgsrc/lang/rust/distinfo
cvs rdiff -u -r0 -r1.1 \
pkgsrc/lang/rust/patches/patch-compiler_rustc__target_src_spec_targets_aarch64__be__unknown__netbsd.rs \
pkgsrc/lang/rust/patches/patch-compiler_rustc__target_src_spec_targets_aarch64__unknown__netbsd.rs \
pkgsrc/lang/rust/patches/patch-compiler_rustc__target_src_spec_targets_sparc64__unknown__netbsd.rs \
pkgsrc/lang/rust/patches/patch-vendor_openssl-src-111.28.1+1.1.1w_openssl_Configurations_10-main.conf \
pkgsrc/lang/rust/patches/patch-vendor_openssl-src-111.28.1+1.1.1w_src_lib.rs \
pkgsrc/lang/rust/patches/patch-vendor_openssl-sys-0.9.102_build_find__normal.rs \
pkgsrc/lang/rust/patches/patch-vendor_openssl-sys-0.9.102_build_main.rs \
pkgsrc/lang/rust/patches/patch-vendor_openssl-sys-0.9.92_build_find__normal.rs \
pkgsrc/lang/rust/patches/patch-vendor_openssl-sys-0.9.92_build_main.rs
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: needs-pullups->pending-pullups
State-Changed-By: riastradh@NetBSD.org
State-Changed-When: Tue, 06 Aug 2024 14:21:38 +0000
State-Changed-Why:
pullup-9 #1864 https://releng.netbsd.org/cgi-bin/req-9.cgi?show=1864
pullup-10 #777 https://releng.netbsd.org/cgi-bin/req-10.cgi?show=777
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/58154 CVS commit: [netbsd-10] src
Date: Wed, 7 Aug 2024 11:00:12 +0000
Module Name: src
Committed By: martin
Date: Wed Aug 7 11:00:12 UTC 2024
Modified Files:
src/libexec/ld.elf_so [netbsd-10]: README.TLS tls.c
src/libexec/ld.elf_so/arch/aarch64 [netbsd-10]: rtld_start.S
src/tests/libexec/ld.elf_so [netbsd-10]: t_tls_extern.c
Log Message:
Pull up following revision(s) (requested by riastradh in ticket #777):
libexec/ld.elf_so/tls.c: revision 1.15
libexec/ld.elf_so/arch/aarch64/rtld_start.S: revision 1.6
libexec/ld.elf_so/arch/aarch64/rtld_start.S: revision 1.7
tests/libexec/ld.elf_so/t_tls_extern.c: revision 1.15
tests/libexec/ld.elf_so/t_tls_extern.c: revision 1.16
libexec/ld.elf_so/README.TLS: revision 1.7
libexec/ld.elf_so/tls.c: revision 1.20
libexec/ld.elf_so/tls.c: revision 1.21
Alignment. NFCI.
ld.elf_so: Sprinkle comments and references for thread-local storage.
Maybe this will help the TLS business to be less mysterious to the
next traveller to pass by here.
Prompted by PR lib/58154.
ld.elf_so: Add comments explaining DTV allocation size.
Patch by pho@ for PR lib/58154.
tests/libexec/ld.elf_so/t_tls_extern: Test PR lib/58154.
ld.elf_so aarch64/rtld_start.S: Sprinkle comments.
No functional change intended.
Prompted by PR lib/58154.
ld.elf_so aarch64/rtld_start.S: Fix dynamic TLS fast path branch.
Bug found and patch prepared by pho@.
PR lib/58154
To generate a diff of this commit:
cvs rdiff -u -r1.5.10.1 -r1.5.10.2 src/libexec/ld.elf_so/README.TLS
cvs rdiff -u -r1.14.8.1 -r1.14.8.2 src/libexec/ld.elf_so/tls.c
cvs rdiff -u -r1.5 -r1.5.2.1 src/libexec/ld.elf_so/arch/aarch64/rtld_start.S
cvs rdiff -u -r1.12.2.2 -r1.12.2.3 src/tests/libexec/ld.elf_so/t_tls_extern.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/58154 CVS commit: [netbsd-9] src
Date: Wed, 7 Aug 2024 11:01:57 +0000
Module Name: src
Committed By: martin
Date: Wed Aug 7 11:01:57 UTC 2024
Modified Files:
src/libexec/ld.elf_so [netbsd-9]: README.TLS tls.c
src/libexec/ld.elf_so/arch/aarch64 [netbsd-9]: rtld_start.S
src/tests/libexec/ld.elf_so [netbsd-9]: t_tls_extern.c
Log Message:
Pull up following revision(s) (requested by riastradh in ticket #1864):
libexec/ld.elf_so/tls.c: revision 1.15
libexec/ld.elf_so/arch/aarch64/rtld_start.S: revision 1.6
libexec/ld.elf_so/arch/aarch64/rtld_start.S: revision 1.7
tests/libexec/ld.elf_so/t_tls_extern.c: revision 1.15
tests/libexec/ld.elf_so/t_tls_extern.c: revision 1.16
libexec/ld.elf_so/README.TLS: revision 1.7
libexec/ld.elf_so/tls.c: revision 1.20
libexec/ld.elf_so/tls.c: revision 1.21
Alignment. NFCI.
ld.elf_so: Sprinkle comments and references for thread-local storage.
Maybe this will help the TLS business to be less mysterious to the
next traveller to pass by here.
Prompted by PR lib/58154.
ld.elf_so: Add comments explaining DTV allocation size.
Patch by pho@ for PR lib/58154.
tests/libexec/ld.elf_so/t_tls_extern: Test PR lib/58154.
ld.elf_so aarch64/rtld_start.S: Sprinkle comments.
No functional change intended.
Prompted by PR lib/58154.
ld.elf_so aarch64/rtld_start.S: Fix dynamic TLS fast path branch.
Bug found and patch prepared by pho@.
PR lib/58154
To generate a diff of this commit:
cvs rdiff -u -r1.5.2.1 -r1.5.2.2 src/libexec/ld.elf_so/README.TLS
cvs rdiff -u -r1.12.2.2 -r1.12.2.3 src/libexec/ld.elf_so/tls.c
cvs rdiff -u -r1.4 -r1.4.2.1 src/libexec/ld.elf_so/arch/aarch64/rtld_start.S
cvs rdiff -u -r1.12.4.2 -r1.12.4.3 src/tests/libexec/ld.elf_so/t_tls_extern.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: pending-pullups->closed
State-Changed-By: riastradh@NetBSD.org
State-Changed-When: Wed, 07 Aug 2024 11:46:34 +0000
State-Changed-Why:
fixed in HEAD, pulled up to 9 and 10
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home