NetBSD Problem Report #60215

From tsutsui@ceres.dti.ne.jp  Tue Apr 28 19:45:28 2026
Return-Path: <tsutsui@ceres.dti.ne.jp>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits)
	 client-signature RSA-PSS (2048 bits))
	(Client CN "mail.netbsd.org", Issuer "R12" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 8ED411A923C
	for <gnats-bugs@gnats.NetBSD.org>; Tue, 28 Apr 2026 19:45:28 +0000 (UTC)
Message-Id: <202604281945.63SJjMH3004816@ceres.dti.ne.jp>
Date: Wed, 29 Apr 2026 04:45:22 +0900 (JST)
From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
Reply-To: tsutsui@ceres.dti.ne.jp
To: gnats-bugs@NetBSD.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: ld.elf_so dlerror() state is not thread-local
X-Send-Pr-Version: 3.95
X-From4GNATS: "Izumi Tsutsui via gnats" <gnats-admin@NetBSD.org>

>Number:         60215
>Category:       lib
>Synopsis:       ld.elf_so dlerror() state is not thread-local
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Apr 28 19:50:00 +0000 2026
>Last-Modified:  Thu Apr 30 13:10:01 +0000 2026
>Originator:     Izumi Tsutsui
>Release:        NetBSD 11.0_RC3
>Organization:
>Environment:
System: NetBSD mirage 11.0_RC3 NetBSD 11.0_RC3 (GENERIC) #22: Sat Apr 25 03:39:02 JST 2026 tsutsui@mirage:/s/netbsd-11/src/sys/arch/i386/compile/GENERIC i386
Architecture: i386
Machine: i386
>Description:
On NetBSD/i386 10.1 and 11.0_RC3, pkgsrc/net/mikutter *sometimes* fails to
start with the following Ruby-GNOME/gobject-introspection error:

```
  GLib::Error Could not locate gtk_events_pending: 'gtk_events_pending':
  /usr/pkg/bin/ruby33: Shared object "nss_files.so.0" not found
```

This error is intermittent.  Running mikutter repeatedly eventually starts
successfully.

ktrace shows that another LWP in the same process reads /etc/nsswitch.conf
and probes NSS modules such as:

```
  /usr/pkg/lib/nss_compat.so.0
  /usr/lib/nss_compat.so.0
  /usr/pkg/lib/nss_nis.so.0
  /usr/lib/nss_nis.so.0
  /usr/pkg/lib/nss_files.so.0
  /usr/lib/nss_files.so.0
  /usr/pkg/lib/nss_dns.so.0
  /usr/lib/nss_dns.so.0
```

These probes fail because these shared objects do not exist.  That failure
is not fatal by itself, because the corresponding NSS backends are provided
by libc on NetBSD.

However, the failure message from the NSS module probe appears to be observed
by an unrelated symbol lookup in another thread.  In this case, GLib/GModule
or gobject-introspection is trying to resolve gtk_events_pending, but receives
the stale or clobbered dlerror() message:

```
  Shared object "nss_files.so.0" not found
```

The gtk_events_pending symbol itself is present in libgtk-3.  The reported
symbol lookup failure therefore appears to be caused by dlerror() state being
shared between threads.

Looking at libexec/ld.elf_so/rtld.c, dlerror() appears to use a single
process-global static variable and dlerror() returns and clears this
global variable:

```
/*
 * Data declarations.
 */
static char    *error_message;	/* Message for dlopen(), or NULL */

```

```
__strong_alias(__dlerror,dlerror)
char *
dlerror(void)
{
	char *msg = error_message;

	error_message = NULL;
	return msg;
}
```

This means that one thread can clear or observe an error produced by another
thread's dlopen()/dlsym() operation.  In a multi-threaded program, this can
make usual sequence as the following unreliable:

```
  dlerror();              /* clear old error */
  sym = dlsym(handle, name);
  error = dlerror();      /* check this lookup */
```

because another thread can call dlopen(), dlsym(), or dlerror()
between these operations.

This seems to explain the intermittent mikutter failure. If the NSS dlopen()
probe failure happens between GLib/GModule's dlerror-clear and dlerror-check
around dlsym("gtk_events_pending"), the unrelated NSS error is treated as a
symbol lookup failure.


>How-To-Repeat:
One reproducer is pkgsrc/net/mikutter on NetBSD/i386 10.1 or 11.0_RC3 with
an /etc/nsswitch.conf that uses compat/nis/files/dns entries, for example:

```
  group:          compat
  group_compat:   nis
  hosts:          files dns
  netgroup:       nis files
  networks:       files nis
  passwd:         compat
  passwd_compat:  nis
  shells:         files
```

Run:

  ktrace -di -f /tmp/mikutter.ktrace mikutter

When it fails, the error dialog reports:

  GLib::Error Could not locate gtk_events_pending: 'gtk_events_pending':
  /usr/pkg/bin/ruby33: Shared object "nss_files.so.0" not found

and kdump shows NSS module probes such as nss_files.so.0, nss_dns.so.0,
nss_compat.so.0, and nss_nis.so.0 from another LWP in the same process.

The failure is intermittent.  Re-running the same command may succeed,
which is consistent with a race on process-global dlerror() state.

>Fix:
If my understanding is correct, error_message in libexec/ld.elf_so/rtld.c
needs to be per-thread rather than process-global.  I would appreciate
comments from people familiar with rtld and libpthread interactions.

---
Izumi Tsutsui

>Audit-Trail:
From: =?UTF-8?Q?J=C3=B6rg_Sonnenberger?= <joerg@bec.de>
To: gnats-bugs@netbsd.org, lib-bug-people@netbsd.org, gnats-admin@netbsd.org,
 netbsd-bugs@netbsd.org
Cc: 
Subject: Re: lib/60215: ld.elf_so dlerror() state is not thread-local
Date: Tue, 28 Apr 2026 22:21:24 +0200

 On 4/28/26 9:50 PM, Izumi Tsutsui via gnats wrote:
 >> Synopsis:       ld.elf_so dlerror() state is not thread-local

 ld.elf_so doesn't itself have any notion of threads, which makes it 
 quite hard to make error messages thread-safe.

 Joerg

From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@netbsd.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: Re: lib/60215: ld.elf_so dlerror() state is not thread-local
Date: Wed, 29 Apr 2026 08:18:16 +0900

 > ld.elf_so doesn't itself have any notion of threads, which makes it 
 > quite hard to make error messages thread-safe.

 >From the application's point of view, this is not only about making
 error messages thread-safe.  If dlerror() state is process-global,
 the usual dlerror()/dlsym()/dlerror() sequence can report an
 unrelated error from another LWP and make the application take
 the wrong error path.

 For comparison, glibc does not use a single process-global dlerror
 state.  Its current implementation stores dlfcn failure state in
 thread-local storage:

 https://sourceware.org/git/?p=glibc.git;a=blob;f=dlfcn/dlerror.h;hb=66f3e9219d8f86b977d9be04ad469b5d72af0da2#l88

 ```
   86 /* Thread-local variable for storing dlfcn failures for subsequent
   87    reporting via dlerror.  */
   88 extern __thread struct dl_action_result *__libc_dlerror_result
   89   attribute_tls_model_ie;
 ```

 ---
 Izumi Tsutsui

From: =?UTF-8?Q?J=C3=B6rg_Sonnenberger?= <joerg@bec.de>
To: gnats-bugs@netbsd.org, lib-bug-people@netbsd.org, gnats-admin@netbsd.org,
 netbsd-bugs@netbsd.org, tsutsui@ceres.dti.ne.jp
Cc: 
Subject: Re: lib/60215: ld.elf_so dlerror() state is not thread-local
Date: Wed, 29 Apr 2026 11:28:14 +0200

 On 4/29/26 1:20 AM, Izumi Tsutsui via gnats wrote:
 > The following reply was made to PR lib/60215; it has been noted by GNATS.
 > 
 > From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
 > To: gnats-bugs@netbsd.org
 > Cc: tsutsui@ceres.dti.ne.jp
 > Subject: Re: lib/60215: ld.elf_so dlerror() state is not thread-local
 > Date: Wed, 29 Apr 2026 08:18:16 +0900
 > 
 >   > ld.elf_so doesn't itself have any notion of threads, which makes it
 >   > quite hard to make error messages thread-safe.
 >   
 >   >From the application's point of view, this is not only about making
 >   error messages thread-safe.  If dlerror() state is process-global,
 >   the usual dlerror()/dlsym()/dlerror() sequence can report an
 >   unrelated error from another LWP and make the application take
 >   the wrong error path.

 Yes, it is possible to return unrelated errors, but for any sane 
 application it shouldn't result in a different error path. That would 
 mean that the application is branching based on dlerror() and not the 
 actual error status of the failing function.

 >   For comparison, glibc does not use a single process-global dlerror
 >   state.  Its current implementation stores dlfcn failure state in
 >   thread-local storage:

 Yes, glibc ties its dynamic linker much tighter to libpthread. We 
 haven't so far. I'm not convinced that the price is worth is here.

 Joerg

From: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>
To: gnats-bugs@netbsd.org
Cc: tsutsui@ceres.dti.ne.jp
Subject: Re: lib/60215: ld.elf_so dlerror() state is not thread-local
Date: Thu, 30 Apr 2026 06:02:23 +0900

 > >   > ld.elf_so doesn't itself have any notion of threads, which makes it
 > >   > quite hard to make error messages thread-safe.
 > >   
 > >   >From the application's point of view, this is not only about making
 > >   error messages thread-safe.  If dlerror() state is process-global,
 > >   the usual dlerror()/dlsym()/dlerror() sequence can report an
 > >   unrelated error from another LWP and make the application take
 > >   the wrong error path.
 > 
 > Yes, it is possible to return unrelated errors, but for any sane 
 > application it shouldn't result in a different error path. That would 
 > mean that the application is branching based on dlerror() and not the 
 > actual error status of the failing function.

 I'd like to make sure I understand your point (i.e. the definitions
 of "sane" and "actual error status") correctly.

 As I understand it, your point is:

  - POSIX specifies dlsym(3) shall return a null pointer if the symbol
    cannot be found

  - POSIX also specifies more detailed diagnostic information shall be
    available through dlerror(3)
     https://pubs.opengroup.org/onlinepubs/9799919799/functions/dlsym.html

  - Therefore, on NetBSD, if dlsym(3) returns a non-NULL pointer,
    GLib/GObject-Introspection should not treat the lookup as failed
    only because dlerror(3) has a non-NULL value

 If that understanding is correct, the immediate ruby/gtk3 failure
 appears to be on the GLib/GObject-Introspection side.  It treats
 the `gtk_events_pending` lookup as failed even though dlsym(3)
 returned a valid non-NULL function pointer.

 Linux dlsym(3) man page claims dlsym() can return NULL without error:
  https://man7.org/linux/man-pages/man3/dlsym.3.html
 Maybe this is the reason why GLib/GObject-Introspection always
 checks dlerror().

 I will look at a pkgsrc-side workaround in that direction.

 One remaining point is documentation on NetBSD side.
 POSIX specifies that it is *implementation-defined*
 whether or not dlerror() is thread-safe:
  https://pubs.opengroup.org/onlinepubs/9799919799/functions/dlerror.html

 If NetBSD's dlerror() state is intentionally process-global,
 I think dlfcn(3) should document that behavior explicitly.
 The current man page does not appear to say whether dlerror(3)
 state is thread-local or process-global in multi-threaded programs:
  https://man.netbsd.org/dlfcn.3

 ---
 Izumi Tsutsui

From: =?UTF-8?Q?J=C3=B6rg_Sonnenberger?= <joerg@bec.de>
To: gnats-bugs@netbsd.org, lib-bug-people@netbsd.org, gnats-admin@netbsd.org,
 netbsd-bugs@netbsd.org, tsutsui@ceres.dti.ne.jp
Cc: 
Subject: Re: lib/60215: ld.elf_so dlerror() state is not thread-local
Date: Thu, 30 Apr 2026 15:09:03 +0200

 On 4/29/26 11:05 PM, Izumi Tsutsui via gnats wrote:
 >   Linux dlsym(3) man page claims dlsym() can return NULL without error:
 >    https://man7.org/linux/man-pages/man3/dlsym.3.html
 >   Maybe this is the reason why GLib/GObject-Introspection always
 >   checks dlerror().

 Please read the NOTES section of that page. I'd argue strongly that none 
 of those conditions exist in sane software and should all just be 
 handled as error without any loss.

 Joerg

Home	PR Database Search