NetBSD Problem Report #57831

From www@netbsd.org  Mon Jan  8 23:12:49 2024
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id DC12A1A9238
	for <gnats-bugs@gnats.NetBSD.org>; Mon,  8 Jan 2024 23:12:49 +0000 (UTC)
Message-Id: <20240108231248.4151C1A9239@mollari.NetBSD.org>
Date: Mon,  8 Jan 2024 23:12:48 +0000 (UTC)
From: marcotte@panix.com
Reply-To: marcotte@panix.com
To: gnats-bugs@NetBSD.org
Subject: Memory leaks in libpthread/libc
X-Send-Pr-Version: www-1.0

>Number:         57831
>Category:       lib
>Synopsis:       Memory leaks in libpthread/libc
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    hannken
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Jan 08 23:15:00 +0000 2024
>Closed-Date:    Fri Jun 28 08:33:44 +0000 2024
>Last-Modified:  Fri Jun 28 08:33:44 +0000 2024
>Originator:     Brian Marcotte
>Release:        10.0
>Organization:
Public Access Networks, Corp
>Environment:
NetBSD mail1.panix.com 10.0_RC2 NetBSD 10.0_RC2 (PANIX-STD) #1: Sat Jan  6 00:27:58 EST 2024  root@xxx.panix.com:/misc/obj64/misc/devel/NO-BACKUPS/release-10/src/sys/arch/amd64/compile/PANIX-STD amd64

>Description:
After upgrading NetBSD-10, I've noticed that some daemons using pthread
have slow memory leaks. I've seen the problem in these daemons:

    gpg-agent
    opendmarc
    dkimpy_milter (python3)
    syslog-ng (only in some cases)

Also, we're getting segfaults in nnrpd (from INN) linked with libperl
when the perl code is called.

I attempted to track down when the problem began but it looks like there
may be more than one issue. I think problems may have been introduced in
these places:

2020/01/27:

  1.156 -> 1.157 src/lib/libpthread/pthread.c
  1.98  -> 1.99  src/lib/libpthread/pthread_int.h

2020/06/04:

  1.2   -> 1.3   external/bsd/jemalloc/dist/src/tsd.c
  1.12  -> 1.13  external/bsd/jemalloc/include/jemalloc/internal/jemalloc_internal_defs.h
  1.173 -> 1.174 lib/libpthread/pthread.c

joerg and ad were working on these libraries at those times.

The second set of commits were easy to back out from modern sources. This
fixed the nnrpd/libperl problem. It made the memory leak issue better,
but not fixed.

I also tried enabling JEMALLOC_PROF, but that made all programs segfault
on startup.

I'm in over my head at this point, so I decided to submit what little I
have.

Thanks.

- Brian

>How-To-Repeat:
I found that when running "gpg-agent --daemon", it's memory usage will
grow even if it's not being used in any way. The other programs required
that they were being used.


>Fix:

>Release-Note:

>Audit-Trail:
From: "Juergen Hannken-Illjes" <hannken@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/57831 CVS commit: src/lib/libpthread
Date: Sat, 8 Jun 2024 08:01:49 +0000

 Module Name:	src
 Committed By:	hannken
 Date:		Sat Jun  8 08:01:49 UTC 2024

 Modified Files:
 	src/lib/libpthread: pthread.c

 Log Message:
 Fix resource leak in pthread_create().  Pthread field "pt_lwpctl"
 must not be accessed after _lwp_exit() as the kernel may free
 it, attach it to another thread and change its "lc_curcpu".

 The "pthread__deadqueue" will fill up with pthreads with
 an invalid "pt_lwpctl" and allocated stack that never get
 reused.

 Replace the test "lc_curcpu == LWPCTL_CPU_EXITED" with
 test "_lwp_kill(newthread->pt_lid, 0) == -1 && errno == ESRCH"
 to make sure this thread has finished its _lwp_exit().

 PR lib/57831 "Memory leaks in libpthread/libc"


 To generate a diff of this commit:
 cvs rdiff -u -r1.184 -r1.185 src/lib/libpthread/pthread.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Jaromir Dolecek" <jdolecek@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/57831 CVS commit: src/doc
Date: Sun, 9 Jun 2024 18:46:52 +0000

 Module Name:	src
 Committed By:	jdolecek
 Date:		Sun Jun  9 18:46:51 UTC 2024

 Modified Files:
 	src/doc: CHANGES

 Log Message:
 note Juergen's pthread memory leak fix in CHANGES

 PR lib/57831


 To generate a diff of this commit:
 cvs rdiff -u -r1.3061 -r1.3062 src/doc/CHANGES

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: Brian Marcotte <marcotte@panix.com>
To: gnats-bugs@netbsd.org
Cc: lib-bug-people@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org
Subject: Re: PR/57831 CVS commit: src/doc
Date: Wed, 12 Jun 2024 20:33:47 -0400

 After nearly a week with this patch, I can say that the leak is fixed
 everywhere I was seeing the problem.

 Thanks!

 --
 - Brian

From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/57831 CVS commit: [netbsd-10] src/lib/libpthread
Date: Thu, 20 Jun 2024 18:22:48 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Thu Jun 20 18:22:48 UTC 2024

 Modified Files:
 	src/lib/libpthread [netbsd-10]: pthread.c

 Log Message:
 Pull up following revision(s) (requested by hannken in ticket #711):

 	lib/libpthread/pthread.c: revision 1.185

 Fix resource leak in pthread_create().  Pthread field "pt_lwpctl"
 must not be accessed after _lwp_exit() as the kernel may free
 it, attach it to another thread and change its "lc_curcpu".

 The "pthread__deadqueue" will fill up with pthreads with
 an invalid "pt_lwpctl" and allocated stack that never get
 reused.

 Replace the test "lc_curcpu == LWPCTL_CPU_EXITED" with
 test "_lwp_kill(newthread->pt_lid, 0) == -1 && errno == ESRCH"
 to make sure this thread has finished its _lwp_exit().

 PR lib/57831 "Memory leaks in libpthread/libc"


 To generate a diff of this commit:
 cvs rdiff -u -r1.181.2.1 -r1.181.2.2 src/lib/libpthread/pthread.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

Responsible-Changed-From-To: lib-bug-people->hannken
Responsible-Changed-By: hannken@NetBSD.org
Responsible-Changed-When: Fri, 28 Jun 2024 08:33:44 +0000
Responsible-Changed-Why:
Fixed it.


State-Changed-From-To: open->closed
State-Changed-By: hannken@NetBSD.org
State-Changed-When: Fri, 28 Jun 2024 08:33:44 +0000
State-Changed-Why:
Fixed in head, pullup to -10 complete.


>Unformatted:

Home	PR Database Search