NetBSD Problem Report #54017

From martin@duskware.de  Tue Feb 26 09:57:23 2019
Return-Path: <martin@duskware.de>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id E559B7A17C
	for <gnats-bugs@gnats.NetBSD.org>; Tue, 26 Feb 2019 09:57:23 +0000 (UTC)
Message-Id: <20190226095712.36A9A5CC89B@emmas.aprisoft.de>
Date: Tue, 26 Feb 2019 10:57:12 +0100 (CET)
From: martin@NetBSD.org
Reply-To: martin@NetBSD.org
To: gnats-bugs@NetBSD.org
Subject: jemalloc deadlock?
X-Send-Pr-Version: 3.95

>Number:         54017
>Category:       lib
>Synopsis:       jemalloc deadlock?
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Feb 26 10:00:00 +0000 2019
>Last-Modified:  Thu Mar 14 21:40:00 +0000 2019
>Originator:     Martin Husemann
>Release:        NetBSD 8.99.34
>Organization:
The NetBSD Foundation, Inc.
>Environment:
System: NetBSD painkiller.duskware.de 8.99.34 NetBSD 8.99.34 (GENERIC64) #43: Mon Feb 25 11:11:53 CET 2019 martin@seven-days-to-the-wolves.aprisoft.de:/work/src/sys/arch/evbarm/compile/GENERIC64 evbarm
Architecture: aarch64
Machine: evbarm
>Description:

Randomly, but right now pretty reproducably, python27 "locks up" when
building Firefox (in my case FF65 on aarch64).

It is not exactly clear what leads to this.

Before this happens, apparently python has forked some other processes,
and not collected one dying child status yet:

  772 pts/3  I+    0:01.76 |     `-- make install 
  778 pts/3  I+    0:00.01 |       `-- /bin/sh -c set -e;\t\t\t\t\t if test -n "" &&  /usr/sbin/pkg_info
  959 pts/3  I+    0:02.03 |         `-- /usr/bin/make _MAKE OPSYS OS_VERSION LOWER_OPSYS _PKGSRCDIR PKG
 3757 pts/3  I+    0:01.21 |           `-- python2.7 /usr/pkgobj/www/firefox/work/firefox-65.0/configure
 3229 pts/3  Il+   1:28.44 |             `-- /usr/pkgobj/www/firefox/work/build/_virtualenvs/init/bin/py
21913 pts/3  Il+   0:00.69 |               |-- /usr/pkgobj/www/firefox/work/build/_virtualenvs/init/bin/
22011 pts/3  I+    0:00.04 |               |-- /usr/pkgobj/www/firefox/work/build/_virtualenvs/init/bin/
22382 pts/3  O+   47:46.57 |               |-- /usr/pkgobj/www/firefox/work/build/_virtualenvs/init/bin/
19425 pts/3  Z+    0:00.00 |               | `-- (python)
22659 pts/3  I+    0:00.04 |               `-- /usr/pkgobj/www/firefox/work/build/_virtualenvs/init/bin/

Attaching gdb to the python process shows two threads trying to malloc()
two identical quite huge memory ranges. The process seems not to be out
of VM nor hit resource limits, but the jemalloc base_mtx is kept.
Unfortunately it was not clear to me how to map the mutex owner to a lwp
or something gdb would know.

Since both threads block on the mutex and the backtrace shows no recursion,
maybe some other thread somehow got the mutex but now is gone without
releasing it?

[Switching to LWP 1 of process 22382]
pthread__mutex_pause () at /work/src/lib/libpthread/pthread_mutex.c:228
228	/work/src/lib/libpthread/pthread_mutex.c: No such file or directory.
(gdb) set pagination off
(gdb) bt
#0  pthread__mutex_pause () at /work/src/lib/libpthread/pthread_mutex.c:228
#1  0x0000fffffbc39070 in pthread__mutex_spin (ptm=ptm@entry=0xfffffbc254f8, owner=owner@entry=0xfffff7113000) at /work/src/lib/libpthread/pthread_mutex.c:253
#2  0x0000fffffbc393a4 in pthread__mutex_lock_slow (ptm=ptm@entry=0xfffffbc254f8, ts=ts@entry=0x0) at /work/src/lib/libpthread/pthread_mutex.c:343
#3  0x0000fffffbc39998 in pthread_mutex_lock (ptm=ptm@entry=0xfffffbc254f8) at /work/src/lib/libpthread/pthread_mutex.c:202
#4  0x0000fffffbb4fd04 in base_chunk_node_alloc () at /work/src/lib/libc/stdlib/jemalloc.c:1167
#5  0x0000fffffbb4fd74 in huge_malloc (size=<optimized out>) at /work/src/lib/libc/stdlib/jemalloc.c:2795
#6  0x0000fffffbb50a44 in malloc (size=1048613) at /work/src/lib/libc/stdlib/jemalloc.c:3776
#7  0x0000fffffbd5bcc0 in PyObject_Malloc (nbytes=<optimized out>, nbytes@entry=1048613) at Objects/obmalloc.c:986
#8  0x0000fffffbd64b60 in PyString_FromStringAndSize (str=str@entry=0x0, size=1048576) at Objects/stringobject.c:88
#9  0x0000fffffbdf0f00 in posix_read (self=<optimized out>, args=<optimized out>) at ./Modules/posixmodule.c:6804
#10 0x0000fffffbdbaec8 in ext_do_call (nk=65535, na=-88950752, flags=<optimized out>, pp_stack=0xffffffff8b58, func=0xfffffb8fa3b0) at Python/ceval.c:4683
#11 PyEval_EvalFrameEx (f=f@entry=0xfffffb82b9d8, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#12 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb826730, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#13 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=3, pp_stack=0xffffffff8d40, func=0xfffffb52ade8) at Python/ceval.c:4467
#14 call_function (oparg=<optimized out>, pp_stack=0xffffffff8d40) at Python/ceval.c:4392
#15 PyEval_EvalFrameEx (f=f@entry=0xfffffab2b820, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#16 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb83b7b0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=0, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0xfffff8460ca0, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#17 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=18, pp_stack=0xffffffff8f30, func=0xfffffb52d5f0) at Python/ceval.c:4467
#18 call_function (oparg=<optimized out>, pp_stack=0xffffffff8f30) at Python/ceval.c:4392
#19 PyEval_EvalFrameEx (f=f@entry=0xfffffab2b020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#20 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb826a30, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff8ba5338, argcount=65535, kws=kws@entry=0xfffff8460ca0, kwcount=kwcount@entry=5, defs=0xfffff849a280, defs@entry=0xfffffb645428, defcount=defcount@entry=13, closure=<optimized out>) at Python/ceval.c:3604
#21 0x0000fffffbd418b4 in function_call (func=0xfffffb52d1b8, arg=0xfffff8ba5320, kw=0xfffff849a280) at Objects/funcobject.c:523
#22 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffffb52d1b8, arg=arg@entry=0xfffff8ba5320, kw=kw@entry=0xfffff849a280) at Objects/abstract.c:2547
#23 0x0000fffffbd2a0d4 in instancemethod_call (func=0xfffffb52d1b8, arg=0xfffff8ba5320, kw=0xfffff849a280) at Objects/classobject.c:2600
#24 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff84dfeb0, arg=arg@entry=0xfffff8470810, kw=kw@entry=0xfffff849a280) at Objects/abstract.c:2547
#25 0x0000fffffbd778c8 in slot_tp_init (self=<optimized out>, args=0xfffff8470810, kwds=0xfffff849a280) at Objects/typeobject.c:5860
#26 0x0000fffffbd745c4 in type_call (type=<optimized out>, args=0xfffff8470810, kwds=0xfffff849a280) at Objects/typeobject.c:765
#27 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffffb76b020, arg=arg@entry=0xfffff8470810, kw=kw@entry=0xfffff849a280) at Objects/abstract.c:2547
#28 0x0000fffffbdb852c in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0xffffffff9520, func=0xfffffb76b020) at Python/ceval.c:4589
#29 call_function (oparg=<optimized out>, pp_stack=0xffffffff9520) at Python/ceval.c:4394
#30 PyEval_EvalFrameEx (f=f@entry=0xfffff87e2820, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#31 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=4, pp_stack=0xffffffff9670, func=0xfffff84d1398) at Python/ceval.c:4457
#32 call_function (oparg=<optimized out>, pp_stack=0xffffffff9670) at Python/ceval.c:4392
#33 PyEval_EvalFrameEx (f=f@entry=0xfffff87e5420, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#34 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85b5b30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=1, closure=<optimized out>) at Python/ceval.c:3604
#35 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=5, pp_stack=0xffffffff9860, func=0xfffff84d1668) at Python/ceval.c:4467
#36 call_function (oparg=<optimized out>, pp_stack=0xffffffff9860) at Python/ceval.c:4392
#37 PyEval_EvalFrameEx (f=f@entry=0xfffff87ca020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#38 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85b5b30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=1, closure=<optimized out>) at Python/ceval.c:3604
#39 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=5, pp_stack=0xffffffff9a50, func=0xfffff84d1668) at Python/ceval.c:4467
#40 call_function (oparg=<optimized out>, pp_stack=0xffffffff9a50) at Python/ceval.c:4392
#41 PyEval_EvalFrameEx (f=f@entry=0xfffff87e2c20, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#42 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85b5b30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=1, closure=<optimized out>) at Python/ceval.c:3604
#43 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=5, pp_stack=0xffffffff9c40, func=0xfffff84d1668) at Python/ceval.c:4467
#44 call_function (oparg=<optimized out>, pp_stack=0xffffffff9c40) at Python/ceval.c:4392
#45 PyEval_EvalFrameEx (f=f@entry=0xfffff87b1420, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#46 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85b5b30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0xfffffb905090, defcount=1, closure=<optimized out>) at Python/ceval.c:3604
#47 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=4, pp_stack=0xffffffff9e30, func=0xfffff84d1668) at Python/ceval.c:4467
#48 call_function (oparg=<optimized out>, pp_stack=0xffffffff9e30) at Python/ceval.c:4392
#49 PyEval_EvalFrameEx (f=f@entry=0xfffff8798820, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#50 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=8, pp_stack=0xffffffff9f80, func=0xfffff84cdf50) at Python/ceval.c:4457
#51 call_function (oparg=<optimized out>, pp_stack=0xffffffff9f80) at Python/ceval.c:4392
#52 PyEval_EvalFrameEx (f=f@entry=0xfffff8798020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#53 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=10, pp_stack=0xffffffffa0d0, func=0xfffff84d4aa0) at Python/ceval.c:4457
#54 call_function (oparg=<optimized out>, pp_stack=0xffffffffa0d0) at Python/ceval.c:4392
#55 PyEval_EvalFrameEx (f=f@entry=0xfffff8791c20, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#56 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85adbb0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff846d4e0, argcount=65535, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=0x6, defs@entry=0xfffff856c458, defcount=defcount@entry=7, closure=<optimized out>) at Python/ceval.c:3604
#57 0x0000fffffbd417d0 in function_call (func=0xfffff881f2a8, arg=0xfffff846d4c8, kw=0x0) at Objects/funcobject.c:523
#58 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff881f2a8, arg=arg@entry=0xfffff846d4c8, kw=kw@entry=0x0) at Objects/abstract.c:2547
#59 0x0000fffffbdb6ce0 in ext_do_call (nk=65535, na=-23664, flags=<optimized out>, pp_stack=0xffffffffa388, func=0xfffff881f2a8) at Python/ceval.c:4686
#60 PyEval_EvalFrameEx (f=f@entry=0xfffff8ac8bc0, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#61 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85ad1b0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff846d2d8, argcount=65535, kws=kws@entry=0xfffffba10068, kwcount=kwcount@entry=0, defs=0xfffff90fbdc4, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>) at Python/ceval.c:3604
#62 0x0000fffffbd418b4 in function_call (func=0xfffff8472b18, arg=0xfffff846d2c0, kw=0xfffff847c398) at Objects/funcobject.c:523
#63 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8472b18, arg=arg@entry=0xfffff846d2c0, kw=kw@entry=0x0) at Objects/abstract.c:2547
#64 0x0000fffffbdb6ce0 in ext_do_call (nk=65535, na=-22976, flags=<optimized out>, pp_stack=0xffffffffa638, func=0xfffff8472b18) at Python/ceval.c:4686
#65 PyEval_EvalFrameEx (f=f@entry=0xfffff8478620, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#66 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff9010db0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff8cbba40, argcount=65535, kws=kws@entry=0xfffffba10068, kwcount=kwcount@entry=0, defs=0xfffff96d12f2, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>) at Python/ceval.c:3604
#67 0x0000fffffbd418b4 in function_call (func=0xfffff901bed8, arg=0xfffff8cbba28, kw=0xfffff8479168) at Objects/funcobject.c:523
#68 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff901bed8, arg=arg@entry=0xfffff8cbba28, kw=kw@entry=0x1) at Objects/abstract.c:2547
#69 0x0000fffffbdb6ce0 in ext_do_call (nk=65535, na=-117282466, flags=<optimized out>, pp_stack=0xffffffffa8e8, func=0xfffff901bed8) at Python/ceval.c:4686
#70 PyEval_EvalFrameEx (f=f@entry=0xfffff8477ad0, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#71 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0xffffffffaa30, func=0xfffff96ee848) at Python/ceval.c:4457
#72 call_function (oparg=<optimized out>, pp_stack=0xffffffffaa30) at Python/ceval.c:4392
#73 PyEval_EvalFrameEx (f=f@entry=0xfffff897de18, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#74 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0xffffffffab80, func=0xfffff96eeed8) at Python/ceval.c:4457
#75 call_function (oparg=<optimized out>, pp_stack=0xffffffffab80) at Python/ceval.c:4392
#76 PyEval_EvalFrameEx (f=f@entry=0xfffff8631220, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#77 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff8e0f630, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff8cbe800, argcount=65535, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=0xfffffbe76000 <_PyImport_DynLoadFiletab>, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>) at Python/ceval.c:3604
#78 0x0000fffffbd417d0 in function_call (func=0xfffff8e0e8c0, arg=0xfffff8cbe7e8, kw=0x0) at Objects/funcobject.c:523
#79 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8e0e8c0, arg=arg@entry=0xfffff8cbe7e8, kw=kw@entry=0x0) at Objects/abstract.c:2547
#80 0x0000fffffbd2a0d4 in instancemethod_call (func=0xfffff8e0e8c0, arg=0xfffff8cbe7e8, kw=0x0) at Objects/classobject.c:2600
#81 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8e37500, arg=arg@entry=0xfffff8638090, kw=kw@entry=0x0) at Objects/abstract.c:2547
#82 0x0000fffffbd778c8 in slot_tp_init (self=<optimized out>, args=0xfffff8638090, kwds=0x0) at Objects/typeobject.c:5860
#83 0x0000fffffbd745c4 in type_call (type=<optimized out>, args=0xfffff8638090, kwds=0x0) at Objects/typeobject.c:765
#84 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8d86c20, arg=arg@entry=0xfffff8638090, kw=kw@entry=0x0) at Objects/abstract.c:2547
#85 0x0000fffffbdb852c in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0xffffffffb170, func=0xfffff8d86c20) at Python/ceval.c:4589
#86 call_function (oparg=<optimized out>, pp_stack=0xffffffffb170) at Python/ceval.c:4394
#87 PyEval_EvalFrameEx (f=f@entry=0xfffff8477210, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#88 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0xffffffffb2c0, func=0xfffff96ee8c0) at Python/ceval.c:4457
#89 call_function (oparg=<optimized out>, pp_stack=0xffffffffb2c0) at Python/ceval.c:4392
#90 PyEval_EvalFrameEx (f=f@entry=0xfffff8478240, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#91 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0xffffffffb410, func=0xfffff901d410) at Python/ceval.c:4457
#92 call_function (oparg=<optimized out>, pp_stack=0xffffffffb410) at Python/ceval.c:4392
#93 PyEval_EvalFrameEx (f=f@entry=0xfffff95bf020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#94 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff90152b0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=0, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x0, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#95 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=8, pp_stack=0xffffffffb600, func=0xfffff901d488) at Python/ceval.c:4467
#96 call_function (oparg=<optimized out>, pp_stack=0xffffffffb600) at Python/ceval.c:4392
#97 PyEval_EvalFrameEx (f=f@entry=0xfffff9552c20, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#98 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85ad4b0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff84d6518, argcount=65535, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=0x1, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>) at Python/ceval.c:3604
#99 0x0000fffffbd417d0 in function_call (func=0xfffff8472c08, arg=0xfffff84d6500, kw=0x0) at Objects/funcobject.c:523
#100 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8472c08, arg=arg@entry=0xfffff84d6500, kw=kw@entry=0x0) at Objects/abstract.c:2547
#101 0x0000fffffbd2a0d4 in instancemethod_call (func=0xfffff8472c08, arg=0xfffff84d6500, kw=0x0) at Objects/classobject.c:2600
#102 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8bc2e60, arg=arg@entry=0xfffff856ce50, kw=kw@entry=0x0) at Objects/abstract.c:2547
#103 0x0000fffffbd778c8 in slot_tp_init (self=<optimized out>, args=0xfffff856ce50, kwds=0x0) at Objects/typeobject.c:5860
#104 0x0000fffffbd745c4 in type_call (type=<optimized out>, args=0xfffff856ce50, kwds=0x0) at Objects/typeobject.c:765
#105 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff9593420, arg=arg@entry=0xfffff856ce50, kw=kw@entry=0x0) at Objects/abstract.c:2547
#106 0x0000fffffbdb852c in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0xffffffffbbf0, func=0xfffff9593420) at Python/ceval.c:4589
#107 call_function (oparg=<optimized out>, pp_stack=0xffffffffbbf0) at Python/ceval.c:4394
#108 PyEval_EvalFrameEx (f=f@entry=0xfffff873e020, throwflag=<optimized out>) at Python/ceval.c:3009
#109 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#110 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8791820, throwflag=<optimized out>) at Python/ceval.c:2828
#111 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#112 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8d93c20, throwflag=<optimized out>) at Python/ceval.c:2828
#113 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#114 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8d93820, throwflag=<optimized out>) at Python/ceval.c:2828
#115 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#116 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8e95de0, throwflag=<optimized out>) at Python/ceval.c:2828
#117 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#118 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8d92c20, throwflag=<optimized out>) at Python/ceval.c:2828
#119 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#120 0x0000fffffbd4694c in listextend (self=self@entry=0xfffff8e1c710, b=<optimized out>) at Objects/listobject.c:872
#121 0x0000fffffbd46b74 in list_init (self=0xfffff8e1c710, args=<optimized out>, kw=<optimized out>) at Objects/listobject.c:2459
#122 0x0000fffffbd745c4 in type_call (type=<optimized out>, args=0xfffffb8526d0, kwds=0x0) at Objects/typeobject.c:765
#123 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffffbe89158 <PyList_Type>, arg=arg@entry=0xfffffb8526d0, kw=kw@entry=0x0) at Objects/abstract.c:2547
#124 0x0000fffffbdb852c in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0xffffffffc630, func=0xfffffbe89158 <PyList_Type>) at Python/ceval.c:4589
#125 call_function (oparg=<optimized out>, pp_stack=0xffffffffc630) at Python/ceval.c:4394
#126 PyEval_EvalFrameEx (f=f@entry=0xfffff8d13420, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#127 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff9335eb0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffffba10068, argcount=65535, kws=kws@entry=0xfffff91dbf80, kwcount=kwcount@entry=7, defs=0xfffffab70e8d, defs@entry=0xfffff91b51d0, defcount=defcount@entry=8, closure=<optimized out>) at Python/ceval.c:3604
#128 0x0000fffffbd418b4 in function_call (func=0xfffff8e8e6e0, arg=0xfffffba10050, kw=0xfffff8e945c8) at Objects/funcobject.c:523
#129 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8e8e6e0, arg=arg@entry=0xfffffba10050, kw=kw@entry=0xfffffac5aed8) at Objects/abstract.c:2547
#130 0x0000fffffbdb6ce0 in ext_do_call (nk=65535, na=-74428272, flags=<optimized out>, pp_stack=0xffffffffc8e8, func=0xfffff8e8e6e0) at Python/ceval.c:4686
#131 PyEval_EvalFrameEx (f=f@entry=0xfffff9516c20, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#132 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb2feb30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=0, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#133 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=1, pp_stack=0xffffffffcad0, func=0xfffffac5aed8) at Python/ceval.c:4467
#134 call_function (oparg=<optimized out>, pp_stack=0xffffffffcad0) at Python/ceval.c:4392
#135 PyEval_EvalFrameEx (f=f@entry=0xfffffaa2d820, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#136 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb2ed3b0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0xfffffb8d5168, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#137 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=1, pp_stack=0xffffffffccc0, func=0xfffffac5a938) at Python/ceval.c:4467
#138 call_function (oparg=<optimized out>, pp_stack=0xffffffffccc0) at Python/ceval.c:4392
#139 PyEval_EvalFrameEx (f=f@entry=0xfffffaeac960, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#140 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=co@entry=0xfffffb2fecb0, globals=globals@entry=0xfffffb8d5168, locals=locals@entry=0xfffffb8d5168, args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=0xfffffbdda4ac <PyRun_SimpleFileExFlags+220>, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>, closure@entry=0x0) at Python/ceval.c:3604
#141 0x0000fffffbdbdbfc in PyEval_EvalCode (co=co@entry=0xfffffb2fecb0, globals=globals@entry=0xfffffb8d5168, locals=locals@entry=0xfffffb8d5168) at Python/ceval.c:669
#142 0x0000fffffbdd80c0 in run_mod (mod=mod@entry=0xfffffab6b588, filename=filename@entry=0xffffffffd908 "/usr/pkgobj/www/firefox/work/firefox-65.0/configure.py", globals=globals@entry=0xfffffb8d5168, locals=locals@entry=0xfffffb8d5168, flags=flags@entry=0xffffffffce40, arena=arena@entry=0xfffffab09060) at Python/pythonrun.c:1385
#143 0x0000fffffbdd92f8 in PyRun_FileExFlags (fp=fp@entry=0xfffffbc25640, filename=filename@entry=0xffffffffd908 "/usr/pkgobj/www/firefox/work/firefox-65.0/configure.py", start=start@entry=257, globals=globals@entry=0xfffffb8d5168, locals=locals@entry=0xfffffb8d5168, closeit=closeit@entry=1, flags=flags@entry=0xffffffffce40) at Python/pythonrun.c:1371
#144 0x0000fffffbdda4ac in PyRun_SimpleFileExFlags (fp=fp@entry=0xfffffbc25640, filename=0xffffffffd908 "/usr/pkgobj/www/firefox/work/firefox-65.0/configure.py", closeit=closeit@entry=1, flags=0xffffffffce40, flags@entry=0xffffffffceb0) at Python/pythonrun.c:957
#145 0x0000fffffbdda9b8 in PyRun_AnyFileExFlags (fp=fp@entry=0xfffffbc25640, filename=<optimized out>, closeit=closeit@entry=1, flags=flags@entry=0xffffffffceb0) at Python/pythonrun.c:761
#146 0x0000fffffbdebd50 in Py_Main (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:645
#147 0x0000000200100968 in ___start ()
#148 0x0000fffffbed0af0 in _rtld_start () from /usr/libexec/ld.elf_so
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) up
#1  0x0000fffffbc39070 in pthread__mutex_spin (ptm=ptm@entry=0xfffffbc254f8, owner=owner@entry=0xfffff7113000) at /work/src/lib/libpthread/pthread_mutex.c:253
253	in /work/src/lib/libpthread/pthread_mutex.c
(gdb) p *ptm
$1 = {ptm_magic = 858980355, ptm_errorcheck = 0 '\000', {ptm_ceiling = 0 '\000', ptm_unused = 0 '\000'}, ptm_owner = 0xfffff7113000, ptm_waiters = 0x0, ptm_recursed = 0, ptm_spare2 = 0x0}
(gdb) up
#2  0x0000fffffbc393a4 in pthread__mutex_lock_slow (ptm=ptm@entry=0xfffffbc254f8, ts=ts@entry=0x0) at /work/src/lib/libpthread/pthread_mutex.c:343
343	in /work/src/lib/libpthread/pthread_mutex.c
(gdb) p base_mtx
$2 = {ptm_magic = 858980355, ptm_errorcheck = 0 '\000', {ptm_ceiling = 0 '\000', ptm_unused = 0 '\000'}, ptm_owner = 0xfffff7113000, ptm_waiters = 0x0, ptm_recursed = 0, ptm_spare2 = 0x0}
(gdb) info thread
  Id   Target Id         Frame 
  1    process 22382     pthread__mutex_pause () at /work/src/lib/libpthread/pthread_mutex.c:228
* 2    LWP 1 of process 22382 pthread__mutex_pause () at /work/src/lib/libpthread/pthread_mutex.c:228
(gdb) thread 1
[Switching to thread 1 (process 22382)]
#0  pthread__mutex_pause () at /work/src/lib/libpthread/pthread_mutex.c:228
228	in /work/src/lib/libpthread/pthread_mutex.c
(gdb) bt
#0  pthread__mutex_pause () at /work/src/lib/libpthread/pthread_mutex.c:228
#1  0x0000fffffbc39070 in pthread__mutex_spin (ptm=ptm@entry=0xfffffbc254f8, owner=owner@entry=0xfffff7113000) at /work/src/lib/libpthread/pthread_mutex.c:253
#2  0x0000fffffbc393a4 in pthread__mutex_lock_slow (ptm=ptm@entry=0xfffffbc254f8, ts=ts@entry=0x0) at /work/src/lib/libpthread/pthread_mutex.c:343
#3  0x0000fffffbc39998 in pthread_mutex_lock (ptm=ptm@entry=0xfffffbc254f8) at /work/src/lib/libpthread/pthread_mutex.c:202
#4  0x0000fffffbb4fd04 in base_chunk_node_alloc () at /work/src/lib/libc/stdlib/jemalloc.c:1167
#5  0x0000fffffbb4fd74 in huge_malloc (size=<optimized out>) at /work/src/lib/libc/stdlib/jemalloc.c:2795
#6  0x0000fffffbb50a44 in malloc (size=1048613) at /work/src/lib/libc/stdlib/jemalloc.c:3776
#7  0x0000fffffbd5bcc0 in PyObject_Malloc (nbytes=<optimized out>, nbytes@entry=1048613) at Objects/obmalloc.c:986
#8  0x0000fffffbd64b60 in PyString_FromStringAndSize (str=str@entry=0x0, size=1048576) at Objects/stringobject.c:88
#9  0x0000fffffbdf0f00 in posix_read (self=<optimized out>, args=<optimized out>) at ./Modules/posixmodule.c:6804
#10 0x0000fffffbdbaec8 in ext_do_call (nk=65535, na=-88950752, flags=<optimized out>, pp_stack=0xffffffff8b58, func=0xfffffb8fa3b0) at Python/ceval.c:4683
#11 PyEval_EvalFrameEx (f=f@entry=0xfffffb82b9d8, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#12 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb826730, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#13 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=3, pp_stack=0xffffffff8d40, func=0xfffffb52ade8) at Python/ceval.c:4467
#14 call_function (oparg=<optimized out>, pp_stack=0xffffffff8d40) at Python/ceval.c:4392
#15 PyEval_EvalFrameEx (f=f@entry=0xfffffab2b820, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#16 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb83b7b0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=0, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0xfffff8460ca0, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#17 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=18, pp_stack=0xffffffff8f30, func=0xfffffb52d5f0) at Python/ceval.c:4467
#18 call_function (oparg=<optimized out>, pp_stack=0xffffffff8f30) at Python/ceval.c:4392
#19 PyEval_EvalFrameEx (f=f@entry=0xfffffab2b020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#20 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb826a30, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff8ba5338, argcount=65535, kws=kws@entry=0xfffff8460ca0, kwcount=kwcount@entry=5, defs=0xfffff849a280, defs@entry=0xfffffb645428, defcount=defcount@entry=13, closure=<optimized out>) at Python/ceval.c:3604
#21 0x0000fffffbd418b4 in function_call (func=0xfffffb52d1b8, arg=0xfffff8ba5320, kw=0xfffff849a280) at Objects/funcobject.c:523
#22 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffffb52d1b8, arg=arg@entry=0xfffff8ba5320, kw=kw@entry=0xfffff849a280) at Objects/abstract.c:2547
#23 0x0000fffffbd2a0d4 in instancemethod_call (func=0xfffffb52d1b8, arg=0xfffff8ba5320, kw=0xfffff849a280) at Objects/classobject.c:2600
#24 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff84dfeb0, arg=arg@entry=0xfffff8470810, kw=kw@entry=0xfffff849a280) at Objects/abstract.c:2547
#25 0x0000fffffbd778c8 in slot_tp_init (self=<optimized out>, args=0xfffff8470810, kwds=0xfffff849a280) at Objects/typeobject.c:5860
#26 0x0000fffffbd745c4 in type_call (type=<optimized out>, args=0xfffff8470810, kwds=0xfffff849a280) at Objects/typeobject.c:765
#27 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffffb76b020, arg=arg@entry=0xfffff8470810, kw=kw@entry=0xfffff849a280) at Objects/abstract.c:2547
#28 0x0000fffffbdb852c in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0xffffffff9520, func=0xfffffb76b020) at Python/ceval.c:4589
#29 call_function (oparg=<optimized out>, pp_stack=0xffffffff9520) at Python/ceval.c:4394
#30 PyEval_EvalFrameEx (f=f@entry=0xfffff87e2820, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#31 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=4, pp_stack=0xffffffff9670, func=0xfffff84d1398) at Python/ceval.c:4457
#32 call_function (oparg=<optimized out>, pp_stack=0xffffffff9670) at Python/ceval.c:4392
#33 PyEval_EvalFrameEx (f=f@entry=0xfffff87e5420, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#34 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85b5b30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=1, closure=<optimized out>) at Python/ceval.c:3604
#35 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=5, pp_stack=0xffffffff9860, func=0xfffff84d1668) at Python/ceval.c:4467
#36 call_function (oparg=<optimized out>, pp_stack=0xffffffff9860) at Python/ceval.c:4392
#37 PyEval_EvalFrameEx (f=f@entry=0xfffff87ca020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#38 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85b5b30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=1, closure=<optimized out>) at Python/ceval.c:3604
#39 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=5, pp_stack=0xffffffff9a50, func=0xfffff84d1668) at Python/ceval.c:4467
#40 call_function (oparg=<optimized out>, pp_stack=0xffffffff9a50) at Python/ceval.c:4392
#41 PyEval_EvalFrameEx (f=f@entry=0xfffff87e2c20, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#42 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85b5b30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=1, closure=<optimized out>) at Python/ceval.c:3604
#43 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=5, pp_stack=0xffffffff9c40, func=0xfffff84d1668) at Python/ceval.c:4467
#44 call_function (oparg=<optimized out>, pp_stack=0xffffffff9c40) at Python/ceval.c:4392
#45 PyEval_EvalFrameEx (f=f@entry=0xfffff87b1420, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#46 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85b5b30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0xfffffb905090, defcount=1, closure=<optimized out>) at Python/ceval.c:3604
#47 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=4, pp_stack=0xffffffff9e30, func=0xfffff84d1668) at Python/ceval.c:4467
#48 call_function (oparg=<optimized out>, pp_stack=0xffffffff9e30) at Python/ceval.c:4392
#49 PyEval_EvalFrameEx (f=f@entry=0xfffff8798820, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#50 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=8, pp_stack=0xffffffff9f80, func=0xfffff84cdf50) at Python/ceval.c:4457
#51 call_function (oparg=<optimized out>, pp_stack=0xffffffff9f80) at Python/ceval.c:4392
#52 PyEval_EvalFrameEx (f=f@entry=0xfffff8798020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#53 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=10, pp_stack=0xffffffffa0d0, func=0xfffff84d4aa0) at Python/ceval.c:4457
#54 call_function (oparg=<optimized out>, pp_stack=0xffffffffa0d0) at Python/ceval.c:4392
#55 PyEval_EvalFrameEx (f=f@entry=0xfffff8791c20, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#56 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85adbb0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff846d4e0, argcount=65535, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=0x6, defs@entry=0xfffff856c458, defcount=defcount@entry=7, closure=<optimized out>) at Python/ceval.c:3604
#57 0x0000fffffbd417d0 in function_call (func=0xfffff881f2a8, arg=0xfffff846d4c8, kw=0x0) at Objects/funcobject.c:523
#58 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff881f2a8, arg=arg@entry=0xfffff846d4c8, kw=kw@entry=0x0) at Objects/abstract.c:2547
#59 0x0000fffffbdb6ce0 in ext_do_call (nk=65535, na=-23664, flags=<optimized out>, pp_stack=0xffffffffa388, func=0xfffff881f2a8) at Python/ceval.c:4686
#60 PyEval_EvalFrameEx (f=f@entry=0xfffff8ac8bc0, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#61 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85ad1b0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff846d2d8, argcount=65535, kws=kws@entry=0xfffffba10068, kwcount=kwcount@entry=0, defs=0xfffff90fbdc4, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>) at Python/ceval.c:3604
#62 0x0000fffffbd418b4 in function_call (func=0xfffff8472b18, arg=0xfffff846d2c0, kw=0xfffff847c398) at Objects/funcobject.c:523
#63 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8472b18, arg=arg@entry=0xfffff846d2c0, kw=kw@entry=0x0) at Objects/abstract.c:2547
#64 0x0000fffffbdb6ce0 in ext_do_call (nk=65535, na=-22976, flags=<optimized out>, pp_stack=0xffffffffa638, func=0xfffff8472b18) at Python/ceval.c:4686
#65 PyEval_EvalFrameEx (f=f@entry=0xfffff8478620, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#66 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff9010db0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff8cbba40, argcount=65535, kws=kws@entry=0xfffffba10068, kwcount=kwcount@entry=0, defs=0xfffff96d12f2, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>) at Python/ceval.c:3604
#67 0x0000fffffbd418b4 in function_call (func=0xfffff901bed8, arg=0xfffff8cbba28, kw=0xfffff8479168) at Objects/funcobject.c:523
#68 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff901bed8, arg=arg@entry=0xfffff8cbba28, kw=kw@entry=0x1) at Objects/abstract.c:2547
#69 0x0000fffffbdb6ce0 in ext_do_call (nk=65535, na=-117282466, flags=<optimized out>, pp_stack=0xffffffffa8e8, func=0xfffff901bed8) at Python/ceval.c:4686
#70 PyEval_EvalFrameEx (f=f@entry=0xfffff8477ad0, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#71 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0xffffffffaa30, func=0xfffff96ee848) at Python/ceval.c:4457
#72 call_function (oparg=<optimized out>, pp_stack=0xffffffffaa30) at Python/ceval.c:4392
#73 PyEval_EvalFrameEx (f=f@entry=0xfffff897de18, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#74 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0xffffffffab80, func=0xfffff96eeed8) at Python/ceval.c:4457
#75 call_function (oparg=<optimized out>, pp_stack=0xffffffffab80) at Python/ceval.c:4392
#76 PyEval_EvalFrameEx (f=f@entry=0xfffff8631220, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#77 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff8e0f630, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff8cbe800, argcount=65535, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=0xfffffbe76000 <_PyImport_DynLoadFiletab>, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>) at Python/ceval.c:3604
#78 0x0000fffffbd417d0 in function_call (func=0xfffff8e0e8c0, arg=0xfffff8cbe7e8, kw=0x0) at Objects/funcobject.c:523
#79 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8e0e8c0, arg=arg@entry=0xfffff8cbe7e8, kw=kw@entry=0x0) at Objects/abstract.c:2547
#80 0x0000fffffbd2a0d4 in instancemethod_call (func=0xfffff8e0e8c0, arg=0xfffff8cbe7e8, kw=0x0) at Objects/classobject.c:2600
#81 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8e37500, arg=arg@entry=0xfffff8638090, kw=kw@entry=0x0) at Objects/abstract.c:2547
#82 0x0000fffffbd778c8 in slot_tp_init (self=<optimized out>, args=0xfffff8638090, kwds=0x0) at Objects/typeobject.c:5860
#83 0x0000fffffbd745c4 in type_call (type=<optimized out>, args=0xfffff8638090, kwds=0x0) at Objects/typeobject.c:765
#84 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8d86c20, arg=arg@entry=0xfffff8638090, kw=kw@entry=0x0) at Objects/abstract.c:2547
#85 0x0000fffffbdb852c in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0xffffffffb170, func=0xfffff8d86c20) at Python/ceval.c:4589
#86 call_function (oparg=<optimized out>, pp_stack=0xffffffffb170) at Python/ceval.c:4394
#87 PyEval_EvalFrameEx (f=f@entry=0xfffff8477210, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#88 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0xffffffffb2c0, func=0xfffff96ee8c0) at Python/ceval.c:4457
#89 call_function (oparg=<optimized out>, pp_stack=0xffffffffb2c0) at Python/ceval.c:4392
#90 PyEval_EvalFrameEx (f=f@entry=0xfffff8478240, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#91 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0xffffffffb410, func=0xfffff901d410) at Python/ceval.c:4457
#92 call_function (oparg=<optimized out>, pp_stack=0xffffffffb410) at Python/ceval.c:4392
#93 PyEval_EvalFrameEx (f=f@entry=0xfffff95bf020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#94 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff90152b0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=0, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x0, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#95 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=8, pp_stack=0xffffffffb600, func=0xfffff901d488) at Python/ceval.c:4467
#96 call_function (oparg=<optimized out>, pp_stack=0xffffffffb600) at Python/ceval.c:4392
#97 PyEval_EvalFrameEx (f=f@entry=0xfffff9552c20, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#98 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85ad4b0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff84d6518, argcount=65535, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=0x1, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>) at Python/ceval.c:3604
#99 0x0000fffffbd417d0 in function_call (func=0xfffff8472c08, arg=0xfffff84d6500, kw=0x0) at Objects/funcobject.c:523
#100 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8472c08, arg=arg@entry=0xfffff84d6500, kw=kw@entry=0x0) at Objects/abstract.c:2547
#101 0x0000fffffbd2a0d4 in instancemethod_call (func=0xfffff8472c08, arg=0xfffff84d6500, kw=0x0) at Objects/classobject.c:2600
#102 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8bc2e60, arg=arg@entry=0xfffff856ce50, kw=kw@entry=0x0) at Objects/abstract.c:2547
#103 0x0000fffffbd778c8 in slot_tp_init (self=<optimized out>, args=0xfffff856ce50, kwds=0x0) at Objects/typeobject.c:5860
#104 0x0000fffffbd745c4 in type_call (type=<optimized out>, args=0xfffff856ce50, kwds=0x0) at Objects/typeobject.c:765
#105 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff9593420, arg=arg@entry=0xfffff856ce50, kw=kw@entry=0x0) at Objects/abstract.c:2547
#106 0x0000fffffbdb852c in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0xffffffffbbf0, func=0xfffff9593420) at Python/ceval.c:4589
#107 call_function (oparg=<optimized out>, pp_stack=0xffffffffbbf0) at Python/ceval.c:4394
#108 PyEval_EvalFrameEx (f=f@entry=0xfffff873e020, throwflag=<optimized out>) at Python/ceval.c:3009
#109 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#110 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8791820, throwflag=<optimized out>) at Python/ceval.c:2828
#111 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#112 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8d93c20, throwflag=<optimized out>) at Python/ceval.c:2828
#113 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#114 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8d93820, throwflag=<optimized out>) at Python/ceval.c:2828
#115 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#116 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8e95de0, throwflag=<optimized out>) at Python/ceval.c:2828
#117 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#118 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8d92c20, throwflag=<optimized out>) at Python/ceval.c:2828
#119 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#120 0x0000fffffbd4694c in listextend (self=self@entry=0xfffff8e1c710, b=<optimized out>) at Objects/listobject.c:872
#121 0x0000fffffbd46b74 in list_init (self=0xfffff8e1c710, args=<optimized out>, kw=<optimized out>) at Objects/listobject.c:2459
#122 0x0000fffffbd745c4 in type_call (type=<optimized out>, args=0xfffffb8526d0, kwds=0x0) at Objects/typeobject.c:765
#123 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffffbe89158 <PyList_Type>, arg=arg@entry=0xfffffb8526d0, kw=kw@entry=0x0) at Objects/abstract.c:2547
#124 0x0000fffffbdb852c in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0xffffffffc630, func=0xfffffbe89158 <PyList_Type>) at Python/ceval.c:4589
#125 call_function (oparg=<optimized out>, pp_stack=0xffffffffc630) at Python/ceval.c:4394
#126 PyEval_EvalFrameEx (f=f@entry=0xfffff8d13420, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#127 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff9335eb0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffffba10068, argcount=65535, kws=kws@entry=0xfffff91dbf80, kwcount=kwcount@entry=7, defs=0xfffffab70e8d, defs@entry=0xfffff91b51d0, defcount=defcount@entry=8, closure=<optimized out>) at Python/ceval.c:3604
#128 0x0000fffffbd418b4 in function_call (func=0xfffff8e8e6e0, arg=0xfffffba10050, kw=0xfffff8e945c8) at Objects/funcobject.c:523
#129 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8e8e6e0, arg=arg@entry=0xfffffba10050, kw=kw@entry=0xfffffac5aed8) at Objects/abstract.c:2547
#130 0x0000fffffbdb6ce0 in ext_do_call (nk=65535, na=-74428272, flags=<optimized out>, pp_stack=0xffffffffc8e8, func=0xfffff8e8e6e0) at Python/ceval.c:4686
#131 PyEval_EvalFrameEx (f=f@entry=0xfffff9516c20, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#132 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb2feb30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=0, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#133 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=1, pp_stack=0xffffffffcad0, func=0xfffffac5aed8) at Python/ceval.c:4467
#134 call_function (oparg=<optimized out>, pp_stack=0xffffffffcad0) at Python/ceval.c:4392
#135 PyEval_EvalFrameEx (f=f@entry=0xfffffaa2d820, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#136 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb2ed3b0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0xfffffb8d5168, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#137 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=1, pp_stack=0xffffffffccc0, func=0xfffffac5a938) at Python/ceval.c:4467
#138 call_function (oparg=<optimized out>, pp_stack=0xffffffffccc0) at Python/ceval.c:4392
#139 PyEval_EvalFrameEx (f=f@entry=0xfffffaeac960, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#140 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=co@entry=0xfffffb2fecb0, globals=globals@entry=0xfffffb8d5168, locals=locals@entry=0xfffffb8d5168, args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=0xfffffbdda4ac <PyRun_SimpleFileExFlags+220>, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>, closure@entry=0x0) at Python/ceval.c:3604
#141 0x0000fffffbdbdbfc in PyEval_EvalCode (co=co@entry=0xfffffb2fecb0, globals=globals@entry=0xfffffb8d5168, locals=locals@entry=0xfffffb8d5168) at Python/ceval.c:669
#142 0x0000fffffbdd80c0 in run_mod (mod=mod@entry=0xfffffab6b588, filename=filename@entry=0xffffffffd908 "/usr/pkgobj/www/firefox/work/firefox-65.0/configure.py", globals=globals@entry=0xfffffb8d5168, locals=locals@entry=0xfffffb8d5168, flags=flags@entry=0xffffffffce40, arena=arena@entry=0xfffffab09060) at Python/pythonrun.c:1385
#143 0x0000fffffbdd92f8 in PyRun_FileExFlags (fp=fp@entry=0xfffffbc25640, filename=filename@entry=0xffffffffd908 "/usr/pkgobj/www/firefox/work/firefox-65.0/configure.py", start=start@entry=257, globals=globals@entry=0xfffffb8d5168, locals=locals@entry=0xfffffb8d5168, closeit=closeit@entry=1, flags=flags@entry=0xffffffffce40) at Python/pythonrun.c:1371
#144 0x0000fffffbdda4ac in PyRun_SimpleFileExFlags (fp=fp@entry=0xfffffbc25640, filename=0xffffffffd908 "/usr/pkgobj/www/firefox/work/firefox-65.0/configure.py", closeit=closeit@entry=1, flags=0xffffffffce40, flags@entry=0xffffffffceb0) at Python/pythonrun.c:957
#145 0x0000fffffbdda9b8 in PyRun_AnyFileExFlags (fp=fp@entry=0xfffffbc25640, filename=<optimized out>, closeit=closeit@entry=1, flags=flags@entry=0xffffffffceb0) at Python/pythonrun.c:761
#146 0x0000fffffbdebd50 in Py_Main (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:645
#147 0x0000000200100968 in ___start ()
#148 0x0000fffffbed0af0 in _rtld_start () from /usr/libexec/ld.elf_so


>How-To-Repeat:

Try to build firefox from pkgsrc.

>Fix:
n/a

>Audit-Trail:
From: Andreas Gustafsson <gson@gson.org>
To: martin@NetBSD.org
Cc: gnats-bugs@NetBSD.org
Subject: Re: lib/54017: jemalloc deadlock?
Date: Tue, 26 Feb 2019 13:14:07 +0200

 Martin wrote:
 > Attaching gdb to the python process shows two threads trying to malloc()
 > two identical quite huge memory ranges

 I suspect there is only one thread and a bug in gdb is causing it to
 be displayed twice.  I've seen something similar before trying to
 debug other multithreaded programs, and the two backtraces in the PR
 are bit-for-bit identical.
 -- 
 Andreas Gustafsson, gson@gson.org

From: Martin Husemann <martin@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: lib/54017: jemalloc deadlock?
Date: Tue, 26 Feb 2019 12:50:34 +0000

 Indeed, the process has only a single lwp.

 Martin

From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: lib/54017: jemalloc deadlock?
Date: Tue, 26 Feb 2019 15:19:51 +0100

 I see these often as well. They also appear when building
 mail/thunderbird or security/tor-browser (all using the firefox build
 system).

 I looked at one backtrace recently. It looks different than martin's.
 I connected gdb to the process using the most CPU and here's the
 output:

 (gdb) thread apply all bt

 Thread 3 (LWP 1 of process 3663):
 #0  0x000079d8a14a9cea in ___lwp_park60 () from /usr/lib/libc.so.12
 #1  0x000079d8a1809dbe in pthread_cond_timedwait (cond=0x79d89b1aa568, mutex=0x79d89b1aa590, abstime=0x0)
     at /disk/6/archive/foreign/src/lib/libpthread/pthread_cond.c:169
 #2  0x000079d8a251df76 in PyThread_acquire_lock (lock=0x79d89b1aa560, waitflag=<optimized out>) at Python/thread_pthread.h:440
 #3  0x000079d8a252236a in lock_PyThread_acquire_lock (self=0x79d89acfb110, args=<optimized out>) at ./Modules/threadmodule.c:52
 #4  0x000079d8a24ef4d4 in call_function (oparg=<optimized out>, pp_stack=0x7f7fff6dfc08) at Python/ceval.c:4372
 #5  PyEval_EvalFrameEx (f=f@entry=0x79d89b19f020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
 #6  0x000079d8a24f09ba in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>,
     argcount=<optimized out>, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x79d8a1ef7568, defcount=1, closure=0x0) at Python/ceval.c:3604
 #7  0x000079d8a24ed936 in fast_function (nk=0, na=<optimized out>, n=<optimized out>, pp_stack=0x7f7fff6dfe08, func=0x79d8a1efe398) at Python/ceval.c:4467
 #8  call_function (oparg=<optimized out>, pp_stack=0x7f7fff6dfe08) at Python/ceval.c:4392
 #9  PyEval_EvalFrameEx (f=f@entry=0x79d89acf1410, throwflag=throwflag@entry=0) at Python/ceval.c:3009
 #10 0x000079d8a24f09ba in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>,
     argcount=<optimized out>, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x79d8a1ef76a8, defcount=1, closure=0x0) at Python/ceval.c:3604
 #11 0x000079d8a24ed936 in fast_function (nk=0, na=<optimized out>, n=<optimized out>, pp_stack=0x7f7fff6e0008, func=0x79d8a1efeb90) at Python/ceval.c:4467
 #12 call_function (oparg=<optimized out>, pp_stack=0x7f7fff6e0008) at Python/ceval.c:4392
 #13 PyEval_EvalFrameEx (f=f@entry=0x79d89acf1230, throwflag=throwflag@entry=0) at Python/ceval.c:3009
 #14 0x000079d8a24ef25f in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0x7f7fff6e0158, func=0x79d8a1efeed8) at Python/ceval.c:4457
 #15 call_function (oparg=<optimized out>, pp_stack=0x7f7fff6e0158) at Python/ceval.c:4392
 #16 PyEval_EvalFrameEx (f=f@entry=0x79d89acf5620, throwflag=throwflag@entry=0) at Python/ceval.c:3009
 #17 0x000079d8a24ef25f in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0x7f7fff6e02a8, func=0x79d89ce92b90) at Python/ceval.c:4457
 #18 call_function (oparg=<optimized out>, pp_stack=0x7f7fff6e02a8) at Python/ceval.c:4392
 #19 PyEval_EvalFrameEx (f=f@entry=0x79d89a8f1410, throwflag=throwflag@entry=0) at Python/ceval.c:3009
 #20 0x000079d8a24f09ba in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>,
     argcount=<optimized out>, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x79d89d26f260, defcount=2, closure=0x0) at Python/ceval.c:3604
 #21 0x000079d8a24ed936 in fast_function (nk=0, na=<optimized out>, n=<optimized out>, pp_stack=0x7f7fff6e04a8, func=0x79d89ce926e0) at Python/ceval.c:4467
 #22 call_function (oparg=<optimized out>, pp_stack=0x7f7fff6e04a8) at Python/ceval.c:4392
 #23 PyEval_EvalFrameEx (f=f@entry=0x79d89acf5810, throwflag=throwflag@entry=0) at Python/ceval.c:3009
 #24 0x000079d8a24f09ba in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x79d89caf7da0,
     argcount=<optimized out>, kws=kws@entry=0x79d8a2854068, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, closure=0x0)
     at Python/ceval.c:3604
 #25 0x000079d8a2473132 in function_call (func=0x79d89d2dad70, arg=0x79d89caf7d88, kw=0x79d89acf37f8) at Objects/funcobject.c:523
 #26 0x000079d8a244b565 in PyObject_Call (func=func@entry=0x79d89d2dad70, arg=arg@entry=0x79d89caf7d88, kw=kw@entry=0x79d89acf37f8) at Objects/abstract.c:2547
 #27 0x000079d8a24e78ed in ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7f7fff6e0790, func=0x79d89d2dad70)
     at Python/ceval.c:4686
 #28 PyEval_EvalFrameEx (f=f@entry=0x79d89acf4c90, throwflag=throwflag@entry=0) at Python/ceval.c:3048
 #29 0x000079d8a24ef25f in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0x7f7fff6e08d8, func=0x79d89ea9e0c8) at Python/ceval.c:4457
 #30 call_function (oparg=<optimized out>, pp_stack=0x7f7fff6e08d8) at Python/ceval.c:4392
 #31 PyEval_EvalFrameEx (f=f@entry=0x79d89acf9050, throwflag=throwflag@entry=0) at Python/ceval.c:3009
 #32 0x000079d8a24ef25f in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0x7f7fff6e0a28, func=0x79d89ea9e758) at Python/ceval.c:4457
 #33 call_function (oparg=<optimized out>, pp_stack=0x7f7fff6e0a28) at Python/ceval.c:4392
 #34 PyEval_EvalFrameEx (f=f@entry=0x79d89acf6050, throwflag=throwflag@entry=0) at Python/ceval.c:3009
 #35 0x000079d8a24f09ba in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x79d89c69a0f8,
     argcount=<optimized out>, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, closure=0x0) at Python/ceval.c:3604
 #36 0x000079d8a247304b in function_call (func=0x79d89ce93500, arg=0x79d89c69a0e0, kw=0x0) at Objects/funcobject.c:523
 #37 0x000079d8a244b565 in PyObject_Call (func=func@entry=0x79d89ce93500, arg=arg@entry=0x79d89c69a0e0, kw=kw@entry=0x0) at Objects/abstract.c:2547
 #38 0x000079d8a245b481 in instancemethod_call (func=0x79d89ce93500, arg=0x79d89c69a0e0, kw=0x0) at Objects/classobject.c:2600
 #39 0x000079d8a244b565 in PyObject_Call (func=func@entry=0x79d89ce71500, arg=arg@entry=0x79d89b0d9b10, kw=kw@entry=0x0) at Objects/abstract.c:2547
 #40 0x000079d8a24ab075 in slot_tp_init (self=<optimized out>, args=0x79d89b0d9b10, kwds=0x0) at Objects/typeobject.c:5860
 #41 0x000079d8a24a747a in type_call (type=<optimized out>, args=0x79d89b0d9b10, kwds=0x0) at Objects/typeobject.c:765
 #42 0x000079d8a244b565 in PyObject_Call (func=func@entry=0x79d89c7fcc20, arg=arg@entry=0x79d89b0d9b10, kw=kw@entry=0x0) at Objects/abstract.c:2547
 #43 0x000079d8a24eaab0 in do_call (nk=0, na=<optimized out>, pp_stack=0x7f7fff6e0ea8, func=0x79d89c7fcc20) at Python/ceval.c:4589
 #44 call_function (oparg=<optimized out>, pp_stack=0x7f7fff6e0ea8) at Python/ceval.c:4394
 #45 PyEval_EvalFrameEx (f=f@entry=0x79d89acf43d0, throwflag=throwflag@entry=0) at Python/ceval.c:3009
 #46 0x000079d8a24ef25f in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0x7f7fff6e0ff8, func=0x79d89ea9e140) at Python/ceval.c:4457
 #47 call_function (oparg=<optimized out>, pp_stack=0x7f7fff6e0ff8) at Python/ceval.c:4392
 #48 PyEval_EvalFrameEx (f=f@entry=0x79d89acf5430, throwflag=throwflag@entry=0) at Python/ceval.c:3009
 #49 0x000079d8a24ef25f in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0x7f7fff6e1148, func=0x79d89d2df2a8) at Python/ceval.c:4457
 #50 call_function (oparg=<optimized out>, pp_stack=0x7f7fff6e1148) at Python/ceval.c:4392
 #51 PyEval_EvalFrameEx (f=f@entry=0x79d89c308c20, throwflag=throwflag@entry=0) at Python/ceval.c:3009
 #52 0x000079d8a24f09ba in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>,
     argcount=<optimized out>, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3604
 #53 0x000079d8a24ed936 in fast_function (nk=0, na=<optimized out>, n=<optimized out>, pp_stack=0x7f7fff6e1348, func=0x79d89d2df320) at Python/ceval.c:4467
 #54 call_function (oparg=<optimized out>, pp_stack=0x7f7fff6e1348) at Python/ceval.c:4392
 #55 PyEval_EvalFrameEx (f=f@entry=0x79d89b145c20, throwflag=throwflag@entry=0) at Python/ceval.c:3009
 #56 0x000079d8a24f09ba in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x79d89acce680,
     argcount=<optimized out>, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, closure=0x0) at Python/ceval.c:3604
 #57 0x000079d8a247304b in function_call (func=0x79d89aceed70, arg=0x79d89acce668, kw=0x0) at Objects/funcobject.c:523
 #58 0x000079d8a244b565 in PyObject_Call (func=func@entry=0x79d89aceed70, arg=arg@entry=0x79d89acce668, kw=kw@entry=0x0) at Objects/abstract.c:2547
 #59 0x000079d8a245b481 in instancemethod_call (func=0x79d89aceed70, arg=0x79d89acce668, kw=0x0) at Objects/classobject.c:2600
 #60 0x000079d8a244b565 in PyObject_Call (func=func@entry=0x79d89cabf190, arg=arg@entry=0x79d89ad4a7c0, kw=kw@entry=0x0) at Objects/abstract.c:2547
 #61 0x000079d8a24ab075 in slot_tp_init (self=<optimized out>, args=0x79d89ad4a7c0, kwds=0x0) at Objects/typeobject.c:5860
 #62 0x000079d8a24a747a in type_call (type=<optimized out>, args=0x79d89ad4a7c0, kwds=0x0) at Objects/typeobject.c:765
 #63 0x000079d8a244b565 in PyObject_Call (func=func@entry=0x79d89b148820, arg=arg@entry=0x79d89ad4a7c0, kw=kw@entry=0x0) at Objects/abstract.c:2547
 #64 0x000079d8a24eaab0 in do_call (nk=0, na=<optimized out>, pp_stack=0x7f7fff6e17c8, func=0x79d89b148820) at Python/ceval.c:4589
 #65 call_function (oparg=<optimized out>, pp_stack=0x7f7fff6e17c8) at Python/ceval.c:4394
 #66 PyEval_EvalFrameEx (f=f@entry=0x79d89b11c420, throwflag=<optimized out>) at Python/ceval.c:3009
 #67 0x000079d8a24680e8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
 #68 0x000079d8a24ea65e in PyEval_EvalFrameEx (f=f@entry=0x79d89b11c020, throwflag=<optimized out>) at Python/ceval.c:2828
 #69 0x000079d8a24680e8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
 #70 0x000079d8a24ea65e in PyEval_EvalFrameEx (f=f@entry=0x79d89c7fa020, throwflag=<optimized out>) at Python/ceval.c:2828
 #71 0x000079d8a24680e8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
 #72 0x000079d8a24ea65e in PyEval_EvalFrameEx (f=f@entry=0x79d89c30ec20, throwflag=<optimized out>) at Python/ceval.c:2828
 #73 0x000079d8a24680e8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
 #74 0x000079d8a24ea65e in PyEval_EvalFrameEx (f=f@entry=0x79d89cea5050, throwflag=<optimized out>) at Python/ceval.c:2828
 #75 0x000079d8a24680e8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
 #76 0x000079d8a24ea65e in PyEval_EvalFrameEx (f=f@entry=0x79d89c30e020, throwflag=<optimized out>) at Python/ceval.c:2828
 #77 0x000079d8a24680e8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
 #78 0x000079d8a247867d in listextend (self=self@entry=0x79d89ce9c758, b=<optimized out>) at Objects/listobject.c:872
 #79 0x000079d8a24788e3 in list_init (self=0x79d89ce9c758, args=<optimized out>, kw=<optimized out>) at Objects/listobject.c:2459
 #80 0x000079d8a24a747a in type_call (type=<optimized out>, args=0x79d8a228f550, kwds=0x0) at Objects/typeobject.c:765
 #81 0x000079d8a244b565 in PyObject_Call (func=func@entry=0x79d8a27a3580 <PyList_Type>, arg=arg@entry=0x79d8a228f550, kw=kw@entry=0x0)
     at Objects/abstract.c:2547
 #82 0x000079d8a24eaab0 in do_call (nk=0, na=<optimized out>, pp_stack=0x7f7fff6e22f8, func=0x79d8a27a3580 <PyList_Type>) at Python/ceval.c:4589
 #83 call_function (oparg=<optimized out>, pp_stack=0x7f7fff6e22f8) at Python/ceval.c:4394
 #84 PyEval_EvalFrameEx (f=f@entry=0x79d89c788420, throwflag=throwflag@entry=0) at Python/ceval.c:3009
 #85 0x000079d8a24f09ba in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x79d8a2854068,
     argcount=<optimized out>, kws=kws@entry=0x79d89e262d88, kwcount=kwcount@entry=7, defs=defs@entry=0x79d89dadaf68, defcount=defcount@entry=8, closure=0x0)
     at Python/ceval.c:3604
 #86 0x000079d8a2473132 in function_call (func=0x79d89d24f578, arg=0x79d8a2854050, kw=0x79d89d252280) at Objects/funcobject.c:523
 #87 0x000079d8a244b565 in PyObject_Call (func=func@entry=0x79d89d24f578, arg=arg@entry=0x79d8a2854050, kw=kw@entry=0x79d89d252280) at Objects/abstract.c:2547
 #88 0x000079d8a24e78ed in ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7f7fff6e25e0, func=0x79d89d24f578)
     at Python/ceval.c:4686
 #89 PyEval_EvalFrameEx (f=f@entry=0x79d89cf05c20, throwflag=throwflag@entry=0) at Python/ceval.c:3048
 #90 0x000079d8a24f09ba in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>,
     argcount=<optimized out>, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3604
 #91 0x000079d8a24ed936 in fast_function (nk=0, na=<optimized out>, n=<optimized out>, pp_stack=0x7f7fff6e27d8, func=0x79d8a12bbf50) at Python/ceval.c:4467
 #92 call_function (oparg=<optimized out>, pp_stack=0x7f7fff6e27d8) at Python/ceval.c:4392
 #93 PyEval_EvalFrameEx (f=f@entry=0x79d8a1bca420, throwflag=throwflag@entry=0) at Python/ceval.c:3009
 #94 0x000079d8a24f09ba in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>,
     argcount=<optimized out>, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3604
 #95 0x000079d8a24ed936 in fast_function (nk=0, na=<optimized out>, n=<optimized out>, pp_stack=0x7f7fff6e29d8, func=0x79d8a12bb500) at Python/ceval.c:4467
 #96 call_function (oparg=<optimized out>, pp_stack=0x7f7fff6e29d8) at Python/ceval.c:4392
 #97 PyEval_EvalFrameEx (f=f@entry=0x79d8a1acd5c0, throwflag=throwflag@entry=0) at Python/ceval.c:3009
 #98 0x000079d8a24f09ba in PyEval_EvalCodeEx (co=co@entry=0x79d8a1e7e9b0, globals=globals@entry=0x79d8a2829168, locals=locals@entry=0x79d8a2829168,
     args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0,
     closure=closure@entry=0x0) at Python/ceval.c:3604
 #99 0x000079d8a24f0ad2 in PyEval_EvalCode (co=co@entry=0x79d8a1e7e9b0, globals=globals@entry=0x79d8a2829168, locals=locals@entry=0x79d8a2829168)
     at Python/ceval.c:669
 #100 0x000079d8a250b2c1 in run_mod (mod=<optimized out>, filename=filename@entry=0x7f7fff6e3715 "/scratch/www/firefox/work/firefox-65.0.1/configure.py",
     globals=globals@entry=0x79d8a2829168, locals=locals@entry=0x79d8a2829168, flags=flags@entry=0x7f7fff6e2cb4, arena=arena@entry=0x79d8a1fe10c0)
     at Python/pythonrun.c:1385
 #101 0x000079d8a250c4e7 in PyRun_FileExFlags (fp=fp@entry=0x79d8a177a120,
     filename=filename@entry=0x7f7fff6e3715 "/scratch/www/firefox/work/firefox-65.0.1/configure.py", start=start@entry=257,
     globals=globals@entry=0x79d8a2829168, locals=locals@entry=0x79d8a2829168, closeit=closeit@entry=1, flags=flags@entry=0x7f7fff6e2cb4)
     at Python/pythonrun.c:1371
 #102 0x000079d8a250d7b4 in PyRun_SimpleFileExFlags (fp=fp@entry=0x79d8a177a120,
     filename=0x7f7fff6e3715 "/scratch/www/firefox/work/firefox-65.0.1/configure.py", closeit=closeit@entry=1, flags=flags@entry=0x7f7fff6e2cb4)
     at Python/pythonrun.c:957
 #103 0x000079d8a250dcb2 in PyRun_AnyFileExFlags (fp=fp@entry=0x79d8a177a120, filename=<optimized out>, closeit=closeit@entry=1,
     flags=flags@entry=0x7f7fff6e2cb4) at Python/pythonrun.c:761
 #104 0x000079d8a252062c in Py_Main (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:645
 #105 0x0000000000400aad in ___start ()
 #106 0x00007f7f7ce02c9e in _rtld_unref_dag (root=<optimized out>, root=<optimized out>) at /disk/6/archive/foreign/src/libexec/ld.elf_so/rtld.c:964
 #107 _rtld_unref_dag (root=<optimized out>, root=<optimized out>) at /disk/6/archive/foreign/src/libexec/ld.elf_so/rtld.c:970
 #108 _rtld_unref_dag (root=<optimized out>, root=<optimized out>) at /disk/6/archive/foreign/src/libexec/ld.elf_so/rtld.c:970
 #109 _rtld_unref_dag (root=<optimized out>, root=<optimized out>) at /disk/6/archive/foreign/src/libexec/ld.elf_so/rtld.c:970
 #110 _rtld_unref_dag (root=<optimized out>, root=<optimized out>) at /disk/6/archive/foreign/src/libexec/ld.elf_so/rtld.c:970
 #111 _rtld_unref_dag (root=<optimized out>, root=<optimized out>) at /disk/6/archive/foreign/src/libexec/ld.elf_so/rtld.c:970
 #112 _rtld_unref_dag (root=<optimized out>, root=<optimized out>) at /disk/6/archive/foreign/src/libexec/ld.elf_so/rtld.c:970
 #113 _rtld_unload_object (do_fini_funcs=true, root=0x7f7fff6e5fe0, mask=0x7f7fff6e2d30) at /disk/6/archive/foreign/src/libexec/ld.elf_so/rtld.c:895
 #114 dlclose (handle=<optimized out>) at /disk/6/archive/foreign/src/libexec/ld.elf_so/rtld.c:997
 #115 0x00007f7fff6e3768 in ?? ()
 #116 0x00007f7fff6e377a in ?? ()
 #117 0x00007f7fff6e3792 in ?? ()
 #118 0x00007f7fff6e37a8 in ?? ()
 #119 0x00007f7fff6e37cc in ?? ()
 #120 0x00007f7fff6e37dd in ?? ()
 #121 0x00007f7fff6e37f0 in ?? ()
 #122 0x00007f7fff6e3809 in ?? ()
 #123 0x00007f7fff6e3819 in ?? ()
 #124 0x00007f7fff6e3830 in ?? ()
 #125 0x00007f7fff6e3845 in ?? ()
 #126 0x00007f7fff6e3859 in ?? ()
 #127 0x00007f7fff6e386b in ?? ()
 #128 0x00007f7fff6e387d in ?? ()
 #129 0x00007f7fff6e3890 in ?? ()
 #130 0x00007f7fff6e38a3 in ?? ()
 #131 0x00007f7fff6e38b5 in ?? ()
 #132 0x00007f7fff6e38d1 in ?? ()
 #133 0x00007f7fff6e38e9 in ?? ()
 #134 0x00007f7fff6e38fe in ?? ()
 #135 0x00007f7fff6e391a in ?? ()
 #136 0x00007f7fff6e3932 in ?? ()
 #137 0x00007f7fff6e3945 in ?? ()
 #138 0x00007f7fff6e3955 in ?? ()
 #139 0x00007f7fff6e3963 in ?? ()
 #140 0x00007f7fff6e3975 in ?? ()
 #141 0x00007f7fff6e3984 in ?? ()
 #142 0x00007f7fff6e398f in ?? ()
 #143 0x00007f7fff6e39a2 in ?? ()
 #144 0x00007f7fff6e39ba in ?? ()
 #145 0x00007f7fff6e39ca in ?? ()
 #146 0x00007f7fff6e39e0 in ?? ()
 #147 0x00007f7fff6e39f7 in ?? ()
 #148 0x00007f7fff6e3a0c in ?? ()
 #149 0x00007f7fff6e3a1a in ?? ()
 #150 0x00007f7fff6e3a35 in ?? ()
 #151 0x0000000000000000 in ?? ()

 Thread 2 (LWP 2 of process 3663):
 #0  pthread__mutex_spin (ptm=ptm@entry=0x79d8a1778340, owner=owner@entry=0x79d89a513000) at /disk/6/archive/foreign/src/lib/libpthread/pthread_mutex.c:252
 #1  0x000079d8a1808b2a in pthread__mutex_lock_slow (ptm=ptm@entry=0x79d8a1778340, ts=ts@entry=0x0)
     at /disk/6/archive/foreign/src/lib/libpthread/pthread_mutex.c:343
 #2  0x000079d8a1809007 in pthread_mutex_lock (ptm=ptm@entry=0x79d8a1778340) at /disk/6/archive/foreign/src/lib/libpthread/pthread_mutex.c:202
 #3  0x000079d8a14b8d0f in choose_arena_hard () at /disk/6/archive/foreign/src/lib/libc/stdlib/jemalloc.c:1601
 #4  0x000079d8a14ba346 in choose_arena () at /disk/6/archive/foreign/src/lib/libc/stdlib/jemalloc.c:1634
 #5  imalloc (size=32) at /disk/6/archive/foreign/src/lib/libc/stdlib/jemalloc.c:3067
 #6  0x000079d8a14ba417 in malloc (size=size@entry=32) at /disk/6/archive/foreign/src/lib/libc/stdlib/jemalloc.c:3776
 #7  0x000079d8a251e1a9 in find_key (key=1, value=value@entry=0x79d89b96f4b0) at Python/thread.c:287
 #8  0x000079d8a251e3ce in PyThread_set_key_value (key=<optimized out>, value=value@entry=0x79d89b96f4b0) at Python/thread.c:347
 #9  0x000079d8a250a32f in _PyGILState_NoteThreadState (tstate=tstate@entry=0x79d89b96f4b0) at Python/pystate.c:570
 #10 0x000079d8a250a497 in _PyGILState_NoteThreadState (tstate=0x79d89b96f4b0) at Python/pystate.c:228
 #11 _PyThreadState_Init (tstate=tstate@entry=0x79d89b96f4b0) at Python/pystate.c:226
 #12 0x000079d8a252281c in t_bootstrap (boot_raw=0x79d89b9b58e0) at ./Modules/threadmodule.c:617
 #13 0x000079d8a180b748 in pthread__create_tramp (cookie=0x79d89b1cc000) at /disk/6/archive/foreign/src/lib/libpthread/pthread.c:593
 #14 0x000079d8a148b1b0 in ?? () from /usr/lib/libc.so.12
 #15 0x0000000000000000 in ?? ()

 Thread 1 (process 3663):
 #0  pthread__mutex_spin (ptm=ptm@entry=0x79d8a1778340, owner=owner@entry=0x79d89a513000) at /disk/6/archive/foreign/src/lib/libpthread/pthread_mutex.c:252
 #1  0x000079d8a1808b2a in pthread__mutex_lock_slow (ptm=ptm@entry=0x79d8a1778340, ts=ts@entry=0x0)
     at /disk/6/archive/foreign/src/lib/libpthread/pthread_mutex.c:343
 #2  0x000079d8a1809007 in pthread_mutex_lock (ptm=ptm@entry=0x79d8a1778340) at /disk/6/archive/foreign/src/lib/libpthread/pthread_mutex.c:202
 #3  0x000079d8a14b8d0f in choose_arena_hard () at /disk/6/archive/foreign/src/lib/libc/stdlib/jemalloc.c:1601
 #4  0x000079d8a14ba346 in choose_arena () at /disk/6/archive/foreign/src/lib/libc/stdlib/jemalloc.c:1634
 #5  imalloc (size=32) at /disk/6/archive/foreign/src/lib/libc/stdlib/jemalloc.c:3067
 #6  0x000079d8a14ba417 in malloc (size=size@entry=32) at /disk/6/archive/foreign/src/lib/libc/stdlib/jemalloc.c:3776
 #7  0x000079d8a251e1a9 in find_key (key=1, value=value@entry=0x79d89b96f4b0) at Python/thread.c:287
 #8  0x000079d8a251e3ce in PyThread_set_key_value (key=<optimized out>, value=value@entry=0x79d89b96f4b0) at Python/thread.c:347
 #9  0x000079d8a250a32f in _PyGILState_NoteThreadState (tstate=tstate@entry=0x79d89b96f4b0) at Python/pystate.c:570
 #10 0x000079d8a250a497 in _PyGILState_NoteThreadState (tstate=0x79d89b96f4b0) at Python/pystate.c:228
 #11 _PyThreadState_Init (tstate=tstate@entry=0x79d89b96f4b0) at Python/pystate.c:226
 #12 0x000079d8a252281c in t_bootstrap (boot_raw=0x79d89b9b58e0) at ./Modules/threadmodule.c:617
 #13 0x000079d8a180b748 in pthread__create_tramp (cookie=0x79d89b1cc000) at /disk/6/archive/foreign/src/lib/libpthread/pthread.c:593
 #14 0x000079d8a148b1b0 in ?? () from /usr/lib/libc.so.12
 #15 0x0000000000000000 in ?? ()
 (gdb)


  Thomas

From: Rin Okuyama <rokuyama.rk@gmail.com>
To: gnats-bugs@NetBSD.org
Cc: Jun Ebihara <jun@soum.co.jp>
Subject: Re: lib/54017: jemalloc deadlock?
Date: Sun, 3 Mar 2019 20:24:49 +0900

 Deadlock also occurs for arm when "make configure" in www/firefox.
 While I've never observed deadlocks yet for amd64 and i386.

 With jemalloc-5.1.0nb1 installed from pkgsrc,

 % env LD_PRELOAD=/usr/pkg/lib/libjemalloc.so.2 make configure

 works fine both for aarch64 and arm.

 So, there seems to be problems in jemalloc in base, at least for
 aarch64 and arm.

 It would be great if we could import the latest jemalloc into
 src/external/bsd and marge our local changes into it.

 However, it would be a heavy work...

 Thanks,
 rin

From: matthew green <mrg@eterna.com.au>
To: gnats-bugs@NetBSD.org
Cc: lib-bug-people@netbsd.org, gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org, martin@NetBSD.org
Subject: re: lib/54017: jemalloc deadlock?
Date: Sun, 03 Mar 2019 23:59:43 +1100

 >  It would be great if we could import the latest jemalloc into
 >  src/external/bsd and marge our local changes into it.

 actually, most of our local changes that aren't about our libc
 ABI should be thrown away and any part of them not already done
 in jemalloc upstream re-done from scratch, but i think that
 there is little else needed.

 >  However, it would be a heavy work...

 probably, but not nearly as much as you may guess :-)


 .mrg.

From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: Christos Zoulas <christos@zoulas.com>
Subject: Re: lib/54017: jemalloc deadlock?
Date: Sun, 10 Mar 2019 00:01:32 +0100

 I have installed a userland with the new jemalloc, but not rebuilt all
 packages. Do I have to?

 For thunderbird I see:

 ld: ../../../../memory/mozalloc/Unified_cpp_memory_mozalloc0.o:(.bss.malloc=
 _conf+0x0): multiple definition of `malloc_conf'; ../../../../memory/build/=
 Unified_cpp_memory_build0.o:(.bss.malloc_conf+0x0): first defined here
 ld: ../../../../memory/mozalloc/Unified_cpp_memory_mozalloc0.o:(.bss.malloc=
 _message+0x0): multiple definition of `malloc_message'; ../../../../memory/=
 build/Unified_cpp_memory_build0.o:(.bss.malloc_message+0x0): first defined =
 here

 Not sure if that is relevant, but the build doesn't finish.


 tor-browser (mostly same code as firefox) failed with:

 error: build failed
 gmake[3]: *** [/scratch/security/tor-browser/work/tor-browser/config/rules.=
 mk:979: force-cargo-library-build] Error 101

 (I had not rebuilt rust with the new jemalloc.)

 And the firefox build seems stuck in cargo:
 gmake[3]: Entering directory '/scratch/www/firefox/work/build/js/src/fronte=
 nd/binsource'
 js/src/frontend/binsource/force-cargo-host-program-build
 env   CARGO_TARGET_DIR=3D. RUSTFLAGS=3D'-C opt-level=3D2 ' RUSTC=3D/usr/pkg=
 /bin/rustc RUSTDOC=3D/usr/pkg/bin/rustdoc RUSTFMT=3D/usr/pkg/bin/rustfmt CC=
 _x86_64_unknown_netbsd=3D" /scratch/www/firefox/work/.cwrapper/bin/gcc" CXX=
 _x86_64_unknown_netbsd=3D" /scratch/www/firefox/work/.cwrapper/bin/c++" CFL=
 AGS_x86_64_unknown_netbsd=3D"-I/scratch/www/firefox/work/build/dist/system_=
 wrappers -include /scratch/www/firefox/work/firefox-65.0.2/config/gcc_hidde=
 n.h -DNDEBUG=3D1 -DTRIMMED=3D1 -I/scratch/www/firefox/work/firefox-65.0.2/j=
 s/src/frontend/binsource -I/scratch/www/firefox/work/build/js/src/frontend/=
 binsource -I/scratch/www/firefox/work/build/dist/include -I/usr/pkg/include=
 /nspr -fPIC -DPIC -include /scratch/www/firefox/work/build/js/src/js-confde=
 fs.h -DMOZILLA_CLIENT -g -I/usr/include -I/usr/pkg/include -I/usr/pkg/inclu=
 de/nspr -I/usr/pkg/include/nss/nss -I/usr/pkg/include/libdrm -I/usr/pkg/inc=
 lude/glib/glib-2.0 -I/usr/pkg/include/glib/gio-unix-2.0 -I/usr/pkg/lib/glib=
 -2.0/include -I/usr/pkg/include/ffmpeg4 -I/usr/pkg/include/freetype2 -U_FOR=
 TIFY_SOURCE -D_FORTIFY_SOURCE=3D2 -O2 -I/usr/pkg/include/nspr -D_GLIBCXX_IN=
 CLUDE_NEXT_C_HEADERS -g -D_FORTIFY_SOURCE=3D2 -fstack-check -I/usr/include =
 -I/usr/pkg/include -I/usr/pkg/include/nss/nss -I/usr/pkg/include/libdrm -I/=
 usr/pkg/include/glib/glib-2.0 -I/usr/pkg/include/glib/gio-unix-2.0 -I/usr/p=
 kg/lib/glib-2.0/include -I/usr/pkg/include/ffmpeg4 -I/usr/pkg/include/freet=
 ype2 -U_FORTIFY_SOURCE
 -D_FORTIFY_SOURCE=3D2 -Dunix -ffunction-sections -fdata-sections -fno-math-=
 errno -pthread -pipe -I/usr/pkg/include -O2 -fomit-frame-pointer -funwind-t=
 ables -Wall -Wempty-body -Wignored-qualifiers -Wpointer-arith -Wsign-compar=
 e -Wtype-limits -Wunreachable-code -Wduplicated-cond -Wno-error=3Dmaybe-uni=
 nitialized -Wno-error=3Ddeprecated-declarations -Wno-error=3Darray-bounds -=
 Wno-error=3Dfree-nonheap-object -Wformat -Wformat-overflow=3D2" CXXFLAGS_x8=
 6_64_unknown_netbsd=3D"-I/scratch/www/firefox/work/build/dist/system_wrappe=
 rs -include /scratch/www/firefox/work/firefox-65.0.2/config/gcc_hidden.h -D=
 NDEBUG=3D1 -DTRIMMED=3D1 -I/scratch/www/firefox/work/firefox-65.0.2/js/src/=
 frontend/binsource -I/scratch/www/firefox/work/build/js/src/frontend/binsou=
 rce -I/scratch/www/firefox/work/build/dist/include -I/usr/pkg/include/nspr =
 -fPIC -DPIC -DMOZILLA_CLIENT -include /scratch/www/firefox/work/build/js/sr=
 c/js-confdefs.h -g -I/usr/include -I/usr/pkg/include -I/usr/pkg/include/nsp=
 r -I/usr/pkg/include/nss/nss -I/usr/pkg/include/libdrm -I/usr/pkg/include/g=
 lib/glib-2.0 -I/usr/pkg/include/glib/gio-unix-2.0 -I/usr/pkg/lib/glib-2.0/i=
 nclude -I/usr/pkg/include/ffmpeg4 -I/usr/pkg/include/freetype2 -U_FORTIFY_S=
 OURCE -D_FORTIFY_SOURCE=3D2 -Wall -Wempty-body -Wignored-qualifiers -Woverl=
 oaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-co=
 de -Wwrite-strings -Wno-invalid-offsetof -Wc++1z-compat -Wduplicated-cond -=
 Wimplicit-fallthrough -Wno-error=3Dmaybe-uninitialized -Wno-error=3Ddepreca=
 ted-declarations -Wno-error=3Darray-bounds -Wno-error=3Dfree-nonheap-object=
  -Wformat -Wformat-overflow=3D2 -Wno-noexcept-type -fno-sized-deallocation =
 -O2 -I/usr/pkg/include/nspr -D_GLIBCXX_INCLUDE_NEXT_C_HEADERS -g -D_FORTIFY=
 _SOURCE=3D2 -fstack-check -I/usr/include -I/usr/pkg/include -I/usr/pkg/incl=
 ude/nss/nss -I/usr/pkg/include/libdrm -I/usr/pkg/include/glib/glib-2.0 -I/u=
 sr/pkg/include/glib/gio-unix-2.0 -I/usr/pkg/lib/glib-2.0/include -I/usr/pkg=
 /include/ffmpeg4 -I/usr/pkg/include/freetype2 -D__HAVE_INLINE___ISINF -U_FO=
 RTIFY_SOURCE -D_FORTIFY_SOURCE=3D2 -Dunix -fno-rtti -ffunction-sections -fd=
 ata-sections -fno-exceptions -fno-math-errno -pthread -pipe -I/usr/pkg/incl=
 ude -O2 -fomit-frame-pointer -funwind-tables" AR_x86_64_unknown_netbsd=3D"a=
 r"  MOZ_SRC=3D/scratch/www/firefox/work/firefox-65.0.2 MOZ_DIST=3D/scratch/=
 www/firefox/work/build/dist LIBCLANG_PATH=3D"/usr/pkg/lib" CLANG_PATH=3D"/u=
 sr/pkg/bin/clang" PKG_CONFIG_ALLOW_CROSS=3D1 RUST_BACKTRACE=3Dfull MOZ_TOPO=
 BJDIR=3D/scratch/www/firefox/work/build   /usr/pkg/bin/cargo rustc  --relea=
 se --frozen --manifest-path
 /scratch/www/firefox/work/firefox-65.0.2/js/src/frontend/binsource/Cargo.to=
 ml --bin binsource --target=3Dx86_64-unknown-netbsd
 (no line for leaving that directory in the log.)
 gmake[3]: Leaving directory '/scratch/www/firefox/work/build/gfx/thebes'
 (nothing after that)

 29378 2000      43    0   312M   42M parked/5   0:02  0.00%  0.00% cargo

 I tried looking at it but:

 # gdb /usr/pkg/bin/cargo 29378
 GNU gdb (GDB) 8.0.1
 Copyright (C) 2017 Free Software Foundation, Inc.
 License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.htm=
 l>
 This is free software: you are free to change and redistribute it.
 There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
 and "show warranty" for details.
 This GDB was configured as "x86_64--netbsd".
 Type "show configuration" for configuration details.
 For bug reporting instructions, please see:
 <http://www.gnu.org/software/gdb/bugs/>.
 Find the GDB manual and other documentation resources online at:
 <http://www.gnu.org/software/gdb/documentation/>.
 For help, type "help".
 Type "apropos word" to search for commands related to "word"...
 Reading symbols from /usr/pkg/bin/cargo...done.
 Attaching to program: /usr/pkg/bin/cargo, process 29378
 ptrace: Operation not permitted.
 /usr/pkgsrc/www/firefox/29378: No such file or directory.
 (gdb)

 Cheers,
  Thomas

From: Christos Zoulas <christos@zoulas.com>
To: "gnats-bugs@netbsd.org" <gnats-bugs@NetBSD.org>
Cc: lib-bug-people@netbsd.org,
 gnats-admin@netbsd.org,
 netbsd-bugs@netbsd.org,
 "martin@netbsd.org" <martin@NetBSD.org>
Subject: Re: lib/54017: jemalloc deadlock?
Date: Sat, 9 Mar 2019 18:08:36 -0500

 1. You should not need to recompile any packages, using the new jemalloc
     or not should be transparent. 
 2. Turn off pax mprotect if you want to attach to running processes with gdb
 3. Compile a libc with JEMALLOC_DEBUG defined; most probably it is a
     memory corruption issue that this might detect.

 christos

From: Thomas Klausner <tk@giga.or.at>
To: gnats-bugs@NetBSD.org
Cc: Christos Zoulas <christos@zoulas.com>
Subject: Re: lib/54017: jemalloc deadlock?
Date: Sun, 10 Mar 2019 00:25:06 +0100

 One more thing, wip/emacs-git (and perhaps other emacses?) fails with:

 Loading /scratch/wip/emacs-git/work/emacs/lisp/files.el (source)...
 Wrong type argument: stringp, []
 gmake[1]: *** [Makefile:810: bootstrap-emacs.pdmp] Error 255
 gmake[1]: Leaving directory '/scratch/wip/emacs-git/work/emacs/src'
 gmake: *** [Makefile:423: src] Error 2
 *** Error code 2


  Thomas

From: matthew green <mrg@eterna.com.au>
To: gnats-bugs@NetBSD.org
Cc: lib-bug-people@netbsd.org, gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org, martin@NetBSD.org, christos@netbsd.org
Subject: re: lib/54017: jemalloc deadlock?
Date: Sun, 10 Mar 2019 11:05:00 +1100

 Thomas Klausner writes:
 >  ld: ../../../../memory/mozalloc/Unified_cpp_memory_mozalloc0.o:(.bss.ma=
 lloc_conf+0x0): multiple definition of `malloc_conf'; ../../../../memory/b=
 uild/Unified_cpp_memory_build0.o:(.bss.malloc_conf+0x0): first defined her=
 e
 >  ld: ../../../../memory/mozalloc/Unified_cpp_memory_mozalloc0.o:(.bss.ma=
 lloc_message+0x0): multiple definition of `malloc_message'; ../../../../me=
 mory/build/Unified_cpp_memory_build0.o:(.bss.malloc_message+0x0): first de=
 fined here

 christos wrote:
 > 1. You should not need to recompile any packages, using the new jemalloc
 >    or not should be transparent.
 > 2. Turn off pax mprotect if you want to attach to running processes with=
  gdb
 > 3. Compile a libc with JEMALLOC_DEBUG defined; most probably it is a
 >    memory corruption issue that this might detect.

 the multiple definition problem seems to indicate that joerg's
 idea of hiding the stats functions by default is going to be
 the right solution.

 all these crashes makes me think that new jemalloc should be
 reverted in -current and delayed until netbsd-9 happens, so
 that we have heaps of time to find all the issues, rather
 than shipping a damaged or delayed netbsd-9.


 .mrg.

From: Jason Thorpe <thorpej@me.com>
To: "gnats-bugs@netbsd.org" <gnats-bugs@NetBSD.org>
Cc: lib-bug-people@netbsd.org,
 gnats-admin@netbsd.org,
 netbsd-bugs@netbsd.org,
 "martin@netbsd.org" <martin@NetBSD.org>
Subject: Re: lib/54017: jemalloc deadlock?
Date: Sat, 9 Mar 2019 16:23:10 -0800

 > On Mar 9, 2019, at 4:10 PM, matthew green <mrg@eterna.com.au> wrote:
 >=20
 > the multiple definition problem seems to indicate that joerg's
 > idea of hiding the stats functions by default is going to be
 > the right solution.

 The only symbols exposed by the malloc library should be the ones =
 described by the relevant standards we conform to.  Any of the optional =
 stuff specific to a particular malloc implementation should be relegated =
 to underscore-oblivion.

 > all these crashes makes me think that new jemalloc should be
 > reverted in -current and delayed until netbsd-9 happens, so
 > that we have heaps of time to find all the issues, rather
 > than shipping a damaged or delayed netbsd-9.

 +1

 -- thorpej

From: Rin Okuyama <rokuyama.rk@gmail.com>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@NetBSD.org>
Cc: 
Subject: Re: lib/54017: jemalloc deadlock?
Date: Mon, 11 Mar 2019 14:38:46 +0900

 Christos, thank you so much for your hard work!

 Now, deadlock with "make configure" in pkgsrc/www/firefox
 disappears on aarch64 (at least for > 10 tries).

 For problems reported by wiz:

 (1) Build failure of Thunderbird due to multiple definition of
 `malloc_foo': already fixed.

 (2) Build failure of Firefox due to stuck in cargo:

 I guess this is not due to new jemalloc. I have observed similar
 failures with old jemalloc; cargo sometimes deadlock randomly,
 and stop it by ^C and re-run make fixes the problem. Also, no
 assertion failures are detected with new jemalloc with
 JEMALLOC_DEBUG.

 (3) Build failure of pkgsrc/wip/emacs-git:

 With JEMALLOC_DEBUG enabled, it crashes earlier:

 ----
 Loading window (source)...
 <jemalloc>: /build/src/external/bsd/jemalloc/lib/../dist/src/arena.c:271: Failed assertion: "bitmap_get(slab_data->bitmap, &bin_info->bitmap_info, regind)"
 gmake[1]: *** [Makefile:810: bootstrap-emacs.pdmp] Abort trap (core dumped)
 gmake[1]: Leaving directory '/build/pkgsrc/wip/emacs-git/work.x86_64/emacs/src'
 gmake: *** [Makefile:423: src] Error 2
 *** Error code 2
 ----

 However, it is unclear whether this is due to bug in new jemalloc
 or that in emacs. Note that pkgsrc/editors/emacs can be build and
 works fine with new jemalloc.

 For test failures:

 https://mail-index.netbsd.org/current-users/2019/03/10/msg035279.html

 I do not observe tmpfs ones on amd64, however test for libc causes
 kernel panic on amd64

 ----
 pmap_unwire: wiring for pmap 0xffffa08d42f93d28 va 0x7827c6b86000did not change!
 panic: kernel diagnostic assertion "pg->wire_count > 0" failed: file "../../../.
 ./uvm/uvm_page.c", line 1594
 cpu8: Begin traceback...
 vpanic() at netbsd:vpanic+0x143
 stge_eeprom_wait.isra.4() at netbsd:stge_eeprom_wait.isra.4
 uvm_fault_upper_enter.isra.5() at netbsd:uvm_fault_upper_enter.isra.5+0x1c6
 uvm_fault_internal() at netbsd:uvm_fault_internal+0x14ae
 uvm_fault_wire() at netbsd:uvm_fault_wire+0x56
 uvm_map_pageable() at netbsd:uvm_map_pageable+0x220
 sys_mlock() at netbsd:sys_mlock+0xa3
 syscall() at netbsd:syscall+0x15b
 --- syscall (number 203) ---
 7827c604307a:
 cpu8: End traceback...
 uvm_fault(0xffffa08cbc513008, 0x0, 2) -> e
 ----

 in a similar manner to i386

 https://releng.netbsd.org/b5reports/i386/2019/2019.03.10.19.47.03/test.log

 I don't know this is related to jemalloc or not.

 Thanks,
 rin

From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: lib/54017: jemalloc deadlock?
Date: Mon, 11 Mar 2019 23:33:42 +0100

 I can confirm that thunderbird and firefox built fine for me.

 I see a new problem with a libc from today, a repeatable pbulk-scan
 core dump that did not happen before the userland update. pbulk-scan
 binary was from 2017, though I've just rebuilt it to get symbols.

 # /usr/pkg_bulk/bin/bulkbuild
 Warning: All log files of the previous pbulk run will be
 removed in 5 seconds. If you want to abort, press Ctrl-C.
 Removing old scan results
 Scanning...
 .................................................. 50/211
 .................................................. 100/211
 .................................................. 150/211
 .................................................. 200/211
 ...........[1]   Segmentation fault (core dumped) ${pscan} -v -l -I ${pscan_start_script} -m ${m...
 # gdb /usr/pkg_bulk/bin/pbulk-scan pbulk-scan.core  
 GNU gdb (GDB) 8.0.1
 Copyright (C) 2017 Free Software Foundation, Inc.
 License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 This is free software: you are free to change and redistribute it.
 There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
 and "show warranty" for details.
 This GDB was configured as "x86_64--netbsd".
 Type "show configuration" for configuration details.
 For bug reporting instructions, please see:
 <http://www.gnu.org/software/gdb/bugs/>.
 Find the GDB manual and other documentation resources online at:
 <http://www.gnu.org/software/gdb/documentation/>.
 For help, type "help".
 Type "apropos word" to search for commands related to "word"...
 Reading symbols from /usr/pkg_bulk/bin/pbulk-scan...done.
 [New process 1]
 Core was generated by `pbulk-scan'.
 Program terminated with signal SIGSEGV, Segmentation fault.
 #0  0x000000004d804c5b in event_dispatch () at event.c:348

 warning: Source file is more recent than executable.
 348                         iter < last_iter && ev && (next = LIST_NEXT(ev, ev_link), 1);
 (gdb) bt
 #0  0x000000004d804c5b in event_dispatch () at event.c:348
 #1  0x000000004d80354b in master_mode (master_port=0x7f7fff20542f "192.168.0.33:2001", 
     start_script=0x7f7fff2053fe "/usr/pkg_bulk/libexec/pbulk/scan-client-start") at master.c:284
 #2  0x000000004d805468 in main (argc=2, argv=0x7f7fff204d88) at pscan.c:155
 (gdb) l
 343             if (ret == 0 && timeout == 0)
 344                     return;
 345             if (ret > 0) {
 346                     iter = poll_list;
 347                     for (ev = LIST_FIRST(&all_events);
 348                         iter < last_iter && ev && (next = LIST_NEXT(ev, ev_link), 1);
 349                         ev = next, ++iter) {
 350                             if (iter->revents) {
 351                                     if (!ev->ev_persistent) {
 352                                             --active_events;
 (gdb) p poll_list
 $16 = (struct pollfd *) 0x788c70bc3000
 (gdb) p iter
 $10 = (struct pollfd *) 0x788c70bc3010
 (gdb) p last_iter
 $9 = (struct pollfd *) 0x788c70bc3018
 (gdb) p ev
 $11 = (struct event *) 0x5a5a5a5a5a5a5a5a
 (gdb) p all_events
 $14 = {lh_first = 0x788c70ba70d8}
 (gdb) p &all_events
 $15 = (struct {...} *) 0x4db871b0 <all_events>
 (gdb) p *all_events.lh_first
 $18 = {ev_link = {le_next = 0x788c70ba7198, le_prev = 0x4db871b0 <all_events>}, ev_fd = 8, ev_write = 1, ev_persistent = 0,
   ev_handler = 0x4d8047c2 <deferred_write_handler>, ev_arg = 0x788c70ba70c0}
 (gdb) p *all_events.lh_first.ev_link.le_next
 $19 = {ev_link = {le_next = 0x788c70ba7018, le_prev = 0x788c70ba70d8}, ev_fd = 7, ev_write = 1, ev_persistent = 0,
   ev_handler = 0x4d8047c2 <deferred_write_handler>, ev_arg = 0x788c70ba7180}
 (gdb) p *all_events.lh_first.ev_link.le_next.ev_link.le_next
 $20 = {ev_link = {le_next = 0x788c70ba7078, le_prev = 0x788c70ba7198}, ev_fd = 9, ev_write = 1, ev_persistent = 0,
   ev_handler = 0x4d8047c2 <deferred_write_handler>, ev_arg = 0x788c70ba7000}
 (gdb) p *all_events.lh_first.ev_link.le_next.ev_link.le_next.ev_link.le_next
 $21 = {ev_link = {le_next = 0x4db87160 <signal_event>, le_prev = 0x788c70ba7018}, ev_fd = 10, ev_write = 1, ev_persistent = 0,
   ev_handler = 0x4d8047c2 <deferred_write_handler>, ev_arg = 0x788c70ba7060}
 (gdb) p *all_events.lh_first.ev_link.le_next.ev_link.le_next.ev_link.le_next.ev_link.le_next
 $22 = {ev_link = {le_next = 0x0, le_prev = 0x788c70ba7078}, ev_fd = 4, ev_write = 0, ev_persistent = 1, ev_handler = 0x4d804430 <signal_read_handler>,
   ev_arg = 0x0}

 ev looks strange. Is that some poisoning?
  Thomas

From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, lib-bug-people@netbsd.org, 
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, martin@NetBSD.org
Cc: 
Subject: Re: lib/54017: jemalloc deadlock?
Date: Mon, 11 Mar 2019 18:50:31 -0400

 On Mar 11, 10:35pm, wiz@NetBSD.org (Thomas Klausner) wrote:
 -- Subject: Re: lib/54017: jemalloc deadlock?

 | The following reply was made to PR lib/54017; it has been noted by GNATS.
 | 
 | From: Thomas Klausner <wiz@NetBSD.org>
 | To: gnats-bugs@NetBSD.org
 | Cc: 
 | Subject: Re: lib/54017: jemalloc deadlock?
 | Date: Mon, 11 Mar 2019 23:33:42 +0100
 | 
 |  I can confirm that thunderbird and firefox built fine for me.
 |  
 |  I see a new problem with a libc from today, a repeatable pbulk-scan
 |  core dump that did not happen before the userland update. pbulk-scan
 |  binary was from 2017, though I've just rebuilt it to get symbols.
 |  
 |  # /usr/pkg_bulk/bin/bulkbuild
 |  Warning: All log files of the previous pbulk run will be
 |  removed in 5 seconds. If you want to abort, press Ctrl-C.
 |  Removing old scan results
 |  Scanning...
 |  .................................................. 50/211
 |  .................................................. 100/211
 |  .................................................. 150/211
 |  .................................................. 200/211
 |  ...........[1]   Segmentation fault (core dumped) ${pscan} -v -l -I ${pscan_start_script} -m ${m...
 |  # gdb /usr/pkg_bulk/bin/pbulk-scan pbulk-scan.core  
 |  GNU gdb (GDB) 8.0.1
 |  Copyright (C) 2017 Free Software Foundation, Inc.
 |  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 |  This is free software: you are free to change and redistribute it.
 |  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
 |  and "show warranty" for details.
 |  This GDB was configured as "x86_64--netbsd".
 |  Type "show configuration" for configuration details.
 |  For bug reporting instructions, please see:
 |  <http://www.gnu.org/software/gdb/bugs/>.
 |  Find the GDB manual and other documentation resources online at:
 |  <http://www.gnu.org/software/gdb/documentation/>.
 |  For help, type "help".
 |  Type "apropos word" to search for commands related to "word"...
 |  Reading symbols from /usr/pkg_bulk/bin/pbulk-scan...done.
 |  [New process 1]
 |  Core was generated by `pbulk-scan'.
 |  Program terminated with signal SIGSEGV, Segmentation fault.
 |  #0  0x000000004d804c5b in event_dispatch () at event.c:348
 |  
 |  warning: Source file is more recent than executable.
 |  348                         iter < last_iter && ev && (next = LIST_NEXT(ev, ev_link), 1);
 |  (gdb) bt
 |  #0  0x000000004d804c5b in event_dispatch () at event.c:348
 |  #1  0x000000004d80354b in master_mode (master_port=0x7f7fff20542f "192.168.0.33:2001", 
 |      start_script=0x7f7fff2053fe "/usr/pkg_bulk/libexec/pbulk/scan-client-start") at master.c:284
 |  #2  0x000000004d805468 in main (argc=2, argv=0x7f7fff204d88) at pscan.c:155
 |  (gdb) l
 |  343             if (ret == 0 && timeout == 0)
 |  344                     return;
 |  345             if (ret > 0) {
 |  346                     iter = poll_list;
 |  347                     for (ev = LIST_FIRST(&all_events);
 |  348                         iter < last_iter && ev && (next = LIST_NEXT(ev, ev_link), 1);
 |  349                         ev = next, ++iter) {
 |  350                             if (iter->revents) {
 |  351                                     if (!ev->ev_persistent) {
 |  352                                             --active_events;
 |  (gdb) p poll_list
 |  $16 = (struct pollfd *) 0x788c70bc3000
 |  (gdb) p iter
 |  $10 = (struct pollfd *) 0x788c70bc3010
 |  (gdb) p last_iter
 |  $9 = (struct pollfd *) 0x788c70bc3018
 |  (gdb) p ev
 |  $11 = (struct event *) 0x5a5a5a5a5a5a5a5a
 |  (gdb) p all_events
 |  $14 = {lh_first = 0x788c70ba70d8}
 |  (gdb) p &all_events
 |  $15 = (struct {...} *) 0x4db871b0 <all_events>
 |  (gdb) p *all_events.lh_first
 |  $18 = {ev_link = {le_next = 0x788c70ba7198, le_prev = 0x4db871b0 <all_events>}, ev_fd = 8, ev_write = 1, ev_persistent = 0,
 |    ev_handler = 0x4d8047c2 <deferred_write_handler>, ev_arg = 0x788c70ba70c0}
 |  (gdb) p *all_events.lh_first.ev_link.le_next
 |  $19 = {ev_link = {le_next = 0x788c70ba7018, le_prev = 0x788c70ba70d8}, ev_fd = 7, ev_write = 1, ev_persistent = 0,
 |    ev_handler = 0x4d8047c2 <deferred_write_handler>, ev_arg = 0x788c70ba7180}
 |  (gdb) p *all_events.lh_first.ev_link.le_next.ev_link.le_next
 |  $20 = {ev_link = {le_next = 0x788c70ba7078, le_prev = 0x788c70ba7198}, ev_fd = 9, ev_write = 1, ev_persistent = 0,
 |    ev_handler = 0x4d8047c2 <deferred_write_handler>, ev_arg = 0x788c70ba7000}
 |  (gdb) p *all_events.lh_first.ev_link.le_next.ev_link.le_next.ev_link.le_next
 |  $21 = {ev_link = {le_next = 0x4db87160 <signal_event>, le_prev = 0x788c70ba7018}, ev_fd = 10, ev_write = 1, ev_persistent = 0,
 |    ev_handler = 0x4d8047c2 <deferred_write_handler>, ev_arg = 0x788c70ba7060}
 |  (gdb) p *all_events.lh_first.ev_link.le_next.ev_link.le_next.ev_link.le_next.ev_link.le_next
 |  $22 = {ev_link = {le_next = 0x0, le_prev = 0x788c70ba7078}, ev_fd = 4, ev_write = 0, ev_persistent = 1, ev_handler = 0x4d804430 <signal_read_handler>,
 |    ev_arg = 0x0}
 |  
 |  ev looks strange. Is that some poisoning?

 Yes, it was probably freed already.

 christos

From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: lib/54017: jemalloc deadlock?
Date: Tue, 12 Mar 2019 00:00:44 +0100

 I also have a not-sure-how-triggered zsh core dump. Happens not so
 rarely too.
 Symptom: zsh dies while I type; I guess while I do command completion.

 (gdb) bt
 #0  0x00007bfd0719a1aa in _lwp_kill () from /usr/lib/libc.so.12
 #1  0x00007bfd07199e59 in abort () at /usr/src/lib/libc/stdlib/abort.c:74
 #2  0x00007bfd070d76cd in rtree_child_leaf_tryread (elm=<optimized out>, dependent=<optimized out>)
     at /usr/src/external/bsd/jemalloc/lib/../dist/src/rtree.c:205
 #3  0x00007bfd070d794d in rtree_leaf_elm_lookup_hard (tsdn=tsdn@entry=0x7bfd08417040, rtree=0x7bfd0741df60 <extents_rtree>,
     rtree_ctx=rtree_ctx@entry=0x7bfd08417068, key=key@entry=11936128518282651045, dependent=dependent@entry=true, init_missing=init_missing@entry=false)
     at /usr/src/external/bsd/jemalloc/lib/../dist/src/rtree.c:292
 #4  0x00007bfd0712eeec in rtree_leaf_elm_lookup (rtree=<optimized out>, init_missing=false, dependent=true, key=11936128518282651045,
     rtree_ctx=0x7bfd08417068, tsdn=0x7bfd08417040) at /usr/src/external/bsd/jemalloc/lib/../include/jemalloc/internal/rtree.h:381
 #5  rtree_read (rtree=<optimized out>, dependent=true, key=11936128518282651045, rtree_ctx=0x7bfd08417068, tsdn=0x7bfd08417040)
     at /usr/src/external/bsd/jemalloc/lib/../include/jemalloc/internal/rtree.h:406
 #6  rtree_szind_slab_read (rtree=<optimized out>, dependent=true, r_slab=0x7f7fff60842c, r_szind=0x7f7fff608428, key=11936128518282651045,
     rtree_ctx=<optimized out>, tsdn=0x7bfd08417040) at /usr/src/external/bsd/jemalloc/lib/../include/jemalloc/internal/rtree.h:458
 #7  ifree (tsd=0x7bfd08417040, ptr=0xa5a5a5a5a5a5a5a5, tcache=0x7bfd08417200, slow_path=<optimized out>)
     at /usr/src/external/bsd/jemalloc/lib/../dist/src/jemalloc.c:2232
 #8  0x00007bfd0713353a in free (ptr=0xa5a5a5a5a5a5a5a5) at /usr/src/external/bsd/jemalloc/lib/../dist/src/jemalloc.c:2431
 #9  0x00000000004545dc in zsfree (p=<optimized out>) at mem.c:1895
 #10 0x0000000000412ecc in bin_shift (name=0x7bfd0828c8a8 "shift", argv=<optimized out>, ops=<optimized out>, func=<optimized out>) at builtin.c:5443
 #11 0x000000000041c0ae in execbuiltin (args=args@entry=0x7bfd0828c8b0, assigns=assigns@entry=0x0, bn=bn@entry=0x6afe20 <builtins+3520>) at builtin.c:507
 #12 0x000000000042a62f in execcmd_exec (state=state@entry=0x7f7fff609390, eparams=eparams@entry=0x7f7fff6088a8, input=input@entry=-10450848,
     output=output@entry=0, how=<optimized out>, how@entry=18, last1=<optimized out>, last1@entry=2, close_if_forked=<optimized out>,
     close_if_forked@entry=-1) at exec.c:4090
 #13 0x000000000042b248 in execpline2 (state=state@entry=0x7f7fff609390, pcode=pcode@entry=1475, how=how@entry=18, input=<optimized out>, output=0,
     last1=last1@entry=0) at exec.c:1927
 #14 0x000000000042b4fd in execpline (state=state@entry=0x7f7fff609390, slcode=<optimized out>, how=how@entry=18, last1=0) at exec.c:1658
 #15 0x000000000042c6f8 in execlist (state=state@entry=0x7f7fff609390, dont_change_job=dont_change_job@entry=1, exiting=0) at exec.c:1413
 #16 0x0000000000450353 in execcase (state=0x7f7fff609390, do_exec=0) at loop.c:675
 #17 0x00000000004294e5 in execcmd_exec (state=state@entry=0x7f7fff609390, eparams=eparams@entry=0x7f7fff608d68, input=input@entry=-10449632,
     output=output@entry=0, how=<optimized out>, how@entry=18, last1=<optimized out>, last1@entry=2, close_if_forked=<optimized out>,
     close_if_forked@entry=-1) at exec.c:3910
 #18 0x000000000042b248 in execpline2 (state=state@entry=0x7f7fff609390, pcode=pcode@entry=963, how=how@entry=18, input=<optimized out>, output=0,
     last1=last1@entry=0) at exec.c:1927
 #19 0x000000000042b4fd in execpline (state=state@entry=0x7f7fff609390, slcode=<optimized out>, how=how@entry=18, last1=0) at exec.c:1658
 #20 0x000000000042c6f8 in execlist (state=state@entry=0x7f7fff609390, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0) at exec.c:1413
 #21 0x000000000044f86d in execwhile (state=0x7f7fff609390, do_exec=<optimized out>) at loop.c:455
 #22 0x00000000004294e5 in execcmd_exec (state=state@entry=0x7f7fff609390, eparams=eparams@entry=0x7f7fff6091f8, input=input@entry=-10448464,
     output=output@entry=0, how=<optimized out>, how@entry=2, last1=<optimized out>, last1@entry=2, close_if_forked=<optimized out>, close_if_forked@entry=-1)
     at exec.c:3910
 #23 0x000000000042b248 in execpline2 (state=state@entry=0x7f7fff609390, pcode=pcode@entry=899, how=how@entry=2, input=<optimized out>, output=0,
     last1=last1@entry=0) at exec.c:1927
 #24 0x000000000042b4fd in execpline (state=state@entry=0x7f7fff609390, slcode=<optimized out>, how=how@entry=2, last1=0) at exec.c:1658
 #25 0x000000000042c6f8 in execlist (state=state@entry=0x7f7fff609390, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0) at exec.c:1413
 #26 0x000000000042cad2 in execode (p=p@entry=0x7bfd08377f80, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0,
     context=context@entry=0x495ef3 "shfunc") at exec.c:1192
 #27 0x000000000042d6ff in runshfunc (prog=prog@entry=0x7bfd08377f80, wrap=wrap@entry=0x0, name=name@entry=0x7bfd0828c170 "_arguments") at exec.c:5974
 #28 0x00007bfd05a0b5dc in comp_wrapper (prog=0x7bfd08377f80, w=0x0, name=0x7bfd0828c170 "_arguments") at complete.c:1524
 #29 0x000000000042d53b in runshfunc (prog=0x7bfd08377f80, wrap=0x7bfd05c240a0 <wrapper>, name=0x7bfd0828c170 "_arguments") at exec.c:5958
 #30 0x000000000042da2e in doshfunc (shfunc=shfunc@entry=0x7bfd083cd700, doshargs=doshargs@entry=0x7bfd082c7478, noreturnval=noreturnval@entry=0)
     at exec.c:5824
 #31 0x000000000042e24b in execshfunc (shf=shf@entry=0x7bfd083cd700, args=args@entry=0x7bfd082c7478) at exec.c:5393
 #32 0x000000000042a0f8 in execshfunc (args=0x7bfd082c7478, shf=0x7bfd083cd700) at exec.c:5359
 #33 execcmd_exec (state=state@entry=0x7f7fff609a40, eparams=eparams@entry=0x7f7fff6098a8, input=input@entry=-10446752, output=output@entry=0,
     how=<optimized out>, how@entry=2, last1=<optimized out>, last1@entry=2, close_if_forked=<optimized out>, close_if_forked@entry=-1) at exec.c:3962
 #34 0x000000000042b248 in execpline2 (state=state@entry=0x7f7fff609a40, pcode=pcode@entry=3587, how=how@entry=2, input=<optimized out>, output=0,
     last1=last1@entry=0) at exec.c:1927
 #35 0x000000000042b4fd in execpline (state=state@entry=0x7f7fff609a40, slcode=<optimized out>, how=how@entry=2, last1=last1@entry=0) at exec.c:1658
 #36 0x000000000042c7df in execlist (state=state@entry=0x7f7fff609a40, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0) at exec.c:1421
 #37 0x000000000042cad2 in execode (p=0x7bfd0822c0c0, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0,
     context=context@entry=0x495e8e "loadautofunc") at exec.c:1192
 #38 0x000000000042cc7d in execautofn_basic (do_exec=<optimized out>, state=<optimized out>) at exec.c:5439
 #39 0x000000000042a244 in execcmd_exec (state=state@entry=0x7f7fff609ef0, eparams=eparams@entry=0x7f7fff609d58, input=input@entry=-10445552,
     output=output@entry=0, how=<optimized out>, how@entry=18, last1=<optimized out>, last1@entry=2, close_if_forked=<optimized out>,
     close_if_forked@entry=-1) at exec.c:3908
 #40 0x000000000042b248 in execpline2 (state=state@entry=0x7f7fff609ef0, pcode=pcode@entry=3, how=how@entry=18, input=<optimized out>, output=0,
     last1=last1@entry=0) at exec.c:1927
 #41 0x000000000042b4fd in execpline (state=state@entry=0x7f7fff609ef0, slcode=<optimized out>, how=how@entry=18, last1=0) at exec.c:1658
 #42 0x000000000042c6f8 in execlist (state=state@entry=0x7f7fff609ef0, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0) at exec.c:1413
 #43 0x000000000042cad2 in execode (p=p@entry=0x7bfd0837f040, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0,
     context=context@entry=0x495ef3 "shfunc") at exec.c:1192
 #44 0x000000000042d6ff in runshfunc (prog=prog@entry=0x7bfd0837f040, wrap=wrap@entry=0x0, name=name@entry=0x7bfd082c6160 "_rm") at exec.c:5974
 #45 0x00007bfd05a0b5dc in comp_wrapper (prog=0x7bfd0837f040, w=0x0, name=0x7bfd082c6160 "_rm") at complete.c:1524
 #46 0x000000000042d53b in runshfunc (prog=0x7bfd0837f040, wrap=0x7bfd05c240a0 <wrapper>, name=0x7bfd082c6160 "_rm") at exec.c:5958
 #47 0x000000000042da2e in doshfunc (shfunc=shfunc@entry=0x7bfd0837f000, doshargs=doshargs@entry=0x7bfd082cab30, noreturnval=noreturnval@entry=0)
     at exec.c:5824
 #48 0x000000000042e24b in execshfunc (shf=shf@entry=0x7bfd0837f000, args=args@entry=0x7bfd082cab30) at exec.c:5393
 #49 0x000000000042a0f8 in execshfunc (args=0x7bfd082cab30, shf=0x7bfd0837f000) at exec.c:5359
 #50 execcmd_exec (state=state@entry=0x7f7fff60a5a0, eparams=eparams@entry=0x7f7fff60a408, input=input@entry=-10443840, output=output@entry=0,
     how=<optimized out>, how@entry=18, last1=<optimized out>, last1@entry=2, close_if_forked=<optimized out>, close_if_forked@entry=-1) at exec.c:3962
 #51 0x000000000042b248 in execpline2 (state=state@entry=0x7f7fff60a5a0, pcode=pcode@entry=131, how=how@entry=18, input=<optimized out>, output=0,
     last1=last1@entry=0) at exec.c:1927
 #52 0x000000000042b4fd in execpline (state=state@entry=0x7f7fff60a5a0, slcode=<optimized out>, how=how@entry=18, last1=0) at exec.c:1658
 #53 0x000000000042c6f8 in execlist (state=state@entry=0x7f7fff60a5a0, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0) at exec.c:1413
 #54 0x000000000042cad2 in execode (p=0x7bfd082caaa8, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0, context=context@entry=0x49df83 "eval")
     at exec.c:1192
 #55 0x0000000000411f8e in eval (argv=0x7bfd082caa68) at builtin.c:5974
 #56 0x000000000041c0ae in execbuiltin (args=args@entry=0x7bfd082caa30, assigns=assigns@entry=0x0, bn=bn@entry=0x6af620 <builtins+1472>) at builtin.c:507
 #57 0x000000000042a62f in execcmd_exec (state=state@entry=0x7f7fff60b030, eparams=eparams@entry=0x7f7fff60aa18, input=input@entry=-10442288,
     output=output@entry=0, how=<optimized out>, how@entry=2, last1=<optimized out>, last1@entry=2, close_if_forked=<optimized out>, close_if_forked@entry=-1)
     at exec.c:4090
 #58 0x000000000042b248 in execpline2 (state=state@entry=0x7f7fff60b030, pcode=pcode@entry=4099, how=how@entry=2, input=<optimized out>, output=0,
     last1=last1@entry=0) at exec.c:1927
 #59 0x000000000042b4fd in execpline (state=state@entry=0x7f7fff60b030, slcode=<optimized out>, how=how@entry=2, last1=last1@entry=0) at exec.c:1658
 #60 0x000000000042c7df in execlist (state=state@entry=0x7f7fff60b030, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0) at exec.c:1421
 #61 0x000000000044fc62 in execif (state=0x7f7fff60b030, do_exec=0) at loop.c:576
 #62 0x00000000004294e5 in execcmd_exec (state=state@entry=0x7f7fff60b030, eparams=eparams@entry=0x7f7fff60ae98, input=input@entry=-10441136,
     output=output@entry=0, how=<optimized out>, how@entry=2, last1=<optimized out>, last1@entry=2, close_if_forked=<optimized out>, close_if_forked@entry=-1)
     at exec.c:3910
 #63 0x000000000042b248 in execpline2 (state=state@entry=0x7f7fff60b030, pcode=pcode@entry=3971, how=how@entry=2, input=<optimized out>, output=0,
     last1=last1@entry=0) at exec.c:1927
 #64 0x000000000042b4fd in execpline (state=state@entry=0x7f7fff60b030, slcode=<optimized out>, how=how@entry=2, last1=0) at exec.c:1658
 #65 0x000000000042c6f8 in execlist (state=state@entry=0x7f7fff60b030, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0) at exec.c:1413
 #66 0x000000000042cad2 in execode (p=p@entry=0x7bfd08349f00, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0,
     context=context@entry=0x495ef3 "shfunc") at exec.c:1192
 #67 0x000000000042d6ff in runshfunc (prog=prog@entry=0x7bfd08349f00, wrap=wrap@entry=0x0, name=name@entry=0x7bfd082ca170 "_dispatch") at exec.c:5974
 #68 0x00007bfd05a0b5dc in comp_wrapper (prog=0x7bfd08349f00, w=0x0, name=0x7bfd082ca170 "_dispatch") at complete.c:1524
 #69 0x000000000042d53b in runshfunc (prog=0x7bfd08349f00, wrap=0x7bfd05c240a0 <wrapper>, name=0x7bfd082ca170 "_dispatch") at exec.c:5958
 #70 0x000000000042da2e in doshfunc (shfunc=shfunc@entry=0x7bfd083f8600, doshargs=doshargs@entry=0x7bfd082de5e8, noreturnval=noreturnval@entry=0)
     at exec.c:5824
 #71 0x000000000042e24b in execshfunc (shf=shf@entry=0x7bfd083f8600, args=args@entry=0x7bfd082de5e8) at exec.c:5393
 #72 0x000000000042a0f8 in execshfunc (args=0x7bfd082de5e8, shf=0x7bfd083f8600) at exec.c:5359
 #73 execcmd_exec (state=state@entry=0x7f7fff60b6e0, eparams=eparams@entry=0x7f7fff60b548, input=input@entry=-10439424, output=output@entry=0,
     how=<optimized out>, how@entry=18, last1=<optimized out>, last1@entry=2, close_if_forked=<optimized out>, close_if_forked@entry=-1) at exec.c:3962
 #74 0x000000000042b248 in execpline2 (state=state@entry=0x7f7fff60b6e0, pcode=pcode@entry=2627, how=how@entry=18, input=<optimized out>, output=0,
     last1=last1@entry=0) at exec.c:1927
 #75 0x000000000042b4fd in execpline (state=state@entry=0x7f7fff60b6e0, slcode=<optimized out>, how=how@entry=18, last1=0) at exec.c:1658
 #76 0x000000000042c6f8 in execlist (state=state@entry=0x7f7fff60b6e0, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0) at exec.c:1413
 #77 0x000000000042cad2 in execode (p=p@entry=0x7bfd08349e80, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0,
     context=context@entry=0x495ef3 "shfunc") at exec.c:1192
 #78 0x000000000042d6ff in runshfunc (prog=prog@entry=0x7bfd08349e80, wrap=wrap@entry=0x0, name=name@entry=0x7bfd082de168 "_normal") at exec.c:5974
 #79 0x00007bfd05a0b5dc in comp_wrapper (prog=0x7bfd08349e80, w=0x0, name=0x7bfd082de168 "_normal") at complete.c:1524
 #80 0x000000000042d53b in runshfunc (prog=0x7bfd08349e80, wrap=0x7bfd05c240a0 <wrapper>, name=0x7bfd082de168 "_normal") at exec.c:5958
 #81 0x000000000042da2e in doshfunc (shfunc=shfunc@entry=0x7bfd08319200, doshargs=doshargs@entry=0x7bfd083108d8, noreturnval=noreturnval@entry=0)
     at exec.c:5824
 #82 0x000000000042e24b in execshfunc (shf=shf@entry=0x7bfd08319200, args=args@entry=0x7bfd083108d8) at exec.c:5393
 #83 0x000000000042a0f8 in execshfunc (args=0x7bfd083108d8, shf=0x7bfd08319200) at exec.c:5359
 #84 execcmd_exec (state=state@entry=0x7f7fff60c210, eparams=eparams@entry=0x7f7fff60bbf8, input=input@entry=-10437712, output=output@entry=0,
     how=<optimized out>, how@entry=2, last1=<optimized out>, last1@entry=2, close_if_forked=<optimized out>, close_if_forked@entry=-1) at exec.c:3962
 #85 0x000000000042b248 in execpline2 (state=state@entry=0x7f7fff60c210, pcode=pcode@entry=7555, how=how@entry=2, input=<optimized out>, output=0,
     last1=last1@entry=0) at exec.c:1927
 #86 0x000000000042b4fd in execpline (state=state@entry=0x7f7fff60c210, slcode=<optimized out>, how=how@entry=2, last1=last1@entry=0) at exec.c:1658
 #87 0x000000000042c7df in execlist (state=state@entry=0x7f7fff60c210, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0) at exec.c:1421
 #88 0x000000000044fc62 in execif (state=0x7f7fff60c210, do_exec=0) at loop.c:576
 #89 0x00000000004294e5 in execcmd_exec (state=state@entry=0x7f7fff60c210, eparams=eparams@entry=0x7f7fff60c078, input=input@entry=-10436560,
     output=output@entry=0, how=<optimized out>, how@entry=2, last1=<optimized out>, last1@entry=2, close_if_forked=<optimized out>, close_if_forked@entry=-1)
     at exec.c:3910
 #90 0x000000000042b248 in execpline2 (state=state@entry=0x7f7fff60c210, pcode=pcode@entry=7427, how=how@entry=2, input=<optimized out>, output=0,
     last1=last1@entry=0) at exec.c:1927
 #91 0x000000000042b4fd in execpline (state=state@entry=0x7f7fff60c210, slcode=<optimized out>, how=how@entry=2, last1=0) at exec.c:1658
 #92 0x000000000042c6f8 in execlist (state=state@entry=0x7f7fff60c210, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0) at exec.c:1413
 #93 0x000000000042cad2 in execode (p=p@entry=0x7bfd08349dc0, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0,
     context=context@entry=0x495ef3 "shfunc") at exec.c:1192
 #94 0x000000000042d6ff in runshfunc (prog=prog@entry=0x7bfd08349dc0, wrap=wrap@entry=0x0, name=name@entry=0x7bfd08310170 "_complete") at exec.c:5974
 #95 0x00007bfd05a0b5dc in comp_wrapper (prog=0x7bfd08349dc0, w=0x0, name=0x7bfd08310170 "_complete") at complete.c:1524
 #96 0x000000000042d53b in runshfunc (prog=0x7bfd08349dc0, wrap=0x7bfd05c240a0 <wrapper>, name=0x7bfd08310170 "_complete") at exec.c:5958
 #97 0x000000000042da2e in doshfunc (shfunc=shfunc@entry=0x7bfd08373e80, doshargs=doshargs@entry=0x7bfd08327428, noreturnval=noreturnval@entry=0)
     at exec.c:5824
 #98 0x000000000042e24b in execshfunc (shf=shf@entry=0x7bfd08373e80, args=args@entry=0x7bfd08327428) at exec.c:5393
 #99 0x000000000042a0f8 in execshfunc (args=0x7bfd08327428, shf=0x7bfd08373e80) at exec.c:5359
 #100 execcmd_exec (state=state@entry=0x7f7fff60db70, eparams=eparams@entry=0x7f7fff60c728, input=input@entry=-10434848, output=output@entry=0,
     how=<optimized out>, how@entry=18, last1=<optimized out>, last1@entry=2, close_if_forked=<optimized out>, close_if_forked@entry=-1) at exec.c:3962
 #101 0x000000000042b248 in execpline2 (state=state@entry=0x7f7fff60db70, pcode=pcode@entry=12995, how=how@entry=18, input=<optimized out>, output=0,
     last1=last1@entry=0) at exec.c:1927
 #102 0x000000000042b4fd in execpline (state=state@entry=0x7f7fff60db70, slcode=<optimized out>, how=how@entry=18, last1=0) at exec.c:1658
 #103 0x000000000042c6f8 in execlist (state=state@entry=0x7f7fff60db70, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0) at exec.c:1413
 #104 0x000000000044fc26 in execif (state=0x7f7fff60db70, do_exec=0) at loop.c:555
 #105 0x00000000004294e5 in execcmd_exec (state=state@entry=0x7f7fff60db70, eparams=eparams@entry=0x7f7fff60cba8, input=input@entry=-10433696,
     output=output@entry=0, how=<optimized out>, how@entry=2, last1=<optimized out>, last1@entry=2, close_if_forked=<optimized out>, close_if_forked@entry=-1)
     at exec.c:3910
 #106 0x000000000042b248 in execpline2 (state=state@entry=0x7f7fff60db70, pcode=pcode@entry=12675, how=how@entry=2, input=<optimized out>, output=0,
     last1=last1@entry=0) at exec.c:1927
 #107 0x000000000042b4fd in execpline (state=state@entry=0x7f7fff60db70, slcode=<optimized out>, how=how@entry=2, last1=0) at exec.c:1658
 #108 0x000000000042c6f8 in execlist (state=state@entry=0x7f7fff60db70, dont_change_job=dont_change_job@entry=1, exiting=0) at exec.c:1413
 #109 0x000000000044ed33 in execfor (state=0x7f7fff60db70, do_exec=<optimized out>) at loop.c:175
 #110 0x00000000004294e5 in execcmd_exec (state=state@entry=0x7f7fff60db70, eparams=eparams@entry=0x7f7fff60d078, input=input@entry=-10432464,
     output=output@entry=0, how=<optimized out>, how@entry=2, last1=<optimized out>, last1@entry=2, close_if_forked=<optimized out>, close_if_forked@entry=-1)
     at exec.c:3910
 #111 0x000000000042b248 in execpline2 (state=state@entry=0x7f7fff60db70, pcode=pcode@entry=12163, how=how@entry=2, input=<optimized out>, output=0,
     last1=last1@entry=0) at exec.c:1927
 #112 0x000000000042b4fd in execpline (state=state@entry=0x7f7fff60db70, slcode=<optimized out>, how=how@entry=2, last1=0) at exec.c:1658
 #113 0x000000000042c6f8 in execlist (state=state@entry=0x7f7fff60db70, dont_change_job=dont_change_job@entry=1, exiting=0) at exec.c:1413
 #114 0x000000000044ed33 in execfor (state=0x7f7fff60db70, do_exec=<optimized out>) at loop.c:175
 #115 0x00000000004294e5 in execcmd_exec (state=state@entry=0x7f7fff60db70, eparams=eparams@entry=0x7f7fff60d548, input=input@entry=-10431232,
     output=output@entry=0, how=<optimized out>, how@entry=2, last1=<optimized out>, last1@entry=2, close_if_forked=<optimized out>, close_if_forked@entry=-1)
     at exec.c:3910
 #116 0x000000000042b248 in execpline2 (state=state@entry=0x7f7fff60db70, pcode=pcode@entry=10691, how=how@entry=2, input=<optimized out>, output=0,
     last1=last1@entry=0) at exec.c:1927
 #117 0x000000000042b4fd in execpline (state=state@entry=0x7f7fff60db70, slcode=<optimized out>, how=how@entry=2, last1=0) at exec.c:1658
 #118 0x000000000042c6f8 in execlist (state=state@entry=0x7f7fff60db70, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0) at exec.c:1413
 #119 0x000000000045043f in exectry (state=0x7f7fff60db70, do_exec=0) at loop.c:743
 #120 0x00000000004294e5 in execcmd_exec (state=state@entry=0x7f7fff60db70, eparams=eparams@entry=0x7f7fff60d9d8, input=input@entry=-10430064,
     output=output@entry=0, how=<optimized out>, how@entry=2, last1=<optimized out>, last1@entry=2, close_if_forked=<optimized out>, close_if_forked@entry=-1)
     at exec.c:3910
 #121 0x000000000042b248 in execpline2 (state=state@entry=0x7f7fff60db70, pcode=pcode@entry=3267, how=how@entry=2, input=<optimized out>, output=0,
     last1=last1@entry=0) at exec.c:1927
 #122 0x000000000042b4fd in execpline (state=state@entry=0x7f7fff60db70, slcode=<optimized out>, how=how@entry=2, last1=0) at exec.c:1658
 #123 0x000000000042c6f8 in execlist (state=state@entry=0x7f7fff60db70, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0) at exec.c:1413
 #124 0x000000000042cad2 in execode (p=p@entry=0x7bfd08349c40, dont_change_job=dont_change_job@entry=1, exiting=exiting@entry=0,
     context=context@entry=0x495ef3 "shfunc") at exec.c:1192
 #125 0x000000000042d6ff in runshfunc (prog=prog@entry=0x7bfd08349c40, wrap=wrap@entry=0x0, name=name@entry=0x7bfd08340178 "_main_complete") at exec.c:5974
 #126 0x00007bfd05a0b5dc in comp_wrapper (prog=0x7bfd08349c40, w=0x0, name=0x7bfd08340178 "_main_complete") at complete.c:1524
 #127 0x000000000042d53b in runshfunc (prog=0x7bfd08349c40, wrap=0x7bfd05c240a0 <wrapper>, name=0x7bfd08340178 "_main_complete") at exec.c:5958
 #128 0x000000000042da2e in doshfunc (shfunc=shfunc@entry=0x7bfd0831ae00, doshargs=doshargs@entry=0x0, noreturnval=noreturnval@entry=1) at exec.c:5824
 #129 0x00007bfd05a13251 in callcompfunc (fn=0x7bfd08316590 "_main_complete", s=<optimized out>) at compcore.c:833
 #130 makecomplist (lst=<optimized out>, incmd=<optimized out>, s=<optimized out>) at compcore.c:988
 #131 do_completion (dummy=<optimized out>, dat=<optimized out>) at compcore.c:343
 #132 0x00007bfd05e37e35 in docompletion (incmd=0, lst=<optimized out>, s=<optimized out>) at zle_tricky.c:2340
 #133 docomplete (lst=<optimized out>) at zle_tricky.c:868
 #134 0x00007bfd05e33092 in completecall (args=args@entry=0x7bfd06053860 <zlenoargs>) at zle_tricky.c:207
 #135 0x00007bfd05e2179a in execzlefunc (func=0x7bfd0604dd30 <thingies+2000>, args=args@entry=0x7bfd06053860 <zlenoargs>, set_bindk=set_bindk@entry=0)
     at zle_main.c:1460
 #136 0x00007bfd05e21aa7 in zlecore () at zle_main.c:1139
 #137 0x00007bfd05e228cd in zleread (lp=<optimized out>, rp=<optimized out>, flags=<optimized out>, context=<optimized out>,
     init=0x7bfd05e42afd "zle-line-init", finish=0x7bfd05e42aed "zle-line-finish") at zle_main.c:1347
 #138 0x0000000000443666 in zleentry (cmd=cmd@entry=1) at init.c:1604
 #139 0x0000000000444673 in inputline () at input.c:295
 #140 ingetc () at input.c:228
 #141 0x0000000000444886 in ingetc () at input.c:234
 #142 0x000000000043bc04 in ihgetc () at hist.c:407
 #143 0x000000000044d3a1 in gettok () at lex.c:611
 #144 zshlex () at lex.c:275
 #145 0x000000000044e00f in zshlex () at lex.c:313
 #146 0x000000000046b206 in parse_event (endtok=endtok@entry=37) at parse.c:581
 #147 0x00000000004407eb in loop (toplevel=toplevel@entry=1, justonce=justonce@entry=0) at init.c:147
 #148 0x0000000000443cb5 in zsh_main (argc=<optimized out>, argv=<optimized out>) at init.c:1758
 #149 0x000000000040f2fd in ___start ()
 #150 0x00007f7fba002c9e in _rtld_unref_dag (root=<optimized out>, root=<optimized out>) at /usr/src/libexec/ld.elf_so/rtld.c:964
 #151 _rtld_unref_dag (root=<optimized out>, root=<optimized out>) at /usr/src/libexec/ld.elf_so/rtld.c:970
 #152 _rtld_unref_dag (root=<optimized out>, root=<optimized out>) at /usr/src/libexec/ld.elf_so/rtld.c:970
 #153 _rtld_unref_dag (root=<optimized out>, root=<optimized out>) at /usr/src/libexec/ld.elf_so/rtld.c:970
 #154 _rtld_unref_dag (root=<optimized out>, root=<optimized out>) at /usr/src/libexec/ld.elf_so/rtld.c:970
 #155 _rtld_unref_dag (root=<optimized out>, root=<optimized out>) at /usr/src/libexec/ld.elf_so/rtld.c:970
 #156 _rtld_unref_dag (root=<optimized out>, root=<optimized out>) at /usr/src/libexec/ld.elf_so/rtld.c:970
 #157 _rtld_unload_object (do_fini_funcs=true, root=0x7f7fff60ffe0, mask=0x7f7fff60e440) at /usr/src/libexec/ld.elf_so/rtld.c:895
 #158 dlclose (handle=<optimized out>) at /usr/src/libexec/ld.elf_so/rtld.c:997
 #159 0x00007bfd08405e21 in ?? ()
 #160 0x00007f7fff60eac5 in ?? ()
 #161 0x00007f7fff60f004 in ?? ()
 #162 0x00007f7fff60f00d in ?? ()
 #163 0x00007f7fff60f01c in ?? ()
 #164 0x00007f7fff60f025 in ?? ()
 #165 0x00007f7fff60f031 in ?? ()
 #166 0x00007f7fff60f040 in ?? ()
 #167 0x00007f7fff60f0da in ?? ()
 #168 0x00007f7fff60f0ed in ?? ()
 #169 0x00007f7fff60f111 in ?? ()
 #170 0x00007f7fff60f146 in ?? ()
 #171 0x00007bfd083ab351 in ?? ()
 #172 0x00007bfd08349c21 in ?? ()
 #173 0x00007bfd084051e1 in ?? ()
 #174 0x00007bfd084065a1 in ?? ()
 #175 0x00007bfd08405d61 in ?? ()
 #176 0x00007bfd083840e1 in ?? ()
 #177 0x00007bfd08384141 in ?? ()
 #178 0x00007bfd083841a1 in ?? ()
 #179 0x00007bfd08384201 in ?? ()
 #180 0x00007bfd08405e61 in ?? ()
 #181 0x00007bfd08405161 in ?? ()
 #182 0x00007bfd08405ee1 in ?? ()
 #183 0x00007f7fff60f2b3 in ?? ()
 #184 0x00007f7fff60f2c2 in ?? ()
 #185 0x00007f7fff60f2ce in ?? ()
 #186 0x00007f7fff60f2da in ?? ()
 #187 0x00007f7fff60f2ef in ?? ()
 #188 0x00007f7fff60f303 in ?? ()
 #189 0x00007f7fff60f31e in ?? ()
 #190 0x00007f7fff60f33c in ?? ()
 #191 0x00007f7fff60f347 in ?? ()
 #192 0x00007f7fff60f354 in ?? ()
 #193 0x00007f7fff60f363 in ?? ()
 #194 0x00007f7fff60f371 in ?? ()
 #195 0x00007f7fff60f388 in ?? ()
 #196 0x0000000000000000 in ?? ()
 (gdb)

From: Rin Okuyama <rokuyama.rk@gmail.com>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@NetBSD.org>,
 Christos Zoulas <christos@astron.com>
Cc: 
Subject: Re: lib/54017: jemalloc deadlock?
Date: Fri, 15 Mar 2019 04:24:31 +0900

 I found it! pkgsrc/wip/emacs-git successfully builds if reverting
 our local hack for sshd:

 ----
 Index: external/bsd/jemalloc/dist/src/jemalloc.c
 ===================================================================
 RCS file: /home/netbsd/src/external/bsd/jemalloc/dist/src/jemalloc.c,v
 retrieving revision 1.6
 diff -p -u -r1.6 jemalloc.c
 --- external/bsd/jemalloc/dist/src/jemalloc.c	12 Mar 2019 15:13:25 -0000	1.6
 +++ external/bsd/jemalloc/dist/src/jemalloc.c	14 Mar 2019 18:52:25 -0000
 @@ -2339,7 +2339,7 @@ je_realloc(void *ptr, size_t size) {
   				tcache = NULL;
   			}
   			ifree(tsd, ptr, tcache, true);
 -#ifdef notyet
 +#if 1
   			/*
   			 * sshd depends on realloc(p, 0) returning non-NULL
   			 * disable for compatibility for now
 ----

 sshd and ssh work fine as far as I can see (usual login and X11
 forwarding). Do we really need that?

 Thanks,
 rin

From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, lib-bug-people@netbsd.org, 
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, martin@NetBSD.org
Cc: 
Subject: Re: lib/54017: jemalloc deadlock?
Date: Thu, 14 Mar 2019 17:39:29 -0400

 On Mar 14,  7:25pm, rokuyama.rk@gmail.com (Rin Okuyama) wrote:
 -- Subject: Re: lib/54017: jemalloc deadlock?

 |  I found it! pkgsrc/wip/emacs-git successfully builds if reverting
 |  our local hack for sshd:

 No, I will remove it!

 Thanks,

 christos

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.43 2018/01/16 07:36:43 maya Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2017 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.