NetBSD Problem Report #54795
From www@netbsd.org Tue Dec 24 00:23:36 2019
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 212167A13E
for <gnats-bugs@gnats.NetBSD.org>; Tue, 24 Dec 2019 00:23:36 +0000 (UTC)
Message-Id: <20191224002334.C08417A1EE@mollari.NetBSD.org>
Date: Tue, 24 Dec 2019 00:23:34 +0000 (UTC)
From: prlw1@cam.ac.uk
Reply-To: prlw1@cam.ac.uk
To: gnats-bugs@NetBSD.org
Subject: rust pbulk build stalls with 2 idle cargo processes
X-Send-Pr-Version: www-1.0
>Number: 54795
>Category: pkg
>Synopsis: rust pbulk build stalls with 2 idle cargo processes
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: pkg-manager
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Tue Dec 24 00:25:00 +0000 2019
>Closed-Date: Fri May 22 23:12:35 +0000 2020
>Last-Modified: Tue Sep 01 08:00:02 +0000 2020
>Originator: Patrick Welche
>Release: NetBSD-9.99.29/amd64 23-Dec-2019 pkgsrc current
>Organization:
>Environment:
>Description:
Still observing the same behaviour described in:
http://mail-index.netbsd.org/pkgsrc-users/2019/11/21/msg029924.html
USER PID %CPU %MEM VSZ RSS TTY STAT STARTED TIME COMMAND
root 27623 0.0 0.0 20172 1240 pts/1 I+ 5:36PM 0:00.00 | `-- /bin/sh /usr/pbulk/bin/bulkbuild
root 884 0.0 0.0 20192 1240 pts/1 I+ 5:46PM 0:00.00 | `-- /bin/sh /usr/pbulk/libexec/pbulk/build
root 13665 0.0 0.0 19440 4264 pts/1 I+ 5:46PM 0:00.03 | `-- /usr/pbulk/bin/pbulk-build -r /usr/pbulk/bulklog/meta/pbuild -v -b /usr/pbulk/libexec/pbulk/pkg-build /usr/pbulk/bulklog/meta/presolve /usr/pbulk/bulklog/meta/success /usr/pbulk/bulklog/meta/error
root 28748 0.0 0.0 20744 1336 pts/1 I+ 6:19PM 0:00.00 | `-- /bin/sh /usr/pbulk/libexec/pbulk/pkg-build
dbus 5169 0.0 0.0 21284 4528 pts/1 I+ 6:23PM 0:00.09 | `-- /usr/bin/make all BATCH DEPENDS_TARGET PYTHON_VERSION_REQD WRKLOG
dbus 29513 0.0 0.0 20196 1244 pts/1 I+ 6:23PM 0:00.00 | `-- /bin/sh -c set -e;\t\t\t\t\t if test -n "" && /usr/sbin/pkg_info -K /var/db/pkg -qe rust-1.39.0nb2; then echo ===\\> "Skipping installation of already handled package"; else cd /usr/pkgsrc/lang/rust\t\t\t\t\t\t\t && /usr/bin/env LD_LIBRARY_PATH=/tmp/pkgsrc/lang/rust/work/rust-bootstrap/lib MAKECONF=/etc/mk.conf PATH=/tmp/pkgsrc/lang/rust/work/.cwrapper/bin:/tmp/pkgsrc/lang/rust/work/.buildlink/bin:/tmp/pkgsrc/lang/rust/work/.gcc/bin:/tmp/pkgsrc/lang/rust/work/.tools/bin:/usr/pkg/bin:/root/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R7/bin:/usr/X11R6/bin:/usr/pkg/bin:/usr/pkg/sbin:/usr/games:/usr/local/bin:/usr/local/sbin /usr/bin/make _MAKE=/usr/bin/make OPSYS=NetBSD OS_VERSION=9.99.29 LOWER_OPSYS=netbsd _PKGSRCDIR=/usr/pkgsrc PKGTOOLS_VERSION=20191008 _CC=/usr/bin/gcc _PATH_ORIG=/root/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R7/bin:/usr/X11R6/bin:/usr/pkg/bin:/usr/pkg/sbin:/usr/games:/usr/local/b
in:/usr/local/sbin _PKGSRC_BARRIER=yes ALLOW_VULNERABLE_PACKAGES= all || {\t\t\t\t\t\t\t\t exitcode="$?";\t\t\t\t\t\t /usr/bin/env LD_LIBRARY_PATH=/tmp/pkgsrc/lang/rust/work/rust-bootstrap/lib MAKECONF=/etc/mk.conf PATH=/tmp/pkgsrc/lang/rust/work/.cwrapper/bin:/tmp/pkgsrc/lang/rust/work/.buildlink/bin:/tmp/pkgsrc/lang/rust/work/.gcc/bin:/tmp/pkgsrc/lang/rust/work/.tools/bin:/usr/pkg/bin:/root/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R7/bin:/usr/X11R6/bin:/usr/pkg/bin:/usr/pkg/sbin:/usr/games:/usr/local/bin:/usr/local/sbin /usr/bin/make _MAKE=/usr/bin/make OPSYS=NetBSD OS_VERSION=9.99.29 LOWER_OPSYS=netbsd _PKGSRCDIR=/usr/pkgsrc PKGTOOLS_VERSION=20191008 _CC=/usr/bin/gcc _PATH_ORIG=/root/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R7/bin:/usr/X11R6/bin:/usr/pkg/bin:/usr/pkg/sbin:/usr/games:/usr/local/bin:/usr/local/sbin _PKGSRC_BARRIER=yes barrier-error-check; exit "$exitcode";\t\t\t\t\t }; fi
dbus 22403 0.0 0.0 21576 4844 pts/1 I+ 6:23PM 0:00.12 | `-- /usr/bin/make _MAKE OPSYS OS_VERSION LOWER_OPSYS _PKGSRCDIR PKGTOOLS_VERSION _CC _PATH_ORIG _PKGSRC_BARRIER ALLOW_VULNERABLE_PACKAGES all
dbus 6022 0.0 0.0 93088 10156 pts/1 I+ 6:23PM 0:00.06 | `-- /usr/pkg/bin/python2.7 ./x.py -v build -j 24
dbus 539 0.0 0.0 26732 3768 pts/1 I+ 6:25PM 0:00.12 | `-- /tmp/pkgsrc/lang/rust/work/rustc-1.39.0-src/build/bootstrap/debug/bootstrap -v build -j 24
dbus 5501 0.0 0.1 184464 36076 pts/1 Il+ 7:01PM 0:00.43 | `-- /tmp/pkgsrc/lang/rust/work/rust-bootstrap/bin/cargo build -Zconfig-profile --target x86_64-unknown-netbsd -Zbinary-dep-depinfo -j 24 --release --frozen --manifest-path /tmp/pkgsrc/lang/rust/work/rustc-1.39.0-src/src/tools/rustfmt/Cargo.toml --message-format json-render-diagnostics
dbus 2423 0.0 0.0 131816 844 pts/1 I+ 7:01PM 0:00.00 | `-- /tmp/pkgsrc/lang/rust/work/rust-bootstrap/bin/cargo build -Zconfig-profile --target x86_64-unknown-netbsd -Zbinary-dep-depinfo -j 24 --release --frozen --manifest-path /tmp/pkgsrc/lang/rust/work/rustc-1.39.0-src/src/tools/rustfmt/Cargo.toml --message-format json-render-diagnostics
[Switching to LWP 16 of process 2423]
0x00007f7ff7c09c8a in ___lwp_park60 () from /usr/libexec/ld.elf_so
(gdb) bt
#0 0x00007f7ff7c09c8a in ___lwp_park60 () from /usr/libexec/ld.elf_so
#1 0x00007f7ff7c010d4 in _rtld_shared_enter () at /usr/src/libexec/ld.elf_so/rtld.c:1687
#2 0x00007f7ff7c00b85 in _rtld_bind (obj=0x7f7ff7efa400, reloff=<optimized out>)
at /usr/src/libexec/ld.elf_so/arch/x86_64/mdreloc.c:359
#3 0x00007f7ff7c007fd in _rtld_bind_start () from /usr/libexec/ld.elf_so
#4 0x0000000000000206 in ?? ()
#5 0x00007f7ff6c8ff3a in dup2 () from /usr/lib/libc.so.12
#6 0x00007f7ff6d194b9 in je_jemalloc_prefork ()
at /usr/src/external/bsd/jemalloc/lib/../dist/src/jemalloc.c:3276
#7 0x00007f7ff7efa400 in ?? ()
#8 0x000000000000009c in ?? ()
#9 0x00007f7ff780a093 in pthread_sigmask (how=<optimized out>, set=<optimized out>, oset=<optimized out>)
at /usr/src/lib/libpthread/pthread_misc.c:146
#10 0x0000000000b8ebf6 in std::sys::unix::process::process_inner::<impl std::sys::unix::process::process_common::Command>::do_exec () at src/libstd/sys/unix/process/process_unix.rs:225
#11 0x0000000000b8e4e0 in std::sys::unix::process::process_inner::<impl std::sys::unix::process::process_common::Command>::spawn () at src/libstd/sys/unix/process/process_unix.rs:50
#12 0x0000000000b80cda in std::process::Command::spawn () at src/libstd/process.rs:768
#13 0x00000000005d2ee4 in cargo::util::process_builder::ProcessBuilder::exec_with_streaming ()
#14 0x000000000067c13a in <cargo::core::compiler::DefaultExecutor as cargo::core::compiler::Executor>::exec ()
#15 0x0000000000671ec8 in <F as cargo::core::compiler::job::FnBox<A,R>>::call_box ()
#16 0x000000000066fcff in <F as cargo::core::compiler::job::FnBox<A,R>>::call_box ()
#17 0x000000000066fcff in <F as cargo::core::compiler::job::FnBox<A,R>>::call_box ()
#18 0x00000000003c73a9 in crossbeam_utils::thread::ScopedThreadBuilder::spawn::{{closure}} ()
#19 0x0000000000417adb in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#20 0x0000000000b903ea in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:80
#21 0x00000000004de289 in core::ops::function::FnOnce::call_once{{vtable-shim}} ()
#22 0x0000000000b7377f in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once ()
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/liballoc/boxed.rs:787
#23 0x0000000000b8f590 in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once ()
at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/liballoc/boxed.rs:787
#24 std::sys_common::thread::start_thread () at src/libstd/sys_common/thread.rs:13
#25 std::sys::unix::thread::Thread::new::thread_start () at src/libstd/sys/unix/thread.rs:79
#26 0x00007f7ff780c170 in pthread__create_tramp (cookie=0x7f7ff4521000)
at /usr/src/lib/libpthread/pthread.c:593
#27 0x00007f7ff6c8fcd0 in ?? () from /usr/lib/libc.so.12
>How-To-Repeat:
Run pbulk on -current/amd64 with firefox or evince in the package list.
>Fix:
>Release-Note:
>Audit-Trail:
From: Patrick Welche <prlw1@cam.ac.uk>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: pkg/54795: rust pbulk build stalls with 2 idle cargo processes
Date: Tue, 14 Jan 2020 09:28:47 +0000
The standard pbulk rust failure seems to have changed for the second time.
Now I see:
thread 'rustc' panicked at 'could not initialize thread_rng: No such file or directory', /cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.7.0/src/rngs/thread.rs:65:17
repeatedly.
This is with
MAKE_JOBS.lang/rust=1
MAKE_JOBS.rust=1
in mk.conf, which apparently has no effect, and also with, in addition
Index: Makefile
===================================================================
RCS file: /cvsroot/pkgsrc/lang/rust/Makefile,v
retrieving revision 1.136
diff -u -r1.136 Makefile
--- Makefile 12 Jan 2020 20:20:27 -0000 1.136
+++ Makefile 14 Jan 2020 09:27:36 -0000
@@ -98,7 +98,7 @@
CONFIGURE_ARGS+= --enable-vendor
# cargo defaults to using the number of available CPUs
-MAKE_ENV+= CARGO_BUILD_JOBS=${MAKE_JOBS:U1}
+MAKE_ENV+= CARGO_BUILD_JOBS=1
CFLAGS.SunOS+= -D_POSIX_PTHREAD_SEMANTICS
@@ -351,7 +351,7 @@
do-build:
cd ${WRKSRC} \
&& ${SETENV} ${MAKE_ENV} \
- ${PYTHONBIN} ./x.py -v ${BUILD_TARGET} -j ${MAKE_JOBS:U1}
+ ${PYTHONBIN} ./x.py -v ${BUILD_TARGET} -j1
do-install:
cd ${WRKSRC} \
Repeatable on various baremetal multicore -current/amd64 (9.99.37 this round)
From: Patrick Welche <prlw1@cam.ac.uk>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: pkg/54795: rust pbulk build stalls with 2 idle cargo processes
Date: Tue, 14 Jan 2020 12:55:40 +0000
On Tue, Jan 14, 2020 at 09:30:02AM +0000, Patrick Welche wrote:
> thread 'rustc' panicked at 'could not initialize thread_rng: No such file or directory', /cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.7.0/src/rngs/thread.rs:65:17
Compiling proc-macro2 v0.4.30
thread 'rustc' panicked at 'could not initialize thread_rng: No such file or directory', /cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.7.0/src/rngs/thread.rs:65
:17
stack backtrace:
0: 0x705cd1a63674 - backtrace::backtrace::libunwind::trace::hda41dbcdfba36aa0
at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.37/src/backtrace/libunwind.rs:88
1: 0x705cd1a63674 - backtrace::backtrace::trace_unsynchronized::h24db1eb743e32b7a
at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.37/src/backtrace/mod.rs:66
2: 0x705cd1a63674 - std::sys_common::backtrace::_print_fmt::h0ecd5de0be695bc9
at src/libstd/sys_common/backtrace.rs:76
3: 0x705cd1a63674 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hf24f776311cdf529
at src/libstd/sys_common/backtrace.rs:60
4: 0x705cd1a9655c - core::fmt::write::h01edf6dd68a42c9c
at src/libcore/fmt/mod.rs:1030
5: 0x705cd1a57737 - std::io::Write::write_fmt::ha096f698ec4136ad
at src/libstd/io/mod.rs:1412
6: 0x705cd1a67785 - std::sys_common::backtrace::_print::h4b33c71a5db06ff0
at src/libstd/sys_common/backtrace.rs:64
7: 0x705cd1a67785 - std::sys_common::backtrace::print::h690a5e4a78826230
at src/libstd/sys_common/backtrace.rs:49
8: 0x705cd1a67785 - std::panicking::default_hook::{{closure}}::h9252dff4eab26d22
at src/libstd/panicking.rs:196
9: 0x705cd1a6744a - std::panicking::default_hook::hf5f567043bc78ff4
at src/libstd/panicking.rs:210
10: 0x705cd20aa923 - rustc_driver::report_ice::h029eb1c685133605
11: 0x705cd1a67f45 - std::panicking::rust_panic_with_hook::h9a662f58cf3f8ffe
at src/libstd/panicking.rs:477
12: 0x705cd1a67a02 - std::panicking::continue_panic_fmt::h9fdc31ae90c76747
at src/libstd/panicking.rs:380
13: 0x705cd1a6795c - std::panicking::begin_panic_fmt::hb07c3a07faf6b5b3
at src/libstd/panicking.rs:335
14: 0x705cd2f47815 - rand::rngs::thread::THREAD_RNG_KEY::__init::{{closure}}::ha4adcba90076ab9d
15: 0x705cd2f477a5 - rand::rngs::thread::THREAD_RNG_KEY::__init::h7e39fdcb92ef8a2d
16: 0x705cd2f47cfd - std::thread::local::os::Key<T>::get::h2ed2fe7c1e2b0261
17: 0x705cd2f473d4 - rand::rngs::thread::thread_rng::h7984a0a450323364
18: 0x705cd2e9402b - tempfile::util::tmpname::h598b103962fed5e7
19: 0x705cd2e94231 - tempfile::util::create_helper::h52477625ed23b083
20: 0x705cd2e94e60 - tempfile::Builder::tempdir::ha28750b29f0d9a55
21: 0x705cc9e6aed4 - <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_utils::codegen_backend::CodegenBackend>::join_codegen_and_link::{{closure}}::h4eae87c
904bd3556
22: 0x705cc9e5f30e - rustc::util::common::time::h55592f1e733949b6
23: 0x705cc9f52c21 - <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_utils::codegen_backend::CodegenBackend>::join_codegen_and_link::h9dc788bfa618867f
24: 0x705cd213a2f6 - rustc_interface::queries::Query<T>::compute::hc3bb31139578310f
25: 0x705cd221da14 - rustc_interface::queries::<impl rustc_interface::interface::Compiler>::link::habe102fd93336e56
26: 0x705cd2077725 - rustc_interface::interface::run_compiler_in_existing_thread_pool::h3a72bbb61e04398e
27: 0x705cd20ac652 - std::thread::local::LocalKey<T>::with::h45bd8a6d7bb8686d
28: 0x705cd20b159e - scoped_tls::ScopedKey<T>::set::h8ddf7511ecdc5a8c
29: 0x705cd20d25a2 - syntax::with_globals::h9bcededffa9387ea
30: 0x705cd203fc70 - std::sys_common::backtrace::__rust_begin_short_backtrace::h80c4b91e3979ec73
31: 0x705cd1a778ea - __rust_maybe_catch_panic
at src/libpanic_unwind/lib.rs:80
32: 0x705cd2069759 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h8fe50374e2f991a9
33: 0x705cd1a4a71f - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h3879f66a40898a4c
at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/liballoc/boxed.rs:922
34: 0x705cd1a763c0 - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h9dc0ea3d0bf5ea71
at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/liballoc/boxed.rs:922
35: 0x705cd1a763c0 - std::sys_common::thread::start_thread::hc03f67963ee443cf
at src/libstd/sys_common/thread.rs:13
36: 0x705cd1a763c0 - std::sys::unix::thread::Thread::new::thread_start::ha181305a817feafa
at src/libstd/sys/unix/thread.rs:79
37: 0x705cd0a0c038 - <unknown>
38: 0x705cd128fcd0 - <unknown>
error: internal compiler error: unexpected panic
note: the compiler unexpectedly panicked. this is a bug.
note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
note: rustc 1.39.0 (4560ea788 2019-11-04) running on x86_64-unknown-netbsd
note: compiler flags: -C debug-assertions=off -C overflow-checks=on -C opt-level=2 --crate-type bin
...
(both before and after today's -current libpthread changes)
From: Patrick Welche <prlw1@cam.ac.uk>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: pkg/54795: rust pbulk build stalls with 2 idle cargo processes
Date: Thu, 16 Jan 2020 15:56:10 +0000
(rust-llvm doesn't change anything)
From: Patrick Welche <prlw1@cam.ac.uk>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: pkg/54795: rust pbulk build stalls with 2 idle cargo processes
Date: Fri, 17 Jan 2020 09:43:32 +0000
Back to the original 2 idle cargo processes problem. (guessing memcmp.S revert)
From: nia <nia@NetBSD.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: pkg/54795: rust pbulk build stalls with 2 idle cargo processes
Date: Fri, 17 Jan 2020 13:11:28 +0000
On Fri, Jan 17, 2020 at 09:45:03AM +0000, Patrick Welche wrote:
> The following reply was made to PR pkg/54795; it has been noted by GNATS.
>
> From: Patrick Welche <prlw1@cam.ac.uk>
> To: gnats-bugs@netbsd.org
> Cc:
> Subject: Re: pkg/54795: rust pbulk build stalls with 2 idle cargo processes
> Date: Fri, 17 Jan 2020 09:43:32 +0000
>
> Back to the original 2 idle cargo processes problem. (guessing memcmp.S revert)
>
Are you sure this isn't the lack of system entropy problem?
I believe it's still present in the bootstrap compiler.
From: Patrick Welche <prlw1@cam.ac.uk>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: pkg/54795: rust pbulk build stalls with 2 idle cargo processes
Date: Fri, 17 Jan 2020 13:25:04 +0000
On Fri, Jan 17, 2020 at 01:15:01PM +0000, nia wrote:
> Are you sure this isn't the lack of system entropy problem?
This is with pbulk, and in my sandbox setup, I have:
ln -sf ${sandboxdir}/dev/urandom ${sandboxdir}/dev/random
From: "Maya Rashish" <maya@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/54795 CVS commit: pkgsrc/lang/rust
Date: Sun, 19 Jan 2020 20:46:40 +0000
Module Name: pkgsrc
Committed By: maya
Date: Sun Jan 19 20:46:40 UTC 2020
Modified Files:
pkgsrc/lang/rust: Makefile
Log Message:
rust: mark MAKE_JOBS_SAFE=no on NetBSD.
PR pkg/54795
To generate a diff of this commit:
cvs rdiff -u -r1.137 -r1.138 pkgsrc/lang/rust/Makefile
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->closed
State-Changed-By: maya@NetBSD.org
State-Changed-When: Fri, 22 May 2020 23:12:35 +0000
State-Changed-Why:
Joerg fixed the real fundamental cause for netbsd-current, and there is a workaround for NetBSD releases (MAKE_JOBS_SAFE=no).
Please bug report if you are still having issues.
From: Patrick Welche <prlw1@cam.ac.uk>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: pkg/54795 (rust pbulk build stalls with 2 idle cargo processes)
Date: Tue, 1 Sep 2020 08:55:28 +0100
Yes it's fixed - I should have closed the bug earlier. Thanks!
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.46 2020/01/03 16:35:01 leot Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2020
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home