NetBSD Problem Report #41893
From mmondor@pulsar-zone.net Sat Aug 15 21:05:02 2009
Return-Path: <mmondor@pulsar-zone.net>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by www.NetBSD.org (Postfix) with ESMTP id 812CF63C284
for <gnats-bugs@gnats.NetBSD.org>; Sat, 15 Aug 2009 21:05:02 +0000 (UTC)
Message-Id: <200908152104.n7FL4cRE004147@ginseng.xisop>
Date: Sat, 15 Aug 2009 17:04:38 -0400 (EDT)
From: Matthew Mondor <mm_lists@pulsar-zone.net>
Reply-To: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@gnats.NetBSD.org
Subject: Should devel/boehm-gc be built with thread support by default?
X-Send-Pr-Version: 3.95
>Number: 41893
>Category: pkg
>Synopsis: Should devel/boehm-gc be built with thread support by default?
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: pkg-manager
>State: closed
>Class: change-request
>Submitter-Id: net
>Arrival-Date: Sat Aug 15 21:10:00 +0000 2009
>Closed-Date: Mon Feb 01 15:18:39 +0000 2021
>Last-Modified: Mon Feb 01 15:18:39 +0000 2021
>Originator: Matthew Mondor
>Release: NetBSD 5.0_STABLE
>Organization:
>Environment:
System: NetBSD ginseng.xisop 5.0_STABLE NetBSD 5.0_STABLE (GENERIC_MM) #4: Thu Aug 6 01:01:17 EDT 2009 root@behemoth.xisop:/usr/obj/sys/arch/i386/compile/GENERIC_MM i386
Architecture: i386
Machine: i386
>Description:
Some packages which can support threads but which are using
boehm-gc need a boehm-gc also built with threads support.
This option exists but is disabled by default.
I've been using ECL with threads built against boehm-gc
built with threads support, and that same thread-enabled
boehm-gc is also used here by inkscape and elinks and it
seems stable (note that these later two do not require a
thread-enabled boehm-gc however).
So should boehm-gc be built with thread support by default,
at least on NetBSD-5+?
>How-To-Repeat:
>Fix:
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: pkg-manager->recht
Responsible-Changed-By: asau@NetBSD.org
Responsible-Changed-When: Sat, 15 Aug 2009 21:46:55 +0000
Responsible-Changed-Why:
Assigned to maintainer.
(I propose turning threads on by default for NetBSD, FreeBSD, Linux
and Darwin at least.)
Responsible-Changed-From-To: recht->pkg-manager
Responsible-Changed-By: wiz@NetBSD.org
Responsible-Changed-When: Mon, 28 Feb 2011 15:08:54 +0000
Responsible-Changed-Why:
Reset responsible field for retired developer.
From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support
by default?
Date: Sun, 28 Aug 2011 11:19:40 -0400
How about this? recht (the previous maintainer) retired since.
Another possibility would be to have separate boehm-gc and
boehm-gc-threads package, and have ECL depend on boehm-gc-threads via
buildlink.
It would also be possible to tell ECL to build and embed its own
boehm-gc, which is part of the tree, although this duplicates things a
little in RAM. Is it a valid concern?
Yet we since force pthread support for everything that uses dlopen(3),
so why not also enable the current thread option by default on
netbsd-5+ and Linux? (boehm-gc doesn't use DLFCN(3), though).
Alternatively, would it be possible for the ECL package, if built with
the threaded option, to also tell its dependency, of not already
present (boehm-gc) to also build with threads support? This is not
perfect though, as boehm-gc might already have been built. In which
case, perhaps that building ECL with threads should fail with an error,
unless using an option to tell it to embed its own? (or, it actually
could enable an option automatically in this case to build its own)?
Thanks,
--
Matt
From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support by
default?
Date: Sat, 5 Nov 2011 13:52:24 +0000
On Sun, Aug 28, 2011 at 03:20:03PM +0000, Matthew Mondor wrote:
> How about this? recht (the previous maintainer) retired since.
>
> Another possibility would be to have separate boehm-gc and
> boehm-gc-threads package, and have ECL depend on boehm-gc-threads via
> buildlink.
>
> It would also be possible to tell ECL to build and embed its own
> boehm-gc, which is part of the tree, although this duplicates things a
> little in RAM. Is it a valid concern?
The primary concern is that if a vulnerability appears in the package,
someone has to remember where the extra private copies are and mark
those packages vulnerable too... and then someone has to apply the
fixes. This is usually a bad idea, so we like to not use included
private copies of things.
> Yet we since force pthread support for everything that uses dlopen(3),
> so why not also enable the current thread option by default on
> netbsd-5+ and Linux? (boehm-gc doesn't use DLFCN(3), though).
Well, the concern I'd have is that thread support may add a lot of
overhead. Threaded garbage collection is painful enough in general
that people go to substantial lengths to avoid it on purpose; I don't
know to what extent this applies to boehm-gc's thread support.
If there's not a significant problem, then we should enable it by
default, as non-threaded/single-core environments are not the future
or even at this point the present.
If there is a problem, a separate package is probably the way to go;
however, I'd go so far as to say the default package should be
threaded and the other package should be "boehm-gc-unthreaded".
As to whether there's a problem... that's the sort of thing we have
package maintainers to know about. I have no idea. If you'd be willing
to look into it, which hopefully by now should require only digging
around and not actually benchmarking things, we can probably move
forward on this...
> Alternatively, would it be possible for the ECL package, if built with
> the threaded option, to also tell its dependency, of not already
> present (boehm-gc) to also build with threads support? This is not
> perfect though, as boehm-gc might already have been built. In which
> case, perhaps that building ECL with threads should fail with an error,
> unless using an option to tell it to embed its own? (or, it actually
> could enable an option automatically in this case to build its own)?
This sort of thing doesn't really work and probably isn't really a
good idea...
--
David A. Holland
dholland@netbsd.org
From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support
by default?
Date: Sat, 5 Nov 2011 17:14:57 -0400
On Sat, 5 Nov 2011 13:55:02 +0000 (UTC)
David Holland <dholland-pbugs@netbsd.org> wrote:
> > It would also be possible to tell ECL to build and embed its own
> > boehm-gc, which is part of the tree, although this duplicates things a
> > little in RAM. Is it a valid concern?
>
> The primary concern is that if a vulnerability appears in the package,
> someone has to remember where the extra private copies are and mark
> those packages vulnerable too... and then someone has to apply the
> fixes. This is usually a bad idea, so we like to not use included
> private copies of things.
True, I agree that dynamic linking has an advantage here.
> > Yet we since force pthread support for everything that uses dlopen(3),
> > so why not also enable the current thread option by default on
> > netbsd-5+ and Linux? (boehm-gc doesn't use DLFCN(3), though).
>
> Well, the concern I'd have is that thread support may add a lot of
> overhead. Threaded garbage collection is painful enough in general
> that people go to substantial lengths to avoid it on purpose; I don't
> know to what extent this applies to boehm-gc's thread support.
>
> If there's not a significant problem, then we should enable it by
> default, as non-threaded/single-core environments are not the future
> or even at this point the present.
>
> If there is a problem, a separate package is probably the way to go;
> however, I'd go so far as to say the default package should be
> threaded and the other package should be "boehm-gc-unthreaded".
>
> As to whether there's a problem... that's the sort of thing we have
> package maintainers to know about. I have no idea. If you'd be willing
> to look into it, which hopefully by now should require only digging
> around and not actually benchmarking things, we can probably move
> forward on this...
I've not checked the code, but according to the tech docs:
According to http://www.hpl.hp.com/personal/Hans_Boehm/gc/gcdescr.html,
it uses a stop-world mark-and-sweep algorithm, with some support or
incremental and generational modes for some operating systems (I'm not
sure if that's true for BSD yet, but no BSD is listed in the OS
enumeration.
It also supports an experimental parallel-mark, but by default,
according to http://www.hpl.hp.com/personal/Hans_Boehm/gc/scale.html:
It causes the garbage collector to stop all other threads when it
needs to see a consistent memory state. It causes the collector to
acquire a lock around essentially all allocation and garbage
collection activity. Since a single lock is used for all
allocation-related activity, only one thread can be allocating or
collecting at one point. This inherently limits performance of
multi-threaded applications on multiprocessors.
Personally, I currently am using a boehm-gc built with threads support,
as I use ECL built with threads support as well. I can't say that it's
a problem on my slowest system (a 2793.16 MHz cache-crippled Celeron).
However, boehm-gc is only used by ECL, w3m and Inkscape on my own
systems. I'm not sure that a boehm-gc-unthreaded would even be
necessary, but who knows...
Thanks,
--
Matt
From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support
by default?
Date: Sat, 5 Nov 2011 19:27:15 -0400
On Sat, 5 Nov 2011 21:15:04 +0000 (UTC)
Matthew Mondor <mm_lists@pulsar-zone.net> wrote:
> Personally, I currently am using a boehm-gc built with threads support,
> as I use ECL built with threads support as well. I can't say that it's
> a problem on my slowest system (a 2793.16 MHz cache-crippled Celeron).
> However, boehm-gc is only used by ECL, w3m and Inkscape on my own
> systems. I'm not sure that a boehm-gc-unthreaded would even be
> necessary, but who knows...
I connected to an Ubuntu system and can see that their boehm-gc deb
package includes thread support in libgc.so too. However, boehm-gc is
certainly more tested on Linux (and tweaked for it) than on BSD...
--
Matt
From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support by
default?
Date: Sun, 29 Dec 2013 20:58:26 +0000
Matthew Mondor wrote:
> So should boehm-gc be built with thread support by default,
> at least on NetBSD-5+?
Did we ever reach a conclusion on this? I recall there's been some
discussion since the last comments on this PR were filed (which was
two years ago) but I can't currently find it. :-/
--
David A. Holland
dholland@netbsd.org
From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support
by default?
Date: Mon, 30 Dec 2013 02:57:25 -0500
On Sun, 29 Dec 2013 21:00:01 +0000 (UTC)
David Holland <dholland-pbugs@netbsd.org> wrote:
> The following reply was made to PR pkg/41893; it has been noted by GNATS.
>
> From: David Holland <dholland-pbugs@netbsd.org>
> To: gnats-bugs@NetBSD.org
> Cc:
> Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support by
> default?
> Date: Sun, 29 Dec 2013 20:58:26 +0000
>
> Matthew Mondor wrote:
> > So should boehm-gc be built with thread support by default,
> > at least on NetBSD-5+?
>
> Did we ever reach a conclusion on this? I recall there's been some
> discussion since the last comments on this PR were filed (which was
> two years ago) but I can't currently find it. :-/
Your memory is right, indeed, a conclusion and consensus was recently
reached on IRC that there should be two packages, one built with
threads support, and the other built without threads, i.e. boehm-gc and
boehm-gc-threads.
There is at least one issue with boehm-gc and threads that I know about
on NetBSD-6/amd64, which appear to have to do with rwlocks. When libgc
attempts to gc-or-grow, it then needs to stop the world, and threads
using rwlocks (such as those using stdio) behave differently than what
libgc expects. Avoiding stdio and rwlocks, thread-enabled ECL with
thread-enabled boehm-gc appear stable. Interestingly, the signal sent
by the stop-world procedure to other threads seems to be received, and
the signal handler called, in rwlocks-using threads, yet those threads
appear to then lock spinning on a mutex, if I remember.
I think that the solution of having two packages would be rather
straightforward to implement; that said, I could not personally put any
time on it yet, unfortunately.
As for the libgc+threads+rwlocks issue itself, it would be nice to fix
as well eventually; I have the impression that the bug is more in libgc
than in libpthread, but I've not looked seriously into it. Among
possible workarounds might be to override pthread_rwlock functions
(i.e. in the case of ECL to build them around ECL's own mutex
implementation), perhaps. When I looked at the libgc source I seemed
to be able to understand pretty much how it works, but the low level
OS-specific part is extremely messy intertwined with many conditionals;
that would require more time to debug and/or fix (and possibly looking
at the results after it passed through the preprocessor).
In my two projects based on ECL, the workaround has been to avoid stdio
and rwlocks. Unfortunately, this would not be possible in all cases;
for instance some libraries used through FFI might require FILE
handles. In fact, one weak point in one of those two projects still
rarely reproduces the issue, because libpcap is used which uses stdio
fread(3) (eventually I'll probably have to replace libpcap by custom
reads instead to make the application fully stable, as it requires
threads, but at this point the application itself is embryonal).
--
Matt
From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support
by default?
Date: Thu, 1 May 2014 19:57:23 -0400
Here is a link to some test cases:
http://cvs.pulsar-zone.net/cgi-bin/cvsweb.cgi/mmondor/mmsoftware/cl/test/gc-bug-test.lisp
I confirm that the gc+threads+stdio issues still occur using libgc 7.2e
and netbsd-6/amd64 with ECL and threads.
--
Matt
From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support
by default?
Date: Fri, 2 May 2014 00:29:21 -0400
A few more related links for reference:
Recent change:
http://mail-index.netbsd.org/pkgsrc-changes/2014/04/30/msg105139.html
Thread "CVS commit: pkgsrc/devel/boehm-gc" at
http://mail-index.netbsd.org/pkgsrc-changes/2014/04/thread7.html
Answers from me:
http://mail-index.netbsd.org/pkgsrc-changes/2014/05/02/msg105200.html
http://mail-index.netbsd.org/pkgsrc-changes/2014/05/02/msg105202.html
--
Matt
From: "Thomas Klausner" <wiz@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/41893 CVS commit: pkgsrc/devel/boehm-gc
Date: Fri, 2 May 2014 07:28:00 +0000
Module Name: pkgsrc
Committed By: wiz
Date: Fri May 2 07:28:00 UTC 2014
Modified Files:
pkgsrc/devel/boehm-gc: Makefile options.mk
Log Message:
Revert previous: thread option only enabled by default on Darwin again
PR 41893.
To generate a diff of this commit:
cvs rdiff -u -r1.76 -r1.77 pkgsrc/devel/boehm-gc/Makefile
cvs rdiff -u -r1.2 -r1.3 pkgsrc/devel/boehm-gc/options.mk
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support
by default?
Date: Fri, 2 May 2014 03:36:37 -0400
Reversion to disable threads by default again except on Darwin:
http://mail-index.netbsd.org/pkgsrc-changes/2014/05/02/msg105206.html
The goal of this PR would still be to eventually enable them by
default, or the split in two packages, boehm-gc and boehm-gc-nothreads,
so it should probably remain open...
Thanks,
--
Matt
From: Thomas Klausner <wiz@NetBSD.org>
To: NetBSD bugtracking <gnats-bugs@NetBSD.org>
Cc:
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support by
default?
Date: Fri, 2 May 2014 13:00:20 +0200
On Fri, May 02, 2014 at 07:40:00AM +0000, Matthew Mondor wrote:
> Reversion to disable threads by default again except on Darwin:
>
> http://mail-index.netbsd.org/pkgsrc-changes/2014/05/02/msg105206.html
>
> The goal of this PR would still be to eventually enable them by
> default, or the split in two packages, boehm-gc and boehm-gc-nothreads,
> so it should probably remain open...
I'm not closing it, but I'll also not be doing the work :)
Thomas
From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support
by default?
Date: Tue, 8 Sep 2015 04:01:26 -0400
Also see: https://gitlab.com/embeddable-common-lisp/ecl/issues/161
--
Matt
State-Changed-From-To: open->closed
State-Changed-By: nia@NetBSD.org
State-Changed-When: Mon, 01 Feb 2021 15:18:39 +0000
State-Changed-Why:
It has been for some time.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.46 2020/01/03 16:35:01 leot Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2020
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.