NetBSD Problem Report #41893

From mmondor@pulsar-zone.net  Sat Aug 15 21:05:02 2009
Return-Path: <mmondor@pulsar-zone.net>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by www.NetBSD.org (Postfix) with ESMTP id 812CF63C284
	for <gnats-bugs@gnats.NetBSD.org>; Sat, 15 Aug 2009 21:05:02 +0000 (UTC)
Message-Id: <200908152104.n7FL4cRE004147@ginseng.xisop>
Date: Sat, 15 Aug 2009 17:04:38 -0400 (EDT)
From: Matthew Mondor <mm_lists@pulsar-zone.net>
Reply-To: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@gnats.NetBSD.org
Subject: Should devel/boehm-gc be built with thread support by default?
X-Send-Pr-Version: 3.95

>Number:         41893
>Category:       pkg
>Synopsis:       Should devel/boehm-gc be built with thread support by default?
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    pkg-manager
>State:          open
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Sat Aug 15 21:10:00 +0000 2009
>Last-Modified:  Mon Dec 30 09:30:00 +0000 2013
>Originator:     Matthew Mondor
>Release:        NetBSD 5.0_STABLE
>Organization:
>Environment:
System: NetBSD ginseng.xisop 5.0_STABLE NetBSD 5.0_STABLE (GENERIC_MM) #4: Thu Aug 6 01:01:17 EDT 2009 root@behemoth.xisop:/usr/obj/sys/arch/i386/compile/GENERIC_MM i386
Architecture: i386
Machine: i386
>Description:
	Some packages which can support threads but which are using
	boehm-gc need a boehm-gc also built with threads support.
	This option exists but is disabled by default.

	I've been using ECL with threads built against boehm-gc
	built with threads support, and that same thread-enabled
	boehm-gc is also used here by inkscape and elinks and it
	seems stable (note that these later two do not require a
	thread-enabled boehm-gc however).

	So should boehm-gc be built with thread support by default,
	at least on NetBSD-5+?
>How-To-Repeat:
>Fix:

>Release-Note:

>Audit-Trail:

Responsible-Changed-From-To: pkg-manager->recht
Responsible-Changed-By: asau@NetBSD.org
Responsible-Changed-When: Sat, 15 Aug 2009 21:46:55 +0000
Responsible-Changed-Why:
Assigned to maintainer.

(I propose turning threads on by default for NetBSD, FreeBSD, Linux
and Darwin at least.)


Responsible-Changed-From-To: recht->pkg-manager
Responsible-Changed-By: wiz@NetBSD.org
Responsible-Changed-When: Mon, 28 Feb 2011 15:08:54 +0000
Responsible-Changed-Why:
Reset responsible field for retired developer.


From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support
 by default?
Date: Sun, 28 Aug 2011 11:19:40 -0400

 How about this?  recht (the previous maintainer) retired since.

 Another possibility would be to have separate boehm-gc and
 boehm-gc-threads package, and have ECL depend on boehm-gc-threads via
 buildlink.

 It would also be possible to tell ECL to build and embed its own
 boehm-gc, which is part of the tree, although this duplicates things a
 little in RAM.  Is it a valid concern?

 Yet we since force pthread support for everything that uses dlopen(3),
 so why not also enable the current thread option by default on
 netbsd-5+ and Linux? (boehm-gc doesn't use DLFCN(3), though).

 Alternatively, would it be possible for the ECL package, if built with
 the threaded option, to also tell its dependency, of not already
 present (boehm-gc) to also build with threads support?  This is not
 perfect though, as boehm-gc might already have been built.  In which
 case, perhaps that building ECL with threads should fail with an error,
 unless using an option to tell it to embed its own?  (or, it actually
 could enable an option automatically in this case to build its own)?

 Thanks,
 -- 
 Matt

From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support by
 default?
Date: Sat, 5 Nov 2011 13:52:24 +0000

 On Sun, Aug 28, 2011 at 03:20:03PM +0000, Matthew Mondor wrote:
  >  How about this?  recht (the previous maintainer) retired since.
  >  
  >  Another possibility would be to have separate boehm-gc and
  >  boehm-gc-threads package, and have ECL depend on boehm-gc-threads via
  >  buildlink.
  >
  >  It would also be possible to tell ECL to build and embed its own
  >  boehm-gc, which is part of the tree, although this duplicates things a
  >  little in RAM.  Is it a valid concern?

 The primary concern is that if a vulnerability appears in the package,
 someone has to remember where the extra private copies are and mark
 those packages vulnerable too... and then someone has to apply the
 fixes. This is usually a bad idea, so we like to not use included
 private copies of things.

  >  Yet we since force pthread support for everything that uses dlopen(3),
  >  so why not also enable the current thread option by default on
  >  netbsd-5+ and Linux? (boehm-gc doesn't use DLFCN(3), though).

 Well, the concern I'd have is that thread support may add a lot of
 overhead. Threaded garbage collection is painful enough in general
 that people go to substantial lengths to avoid it on purpose; I don't
 know to what extent this applies to boehm-gc's thread support.

 If there's not a significant problem, then we should enable it by
 default, as non-threaded/single-core environments are not the future
 or even at this point the present.

 If there is a problem, a separate package is probably the way to go;
 however, I'd go so far as to say the default package should be
 threaded and the other package should be "boehm-gc-unthreaded".

 As to whether there's a problem... that's the sort of thing we have
 package maintainers to know about. I have no idea. If you'd be willing
 to look into it, which hopefully by now should require only digging
 around and not actually benchmarking things, we can probably move
 forward on this...

  >  Alternatively, would it be possible for the ECL package, if built with
  >  the threaded option, to also tell its dependency, of not already
  >  present (boehm-gc) to also build with threads support?  This is not
  >  perfect though, as boehm-gc might already have been built.  In which
  >  case, perhaps that building ECL with threads should fail with an error,
  >  unless using an option to tell it to embed its own?  (or, it actually
  >  could enable an option automatically in this case to build its own)?

 This sort of thing doesn't really work and probably isn't really a
 good idea...

 -- 
 David A. Holland
 dholland@netbsd.org

From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support
 by default?
Date: Sat, 5 Nov 2011 17:14:57 -0400

 On Sat,  5 Nov 2011 13:55:02 +0000 (UTC)
 David Holland <dholland-pbugs@netbsd.org> wrote:

 >   >  It would also be possible to tell ECL to build and embed its own
 >   >  boehm-gc, which is part of the tree, although this duplicates things a
 >   >  little in RAM.  Is it a valid concern?
 >  
 >  The primary concern is that if a vulnerability appears in the package,
 >  someone has to remember where the extra private copies are and mark
 >  those packages vulnerable too... and then someone has to apply the
 >  fixes. This is usually a bad idea, so we like to not use included
 >  private copies of things.

 True, I agree that dynamic linking has an advantage here.

 >   >  Yet we since force pthread support for everything that uses dlopen(3),
 >   >  so why not also enable the current thread option by default on
 >   >  netbsd-5+ and Linux? (boehm-gc doesn't use DLFCN(3), though).
 >  
 >  Well, the concern I'd have is that thread support may add a lot of
 >  overhead. Threaded garbage collection is painful enough in general
 >  that people go to substantial lengths to avoid it on purpose; I don't
 >  know to what extent this applies to boehm-gc's thread support.
 >  
 >  If there's not a significant problem, then we should enable it by
 >  default, as non-threaded/single-core environments are not the future
 >  or even at this point the present.
 >  
 >  If there is a problem, a separate package is probably the way to go;
 >  however, I'd go so far as to say the default package should be
 >  threaded and the other package should be "boehm-gc-unthreaded".
 >  
 >  As to whether there's a problem... that's the sort of thing we have
 >  package maintainers to know about. I have no idea. If you'd be willing
 >  to look into it, which hopefully by now should require only digging
 >  around and not actually benchmarking things, we can probably move
 >  forward on this...

 I've not checked the code, but according to the tech docs:

 According to http://www.hpl.hp.com/personal/Hans_Boehm/gc/gcdescr.html,
 it uses a stop-world mark-and-sweep algorithm, with some support or
 incremental and generational modes for some operating systems (I'm not
 sure if that's true for BSD yet, but no BSD is listed in the OS
 enumeration.

 It also supports an experimental parallel-mark, but by default,
 according to http://www.hpl.hp.com/personal/Hans_Boehm/gc/scale.html:

     It causes the garbage collector to stop all other threads when it
     needs to see a consistent memory state. It causes the collector to
     acquire a lock around essentially all allocation and garbage
     collection activity. Since a single lock is used for all
     allocation-related activity, only one thread can be allocating or
     collecting at one point. This inherently limits performance of
     multi-threaded applications on multiprocessors.

 Personally, I currently am using a boehm-gc built with threads support,
 as I use ECL built with threads support as well.  I can't say that it's
 a problem on my slowest system (a 2793.16 MHz cache-crippled Celeron).
 However, boehm-gc is only used by ECL, w3m and Inkscape on my own
 systems.  I'm not sure that a boehm-gc-unthreaded would even be
 necessary, but who knows...

 Thanks,
 -- 
 Matt

From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support
 by default?
Date: Sat, 5 Nov 2011 19:27:15 -0400

 On Sat,  5 Nov 2011 21:15:04 +0000 (UTC)
 Matthew Mondor <mm_lists@pulsar-zone.net> wrote:

 >  Personally, I currently am using a boehm-gc built with threads support,
 >  as I use ECL built with threads support as well.  I can't say that it's
 >  a problem on my slowest system (a 2793.16 MHz cache-crippled Celeron).
 >  However, boehm-gc is only used by ECL, w3m and Inkscape on my own
 >  systems.  I'm not sure that a boehm-gc-unthreaded would even be
 >  necessary, but who knows...

 I connected to an Ubuntu system and can see that their boehm-gc deb
 package includes thread support in libgc.so too.  However, boehm-gc is
 certainly more tested on Linux (and tweaked for it) than on BSD...
 -- 
 Matt

From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support by
 default?
Date: Sun, 29 Dec 2013 20:58:26 +0000

 Matthew Mondor wrote:
  > 	So should boehm-gc be built with thread support by default,
  > 	at least on NetBSD-5+?

 Did we ever reach a conclusion on this? I recall there's been some
 discussion since the last comments on this PR were filed (which was
 two years ago) but I can't currently find it. :-/

 -- 
 David A. Holland
 dholland@netbsd.org

From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support
 by default?
Date: Mon, 30 Dec 2013 02:57:25 -0500

 On Sun, 29 Dec 2013 21:00:01 +0000 (UTC)
 David Holland <dholland-pbugs@netbsd.org> wrote:

 > The following reply was made to PR pkg/41893; it has been noted by GNATS.
 > 
 > From: David Holland <dholland-pbugs@netbsd.org>
 > To: gnats-bugs@NetBSD.org
 > Cc: 
 > Subject: Re: pkg/41893: Should devel/boehm-gc be built with thread support by
 >  default?
 > Date: Sun, 29 Dec 2013 20:58:26 +0000
 > 
 >  Matthew Mondor wrote:
 >   > 	So should boehm-gc be built with thread support by default,
 >   > 	at least on NetBSD-5+?
 >  
 >  Did we ever reach a conclusion on this? I recall there's been some
 >  discussion since the last comments on this PR were filed (which was
 >  two years ago) but I can't currently find it. :-/

 Your memory is right, indeed, a conclusion and consensus was recently
 reached on IRC that there should be two packages, one built with
 threads support, and the other built without threads, i.e. boehm-gc and
 boehm-gc-threads.

 There is at least one issue with boehm-gc and threads that I know about
 on NetBSD-6/amd64, which appear to have to do with rwlocks.  When libgc
 attempts to gc-or-grow, it then needs to stop the world, and threads
 using rwlocks (such as those using stdio) behave differently than what
 libgc expects.  Avoiding stdio and rwlocks, thread-enabled ECL with
 thread-enabled boehm-gc appear stable.  Interestingly, the signal sent
 by the stop-world procedure to other threads seems to be received, and
 the signal handler called, in rwlocks-using threads, yet those threads
 appear to then lock spinning on a mutex, if I remember.

 I think that the solution of having two packages would be rather
 straightforward to implement; that said, I could not personally put any
 time on it yet, unfortunately.

 As for the libgc+threads+rwlocks issue itself, it would be nice to fix
 as well eventually; I have the impression that the bug is more in libgc
 than in libpthread, but I've not looked seriously into it.  Among
 possible workarounds might be to override pthread_rwlock functions
 (i.e. in the case of ECL to build them around ECL's own mutex
 implementation), perhaps.  When I looked at the libgc source I seemed
 to be able to understand pretty much how it works, but the low level
 OS-specific part is extremely messy intertwined with many conditionals;
 that would require more time to debug and/or fix (and possibly looking
 at the results after it passed through the preprocessor).

 In my two projects based on ECL, the workaround has been to avoid stdio
 and rwlocks.  Unfortunately, this would not be possible in all cases;
 for instance some libraries used through FFI might require FILE
 handles.  In fact, one weak point in one of those two projects still
 rarely reproduces the issue, because libpcap is used which uses stdio
 fread(3) (eventually I'll probably have to replace libpcap by custom
 reads instead to make the application fully stable, as it requires
 threads, but at this point the application itself is embryonal).
 -- 
 Matt

>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.