NetBSD Problem Report #46661

From root@infra.vm.internal.torchbox.com  Thu Jul  5 15:05:19 2012
Return-Path: <root@infra.vm.internal.torchbox.com>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
	by www.NetBSD.org (Postfix) with ESMTP id 135B763B882
	for <gnats-bugs@gnats.NetBSD.org>; Thu,  5 Jul 2012 15:05:19 +0000 (UTC)
Message-Id: <20120705134651.BD23DB3DA8@infra.vm.internal.torchbox.com>
Date: Thu,  5 Jul 2012 14:46:51 +0100 (BST)
From: River Tarnell <river@RT.UK.EU.ORG>
Reply-To: River Tarnell <river@RT.UK.EU.ORG>
To: gnats-bugs@gnats.NetBSD.org
Subject: libpthread shouldn't provide __res_state()
X-Send-Pr-Version: 3.95

>Number:         46661
>Category:       lib
>Synopsis:       libpthread shouldn't provide __res_state()
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Jul 05 15:10:01 +0000 2012
>Last-Modified:  Thu Jul 05 17:50:02 +0000 2012
>Originator:     River Tarnell <river@RT.UK.EU.ORG>
>Release:        NetBSD 6.0_BETA2
>Organization:
>Environment:
System: NetBSD infra.vm.internal.torchbox.com 6.0_BETA2 NetBSD 6.0_BETA2 (XEN3_DOMU) amd64
Architecture: x86_64
Machine: amd64
>Description:

libpthread (src/lib/libpthread/res_state.c) provides its own version of 
__res_state() that aborts when called.  This breaks programs that use _res
and are linked to libpthread (examples include Postfix and Exim when using
libmysqlclient or libpq, and the DCC anti-spam system).

I've looked through the code and I don't see any reason why _res can't be
used from a program that links with -lpthread.  While it's not a thread-safe
interface, using it from a single thread would appear to work fine.

The libpthread __res_state should be removed to allow this to work again.

>How-To-Repeat:

yvaine# cat test.c
#include        <sys/types.h>
#include        <sys/socket.h>
#include        <netinet/in.h>
#include        <arpa/inet.h>
#include        <resolv.h>

int main() {
        void *p = &_res;
}
yvaine# cc test.c -o test -lpthread
yvaine# ./test
_res is not supported for multi-threaded programs.
zsh: abort (core dumped)  ./test

>Fix:
Remove __res_state from src/lib/libpthread/res_state.c.

>Audit-Trail:
From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, lib-bug-people@netbsd.org, 
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Cc: 
Subject: Re: lib/46661: libpthread shouldn't provide __res_state()
Date: Thu, 5 Jul 2012 11:20:50 -0400

 On Jul 5,  3:10pm, river@RT.UK.EU.ORG (River Tarnell) wrote:
 -- Subject: lib/46661: libpthread shouldn't provide __res_state()
 | 
 | I've looked through the code and I don't see any reason why _res can't be
 | used from a program that links with -lpthread.  While it's not a thread-safe
 | interface, using it from a single thread would appear to work fine.

 There is plenty good reason for not doing that; using the non-threaded
 versions of the resolver variable will not just "work fine". Just read
 the code (in res_send.c).

 Instead start using the thread-safe versions of the functions (it is
 as simple as changing res_send() to res_nsend() for example and managing
 your own _res_state with res_ninit()/res_nclose()). They've been there for
 almost a decade now, and there is no excuse for dicking with _res directly
 anymore!

 christos

From: River Tarnell <river@RT.UK.EU.ORG>
To: gnats-bugs@NetBSD.org
Cc: lib-bug-people@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org
Subject: Re: lib/46661: libpthread shouldn't provide __res_state()
Date: Thu, 5 Jul 2012 16:37:07 +0100

 Christos Zoulas:
 >  | I've looked through the code and I don't see any reason why _res can't=
  be
 >  | used from a program that links with -lpthread.  While it's not a threa=
 d-safe
 >  | interface, using it from a single thread would appear to work fine.
  =20
 >  There is plenty good reason for not doing that; using the non-threaded
 >  versions of the resolver variable will not just "work fine". Just read
 >  the code (in res_send.c).
  =20
 I've looked through the code but I don't see an obvious problem (which=20
 is not to say there isn't one, I just can't see it).  As I understand=20
 it, _res is not really used except in res_init() where it's copied to=20
 _nres, the new global structure for the non-thread-safe resolver=20
 functions (which still appear to work when -lpthread is linked, as long=20
 as the user doesn't touch _res -- at least, they don't abort).

 =46rom an API point of view I don't see why _res shouldn't work in=20
 programs linked against -lpthread; of course in multi-threaded programs=20
 there is a re-entrancy problem, but the programs in question do not=20
 create any threads other than the main thread (i.e., they pull in=20
 -lpthread because they are linked against other shared libraries that=20
 do, but they are single-threaded).  Is there an issue with NetBSD's=20
 implementation that prevents this from working?

 >  Instead start using the thread-safe versions of the functions (it is
 >  as simple as changing res_send() to res_nsend() for example and managing
 >  your own _res_state with res_ninit()/res_nclose()). They've been there f=
 or
 >  almost a decade now, and there is no excuse for dicking with _res direct=
 ly
 >  anymore!

 I don't disagree, but these functions are used in current, existing code=20
 and break popular software (such as Postfix) in certain configurations=20
 on NetBSD.  For that reason alone it would be nice if they worked as=20
 they do on other platforms.

 	- river.

From: christos@zoulas.com (Christos Zoulas)
To: River Tarnell <river@RT.UK.EU.ORG>, gnats-bugs@NetBSD.org
Cc: lib-bug-people@netbsd.org, gnats-admin@netbsd.org, 
	netbsd-bugs@netbsd.org
Subject: Re: lib/46661: libpthread shouldn't provide __res_state()
Date: Thu, 5 Jul 2012 11:52:39 -0400

 On Jul 5,  4:37pm, river@RT.UK.EU.ORG (River Tarnell) wrote:
 -- Subject: Re: lib/46661: libpthread shouldn't provide __res_state()

 | From an API point of view I don't see why _res shouldn't work in
 | programs linked against -lpthread; of course in multi-threaded programs
 | there is a re-entrancy problem, but the programs in question do not
 | create any threads other than the main thread (i.e., they pull in
 | -lpthread because they are linked against other shared libraries that
 | do, but they are single-threaded).  Is there an issue with NetBSD's
 | implementation that prevents this from working?

 No there is not, but there is no way to know that the program will use
 one thread to call the resolver once linking with -lpthread.

 | I don't disagree, but these functions are used in current, existing code
 | and break popular software (such as Postfix) in certain configurations
 | on NetBSD.  For that reason alone it would be nice if they worked as
 | they do on other platforms.

 Postfix itself is not broken, other things linking in it are. This
 is why we have pkgsrc, so that we can patch them, fix them and send
 the fixes back to mainline. This is similar to the issue with have
 with mutexes: requiring pthread_mutex_init() before using them;
 other OS's don't and people claim NetBSD is broken. If programs want to
 play with _res from a threaded context, they are broken and should be
 fixed.

 christos

From: River Tarnell <river@RT.UK.EU.ORG>
To: gnats-bugs@NetBSD.org
Cc: lib-bug-people@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org
Subject: Re: lib/46661: libpthread shouldn't provide __res_state()
Date: Thu, 5 Jul 2012 17:08:00 +0100

 Christos Zoulas:
 >  | From an API point of view I don't see why _res shouldn't work in
 >  | programs linked against -lpthread[...]
 >  | Is there an issue with NetBSD's
 >  | implementation that prevents this from working?

 >  No there is not, but there is no way to know that the program will use
 >  one thread to call the resolver once linking with -lpthread.

 So because a program *may* use _res wrongly, *all* programs that use 
 _res (and libpthread) are broken, even those that use it correctly?

 >  | I don't disagree, but these functions are used in current, existing code
 >  | and break popular software (such as Postfix) in certain configurations
 >  | on NetBSD.  For that reason alone it would be nice if they worked as
 >  | they do on other platforms.
 >  
 >  Postfix itself is not broken, other things linking in it are.

 Postfix is broken in the sense that it doesn't work (e.g. pkg/44656).

 The problem is that it's not clear whose fault it is.  Postfix is the 
 one calling _res, but it's entirely single-threaded, so there's nothing 
 wrong with that.  It links against the PostgreSQL client library, libpq.  
 libpq does not use _res (it uses the MP-safe resolv interface), but it 
 is MP-safe itself, and pulls in libpthread.  As a result, Postfix 
 doesn't work.

 >  This is similar to the issue with have
 >  with mutexes: requiring pthread_mutex_init() before using them;
 >  other OS's don't and people claim NetBSD is broken.

 The pthread API requires that pthread_mutex_init() be called, doesn't 
 it?  So, a program not doing so is clearly wrong.

 >  If programs want to play with _res from a threaded context, they are
 >  broken and should be fixed.

 But the program is single-threaded; it just happens to link against 
 libpthread.  Why shouldn't that work?

 	- river.

From: Matthew Mondor <mm_lists@pulsar-zone.net>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: lib/46661: libpthread shouldn't provide __res_state()
Date: Thu, 5 Jul 2012 13:11:07 -0400

 On Thu,  5 Jul 2012 16:10:06 +0000 (UTC)
 River Tarnell <river@RT.UK.EU.ORG> wrote:

 >  But the program is single-threaded; it just happens to link against 
 >  libpthread.  Why shouldn't that work?

 Does the build system link it implicitely to libpthread because it makes
 use of dlfcn(3) to load modules?

 I think I remember about some such issue with some software (i.e.
 Apache 1.3 + PHP + PHP-GD, but on netbsd-5).  GD somehow wasn't working
 well if linked against libpthread (I forgot why) but pkgsrc implicitely
 linked everything that used the module system against libpthread.

 There is more information about it under pkgsrc/mk/dlopen.builtin.mk, I
 wonder if some packages could benefit from a custom
 DLOPEN_REQUIRE_PTHREADS setting, when they're known not to use
 threads...
 -- 
 Matt

From: River Tarnell <river@RT.UK.EU.ORG>
To: gnats-bugs@NetBSD.org
Cc: lib-bug-people@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org
Subject: Re: lib/46661: libpthread shouldn't provide __res_state()
Date: Thu, 5 Jul 2012 18:19:24 +0100

 Matthew Mondor:
 >  Does the build system link it implicitely to libpthread because it makes
 >  use of dlfcn(3) to load modules?

 In this case it's because it links against libpq:

 yvaine# ldd /usr/pkg/lib/libpq.so
 /usr/pkg/lib/libpq.so:
 	...
         -lpthread.0 => /usr/lib/libpthread.so.0

 yvaine# ldd /usr/pkg/libexec/postfix/smtp   
 /usr/pkg/libexec/postfix/smtp:
 	...
         -lpq.5 => /usr/pkg/lib/libpq.so.5
         -lpthread.0 => /usr/lib/libpthread.so.0

 >  I think I remember about some such issue with some software (i.e.
 >  Apache 1.3 + PHP + PHP-GD, but on netbsd-5).  GD somehow wasn't working
 >  well if linked against libpthread (I forgot why) but pkgsrc implicitely
 >  linked everything that used the module system against libpthread.

 In the MySQL case (libmysqlclient) libpthread is really required because 
 the MySQL client is thread safe and uses pthread primitives for that.  I 
 don't know if that's the case in PostgreSQL as well off hand.

 	- river.

From: christos@zoulas.com (Christos Zoulas)
To: River Tarnell <river@RT.UK.EU.ORG>, gnats-bugs@NetBSD.org
Cc: lib-bug-people@netbsd.org, gnats-admin@netbsd.org, 
	netbsd-bugs@netbsd.org
Subject: Re: lib/46661: libpthread shouldn't provide __res_state()
Date: Thu, 5 Jul 2012 13:42:30 -0400

 On Jul 5,  5:08pm, river@RT.UK.EU.ORG (River Tarnell) wrote:
 -- Subject: Re: lib/46661: libpthread shouldn't provide __res_state()

 | So because a program *may* use _res wrongly, *all* programs that use 
 | _res (and libpthread) are broken, even those that use it correctly?

 Define correctly. If a program is linked against -lpthread, nothing
 prevents other things that it is linked with to use threads and/or
 the resolver functions.

 | Postfix is broken in the sense that it doesn't work (e.g. pkg/44656).
 | 
 | The problem is that it's not clear whose fault it is.  Postfix is the 
 | one calling _res, but it's entirely single-threaded, so there's nothing 
 | wrong with that.  It links against the PostgreSQL client library, libpq.  
 | libpq does not use _res (it uses the MP-safe resolv interface), but it 
 | is MP-safe itself, and pulls in libpthread.  As a result, Postfix 
 | doesn't work.
 | 
 | >  This is similar to the issue with have
 | >  with mutexes: requiring pthread_mutex_init() before using them;
 | >  other OS's don't and people claim NetBSD is broken.
 | 
 | The pthread API requires that pthread_mutex_init() be called, doesn't 
 | it?  So, a program not doing so is clearly wrong.

 Correct.

 | >  If programs want to play with _res from a threaded context, they are
 | >  broken and should be fixed.
 | 
 | But the program is single-threaded; it just happens to link against 
 | libpthread.  Why shouldn't that work?

 Because it cannot possibly know what other things it calls that use pthreads
 are doing. Anyway, the I just fixed postfix, and it will probably get fixed
 upstream too.

 christos

From: David Laight <david@l8s.co.uk>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: lib/46661: libpthread shouldn't provide __res_state()
Date: Thu, 5 Jul 2012 18:38:32 +0100

 On Thu, Jul 05, 2012 at 03:10:01PM +0000, River Tarnell wrote:
 > >Number:         46661
 > >Category:       lib
 > >Synopsis:       libpthread shouldn't provide __res_state()
 ...
 > yvaine# cc test.c -o test -lpthread
 > yvaine# ./test
 > _res is not supported for multi-threaded programs.
 > zsh: abort (core dumped)  ./test

 Where does that printf() come from?
 I though that libraries weren't allowed to write to stdout/stderr
 or assume that fds 1 or 2 were suitable places to write messages to.

 Any controlling terminal is probably safe, but not much else.

 	David

 -- 
 David Laight: david@l8s.co.uk

From: River Tarnell <river@RT.UK.EU.ORG>
To: gnats-bugs@NetBSD.org
Cc: lib-bug-people@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org
Subject: Re: lib/46661: libpthread shouldn't provide __res_state()
Date: Thu, 5 Jul 2012 18:47:04 +0100

 David Laight:
 >  > yvaine# ./test
 >  > _res is not supported for multi-threaded programs.
 >  > zsh: abort (core dumped)  ./test
 >  
 >  Where does that printf() come from?

 src/lib/libpthread/res_state.c:

 res_state
 __res_state(void)
 {
         static const char res[] = "_res is not supported for multi-threaded"
             " programs.\n";
         (void)write(STDERR_FILENO, res, sizeof(res) - 1);
         abort();
         return NULL;
 }

 	- river.

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.