NetBSD Problem Report #52755

From khorben@defora.org  Wed Nov 22 23:52:55 2017
Return-Path: <khorben@defora.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 55D287A1F8
	for <gnats-bugs@gnats.NetBSD.org>; Wed, 22 Nov 2017 23:52:55 +0000 (UTC)
Message-Id: <20171122235414.E25E8109C5@tungsten.defora.rom>
Date: Thu, 23 Nov 2017 00:54:14 +0100 (CET)
From: Pierre Pronchery <khorben@defora.org>
Reply-To:
To: gnats-bugs@NetBSD.org
Subject: lang/python27 needs comp.tgz to find shared libraries
X-Send-Pr-Version: 3.95

>Number:         52755
>Category:       pkg
>Synopsis:       ctypes.util.find_library() fails without the "comp" set
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    pkg-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Nov 22 23:55:00 +0000 2017
>Last-Modified:  Tue Feb 20 09:35:01 +0000 2018
>Originator:     Pierre Pronchery <khorben@defora.org>
>Release:        NetBSD 7.1_STABLE
>Organization:
>Environment:
System: NetBSD hostname 7.1_STABLE NetBSD 7.1_STABLE (CGDROOT_XEN3_DOM0) amd64
Architecture: x86_64
Machine: amd64
>Description:
Many Python-based programs and packages will fail to work without the "comp"
set installed, since ctypes.util.find_library() will not find any library
without it being installed.

This is notably the case with sysutils/salt:

hostname# salt-key -F
Traceback (most recent call last):
  File "/usr/pkg/bin/salt-key", line 10, in <module>
    salt_key()
  File "/usr/pkg/lib/python2.7/site-packages/salt/scripts.py", line 361, in salt
_key
[...]
  File "/usr/pkg/lib/python2.7/site-packages/salt/utils/rsax931.py", line 49, in
 _load_libcrypto
    raise OSError('Cannot locate OpenSSL libcrypto')
OSError: Cannot locate OpenSSL libcrypto

The documentation for Python's ctypes says:

> On Linux, find_library tries to run external programs (/sbin/ldconfig, gcc,
> and objdump) to find the library file. It returns the filename of the library
> file. Here are sone examples: 

So I suppose on NetBSD it relies on gcc and objdump.

>How-To-Repeat:

hostname# ls /etc/mtree/set.comp
ls: /etc/mtree/set.comp: No such file or directory
hostname# python2.7 
Python 2.7.14 (default, Oct 24 2017, 00:11:35) 
[GCC 4.8.5] on netbsd7
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes.util
>>> print ctypes.util.find_library("crypto")
None
>>> ^D
hostname# progress -f comp.tgz tar xzpf -        
100% |***********************************|   104 MiB   11.44 MiB/s    00:00 ETA
hostname# python2.7 
>>> import ctypes.util
>>> print ctypes.util.find_library("crypto")
libcrypto.so.8

>Fix:
To work around the problem, install the "comp" set. However I do not
think we can expect users to have to install it. I avoid installing it
as often as possible myself (notably for security reasons).

I am not sure what the proper fix should be:
- let lang/python* depend on an implementation of lang/gcc
- adding a message when installing lang/python*

I would clearly prefer the former.

>Audit-Trail:
From: Hauke Fath <hf@spg.tu-darmstadt.de>
To: gnats-bugs@NetBSD.org
Cc: pkg-manager@NetBSD.org, gnats-admin@NetBSD.org
Subject: Re: pkg/52755: lang/python27 needs comp.tgz to find shared libraries
Date: Fri, 24 Nov 2017 12:20:28 +0100

 On 11/23/17 00:55, Pierre Pronchery wrote:
 > Many Python-based programs and packages will fail to work without the "comp"
 > set installed, since ctypes.util.find_library() will not find any library
 > without it being installed.

 So you are installing a programming environment (python), but 
 explicitely not the native toolchain. Then you find that some aspects of 
 the programming environment rely on native toolchain bits.

 While python's autohell build system is perfectly capable of finding the 
 system libraries, a runtime module (ctypes) is not, and relies on 
 external tools for the task. Looks like a python ctypes bug to me - have 
 you taken the issue upstream?

 >> Fix:
 > To work around the problem, install the "comp" set. However I do not
 > think we can expect users to have to install it. 

 Why? How is it unreasonable to expect the native toolchain around when 
 you install a programming environment?

 > I avoid installing it
 > as often as possible myself (notably for security reasons).

 Fun fact: The last "malware installation" I have seen in the wild was a 
 SIP scanner, implemented in - python.

 > I am not sure what the proper fix should be:
 > - let lang/python* depend on an implementation of lang/gcc
 > - adding a message when installing lang/python*

    - ask upstream to let ctypes fall back to a compile-time search path 
 to find its libraries. It does that on Windows and Mac OS X, already 
 <http://docs.activestate.com/activepython/2.5/python/lib/ctypes-finding-shared-libraries.html>.

 > I would clearly prefer the former.

 You refuse to install the comp set "because security", and then turn 
 around to build and install gcc (binutils rather, if you want 
 objdump(1)) for every python installation? Apart from the bloat and 
 churn this adds - how is it more secure?

 Cheerio,
 hauke

 -- 
       The ASCII Ribbon Campaign                    Hauke Fath
 ()     No HTML/RTF in email	        Institut für Nachrichtentechnik
 /\     No Word docs in email                     TU Darmstadt
       Respect for open standards              Ruf +49-6151-16-21344

From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/52755: lang/python27 needs comp.tgz to find shared libraries
Date: Tue, 20 Feb 2018 09:31:19 +0000

 On Wed, Nov 22, 2017 at 11:55:01PM +0000, Pierre Pronchery wrote:
  > Many Python-based programs and packages will fail to work without the "comp"
  > set installed, since ctypes.util.find_library() will not find any library
  > without it being installed.

 I don't think it's reasonable to expect 3rd-party software to work
 with parts of base missing.

  > To work around the problem, install the "comp" set. However I do not
  > think we can expect users to have to install it. I avoid installing it
  > as often as possible myself (notably for security reasons).

 ?

  > I am not sure what the proper fix should be:
  > - let lang/python* depend on an implementation of lang/gcc
  > - adding a message when installing lang/python*
  > 
  > I would clearly prefer the former.

 Perhaps you should figure out what it's doing that's not working, first...

 -- 
 David A. Holland
 dholland@netbsd.org

Home	PR Database Search