NetBSD Problem Report #55126

From www@netbsd.org  Mon Mar 30 16:42:34 2020
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id C20C81A9213
	for <gnats-bugs@gnats.NetBSD.org>; Mon, 30 Mar 2020 16:42:34 +0000 (UTC)
Message-Id: <20200330164234.1E0D21A9242@mollari.NetBSD.org>
Date: Mon, 30 Mar 2020 16:42:33 +0000 (UTC)
From: mj@mjturner.net
Reply-To: mj@mjturner.net
To: gnats-bugs@NetBSD.org
Subject: lang/python38 fails to build on macOS 10.14.6 (Xcode 11.3)
X-Send-Pr-Version: www-1.0

>Number:         55126
>Category:       pkg
>Synopsis:       lang/python38 fails to build on macOS 10.14.6 (Xcode 11.3)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    macos-pkg-people
>State:          feedback
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Mar 30 16:45:00 +0000 2020
>Closed-Date:    
>Last-Modified:  Tue Jul 13 03:55:01 +0000 2021
>Originator:     Michael-John Turner
>Release:        pkgsrc -current
>Organization:
>Environment:
Darwin <redacted> 18.7.0 Darwin Kernel Version 18.7.0: Thu Jan 23 06:52:12 PST 2020; root:xnu-4903.278.25~1/RELEASE_X86_64 x86_64
>Description:
When attempting to build lang/python38 (and lang/python37 and lang/python27 - lang/python36 builds fine) on macOS 10.14.6 with Xcode 11.3, the build fails with the following errors:
---
=> Checking file-check results for python38-3.8.2nb1
ERROR: ************************************************************
ERROR: The following files are in the PLIST but not in /usr/obj/pkg/lang/python38/work/.destdir/usr/pkg:
ERROR:         /usr/obj/pkg/lang/python38/work/.destdir/usr/pkg/lib/python3.8/lib-dynload/_ctypes.so
ERROR:         /usr/obj/pkg/lang/python38/work/.destdir/usr/pkg/lib/python3.8/lib-dynload/_hashlib.so
ERROR:         /usr/obj/pkg/lang/python38/work/.destdir/usr/pkg/lib/python3.8/lib-dynload/_ssl.so
---


The issue appears to be related to the use of Xcode 11.3, which uses the macOS 10.15 SDK, even though the host OS is 11.14 in this case. adam@ tested and was able to build the package on macOS 10.14 with Xcode 10.3 (which uses the macOS 10.14 SDK).

Xcode 11.3 is the default on macOS 10.14 when installing from the App Store so it would be ideal if the build worked on that release (ie it doesn't require the manual installation of Xcode 10.x).



>How-To-Repeat:
1. Update pkgsrc to head.
2. On a machine running macOS 10.14.6 and Xcode 11.3.x (eg, 11.3.1), attempt to build lang/python38.
3. The build will fail with the error above.
>Fix:
There are two fixes that resolve this issue:
1) To ensure that _ctypes.so is built, the following diff can be applied to the package Makefile (thanks to adam@ for the pointer):

--- Makefile.orig	2020-03-12 20:08:20.000000000 +0000
+++ Makefile	2020-03-26 11:30:10.000000000 +0000
@@ -51,6 +51,7 @@
 .if ${OPSYS} == "Darwin"
 PY_PLATNAME=	${LOWER_OPSYS}
 USE_TOOLS+=	gmake
+USE_TOOLS+= pkg-config
 .elif ${OPSYS} == "IRIX"
 PY_PLATNAME=	${LOWER_OPSYS:C/\..*//}
 .elif ${OPSYS} == "SunOS"


2) To ensure that _hashlib.so and _ssl.so are built, a patch to setup.py is required. I'm not 100% certain of the correct fix, but a change to function detect_openssl_hashlib() corrected the issue. Removing the code that adds _ssl and _hashlib to self.missing results in both modules being built. 

In particular, the problematic code is:

---
openssl_includes = split_var('OPENSSL_INCLUDES', '-I')
...
# Find OpenSSL includes
ssl_incs = find_file(
	'openssl/ssl.h', self.inc_dirs, openssl_includes
)
if ssl_incs is None:
	self.missing.extend(['_ssl', '_hashlib'])
	return None, None
---

OPENSSL_INCLUDE has value "-I/usr/pkg/include". Checking the .buildlink
directory I see include/openssl/ssl.h but no usr/pkg. Not sure if it's supposed to be there or not?

Hope the above helps - should hopefully be a minor fix for someone more skilled in buildlink than me!


>Release-Note:

>Audit-Trail:

Responsible-Changed-From-To: pkg-manager->macos-pkg-people
Responsible-Changed-By: leot@NetBSD.org
Responsible-Changed-When: Tue, 31 Mar 2020 00:06:39 +0000
Responsible-Changed-Why:
macOS problem


State-Changed-From-To: open->feedback
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Sat, 10 Jul 2021 22:13:09 +0000
State-Changed-Why:
Does this happen with 3.8.11? At least the pkg-config part seems to have been
fixed.


From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: pkg/55126 (lang/python38 fails to build on macOS 10.14.6 (Xcode
 11.3))
Date: Sat, 10 Jul 2021 22:14:19 +0000

 On Sat, Jul 10, 2021 at 10:13:09PM +0000, dholland@NetBSD.org wrote:
  > Does this happen with 3.8.11? At least the pkg-config part seems to
  > have been fixed.

 If the other part is not, it sounds like things are confused about
 whether openssl is built-in or coming from pkgsrc; how do you have
 that set?

 -- 
 David A. Holland
 dholland@netbsd.org

From: Michael-John Turner <mj@mjturner.net>
To: gnats-bugs@netbsd.org
Cc: macos-pkg-people@netbsd.org, gnats-admin@netbsd.org,
	pkgsrc-bugs@netbsd.org
Subject: Re: pkg/55126 (lang/python38 fails to build on macOS 10.14.6 (Xcode
 11.3))
Date: Mon, 12 Jul 2021 17:18:28 +0100

 On Sat, Jul 10, 2021 at 10:15:01PM +0000, David Holland wrote:
 > On Sat, Jul 10, 2021 at 10:13:09PM +0000, dholland@NetBSD.org wrote:
 >  > Does this happen with 3.8.11? At least the pkg-config part seems to
 >  > have been fixed.
 >
 > If the other part is not, it sounds like things are confused about
 > whether openssl is built-in or coming from pkgsrc; how do you have
 > that set?

 Thanks for following up. Unfortunately I no longer have access to a macOS
 system running 10.14, all of my systems have been upgraded. I'll see if I
 can easily install a 10.14 VM and try and reproduce the issue. Will provide
 an update in the next few days.


From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: pkg/55126 (lang/python38 fails to build on macOS 10.14.6 (Xcode
 11.3))
Date: Tue, 13 Jul 2021 03:50:27 +0000

 On Mon, Jul 12, 2021 at 04:20:02PM +0000, Michael-John Turner wrote:
  >  On Sat, Jul 10, 2021 at 10:15:01PM +0000, David Holland wrote:
  >  > On Sat, Jul 10, 2021 at 10:13:09PM +0000, dholland@NetBSD.org wrote:
  >  >  > Does this happen with 3.8.11? At least the pkg-config part seems to
  >  >  > have been fixed.
  >  >
  >  > If the other part is not, it sounds like things are confused about
  >  > whether openssl is built-in or coming from pkgsrc; how do you have
  >  > that set?
  >  
  >  Thanks for following up. Unfortunately I no longer have access to a macOS
  >  system running 10.14, all of my systems have been upgraded. I'll see if I
  >  can easily install a 10.14 VM and try and reproduce the issue. Will provide
  >  an update in the next few days.

 If it works on later versions chasing after 10.14 in particular may
 not be worthwhile. So, only if you want to bother...

 -- 
 David A. Holland
 dholland@netbsd.org

>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.46 2020/01/03 16:35:01 leot Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2020 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.