NetBSD Problem Report #53170

From www@NetBSD.org  Sun Apr  8 17:22:04 2018
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 920187A1B6
	for <gnats-bugs@gnats.NetBSD.org>; Sun,  8 Apr 2018 17:22:04 +0000 (UTC)
Message-Id: <20180408172203.6BE577A202@mollari.NetBSD.org>
Date: Sun,  8 Apr 2018 17:22:03 +0000 (UTC)
From: keivan@motavalli.me
Reply-To: keivan@motavalli.me
To: gnats-bugs@NetBSD.org
Subject: get_compiler fails to detect clang if it's not named "gcc" in $PATH
X-Send-Pr-Version: www-1.0

>Number:         53170
>Category:       pkg
>Synopsis:       get_compiler fails to detect clang if it's not named "gcc" in $PATH
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    openbsd-pkg-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Apr 08 17:25:00 +0000 2018
>Closed-Date:    Wed Sep 12 20:18:57 +0000 2018
>Last-Modified:  Wed Sep 12 20:18:57 +0000 2018
>Originator:     keivan motavalli
>Release:        trunk/06-apr-2018
>Organization:
>Environment:
OpenBSD 6.3 GENERIC#100 amd64
>Description:
pkgsrc fails to bootstrap on recent OpenBSD releases because of a circular dependency on gcc3-c when compiling cwrappers as part of the procedure.

further fixes already discussed on pkgsrc-users are then needed to compile pkgtools/libarchive against LibreSSL >= 2.7, included in OpenBSD 6.3.

a known workaround for the circular dependency problem is to run the bootstrap script with --compiler clang, default C compiler since OpenBSD 6.2

modifying the bootstrap script to set check_compiler=yes when running in OpenBSD fails to detect clang and sets pkgsrc to use gcc.

this is because the system comes with both gcc (/usr/bin/gcc) and clang (/usr/bin/cc, /usr/bin/clang) installed, while the get_compiler() function first checks if a gcc executable exists in $PATH, then tests if it identifies as clang, so it tests the real gcc compiler reporting that clang is not present on the system.

the attached patch avoids searching for gcc and tests the default cc to identify it as either gcc or clang.

it doesn't explicitly search for gcc in $PATH anymore, but that search could be restored after the check on 'cc' if required.

>How-To-Repeat:
set check_compiler=yes when running on OpenBSD 6.2+ and run the bootstrap script
>Fix:
--- a/bootstrap/bootstrap
+++ b/bootstrap/bootstrap
@@ -134,37 +134,21 @@ get_compiler()
 	# normally, we'd just use 'cc', but certain configure tools look
 	# for gcc specifically, so we have to see if that comes first
 	if [ -z "${testcc}" ]; then
-		save_IFS="${IFS}"
-		IFS=':'
-		for dir in ${PATH}; do
-			test -z "$dir" && dir=.
-			if [ -x "$dir/gcc" ]; then
-				testcc="$dir/gcc"
-				break
-			fi
-		done
-		IFS="${save_IFS}"
-	fi
-
 	# Clang compiler pretends to be GCC, so we have to check that
-	cat >${wrkdir}/$$.c <<EOF
-#ifdef __clang__
-indeed
-#endif
-EOF
-	compiler_is_clang=`${testcc:-cc} -E ${wrkdir}/$$.c 2>/dev/null | grep -c indeed`
-	rm -f ${wrkdir}/$$.c
-
-	cat >${wrkdir}/$$.c <<EOF
-#ifdef __GNUC__
-#ifndef __clang__
-indeed
+		cat >${wrkdir}/$$.c <<EOF
+#define GCC_COMPILER (defined(__GNUC__) && !defined(__clang__))indeed
+#if GCC_COMPILER
+gcc
 #endif
+#ifdef __clang__
+clang
 #endif
 EOF
-	compiler_is_gnu=`${testcc:-cc} -E ${wrkdir}/$$.c 2>/dev/null | grep -c indeed`
-	rm -f ${wrkdir}/$$.c
-
+		compiler_is_gnu=`${testcc:-cc} -E ${wrkdir}/$$.c 2>/dev/null | grep -c gcc`
+		compiler_is_clang=`${testcc:-cc} -E ${wrkdir}/$$.c 2>/dev/null | grep -c clang`
+		rm -f ${wrkdir}/$$.c
+	fi
+	
 }
 get_abi()
 {
@@ -776,6 +760,7 @@ OpenBSD)
 	need_awk=no
 	need_sed=no
 	set_opsys=no
+	check_compiler=yes
 	machine_arch=`arch -s`
 	;;
 OSF1)

>Release-Note:

>Audit-Trail:

Responsible-Changed-From-To: pkg-manager->openbsd-pkg-people
Responsible-Changed-By: leot@NetBSD.org
Responsible-Changed-When: Sun, 08 Apr 2018 17:35:33 +0000
Responsible-Changed-Why:
OpenBSD problem.


State-Changed-From-To: open->feedback
State-Changed-By: leot@NetBSD.org
State-Changed-When: Sun, 08 Apr 2018 17:37:58 +0000
State-Changed-Why:
Possible patch shared.


From: Leonardo Taccari <leot@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: pkg/53170: get_compiler fails to detect clang if it's not named "gcc" in $PATH
Date: Sun, 08 Apr 2018 19:33:35 +0200

 Hello Keivan,

 keivan@motavalli.me writes:
 > >Number:         53170
 > >Category:       pkg
 > >Synopsis:       get_compiler fails to detect clang if it's not named "g=
 cc" in $PATH
 > >Confidential:   no
 > >Severity:       serious
 > >Priority:       low
 > >Responsible:    pkg-manager
 > >State:          open
 > >Class:          sw-bug
 > >Submitter-Id:   net
 > >Arrival-Date:   Sun Apr 08 17:25:00 +0000 2018
 > >Originator:     keivan motavalli
 > >Release:        trunk/06-apr-2018
 > >Organization:
 > >Environment:
 > OpenBSD 6.3 GENERIC#100 amd64
 > >Description:
 > pkgsrc fails to bootstrap on recent OpenBSD releases because of a circul=
 ar dependency on gcc3-c when compiling cwrappers as part of the procedure.
 >
 > further fixes already discussed on pkgsrc-users are then needed to compi=
 le pkgtools/libarchive against LibreSSL >=3D 2.7, included in OpenBSD 6.3.
 >
 > a known workaround for the circular dependency problem is to run the boo=
 tstrap script with --compiler clang, default C compiler since OpenBSD 6.2
 >
 > modifying the bootstrap script to set check_compiler=3Dyes when running =
 in OpenBSD fails to detect clang and sets pkgsrc to use gcc.
 > [...]

 Can you please try the following?:


 Index: bootstrap
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 RCS file: /cvsroot/pkgsrc/bootstrap/bootstrap,v
 retrieving revision 1.248
 diff -u -p -r1.248 bootstrap
 --- bootstrap	7 Mar 2018 07:34:07 -0000	1.248
 +++ bootstrap	8 Apr 2018 17:21:51 -0000
 @@ -777,6 +777,8 @@ OpenBSD)
  	need_sed=3Dno
  	set_opsys=3Dno
  	machine_arch=3D`arch -s`
 +	CC=3D${CC:-cc}; export CC
 +	check_compiler=3Dyes
  	;;
  OSF1)
  	root_group=3Dsystem


 (it is less intrusive and should do the trick!)


 Please let us know if this make pkgsrc bootstrap-able on recent OpenBSD!
 Thank you!

From: Keivan Motavalli <keivan@motavalli.me>
To: Leonardo Taccari <leot@NetBSD.org>
Cc: gnats-bugs@NetBSD.org, openbsd-pkg-people@NetBSD.org,
 gnats-admin@netbsd.org, pkgsrc-bugs@netbsd.org
Subject: Re: pkg/53170: get_compiler fails to detect clang if it's not named
 "gcc" in $PATH
Date: Sun, 8 Apr 2018 20:35:20 +0200

 On Sun,  8 Apr 2018 17:50:01 +0000 (UTC)
 Leonardo Taccari <leot@NetBSD.org> wrote:


 >  Can you please try the following?:

 >  Index: bootstrap
 >  
 >  --- bootstrap	7 Mar 2018 07:34:07 -0000	1.248
 >  +++ bootstrap	8 Apr 2018 17:21:51 -0000
 >  @@ -777,6 +777,8 @@ OpenBSD)
 >   	need_sed=3Dno
 >   	set_opsys=3Dno
 >   	machine_arch=3D`arch -s`
 >  +	CC=${CC:-cc}; export CC
 >  +	check_compiler=yes
 >   	;;
 >  
 >  (it is less intrusive and should do the trick!)

 it does the trick on both OpenBSD 6.2 and 6.3, but on 6.3
 pkgtools/libarchive needs to be patched to avoid redefining functions
 already implemented in LibreSSL 2.7

 I also tested my patch as working on OpenBSD 6.2 and 6.3 and it doesn't
 break FreeBSD 11.1. I haven't tested for Darwin yet

From: Keivan Motavalli <keivan@motavalli.me>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: pkg/53170: get_compiler fails to detect clang if it's not named "gcc" in $PATH
Date: Mon, 09 Apr 2018 11:10:32 +0200

 On Sunday, April 8, 2018 8:40:01 PM CEST you wrote:
 >  I also tested my patch as working on OpenBSD 6.2 and 6.3 and it doesn't
 >  break FreeBSD 11.1. I haven't tested for Darwin yet

 It doesn't work under Darwin (Mac OS 10.13): cc -E doesn't output anything to 
 stdout when running the preprocessor.

 furthermore, I had left a stray "indeed" in the original patch.

 --- a/bootstrap/bootstrap
 +++ b/bootstrap/bootstrap
 @@ -134,37 +134,21 @@ get_compiler()
  	# normally, we'd just use 'cc', but certain configure tools look
  	# for gcc specifically, so we have to see if that comes first
  	if [ -z "${testcc}" ]; then
 -		save_IFS="${IFS}"
 -		IFS=':'
 -		for dir in ${PATH}; do
 -			test -z "$dir" && dir=.
 -			if [ -x "$dir/gcc" ]; then
 -				testcc="$dir/gcc"
 -				break
 -			fi
 -		done
 -		IFS="${save_IFS}"
 -	fi
 -
  	# Clang compiler pretends to be GCC, so we have to check that
 -	cat >${wrkdir}/$$.c <<EOF
 -#ifdef __clang__
 -indeed
 -#endif
 -EOF
 -	compiler_is_clang=`${testcc:-cc} -E ${wrkdir}/$$.c 2>/dev/null | grep -c 
 indeed`
 -	rm -f ${wrkdir}/$$.c
 -
 -	cat >${wrkdir}/$$.c <<EOF
 -#ifdef __GNUC__
 -#ifndef __clang__
 -indeed
 +		cat >${wrkdir}/$$.c <<EOF
 +#define GCC_COMPILER (defined(__GNUC__) && !defined(__clang__))
 +#if GCC_COMPILER
 +gcc
  #endif
 +#ifdef __clang__
 +clang
  #endif
  EOF
 -	compiler_is_gnu=`${testcc:-cc} -E ${wrkdir}/$$.c 2>/dev/null | grep -c 
 indeed`
 -	rm -f ${wrkdir}/$$.c
 -
 +		compiler_is_gnu=`${testcc:-cc} -E ${wrkdir}/$$.c 2>/dev/null | grep -c 
 gcc`
 +		compiler_is_clang=`${testcc:-cc} -E ${wrkdir}/$$.c 2>/dev/null | grep -c 
 clang`
 +		rm -f ${wrkdir}/$$.c
 +	fi
 +	
  }
  get_abi()
  {
 @@ -776,6 +760,7 @@ OpenBSD)
  	need_awk=no
  	need_sed=no
  	set_opsys=no
 +	check_compiler=yes
  	machine_arch=`arch -s`
  	;;
  OSF1)



From: Keivan Motavalli <keivan@motavalli.me>
To: gnats-bugs@netbsd.org
Cc: Leonardo Taccari <leot@netbsd.org>,
 openbsd-pkg-people@NetBSD.org,
 pkgsrc-bugs@netbsd.org
Subject: Re: pkg/53170: get_compiler fails to detect clang if it's not named
 "gcc" in $PATH
Date: Sat, 5 May 2018 16:46:42 +0200

 Patch updated and tested in Mac OS 10.13, it also works on OpenBSD 6.2, =
 OpenBSD 6.3, FreeBSD and DragonflyBSD.

 I had left a useless check on testcc being set which excluded the =
 preprocessor check from being run on platforms where check_compiler and =
 cc were both set (Drawing, IRIX).

 Index: =
 archivers/libarchive/files/libarchive/archive_openssl_hmac_private.h
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 RCS file: =
 /cvsroot/pkgsrc/archivers/libarchive/files/libarchive/archive_openssl_hmac=
 _private.h,v
 retrieving revision 1.1.1.2
 diff -u -r1.1.1.2 archive_openssl_hmac_private.h
 --- archivers/libarchive/files/libarchive/archive_openssl_hmac_private.h	=
 1 Aug 2017 22:21:17 -0000	1.1.1.2
 +++ archivers/libarchive/files/libarchive/archive_openssl_hmac_private.h	=
 5 May 2018 14:37:30 -0000
 @@ -28,7 +28,7 @@
  #include <openssl/hmac.h>
  #include <openssl/opensslv.h>
 =20
 -#if OPENSSL_VERSION_NUMBER < 0x10100000L || =
 defined(LIBRESSL_VERSION_NUMBER)
 +#if OPENSSL_VERSION_NUMBER < 0x10100000L || =
 (defined(LIBRESSL_VERSION_NUMBER) && LIBRESSL_VERSION_NUMBER < =
 0x2070000fL)
  #include <stdlib.h> /* malloc, free */
  #include <string.h> /* memset */
  static inline HMAC_CTX *HMAC_CTX_new(void)
 Index: bootstrap/bootstrap
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 RCS file: /cvsroot/pkgsrc/bootstrap/bootstrap,v
 retrieving revision 1.248
 diff -u -r1.248 bootstrap
 --- bootstrap/bootstrap	7 Mar 2018 07:34:07 -0000	1.248
 +++ bootstrap/bootstrap	5 May 2018 14:37:33 -0000
 @@ -131,40 +131,20 @@
  get_compiler()
  {
  	testcc=3D"${CC}"
 -	# normally, we'd just use 'cc', but certain configure tools look
 -	# for gcc specifically, so we have to see if that comes first
 -	if [ -z "${testcc}" ]; then
 -		save_IFS=3D"${IFS}"
 -		IFS=3D':'
 -		for dir in ${PATH}; do
 -			test -z "$dir" && dir=3D.
 -			if [ -x "$dir/gcc" ]; then
 -				testcc=3D"$dir/gcc"
 -				break
 -			fi
 -		done
 -		IFS=3D"${save_IFS}"
 -	fi
 -
  	# Clang compiler pretends to be GCC, so we have to check that
  	cat >${wrkdir}/$$.c <<EOF
 -#ifdef __clang__
 -indeed
 -#endif
 -EOF
 -	compiler_is_clang=3D`${testcc:-cc} -E ${wrkdir}/$$.c 2>/dev/null =
 | grep -c indeed`
 -	rm -f ${wrkdir}/$$.c
 -
 -	cat >${wrkdir}/$$.c <<EOF
 -#ifdef __GNUC__
 -#ifndef __clang__
 -indeed
 +#define GCC_COMPILER (defined(__GNUC__) && !defined(__clang__))
 +#if GCC_COMPILER
 +gcc
  #endif
 +#ifdef __clang__
 +clang
  #endif
  EOF
 -	compiler_is_gnu=3D`${testcc:-cc} -E ${wrkdir}/$$.c 2>/dev/null | =
 grep -c indeed`
 +	compiler_is_gnu=3D`${testcc:-cc} -E ${wrkdir}/$$.c 2>/dev/null | =
 grep -c gcc`
 +	compiler_is_clang=3D`${testcc:-cc} -E ${wrkdir}/$$.c 2>/dev/null =
 | grep -c clang`
  	rm -f ${wrkdir}/$$.c
 -
 +=09
  }
  get_abi()
  {
 @@ -776,6 +756,7 @@
  	need_awk=3Dno
  	need_sed=3Dno
  	set_opsys=3Dno
 +	check_compiler=3Dyes
  	machine_arch=3D`arch -s`
  	;;
  OSF1)

From: Keivan Motavalli <keivan@motavalli.me>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: pkg/53170: get_compiler patch, text only
Date: Sat, 5 May 2018 16:54:01 +0200

 Index: =
 archivers/libarchive/files/libarchive/archive_openssl_hmac_private.h
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 RCS file: =
 /cvsroot/pkgsrc/archivers/libarchive/files/libarchive/archive_openssl_hmac=
 _private.h,v
 retrieving revision 1.1.1.2
 diff -u -r1.1.1.2 archive_openssl_hmac_private.h
 --- archivers/libarchive/files/libarchive/archive_openssl_hmac_private.h	=
 1 Aug 2017 22:21:17 -0000	1.1.1.2
 +++ archivers/libarchive/files/libarchive/archive_openssl_hmac_private.h	=
 5 May 2018 14:37:30 -0000
 @@ -28,7 +28,7 @@
  #include <openssl/hmac.h>
  #include <openssl/opensslv.h>
 =20
 -#if OPENSSL_VERSION_NUMBER < 0x10100000L || =
 defined(LIBRESSL_VERSION_NUMBER)
 +#if OPENSSL_VERSION_NUMBER < 0x10100000L || =
 (defined(LIBRESSL_VERSION_NUMBER) && LIBRESSL_VERSION_NUMBER < =
 0x2070000fL)
  #include <stdlib.h> /* malloc, free */
  #include <string.h> /* memset */
  static inline HMAC_CTX *HMAC_CTX_new(void)
 Index: bootstrap/bootstrap
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 RCS file: /cvsroot/pkgsrc/bootstrap/bootstrap,v
 retrieving revision 1.248
 diff -u -r1.248 bootstrap
 --- bootstrap/bootstrap	7 Mar 2018 07:34:07 -0000	1.248
 +++ bootstrap/bootstrap	5 May 2018 14:37:33 -0000
 @@ -131,40 +131,20 @@
  get_compiler()
  {
  	testcc=3D"${CC}"
 -	# normally, we'd just use 'cc', but certain configure tools look
 -	# for gcc specifically, so we have to see if that comes first
 -	if [ -z "${testcc}" ]; then
 -		save_IFS=3D"${IFS}"
 -		IFS=3D':'
 -		for dir in ${PATH}; do
 -			test -z "$dir" && dir=3D.
 -			if [ -x "$dir/gcc" ]; then
 -				testcc=3D"$dir/gcc"
 -				break
 -			fi
 -		done
 -		IFS=3D"${save_IFS}"
 -	fi
 -
  	# Clang compiler pretends to be GCC, so we have to check that
  	cat >${wrkdir}/$$.c <<EOF
 -#ifdef __clang__
 -indeed
 -#endif
 -EOF
 -	compiler_is_clang=3D`${testcc:-cc} -E ${wrkdir}/$$.c 2>/dev/null =
 | grep -c indeed`
 -	rm -f ${wrkdir}/$$.c
 -
 -	cat >${wrkdir}/$$.c <<EOF
 -#ifdef __GNUC__
 -#ifndef __clang__
 -indeed
 +#define GCC_COMPILER (defined(__GNUC__) && !defined(__clang__))
 +#if GCC_COMPILER
 +gcc
  #endif
 +#ifdef __clang__
 +clang
  #endif
  EOF
 -	compiler_is_gnu=3D`${testcc:-cc} -E ${wrkdir}/$$.c 2>/dev/null | =
 grep -c indeed`
 +	compiler_is_gnu=3D`${testcc:-cc} -E ${wrkdir}/$$.c 2>/dev/null | =
 grep -c gcc`
 +	compiler_is_clang=3D`${testcc:-cc} -E ${wrkdir}/$$.c 2>/dev/null =
 | grep -c clang`
  	rm -f ${wrkdir}/$$.c
 -
 +=09
  }
  get_abi()
  {
 @@ -776,6 +756,7 @@
  	need_awk=3Dno
  	need_sed=3Dno
  	set_opsys=3Dno
 +	check_compiler=3Dyes
  	machine_arch=3D`arch -s`
  	;;
  OSF1)

State-Changed-From-To: feedback->closed
State-Changed-By: schmonz@NetBSD.org
State-Changed-When: Wed, 12 Sep 2018 20:18:57 +0000
State-Changed-Why:
Committed a fix for OpenBSD to detect clang and prefer it over gcc.
Haven't encountered any problems with libarchive. If you're still having htem,
please submit a PR for that. Thanks for this PR!


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.43 2018/01/16 07:36:43 maya Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2017 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.