NetBSD Problem Report #47503

From www@NetBSD.org  Fri Jan 25 02:39:22 2013
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
	by www.NetBSD.org (Postfix) with ESMTP id 5524A63C07C
	for <gnats-bugs@gnats.NetBSD.org>; Fri, 25 Jan 2013 02:39:22 +0000 (UTC)
Message-Id: <20130125023921.1B74163C07C@www.NetBSD.org>
Date: Fri, 25 Jan 2013 02:39:21 +0000 (UTC)
From: jwbacon@tds.net
Reply-To: jwbacon@tds.net
To: gnats-bugs@NetBSD.org
Subject: Request automated addition of gfortran to base compiler set
X-Send-Pr-Version: www-1.0

>Number:         47503
>Category:       toolchain
>Synopsis:       Request automated addition of gfortran to base compiler set
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    toolchain-manager
>State:          open
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Fri Jan 25 02:40:00 +0000 2013
>Last-Modified:  Fri May 27 06:10:00 +0000 2022
>Originator:     Jason Bacon
>Release:        6.0
>Organization:
UW - Milwaukee
>Environment:
NetBSD netbsd6-pkgsrc.ceas.uwm.edu 6.0 NetBSD 6.0 (GENERIC) amd64

>Description:
This might be considered a followup to PR 41220, but I'd like to propose a higher priority and a different solution.

I've been developing pkgsrc packages to facilitate cross-platform scientific research.  There are many applications that I would like to package that require Fortran 90, and this has proven to be a major barrier to portable package development.

In order to fully realize the benefits of pkgsrc in my work, I need to package Fortran programs for NetBSD, CentOS, and OS X.

CentOS 6 has proven to be the easiest to work with because of the presence of a mature gfortran in the base compiler set.

Since NetBSD 6 is now using gcc 4.5 as a base compiler, I think it would make sense to provide the ability to add gfortran to the base toolchain.  I'm guessing that this wouldn't require much more than rebuilding gcc within /usr/src with

--enable-languages=c,c++,fortran

I'm not suggesting that gfortran necessarily be added to generic build, although I'd be fine with that as well if ISO size is not a concern.  It would suffice for my purposes to have the ability to easily add it.  

The ideal solution I envision would be an automated script that the pkgsrc base could run when it detects a Fortran 90 dependency.  Rather than add a gcc package, it would rebuild the base gcc and install /usr/bin/gfortran, leaving a clean, matching set of compilers for pkgsrc to use.  I'd be happy to help develop such a script if I could get a little guidance from the core developers on the best approach.

The pkgsrc gcc packages are not viable at this point, since they only work on NetBSD.  It would be hard to match the various base toolchain versions anyway, which leads to mixing toolchains in package builds.  Finally, it would be messy to use the base gfortran on some platforms and a gcc package on others.

For OS X, the solution I've settled on is to build a complete base gcc 4.6 or later to use as a base compiler instead of gcc 4.2.1 or clang.  I'm in the process of scripting the entire setup process (download and install gcc the old fashioned way, then bootstrap pkgsrc using the new gcc).  Eventually, I assume this approach could be adapted to other platforms that lack gfortran in the base toolchain.

>How-To-Repeat:

>Fix:

>Audit-Trail:
From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: toolchain/47503: Request automated addition of gfortran to base
 compiler set
Date: Fri, 25 Jan 2013 05:30:02 +0000

 On Fri, Jan 25, 2013 at 02:40:01AM +0000, jwbacon@tds.net wrote:
  > The pkgsrc gcc packages are not viable at this point, since they
  > only work on NetBSD.

 Not AFAIK...

  > It would be hard to match the various base toolchain versions
  > anyway, which leads to mixing toolchains in package builds.

 At least in theory this shouldn't matter.

  > Finally, it would be messy to use the base
  > gfortran on some platforms and a gcc package on others.

 This is the whole point of the compiler logic in pkgsrc, though.

 pkgsrc is at least in theory capable of installing a FORTRAN compiler
 of your choice, more or less on its own, if and only if the base
 FORTRAN compiler is nonexistent or unsuitable.

 If this isn't working (or doesn't know about gfortran vs. g77 vs. f2c
 or whatnot) it should be fixed.

 -- 
 David A. Holland
 dholland@netbsd.org

From: Jason Bacon <jwbacon@tds.net>
To: gnats-bugs@NetBSD.org
Cc: David Holland <dholland-bugs@netbsd.org>, 
 toolchain-manager@netbsd.org, gnats-admin@netbsd.org, 
 netbsd-bugs@netbsd.org
Subject: Re: toolchain/47503: Request automated addition of gfortran to base
 compiler set
Date: Fri, 25 Jan 2013 09:00:17 -0600

 On 01/24/2013 23:35, David Holland wrote:
 > The following reply was made to PR toolchain/47503; it has been noted by GNATS.
 >
 > From: David Holland<dholland-bugs@netbsd.org>
 > To: gnats-bugs@NetBSD.org
 > Cc:
 > Subject: Re: toolchain/47503: Request automated addition of gfortran to base
 >   compiler set
 > Date: Fri, 25 Jan 2013 05:30:02 +0000
 >
 >   On Fri, Jan 25, 2013 at 02:40:01AM +0000, jwbacon@tds.net wrote:
 >    >  The pkgsrc gcc packages are not viable at this point, since they
 >    >  only work on NetBSD.
 >
 >   Not AFAIK...
 I misspoke.  They may also work on DragonFly.

 Presently (or at least recently) none of the gcc packages build on RHEL 
 5, CentOS 6, or Darwin.  I put some effort into patching them and got 
 one to build on RHEL, but still experienced runtime errors.  I abandoned 
 the effort when we started moving toward CentOS 6, which has a mature 
 base compiler suite.


    >  It would be hard to match the various base toolchain versions
    >  anyway, which leads to mixing toolchains in package builds.

   At least in theory this shouldn't matter.

 Actually, the gcc project doesn't guarantee object code compatibility 
 across versions, so it does matter in theory.  In practice, it's 
 generally OK to mix two GNU compilers with the same major version, but I 
 have run into link problems in the past when mixing gcc versions.

 There's a chance that relying on gcc packages for Fortran support while 
 using the base C/C++ compilers will bite pkgsrc users down the road.  A 
 gfortran in the base would eliminate that risk.

 >
 >    >  Finally, it would be messy to use the base
 >    >  gfortran on some platforms and a gcc package on others.
 >
 >   This is the whole point of the compiler logic in pkgsrc, though.
 >
 >   pkgsrc is at least in theory capable of installing a FORTRAN compiler
 >   of your choice, more or less on its own, if and only if the base
 >   FORTRAN compiler is nonexistent or unsuitable.
 >
 >   If this isn't working (or doesn't know about gfortran vs. g77 vs. f2c
 >   or whatnot) it should be fixed.
 I would agree that it *should* be fixed, but it's an awfully squirrelly 
 problem and I think it would be much easier and much safer to build 
 gfortran as part of the base.

 I've been wrangling with these issues for over a year and a complete 
 base compiler suite seems like the only clean, safe solution.

 Thanks,

      -J

 -- 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Jason W. Bacon
 jwbacon@tds.net
 http://personalpages.tds.net/~jwbacon
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


From: Mark Davies <mark@ecs.vuw.ac.nz>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: toolchain/47503: Request automated addition of gfortran to base compiler set
Date: Tue, 5 Feb 2013 10:15:29 +1300

 On Saturday 26 January 2013 04:05:03 you wrote:
 >  I would agree that it *should* be fixed, but it's an awfully squirrelly
 >  problem and I think it would be much easier and much safer to build
 >  gfortran as part of the base.
 > 
 >  I've been wrangling with these issues for over a year and a complete
 >  base compiler suite seems like the only clean, safe solution.

 By default pkgsrc uses g95 as its fortran in the situation that gcc is the 
 base compiler but no fortran is provided.  Does this not work for you?  
 g95 builds and works fine for me on ArchLinux as well as NetBSD.

 cheers
 mark

From: Jason Bacon <jwbacon@tds.net>
To: gnats-bugs@NetBSD.org
Cc: Mark Davies <mark@ecs.vuw.ac.nz>, toolchain-manager@netbsd.org, 
 gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
 Aleksej Saushev <asau@inbox.ru>
Subject: Re: toolchain/47503: Request automated addition of gfortran to base
 compiler set
Date: Mon, 04 Feb 2013 16:54:01 -0600

 On 02/04/2013 15:20, Mark Davies wrote:
 > The following reply was made to PR toolchain/47503; it has been noted by GNATS.
 >
 > From: Mark Davies<mark@ecs.vuw.ac.nz>
 > To: gnats-bugs@netbsd.org
 > Cc:
 > Subject: Re: toolchain/47503: Request automated addition of gfortran to base compiler set
 > Date: Tue, 5 Feb 2013 10:15:29 +1300
 >
 >   On Saturday 26 January 2013 04:05:03 you wrote:
 >   >   I would agree that it *should* be fixed, but it's an awfully squirrelly
 >   >   problem and I think it would be much easier and much safer to build
 >   >   gfortran as part of the base.
 >   >
 >   >   I've been wrangling with these issues for over a year and a complete
 >   >   base compiler suite seems like the only clean, safe solution.
 >
 >   By default pkgsrc uses g95 as its fortran in the situation that gcc is the
 >   base compiler but no fortran is provided.  Does this not work for you?
 >   g95 builds and works fine for me on ArchLinux as well as NetBSD.

 Hello Mark,

 Thanks for following up.

 Let me fill you in on where I've been and where I hope to go with this.  
 Sorry for the novel, but I want to make sure my motivations are clear.

 I support high performance computing at UW - Milwaukee, including a 
 fairly large RHEL cluster, as well as some open source workstations and 
 Macs.

 HPC clusters are dominated by CentOS and RHEL, due to the need to run 
 commercial software like Abaqus and ANSYS.  They have very poor support 
 for running open source applications, though.  The RPMs in the Yum 
 repository tend to be outdated due to Redhat's need to maintain binary 
 compatibility for older closed-source applications.  The Yum repo is 
 rather small anyway, and doesn't contain much to support scientific 
 computing.

 At most HPC sites, they do cave-man installations of most of the open 
 source software and use "modules" (http://modules.sourceforge.net/) to 
 control PATH and other environment variables in order to enable 
 individual applications.

 I see a huge potential for pkgsrc to improve this situation.  It has the 
 potential to:

 a) Save countless man-hours that are currently being wasted on manual 
 installations.
 b) Enable the use of software that might not otherwise be explored 
 because it's too difficult for a physicist, biologist, or engineer to 
 install.
 c) Allow researchers to easily have the same software installed on their 
 Mac, open source desktop, and the cluster.  This is going to be a key 
 selling point for pkgsrc in research computing.

 However, there are some basic foundational issues like Fortran support 
 that need to be solidified across platforms first.

 G95 is based on a very old (actually experimental) version of the 
 gfortran compiler.  Fortran support in GCC didn't really mature until 
 around 4.4.  Earlier versions are generally untrusted in the HPC 
 community, and a lot of code is still being built with commercial 
 compilers.  Some recent versions of software won't even compile under G95.

 Also, for HPC, a good optimizer is indispensable, since it can save 
 thousands of CPU-hours on a busy cluster.  GCC made huge improvements to 
 the optimizer between 4.2 and 4.5, and most HPC sites are now running 
 4.4 or later.

 If we're going to use a package to support Fortran, it would have to be 
 a recent GCC package, not G95.

 We could get by with a GCC package on NetBSD, but since GCC 4.5 is now 
 being used as the base compiler, and pkgsrc already contains the logic 
 to use a base gfortran if it's present, I think it would make sense to 
 have gfortran in the base (at least as an option).  This would ensure 
 object code compatibility with code compiled by the base gcc and g++ as 
 well.  I think it would take fewer man-hours in the long run to put it 
 in the base, and the end result would be cleaner and safer.  I've used 
 GCC ports on FreeBSD for a long time and it mostly works fine, but I 
 have run into a few issues due to mixing gcc versions.

 RHEL and CentOS already include gfortran in the base, so a GCC package 
 is unnecessary here.

 OS X presents a different problem.  There, the base compiler is either 
 GCC 4.2 or earlier, or clang.  Even if we had working GCC packages on OS 
 X, using it would guarantee that we're mixing tool chains.

 At best, we'd be using an old GCC on Snow Leopard and earlier for most 
 of the packages, which doesn't optimize nearly as well as later GCC.

 Hence, a gfortran package that works on OS X isn't of much interest, and 
 we don't need one at all on CentOS and RHEL.

 I'm developing a solution for OS X that installs the GCC 4.6 collection 
 outside of pkgsrc, from which pkgsrc can be bootstrapped.  That way, 
 everything in pkgsrc (C, C++, and Fortran) will be built with a modern 
 compiler from the same build, and will benefit from a good optimizer, 
 which is what the HPC community requires.

 We're pretty close to having a solid foundation for scientific packages 
 on NetBSD, CentOS 6, and OS X.  I have a lot of packages in wip already, 
 but we need to clean up a few more things before we start committing 
 most of them.

 Once we get a few HPC packages working well (including some with 
 OpenMPI), I plan to start promoting pkgsrc to the research computing 
 community via our website and presentations at HPC conferences.

  From there, I think things could really take off.  We'd be able to 
 recruit more Linux and Mac users to become packagers, as well as promote 
 NetBSD as a research computing platform.

 Thanks again.

 -- 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Jason W. Bacon
 jwbacon@tds.net
 http://personalpages.tds.net/~jwbacon
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


From: "Valeriy E. Ushakov" <uwe@stderr.spb.ru>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: toolchain/47503: Request automated addition of gfortran to base compiler set
Date: Tue, 5 Feb 2013 17:18:38 +0400

 On Mon, Feb 04, 2013 at 22:55:08 +0000, Jason Bacon wrote:

 >  RHEL and CentOS already include gfortran in the base, so a GCC package 
 >  is unnecessary here.

 Does a typical Linux distribution even have a "base" in the same sense
 we speak about base system in BSDs?  My impression from my limited
 exposure is that in NetBSD terms their "base" gcc is what we would
 call a gcc package.


 >  We could get by with a GCC package on NetBSD, but since GCC 4.5 is now 
 >  being used as the base compiler, and pkgsrc already contains the logic 
 >  to use a base gfortran if it's present, I think it would make sense to 
 >  have gfortran in the base (at least as an option).  This would ensure 
 >  object code compatibility with code compiled by the base gcc and g++ as 
 >  well. 

 Are there some known ABI issues between recent 4.x series, or is it
 more in the bad vibes area? (I use it as a neutral technical (sic!:)
 term here)


 >  I think it would take fewer man-hours in the long run to put it 
 >  in the base, and the end result would be cleaner and safer. 

 ... for as long as gfortran in gcc 4.5 is considered ok, until some
 horrible bug that triggers only under rare conditions makes it
 unsuitable because you can't trust it any more.

 ... or when gcc 4.N+1 has optimizer improvements that cut you batch
 wall time by days and you absolutely want to use 4.N+1 instead of that
 old base 4.5

 ... or until we decide to upgrade the base compiler to, say, gcc 4.7
 and it turns out newer gfortran have issues.

 I'm sure pkgsrc folks can continute the list.

 From the base system persepective I'd say it would take fewer
 man-hours in the long run to make sure gfortran packages work smoothly.

 . de facto, optional code tends to bit-root

 . de jure, once in base, optional code is, effectively, mandatory

 . it's "mandatory optional" on *all* ports, including those no-one
   would ever going to use for HPC (vax heyday is over; jornada 680
   cluster?)

 . it will have to be in the standard builds, so limited resources of
   netbsd autobuid cluster will be spent on building it (for all
   ports).

 . we will have to solve *exactly the same* ABI compatibility issues
   (if any) during base gcc upgrade as pkgsrc needs to solve for a gcc
   pacakge.

 So the last item is the same for base and pkgsrc and all previous
 items are extra work for base.

 It's most likely that my perception of this problem is affected by my
 wearing base-hacker hat (mine might have -5 penalty to wisdom and
 cursed), so, please, excuse me if this mail might come across as a bit
 antagonistic - it's not inteded that way.

 -uwe

From: Jason bacon <jwbacon@tds.net>
To: gnats-bugs@NetBSD.org
Cc: "Valeriy E. Ushakov" <uwe@stderr.spb.ru>, 
 toolchain-manager@netbsd.org, gnats-admin@netbsd.org, 
 netbsd-bugs@netbsd.org, Aleksej Saushev <asau@inbox.ru>
Subject: Re: toolchain/47503: Request automated addition of gfortran to base
 compiler set
Date: Tue, 05 Feb 2013 12:42:17 -0600

 On 2/5/13 7:20 AM, Valeriy E. Ushakov wrote:
 > The following reply was made to PR toolchain/47503; it has been noted by GNATS.
 >
 > From: "Valeriy E. Ushakov"<uwe@stderr.spb.ru>
 > To: gnats-bugs@NetBSD.org
 > Cc:
 > Subject: Re: toolchain/47503: Request automated addition of gfortran to base compiler set
 > Date: Tue, 5 Feb 2013 17:18:38 +0400
 >
 >   On Mon, Feb 04, 2013 at 22:55:08 +0000, Jason Bacon wrote:
 >
 >   >   RHEL and CentOS already include gfortran in the base, so a GCC package
 >   >   is unnecessary here.
 >
 >   Does a typical Linux distribution even have a "base" in the same sense
 >   we speak about base system in BSDs?  My impression from my limited
 >   exposure is that in NetBSD terms their "base" gcc is what we would
 >   call a gcc package.
 Not exactly.  RHEL/CentOS don't necessarily have any development tools 
 out of the box,
 but a Yum package in the 'base' category installs them into /usr/bin.  
 This would be the first
 and usually only compiler suite on the system, so it's akin to the BSD 
 base compilers.

 The development tools "group" (like a metapackage) is what most people 
 use.  It installs all
 the compilers, flex, bison, subversion, etc.

 https://support.eapps.com/index.php?/Knowledgebase/Article/View/438/55/user-guide---installing-the-centos-development-tools-gcc-flex-etc
 >
 >
 >   >   We could get by with a GCC package on NetBSD, but since GCC 4.5 is now
 >   >   being used as the base compiler, and pkgsrc already contains the logic
 >   >   to use a base gfortran if it's present, I think it would make sense to
 >   >   have gfortran in the base (at least as an option).  This would ensure
 >   >   object code compatibility with code compiled by the base gcc and g++ as
 >   >   well.
 >
 >   Are there some known ABI issues between recent 4.x series, or is it
 >   more in the bad vibes area? (I use it as a neutral technical (sic!:)
 >   term here)
 I'm not aware of any issues between recent 4.x compilers, but that does 
 not make me
 confident that they don't exist.  I'll be more worried with GCC 5.0 
 becomes the preferred
 package.
 >
 >
 >   >   I think it would take fewer man-hours in the long run to put it
 >   >   in the base, and the end result would be cleaner and safer.
 >
 >   ... for as long as gfortran in gcc 4.5 is considered ok, until some
 >   horrible bug that triggers only under rare conditions makes it
 >   unsuitable because you can't trust it any more.
 >
 >   ... or when gcc 4.N+1 has optimizer improvements that cut you batch
 >   wall time by days and you absolutely want to use 4.N+1 instead of that
 >   old base 4.5
 >
 >   ... or until we decide to upgrade the base compiler to, say, gcc 4.7
 >   and it turns out newer gfortran have issues.
 >
 >   I'm sure pkgsrc folks can continute the list.
 >
 >   From the base system persepective I'd say it would take fewer
 >   man-hours in the long run to make sure gfortran packages work smoothly.
 >
 >   . de facto, optional code tends to bit-root
 >
 >   . de jure, once in base, optional code is, effectively, mandatory
 >
 >   . it's "mandatory optional" on *all* ports, including those no-one
 >     would ever going to use for HPC (vax heyday is over; jornada 680
 >     cluster?)
 >
 >   . it will have to be in the standard builds, so limited resources of
 >     netbsd autobuid cluster will be spent on building it (for all
 >     ports).
 This is why I initially suggested only providing instructions or a 
 script for rebuilding the
 base compiler suite with gfortran enabled, rather than including it in 
 the generic installation.
 Based on my experience with manual GCC builds, once you get the build 
 working, adding
 Fortran is unlikely to cause many issues.
 >
 >   . we will have to solve *exactly the same* ABI compatibility issues
 >     (if any) during base gcc upgrade as pkgsrc needs to solve for a gcc
 >     pacakge.
 The ABI issues I'm worried about stem from the fact that the GCC package 
 chosen
 as a prerequisite might be a different version than the base compilers 
 and might
 be built with different options.  Rebuilding the base suite guarantees 
 the same
 version, --build, --host, --target, etc.
 >
 >   So the last item is the same for base and pkgsrc and all previous
 >   items are extra work for base.
 >
 >   It's most likely that my perception of this problem is affected by my
 >   wearing base-hacker hat (mine might have -5 penalty to wisdom and
 >   cursed), so, please, excuse me if this mail might come across as a bit
 >   antagonistic - it's not inteded that way.
 >
 >   -uwe
 >
 No worries.  I see this as collaborative discourse, not a flame war.  
 The least
 that can come of it is we all end up with a better understanding of the 
 issues.

 I won't be offended if the developers doesn't agree with my suggestions.  I
 just want to have a discussion to raise awareness of the issues, and
 offer some ideas for solving them.

 Regards,

 -- 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Jason W. Bacon
 jwbacon@tds.net
 http://personalpages.tds.net/~jwbacon
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: toolchain/47503: Request automated addition of gfortran to base
 compiler set
Date: Thu, 26 May 2022 03:57:24 +0000

 On Fri, Jan 25, 2013 at 02:40:01AM +0000, jwbacon@tds.net wrote:
  > I think it would make sense to provide the ability to add gfortran
  > to the base toolchain.

 I think everything else in this PR is out of date (pkgsrc fortran now
 works more or less properly and provides gfortran)... but yes, we'd
 like to have fortran in base, since the hope in the 90s that fortran
 would go away eventually didn't pan out.

 Unfortunately, it's not so simple to do...

 -- 
 David A. Holland
 dholland@netbsd.org

From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: toolchain/47503: Request automated addition of gfortran to base
 compiler set
Date: Thu, 26 May 2022 08:41:25 +0200

 I don't agree that we want fortran in base - it's in pkgsrc and
 working well enough there.
  Thomas


From: matthew green <mrg@eterna.com.au>
To: gnats-bugs@netbsd.org
Cc: toolchain-manager@netbsd.org, gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org, jwbacon@tds.net
Subject: re: toolchain/47503: Request automated addition of gfortran to base compiler set
Date: Fri, 27 May 2022 16:05:00 +1000

 >  I don't agree that we want fortran in base - it's in pkgsrc and
 >  working well enough there.

 i've always considered this a regression - as the person
 who upgraded netbsd gcc to "no g77" and then to "with
 gfortran"...

 it's merely a lack of time spent on it from my (or anyone
 else willing to do the work :-) side.


 .mrg.

Home	PR Database Search