NetBSD Problem Report #56924

From stix@stix.id.au  Thu Jul 14 07:19:38 2022
Return-Path: <stix@stix.id.au>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 78B941A9239
	for <gnats-bugs@gnats.NetBSD.org>; Thu, 14 Jul 2022 07:19:38 +0000 (UTC)
Message-Id: <20220714071038.EB5B719E7D@stix.id.au>
Date: Thu, 14 Jul 2022 17:10:38 +1000 (AEST)
From: stix@stix.id.au
Reply-To: stix@stix.id.au
To: gnats-bugs@NetBSD.org
Subject: Many packages ignore package specific MAKE_JOBS overrides
X-Send-Pr-Version: 3.95

>Number:         56924
>Category:       pkg
>Synopsis:       Many packages ignore package specific MAKE_JOBS overrides
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    dholland
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Jul 14 07:20:00 +0000 2022
>Last-Modified:  Fri Jul 15 00:15:01 +0000 2022
>Originator:     Paul Ripke
>Release:        NetBSD 9.2
>Organization:
Paul Ripke
"Great minds discuss ideas, average minds discuss events, small minds
 discuss people."
-- Disputed: Often attributed to Eleanor Roosevelt. 1948.
>Environment:
System: NetBSD slave 9.2 NetBSD 9.2 (SLAVE) #9: Sat May 15 17:19:33 AEST 2021 stix@slave:/home/netbsd/netbsd-9/obj.amd64/home/netbsd/netbsd-9/src/sys/arch/amd64/compile/SLAVE amd64
Architecture: x86_64
Machine: amd64
pkgsrc-2022Q2
>Description:
See thread:
http://mail-index.netbsd.org/tech-pkg/2020/11/04/msg024021.html

Many packages reference MAKE_JOBS in their makefiles, eg:

ksh$ grep MAKE_JOBS x11/qt5-qtwebengine/Makefile
MAKE_ENV+=      NINJAJOBS="-j ${MAKE_JOBS:U1}"

However, if setting a manual override in mk.conf:

MAKE_JOBS.x11/qt5-qtwebengine=4

the override is ignored, and the default used. Substituting _MAKE_JOBS_N
for MAKE_JOBS works, and is used in many other packages, eg:

ksh$ grep _MAKE_JOBS_N devel/cmake/Makefile
CONFIGURE_ARGS+=        --parallel=${_MAKE_JOBS_N:U1}

but this (I believe) is supposed to be an internal variable.

>How-To-Repeat:
Build x11/qt5-qtwebengine and watch swap fill up?
>Fix:
All packages should use _MAKE_JOBS_N or a new appropriate variable.

See possible patch referenced in the mail thread above:
https://gist.github.com/jperkin/b390f0d922e5c22430ae7f2d8131a158

>Release-Note:

>Audit-Trail:
From: Paul Goyette <paul@whooppee.com>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: pkg/56924 Many packages ignore package specific MAKE_JOBS
 overrides
Date: Thu, 14 Jul 2022 06:02:10 -0700 (PDT)

 >> Fix:
 >  All packages should use _MAKE_JOBS_N or a new appropriate variable.

 Alternatively, each Makefile can be updated similarly to what I did
 recently for misc/libreoffice:

  	...
  	.if !empty(MAKE_JOBS.${PKGPATH})
  	CONFIGURE_ARGS+=        --with-parallelism=${MAKE_JOBS.${PKGPATH}}
  	.elif !empty(MAKE_JOBS)
  	CONFIGURE_ARGS+=        --with-parallelism=${MAKE_JOBS}
  	.else
  	CONFIGURE_ARGS+=        --with-parallelism=1
  	.endif
  	...

 +--------------------+--------------------------+----------------------+
 | Paul Goyette       | PGP Key fingerprint:     | E-mail addresses:    |
 | (Retired)          | FA29 0E3B 35AF E8AE 6651 | paul@whooppee.com    |
 | Software Developer | 0786 F758 55DE 53BA 7731 | pgoyette@netbsd.org  |
 | & Network Engineer |                          | pgoyette99@gmail.com |
 +--------------------+--------------------------+----------------------+

Responsible-Changed-From-To: pkg-manager->dholland
Responsible-Changed-By: dholland@NetBSD.org
Responsible-Changed-When: Fri, 15 Jul 2022 00:02:49 +0000
Responsible-Changed-Why:
my fault

(this doesn't mean that someone else shouldn't take care of it if they
see fit)


From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: pkg-manager@netbsd.org, gnats-admin@netbsd.org, pkgsrc-bugs@netbsd.org
Subject: Re: pkg/56924: Many packages ignore package specific MAKE_JOBS
 overrides
Date: Fri, 15 Jul 2022 00:07:45 +0000

 On Thu, Jul 14, 2022 at 07:20:00AM +0000, stix@stix.id.au wrote:
  > Many packages reference MAKE_JOBS in their makefiles, eg:
  > 
  > ksh$ grep MAKE_JOBS x11/qt5-qtwebengine/Makefile
  > MAKE_ENV+=      NINJAJOBS="-j ${MAKE_JOBS:U1}"

 For ninja and other build tools I would think that this ought to
 happen somewhere in the infrastructure, not in every package. But I
 suppose there isn't any such place currently.

  > However, if setting a manual override in mk.conf:
  > 
  > MAKE_JOBS.x11/qt5-qtwebengine=4
  > 
  > the override is ignored, and the default used. Substituting _MAKE_JOBS_N
  > for MAKE_JOBS works, and is used in many other packages, eg:
  > 
  > ksh$ grep _MAKE_JOBS_N devel/cmake/Makefile
  > CONFIGURE_ARGS+=        --parallel=${_MAKE_JOBS_N:U1}
  > 
  > but this (I believe) is supposed to be an internal variable.

 Yes, the right way is for the infrastructure to compute a variable
 PKG_MAKE_JOBS and then for packages to use that.

  > See possible patch referenced in the mail thread above:
  > https://gist.github.com/jperkin/b390f0d922e5c22430ae7f2d8131a158

 ...why didn't that get committed?

 -- 
 David A. Holland
 dholland@netbsd.org

From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: pkg/56924: Many packages ignore package specific MAKE_JOBS
 overrides
Date: Fri, 15 Jul 2022 00:12:28 +0000

 On Fri, Jul 15, 2022 at 12:07:45AM +0000, David Holland wrote:
  >  > See possible patch referenced in the mail thread above:
  >  > https://gist.github.com/jperkin/b390f0d922e5c22430ae7f2d8131a158
  > 
  > ...why didn't that get committed?

 ah, I see, it doesn't handle the difference between "1 job" and "don't
 parallelize" correctly.

 (also I see a couple places in the patch that change -j N to -jN in
 python stuff that will likely choke on it)

 -- 
 David A. Holland
 dholland@netbsd.org

>Unformatted:

Home	PR Database Search