NetBSD Problem Report #57238

From paul@whooppee.com  Sat Feb 18 20:16:36 2023
Return-Path: <paul@whooppee.com>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 757B01A923A
	for <gnats-bugs@gnats.NetBSD.org>; Sat, 18 Feb 2023 20:16:36 +0000 (UTC)
Message-Id: <20230218201631.80EDA999C5@speedy.whooppee.com>
Date: Sat, 18 Feb 2023 12:16:31 -0800 (PST)
From: Paul Goyette <paul@whooppee.com>
To: gnats-bugs@NetBSD.org
Subject: Would be nice to have tool-chain nbxz support multi-threaded

>Number:         57238
>Category:       toolchain
>Synopsis:       Would be nice to have tool-chain nbxz support multi-threaded
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    toolchain-manager
>State:          open
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Sat Feb 18 20:20:00 +0000 2023
>Originator:     Paul Goyette
>Release:        NetBSD 10.99.2
>Organization:
+--------------------+--------------------------+----------------------+
| Paul Goyette       | PGP Key fingerprint:     | E-mail addresses:    |
| (Retired)          | FA29 0E3B 35AF E8AE 6651 | paul@whooppee.com    |
| Software Developer | 0786 F758 55DE 53BA 7731 | pgoyette@netbsd.org  |
| & Network Engineer |                          | pgoyette99@gmail.com |
+--------------------+--------------------------+----------------------+
>Environment:


System: NetBSD speedy.whooppee.com 10.99.2 NetBSD 10.99.2 (SPEEDY 2023-01-30 15:47:31 UTC) #0: Tue Jan 31 02:33:14 UTC 2023 paul@speedy.whooppee.com:/build/netbsd-local/obj/amd64/sys/arch/amd64/compile/SPEEDY amd64
Architecture: x86_64
Machine: amd64
>Description:
	Using xz is great for improving compression ratios, but getting
	good compression is very compute-intensive.  It would be nice if
	the toolchain build for nbxz could detect availability of pthread
	and include multi-threaded options when possible.

	Note that USE_PIGZGZIP is inadequate for this task, as its
	compression ratios are significantly worse than xz.
>How-To-Repeat:
>Fix:
The following hack works for me on NetBSD/amd64 host, but is insufficient
as it simply assumes that pthread is available.  There's also a question
of whether librt might also be needed in some cases.


Index: distrib/sets/Makefile
===================================================================
RCS file: /cvsroot/src/distrib/sets/Makefile,v
retrieving revision 1.108
diff -u -p -r1.108 Makefile
--- distrib/sets/Makefile	21 Aug 2022 07:10:03 -0000	1.108
+++ distrib/sets/Makefile	25 Dec 2022 16:38:38 -0000
@@ -18,6 +18,7 @@ all:
 COMPRESS_PROGRAM=${"${USE_XZ_SETS:Uno}"!="no":?${TOOL_XZ}:${TOOL_GZIP}}
 TAR_SUFF=${"${USE_XZ_SETS:Uno}"!="no":?tar.xz:tgz}
 XZ_OPT=		-9
+XZ_OPT+= 	-T0	# XXX-PRG Use parallel compression to speed things up
 GZIP_FLAGS=	${GZIP_N_FLAG}

 SETSENV=	DESTDIR=${DESTDIR:Q} \
Index: external/public-domain/xz/bin/xz/Makefile
===================================================================
RCS file: /cvsroot/src/external/public-domain/xz/bin/xz/Makefile,v
retrieving revision 1.6
diff -u -p -r1.6 Makefile
--- external/public-domain/xz/bin/xz/Makefile	12 Apr 2021 02:54:08 -0000	1.6
+++ external/public-domain/xz/bin/xz/Makefile	25 Dec 2022 16:38:46 -0000
@@ -45,6 +45,7 @@ FILESNAME_${XZSRCDIR}/po/${lang}.gmo=	xz
 HOST_CPPFLAGS+=	${CPPFLAGS:N-Wp,-iremap,*}
 XZLIBDIR!=	cd ${NETBSDSRCDIR}/tools/xz-lib && ${PRINTOBJDIR}
 LDADD+=	-L${XZLIBDIR} -llzma
+LDADD+=	-lpthread		# XXX-PRG
 DPADD+=	${XZLIBDIR}/liblzma.a
 .else
 DPADD+=	${LIBLZMA} ${LIBINTL} ${LIBPTHREAD}
Index: external/public-domain/xz/lib/Makefile
===================================================================
RCS file: /cvsroot/src/external/public-domain/xz/lib/Makefile,v
retrieving revision 1.11
diff -u -p -r1.11 Makefile
--- external/public-domain/xz/lib/Makefile	22 Jul 2022 15:43:37 -0000	1.11
+++ external/public-domain/xz/lib/Makefile	25 Dec 2022 16:38:47 -0000
@@ -57,9 +57,9 @@ SRCS+=	common.c block_util.c easy_preset
 	index_decoder.c index_hash.c stream_buffer_decoder.c \
 	stream_decoder.c stream_flags_decoder.c vli_decoder.c

-.if !defined(HOSTLIB)
+# .if !defined(HOSTLIB)	XXX-PRG
 SRCS+=	 stream_encoder_mt.c
-.endif
+# .endif		XXX-PRG

 .PATH:	${XZSRCDIR}/src/liblzma/delta
 SRCS+=	delta_common.c delta_encoder.c delta_decoder.c
Index: tools/xz-include/Makefile
===================================================================
RCS file: /cvsroot/src/tools/xz-include/Makefile,v
retrieving revision 1.4
diff -u -p -r1.4 Makefile
--- tools/xz-include/Makefile	18 Sep 2021 01:47:11 -0000	1.4
+++ tools/xz-include/Makefile	25 Dec 2022 16:38:58 -0000
@@ -8,7 +8,9 @@
 #
 .include "Makefile.inc"

-CONFIGURE_ARGS+=	--enable-threads=no --disable-nls
+# XXX-PRG
+# CONFIGURE_ARGS+=	--enable-threads=no --disable-nls
+CONFIGURE_ARGS+=	--enable-threads=posix --disable-nls
 .if ${MAKEVERBOSE} == 0
 CONFIGURE_ARGS+=	--silent
 .endif

>Unformatted:
 compression if the host supports it.
 From: paul@whooppee.com
 Reply-To: paul@whooppee.com
 X-send-pr-version: 3.95

Home	PR Database Search