NetBSD Problem Report #54844
From paul@whooppee.com Wed Jan 8 13:39:24 2020
Return-Path: <paul@whooppee.com>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 9BC4B7A154
for <gnats-bugs@gnats.NetBSD.org>; Wed, 8 Jan 2020 13:39:24 +0000 (UTC)
Message-Id: <20200108133852.7434A30F2C3@speedy.whooppee.com>
Date: Wed, 8 Jan 2020 05:38:52 -0800 (PST)
From: paul@whooppee.com
Reply-To: paul@whooppee.com
To: gnats-bugs@NetBSD.org
Subject: ``build.sh install-image'' fails with -V USE_PIGZGZIP=yes
X-Send-Pr-Version: 3.95
>Number: 54844
>Category: install
>Synopsis: ``build.sh install-image'' fails with -V USE_PIGZGZIP=yes
>Confidential: no
>Severity: non-critical
>Priority: medium
>Responsible: install-manager
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Wed Jan 08 13:40:00 +0000 2020
>Closed-Date:
>Last-Modified: Sat Feb 18 17:25:01 +0000 2023
>Originator: Paul Goyette
>Release: NetBSD 9.99.32
>Organization:
+--------------------+--------------------------+-----------------------+
| Paul Goyette | PGP Key fingerprint: | E-mail addresses: |
| (Retired) | FA29 0E3B 35AF E8AE 6651 | paul@whooppee.com |
| Software Developer | 0786 F758 55DE 53BA 7731 | pgoyette@netbsd.org |
+--------------------+--------------------------+-----------------------+
>Environment:
System: NetBSD speedy.whooppee.com 9.99.32 NetBSD 9.99.32 (SPEEDY 2020-01-01 13:34:35 UTC) #0: Wed Jan 1 16:22:29 UTC 2020 paul@speedy.whooppee.com:/build/netbsd-local/obj/amd64/sys/arch/amd64/compile/SPEEDY amd64
Architecture: x86_64
Machine: amd64
>Description:
``build.sh install-image'' fails with -V USE_PIGZGZIP=yes
===> build.sh command: ./build.sh -T /build/netbsd-local/tools/x86_64/amd64 -D /build/netbsd-local/dest/amd64 -O /build/netbsd-local/obj/amd64 -R /build/netbsd-local/release -V RELEASEMACHINEDIR=amd64 -V MKDEBUG=yes -V MKKDEBUG=yes -V USE_PIGZGZIP=yes -U -x -N0 -m amd64 -j10 install-image
===> build.sh started: Wed Jan 8 01:13:17 UTC 2020
===> NetBSD version: 9.99.34
===> MACHINE: amd64
===> MACHINE_ARCH: x86_64
===> Build platform: NetBSD 9.99.32 amd64
===> HOST_SH: /bin/sh
===> MAKECONF file: /etc/mk.conf
===> TOOLDIR path: /build/netbsd-local/tools/x86_64/amd64
===> DESTDIR path: /build/netbsd-local/dest/amd64
===> RELEASEDIR path: /build/netbsd-local/release
===> Updated makewrapper: /build/netbsd-local/tools/x86_64/amd64/bin/nbmake-amd64
install-image ===> etc
install_image ===> /build/netbsd-local/src_ro/distrib (with: INSTIMG_RELEASEDIR=/build/netbsd-local/release/images)
install_image ===> /build/netbsd-local/src_ro/distrib/notes
install_image ===> /build/netbsd-local/src_ro/distrib/utils
install_image ===> /build/netbsd-local/src_ro/distrib/amd64
install_image ===> /build/netbsd-local/src_ro/distrib/amd64/installimage
creating MBR labels...
1+0 records in
1+0 records out
512 bytes transferred in 0.001 secs (512000 bytes/sec)
Making partition 0 active.
2048+0 records in
2048+0 records out
1048576 bytes transferred in 0.031 secs (33825032 bytes/sec)
Preparing /etc/fstab ...
Extracting kern-GENERIC.tgz ...
Extracting modules.tgz ...
Extracting base.tgz ...
Extracting etc.tgz ...
Copying secondary boot...
Setting rc_configured=YES in /etc/rc.conf ...
Copying extra dirs...
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/.
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/kern-GENERIC_KASLR.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/kern-XEN3_DOM0.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/kern-GENERIC.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/kern-XEN3_DOMU.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/tests.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/xbase.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/rescue.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/games.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/xdebug.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/MD5
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/SHA512
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/comp.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/etc.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/modules.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/misc.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/man.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/xserver.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/debug.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/xfont.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/text.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/xetc.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/base.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/sets/xcomp.tgz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/kernel
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/kernel/netbsd-XEN3_DOMU.gz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/kernel/netbsd-XEN3_DOM0.gz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/kernel/netbsd-GENERIC.gz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/kernel/netbsd-GENERIC_KASLR.gz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/kernel/netbsd-INSTALL_XEN3_DOMU.gz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/kernel/netbsd-INSTALL.gz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/kernel/MD5
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/kernel/SHA512
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/kernel/netbsd-INSTALL.symbols.gz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./binary/kernel/netbsd-GENERIC.symbols.gz
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./installation
./installation/cdrom >> <empty string>
./installation/cdrom/boot.iso >> <empty string>
./installation/cdrom/MD5 >> <empty string>
./installation/cdrom/SHA512 >> <empty string>
./installation/cdrom/boot-com.iso >> <empty string>
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./installation/misc
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./installation/misc/MD5
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./installation/misc/SHA512
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./installation/misc/pxeboot_ia32.bin
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./installation/miniroot
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./installation/miniroot/MD5
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./installation/miniroot/SHA512
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./installation/miniroot/miniroot.kmod
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./installation/miniroot/cgdroot.kmod
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./INSTALL.html
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./INSTALL.ps
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./INSTALL.txt
/build/netbsd-local/obj/amd64/distrib/amd64/installimage/work/amd64/./INSTALL.more
Copying extra files...
/build/netbsd-local/tools/x86_64/amd64/bin/x86_64--netbsd-install -c -m 0644 /build/netbsd-local/src_ro/distrib/amd64/installimage/etc.ttys work/etc/ttys
/build/netbsd-local/tools/x86_64/amd64/bin/x86_64--netbsd-install -c -m 0644 /build/netbsd-local/src_ro/distrib/amd64/installimage/etc.rc work/etc/rc
/build/netbsd-local/tools/x86_64/amd64/bin/x86_64--netbsd-install -c -m 0644 /build/netbsd-local/src_ro/distrib/amd64/installimage/install.sh work/.
/build/netbsd-local/tools/x86_64/amd64/bin/x86_64--netbsd-install -c -m 0644 /build/netbsd-local/obj/amd64/distrib/amd64/installimage/boot.cfg work/.
/build/netbsd-local/tools/x86_64/amd64/bin/x86_64--netbsd-install -c -m 0644 /build/netbsd-local/obj/amd64/usr.sbin/sysinst/arch/amd64/sysinstmsgs.de work/.
/build/netbsd-local/tools/x86_64/amd64/bin/x86_64--netbsd-install -c -m 0644 /build/netbsd-local/obj/amd64/usr.sbin/sysinst/arch/amd64/sysinstmsgs.es work/.
/build/netbsd-local/tools/x86_64/amd64/bin/x86_64--netbsd-install -c -m 0644 /build/netbsd-local/obj/amd64/usr.sbin/sysinst/arch/amd64/sysinstmsgs.fr work/.
/build/netbsd-local/tools/x86_64/amd64/bin/x86_64--netbsd-install -c -m 0644 /build/netbsd-local/obj/amd64/usr.sbin/sysinst/arch/amd64/sysinstmsgs.pl work/.
/build/netbsd-local/tools/x86_64/amd64/bin/x86_64--netbsd-install -c -m 0644 /build/netbsd-local/obj/amd64/usr.sbin/sysinst/arch/amd64/sysinst work/.
Preparing spec files for makefs...
Creating rootfs...
nbmakefs: `work' size of 1994883072 is larger than the maxsize of 1519386624.
*** Failed target: imgroot.fs
*** Failed command: /build/netbsd-local/tools/x86_64/amd64/bin/nbmakefs -M 1519386624 -m 1519386624 -B 1234 -F work.spec -N work/etc -o bsize=16384,fsize=2048,density=8192 work.rootfs work
*** Error code 1
Stop.
nbmake[4]: stopped in /build/netbsd-local/src_ro/distrib/amd64/installimage
--- install_image ---
*** [install_image] Error code 1
nbmake[3]: stopped in /build/netbsd-local/src_ro/distrib/amd64
1 error
nbmake[3]: stopped in /build/netbsd-local/src_ro/distrib/amd64
*** Failed target: install_image-amd64
*** Failed command: _makedirtarget() { dir="$1"; shift; target="$1"; shift; case "${dir}" in /*) this="${dir}/"; real="${dir}" ;; .) this="/build/netbsd-local/src_ro/distrib/"; real="/build/netbsd-local/src_ro/distrib" ;; *) this="/build/netbsd-local/src_ro/distrib/${dir}/"; real="/build/netbsd-local/src_ro/distrib/${dir}" ;; esac; show=${this:-.}; echo "${target} ===> ${show%/}${1:+ (with: $@)}"; cd "${real}" && /build/netbsd-local/tools/x86_64/amd64/bin/nbmake _THISDIR_="${this}" "$@" ${target}; }; _makedirtarget amd64 install_image
*** Error code 2
Stop.
nbmake[2]: stopped in /build/netbsd-local/src_ro/distrib
--- install-image ---
*** [install-image] Error code 1
nbmake[1]: stopped in /build/netbsd-local/src_ro/etc
1 error
nbmake[1]: stopped in /build/netbsd-local/src_ro/etc
--- install-image ---
*** [install-image] Error code 2
nbmake: stopped in /build/netbsd-local/src_ro
nbmake: stopped in /build/netbsd-local/src_ro
>How-To-Repeat:
See above
>Fix:
Unknown
>Release-Note:
>Audit-Trail:
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: install/54844: ``build.sh install-image'' fails with -V
USE_PIGZGZIP=yes
Date: Wed, 8 Jan 2020 15:05:37 +0100
Not sure if we should support non standard options to this level, but
you could try something like below.
Martin
Index: Makefile
===================================================================
RCS file: /cvsroot/src/distrib/amd64/installimage/Makefile,v
retrieving revision 1.10
diff -u -r1.10 Makefile
--- Makefile 15 Dec 2018 18:03:16 -0000 1.10
+++ Makefile 8 Jan 2020 14:04:06 -0000
@@ -4,7 +4,11 @@
INSTIMGBASE= NetBSD-${DISTRIBVER}-amd64-install # gives ${IMGBASE}.img
+.if ${USE_XZ_SETS:Uno} == "yes"
INSTIMAGEMB?= 1450 # for all installation binaries
+.else
+INSTIMAGEMB?= 1950 # for all installation binaries
+.endif
PRIMARY_BOOT= bootxx_ffsv1
SECONDARY_BOOT= boot
From: Paul Goyette <paul@whooppee.com>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: install/54844: ``build.sh install-image'' fails with -V
USE_PIGZGZIP=yes (fwd)
Date: Wed, 8 Jan 2020 06:59:18 -0800 (PST)
+--------------------+--------------------------+-----------------------+
| Paul Goyette | PGP Key fingerprint: | E-mail addresses: |
| (Retired) | FA29 0E3B 35AF E8AE 6651 | paul@whooppee.com |
| Software Developer | 0786 F758 55DE 53BA 7731 | pgoyette@netbsd.org |
+--------------------+--------------------------+-----------------------+
---------- Forwarded message ----------
Date: Wed, 8 Jan 2020 06:58:30 -0800 (PST)
From: Paul Goyette <paul@whooppee.com>
To: Martin Husemann <martin@duskware.de>
Subject: Re: install/54844: ``build.sh install-image'' fails with -V
USE_PIGZGZIP=yes
I'm not so sure this is just a size issue. At one point I actually
looked at the contents of the work directory, and found _another_
work directory a couple levels down. The lower-level work directory
seemed to be the correct size.
Perhaps there are some conflicting command-line args that work for
xz but not for pigz? (Guesing: Perhaps related to sym-links?)
On Wed, 8 Jan 2020, Martin Husemann wrote:
> The following reply was made to PR install/54844; it has been noted by GNATS.
>
> From: Martin Husemann <martin@duskware.de>
> To: gnats-bugs@netbsd.org
> Cc:
> Subject: Re: install/54844: ``build.sh install-image'' fails with -V
> USE_PIGZGZIP=yes
> Date: Wed, 8 Jan 2020 15:05:37 +0100
>
> Not sure if we should support non standard options to this level, but
> you could try something like below.
>
> Martin
>
> Index: Makefile
> ===================================================================
> RCS file: /cvsroot/src/distrib/amd64/installimage/Makefile,v
> retrieving revision 1.10
> diff -u -r1.10 Makefile
> --- Makefile 15 Dec 2018 18:03:16 -0000 1.10
> +++ Makefile 8 Jan 2020 14:04:06 -0000
> @@ -4,7 +4,11 @@
>
> INSTIMGBASE= NetBSD-${DISTRIBVER}-amd64-install # gives ${IMGBASE}.img
>
> +.if ${USE_XZ_SETS:Uno} == "yes"
> INSTIMAGEMB?= 1450 # for all installation binaries
> +.else
> +INSTIMAGEMB?= 1950 # for all installation binaries
> +.endif
>
> PRIMARY_BOOT= bootxx_ffsv1
> SECONDARY_BOOT= boot
>
>
> !DSPAM:5e15e2f6268761828113096!
>
>
+--------------------+--------------------------+-----------------------+
| Paul Goyette | PGP Key fingerprint: | E-mail addresses: |
| (Retired) | FA29 0E3B 35AF E8AE 6651 | paul@whooppee.com |
| Software Developer | 0786 F758 55DE 53BA 7731 | pgoyette@netbsd.org |
+--------------------+--------------------------+-----------------------+
From: Paul Goyette <paul@whooppee.com>
To: gnats-bugs@netbsd.org
Cc: martin@duskware.de
Subject: Re: install/54844: ``build.sh install-image'' fails with -V
USE_PIGZGZIP=yes
Date: Wed, 8 Jan 2020 08:42:50 -0800 (PST)
On Wed, 8 Jan 2020, Paul Goyette wrote:
> I'm not so sure this is just a size issue. At one point I actually
> looked at the contents of the work directory, and found _another_
> work directory a couple levels down. The lower-level work directory
> seemed to be the correct size.
>
> Perhaps there are some conflicting command-line args that work for
> xz but not for pigz? (Guesing: Perhaps related to sym-links?)
Well, I just checked again, and I don't see the "nested" work directory.
So perhapas setting the size as you suggest is appropriate.
+--------------------+--------------------------+-----------------------+
| Paul Goyette | PGP Key fingerprint: | E-mail addresses: |
| (Retired) | FA29 0E3B 35AF E8AE 6651 | paul@whooppee.com |
| Software Developer | 0786 F758 55DE 53BA 7731 | pgoyette@netbsd.org |
+--------------------+--------------------------+-----------------------+
From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@netbsd.org, install-manager@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, paul@whooppee.com
Cc:
Subject: Re: install/54844: ``build.sh install-image'' fails with -V USE_PIGZGZIP=yes
Date: Wed, 8 Jan 2020 12:37:03 -0500
On Jan 8, 4:45pm, paul@whooppee.com (Paul Goyette) wrote:
-- Subject: Re: install/54844: ``build.sh install-image'' fails with -V USE_P
| The following reply was made to PR install/54844; it has been noted by GNATS.
|
| From: Paul Goyette <paul@whooppee.com>
| To: gnats-bugs@netbsd.org
| Cc: martin@duskware.de
| Subject: Re: install/54844: ``build.sh install-image'' fails with -V
| USE_PIGZGZIP=yes
| Date: Wed, 8 Jan 2020 08:42:50 -0800 (PST)
|
| On Wed, 8 Jan 2020, Paul Goyette wrote:
|
| > I'm not so sure this is just a size issue. At one point I actually
| > looked at the contents of the work directory, and found _another_
| > work directory a couple levels down. The lower-level work directory
| > seemed to be the correct size.
| >
| > Perhaps there are some conflicting command-line args that work for
| > xz but not for pigz? (Guesing: Perhaps related to sym-links?)
|
| Well, I just checked again, and I don't see the "nested" work directory.
|
| So perhapas setting the size as you suggest is appropriate.
But why is it that much bigger?
christos
From: Paul Goyette <paul@whooppee.com>
To: gnats-bugs@netbsd.org
Cc: martin@duskware.de
Subject: Re: install/54844: ``build.sh install-image'' fails with -V
USE_PIGZGZIP=yes
Date: Wed, 8 Jan 2020 12:07:51 -0800 (PST)
On Wed, 8 Jan 2020, Paul Goyette wrote:
> On Wed, 8 Jan 2020, Paul Goyette wrote:
>
>> I'm not so sure this is just a size issue. At one point I actually
>> looked at the contents of the work directory, and found _another_
>> work directory a couple levels down. The lower-level work directory
>> seemed to be the correct size.
>>
>> Perhaps there are some conflicting command-line args that work for
>> xz but not for pigz? (Guesing: Perhaps related to sym-links?)
>
> Well, I just checked again, and I don't see the "nested" work directory.
>
> So perhaps setting the size as you suggest is appropriate.
Actually, the patch as-is doesn't work. It seems that USE_XZ_SETS
is still defined - for some reason USE_PIGZGZIP didn't unset it (at
least, not for the install-image target).
+--------------------+--------------------------+-----------------------+
| Paul Goyette | PGP Key fingerprint: | E-mail addresses: |
| (Retired) | FA29 0E3B 35AF E8AE 6651 | paul@whooppee.com |
| Software Developer | 0786 F758 55DE 53BA 7731 | pgoyette@netbsd.org |
+--------------------+--------------------------+-----------------------+
From: Paul Goyette <paul@whooppee.com>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: install/54844: ``build.sh install-image'' fails with -V
USE_PIGZGZIP=yes
Date: Mon, 18 Apr 2022 12:20:35 -0700 (PDT)
OK, I've been looking a little deper into this. As christos@ asked
some time ago
But why is it that much bigger?
The answer seems to be that pigz(1) is not nearly as good as xz(1)
at compressing NetBSD distribution sets. Here is a comparison of
sizes of some distribution sets:
pigz xz
---- ----
base.tgz 77M 44M
comp.tgz 160M 69M
debug.tgz 1.2G 688M
Since the distribution sets themselves are included in the install
image, the image's filesystem needs to be a lot bigger.
So, since (1) there's no real bug here, (2) using pigz is not a
support option, and (3) the problem can be worked around by
including ``-V INSTIMAGEMB=<size-needed>'' on the build.sh command
line, I'm going to close this PR.
+--------------------+--------------------------+----------------------+
| Paul Goyette | PGP Key fingerprint: | E-mail addresses: |
| (Retired) | FA29 0E3B 35AF E8AE 6651 | paul@whooppee.com |
| Software Developer | 0786 F758 55DE 53BA 7731 | pgoyette@netbsd.org |
| & Network Engineer | | pgoyette99@gmail.com |
+--------------------+--------------------------+----------------------+
State-Changed-From-To: open->closed
State-Changed-By: pgoyette@NetBSD.org
State-Changed-When: Mon, 18 Apr 2022 19:27:55 +0000
State-Changed-Why:
Not a bug, involves unsupported option, and workaround exists.
XXX It would be nice, however, if TOOL_XZ could be built to include
multi-CPU / thread support. :-)
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@netbsd.org
Cc: install-manager@netbsd.org, netbsd-bugs@netbsd.org,
gnats-admin@netbsd.org, pgoyette@NetBSD.org, paul@whooppee.com
Subject: Re: install/54844 (``build.sh install-image'' fails with -V
USE_PIGZGZIP=yes)
Date: Tue, 19 Apr 2022 06:30:29 +0200
On Mon, Apr 18, 2022 at 07:27:56PM +0000, pgoyette@NetBSD.org wrote:
> Not a bug, involves unsupported option, and workaround exists.
I wouldn't mind making the image size depend on USE_XZ_SETS, the problem
here is that in many places there is a pair of knobs that need to be
tuned togethere, one in a Makefile (easy to use USE_XZ_SETS), and one in
a config(8) file - and for the latter we have no easy conditional.
If someone would enhance config(8) input language so we could override
some of the defines in there on the command line, we could solve this
properly.
Martin
From: Paul Goyette <paul@whooppee.com>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: install/54844: ``build.sh install-image'' fails with -V
USE_PIGZGZIP=yes
Date: Tue, 19 Apr 2022 14:35:32 -0700 (PDT)
The following patches provided by tnn@ will enable posxi-thread mode
when building TOOL_XZ. The final patch adds the ``-T0'' option when
invoking TOOL_XZ to build the distribution sets. Together, these
changes reduce my time-to-build from 2 hours to 90 minutes, a 25%
reduction.
It would be nice if someone with more make-foo than I could get
this a bit prettier and controllable with a USE_XZ_TOOL_THREADS
knob (or similar).
Index: external/public-domain/xz/bin/xz/Makefile
===================================================================
RCS file: /cvsroot/src/external/public-domain/xz/bin/xz/Makefile,v
retrieving revision 1.6
diff -p -u -r1.6 Makefile
--- external/public-domain/xz/bin/xz/Makefile 12 Apr 2021 02:54:08 -0000 1.6
+++ external/public-domain/xz/bin/xz/Makefile 19 Apr 2022 18:03:58 -0000
@@ -44,7 +44,7 @@ FILESNAME_${XZSRCDIR}/po/${lang}.gmo= xz
.if defined(HOSTPROG)
HOST_CPPFLAGS+= ${CPPFLAGS:N-Wp,-iremap,*}
XZLIBDIR!= cd ${NETBSDSRCDIR}/tools/xz-lib && ${PRINTOBJDIR}
-LDADD+= -L${XZLIBDIR} -llzma
+LDADD+= -L${XZLIBDIR} -llzma -lpthread
DPADD+= ${XZLIBDIR}/liblzma.a
.else
DPADD+= ${LIBLZMA} ${LIBINTL} ${LIBPTHREAD}
Index: external/public-domain/xz/lib/Makefile
===================================================================
RCS file: /cvsroot/src/external/public-domain/xz/lib/Makefile,v
retrieving revision 1.10
diff -p -u -r1.10 Makefile
--- external/public-domain/xz/lib/Makefile 25 Sep 2018 05:42:08 -0000 1.10
+++ external/public-domain/xz/lib/Makefile 19 Apr 2022 18:03:58 -0000
@@ -57,9 +57,7 @@ SRCS+= common.c block_util.c easy_preset
index_decoder.c index_hash.c stream_buffer_decoder.c \
stream_decoder.c stream_flags_decoder.c vli_decoder.c
-.if !defined(HOSTLIB)
SRCS+= stream_encoder_mt.c
-.endif
.PATH: ${XZSRCDIR}/src/liblzma/delta
SRCS+= delta_common.c delta_encoder.c delta_decoder.c
Index: tools/xz-include/Makefile
===================================================================
RCS file: /cvsroot/src/tools/xz-include/Makefile,v
retrieving revision 1.4
diff -p -u -r1.4 Makefile
--- tools/xz-include/Makefile 18 Sep 2021 01:47:11 -0000 1.4
+++ tools/xz-include/Makefile 19 Apr 2022 18:03:59 -0000
@@ -8,7 +8,7 @@
#
.include "Makefile.inc"
-CONFIGURE_ARGS+= --enable-threads=no --disable-nls
+CONFIGURE_ARGS+= --enable-threads=posix --disable-nls
.if ${MAKEVERBOSE} == 0
CONFIGURE_ARGS+= --silent
.endif
Index: distrib/sets/Makefile
===================================================================
RCS file: /cvsroot/src/distrib/sets/Makefile,v
retrieving revision 1.107
diff -u -p -r1.107 Makefile
--- Makefile 10 Mar 2019 11:04:08 -0000 1.107
+++ Makefile 19 Apr 2022 21:28:32 -0000
@@ -18,6 +18,7 @@ all:
COMPRESS_PROGRAM=${"${USE_XZ_SETS:Uno}"!="no":?${TOOL_XZ}:${TOOL_GZIP}}
TAR_SUFF=${"${USE_XZ_SETS:Uno}"!="no":?tar.xz:tgz}
XZ_OPT= -9
+XZ_OPT+= -T0
GZIP_FLAGS= ${GZIP_N_FLAG}
SETSENV= DESTDIR=${DESTDIR:Q} \
+--------------------+--------------------------+----------------------+
| Paul Goyette | PGP Key fingerprint: | E-mail addresses: |
| (Retired) | FA29 0E3B 35AF E8AE 6651 | paul@whooppee.com |
| Software Developer | 0786 F758 55DE 53BA 7731 | pgoyette@netbsd.org |
| & Network Engineer | | pgoyette99@gmail.com |
+--------------------+--------------------------+----------------------+
State-Changed-From-To: closed->open
State-Changed-By: pgoyette@NetBSD.org
State-Changed-When: Sat, 18 Feb 2023 17:16:50 +0000
State-Changed-Why:
Per discussion on IRC, this should be looked at as an issue with
USE_XZ_SETS, which (when building install-image target) does not
get undefined when USE_PIGZGZIP is enabled.
From: Joerg Sonnenberger <joerg@bec.de>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: install/54844 (``build.sh install-image'' fails with -V
USE_PIGZGZIP=yes)
Date: Sat, 18 Feb 2023 18:22:37 +0100
Am Sat, Feb 18, 2023 at 05:16:51PM +0000 schrieb pgoyette@NetBSD.org:
> Synopsis: ``build.sh install-image'' fails with -V USE_PIGZGZIP=yes
>
> State-Changed-From-To: closed->open
> State-Changed-By: pgoyette@NetBSD.org
> State-Changed-When: Sat, 18 Feb 2023 17:16:50 +0000
> State-Changed-Why:
> Per discussion on IRC, this should be looked at as an issue with
> USE_XZ_SETS, which (when building install-image target) does not
> get undefined when USE_PIGZGZIP is enabled.
No, that's not what I said. USE_XZ_SETS should be what decides the
compression method (xz vs gzip). USE_PIGZGZIP decides what
implementation of gzip is used for tools purposes.
Joerg
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.47 2022/09/11 19:34:41 kim Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2023
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home