NetBSD Problem Report #59195
From www@netbsd.org Tue Mar 18 00:03:50 2025
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
client-signature RSA-PSS (2048 bits) client-digest SHA256)
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id F40831A9239
for <gnats-bugs@gnats.NetBSD.org>; Tue, 18 Mar 2025 00:03:49 +0000 (UTC)
Message-Id: <20250318000348.A63311A923C@mollari.NetBSD.org>
Date: Tue, 18 Mar 2025 00:03:48 +0000 (UTC)
From: logix@foobar.franken.de
Reply-To: logix@foobar.franken.de
To: gnats-bugs@NetBSD.org
Subject: Hashes missing in gzimg directories in FTP tree
X-Send-Pr-Version: www-1.0
>Number: 59195
>Notify-List: rwhitlock22@gmail.com
>Category: install
>Synopsis: Hashes missing in gzimg directories in FTP tree
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: install-manager
>State: needs-pullups
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Tue Mar 18 00:05:00 +0000 2025
>Closed-Date:
>Last-Modified: Fri Oct 09 01:56:03 +0000 2026
>Originator: Harold Gutch
>Release: head, 10, 9, ...
>Organization:
The Nohash Foundation
>Environment:
>Description:
The "gzimg" directories in the FTP tree at https://nycdn.netbsd.org/pub/NetBSD-daily/*/*/*/binary/gzimg/ (for example https://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/latest/evbarm-aarch64/binary/gzimg/ ) don't contain SHA512 and MD5 hash files.
>How-To-Repeat:
ftp -o - https://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/latest/evbarm-aarch64/binary/gzimg/
>Fix:
Unsure, src/etc/Makefile runs ${MAKESUMS} for all sorts of targets/directories, but not for INSTALLATION_DIRS which (among other things) contains binary/gzimg in src/etc/etc.evb{arm,mips,ppc}/Makefile.inc .
>Release-Note:
>Audit-Trail:
From: Harold Gutch <logix@foobar.franken.de>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: install/59195: Hashes missing in gzimg directories in FTP tree
Date: Thu, 20 Mar 2025 10:27:09 +0100
--3hAdtgBjtgL7p0NQ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Only did some rudimentary testing, but this seems to do the job.
--3hAdtgBjtgL7p0NQ
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment; filename="gzimg_hashes.patch"
--- src/distrib/utils/embedded/mkimage.orig 2025-03-17 10:15:42.882887666 +0100
+++ src/distrib/utils/embedded/mkimage 2025-03-20 10:14:04.467803455 +0100
@@ -415,4 +415,8 @@
image="${image}.gz"
fi
+cd "${IMAGEDIR}"
+${CKSUM} -a MD5 "$(basename "${image}")" > MD5
+${CKSUM} -a SHA512 "$(basename "${image}")" > SHA512
+
echo ${bar} Image is ${image} ${bar}
--- src/etc/etc.evbarm/Makefile.inc.orig 2025-03-17 10:15:55.623898568 +0100
+++ src/etc/etc.evbarm/Makefile.inc 2025-03-20 09:39:15.468388551 +0100
@@ -95,6 +95,8 @@
MACHINE=${MACHINE} \
MKDTB=${MKDTB} \
MKPOSTFIX=${MKPOSTFIX} \
+ CKSUM=${CKSUM} \
+ IMAGEDIR=${IMAGE.dir} \
${HOST_SH} ${MKIMAGE} -h ${.TARGET:S/smp_//} -D ${DESTDIR} \
-S ${NETBSDSRCDIR} -B ${IMAGEENDIAN} ${MKI_OPTS.${.TARGET}} \
${IMAGE.dir}/${.TARGET:S/smp_//}.img
--- src/etc/etc.evbmips/Makefile.inc.orig 2025-03-17 10:16:09.826910721 +0100
+++ src/etc/etc.evbmips/Makefile.inc 2025-03-20 09:39:31.865405784 +0100
@@ -70,6 +70,8 @@
MACHINE=${MACHINE} \
MKDTB=${MKDTB} \
MKPOSTFIX=${MKPOSTFIX} \
+ CKSUM=${CKSUM} \
+ IMAGEDIR=${IMAGE.dir} \
${HOST_SH} ${MKIMAGE} -h ${.TARGET:S/smp_//} -D ${DESTDIR} \
-S ${NETBSDSRCDIR} -B ${IMAGEENDIAN} ${MKI_OPTS.${.TARGET}} \
${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz
--- src/etc/etc.evbppc/Makefile.inc.orig 2025-03-17 10:16:24.527923303 +0100
+++ src/etc/etc.evbppc/Makefile.inc 2025-03-20 09:54:03.747337797 +0100
@@ -58,6 +58,8 @@
MACHINE=${MACHINE} \
MKDTB=${MKDTB} \
MKPOSTFIX=${MKPOSTFIX} \
+ CKSUM=${CKSUM} \
+ IMAGEDIR=${IMAGE.dir} \
${HOST_SH} -x ${MKIMAGE} -h ${.TARGET:S/smp_//} -D ${DESTDIR} \
-S ${NETBSDSRCDIR} -B ${IMAGEENDIAN} ${MKI_OPTS.${.TARGET}} \
${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz
--- src/etc/etc.riscv/Makefile.inc.orig 2025-03-17 10:16:40.871937301 +0100
+++ src/etc/etc.riscv/Makefile.inc 2025-03-20 09:39:49.245424050 +0100
@@ -34,6 +34,8 @@
MACHINE=${MACHINE} \
MKDTB=${MKDTB} \
MKPOSTFIX=${MKPOSTFIX} \
+ CKSUM=${CKSUM} \
+ IMAGEDIR=${IMAGE.dir} \
${HOST_SH} ${MKIMAGE} -h ${.TARGET:S/smp_//} -D ${DESTDIR} \
-S ${NETBSDSRCDIR} ${MKI_OPTS.${.TARGET}} \
${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz
--3hAdtgBjtgL7p0NQ--
State-Changed-From-To: open->pending-pullups
State-Changed-By: hgutch@NetBSD.org
State-Changed-When: Sat, 22 Mar 2025 10:36:00 +0000
State-Changed-Why:
Addressed in https://mail-index.netbsd.org/source-changes/2025/03/22/msg156023.html
Waiting for one successful HEAD run before requesting pullups.
From: "David H. Gutteridge" <david@gutteridge.ca>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: install/59195: Hashes missing in gzimg directories in FTP tree
Date: Wed, 02 Apr 2025 22:16:17 -0400
I've noticed a couple of issues with this. First, if there's more than
one file to generate a checksum for, the final one is basically missing
in the output. That is, all but two characters of the line are lost.
E.g., from
https://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/202504020520Z/evbarm-aarch64=
/binary/gzimg/SHA512
where there are the files arm64.img.gz and arm64mbr.img.gz we have:
SHA512 (arm64.img) =3D
5b1a2ce7ce6a5401f8fc2563953d12351143e24a3f7706f09bb1ef084b2ec50338f1f455
bbf43dea7c069de92960e7e4d86e1dbb8a8fcad2e67230ac1127fe97
f3
(The "f3" is the last two characters of the SHA512 checksum for
arm64mbr.img.)
Since the latter file is missing, if one runs cksum on MD5 or SHA512,
it isn't actually checked at all. (Leading to a false sense of success,
potentially, if one doesn't use -w with cksum.)
Separately, the current output kind of violates the POLA, since the
files themselves are gzipped, but the checksums are generated on the
raw images. This is confusing, since it's not the practice used for
similar checksum files provided by TNF, which are calculated against
the gzipped content.
Dave
From: "David H. Gutteridge" <david@gutteridge.ca>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: install/59195: Hashes missing in gzimg directories in FTP tree
Date: Wed, 15 Jul 2026 17:48:34 -0400
Further proposed changes are under discussion/review on the
current-users mailing list:
https://mail-index.netbsd.org/current-users/2026/07/15/msg047737.html
From: "Brook Milligan" <brook@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/59195 CVS commit: src
Date: Sun, 2 Aug 2026 02:59:14 +0000
Module Name: src
Committed By: brook
Date: Sun Aug 2 02:59:14 UTC 2026
Modified Files:
src/distrib/utils/embedded: mkimage
src/etc: Makefile
src/etc/etc.evbarm: Makefile.inc
src/etc/etc.evbmips: Makefile.inc
src/etc/etc.evbppc: Makefile.inc
src/etc/etc.riscv: Makefile.inc
Log Message:
PR install/59195 and PR port-evbmips/60302: fix hashes in gzimg directories
Hashes in gzimg directories for some ports either correspond to the
wrong (i.e., uncompressed) files or do not cover all files. These
problems are described in PR install/59195 and PR port-evbmips/60302,
respectively. The potential solution offered for PR 59195 does not
work if multiple image files exist in a single gzimg directory.
The correct solution involves three aspects: (i) make checksums
independently of mkimage, which allows a common list of hashes in
etc/Makefile, (ii) for directories producing more than one compressed
image, make checksums in individual temporary files, one per
compressed image, to combine at the end, and (iii) add make targets to
allow parallelism across multiple images within each directory.
This patch implements those steps for the evbarm, evbmips, evbppc, and
riscv ports, which were the only ones previously using mkimage for
checksums.
To generate a diff of this commit:
cvs rdiff -u -r1.97 -r1.98 src/distrib/utils/embedded/mkimage
cvs rdiff -u -r1.477 -r1.478 src/etc/Makefile
cvs rdiff -u -r1.139 -r1.140 src/etc/etc.evbarm/Makefile.inc
cvs rdiff -u -r1.32 -r1.33 src/etc/etc.evbmips/Makefile.inc
cvs rdiff -u -r1.22 -r1.23 src/etc/etc.evbppc/Makefile.inc
cvs rdiff -u -r1.7 -r1.8 src/etc/etc.riscv/Makefile.inc
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: pending-pullups->needs-pullups
State-Changed-By: gutteridge@NetBSD.org
State-Changed-When: Wed, 23 Sep 2026 15:19:34 +0000
State-Changed-Why:
Put into correct state; relevant pullups have not been submitted.
From: "David H. Gutteridge" <gutteridge@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/59195 CVS commit: src/etc/etc.evbarm
Date: Sun, 4 Oct 2026 03:44:46 +0000
Module Name: src
Committed By: gutteridge
Date: Sun Oct 4 03:44:46 UTC 2026
Modified Files:
src/etc/etc.evbarm: Makefile.inc
Log Message:
etc/etc.evbarm/Makefile.inc: follow-on hash generation fix
Revs. 141 & 142 introduced a guard that tried to prevent checksum
consolidation from running in contexts where there's nothing to process
-- that is, images aren't created at all. However, it then also
prevented clean builds that do include images from consolidating their
checksums as intended, since the "exists" check that was added reflects
too early a file system state on clean builds ("gzimg" doesn't exist
yet). It also didn't address the fact that there can be targets that
do in fact create and populate a "gzimg" directory, yet don't actually
generate any images as presently construed/defined are relevant for
hash generation, as happens with the evbearmv4-el arch.
Another fix for PR install/59195. Tested with earmv5hf (no images, no
"gzimg" directory created), evbearmv4-el (no images, "gzimg" directory
is created), and aarch64 (images and "gzimg" directory are created),
with clean builds into new directory structures and subsequent update
builds tested for each arch.
To generate a diff of this commit:
cvs rdiff -u -r1.144 -r1.145 src/etc/etc.evbarm/Makefile.inc
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: needs-pullups->pending-pullups
State-Changed-By: gutteridge@NetBSD.org
State-Changed-When: Tue, 06 Oct 2026 03:17:00 +0000
State-Changed-Why:
Pullup to netbsd-11 submitted, #511.
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/59195 CVS commit: [netbsd-11] src
Date: Thu, 8 Oct 2026 18:56:27 +0000
Module Name: src
Committed By: martin
Date: Thu Oct 8 18:56:27 UTC 2026
Modified Files:
src/distrib/utils/embedded [netbsd-11]: mkimage
src/etc [netbsd-11]: Makefile
src/etc/etc.evbarm [netbsd-11]: Makefile.inc
src/etc/etc.evbmips [netbsd-11]: Makefile.inc
src/etc/etc.evbppc [netbsd-11]: Makefile.inc
src/etc/etc.riscv [netbsd-11]: Makefile.inc
Log Message:
Pull up following revision(s) (requested by gutteridge in ticket #511):
etc/etc.riscv/Makefile.inc: revision 1.8
etc/etc.evbarm/Makefile.inc: revision 1.140
etc/etc.evbarm/Makefile.inc: revision 1.141
etc/etc.evbarm/Makefile.inc: revision 1.142
etc/etc.evbarm/Makefile.inc: revision 1.145
etc/etc.evbppc/Makefile.inc: revision 1.23
etc/etc.evbmips/Makefile.inc: revision 1.33
etc/etc.evbarm/Makefile.inc: revision 1.139
etc/Makefile: revision 1.478
distrib/utils/embedded/mkimage: revision 1.98
Generate hashes for bootable images in gzimg directory.
At one point this should probably be unified at least a bit with the hash
generation of regular sets, so that there is a single place that defines
the types of hashes generated for sets and gzimg.
PR install/59195 and PR port-evbmips/60302: fix hashes in gzimg directories
Hashes in gzimg directories for some ports either correspond to the
wrong (i.e., uncompressed) files or do not cover all files. These
problems are described in PR install/59195 and PR port-evbmips/60302,
respectively. The potential solution offered for PR 59195 does not
work if multiple image files exist in a single gzimg directory.
The correct solution involves three aspects: (i) make checksums
independently of mkimage, which allows a common list of hashes in
etc/Makefile, (ii) for directories producing more than one compressed
image, make checksums in individual temporary files, one per
compressed image, to combine at the end, and (iii) add make targets to
allow parallelism across multiple images within each directory.
This patch implements those steps for the evbarm, evbmips, evbppc, and
riscv ports, which were the only ones previously using mkimage for
checksums.
Do not try to make image hashes if we created no images
Fix indent.
etc/etc.evbarm/Makefile.inc: follow-on hash generation fix
Revs. 141 & 142 introduced a guard that tried to prevent checksum
consolidation from running in contexts where there's nothing to process
-- that is, images aren't created at all. However, it then also
prevented clean builds that do include images from consolidating their
checksums as intended, since the "exists" check that was added reflects
too early a file system state on clean builds ("gzimg" doesn't exist
yet). It also didn't address the fact that there can be targets that
do in fact create and populate a "gzimg" directory, yet don't actually
generate any images as presently construed/defined are relevant for
hash generation, as happens with the evbearmv4-el arch.
Another fix for PR install/59195. Tested with earmv5hf (no images, no
"gzimg" directory created), evbearmv4-el (no images, "gzimg" directory
is created), and aarch64 (images and "gzimg" directory are created),
with clean builds into new directory structures and subsequent update
builds tested for each arch.
To generate a diff of this commit:
cvs rdiff -u -r1.92 -r1.92.2.1 src/distrib/utils/embedded/mkimage
cvs rdiff -u -r1.475.2.2 -r1.475.2.3 src/etc/Makefile
cvs rdiff -u -r1.136 -r1.136.2.1 src/etc/etc.evbarm/Makefile.inc
cvs rdiff -u -r1.32 -r1.32.2.1 src/etc/etc.evbmips/Makefile.inc
cvs rdiff -u -r1.19.2.1 -r1.19.2.2 src/etc/etc.evbppc/Makefile.inc
cvs rdiff -u -r1.7 -r1.7.2.1 src/etc/etc.riscv/Makefile.inc
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: pending-pullups->needs-pullups
State-Changed-By: gutteridge@NetBSD.org
State-Changed-When: Fri, 09 Oct 2026 01:56:03 +0000
State-Changed-Why:
Pulled up to 11. Parking back in needs-pullups in case someone cares about 10.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home