NetBSD Problem Report #58226

From martin@duskware.de  Mon May  6 14:38:55 2024
Return-Path: <martin@duskware.de>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 5CA631A9238
	for <gnats-bugs@gnats.NetBSD.org>; Mon,  6 May 2024 14:38:55 +0000 (UTC)
From: martin@NetBSD.org
Reply-To: martin@NetBSD.org
To: gnats-bugs@NetBSD.org
Subject: errors in set tests
X-Send-Pr-Version: 3.95

>Number:         58226
>Category:       toolchain
>Synopsis:       errors in set tests
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    toolchain-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon May 06 14:40:00 +0000 2024
>Last-Modified:  Tue Jun 10 15:15:01 +0000 2025
>Originator:     Martin Husemann
>Release:        NetBSD 10.99.10
>Organization:
The NetBSD Foundation, Inc.
>Environment:
System: NetBSD space-truckin.duskware.de 10.99.10 NetBSD 10.99.10 (GENERIC) #346: Mon May 6 11:01:29 CEST 2024 martin@seven-days-to-the-wolves.aprisoft.de:/work/src/sys/arch/evbarm/compile/GENERIC evbarm
Architecture: earmv7hfeb
Machine: evbarm
>Description:

Somehow I sometimes get inconsistent mtree data generated for the sets build
by update build.sh -u runs.

Example failure from the sets tests:

tc-so:Executing command [ mtree -e ]
tc-se:Fail: incorrect exit status: 2, expected: 0
tc-se:stdout:
tc-se:rescue/[: 
tc-se:  size (8412280, 8412304)
tc-se:  sha256 (0xc397eb3e182ba4df2af5b09f7c1004fd5f09f93b89fed819451d84e69f907af9, 0xee14402ac2a37e0f270b7d16feef0671f66451a8cd3541a267f269d5868ec257)

but the build machines $DESTDIR/rescue/[ and the installed version
agree:

> ll \[
-r-xr-xr-x  154 builds  builds  8412304 May  6 09:30 [*
> cksum -a sha256 \[
SHA256 ([) = ee14402ac2a37e0f270b7d16feef0671f66451a8cd3541a267f269d5868ec257

and I don't see where the bogus size/checksum recorded in the mtree file
come from.

>How-To-Repeat:
build sets with -u, install, run usr.bin/mtree/t_sets.

Full build.sh command line used in this case:

./build.sh -u -U -m evbearmv7hf-eb -M /work/obj/evbearmv7hf-eb -R /branches/rel-current/evbearmv7hf-eb -V BUILDINFO=Build locally, cvs.netbsd.org source tree date: 2024-05-06 06:16 UTC -V MKDEBUG=yes -V TMPDIR=/var/shm/btmp -V MKCROSSGDB=yes -T /work/tools -D /branches/hosts-current/evbearmv7hf-eb -j 8 -x -X /work/xsrc distribution sets


>Fix:
n/a

>Audit-Trail:
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: toolchain/58226: building crunched binaries is not MAKE_JOBS save
Date: Wed, 15 May 2024 14:18:44 +0200

 mrg identified the culprit in the crunchgen generated .mk fragment,
 e.g. $OBJ/rescue/rescue.mk:

 all: ${PROG}.crunched
 ${PROG}.crunched: ${SUBMAKE_TARGETS} .WAIT ${PROG}.strip
 ${PROG}.strip:
         ${MAKE} -f ${PROG}.mk ${PROG}
         @[ -f ${PROG}.unstripped -a ! ${PROG} -nt ${PROG}.unstripped ] || { \
                 ${_MKSHMSG:Uecho} "  strip and clear PaX flags " ${PROG}; \
                 cp ${PROG} ${PROG}.unstripped && \
                 ${OBJCOPY} -S ${OBJCOPY_REMOVE_FLAGS} ${PROG} && \
                 ${PAXCTL} -0 ${PROG} && \
                 touch ${PROG}.unstripped; \
         }


 This creates ${PROG}.unstripped and modifies it in-place, racing with other
 make instances getting at this job.

 Martin

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: toolchain/58226: building crunched binaries is not MAKE_JOBS save
Date: Wed, 15 May 2024 15:33:19 +0200

 ... but on closer observation I also see broken sha values on other
 links in non-crunched parts of the tree:

 [/tmp] h-pulse # ls -lisa /bin/[ /bin/test 
 651458 72 -r-xr-xr-x  2 root  wheel  73600 May 15 08:41 /bin/[*
 651458 72 -r-xr-xr-x  2 root  wheel  73600 May 15 08:41 /bin/test*
 [/tmp] h-pulse # egrep 'bin/\\133|bin/test' /etc/mtree/set.base
 ./bin/\133 type=file uname=root gname=wheel mode=0555 size=73600 sha256=ce23e5de0cf34f18c332708ec5bbd8380b5448e5b94eaa4a609ba3a17dac75ae
 ./bin/test type=file uname=root gname=wheel mode=0555 size=73600 sha256=2499d6f817197cbc0f5557c48e376023582ba54a3316dc3544a9eb5b91bf7b57
 [/tmp] h-pulse # cksum -a sha256 /bin/[ /bin/test 
 SHA256 (/bin/[) = 2499d6f817197cbc0f5557c48e376023582ba54a3316dc3544a9eb5b91bf7b57
 SHA256 (/bin/test) = 2499d6f817197cbc0f5557c48e376023582ba54a3316dc3544a9eb5b91bf7b57


 so the crunchgen issue may be totally related (and would probably not hit
 as persistently this one does).

 Martin

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: toolchain/58226: errors in set tests
Date: Mon, 9 Jun 2025 09:34:05 +0200

 The reason we do not see this issues on non-update builds
 is probably this section from bsd.links.mk:

 # Handle case conflicts carefully, when _dst occurs
 # more than once after case flattening
 .if ${MKUPDATE} == "no" || ${LINKS:tl:M${_dst:tl:Q}:[\#]} > 1
 ${_t}!          ${_l} __linkinstall
 .else
 ${_t}:          ${_l} __linkinstall
 .endif


 so by always recreating the target ${_t} in the MKUPDATE=no case we avoid
 partial check sums.

 Maybe we should also use that branch when INSTPRIV.unpriv is not empty?

 Martin

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: toolchain/58226: errors in set tests
Date: Tue, 10 Jun 2025 17:12:12 +0200

 It looks like we have two kinds of problems here, all related to LINK
 or FILES targets.

 In the last run I got several failures, among them:

 rescue/[: 
   size (10665136, 10913480)
   sha256 (0x26bbf9c48470c2eccaf01bc4ad6c7c4c43a84bd047d84eb145e620e6c58a692b, 0x724dc0a71099e5a720bc30ef8ccdddc5592669e1f051e3a1a8151082024170ba)

 usr/share/examples/dhcpcd/dhcpcd.conf: 
   size (1429, 1276)
   sha256 (0x5c0e35f2b53f7ef1d9876042f170363e2a00e3f41272a73cd547c2d8027531f4, 0x0eb37ddedf4a19609a7961b2be42e4336ff81da8f555b98c9cadde603bcc4e64)


 The first one could be a race condition, as the end result in METALOG
 looks correct:

  > fgrep rescue/\[ METALOG
 ./rescue/[ type=file uname=root gname=wheel mode=0555 size=10913480 sha256=724dc0a71099e5a720bc30ef8ccdddc5592669e1f051e3a1a8151082024170ba
  > ll rescue/[
 -r-xr-xr-x  154 builds  builds  10913480 Jun 10 11:16 rescue/[*
  > sha256  rescue/[
 SHA256 (rescue/[) = 724dc0a71099e5a720bc30ef8ccdddc5592669e1f051e3a1a8151082024170ba


 but the other one is strange - looks like something did not update METALOG
 and we now have an inconsitent state:

  > fgrep examples/dhcpcd/dhcpcd.conf METALOG 
 ./usr/share/examples/dhcpcd/dhcpcd.conf type=file uname=root gname=wheel mode=0444 size=1429 sha256=5c0e35f2b53f7ef1d9876042f170363e2a00e3f41272a73cd547c2d8027531f4
   > ll ./usr/share/examples/dhcpcd/dhcpcd.conf
 -r--r--r--  1 builds  builds  1276 May 19 10:10 ./usr/share/examples/dhcpcd/dhcpcd.conf
  > sha256 ./usr/share/examples/dhcpcd/dhcpcd.conf
 SHA256 (./usr/share/examples/dhcpcd/dhcpcd.conf) = 0eb37ddedf4a19609a7961b2be42e4336ff81da8f555b98c9cadde603bcc4e64

 note that not even the size matches.

 Martin

Home	PR Database Search