NetBSD Problem Report #55757

From gson@gson.org  Tue Oct 27 10:06:37 2020
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 4E9531A9239
	for <gnats-bugs@gnats.NetBSD.org>; Tue, 27 Oct 2020 10:06:37 +0000 (UTC)
Message-Id: <20201027100633.F2F6A253EDE@guava.gson.org>
Date: Tue, 27 Oct 2020 12:06:33 +0200 (EET)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@NetBSD.org
Subject: Builds fail to clean up temporary files
X-Send-Pr-Version: 3.95

>Number:         55757
>Category:       misc
>Synopsis:       Builds fail to clean up temporary files
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    lukem
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Oct 27 10:10:00 +0000 2020
>Closed-Date:    Sun May 21 18:09:07 +0000 2023
>Last-Modified:  Mon May 22 10:05:00 +0000 2023
>Originator:     Andreas Gustafsson
>Release:        NetBSD-current (target), NetBSD-9 (host)
>Organization:
>Environment:
System: NetBSD
Architecture: x86_64
Machine: amd64
>Description:

Both the TNF testbed and my own are accumulating new files in /tmp
every time they do a build.  For example, on my testbed, after the
latest amd64 release build, which was successful, the following files
remain in /tmp:

  -rw-------    1 gson  wheel     126453 Oct 27 01:02 ccpGAqy9.ld
  -rw-------    1 gson  wheel          0 Oct 27 01:02 ccdid22o.le
  -rw-------    1 gson  wheel          0 Oct 27 01:02 ccHFelWv.cdtor.o
  -rw-------    1 gson  wheel          0 Oct 27 01:02 ccCBtXvF.cdtor.c

The build command was one with TMPDIR=/tmp; the full build command was:

   build.sh -j 24 -D /tmp/build/2020.10.26.21.34.10-amd64-baremetal/destdir -R /bracket/amd64-baremetal/test/2020.10.26.21.34.10/release -T /tmp/build/2020.10.26.21.34.10-amd64-baremetal/tools -O /tmp/build/2020.10.26.21.34.10-amd64-baremetal/obj -m amd64 -V TMPDIR=/tmp -x -U release sourcesets

>How-To-Repeat:

>Fix:

>Release-Note:

>Audit-Trail:
From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: misc/55757: Builds fail to clean up temporary files
Date: Tue, 27 Oct 2020 21:52:24 +0200

 This is an old problem - it also happens with sources from
 2016.12.10.09.51.43, for example.  I can't easily test versions
 much older than that because they tend to no longer build on 9.0.
 -- 
 Andreas Gustafsson, gson@gson.org

From: matthew green <mrg@eterna.com.au>
To: gnats-bugs@netbsd.org
Cc: misc-bug-people@netbsd.org, gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org, gson@gson.org (Andreas Gustafsson)
Subject: re: misc/55757: Builds fail to clean up temporary files
Date: Wed, 28 Oct 2020 14:45:57 +1100

 > -rw-------    1 gson  wheel     126453 Oct 27 01:02 ccpGAqy9.ld
 > -rw-------    1 gson  wheel          0 Oct 27 01:02 ccdid22o.le
 > -rw-------    1 gson  wheel          0 Oct 27 01:02 ccHFelWv.cdtor.o
 > -rw-------    1 gson  wheel          0 Oct 27 01:02 ccCBtXvF.cdtor.c

 can you determine the phase of build that 1:02 am was?

 these are compiler temp files, but not the normal ones.
 this is most likely a bug in GCC, but i don't know what
 options generate those files off hand.

 thanks.


 .mrg.

From: Andreas Gustafsson <gson@gson.org>
To: matthew green <mrg@eterna.com.au>
Cc: gnats-bugs@netbsd.org
Subject: re: misc/55757: Builds fail to clean up temporary files
Date: Thu, 29 Oct 2020 09:53:25 +0200

 matthew green wrote:
 > can you determine the phase of build that 1:02 am was?

 Too late for that build, but I ran a new one with the output piped
 through the timestamping tool "ts" from pkgsrc/misc/moreutils, and
 with -j 1 to reduce ambiguity.  This left the following files in /tmp:

 -rw-------    1 gson  wheel     119653 Oct 29 02:01:34 2020 ccF9Yr0e.ld
 -rw-------    1 gson  wheel          0 Oct 29 02:01:34 2020 ccwyu7pT.le
 -rw-------    1 gson  wheel          0 Oct 29 02:01:34 2020 cct7U4rZ.cdtor.o
 -rw-------    1 gson  wheel          0 Oct 29 02:01:34 2020 cc5bxOgO.cdtor.c

 There was more than 30 kB of log output during the second when the
 files in /tmp were created, so I'm not including it here, but it
 was building biosboot. The full build log is at

   https://www.gson.org/netbsd/bugs/build-tmpfile/buildlog.gz

 There may also be some clues in the contents of the .ld file, which
 are human-readable text:

   https://www.gson.org/netbsd/bugs/build-tmpfile/ccF9Yr0e.ld

 -- 
 Andreas Gustafsson, gson@gson.org

From: "John D. Baker" <jdbaker@consolidated.net>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/55757: Builds fail to clean up temporary files
Date: Thu, 29 Oct 2020 11:51:36 -0500 (CDT)

 I just checked "/tmp" on my build hosts and indeed the leftover files
 present, namely the "*.ld" files, indicate "biosboot" as the target
 being built.  The other files are from about the same time.  In
 timestamp order they are:

 netbsd-9/i386:  cc*.c, cc*.o, cc*.le, cc*.ld

 -current/{amd64,i386}:  cc*.cdtor.c, cc*.cdtor.o, cc*.le, cc*.ld

 All except the "cc*.ld" files are zero length.

 -- 
 |/"\ John D. Baker, KN5UKS               NetBSD     Darwin/MacOS X
 |\ / jdbaker[snail]consolidated[flyspeck]net  OpenBSD            FreeBSD
 | X  No HTML/proprietary data in email.   BSD just sits there and works!
 |/ \ GPGkeyID:  D703 4A7E 479F 63F8 D3F4  BD99 9572 8F23 E4AD 1645

State-Changed-From-To: open->feedback
State-Changed-By: lukem@NetBSD.org
State-Changed-When: Fri, 19 May 2023 11:54:22 +0000
State-Changed-Why:

Is this still an issue building -current?

I've looked through the sources for "cdtor.c" references and it's
just in gcc's collect2.c. There were some changes in how that
file was generated in the import of gcc 9.3.0 on 2020-09-05.
As far as I can tell, the *cdtor.c and *cdtor.o files are
saved persistently with -save-temps, or are created with mkstemps()
by make_temp_file_with_prefix() (backend for make_temp_file())
in libiberty/make-temp-file.c.
Files created by mkstemps() should be cleaned up on process exit,
unless the (gcc) process terminates abnormally.

If the files are still occurring, are they from the tools gcc,
or the hosts's native gcc?


Responsible-Changed-From-To: misc-bug-people->lukem
Responsible-Changed-By: lukem@NetBSD.org
Responsible-Changed-When: Sat, 20 May 2023 08:40:31 +0000
Responsible-Changed-Why:


From: Andreas Gustafsson <gson@gson.org>
To: lukem@NetBSD.org
Cc: gnats-bugs@netbsd.org
Subject: Re: misc/55757 (Builds fail to clean up temporary files)
Date: Sun, 21 May 2023 10:01:41 +0300

 lukem@NetBSD.org asked:
 > Is this still an issue building -current?

 Thank you for looking into this.  Looks like it is no longer an issue
 building -current.  I just checked the /tmp on babylon5.netbsd.org,
 and although it is still accumulating temporary files, it is now doing
 so at a much slower rate than when the PR was filed, and all the ones
 I examined were either from builds of branches other than HEAD or from
 builds of historical versions of -current.

 > I've looked through the sources for "cdtor.c" references and it's
 > just in gcc's collect2.c. There were some changes in how that
 > file was generated in the import of gcc 9.3.0 on 2020-09-05.
 > As far as I can tell, the *cdtor.c and *cdtor.o files are
 > saved persistently with -save-temps, or are created with mkstemps()
 > by make_temp_file_with_prefix() (backend for make_temp_file())
 > in libiberty/make-temp-file.c.
 > Files created by mkstemps() should be cleaned up on process exit,
 > unless the (gcc) process terminates abnormally.

 I don't think it was fixed by the gcc 9.3.0 import specifically,
 because the PR said the problem was present in an amd64 build of
 sources from 2020-10-26, which is after gcc 9.3.0 was enabled on
 amd64.  I'm currently running an automated bisection to find out when
 it was actually fixed, but that will take another day or so to
 complete.

 > If the files are still occurring, are they from the tools gcc,
 > or the hosts's native gcc?

 According to the PR, the .ld files are from the build of biosboot,
 and since biosboot is not itself a tool, presumably the files are
 from the tools gcc.
 -- 
 Andreas Gustafsson, gson@gson.org

From: Luke Mewburn <luke@mewburn.net>
To: gnats-bugs@netbsd.org
Cc: lukem@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
	Andreas Gustafsson <gson@gson.org>
Subject: Re: misc/55757 (Builds fail to clean up temporary files)
Date: Mon, 22 May 2023 00:42:09 +1000

 On 23-05-21 07:05, Andreas Gustafsson wrote:
   |  lukem@NetBSD.org asked:
   |  > Is this still an issue building -current?
   |  
   |  Thank you for looking into this.  Looks like it is no longer an issue
   |  building -current.  I just checked the /tmp on babylon5.netbsd.org,
   |  and although it is still accumulating temporary files, it is now doing
   |  so at a much slower rate than when the PR was filed, and all the ones
   |  I examined were either from builds of branches other than HEAD or from
   |  builds of historical versions of -current.

 Are you happy for the PR to be closed now, or do you want to wait until
 you've done root cause analysis using automated bisection as described
 below?


   |  > I've looked through the sources for "cdtor.c" references and it's
   |  > just in gcc's collect2.c. There were some changes in how that
   |  > file was generated in the import of gcc 9.3.0 on 2020-09-05.
   |  > As far as I can tell, the *cdtor.c and *cdtor.o files are
   |  > saved persistently with -save-temps, or are created with mkstemps()
   |  > by make_temp_file_with_prefix() (backend for make_temp_file())
   |  > in libiberty/make-temp-file.c.
   |  > Files created by mkstemps() should be cleaned up on process exit,
   |  > unless the (gcc) process terminates abnormally.
   |  
   |  I don't think it was fixed by the gcc 9.3.0 import specifically,
   |  because the PR said the problem was present in an amd64 build of
   |  sources from 2020-10-26, which is after gcc 9.3.0 was enabled on
   |  amd64.  I'm currently running an automated bisection to find out when
   |  it was actually fixed, but that will take another day or so to
   |  complete.
   |  
   |  > If the files are still occurring, are they from the tools gcc,
   |  > or the hosts's native gcc?
   |  
   |  According to the PR, the .ld files are from the build of biosboot,
   |  and since biosboot is not itself a tool, presumably the files are
   |  from the tools gcc.
   |  -- 
   |  Andreas Gustafsson, gson@gson.org
   |  

 It would be interesting to know where the issue got resolved.

 That said, if it was an issue in the host compiler (e.g., older gcc
 with a bug in temp file handling), then it's quite feasible that
 the bug is observed in a build of -current after the 9.3.0 gcc was
 imported - i.e., the PR being 6 weeks after said import - depending
 upon the age of the host compiler.


 regards,
 Luke.

State-Changed-From-To: feedback->closed
State-Changed-By: lukem@NetBSD.org
State-Changed-When: Sun, 21 May 2023 18:09:07 +0000
State-Changed-Why:
Submitter is fine with closing the PR


From: Andreas Gustafsson <gson@gson.org>
To: Luke Mewburn <luke@mewburn.net>
Cc: gnats-bugs@netbsd.org
Subject: Re: misc/55757 (Builds fail to clean up temporary files)
Date: Sun, 21 May 2023 21:06:28 +0300

 Luke Mewburn wrote:
 > Are you happy for the PR to be closed now, or do you want to wait until
 > you've done root cause analysis using automated bisection as described
 > below?

 Yes, closing it now is fine by me - I can comment on the PR when the
 bisection finishes even if it's already closed.

 > That said, if it was an issue in the host compiler (e.g., older gcc
 > with a bug in temp file handling), then it's quite feasible that
 > the bug is observed in a build of -current after the 9.3.0 gcc was
 > imported - i.e., the PR being 6 weeks after said import - depending
 > upon the age of the host compiler.

 In principle, sure, but the bisection has already encountered several
 versions both with and without the bug, all using the same host
 compiler.
 -- 
 Andreas Gustafsson, gson@gson.org

From: Andreas Gustafsson <gson@gson.org>
To: Luke Mewburn <luke@mewburn.net>
Cc: gnats-bugs@netbsd.org
Subject: Re: misc/55757 (Builds fail to clean up temporary files)
Date: Mon, 22 May 2023 12:59:57 +0300

 Luke Mewburn wrote:
 > It would be interesting to know where the issue got resolved.

 For amd64, it was resolved by one of these commits:

   2021.04.17.04.07.16 mrg src/external/gpl3/gcc/README.gcc10 1.11
   2021.04.17.04.07.16 mrg src/share/mk/bsd.own.mk 1.1240
   2021.04.17.04.08.32 mrg src/UPDATING 1.318
   2021.04.17.04.11.52 mrg src/doc/3RDPARTY 1.1795
   2021.04.17.04.11.52 mrg src/doc/CHANGES 1.2796
   2021.04.17.05.57.11 simonb src/common/lib/libc/string/memset2.c 1.6
   2021.04.17.06.02.35 simonb src/common/lib/libc/string/memset2.c 1.7
   2021.04.17.06.14.15 maya src/usr.bin/w/pr_time.c 1.19
   2021.04.17.06.14.15 maya src/usr.bin/w/w.c 1.91
   2021.04.17.08.06.58 simonb src/common/lib/libc/string/memset2.c 1.8
   2021.04.17.08.34.27 maya src/usr.bin/systat/ps.c 1.40
   2021.04.17.08.35.33 maya src/bin/ps/print.c 1.135
   2021.04.17.09.22.28 rin src/sys/arch/powerpc/include/oea/vmparam.h 1.21
   2021.04.17.09.39.33 mlelstv src/share/mk/bsd.own.mk 1.1241

 The bisection couldn't pin it down further because of build breakage,
 but the obvious candidate is src/share/mk/bsd.own.mk 1.1240 which
 enabled gcc10 for amd64.
 -- 
 Andreas Gustafsson, gson@gson.org

>Unformatted:

Home	PR Database Search