NetBSD Problem Report #50119

From wiz@yt.nih.at  Mon Aug  3 13:57:00 2015
Return-Path: <wiz@yt.nih.at>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 578E9A654B
	for <gnats-bugs@gnats.NetBSD.org>; Mon,  3 Aug 2015 13:57:00 +0000 (UTC)
Message-Id: <20150803135652.D50D42AC113@yt.nih.at>
Date: Mon,  3 Aug 2015 15:56:52 +0200 (CEST)
From: Thomas Klausner <wiz@NetBSD.org>
Reply-To: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Subject: MKREPRO: get repeatable timestamps for output files
X-Send-Pr-Version: 3.95

>Number:         50119
>Category:       misc
>Synopsis:       MKREPRO: get repeatable timestamps for output files
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    misc-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Aug 03 14:00:00 +0000 2015
>Closed-Date:    Mon Dec 21 14:27:13 +0000 2015
>Last-Modified:  Mon Dec 21 14:27:13 +0000 2015
>Originator:     Thomas Klausner
>Release:        NetBSD 7.99.20
>Organization:
Curiosity is the very basis of education and if you tell me that 
curiosity killed the cat, I say only that the cat died nobly.
- Arnold Edinborough
>Environment:


Architecture: x86_64
Machine: amd64
>Description:
Found in the debian MKREPRO builds for NetBSD:
When using the same tree to build on different dates, you will get different *.tgz
files since the created files get the current timestamp at the time of the build.

This makes consecutive builds of the same tree differ.
>How-To-Repeat:
Build the same tree now and five minutes/hours/days later.
See that base.tgz for the two builds is not the same.
>Fix:
A number of solutions are possible (from IRC), e.g.:
* make the output timestamp the date of the newest used input file
* use a fixed time stamp for a build, e.g.
  MKREPROTIMESTAMP=20150802T151413.12+0000
  either by touch(1)ing the files, or by making tar(1) use a fixed
  date for all files it includes

>Release-Note:

>Audit-Trail:
From: Joerg Sonnenberger <joerg@britannica.bec.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
Date: Tue, 18 Aug 2015 21:01:21 +0200

 On Mon, Aug 03, 2015 at 02:00:00PM +0000, Thomas Klausner wrote:
 > >Number:         50119
 > >Category:       misc
 > >Synopsis:       MKREPRO: get repeatable timestamps for output files
 > >Confidential:   no
 > >Severity:       non-critical
 > >Priority:       medium
 > >Responsible:    misc-bug-people
 > >State:          open
 > >Class:          sw-bug
 > >Submitter-Id:   net
 > >Arrival-Date:   Mon Aug 03 14:00:00 +0000 2015
 > >Originator:     Thomas Klausner
 > >Release:        NetBSD 7.99.20
 > >Organization:
 > Curiosity is the very basis of education and if you tell me that 
 > curiosity killed the cat, I say only that the cat died nobly.
 > - Arnold Edinborough
 > >Environment:
 > 	
 > 	
 > Architecture: x86_64
 > Machine: amd64
 > >Description:
 > Found in the debian MKREPRO builds for NetBSD:
 > When using the same tree to build on different dates, you will get different *.tgz
 > files since the created files get the current timestamp at the time of the build.

 I don't consider this a bug and I don't think there is any sane way to
 avoid it in general. For point release builds, some magic timestamp
 could be decided upon, but that's about it.

 Joegr

From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
Date: Wed, 19 Aug 2015 01:00:16 +0200

 On Tue, Aug 18, 2015 at 07:05:00PM +0000, Joerg Sonnenberger wrote:
 >  I don't consider this a bug and I don't think there is any sane way to
 >  avoid it in general. For point release builds, some magic timestamp
 >  could be decided upon, but that's about it.

 A magic timestamp is fine with me. I just want to get to a situation
 where I can rebuild a particular NetBSD version identically on a
 different machine without having to set the time on that machine and
 have the exact same runtime for the build (parts).
  Thomas

From: Joerg Sonnenberger <joerg@britannica.bec.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
Date: Wed, 19 Aug 2015 01:13:34 +0200

 On Tue, Aug 18, 2015 at 11:05:01PM +0000, Thomas Klausner wrote:
 > The following reply was made to PR misc/50119; it has been noted by GNATS.
 > 
 > From: Thomas Klausner <wiz@NetBSD.org>
 > To: gnats-bugs@NetBSD.org
 > Cc: 
 > Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
 > Date: Wed, 19 Aug 2015 01:00:16 +0200
 > 
 >  On Tue, Aug 18, 2015 at 07:05:00PM +0000, Joerg Sonnenberger wrote:
 >  >  I don't consider this a bug and I don't think there is any sane way to
 >  >  avoid it in general. For point release builds, some magic timestamp
 >  >  could be decided upon, but that's about it.
 >  
 >  A magic timestamp is fine with me. I just want to get to a situation
 >  where I can rebuild a particular NetBSD version identically on a
 >  different machine without having to set the time on that machine and
 >  have the exact same runtime for the build (parts).

 But you do get the same NetBSD version. Just like most IDS don't care
 about the timestamps in the filesystem, I don't see why we should care
 about them in the release sets.

 Joerg

From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
Date: Wed, 19 Aug 2015 01:17:17 +0200

 On Tue, Aug 18, 2015 at 11:15:01PM +0000, Joerg Sonnenberger wrote:
 >  But you do get the same NetBSD version. Just like most IDS don't care
 >  about the timestamps in the filesystem, I don't see why we should care
 >  about them in the release sets.

 Convenience: because we distribute base.tgz and not the single files,
 so it's much easy to run checksums on the couple of *.tgz files we
 distribute instead of all the single files.
  Thomas

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
Date: Wed, 19 Aug 2015 08:54:11 +0200

 On Tue, Aug 18, 2015 at 11:20:00PM +0000, Thomas Klausner wrote:
 >  Convenience: because we distribute base.tgz and not the single files,
 >  so it's much easy to run checksums on the couple of *.tgz files we
 >  distribute instead of all the single files.

 But on the other hand you lose important information - I want to know
 whether some file changed contents (easy to verify by sha1) and from
 what attempt to build the reproducable content I got it (timestamp).

 Martin

From: Joerg Sonnenberger <joerg@britannica.bec.de>
To: gnats-bugs@NetBSD.org
Cc: misc-bug-people@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org, Thomas Klausner <wiz@NetBSD.org>
Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
Date: Wed, 19 Aug 2015 12:53:03 +0200

 On Tue, Aug 18, 2015 at 11:20:00PM +0000, Thomas Klausner wrote:
 > The following reply was made to PR misc/50119; it has been noted by GNATS.
 > 
 > From: Thomas Klausner <wiz@NetBSD.org>
 > To: gnats-bugs@NetBSD.org
 > Cc: 
 > Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
 > Date: Wed, 19 Aug 2015 01:17:17 +0200
 > 
 >  On Tue, Aug 18, 2015 at 11:15:01PM +0000, Joerg Sonnenberger wrote:
 >  >  But you do get the same NetBSD version. Just like most IDS don't care
 >  >  about the timestamps in the filesystem, I don't see why we should care
 >  >  about them in the release sets.
 >  
 >  Convenience: because we distribute base.tgz and not the single files,
 >  so it's much easy to run checksums on the couple of *.tgz files we
 >  distribute instead of all the single files.

 The mtree files have the per-file checksums and are sorted, just compare
 those or filter them for checksum and file name.

 Joerg

From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
Date: Wed, 19 Aug 2015 22:07:57 +0200

 On Wed, Aug 19, 2015 at 06:55:01AM +0000, Martin Husemann wrote:
 >  But on the other hand you lose important information - I want to know
 >  whether some file changed contents (easy to verify by sha1) and from
 >  what attempt to build the reproducable content I got it (timestamp).

 Are you talking about update builds?

 I have no problem with base.tgz itself having the timestamp of the
 actual build. I just want base.tgz itself to be the same between
 MKREPRO builds (perhaps with an additional MKREPROTIMESTAMP set).
  Thomas

From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
Date: Wed, 19 Aug 2015 22:08:53 +0200

 On Wed, Aug 19, 2015 at 10:55:01AM +0000, Joerg Sonnenberger wrote:
 >  The mtree files have the per-file checksums and are sorted, just compare
 >  those or filter them for checksum and file name.

 Yes, but they don't tell me anything -- just because the mtree files
 are the same does not tell me anything about the files they are
 _supposed_ to checksum. They could contain wrong checksums.
  Thomas

From: Joerg Sonnenberger <joerg@britannica.bec.de>
To: gnats-bugs@NetBSD.org
Cc: misc-bug-people@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org, Thomas Klausner <wiz@NetBSD.org>
Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
Date: Wed, 19 Aug 2015 22:30:57 +0200

 On Wed, Aug 19, 2015 at 08:10:08PM +0000, Thomas Klausner wrote:
 > The following reply was made to PR misc/50119; it has been noted by GNATS.
 > 
 > From: Thomas Klausner <wiz@NetBSD.org>
 > To: gnats-bugs@NetBSD.org
 > Cc: 
 > Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
 > Date: Wed, 19 Aug 2015 22:08:53 +0200
 > 
 >  On Wed, Aug 19, 2015 at 10:55:01AM +0000, Joerg Sonnenberger wrote:
 >  >  The mtree files have the per-file checksums and are sorted, just compare
 >  >  those or filter them for checksum and file name.
 >  
 >  Yes, but they don't tell me anything -- just because the mtree files
 >  are the same does not tell me anything about the files they are
 >  _supposed_ to checksum. They could contain wrong checksums.

 I don't understand what you are trying to say. You can check the rest of
 the tarball against the mtree spec.

 Joerg

From: Thomas Klausner <wiz@NetBSD.org>
To: NetBSD bugtracking <gnats-bugs@NetBSD.org>
Cc: 
Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
Date: Thu, 20 Aug 2015 00:07:43 +0200

 On Wed, Aug 19, 2015 at 10:30:57PM +0200, Joerg Sonnenberger wrote:
 > I don't understand what you are trying to say. You can check the rest of
 > the tarball against the mtree spec.

 For that, I have to unpack the distfile completely. And we're back to
 my previous point where I said I just want to diff the set tgzs
 directly, because it's much easier.
  Thomas

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
Date: Thu, 20 Aug 2015 09:12:06 +0200

 On Wed, Aug 19, 2015 at 08:10:01PM +0000, Thomas Klausner wrote:
 >  Are you talking about update builds?
 >  
 >  I have no problem with base.tgz itself having the timestamp of the
 >  actual build. I just want base.tgz itself to be the same between
 >  MKREPRO builds (perhaps with an additional MKREPROTIMESTAMP set).

 I am talking about partialy updated systems; think binary security
 patches.

 Martin

From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
Date: Fri, 21 Aug 2015 10:58:03 +0200

 On Thu, Aug 20, 2015 at 07:15:01AM +0000, Martin Husemann wrote:
 >  I am talking about partialy updated systems; think binary security
 >  patches.

 If you use different sources, you should use a different MKREPRODATE
 value.
  Thomas

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
Date: Fri, 21 Aug 2015 11:05:16 +0200

 On Fri, Aug 21, 2015 at 09:00:01AM +0000, Thomas Klausner wrote:
 >  If you use different sources, you should use a different MKREPRODATE
 >  value.

 Yes, of course - that works for me!

 Martin

From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: misc/50119: MKREPRO: get repeatable timestamps for output files
Date: Fri, 18 Sep 2015 16:37:14 +0200

 On Fri, Aug 21, 2015 at 09:10:01AM +0000, Martin Husemann wrote:
 >  On Fri, Aug 21, 2015 at 09:00:01AM +0000, Thomas Klausner wrote:
 >  >  If you use different sources, you should use a different MKREPRODATE
 >  >  value.
 >  
 >  Yes, of course - that works for me!

 Debian suggests using a SOURCE_DATE_EPOCH variable for this:

 https://reproducible-builds.org/specs/source-date-epoch/

 We should use that.
  Thomas

From: "Christos Zoulas" <christos@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/50119 CVS commit: src/bin/pax
Date: Sat, 19 Dec 2015 13:28:55 -0500

 Module Name:	src
 Committed By:	christos
 Date:		Sat Dec 19 18:28:54 UTC 2015

 Modified Files:
 	src/bin/pax: options.c tar.1 tar.c

 Log Message:
 PR/50119: Thomas Klausner: Add --timestamp option to tar.


 To generate a diff of this commit:
 cvs rdiff -u -r1.116 -r1.117 src/bin/pax/options.c
 cvs rdiff -u -r1.35 -r1.36 src/bin/pax/tar.1
 cvs rdiff -u -r1.72 -r1.73 src/bin/pax/tar.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->closed
State-Changed-By: wiz@NetBSD.org
State-Changed-When: Mon, 21 Dec 2015 14:27:13 +0000
State-Changed-Why:
christos fixed it -- thank you!


>Unformatted:

Home	PR Database Search