NetBSD Problem Report #39454

From he@smistad.uninett.no  Wed Sep  3 14:29:55 2008
Return-Path: <he@smistad.uninett.no>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by narn.NetBSD.org (Postfix) with ESMTP id D92FB63BC80
	for <gnats-bugs@gnats.NetBSD.org>; Wed,  3 Sep 2008 14:29:54 +0000 (UTC)
Message-Id: <20080903142953.4FBBB21DC44@smistad.uninett.no>
Date: Wed,  3 Sep 2008 16:29:53 +0200 (CEST)
From: he@NetBSD.org
Reply-To: he@NetBSD.org
To: gnats-bugs@gnats.NetBSD.org
Subject: UPDATE build of amd64 bootcd fails to pick up new kernel
X-Send-Pr-Version: 3.95

>Number:         39454
>Category:       misc
>Synopsis:       UPDATE build of amd64 bootcd fails to pick up new kernel
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    misc-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Sep 03 14:30:00 +0000 2008
>Last-Modified:  Wed Sep 03 19:50:02 +0000 2008
>Originator:     Havard Eidnes
>Release:        NetBSD 4.99.72
>Organization:
	I try...
>Environment:
System: Target: amd64 4.99.72 -current as of yesterday
Architecture: amd64
Machine: amd64
>Description:
	I'm trying to fix bugs preventing the installation of NetBSD
	on a HP ProLiant BL460c G1 system, using the amd64 port.

	This system typically wants an ISO image to boot from.

	I'm trying to use the in-tree methods to replicate the parts
	of the build which is creating the boot.iso image, but for
	some reason that I can't seem to fathom, I appear to be unable
	to make the build work as intended.

	I've found distrib/amd64/cdrom/bootcd/, which appears to
	describe the build procedure used.  However, doing an UPDATE
	build using "nbmake-amd64" in distrib/amd64/, the build
	creates the new ramdisk and the new netbsd-INSTALL.gz file,
	but the build in cdroms/bootcd/ etc. fail to pick up the new
	kernel and do not appear to do any work.  Also, doing
	"nbmake-amd64 release" does not install the new (or old) iso
	images.  Here's excerpts of the build log:

quattro: {108} /u/build/HEAD/tools/amd64/bin/nbmake-amd64 dependall
dependall ===> ramdisks
dependall ===> ramdisks/ramdisk
`cat.ro' is up to date.
`chmod.ro' is up to date.
`cp.ro' is up to date.
`dd.ro' is up to date.
`df.ro' is up to date.
`ed.ro' is up to date.
`ln.ro' is up to date.
...
depend ===> ramdisks/ramdisk/mbr
`fdisk.ro' is up to date.
`fsck.ro' is up to date.
`fsck_ffs.ro' is up to date.
...
`rtsol.ro' is up to date.
       link  ramdisk/ramdiskbin
      strip  ramdiskbin
      build  ramdisk/work
missing: ./bin (created)
missing: ./dev (created)
...
missing: ./var/spool/lock (created)
     create  ramdisk/ramdisk.fs
Calculated size of `ramdisk.fs.tmp': 5120000 bytes, 60 inodes
Extent size set to 4096
ramdisk.fs.tmp: 4.9MB (10000 sectors) block size 4096, fragment size 512
        using 1 cylinder groups of 4.88MB, 1250 blks, 64 inodes.
super-block backups (for fsck -b #) at:
 32,
Populating `ramdisk.fs.tmp'
Image `ramdisk.fs.tmp' complete
dependall ===> instkernel
     create  instkernel/netbsd-INSTALL (from: /u/build/HEAD/obj/amd64/sys/arch/amd64/compile/INSTALL/netbsd /u/build/HEAD/obj/amd64/distrib/amd64/ramdisks/ramdisk/ramdisk.fs)
got symbols from netbsd-INSTALL.tmp
mapped netbsd-INSTALL.tmp
copying image /u/build/HEAD/obj/amd64/distrib/amd64/ramdisks/ramdisk/ramdisk.fs into netbsd-INSTALL.tmp
done copying image
exiting
     create  instkernel/netbsd-INSTALL.gz
...
dependall ===> cdroms
dependall ===> cdroms/bootcd
dependall ===> cdroms/bootcd-com
dependall ===> cdroms/installcd
dependall ===> floppies
dependall ===> floppies/bootfloppy
     create  bootfloppy/netbsd (from: /u/build/HEAD/obj/amd64/distrib/amd64/instkernel/netbsd-INSTALL.gz)
     create  bootfloppy/boot1.fs (from: boot /u/build/HEAD/src/distrib/amd64/floppies/common/boot.cfg  netbsd)
pax: Removing leading / from absolute path names in the archive
Running instboot: /u/build/HEAD/tools/amd64/bin/nbinstallboot -mamd64 floppy.11754.tar /u/build/HEAD/dest/amd64/usr/mdec/bootxx_ustarfs
Free space in last tar block: boot 268, /u/build/HEAD/src/distrib/amd64/floppies/common/boot.cfg 46, netbsd 307,
Writing 5603328 bytes (5472 KB) on 4 disks, free space 270336 bytes (264 KB)
Creating disk 1 to boot1.fs
Creating disk 2 to boot2.fs
Creating disk 3 to boot3.fs
Creating disk 4 to boot4.fs
Final result:
-rw-r--r--  1 he  wheel  1474560 Sep  3 16:18 boot1.fs
-rw-r--r--  1 he  wheel  1474560 Sep  3 16:18 boot2.fs
-rw-r--r--  1 he  wheel  1474560 Sep  3 16:18 boot3.fs
-rw-r--r--  1 he  wheel  1204224 Sep  3 16:18 boot4.fs

	You'll note that it dutifully created the new boot floppies,
	but alas, no new bootcd iso image.

	Doing "nbmake-amd64 prepare" in cdroms/bootcd/ does indeed
	copy in the new netbsd INSTALL kernel to the OBJDIR, but
	if I thereafter do an "nbmake-amd64 dependall" in the same
	directory, nothing is done (no new iso image is created).
	I smell a missing dependency tracking allowing UPDATE builds
	to work as intended.

	I'm also somewhat surprised that there's no top-level target
	to build.sh to create the boot images; the iso-image target
	from build.sh bundles up the RELEASEDIR contents, and does not
	pick up the new INSTALL kernel I just built...

>How-To-Repeat:
	See above.

>Fix:
	Sorry, this appears to be above what I'm able to figure out.

>Audit-Trail:
From: Manuel Bouyer <bouyer@antioche.eu.org>
To: gnats-bugs@NetBSD.org
Cc: misc-bug-people@NetBSD.org, gnats-admin@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: misc/39454: UPDATE build of amd64 bootcd fails to pick up new kernel
Date: Wed, 3 Sep 2008 16:39:42 +0200

 On Wed, Sep 03, 2008 at 02:30:00PM +0000, he@NetBSD.org wrote:
 > >Description:
 > 	I'm trying to fix bugs preventing the installation of NetBSD
 > 	on a HP ProLiant BL460c G1 system, using the amd64 port.
 > 
 > 	This system typically wants an ISO image to boot from.
 > 
 > 	I'm trying to use the in-tree methods to replicate the parts
 > 	of the build which is creating the boot.iso image, but for
 > 	some reason that I can't seem to fathom, I appear to be unable
 > 	to make the build work as intended.
 > 
 > 	I've found distrib/amd64/cdrom/bootcd/, which appears to
 > 	describe the build procedure used.  However, doing an UPDATE
 > 	build using "nbmake-amd64" in distrib/amd64/, the build
 > 	creates the new ramdisk and the new netbsd-INSTALL.gz file,
 > 	but the build in cdroms/bootcd/ etc. fail to pick up the new
 > 	kernel and do not appear to do any work.  Also, doing
 > 	"nbmake-amd64 release" does not install the new (or old) iso
 > 	images.  Here's excerpts of the build log:
 > 
 > quattro: {108} /u/build/HEAD/tools/amd64/bin/nbmake-amd64 dependall

 I think the right target for this would be "iso_image" and not "dependall".
 This is typically done after dependall have been run though the whole tree.


 > [...]
 > 	I'm also somewhat surprised that there's no top-level target
 > 	to build.sh to create the boot images; the iso-image target
 > 	from build.sh bundles up the RELEASEDIR contents, and does not
 > 	pick up the new INSTALL kernel I just built...

 'build.sh release' will build a release, including binary sets and INSTALL
 kernel with embeded ramdisk.
 'build.sh iso-image' will take whatever exists in the sets directory and
 build a cd image with it. It won't try to update kernels or binaries.

 -- 
 Manuel Bouyer, LIP6, Universite Paris VI.           Manuel.Bouyer@lip6.fr
      NetBSD: 26 ans d'experience feront toujours la difference
 --

From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org, bouyer@antioche.eu.org
Cc: misc-bug-people@netbsd.org, gnats-admin@netbsd.org,
 netbsd-bugs@netbsd.org
Subject: Re: misc/39454: UPDATE build of amd64 bootcd fails to pick up new
 kernel
Date: Wed, 03 Sep 2008 20:48:34 +0200 (CEST)

 >  > quattro: {108} /u/build/HEAD/tools/amd64/bin/nbmake-amd64 dependal=
 l
 >
 >  I think the right target for this would be "iso_image" and not "depe=
 ndall".
 >  This is typically done after dependall have been run though the whol=
 e tree.
 ...
 >  'build.sh release' will build a release, including binary sets and I=
 NSTALL
 >  kernel with embeded ramdisk.
 >  'build.sh iso-image' will take whatever exists in the sets directory=
  and
 >  build a cd image with it. It won't try to update kernels or binaries=
 .=


 Is there a good reason the distrib/amd64/cdrom/ make target needs to
 be different from the floppy targets in this respect?  And why does
 not "release" dump the built bits into the appropriate RELEASEDIR
 subdirectory?  I think the Principle Of Least Astonishment has been
 seriously violated here.

 Regards,

 - H=E5vard

From: Manuel Bouyer <bouyer@antioche.eu.org>
To: Havard Eidnes <he@NetBSD.org>
Cc: gnats-bugs@NetBSD.org, misc-bug-people@NetBSD.org, gnats-admin@NetBSD.org,
        netbsd-bugs@NetBSD.org
Subject: Re: misc/39454: UPDATE build of amd64 bootcd fails to pick up new kernel
Date: Wed, 3 Sep 2008 20:52:21 +0200

 On Wed, Sep 03, 2008 at 08:48:34PM +0200, Havard Eidnes wrote:
 > >  > quattro: {108} /u/build/HEAD/tools/amd64/bin/nbmake-amd64 dependall
 > >
 > >  I think the right target for this would be "iso_image" and not "dependall".
 > >  This is typically done after dependall have been run though the whole tree.
 > ...
 > >  'build.sh release' will build a release, including binary sets and INSTALL
 > >  kernel with embeded ramdisk.
 > >  'build.sh iso-image' will take whatever exists in the sets directory and
 > >  build a cd image with it. It won't try to update kernels or binaries.
 > 
 > Is there a good reason the distrib/amd64/cdrom/ make target needs to
 > be different from the floppy targets in this respect?

 Yes, because it can't be done at the same time (it has to be run after
 release).

 > And why does
 > not "release" dump the built bits into the appropriate RELEASEDIR
 > subdirectory?

 I think it does, but maybe not when run from the distrib/ dir.

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

From: Manuel Bouyer <bouyer@antioche.eu.org>
To: Havard Eidnes <he@NetBSD.org>
Cc: gnats-bugs@NetBSD.org, misc-bug-people@NetBSD.org, gnats-admin@NetBSD.org,
        netbsd-bugs@NetBSD.org
Subject: Re: misc/39454: UPDATE build of amd64 bootcd fails to pick up new kernel
Date: Wed, 3 Sep 2008 21:17:51 +0200

 On Wed, Sep 03, 2008 at 08:52:21PM +0200, Manuel Bouyer wrote:
 > On Wed, Sep 03, 2008 at 08:48:34PM +0200, Havard Eidnes wrote:
 > > >  > quattro: {108} /u/build/HEAD/tools/amd64/bin/nbmake-amd64 dependall
 > > >
 > > >  I think the right target for this would be "iso_image" and not "dependall".
 > > >  This is typically done after dependall have been run though the whole tree.
 > > ...
 > > >  'build.sh release' will build a release, including binary sets and INSTALL
 > > >  kernel with embeded ramdisk.
 > > >  'build.sh iso-image' will take whatever exists in the sets directory and
 > > >  build a cd image with it. It won't try to update kernels or binaries.
 > > 
 > > Is there a good reason the distrib/amd64/cdrom/ make target needs to
 > > be different from the floppy targets in this respect?
 > 
 > Yes, because it can't be done at the same time (it has to be run after
 > release).
 > 
 > > And why does
 > > not "release" dump the built bits into the appropriate RELEASEDIR
 > > subdirectory?
 > 
 > I think it does, but maybe not when run from the distrib/ dir.

 To make is clear; here is what happens:

 - 'make distribution' builds from sources, and update binairies in $DESTDIR,
   and builds INSTALL kernels/ramdisks/boot media. This may include bootable
   iso images without binary sets.
 - 'make release' takes whatever is in $DESTDIR and build the release bits
   from it. It won't attempts to rebuild outdated binairies from sources.
 - 'make iso_image' takes whatever is in $RELEASEDIR and makes a
   (possibly bootable) iso image from it.

 You have to go though these 3 steps in order to have an updated from sources
 iso image with binary sets.

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

From: Havard Eidnes <he@NetBSD.org>
To: bouyer@antioche.eu.org
Cc: gnats-bugs@NetBSD.org, misc-bug-people@NetBSD.org,
 gnats-admin@NetBSD.org, netbsd-bugs@NetBSD.org
Subject: Re: misc/39454: UPDATE build of amd64 bootcd fails to pick up new
 kernel
Date: Wed, 03 Sep 2008 21:24:21 +0200 (CEST)

 > > Is there a good reason the distrib/amd64/cdrom/ make target needs t=
 o
 > > be different from the floppy targets in this respect?
 >
 > Yes, because it can't be done at the same time (it has to be run afte=
 r
 > release).

 Also when doing what's done under "build.sh ... release", which
 is to construct an ISO image containing the boot loader and the
 "fat" INSTALL kernel?  I find that somewhat hard to beleive, and
 would still expect "dependall" and "release" to do what they do
 elsewhere to avoid violating the Principle of Least Astonishment,
 not like now that when CDRELEASE is not set, the actual build
 *and* installation is done under "make release", and is hidden
 from view (actions are performed even though nothing shows up in
 the make log!) when MAKEVERBOSE=3D1...  Again, POLA violation.

 > > And why does not "release" dump the built bits into the
 > > appropriate RELEASEDIR subdirectory?
 >
 > I think it does, but maybe not when run from the distrib/ dir.

 Why would that make a difference?!?  If it does, this is yet
 another POLA violation.

 Regards,

 - H=E5vard

From: Havard Eidnes <he@NetBSD.org>
To: gnats-bugs@NetBSD.org, bouyer@antioche.eu.org
Cc: misc-bug-people@netbsd.org, gnats-admin@netbsd.org,
 netbsd-bugs@netbsd.org
Subject: Re: misc/39454: UPDATE build of amd64 bootcd fails to pick up new
 kernel
Date: Wed, 03 Sep 2008 21:34:30 +0200 (CEST)

 >  > > And why does not "release" dump the built bits into the
 >  > > appropriate RELEASEDIR subdirectory?
 >  >
 >  > I think it does, but maybe not when run from the distrib/ dir.
 >
 >  To make is clear; here is what happens:
 >
 >  - 'make distribution' builds from sources, and update binairies in $=
 DESTDIR,
 >    and builds INSTALL kernels/ramdisks/boot media. This may include b=
 ootable
 >    iso images without binary sets.
 >  - 'make release' takes whatever is in $DESTDIR and build the release=
  bits
 >    from it. It won't attempts to rebuild outdated binairies from sour=
 ces.
 >  - 'make iso_image' takes whatever is in $RELEASEDIR and makes a
 >    (possibly bootable) iso image from it.
 >
 >  You have to go though these 3 steps in order to have an updated from=
  sources
 >  iso image with binary sets.

 Yes, I'm well aware of that, but that's not what started this
 discussion.  I was asking for "how do I produce the boot-CD ISO images
 with minimal expenditure of resources", and doing a full build does not=

 count as "minimal expenditure of resources" in my book.

 I wanted to replicate what's done under "build.sh ... release".  The IS=
 O
 images which come out from this typically only contain a bootloader and=

 a "fat" INSTALL kernel, i.e. one with an embedded ramdisk with all the
 install tools.

 Such a method would be needed when repeatedly testing out fixes to the
 kernel, and making a new boot-CD ISO image to try out the new INSTALL
 kernel.

 I think I've now understood that at least the lack of any output under
 MAKEVERBOSE=3D1 in distrib/amd64/cdroms/ when doing "make release"
 seriously threw me off course, as well as the "do nothing under
 dependall, do everything under release" Makefile setup, both of them
 serious and as far as I can see completely unneccessary POLA violations=
 !

 Regards,

 - H=E5vard

From: Manuel Bouyer <bouyer@antioche.eu.org>
To: Havard Eidnes <he@NetBSD.org>
Cc: gnats-bugs@NetBSD.org, misc-bug-people@NetBSD.org, gnats-admin@NetBSD.org,
        netbsd-bugs@NetBSD.org
Subject: Re: misc/39454: UPDATE build of amd64 bootcd fails to pick up new kernel
Date: Wed, 3 Sep 2008 21:42:14 +0200

 On Wed, Sep 03, 2008 at 09:24:21PM +0200, Havard Eidnes wrote:
 > > > Is there a good reason the distrib/amd64/cdrom/ make target needs to
 > > > be different from the floppy targets in this respect?
 > >
 > > Yes, because it can't be done at the same time (it has to be run after
 > > release).
 > 
 > Also when doing what's done under "build.sh ... release", which
 > is to construct an ISO image containing the boot loader and the
 > "fat" INSTALL kernel?

 No, this one is built as part of 'release', I think.

 > I find that somewhat hard to beleive, and
 > would still expect "dependall" and "release" to do what they do
 > elsewhere to avoid violating the Principle of Least Astonishment,
 > not like now that when CDRELEASE is not set, the actual build
 > *and* installation is done under "make release", and is hidden
 > from view (actions are performed even though nothing shows up in
 > the make log!) when MAKEVERBOSE=1...  Again, POLA violation.

 I think they do. I can't see why it wouldn't work for a cdboot.
 Reading the makefiles, I think it should always copy the updated binaries;
 even if they didn't change. The make targets don't correspond to created
 files.

 > 
 > > > And why does not "release" dump the built bits into the
 > > > appropriate RELEASEDIR subdirectory?
 > >
 > > I think it does, but maybe not when run from the distrib/ dir.
 > 
 > Why would that make a difference?!?

 Because the bits to do it is in src/etc/ ?

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

From: Manuel Bouyer <bouyer@antioche.eu.org>
To: Havard Eidnes <he@NetBSD.org>
Cc: gnats-bugs@NetBSD.org, misc-bug-people@NetBSD.org, gnats-admin@NetBSD.org,
        netbsd-bugs@NetBSD.org
Subject: Re: misc/39454: UPDATE build of amd64 bootcd fails to pick up new kernel
Date: Wed, 3 Sep 2008 21:48:49 +0200

 On Wed, Sep 03, 2008 at 09:34:30PM +0200, Havard Eidnes wrote:
 > >  > > And why does not "release" dump the built bits into the
 > >  > > appropriate RELEASEDIR subdirectory?
 > >  >
 > >  > I think it does, but maybe not when run from the distrib/ dir.
 > >
 > >  To make is clear; here is what happens:
 > >
 > >  - 'make distribution' builds from sources, and update binairies in $DESTDIR,
 > >    and builds INSTALL kernels/ramdisks/boot media. This may include bootable
 > >    iso images without binary sets.
 > >  - 'make release' takes whatever is in $DESTDIR and build the release bits
 > >    from it. It won't attempts to rebuild outdated binairies from sources.
 > >  - 'make iso_image' takes whatever is in $RELEASEDIR and makes a
 > >    (possibly bootable) iso image from it.
 > >
 > >  You have to go though these 3 steps in order to have an updated from sources
 > >  iso image with binary sets.
 > 
 > Yes, I'm well aware of that, but that's not what started this
 > discussion.  I was asking for "how do I produce the boot-CD ISO images
 > with minimal expenditure of resources", and doing a full build does not
 > count as "minimal expenditure of resources" in my book.
 > 
 > I wanted to replicate what's done under "build.sh ... release".  The ISO
 > images which come out from this typically only contain a bootloader and
 > a "fat" INSTALL kernel, i.e. one with an embedded ramdisk with all the
 > install tools.
 > 
 > Such a method would be needed when repeatedly testing out fixes to the
 > kernel, and making a new boot-CD ISO image to try out the new INSTALL
 > kernel.
 > 
 > I think I've now understood that at least the lack of any output under
 > MAKEVERBOSE=1 in distrib/amd64/cdroms/ when doing "make release"
 > seriously threw me off course, as well as the "do nothing under
 > dependall, do everything under release" Makefile setup, both of them
 > serious and as far as I can see completely unneccessary POLA violations!

 Actually, the "do nothing under dependall, do everything under release" part
 can't easily be avoided; because distrib/ isn't self-contained. It depends
 on 'dependall' having being run over the whole tree to work.

 I'm not sure why MAKEVERBOSE=1 doens't work; the commands do show up in my
 make log even without MAKEVERBOSE=1.

 -- 
 Manuel Bouyer <bouyer@antioche.eu.org>
      NetBSD: 26 ans d'experience feront toujours la difference
 --

Home	PR Database Search