NetBSD Problem Report #41215

From salo@Xtrmntr.org  Tue Apr 14 21:45:16 2009
Return-Path: <salo@Xtrmntr.org>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by www.NetBSD.org (Postfix) with ESMTP id C94E463B8A5
	for <gnats-bugs@gnats.NetBSD.org>; Tue, 14 Apr 2009 21:45:15 +0000 (UTC)
Message-Id: <20090414214513.8BD6748B9@otaku.Xtrmntr.org>
Date: Tue, 14 Apr 2009 23:45:13 +0200 (CEST)
From: Lubomir Sedlacik <salo@Xtrmntr.org>
Reply-To: salo@Xtrmntr.org
To: gnats-bugs@gnats.NetBSD.org
Subject: cannot load miniroot as kmod, kernel heap full
X-Send-Pr-Version: 3.95

>Number:         41215
>Category:       kern
>Synopsis:       cannot load miniroot as kmod, kernel heap full
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Apr 14 21:50:00 +0000 2009
>Last-Modified:  Mon Feb 28 17:30:03 +0000 2011
>Originator:     Lubomir Sedlacik
>Release:        NetBSD 5.99.10
>Organization:
>Environment:
System: NetBSD 5.99.10
Architecture: i386
Machine: i386
>Description:
I got the custom miniroot.kmod built pretty easily but it's failing at boot
with full kernel heap.

(capured in qemu, since it's very hard to stop before it scrolls away on real
hardware)

 (qemu) >> VESA VBE Version 2.0 4096 k

 >> NetBSD/x86 BIOS Boot, Revision 5.3 (from NetBSD 5.99.10)
 >> Memory: 639/130048 k

      1. Boot normally
      2. Boot single user
      3. Disable ACPI
      4. Disable ACPI and SMP
      5. Drop to boot prompt

 Choose an option; RETURN for default; SPACE to stop countdown.
 Option 1 will be chosen in 0
 8509704+406516+538152 [462032+449388]=0x9e473c
 Loading /miniroot.kmod heap full (0x4ae64+32768)
 exit
 exit
 exit
 exit
 exit
 exit
 exit
 exit
 exit
 exit
 exit
 [...]

The boot.cfg is default with added line:

 load=/miniroot.kmod

I've tried GENERIC kernel with both my custom and the default provided
miniroot.kmod from the NetBSD-daily snapshot, both fail in the same fashion.

I found that it was already reported last month:
http://mail-index.netbsd.org/current-users/2008/12/17/msg006699.html

>How-To-Repeat:

# dd if=/dev/zero of=/tmp/disk.img bs=1m count=16
# vnconfig vnd0 /tmp/disk.img
# disklabel vnd0
[...]
 #       size    offset     fstype [fsize bsize cpg/sgs]
 a:     32705        63     4.2BSD      0     0     0  # (Cyl.      0*- 15)
 d:     32768         0     unused      0     0        # (Cyl.      0 - 15)
# fdisk -u vnd0
[...]
# fdisk -a vnd0
[...]
# fdisk vnd0
[...]
Partition table:
0: NetBSD (sysid 169)
    start 63, size 32705 (16 MB, Cyls 0/1/32-15), Active
# installboot -v -o console=com0 /dev/rvnd0a .../usr/mdec/bootxx_ffsv1
# newfs_ffs /dev/rvnd0a
# mount /dev/vnd0a /mnt
# cp .../netbsd /mnt
# cp .../usr/mdec/boot /mnt
# cp -r .../stand /mnt
# cp .../miniroot.kmod /mnt
# cp .../boot.cfg /mnt
# sync
# qemu -hda /dev/vnd0d -boot c -nographic

>Fix:

>Audit-Trail:
From: Andrew Doran <ad@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/41215: cannot load miniroot as kmod, kernel heap full
Date: Wed, 15 Apr 2009 08:12:42 +0000

 Hi Lubomir,

 Can you try increasing the heap size in the boot loader?
 IIRC it is in CPPFLAGS in one of the Makefiles under i386/stand/boot.

From: Lubomir Sedlacik <salo@Xtrmntr.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/41215: cannot load miniroot as kmod, kernel heap full
Date: Thu, 16 Apr 2009 01:20:52 +0200

 On Wed, Apr 15, 2009 at 08:15:05AM +0000, Andrew Doran wrote:
 >  Can you try increasing the heap size in the boot loader?
 >  IIRC it is in CPPFLAGS in one of the Makefiles under i386/stand/boot.

 Tried that, got a bit further but still no dice:

 Index: Makefile.boot
 ===================================================================
 RCS file: /cvsroot/src/sys/arch/i386/stand/boot/Makefile.boot,v
 retrieving revision 1.40
 diff -u -p -r1.40 Makefile.boot
 --- Makefile.boot       30 Mar 2009 09:22:52 -0000      1.40
 +++ Makefile.boot       15 Apr 2009 23:08:43 -0000
 @@ -81,7 +81,7 @@ CPPFLAGS+= -DEPIA_HACK
  # The biosboot code is linked to 'virtual' address of zero and is
  # loaded at physical address 0x10000.
  # XXX The heap values should be determined from _end.
 -SAMISCCPPFLAGS+= -DHEAP_START=0x30000 -DHEAP_LIMIT=0x50000
 +SAMISCCPPFLAGS+= -DHEAP_START=0x30000 -DHEAP_LIMIT=0x60000
  SAMISCMAKEFLAGS+= SA_USE_CREAD=yes     # Read compressed kernels
  SAMISCMAKEFLAGS+= SA_INCLUDE_NET=no    # Netboot via TFTP, NFS

 ---

 8509704+406516+538152 [462032+449388]=0x9e473c
 Loading /miniroot.kmod  
 Loading ffs  
 Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
     2006, 2007, 2008, 2009
     The NetBSD Foundation, Inc.  All rights reserved.
 Copyright (c) 1982, 1986, 1989, 1991, 1993
     The Regents of the University of California.  All rights reserved.
 [...]
 boot device: wd0
 root on md0a dumps on md0b
 root file system type: ffs
 exec /sbin/init: error 8
 init: trying /sbin/oinit
 exec /sbin/oinit: error 2
 init: trying /sbin/init.bak
 exec /sbin/init.bak: error 2
 init path (default /sbin/init): 

 ---

 Given that errno 8 is "Exec format error", I tried to force load exec_elf32:

 8509704+406516+538152 [462032+449388]=0x9e473c
 Loading /miniroot.kmod  
 Loading exec_elf32  
 Loading ffs  
 Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
     2006, 2007, 2008, 2009
     The NetBSD Foundation, Inc.  All rights reserved.
 Copyright (c) 1982, 1986, 1989, 1991, 1993
     The Regents of the University of California.  All rights reserved.
 [...]
 attimer0: attached to pcppi0
 uvm_fault(0xc09db680, 0, 1) -> 0xe
 fatal page fault in supervisor mode
 trap type 6 code 0 eip c069d5b5 cs 8 eflags 206 cr2 14 ilevel 0
 kernel: supervisor trap page fault, code=0
 Stopped in pid 0.1 (system) at  netbsd:_atomic_inc_32+0x5:      incl    0(%edx)
 db{0}> t
 _atomic_inc_32(c0e1fbc4,ca420d80,0,c090a90f,73,c0e1fcb4,0,32,c0976480,ca42af00) at netbsd:_atomic_inc_32+0x5
 vn_open(c0e1fbc4,1,0,ca41faa4,0,c09c1be4,c0e1fbe8,c044fa2c,c0976480,cb0a2c00) at netbsd:vn_open+0x80
 kobj_load_file(ca41faac,c0e1fcac,c09bc020,1,140cd80,0,1e1fc70,c0e1fc6c,c09c1be4, 0) at netbsd:kobj_load_file+0x94
 module_do_load(0,ca41f968,0,1,0,0,e1fce8,c04429b8,c0de7b68,c0de7b42) at netbsd:module_do_load+0x2c5
 module_do_load(0,0,0,0,0,e1d000,c0e1fd38,c03d41a7,0,c09c3300) at netbsd:module_do_load+0x435
 module_init_class(0,c09c3300,c0e1fd38,c03d4134,0,0,0,0,0,0) at netbsd:module_init_class+0xa2
 main(0,c01002cd,0,0,0,0,0,0,0,0) at netbsd:main+0x227

 ---

 Not sure if the heap is too big now or whether there is something else
 going wrong.  E.g., 0x55000 was enough to load miniroot and ffs but not
 enough for miniroot, ffs and exec_elf32.


 -- 
 -- Lubomir Sedlacik <salo@{NetBSD,Xtrmntr,silcnet}.org>   --

From: David Laight <david@l8s.co.uk>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/41215: cannot load miniroot as kmod, kernel heap full
Date: Thu, 16 Apr 2009 07:41:13 +0100

 On Tue, Apr 14, 2009 at 09:50:00PM +0000, Lubomir Sedlacik wrote:
 > >Number:         41215
 > >Synopsis:       cannot load miniroot as kmod, kernel heap full
 ...
 > I got the custom miniroot.kmod built pretty easily but it's failing at boot
 > with full kernel heap.
 ...
 >  8509704+406516+538152 [462032+449388]=0x9e473c
 >  Loading /miniroot.kmod heap full (0x4ae64+32768)
 >  exit

 What is the filesystem block size ?
 Possibly the file code is leaking a large buffer.
 (It hadn't used to matter since only 1 file was ever processed)

 	David

 -- 
 David Laight: david@l8s.co.uk

From: dieter roelants <dieter.NetBSD@pandora.be>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/41215: cannot load miniroot as kmod, kernel heap full
Date: Thu, 16 Apr 2009 15:33:20 +0200

 On Tue, 14 Apr 2009 21:50:00 +0000 (UTC)
 Lubomir Sedlacik <salo@Xtrmntr.org> wrote:

 > I got the custom miniroot.kmod built pretty easily but it's failing at boot
 > with full kernel heap.

 I don't know if it's the same cause, but booting Xen fails for me with
 the same error. But see below.

 > (capured in qemu, since it's very hard to stop before it scrolls away on real
 > hardware)

 My way of working around it (I can't use qemu, obviously), was to set
 the vesa mode to 1280x1024 because it scrolls slow enough to make me
 able to read it. To increase the time the error was on-screen, I filled
 the display first with a `vesa list`, a `help` and finally a `ls`. And
 guess what... when I run ls from the boot prompt, it succeeds loading
 the netbsd and xen kernel afterwards (I tried this several times). I
 have no clue why this is so and if it's a workaround in your case as
 well.

 dieter

From: Lubomir Sedlacik <salo@Xtrmntr.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/41215: cannot load miniroot as kmod, kernel heap full
Date: Fri, 17 Apr 2009 00:49:00 +0200

 On Thu, Apr 16, 2009 at 06:40:03AM +0000, David Laight wrote:
 >  What is the filesystem block size ?
 >  Possibly the file code is leaking a large buffer.
 >  (It hadn't used to matter since only 1 file was ever processed)

 4096 bytes, according to dumpfs.

 -- 
 -- Lubomir Sedlacik <salo@{NetBSD,Xtrmntr,silcnet}.org>   --

From: David Laight <david@l8s.co.uk>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org, salo@Xtrmntr.org
Subject: Re: kern/41215: cannot load miniroot as kmod, kernel heap full
Date: Sat, 18 Apr 2009 15:03:00 +0100

 On Thu, Apr 16, 2009 at 10:50:04PM +0000, Lubomir Sedlacik wrote:
 > The following reply was made to PR kern/41215; it has been noted by GNATS.
 > 
 > From: Lubomir Sedlacik <salo@Xtrmntr.org>
 > To: gnats-bugs@NetBSD.org
 > Cc: 
 > Subject: Re: kern/41215: cannot load miniroot as kmod, kernel heap full
 > Date: Fri, 17 Apr 2009 00:49:00 +0200
 > 
 >  On Thu, Apr 16, 2009 at 06:40:03AM +0000, David Laight wrote:
 >  >  What is the filesystem block size ?
 >  >  Possibly the file code is leaking a large buffer.
 >  >  (It hadn't used to matter since only 1 file was ever processed)
 >  
 >  4096 bytes, according to dumpfs.

 That sounds like the fragment size - giving a block size of 32k,
 rather than the more likely defaults of 1k and 8k.
 The standalons fs code allocates block sized buffers for allocation
 maps and (IIRC inodes), so I'd guess it isn't freeing them.

 It ought to be possibly to only need disk block sized buffers - which
 is also necessary in order to load /boot from a fs with large blocks.
 But that requires a full understanding of the fs layout which isn't
 obvious from the current #defines.

 	David

 -- 
 David Laight: david@l8s.co.uk

From: David Laight <david@l8s.co.uk>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/41215: cannot load miniroot as kmod, kernel heap full
Date: Mon, 20 Apr 2009 07:41:30 +0100

 On Thu, Apr 16, 2009 at 06:40:03AM +0000, David Laight wrote:
 >  
 >  Possibly the file code is leaking a large buffer.

 Just a thought, does the standalone code even implement free() ?
 A very trivial version might be useful here!

 	David

 -- 
 David Laight: david@l8s.co.uk

From: =?ISO-8859-1?Q?Gr=E9goire_Sutre?= <gregoire.sutre@gmail.com>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/41215: cannot load miniroot as kmod, kernel heap full
Date: Wed, 10 Mar 2010 10:37:11 +0100

 Hi,

 [ This is a copy of a message I sent to port-i386. ]

 With -current boot(8) on i386, loading a (large) compressed module, such
 as a kernel or an installation miniroot.kmod, leads to an error heap
 full.  In qemu:

  >> NetBSD/x86 BIOS Boot, Revision 5.4 (from NetBSD 5.99.24)
  >> Memory: 639/130048 k
  > load /miniroot.kmod
  > boot /netbsd-GENERIC.gz
 8982376+493364+539784 [476288+461807]=0xa73910
 WARNING: couldn't open ffs (/stand/i386/5.99.24/modules/ffs/ffs.kmod)
 Loading /miniroot.kmod heap full (0x48f9c+32768)
 exit
 exit
 [many more exit]

 This is likely related to PRs #41215 and #41524, and to [1].  The
 following patch seems to work, but it's a wild guess:


 diff -u -r1.42 Makefile.boot
 --- sys/arch/i386/stand/boot/Makefile.boot
 +++ sys/arch/i386/stand/boot/Makefile.boot
 @@ -81,7 +81,7 @@
   # The biosboot code is linked to 'virtual' address of zero and is
   # loaded at physical address 0x10000.
   # XXX The heap values should be determined from _end.
 -SAMISCCPPFLAGS+= -DHEAP_START=0x30000 -DHEAP_LIMIT=0x50000
 +SAMISCCPPFLAGS+= -DHEAP_START=0x28000 -DHEAP_LIMIT=0x50000
   SAMISCMAKEFLAGS+= SA_USE_CREAD=yes     # Read compressed kernels
   SAMISCMAKEFLAGS+= SA_INCLUDE_NET=no    # Netboot via TFTP, NFS


 According to the CVS logs, the value of HEAP_START was bumped from
 0x20000 to 0x30000 a year ago since the size of /boot was getting
 dangerously close to the limit (HEAP_START - 0x10000).  But the
 downside is that the heap size was reduced from 0x30000 to 0x20000, and
 the latter is apparently not enough (for compressed modules).  Using
 0x28000 for HEAP_START leaves enough space for /boot, and enough room
 for the heap in my tests.

 I don't know what are the right values here, and why the HEAP_LIMIT is
 set to 0x50000.  Some help would be welcome :-)

 Thanks,

 Grégoire

 p.s. In my tests, the problem does not occur if the modules are
       uncompressed.

 [1] http://mail-index.netbsd.org/port-xen/2009/10/29/msg005516.html

From: =?ISO-8859-1?Q?Gr=E9goire_Sutre?= <gregoire.sutre@gmail.com>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/41215: cannot load miniroot as kmod, kernel heap full
Date: Mon, 28 Feb 2011 18:27:23 +0100

 For the record, the problem I experienced does not occur anymore with
 a recent /boot (5.99.47).  This is likely due to:

  > Module Name:	src
  > Committed By:	jakllsch
  > Date:		Mon Dec 20 00:55:10 UTC 2010
  >
  > Modified Files:
  > 	src/sys/arch/i386/stand/boot: Makefile.boot
  >
  > Log Message:
  > Move the bottom of the heap off the top of the stack. Also, increase the heap
  > size by 64KiB.  These changes allows gzipped Xen with gzipped Dom0 kernels
  > to successfully boot.  I also suspect this will cure the amd64 CD boot issue
  > without disabling ext2fs support.

 Grégoire

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.