NetBSD Problem Report #31112

From raeburn@MIT.EDU  Thu Sep  1 00:00:59 2005
Return-Path: <raeburn@MIT.EDU>
Received: from biscayne-one-station.mit.edu (BISCAYNE-ONE-STATION.MIT.EDU [18.7.7.80])
	by narn.netbsd.org (Postfix) with ESMTP id 35D9F63B84C
	for <gnats-bugs@gnats.NetBSD.org>; Thu,  1 Sep 2005 00:00:59 +0000 (UTC)
Message-Id: <tx1fyspmzbi.fsf@mit.edu>
Date: Wed, 31 Aug 2005 20:00:49 -0400
From: Ken Raeburn <raeburn@MIT.EDU>
To: gnats-bugs@netbsd.org
Subject: malloc.conf and system programs
X-Send-Pr-Version: 3.95

>Number:         31112
>Category:       lib
>Synopsis:       malloc.conf and system programs
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    lib-bug-people
>State:          closed
>Class:          doc-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Sep 01 00:01:00 +0000 2005
>Closed-Date:    Tue Jun 21 03:03:35 +0000 2011
>Last-Modified:  Tue Jun 21 03:05:01 +0000 2011
>Originator:     Ken Raeburn
>Release:        NetBSD 2.0
>Organization:
	MIT
>Environment:
System: NetBSD venix.mit.edu 2.0 NetBSD 2.0 (GENERIC) #0: Tue Nov 30 21:04:03 UTC 2004 builds@build:/big/builds/ab/netbsd-2-0-RELEASE/alpha/200411300000Z-obj/big/builds/ab/netbsd-2-0-RELEASE/src/sys/arch/alpha/compile/GENERIC alpha
Architecture: alpha
Machine: alpha
>Description:

The malloc man page says:

 EXAMPLES
      To set a systemwide reduction of cache size, and to dump core whenever a
      problem occurs:

            ln -s 'A<' /etc/malloc.conf

I had set malloc.conf->AJ, and many months later, rebooted.  On
startup, fsck_ffs died on an unclean file system with an "allocation
failed" error.  The file system is about 4G, and the machine has >1G
RAM, and unlimiting the process memory size didn't help.  Removing
malloc.conf made it work just fine: It cleaned up some unreferenced
files and stuff, and my system came up.

While the man page describes how to set the 'A' option system-wide,
persistent across reboots, this may be a poor idea.  One thing the man
page does not mention is that the 'A' option is very bad for a program
which tries to make allocation requests but can handle failures
cleanly and continue.  For example, a program that runs a garbage
collector on malloc failure, or a program that caches information it's
read if it can get the extra memory to do so -- like fsck_ffs appears
to do at first glance with inode data.

I have no reason to believe that the 'J' option was causing any
problems, or that fsck_ffs has a bug relating to this situation.  (It
is possible that there is a bug, and that the failed allocation was
for some absurdly large size that was read out of memory scribbled on
by enabling the 'J' option.  But it appears that there are parts of
the fsck_ffs code that cope well with running out of storage.)

>How-To-Repeat:

The fsck_ffs failure only happened when I had an unclean file system
with unreferenced files.  I don't know what other cases will trigger
it; a forced check of another, clean file system did not.

>Fix:

Don't recommend 'A' in system-wide options, only in MALLOC_OPTIONS and
malloc_options.

Describe the sorts of cases where 'A' may be inappropriate, and how
one should be extra careful with options set via malloc.conf for
system-wide effect.  (Is 'A' the only one likely to be a problem?)

>Release-Note:

>Audit-Trail:
From: Jukka Ruohonen <jruohonen@iki.fi>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: lib/31112
Date: Fri, 22 May 2009 11:55:52 +0300

 I wonder if the following small addition would be sufficient as a general
 warning?

 It would be difficult to document all potential consequences from tuning the
 behavior of malloc().


 Index: malloc.3
 ===================================================================
 RCS file: /cvsroot/src/lib/libc/stdlib/malloc.3,v
 retrieving revision 1.26
 diff -u -p -r1.26 malloc.3
 --- malloc.3	15 Oct 2007 11:18:44 -0000	1.26
 +++ malloc.3	22 May 2009 08:50:08 -0000
 @@ -234,7 +234,8 @@ and
  .Dq Z
  options are intended for testing and debugging.
  An application which changes its behavior when these options are used
 -is flawed.
 +is flawed. Extra care should be taken in case any of these flags is set on
 +production machines.
  .Sh IMPLEMENTATION NOTES
  This allocator uses multiple arenas in order to reduce lock contention for
  threaded programs on multi-processor systems.

State-Changed-From-To: open->closed
State-Changed-By: jruoho@NetBSD.org
State-Changed-When: Tue, 21 Jun 2011 03:03:35 +0000
State-Changed-Why:

Finally fixed, thanks.



From: "Jukka Ruohonen" <jruoho@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/31112 CVS commit: src/lib/libc/stdlib
Date: Tue, 21 Jun 2011 03:02:58 +0000

 Module Name:	src
 Committed By:	jruoho
 Date:		Tue Jun 21 03:02:58 UTC 2011

 Modified Files:
 	src/lib/libc/stdlib: jemalloc.3

 Log Message:
 Note the potential danger in the options. Fixes PR lib/31112.


 To generate a diff of this commit:
 cvs rdiff -u -r1.6 -r1.7 src/lib/libc/stdlib/jemalloc.3

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

>Unformatted:

Home	PR Database Search