NetBSD Problem Report #8889

Received: (qmail 22577 invoked from network); 26 Nov 1999 23:29:53 -0000
Message-Id: <199911262327.QAA11630@zoo>
Date: Fri, 26 Nov 1999 16:27:53 -0700 (MST)
From: jbernard@mines.edu
Reply-To: jbernard@mines.edu
To: gnats-bugs@gnats.netbsd.org
Subject: -current LFS corruption
X-Send-Pr-Version: 3.95

>Number:         8889
>Category:       kern
>Synopsis:       LFS corruption: null files, dirty files in clean segments, non-removable directories
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    kern-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Nov 26 15:30:01 +0000 1999
>Closed-Date:    Wed Mar 31 16:04:13 +0000 2004
>Last-Modified:  Wed Mar 31 16:04:13 +0000 2004
>Originator:     Jim Bernard
>Release:        Nov. 25, 1999
>Organization:
	Speaking for myself
>Environment:
1.4P, Nov. 25, 1999, i386


>Description:
	LFS is exhibiting several forms of corruption.  I don't know exactly
	what minimum sequence of events leads to each, so I'll describe the
	sequence that has so far caused the problems.

	I rebuilt the LFS (3 GB in size) under a Nov. 14 kernel with 
	userland from the Nov. 13 snapshot, and unpacked a -current source
	tree onto the filesystem.  No corruption was evident (except for
	the appearance of UNREF FILE messages from fsck_lfs, which is
	apparently harmless) in a couple of days of relatively light
	operation.  At some point (not sure exactly when, but I think it was
	at this point), I noticed a couple of files with wrong timestamps
	(more recent than they should be), but dismissed that as not critical.
	I then built a new kernel (sources supped after Nov. 25 supscan; built
	in a scratch directory on the LFS, with the source tree, also on LFS,
	union mounted beneath the scratch directory), and booted it, with no
	immediately apparent problems.  I then did a full system build (again
	with sources on the LFS filesystem union mounted beneath a scratch
	directory on the LFS), with no immediately apparent problems, and
	rebooted.  Then "fsck_lfs -n -d" reported in phase 1 some 15,000
	messages like:

	  ! INO 1318: daddr 0x2779b3 is in clean segment 1263

	(The number of these has decreased a bit with time, over a period of
	a bit less than 24 hours---down to a minimum of about 7,000, then
	rising slightly.)  No other corruption, besides the UNREF FILE's
	was found by fsck_lfs.

	I then unpacked the latest xsrc tarball, and supped pkgsrc (which was
	about a week out of date) and xsrc (all onto the LFS), with no apparent
	problems.  BTW: at this point, df reported something like 600 MB of
	space in use.  I then successfully built and installed the tcl80
	package (the installed files go on a different filesystem, but the
	source tree and build space were on the LFS---no union mount was used
	here).  An immediately subsequent attempt to build tk80 failed
	miserably, because some directories and files in the work subdirectory
	(where the source tarball gets unpacked) were null (ls showed all mode
	bits off and 0 link count):

	  ----------  0 someuser  somegroup   (not sure of the rest)

	Furthermore, some directories could not be removed, reminiscent of
	the problem reported in kern/8815, which has, however been fixed
	(the current directory-removal problem evidently occurs far less
	frequently).

	Altogether, I tried building tk80 three times; finding different
	problem directories and files each time (only on the second try
	were there non-removable directories).  The third try crashed the
	machine (and I'm miles from the machine at the moment, so can't
	get a traceback right now---I'll submit an addendum when I get one).

>How-To-Repeat:
	I imagine any use of LFS for a while would eventually lead to these
	problems, but the sequence that did it for me is described above.

>Fix:
	Unknown.
>Release-Note:
>Audit-Trail:

From: Jim Bernard <jbernard@ox.mines.edu>
To: gnats-bugs@gnats.netbsd.org
Cc: jbernard@mines.edu
Subject: Re: kern/8889: -current LFS corruption
Date: Sat, 27 Nov 1999 10:49:32 -0700

 On Fri, Nov 26, 1999 at 04:27:53PM -0700, jbernard@Mines.EDU wrote:
 > 	were there non-removable directories).  The third try crashed the
 > 	machine (and I'm miles from the machine at the moment, so can't
 > 	get a traceback right now---I'll submit an addendum when I get one).

   Here's info from ddb (no crash dump):

 panic: lfs_ialloc: in use inode 101860 on the free list
 stopped in tar at cpu_Debugger+0x4: leave

 trace:
 cpu_Debugger (f1218b58,2d6,f2,f5e31c6c,f01ca0c6) at cpu_Debugger+0x4
 panic (f023f120,18de4,f5e31e1c,f5e31e08,81a4) at panic+0x55
 lfs_valloc (f5e31c90,f5e31ec1,f5e31e08,0,f5e31c90) at lfs_valloc+0xc2
 ufs_makeinode (81a4,f5e3d68c,f5e31ee8,f5e31efc,f5e31de8) at ufs_makeinode+0x96
 ufs_create (f5e31e08,f5e31e1c,f5e31ed8,f5e31e08,6) at ufs_create+0x2a
 lfs_create (f5e31e08,0,f5e31f88,fffffffc,f05fbf80) at lfs_create+0x52
 vn_open (f5e31ed8,60e,1a4,f5e31f88,f563c78c) at vn_open+0xe8
 sys_open (f563c78c,f5e31f88,f5e31f80,0,8091000) at sys_open+0xd0
 syscall() at syscall+0x206
 ---syscall (number 5) ---
 0x8071fd7:


 ps shows tar as the running process at the time of the panic

From: Konrad Schroder <perseant@hhhh.org>
To: jbernard@mines.edu
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: kern/8889: -current LFS corruption
Date: Sun, 28 Nov 1999 21:22:33 -0800 (PST)

 On Fri, 26 Nov 1999 jbernard@mines.edu wrote:

 [...]
 > 	operation.  At some point (not sure exactly when, but I think it was
 > 	at this point), I noticed a couple of files with wrong timestamps
 > 	(more recent than they should be), but dismissed that as not critical.

 Interesting.  This may mean that the cleaning syscalls are not properly
 separating cleaning time from update time.  But if so this is, as you
 say, not critical.

 [...]
 > 	directory on the LFS), with no immediately apparent problems, and
 > 	rebooted.  Then "fsck_lfs -n -d" reported in phase 1 some 15,000
 > 	messages like:
 > 
 > 	  ! INO 1318: daddr 0x2779b3 is in clean segment 1263

 At this point on the other hand you are screwed.  (But before this you
 didn't have any hangs or crashes?)  Until this segment is reallocated you
 will see this number decrease (without any apparent corruption except for
 this message from fsck); when it is reallocated for use suddenly lots of
 Bad Things will happen.

 Try applying the appended diff and see if that stabilizes things.  If it
 doesn't, crash dumps will not be so useful for this kind of problem, but
 filesystem images will (if you have enough space to store a 3G fs image,
 ugh.)  But please do let me know either way.

 						Konrad Schroder
 						perseant@hhhh.org

 Index: lfs_syscalls.c
 ===================================================================
 RCS file: /cvsroot/syssrc/sys/ufs/lfs/lfs_syscalls.c,v
 retrieving revision 1.37
 diff -u -r1.37 lfs_syscalls.c
 --- lfs_syscalls.c	1999/11/23 23:52:42	1.37
 +++ lfs_syscalls.c	1999/11/29 05:03:32
 @@ -399,13 +399,14 @@
  			}
  			splx(s);
  		}
 -		if (blkp->bi_lbn >= 0)	{ /* Data Block */
 -			/* XXX KS - should we use incore here, or just always use getblk()? */
 +		if (blkp->bi_lbn >= 0 && vp != fs->lfs_ivnode)	{
 +			/* Data block */
  			bp = lfs_fakebuf(vp, blkp->bi_lbn,
  					 blkp->bi_size, blkp->bi_bp);
  			/* Pretend we used bread() to get it */
  			bp->b_blkno = blkp->bi_daddr;
 -		} else {	/* Indirect block */
 +		} else {
 +			/* Metadata */
  			bp = getblk(vp, blkp->bi_lbn, blkp->bi_size, 0, 0);
  			if (!(bp->b_flags & (B_DONE|B_DELWRI))) { /* B_CACHE */
  				/*


From: Jim Bernard <jbernard@ox.mines.edu>
To: Konrad Schroder <perseant@hhhh.org>
Cc: gnats-bugs@gnats.netbsd.org, jbernard@mines.edu
Subject: Re: kern/8889: -current LFS corruption
Date: Tue, 30 Nov 1999 19:21:28 -0700

 On Sun, Nov 28, 1999 at 09:22:33PM -0800, Konrad Schroder wrote:
 > On Fri, 26 Nov 1999 jbernard@mines.edu wrote:
 > [...]
 > > 	directory on the LFS), with no immediately apparent problems, and
 > > 	rebooted.  Then "fsck_lfs -n -d" reported in phase 1 some 15,000
 > > 	messages like:
 > > 
 > > 	  ! INO 1318: daddr 0x2779b3 is in clean segment 1263
 > 
 > At this point on the other hand you are screwed.  (But before this you
 > didn't have any hangs or crashes?)  Until this segment is reallocated you

   Nope; it got through an entire userland build successfully.

 > Try applying the appended diff and see if that stabilizes things.  If it
 > doesn't, crash dumps will not be so useful for this kind of problem, but
 > filesystem images will (if you have enough space to store a 3G fs image,
 > ugh.)  But please do let me know either way.

   I rebuilt the filesystem a few days ago, and minor activity didn't elicit
 any reports of problems from fsck_lfs, so I built a kernel with your patch
 (Nov. 29 sources, source and build tree on the LFS, but without the union
 mount this time), and that went well.  The resulting kernel booted fine, so
 I tried a userland build.  First I copied the entire source tree to another
 directory (both original and copy on the LFS), so I could avoid the use of
 the union mount, just in case that played a role in causing the problems.
 Then, I applied a few local patches and checked the filesystem with
 fsck_lfs -n -d.  The result showed two new problems that weren't present
 immediately before the copy:
 ** /dev/rwd0g (NO WRITE)
 sb0 943928539, sb1 943928539
 dev_bsize = 512
 lfs_bsize = 8192
 lfs_fsize = 1024
 lfs_frag  = 8
 INOPB(fs) = 64
 maxino=126169
 ** Last Mounted on /home
 ** Phase 0 - Check Segment Summaries
 ** Phase 1 - Check Blocks and Sizes
 ** Phase 2 - Check Pathnames
 ** Phase 3 - Check Connectivity
 ** Phase 4 - Check Reference Counts
 UNREF FILE  I=114286  OWNER=local MODE=100644
 SIZE=29694 MTIME=Nov 17 05:32 1999 
 RECONNECT? no


 CLEAR? no

 LINK COUNT FILE I=114287  OWNER=local MODE=100644
 SIZE=6155 MTIME=Jul 31 05:10 1998  COUNT 1 SHOULD BE 2
 ADJUST? no

 126041 files, 764500 used, 0 free

 Now, here's a curiosity: inode 114287 appears *twice* in the copy of the
 source tree, in sys/dev/:

 114287 -rw-r--r--  1 local  staff  -  6155 Jul 31  1998 vnd.c
 114287 -rw-r--r--  1 local  staff  -  6155 Jul 31  1998 vndvar.h

 This, aside from the obvious bogosity, is incorrect in that vnd.c has
 the contents of vndvar.h.  The original copy of the tree has the correct
 files.

 The build failed at the installation of /usr/lib/libg2c.a, with ranlib
 complaining that the format of the file was unrecognized.  In fact the
 file had zero length, in spite of the fact that there was an apparently
 viable original in the build tree (which I later installed by hand).
 There were no messages in the build log or system logs that indicated
 a failure of install or of the copy it would have done.  There were also
 the following messages earlier in the build log (relevant to libg2c):
   /usr/bin/nm: ef1asc.o: File truncated
   /usr/bin/nm: erf.o: File truncated
   /usr/bin/nm: fseek.o: File truncated
 I checked these with nm by hand, and there is no complaint now.  So, there
 apparently were some transient failures during the build that disappeared
 (by virtue of lfs_cleanerd??) later.

 A check of the filesystem with fsck_lfs -n -d showed that it was trashed,
 with lots of these:
   ! INO 246: daddr 0x162eae is in clean segment 709
 but no new "UNREF FILE" or "LINK COUNT" errors.  The system stayed up.

 At the moment, I don't have any place to put a filesystem image, but I may
 be able to manage it in a few days.  BTW: by filesystem image, do you mean
 a literal copy made via dd, or a dump made with dump_lfs (I imagine the
 former, but wanted to be sure)?  In the mean time, would output from dumplfs
 (sans underscore) be of any use?  It looks like a pretty comprehensive picture
 of the state of the filesystem, so I imagine it might be useful, but don't
 know for sure.

 Thanks.

 --Jim Bernard
Responsible-Changed-From-To: kern-bug-people->perseant 
Responsible-Changed-By: perseant 
Responsible-Changed-When: Thu Jun 29 10:56:39 PDT 2000 
Responsible-Changed-Why:  
I am responsible for LFS problems right now. 
Responsible-Changed-From-To: perseant->kern-bug-people 
Responsible-Changed-By: perseant 
Responsible-Changed-When: Thu Nov 20 19:52:50 UTC 2003 
Responsible-Changed-Why:  
Trying to be realistic 
State-Changed-From-To: open->feedback 
State-Changed-By: tls 
State-Changed-When: Wed Mar 31 06:38:02 UTC 2004 
State-Changed-Why:  
I believe this bug is fixed in -current and on the 2.0 branch.  Can you confirm? 

From: Jim Bernard <jbernard@mines.edu>
To: gnats-bugs@gnats.netbsd.org
Cc: tls@netbsd.org, kern-bug-people@netbsd.org
Subject: Re: kern/8889
Date: Wed, 31 Mar 2004 06:38:40 -0700

 On Wed, Mar 31, 2004 at 06:38:24AM -0000, tls@netbsd.org wrote:
 > Synopsis: LFS corruption: null files, dirty files in clean segments, non-removable directories
 > 
 > State-Changed-From-To: open->feedback
 > State-Changed-By: tls
 > State-Changed-When: Wed Mar 31 06:38:02 UTC 2004
 > State-Changed-Why: 
 > I believe this bug is fixed in -current and on the 2.0 branch.  Can you confirm?

   Unfortunately, I don't have any free partitions on which I could build
 an LFS file system now, so I can't test it any time soon.  I suggest closing
 the PR if nobody else reports seeing the problem.

   Thanks for taking a look at this.
State-Changed-From-To: feedback->closed 
State-Changed-By: tls 
State-Changed-When: Wed Mar 31 16:03:51 UTC 2004 
State-Changed-Why:  
Submitter no longer has ability to reproduce; suggests closing PR. 
>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.