NetBSD Problem Report #8889
Received: (qmail 22577 invoked from network); 26 Nov 1999 23:29:53 -0000
Message-Id: <199911262327.QAA11630@zoo>
Date: Fri, 26 Nov 1999 16:27:53 -0700 (MST)
From: jbernard@mines.edu
Reply-To: jbernard@mines.edu
To: gnats-bugs@gnats.netbsd.org
Subject: -current LFS corruption
X-Send-Pr-Version: 3.95
>Number: 8889
>Category: kern
>Synopsis: LFS corruption: null files, dirty files in clean segments, non-removable directories
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: kern-bug-people
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Fri Nov 26 15:30:01 +0000 1999
>Closed-Date: Wed Mar 31 16:04:13 +0000 2004
>Last-Modified: Wed Mar 31 16:04:13 +0000 2004
>Originator: Jim Bernard
>Release: Nov. 25, 1999
>Organization:
Speaking for myself
>Environment:
1.4P, Nov. 25, 1999, i386
>Description:
LFS is exhibiting several forms of corruption. I don't know exactly
what minimum sequence of events leads to each, so I'll describe the
sequence that has so far caused the problems.
I rebuilt the LFS (3 GB in size) under a Nov. 14 kernel with
userland from the Nov. 13 snapshot, and unpacked a -current source
tree onto the filesystem. No corruption was evident (except for
the appearance of UNREF FILE messages from fsck_lfs, which is
apparently harmless) in a couple of days of relatively light
operation. At some point (not sure exactly when, but I think it was
at this point), I noticed a couple of files with wrong timestamps
(more recent than they should be), but dismissed that as not critical.
I then built a new kernel (sources supped after Nov. 25 supscan; built
in a scratch directory on the LFS, with the source tree, also on LFS,
union mounted beneath the scratch directory), and booted it, with no
immediately apparent problems. I then did a full system build (again
with sources on the LFS filesystem union mounted beneath a scratch
directory on the LFS), with no immediately apparent problems, and
rebooted. Then "fsck_lfs -n -d" reported in phase 1 some 15,000
messages like:
! INO 1318: daddr 0x2779b3 is in clean segment 1263
(The number of these has decreased a bit with time, over a period of
a bit less than 24 hours---down to a minimum of about 7,000, then
rising slightly.) No other corruption, besides the UNREF FILE's
was found by fsck_lfs.
I then unpacked the latest xsrc tarball, and supped pkgsrc (which was
about a week out of date) and xsrc (all onto the LFS), with no apparent
problems. BTW: at this point, df reported something like 600 MB of
space in use. I then successfully built and installed the tcl80
package (the installed files go on a different filesystem, but the
source tree and build space were on the LFS---no union mount was used
here). An immediately subsequent attempt to build tk80 failed
miserably, because some directories and files in the work subdirectory
(where the source tarball gets unpacked) were null (ls showed all mode
bits off and 0 link count):
---------- 0 someuser somegroup (not sure of the rest)
Furthermore, some directories could not be removed, reminiscent of
the problem reported in kern/8815, which has, however been fixed
(the current directory-removal problem evidently occurs far less
frequently).
Altogether, I tried building tk80 three times; finding different
problem directories and files each time (only on the second try
were there non-removable directories). The third try crashed the
machine (and I'm miles from the machine at the moment, so can't
get a traceback right now---I'll submit an addendum when I get one).
>How-To-Repeat:
I imagine any use of LFS for a while would eventually lead to these
problems, but the sequence that did it for me is described above.
>Fix:
Unknown.
>Release-Note:
>Audit-Trail:
From: Jim Bernard <jbernard@ox.mines.edu>
To: gnats-bugs@gnats.netbsd.org
Cc: jbernard@mines.edu
Subject: Re: kern/8889: -current LFS corruption
Date: Sat, 27 Nov 1999 10:49:32 -0700
On Fri, Nov 26, 1999 at 04:27:53PM -0700, jbernard@Mines.EDU wrote:
> were there non-removable directories). The third try crashed the
> machine (and I'm miles from the machine at the moment, so can't
> get a traceback right now---I'll submit an addendum when I get one).
Here's info from ddb (no crash dump):
panic: lfs_ialloc: in use inode 101860 on the free list
stopped in tar at cpu_Debugger+0x4: leave
trace:
cpu_Debugger (f1218b58,2d6,f2,f5e31c6c,f01ca0c6) at cpu_Debugger+0x4
panic (f023f120,18de4,f5e31e1c,f5e31e08,81a4) at panic+0x55
lfs_valloc (f5e31c90,f5e31ec1,f5e31e08,0,f5e31c90) at lfs_valloc+0xc2
ufs_makeinode (81a4,f5e3d68c,f5e31ee8,f5e31efc,f5e31de8) at ufs_makeinode+0x96
ufs_create (f5e31e08,f5e31e1c,f5e31ed8,f5e31e08,6) at ufs_create+0x2a
lfs_create (f5e31e08,0,f5e31f88,fffffffc,f05fbf80) at lfs_create+0x52
vn_open (f5e31ed8,60e,1a4,f5e31f88,f563c78c) at vn_open+0xe8
sys_open (f563c78c,f5e31f88,f5e31f80,0,8091000) at sys_open+0xd0
syscall() at syscall+0x206
---syscall (number 5) ---
0x8071fd7:
ps shows tar as the running process at the time of the panic
From: Konrad Schroder <perseant@hhhh.org>
To: jbernard@mines.edu
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: kern/8889: -current LFS corruption
Date: Sun, 28 Nov 1999 21:22:33 -0800 (PST)
On Fri, 26 Nov 1999 jbernard@mines.edu wrote:
[...]
> operation. At some point (not sure exactly when, but I think it was
> at this point), I noticed a couple of files with wrong timestamps
> (more recent than they should be), but dismissed that as not critical.
Interesting. This may mean that the cleaning syscalls are not properly
separating cleaning time from update time. But if so this is, as you
say, not critical.
[...]
> directory on the LFS), with no immediately apparent problems, and
> rebooted. Then "fsck_lfs -n -d" reported in phase 1 some 15,000
> messages like:
>
> ! INO 1318: daddr 0x2779b3 is in clean segment 1263
At this point on the other hand you are screwed. (But before this you
didn't have any hangs or crashes?) Until this segment is reallocated you
will see this number decrease (without any apparent corruption except for
this message from fsck); when it is reallocated for use suddenly lots of
Bad Things will happen.
Try applying the appended diff and see if that stabilizes things. If it
doesn't, crash dumps will not be so useful for this kind of problem, but
filesystem images will (if you have enough space to store a 3G fs image,
ugh.) But please do let me know either way.
Konrad Schroder
perseant@hhhh.org
Index: lfs_syscalls.c
===================================================================
RCS file: /cvsroot/syssrc/sys/ufs/lfs/lfs_syscalls.c,v
retrieving revision 1.37
diff -u -r1.37 lfs_syscalls.c
--- lfs_syscalls.c 1999/11/23 23:52:42 1.37
+++ lfs_syscalls.c 1999/11/29 05:03:32
@@ -399,13 +399,14 @@
}
splx(s);
}
- if (blkp->bi_lbn >= 0) { /* Data Block */
- /* XXX KS - should we use incore here, or just always use getblk()? */
+ if (blkp->bi_lbn >= 0 && vp != fs->lfs_ivnode) {
+ /* Data block */
bp = lfs_fakebuf(vp, blkp->bi_lbn,
blkp->bi_size, blkp->bi_bp);
/* Pretend we used bread() to get it */
bp->b_blkno = blkp->bi_daddr;
- } else { /* Indirect block */
+ } else {
+ /* Metadata */
bp = getblk(vp, blkp->bi_lbn, blkp->bi_size, 0, 0);
if (!(bp->b_flags & (B_DONE|B_DELWRI))) { /* B_CACHE */
/*
From: Jim Bernard <jbernard@ox.mines.edu>
To: Konrad Schroder <perseant@hhhh.org>
Cc: gnats-bugs@gnats.netbsd.org, jbernard@mines.edu
Subject: Re: kern/8889: -current LFS corruption
Date: Tue, 30 Nov 1999 19:21:28 -0700
On Sun, Nov 28, 1999 at 09:22:33PM -0800, Konrad Schroder wrote:
> On Fri, 26 Nov 1999 jbernard@mines.edu wrote:
> [...]
> > directory on the LFS), with no immediately apparent problems, and
> > rebooted. Then "fsck_lfs -n -d" reported in phase 1 some 15,000
> > messages like:
> >
> > ! INO 1318: daddr 0x2779b3 is in clean segment 1263
>
> At this point on the other hand you are screwed. (But before this you
> didn't have any hangs or crashes?) Until this segment is reallocated you
Nope; it got through an entire userland build successfully.
> Try applying the appended diff and see if that stabilizes things. If it
> doesn't, crash dumps will not be so useful for this kind of problem, but
> filesystem images will (if you have enough space to store a 3G fs image,
> ugh.) But please do let me know either way.
I rebuilt the filesystem a few days ago, and minor activity didn't elicit
any reports of problems from fsck_lfs, so I built a kernel with your patch
(Nov. 29 sources, source and build tree on the LFS, but without the union
mount this time), and that went well. The resulting kernel booted fine, so
I tried a userland build. First I copied the entire source tree to another
directory (both original and copy on the LFS), so I could avoid the use of
the union mount, just in case that played a role in causing the problems.
Then, I applied a few local patches and checked the filesystem with
fsck_lfs -n -d. The result showed two new problems that weren't present
immediately before the copy:
** /dev/rwd0g (NO WRITE)
sb0 943928539, sb1 943928539
dev_bsize = 512
lfs_bsize = 8192
lfs_fsize = 1024
lfs_frag = 8
INOPB(fs) = 64
maxino=126169
** Last Mounted on /home
** Phase 0 - Check Segment Summaries
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
UNREF FILE I=114286 OWNER=local MODE=100644
SIZE=29694 MTIME=Nov 17 05:32 1999
RECONNECT? no
CLEAR? no
LINK COUNT FILE I=114287 OWNER=local MODE=100644
SIZE=6155 MTIME=Jul 31 05:10 1998 COUNT 1 SHOULD BE 2
ADJUST? no
126041 files, 764500 used, 0 free
Now, here's a curiosity: inode 114287 appears *twice* in the copy of the
source tree, in sys/dev/:
114287 -rw-r--r-- 1 local staff - 6155 Jul 31 1998 vnd.c
114287 -rw-r--r-- 1 local staff - 6155 Jul 31 1998 vndvar.h
This, aside from the obvious bogosity, is incorrect in that vnd.c has
the contents of vndvar.h. The original copy of the tree has the correct
files.
The build failed at the installation of /usr/lib/libg2c.a, with ranlib
complaining that the format of the file was unrecognized. In fact the
file had zero length, in spite of the fact that there was an apparently
viable original in the build tree (which I later installed by hand).
There were no messages in the build log or system logs that indicated
a failure of install or of the copy it would have done. There were also
the following messages earlier in the build log (relevant to libg2c):
/usr/bin/nm: ef1asc.o: File truncated
/usr/bin/nm: erf.o: File truncated
/usr/bin/nm: fseek.o: File truncated
I checked these with nm by hand, and there is no complaint now. So, there
apparently were some transient failures during the build that disappeared
(by virtue of lfs_cleanerd??) later.
A check of the filesystem with fsck_lfs -n -d showed that it was trashed,
with lots of these:
! INO 246: daddr 0x162eae is in clean segment 709
but no new "UNREF FILE" or "LINK COUNT" errors. The system stayed up.
At the moment, I don't have any place to put a filesystem image, but I may
be able to manage it in a few days. BTW: by filesystem image, do you mean
a literal copy made via dd, or a dump made with dump_lfs (I imagine the
former, but wanted to be sure)? In the mean time, would output from dumplfs
(sans underscore) be of any use? It looks like a pretty comprehensive picture
of the state of the filesystem, so I imagine it might be useful, but don't
know for sure.
Thanks.
--Jim Bernard
Responsible-Changed-From-To: kern-bug-people->perseant
Responsible-Changed-By: perseant
Responsible-Changed-When: Thu Jun 29 10:56:39 PDT 2000
Responsible-Changed-Why:
I am responsible for LFS problems right now.
Responsible-Changed-From-To: perseant->kern-bug-people
Responsible-Changed-By: perseant
Responsible-Changed-When: Thu Nov 20 19:52:50 UTC 2003
Responsible-Changed-Why:
Trying to be realistic
State-Changed-From-To: open->feedback
State-Changed-By: tls
State-Changed-When: Wed Mar 31 06:38:02 UTC 2004
State-Changed-Why:
I believe this bug is fixed in -current and on the 2.0 branch. Can you confirm?
From: Jim Bernard <jbernard@mines.edu>
To: gnats-bugs@gnats.netbsd.org
Cc: tls@netbsd.org, kern-bug-people@netbsd.org
Subject: Re: kern/8889
Date: Wed, 31 Mar 2004 06:38:40 -0700
On Wed, Mar 31, 2004 at 06:38:24AM -0000, tls@netbsd.org wrote:
> Synopsis: LFS corruption: null files, dirty files in clean segments, non-removable directories
>
> State-Changed-From-To: open->feedback
> State-Changed-By: tls
> State-Changed-When: Wed Mar 31 06:38:02 UTC 2004
> State-Changed-Why:
> I believe this bug is fixed in -current and on the 2.0 branch. Can you confirm?
Unfortunately, I don't have any free partitions on which I could build
an LFS file system now, so I can't test it any time soon. I suggest closing
the PR if nobody else reports seeing the problem.
Thanks for taking a look at this.
State-Changed-From-To: feedback->closed
State-Changed-By: tls
State-Changed-When: Wed Mar 31 16:03:51 UTC 2004
State-Changed-Why:
Submitter no longer has ability to reproduce; suggests closing PR.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.