NetBSD Problem Report #8863
Received: (qmail 833 invoked from network); 23 Nov 1999 20:28:12 -0000
Message-Id: <199911232022.VAA00439@rochebonne.antioche.eu.org>
Date: Tue, 23 Nov 1999 21:22:36 +0100 (MET)
From: Manuel Bouyer <bouyer@antioche.lip6.fr>
Reply-To: bouyer@antioche.lip6.fr
To: gnats-bugs@gnats.netbsd.org
Subject: filesystem lookup under heavy LFS write
X-Send-Pr-Version: 3.95
>Number: 8863
>Category: kern
>Synopsis: filesystem lookup under heavy LFS write
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: perseant
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Tue Nov 23 12:30:00 +0000 1999
>Closed-Date: Fri Apr 07 06:21:58 +0000 2000
>Last-Modified: Fri Apr 07 06:21:58 +0000 2000
>Originator: Manuel Bouyer
>Release: -current as of yesterday
>Organization:
none
>Environment:
System: NetBSD rochebonne.antioche.eu.org 1.4P NetBSD 1.4P (ROCHEBONNE) #0: Mon Nov 22 20:33:51 MET 1999 bouyer@rochebonne.antioche.eu.org:/usr/src/1.4M/src/sys/arch/i386/compile/ROCHEBONNE i386
rochebonne#/home/bouyer>mount
/dev/wd0a on / type ffs (NFS exported, local)
/dev/wd0e on /usr type ffs (NFS exported, local)
/dev/sd0g on /home type ffs (NFS exported, local)
/dev/wd0g on /usr/src type lfs (local)
mfs:161 on /tmp type mfs (asynchronous, local)
pid192@rochebonne:/auto on /auto type nfs
/dev/sd1d on /magn type ffs (asynchronous, NFS exported, local, read-only)
This system is configured with a large buffer cache:
total memory = 65148 KB
avail memory = 41364 KB
using 2822 buffers containing 20000 KB of memory
kernel compiled without SOFTDEP.
wd0 is a realy fast HD (> 22MB/s on the raw device), the CPU (Cyrix 133Mhz)
has difficulties keeping up.
>Description:
After a frech newfs_lfs of /usr/src (without any special options)
I was running 'zcat /magn/1.4P.tgz | tar xf -' in /usr/src (1.4P.tgz
being the sources tarballs of today's -current). 'systat vm' shows
that the CPU is busy (less than 1% idle), and spending more than
50% in the kernel. wd0 gets about 2MB/s, disk is clearly not the
bottleneck. 'top' show zcat, tar and lfs_cleanerd being the processes
eating CPU (lfs_cleanerd for about 1%, but constantly running).
after a while all activity stopped, any access to the filesystem
hangs the process. I've been able to enter the debugger and get a
core dump. Following are infos I've been able to get from it.
rochebonne#/var/crash>ps -M netbsd.3.core -axl
UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TT TIME COMMAND
0 0 173 0 -18 0 0 0 schedu DLs ?? 0:00.01 (swapper)
0 1 173 0 10 0 236 0 wait Is ?? 0:00.03 (init)
0 2 173 0 -18 0 0 0 daemon DL ?? 0:00.01 (pagedaemon)
0 3 173 0 -18 0 0 0 reaper DL ?? 0:00.03 (reaper)
0 4 173 0 -5 0 0 0 lfs_di DL ?? 0:00.10 (ioflush)
0 150 173 0 2 0 92 0 - Rs ?? 0:00.10 (syslogd)
0 154 173 25 2 0 60 0 select Is ?? 0:00.05 (portmap)
0 161 173 0 10 0 20936 0 mfsidl Is ?? 0:00.03 (mount_mfs)
0 165 173 0 2 0 116 0 select Is ?? 0:00.17 (mountd)
0 167 173 18 2 0 40 0 netcon Is ?? 0:00.02 (nfsd)
0 171 173 25 10 0 16 0 nfsidl IL ?? 0:00.00 (nfsiod)
0 172 173 25 10 0 16 0 nfsidl IL ?? 0:00.00 (nfsiod)
0 173 173 25 10 0 16 0 nfsidl IL ?? 0:00.00 (nfsiod)
0 174 173 25 10 0 16 0 nfsidl IL ?? 0:00.00 (nfsiod)
0 177 173 25 2 0 20 0 nfsd IL ?? 0:00.01 (nfsd)
0 178 173 25 2 0 20 0 nfsd IL ?? 0:00.01 (nfsd)
0 179 173 25 2 0 20 0 nfsd IL ?? 0:00.01 (nfsd)
0 181 173 25 2 0 20 0 nfsd IL ?? 0:00.00 (nfsd)
0 192 173 0 2 0 260 0 select Ss ?? 0:00.02 (amd)
0 219 173 0 -14 0 220 0 ufs_ha Ds ?? 0:00.02 (cron)
0 222 173 25 2 0 212 0 select Is ?? 0:00.01 (dhcpd)
0 225 173 25 2 0 64 0 select Is ?? 0:00.01 (inetd)
0 231 173 19 2 0 240 0 select Is ?? 0:02.89 (sshd1)
0 257 173 0 10 0 44 0 wait Is ?? 0:00.00 (lfs_cleanerd
0 258 173 0 28 0 576 0 segmen I ?? 0:01.73 (lfs_cleanerd
1210 233 173 0 18 0 524 0 pause Is E0 0:00.22 (tcsh)
1210 266 173 20 2 0 500 0 netio I+ E0 0:47.33 (zcat)
1210 267 173 27 -5 0 588 0 lfs wr D+ E0 1:00.80 (tar)
0 234 173 2 18 0 328 0 pause Is E1 0:00.12 (csh)
0 245 173 0 18 0 548 0 pause I E1 0:00.16 (tcsh)
0 265 173 0 3 0 292 0 ttyin S+ E1 0:00.46 (systat)
0 235 173 0 18 0 328 0 pause Is E2 0:00.12 (csh)
0 270 173 0 2 0 204 0 select S+ E2 0:00.56 (top)
0 236 173 17 3 0 48 0 ttyin Is+ E3 0:00.03 (getty)
0 237 173 17 3 0 48 0 ttyin Is+ E4 0:00.03 (getty)
0 238 173 17 3 0 48 0 ttyin Is+ E5 0:00.03 (getty)
0 239 173 17 3 0 48 0 ttyin Is+ E6 0:00.03 (getty)
Note that tar, lfs_cleanerd and ioflush are all waiting in lfs code.
Other hung processes seems to wait on ufs_halock (verified in another case
I reproduced).
stack trace for lfs_cleanerd (pid 258):
(gdb) proc 0xfdf78c80
(gdb) where
#0 mi_switch () at ../../../../kern/kern_synch.c:748
#1 0xf0136495 in bpendtsleep () at ../../../../kern/kern_synch.c:415
#2 0xf01d9582 in sys_lfs_segwait (p=0xfdf78c80, v=0xfe062f88,
retval=0xfe062f80) at ../../../../ufs/lfs/lfs_syscalls.c:862
#3 0xf01fa86e in syscall (frame={tf_es = 134479903, tf_ds = -272695265,
tf_edi = -272638948, tf_esi = -272638956, tf_ebp = -272638936,
tf_ebx = 134549504, tf_edx = 1835, tf_ecx = 3, tf_eax = 187,
tf_trapno = 3, tf_err = 2, tf_eip = 1208583703, tf_cs = 23,
tf_eflags = 663, tf_esp = -272638996, tf_ss = 31, tf_vm86_es = 0,
tf_vm86_ds = 0, tf_vm86_fs = 0, tf_vm86_gs = 0})
at ../../../../arch/i386/i386/trap.c:760
#4 0xf0100d7d in syscall1 ()
can not access 0xefbfdc28, invalid translation (invalid PDE)
can not access 0xefbfdc28, invalid translation (invalid PDE)
Cannot access memory at address 0xefbfdc28.
stack trace for tar (pid 267):
#0 mi_switch () at ../../../../kern/kern_synch.c:748
#1 0xf0136495 in bpendtsleep () at ../../../../kern/kern_synch.c:415
#2 0xf01d64ff in lfs_segwrite (mp=0xf04d4000, flags=5)
at ../../../../ufs/lfs/lfs_segment.c:501
#3 0xf01d611c in lfs_vflush (vp=0xfe1dd700)
at ../../../../ufs/lfs/lfs_segment.c:253
#4 0xf01d4ae3 in lfs_update (v=0xfe0beac4)
at ../../../../ufs/lfs/lfs_inode.c:178
#5 0xf01db0d3 in lfs_fsync (v=0xfe0beb04) at ../../../../sys/vnode_if.h:1027
#6 0xf01542e7 in vinvalbuf (vp=0xfe1dd700, flags=1, cred=0xffffffff,
p=0xfe096c88, slpflag=0, slptimeo=0) at ../../../../sys/vnode_if.h:449
#7 0xf0154ea5 in vclean (vp=0xfe1dd700, flags=8, p=0xfe096c88)
at ../../../../kern/vfs_subr.c:1349
#8 0xf0155083 in vgonel (vp=0xfe1dd700, p=0xfe096c88)
at ../../../../kern/vfs_subr.c:1479
#9 0xf0154026 in getnewvnode (tag=VT_LFS, mp=0xf04d4000, vops=0xf0451c00,
vpp=0xfe0bec30) at ../../../../kern/vfs_subr.c:503
#10 0xf01d367e in lfs_vcreate (mp=0xf04d4000, ino=33107, vpp=0xfe0bec30)
at ../../../../ufs/lfs/lfs_alloc.c:255
#11 0xf01d357b in lfs_valloc (v=0xfe0bec90)
at ../../../../ufs/lfs/lfs_alloc.c:201
#12 0xf01e1792 in ufs_makeinode (mode=33188, dvp=0xfe1db09c, vpp=0xfe0beee8,
cnp=0xfe0beefc) at ../../../../sys/vnode_if.h:933
#13 0xf01deb7a in ufs_create (v=0xfe0bee08)
at ../../../../ufs/ufs/ufs_vnops.c:118
#14 0xf01db462 in lfs_create (v=0xfe0bee08)
at ../../../../ufs/lfs/lfs_vnops.c:449
#15 0xf015ae28 in vn_open (ndp=0xfe0beed8, fmode=1550, cmode=420)
at ../../../../sys/vnode_if.h:96
#16 0xf01573a0 in sys_open (p=0xfe096c88, v=0xfe0bef88, retval=0xfe0bef80)
at ../../../../kern/vfs_syscalls.c:999
#17 0xf01fa86e in syscall (frame={tf_es = 31, tf_ds = 31, tf_edi = 0,
tf_esi = 134813696, tf_ebp = -272639140, tf_ebx = 1549, tf_edx = -1,
tf_ecx = 2, tf_eax = 5, tf_trapno = 3, tf_err = 2, tf_eip = 134684535,
tf_cs = 23, tf_eflags = 514, tf_esp = -272639504, tf_ss = 31,
tf_vm86_es = 0, tf_vm86_ds = 0, tf_vm86_fs = 0, tf_vm86_gs = 0})
at ../../../../arch/i386/i386/trap.c:760
#18 0xf0100d7d in syscall1 ()
can not access 0xefbfdb5c, invalid translation (invalid PDE)
can not access 0xefbfdb5c, invalid translation (invalid PDE)
Cannot access memory at address 0xefbfdb5c.
stack trace for ioflush (pid 4):
#0 mi_switch () at ../../../../kern/kern_synch.c:748
#1 0xf0136495 in bpendtsleep () at ../../../../kern/kern_synch.c:415
#2 0xf01da5ab in lfs_sync (mp=0xf04d4000, waitfor=3, cred=0xf044cf80,
p=0xfdf783c0) at ../../../../ufs/lfs/lfs_vfsops.c:616
#3 0xf015e957 in sync_fsync (v=0xfdf84f80)
at ../../../../miscfs/syncfs/sync_vnops.c:136
#4 0xf015ec61 in sched_sync (v=0x0) at ../../../../sys/vnode_if.h:449
#5 0xf0100327 in proc_trampoline ()
#6 0xf02c6ed8 in ?? ()
>How-To-Repeat:
On my system, untarring a large source tree to a LFS filesystem
will reproduce it reliably, after variable time (but always before
extract is done).
>Fix:
Sounds like a deadlock condition between lfs_sync() and another part of
the LFS code. Unfortunably I can track this down better for now
(doing so would require for me to understand the LFS code first which
is require more time than I can spend on it) but I can test
any patch or provide any additionnal informations.
I can easily reproduce the problem.
>Release-Note:
>Audit-Trail:
From: Konrad Schroder <perseant@hhhh.org>
To: Manuel Bouyer <bouyer@antioche.lip6.fr>
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: kern/8863: filesystem lookup under heavy LFS write
Date: Tue, 23 Nov 1999 13:08:26 -0800 (PST)
I see the problem: lfs_create declares a dirop, then goes on to call
getnewvnode; an lfs vnode has to be flushed, but oops, dirops are active,
so we can't flush anything or we risk corrupting the filesystem. The
other processes are backed up waiting on this one.
(And lfs_{mkdir,mknod,symlink} will also have the same problem.)
It may be possible to flush non-dirop vnodes without corrupting anything,
but I'll have to think more on that to be sure. If not, this can still be
fixed in a couple of ways: Rewrite ufs_makeinode and ufs_mkdir in two
parts, so that the lfs function can declare dirops *after* getnewvnode is
called; or simply inline it into the lfs functions.
I'll try and get one of these three solutions worked into a patch this
week.
Konrad Schroder
perseant@hhhh.org
On Tue, 23 Nov 1999, Manuel Bouyer wrote:
> stack trace for tar (pid 267):
> #0 mi_switch () at ../../../../kern/kern_synch.c:748
> #1 0xf0136495 in bpendtsleep () at ../../../../kern/kern_synch.c:415
> #2 0xf01d64ff in lfs_segwrite (mp=0xf04d4000, flags=5)
> at ../../../../ufs/lfs/lfs_segment.c:501
> #3 0xf01d611c in lfs_vflush (vp=0xfe1dd700)
> at ../../../../ufs/lfs/lfs_segment.c:253
> #4 0xf01d4ae3 in lfs_update (v=0xfe0beac4)
> at ../../../../ufs/lfs/lfs_inode.c:178
> #5 0xf01db0d3 in lfs_fsync (v=0xfe0beb04) at ../../../../sys/vnode_if.h:1027
> #6 0xf01542e7 in vinvalbuf (vp=0xfe1dd700, flags=1, cred=0xffffffff,
> p=0xfe096c88, slpflag=0, slptimeo=0) at ../../../../sys/vnode_if.h:449
> #7 0xf0154ea5 in vclean (vp=0xfe1dd700, flags=8, p=0xfe096c88)
> at ../../../../kern/vfs_subr.c:1349
> #8 0xf0155083 in vgonel (vp=0xfe1dd700, p=0xfe096c88)
> at ../../../../kern/vfs_subr.c:1479
> #9 0xf0154026 in getnewvnode (tag=VT_LFS, mp=0xf04d4000, vops=0xf0451c00,
> vpp=0xfe0bec30) at ../../../../kern/vfs_subr.c:503
> #10 0xf01d367e in lfs_vcreate (mp=0xf04d4000, ino=33107, vpp=0xfe0bec30)
> at ../../../../ufs/lfs/lfs_alloc.c:255
> #11 0xf01d357b in lfs_valloc (v=0xfe0bec90)
> at ../../../../ufs/lfs/lfs_alloc.c:201
> #12 0xf01e1792 in ufs_makeinode (mode=33188, dvp=0xfe1db09c, vpp=0xfe0beee8,
> cnp=0xfe0beefc) at ../../../../sys/vnode_if.h:933
> #13 0xf01deb7a in ufs_create (v=0xfe0bee08)
> at ../../../../ufs/ufs/ufs_vnops.c:118
> #14 0xf01db462 in lfs_create (v=0xfe0bee08)
> at ../../../../ufs/lfs/lfs_vnops.c:449
> #15 0xf015ae28 in vn_open (ndp=0xfe0beed8, fmode=1550, cmode=420)
> at ../../../../sys/vnode_if.h:96
> #16 0xf01573a0 in sys_open (p=0xfe096c88, v=0xfe0bef88, retval=0xfe0bef80)
> at ../../../../kern/vfs_syscalls.c:999
> #17 0xf01fa86e in syscall (frame={tf_es = 31, tf_ds = 31, tf_edi = 0,
> tf_esi = 134813696, tf_ebp = -272639140, tf_ebx = 1549, tf_edx = -1,
> tf_ecx = 2, tf_eax = 5, tf_trapno = 3, tf_err = 2, tf_eip = 134684535,
> tf_cs = 23, tf_eflags = 514, tf_esp = -272639504, tf_ss = 31,
> tf_vm86_es = 0, tf_vm86_ds = 0, tf_vm86_fs = 0, tf_vm86_gs = 0})
> at ../../../../arch/i386/i386/trap.c:760
> #18 0xf0100d7d in syscall1 ()
From: Konrad Schroder <perseant@hhhh.org>
To: Manuel Bouyer <bouyer@antioche.lip6.fr>
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: kern/8863: filesystem lookup under heavy LFS write
Date: Wed, 24 Nov 1999 14:43:46 -0800 (PST)
Please try the enclosed patch, which I think should fix this problem.
Konrad Schroder
perseant@hhhh.org
Index: lfs_segment.c
===================================================================
RCS file: /cvsroot/syssrc/sys/ufs/lfs/lfs_segment.c,v
retrieving revision 1.36
diff -u -r1.36 lfs_segment.c
--- lfs_segment.c 1999/11/17 20:08:30 1.36
+++ lfs_segment.c 1999/11/24 22:37:42
@@ -484,32 +484,30 @@
/*
* If lfs_flushvp is non-NULL, we are called from lfs_vflush,
* in which case we have to flush *all* buffers off of this vnode.
+ * We don't care about other nodes, but write any non-dirop nodes
+ * anyway in anticipation of another getnewvnode().
+ *
+ * If we're cleaning we only write cleaning and ifile blocks, and
+ * no dirops, since otherwise we'd risk corruption in a crash.
*/
- if((sp->seg_flags & SEGM_CLEAN) && !(fs->lfs_flushvp))
+ if(fs->lfs_flushvp)
+ lfs_writevnodes(fs, mp, sp, VN_REG);
+ else if(sp->seg_flags & SEGM_CLEAN)
lfs_writevnodes(fs, mp, sp, VN_CLEAN);
else {
lfs_writevnodes(fs, mp, sp, VN_REG);
- /*
- * XXX KS - If we're cleaning, we can't wait for dirops,
- * because they might be waiting on us. The downside of this
- * is that, if we write anything besides cleaning blocks
- * while cleaning, the checkpoint is not completely
- * consistent.
- */
- if(!(sp->seg_flags & SEGM_CLEAN)) {
- while(fs->lfs_dirops)
- if((error = tsleep(&fs->lfs_writer, PRIBIO + 1,
- "lfs writer", 0)))
- {
- free(sp->bpp, M_SEGMENT);
- free(sp, M_SEGMENT);
- return (error);
- }
- fs->lfs_writer++;
- writer_set=1;
- lfs_writevnodes(fs, mp, sp, VN_DIROP);
- ((SEGSUM *)(sp->segsum))->ss_flags &= ~(SS_CONT);
- }
+ while(fs->lfs_dirops)
+ if((error = tsleep(&fs->lfs_writer, PRIBIO + 1,
+ "lfs writer", 0)))
+ {
+ free(sp->bpp, M_SEGMENT);
+ free(sp, M_SEGMENT);
+ return (error);
+ }
+ fs->lfs_writer++;
+ writer_set=1;
+ lfs_writevnodes(fs, mp, sp, VN_DIROP);
+ ((SEGSUM *)(sp->segsum))->ss_flags &= ~(SS_CONT);
}
/*
From: Manuel Bouyer <bouyer@antioche.lip6.fr>
To: Konrad Schroder <perseant@hhhh.org>
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: kern/8863: filesystem lookup under heavy LFS write
Date: Thu, 25 Nov 1999 21:41:04 +0100
On Wed, Nov 24, 1999 at 02:43:46PM -0800, Konrad Schroder wrote:
> Please try the enclosed patch, which I think should fix this problem.
I seems to, but I have now another one. tar hang, as well as any access to
the LFS but access to other filesystems still works (this is with fresh sources,
updated from cvs.netbsd.org today, at about 17H GMT).
I've got a core dump, here's the ps output:
rochebonne#/var/crash>ps -axl -M netbsd.6.core
UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TT TIME COMMAND
0 0 173 0 -18 0 0 0 schedu DLs ?? 0:00.01 (swapper)
0 1 173 0 10 0 236 0 wait Is ?? 0:00.04 (init)
0 2 173 0 -18 0 0 0 daemon DL ?? 0:00.08 (pagedaemon)
0 3 173 0 -18 0 0 0 reaper DL ?? 0:00.03 (reaper)
0 4 173 0 18 0 0 0 syncer DL ?? 0:00.97 (ioflush)
0 150 173 0 2 0 92 0 - Rs ?? 0:00.10 (syslogd)
0 154 173 24 2 0 60 0 select IWs ?? 0:00.00 (portmap)
0 161 173 0 10 0 20936 0 mfsidl Is ?? 0:00.04 (mount_mfs)
0 165 173 0 2 0 116 0 select Is ?? 0:00.18 (mountd)
0 167 173 24 2 0 40 0 netcon IWs ?? 0:00.00 (nfsd)
0 171 173 24 10 0 16 0 nfsidl IL ?? 0:00.00 (nfsiod)
0 172 173 24 10 0 16 0 nfsidl IL ?? 0:00.00 (nfsiod)
0 173 173 24 10 0 16 0 nfsidl IL ?? 0:00.00 (nfsiod)
0 174 173 24 10 0 16 0 nfsidl IL ?? 0:00.00 (nfsiod)
0 177 173 24 2 0 20 0 nfsd IL ?? 0:00.01 (nfsd)
0 178 173 24 2 0 20 0 nfsd IL ?? 0:00.01 (nfsd)
0 179 173 24 2 0 20 0 nfsd IL ?? 0:00.00 (nfsd)
0 180 173 24 2 0 20 0 nfsd IL ?? 0:00.00 (nfsd)
0 192 173 0 2 0 260 0 select Is ?? 0:00.03 (amd)
0 219 173 0 10 0 220 0 nanosl Ss ?? 0:00.03 (cron)
0 222 173 16 2 0 212 0 select IWs ?? 0:00.00 (dhcpd)
0 225 173 24 2 0 64 0 select IWs ?? 0:00.00 (inetd)
0 231 173 24 2 0 240 0 select IWs ?? 0:00.00 (sshd1)
0 265 173 0 2 0 3988 0 select S ?? 0:14.99 (XF86_SVGA)
0 365 173 2 10 0 44 0 wait Is ?? 0:00.00 (lfs_cleanerd
0 366 173 1 29 0 128 0 segmen I ?? 0:00.01 (lfs_cleanerd
1210 287 173 2 3 0 532 0 ttyin Is+ p0 0:00.13 (tcsh)
1210 288 173 8 3 0 524 0 ttyin IWs+ p1 0:00.00 (tcsh)
1210 294 173 10 3 0 524 0 ttyin IWs+ p2 0:00.00 (tcsh)
1210 283 173 7 3 0 524 0 ttyin IWs+ p3 0:00.00 (tcsh)
1210 282 173 6 3 0 524 0 ttyin IWs+ p4 0:00.00 (tcsh)
0 290 173 0 18 0 512 0 pause Is p5 0:00.19 (tcsh)
0 355 173 0 2 0 220 0 select S+ p5 0:00.68 (top)
1210 296 173 13 3 0 524 0 ttyin IWs+ p6 0:00.00 (tcsh)
0 313 173 0 18 0 552 0 pause Is p7 0:00.29 (tcsh)
0 354 173 0 3 0 296 0 ttyin S+ p7 0:00.52 (systat)
0 326 173 2 18 0 552 0 pause Is p8 0:00.24 (tcsh)
0 367 173 6 2 0 496 0 netio I+ p8 0:00.33 (zcat)
0 368 173 6 34 0 340 0 lfs_ma I+ p8 0:00.30 (tar)
1210 254 173 4 18 0 524 0 pause IWs E0 0:00.00 (tcsh)
1210 259 173 18 10 0 364 0 wait IW+ E0 0:00.00 (sh)
1210 264 173 4 10 0 68 0 wait IW+ E0 0:00.00 (xinit)
0 268 173 0 2 0 524 0 - R E0 0:00.28 (xterm)
1210 270 173 0 2 0 236 0 select I E0 0:00.42 (fvwm)
0 271 173 0 2 0 536 0 select I E0 0:01.29 (xterm)
1210 272 173 0 2 0 156 0 select S E0 0:00.29 (xload)
1210 273 173 0 2 0 124 0 select S E0 0:00.16 (oclock)
0 276 173 1 2 0 532 0 select IW E0 0:00.00 (xterm)
0 277 173 1 2 0 532 0 select IW E0 0:00.00 (xterm)
0 278 173 1 2 0 532 0 select IW E0 0:00.00 (xterm)
0 279 173 1 2 0 532 0 select IW E0 0:00.00 (xterm)
0 280 173 0 2 0 532 0 select S E0 0:00.72 (xterm)
0 312 173 0 2 0 532 0 select S E0 0:00.56 (xterm)
0 325 173 0 2 0 532 0 select S E0 0:00.31 (xterm)
0 234 173 16 3 0 48 0 ttyin IWs+ E1 0:00.00 (getty)
0 235 173 16 3 0 48 0 ttyin IWs+ E2 0:00.00 (getty)
0 236 173 16 3 0 48 0 ttyin IWs+ E3 0:00.00 (getty)
0 237 173 16 3 0 48 0 ttyin IWs+ E4 0:00.00 (getty)
0 238 173 16 3 0 48 0 ttyin IWs+ E5 0:00.00 (getty)
0 239 173 16 3 0 48 0 ttyin IWs+ E6 0:00.00 (getty)
Stack trace for 'tar':
(gdb) where
#0 mi_switch () at ../../../../kern/kern_synch.c:748
#1 0xf0136495 in bpendtsleep () at ../../../../kern/kern_synch.c:415
#2 0xf01db2b5 in lfs_set_dirop (fs=0xf04fdc00)
at ../../../../ufs/lfs/lfs_vnops.c:311
#3 0xf01db59d in lfs_create (v=0xfed79e08)
at ../../../../ufs/lfs/lfs_vnops.c:446
#4 0xf015aee8 in vn_open (ndp=0xfed79ed8, fmode=1550, cmode=420)
at ../../../../sys/vnode_if.h:96
#5 0xf0157460 in sys_open (p=0xfe0f8dd0, v=0xfed79f88, retval=0xfed79f80)
at ../../../../kern/vfs_syscalls.c:999
#6 0xf01fa9fa in syscall (frame={tf_es = 31, tf_ds = 31, tf_edi = 0,
tf_esi = 134811648, tf_ebp = -272640156, tf_ebx = 1549, tf_edx = -1,
tf_ecx = 2, tf_eax = 5, tf_trapno = 3, tf_err = 2, tf_eip = 134684535,
tf_cs = 23, tf_eflags = 514, tf_esp = -272640520, tf_ss = 31,
tf_vm86_es = 0, tf_vm86_ds = 0, tf_vm86_fs = 0, tf_vm86_gs = 0})
at ../../../../arch/i386/i386/trap.c:760
#7 0xf0100d7d in syscall1 ()
can not access 0xefbfd764, invalid translation (invalid PDE)
can not access 0xefbfd764, invalid translation (invalid PDE)
Cannot access memory at address 0xefbfd764.
stack trace for 'lfs_cleanerd' (pid 366):
(gdb) where
#0 mi_switch () at ../../../../kern/kern_synch.c:748
#1 0xf0136495 in bpendtsleep () at ../../../../kern/kern_synch.c:415
#2 0xf01d96f2 in sys_lfs_segwait (p=0xfdf78c80, v=0xfe0d4f88,
retval=0xfe0d4f80) at ../../../../ufs/lfs/lfs_syscalls.c:864
#3 0xf01fa9fa in syscall (frame={tf_es = -272695265, tf_ds = 31,
tf_edi = -272640140, tf_esi = -272640148, tf_ebp = -272640128,
tf_ebx = 134549504, tf_edx = 1835, tf_ecx = 3, tf_eax = 187,
tf_trapno = 3, tf_err = 2, tf_eip = 1208583703, tf_cs = 23,
tf_eflags = 663, tf_esp = -272640188, tf_ss = 31, tf_vm86_es = 0,
tf_vm86_ds = 0, tf_vm86_fs = 0, tf_vm86_gs = 0})
at ../../../../arch/i386/i386/trap.c:760
#4 0xf0100d7d in syscall1 ()
can not access 0xefbfd780, invalid translation (invalid PDE)
can not access 0xefbfd780, invalid translation (invalid PDE)
Cannot access memory at address 0xefbfd780.
The other lfs_cleanerd proc is waiting in sys_wait4(), nothing interesting
here :)
When in this situation, an attempt to 'reboot' (I got the core dump from
ddb with 'call cpu_reboot 0x104', i.e. no sync and no umount) causes a panic:
"Panic: bawrite LFS buffer" when the kernel umounts the LFS.
A userland umount will just return EBUSY.
Also I still get a panic when running a 'rm -rf' on a LFS filesystem
(even a fresh one, just populated with kernel sources):
panic: pool_get(lfsinopl): free list modified: magic=0; page 0xfe0cc000; item addr 0xfe0cc850
syncing disks... 5 3 done
I have a crash dump, here's the output of 'ps -alx':
rochebonne#/var/crash>ps -axl -M netbsd.5.core
UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TT TIME COMMAND
0 0 173 0 -18 0 0 0 schedu DLs ?? 0:00.00 (swapper)
0 1 173 1 10 0 236 0 wait Is ?? 0:00.03 (init)
0 2 173 0 -18 0 0 0 daemon DL ?? 0:00.00 (pagedaemon)
0 3 173 0 -18 0 0 0 reaper DL ?? 0:00.02 (reaper)
0 4 173 0 18 0 0 0 syncer DL ?? 0:00.09 (ioflush)
0 150 173 0 2 0 92 0 select Ss ?? 0:00.08 (syslogd)
0 154 173 18 2 0 60 0 select Is ?? 0:00.05 (portmap)
0 158 173 15 10 0 44 0 wait Is ?? 0:00.00 (lfs_cleanerd
0 160 173 0 28 0 320 0 segmen I ?? 0:00.02 (lfs_cleanerd
0 161 173 0 10 0 20936 0 mfsidl Ss ?? 0:00.03 (mount_mfs)
0 165 173 15 2 0 100 0 select Is ?? 0:00.01 (mountd)
0 167 173 15 2 0 40 0 netcon Is ?? 0:00.02 (nfsd)
0 169 173 15 2 0 20 0 nfsd IL ?? 0:00.01 (nfsd)
0 170 173 15 2 0 20 0 nfsd IL ?? 0:00.01 (nfsd)
0 171 173 15 2 0 20 0 nfsd IL ?? 0:00.01 (nfsd)
0 172 173 15 2 0 20 0 nfsd IL ?? 0:00.01 (nfsd)
0 177 173 15 10 0 16 0 nfsidl IL ?? 0:00.00 (nfsiod)
0 178 173 15 10 0 16 0 nfsidl IL ?? 0:00.00 (nfsiod)
0 179 173 15 10 0 16 0 nfsidl IL ?? 0:00.00 (nfsiod)
0 180 173 15 10 0 16 0 nfsidl IL ?? 0:00.00 (nfsiod)
0 192 173 0 2 0 260 0 select Is ?? 0:00.02 (amd)
0 220 173 0 10 0 220 0 nanosl Ss ?? 0:00.02 (cron)
0 223 173 18 2 0 212 0 select Is ?? 0:00.01 (dhcpd)
0 226 173 18 2 0 64 0 select Is ?? 0:00.01 (inetd)
0 232 173 9 2 0 240 0 select Is ?? 0:05.81 (sshd1)
1210 234 173 0 18 0 524 0 pause Ss E0 0:00.22 (tcsh)
1210 274 173 5 33 0 236 0 - R+ E0 0:00.11 (rm)
0 235 173 9 3 0 48 0 ttyin Is+ E1 0:00.03 (getty)
0 236 173 9 3 0 48 0 ttyin Is+ E2 0:00.03 (getty)
0 237 173 9 3 0 48 0 ttyin Is+ E3 0:00.03 (getty)
0 238 173 9 3 0 48 0 ttyin Is+ E4 0:00.03 (getty)
0 239 173 9 3 0 48 0 ttyin Is+ E5 0:00.03 (getty)
0 240 173 9 3 0 48 0 ttyin Is+ E6 0:00.03 (getty)
The stack trace look like:
#0 cpu_reboot (howto=11, bootstr=0xfe0b2af8 " +\013þø©\020ðD+\013þ")
at ../../../../arch/i386/i386/machdep.c:1192
#1 0xf01f4647 in cpu_reboot (howto=256, bootstr=0x0)
at ../../../../arch/i386/i386/machdep.c:1215
#2 0xf010ac71 in db_fncall (addr=-32822460, have_addr=0, count=0,
modif=0xfe0b2b28 " ð&ð\001") at ../../../../ddb/db_command.c:561
#3 0xf010a9f8 in db_command (last_cmdp=0xf025d878, cmd_table=0xf025d6b8)
at ../../../../ddb/db_command.c:295
#4 0xf010ab52 in db_command_loop () at ../../../../ddb/db_command.c:486
#5 0xf010d212 in db_trap (type=1, code=0) at ../../../../ddb/db_trap.c:78
#6 0xf01f2570 in kdb_trap (type=1, code=0, regs=0xfe0b2c54)
at ../../../../arch/i386/i386/db_interface.c:120
#7 0xf01fa338 in trap (frame={tf_es = 1667432464, tf_ds = -32833520,
tf_edi = -266090624, tf_esi = 256, tf_ebp = -32822132,
tf_ebx = -32822100, tf_edx = 981, tf_ecx = 768, tf_eax = 0,
tf_trapno = 1, tf_err = 0, tf_eip = -266394068, tf_cs = -266141688,
tf_eflags = 514, tf_esp = -32822112, tf_ss = -267139359,
tf_vm86_es = -32714792, tf_vm86_ds = -265718368, tf_vm86_fs = -32716720,
tf_vm86_gs = -32822044}) at ../../../../arch/i386/i386/trap.c:298
#8 0xf0100cc5 in calltrap ()
#9 0xf013c6e1 in panic (
fmt=0xf023c780 "pool_get(%s): free list modified: magic=%x; page %p; item addr %p\n") at ../../../../kern/subr_prf.c:215
#10 0xf013b708 in _pool_get (pp=0xf02975a0, flags=2,
file=0xf0243f20 "../../../../ufs/lfs/lfs_alloc.c", line=264)
at ../../../../kern/subr_pool.c:745
#11 0xf01d3761 in lfs_vcreate (mp=0xf04a9400, ino=282, vpp=0xfe0b2d34)
at ../../../../ufs/lfs/lfs_alloc.c:264
#12 0xf01da870 in lfs_vget (mp=0xf04a9400, ino=282, vpp=0xfe0b2dcc)
at ../../../../ufs/lfs/lfs_vfsops.c:681
#13 0xf01dda72 in ufs_lookup (v=0xfe0b2e3c)
at ../../../../ufs/ufs/ufs_lookup.c:494
#14 0xf0153623 in lookup (ndp=0xfe0b2ef0) at ../../../../sys/vnode_if.h:71
#15 0xf0153317 in namei (ndp=0xfe0b2ef0) at ../../../../kern/vfs_lookup.c:153
#16 0xf01588a1 in sys_unlink (p=0xfe0998c8, v=0xfe0b2f88, retval=0xfe0b2f80)
at ../../../../kern/vfs_syscalls.c:1560
#17 0xf01fa9fa in syscall (frame={tf_es = 31, tf_ds = 31, tf_edi = 0,
tf_esi = 134713344, tf_ebp = -272639128, tf_ebx = 134736512,
tf_edx = 134734208, tf_ecx = 0, tf_eax = 10, tf_trapno = 3, tf_err = 2,
tf_eip = 134606939, tf_cs = 23, tf_eflags = 582, tf_esp = -272639148,
tf_ss = 31, tf_vm86_es = 0, tf_vm86_ds = 0, tf_vm86_fs = 0,
tf_vm86_gs = 0}) at ../../../../arch/i386/i386/trap.c:760
#18 0xf0100d7d in syscall1 ()
can not access 0xefbfdb68, invalid translation (invalid PDE)
can not access 0xefbfdb68, invalid translation (invalid PDE)
Cannot access memory at address 0xefbfdb68.
I'm not sure if this is related to the previous problem or not.
Let me know if you need trace of other processes (but I'm not sure we'll
have something interesting here), or other infos.
--
Manuel Bouyer <bouyer@antioche.eu.org>
--
From: Konrad Schroder <perseant@hhhh.org>
To: Manuel Bouyer <bouyer@antioche.lip6.fr>
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: kern/8863: filesystem lookup under heavy LFS write
Date: Mon, 29 Nov 1999 11:17:12 -0800 (PST)
On Thu, 25 Nov 1999, Manuel Bouyer wrote:
> I seems to, but I have now another one. tar hang, as well as any access to
> the LFS but access to other filesystems still works (this is with fresh sources,
> updated from cvs.netbsd.org today, at about 17H GMT).
> 0 368 173 6 34 0 340 0 lfs_ma I+ p8 0:00.30 (tar)
Here's a new patch, which seems to fix the problem here.
(OTOH I still have no idea on the pool panic problem.)
Konrad Schroder
perseant@hhhh.org
Index: lfs_vnops.c
===================================================================
RCS file: /cvsroot/syssrc/sys/ufs/lfs/lfs_vnops.c,v
retrieving revision 1.32
diff -u -r1.32 lfs_vnops.c
--- lfs_vnops.c 1999/11/15 18:49:14 1.32
+++ lfs_vnops.c 1999/11/29 19:14:44
@@ -304,9 +304,16 @@
while (fs->lfs_writer || lfs_dirvcount>LFS_MAXDIROP) {
if(fs->lfs_writer)
tsleep(&fs->lfs_dirops, PRIBIO + 1, "lfs_dirop", 0);
+ if(lfs_dirvcount > LFS_MAXDIROP) {
+ lfs_flush(fs,0);
+ /* We might be busying the fs, flush it explicitly */
+ lfs_flush_fs(fs->lfs_ivnode->v_mount,0);
+ }
+
+ /* If it's still too high, e.g. other dirops are active, sleep */
if(lfs_dirvcount > LFS_MAXDIROP) {
#ifdef DEBUG_LFS
- printf("(dirvcount=%d)\n",lfs_dirvcount);
+ printf("lfs_set_dirop: sleeping with dirops=%d, dirvcount=%d\n",fs->lfs_dirops,lfs_dirvcount);
#endif
if((error=tsleep(&lfs_dirvcount, PCATCH|PUSER, "lfs_maxdirop", 0))!=0)
return error;
From: Manuel Bouyer <bouyer@antioche.lip6.fr>
To: Konrad Schroder <perseant@hhhh.org>
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: kern/8863: filesystem lookup under heavy LFS write
Date: Wed, 1 Dec 1999 10:49:31 +0100
On Mon, Nov 29, 1999 at 11:17:12AM -0800, Konrad Schroder wrote:
> Here's a new patch, which seems to fix the problem here.
Ok, with this one + the one you sent me previsoulsy I've been able to
extract a whole src tree several times without problems.
> (OTOH I still have no idea on the pool panic problem.)
I looked a bit at this. The pool magic number has been set to 0, which
in the inode structure correspond to the le_next member of i_hash.
This one is set to 0 in ufs_ihashrem().
I'll try to add more code to confirm this theory and detect the corruption
earlier. Maybe this way I'll get a usefull stack trace.
State-Changed-From-To: open->feedback
State-Changed-By: perseant
State-Changed-When: Sat Jan 15 15:27:13 PST 2000
State-Changed-Why:
Problem solved, patch committed some time ago
Responsible-Changed-From-To: kern-bug-people->perseant
Responsible-Changed-By: perseant
Responsible-Changed-When: Sat Jan 15 15:27:13 PST 2000
Responsible-Changed-Why:
I have responsibility for LFS-related problems.
State-Changed-From-To: feedback->closed
State-Changed-By: bouyer
State-Changed-When: Thu Apr 6 23:21:11 PDT 2000
State-Changed-Why:
Solved, I haven't got this problem for months now.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.