NetBSD Problem Report #8863

Received: (qmail 833 invoked from network); 23 Nov 1999 20:28:12 -0000
Message-Id: <199911232022.VAA00439@rochebonne.antioche.eu.org>
Date: Tue, 23 Nov 1999 21:22:36 +0100 (MET)
From: Manuel Bouyer <bouyer@antioche.lip6.fr>
Reply-To: bouyer@antioche.lip6.fr
To: gnats-bugs@gnats.netbsd.org
Subject: filesystem lookup under heavy LFS write
X-Send-Pr-Version: 3.95

>Number:         8863
>Category:       kern
>Synopsis:       filesystem lookup under heavy LFS write
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    perseant
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Nov 23 12:30:00 +0000 1999
>Closed-Date:    Fri Apr 07 06:21:58 +0000 2000
>Last-Modified:  Fri Apr 07 06:21:58 +0000 2000
>Originator:     Manuel Bouyer
>Release:        -current as of yesterday
>Organization:

none

>Environment:

System: NetBSD rochebonne.antioche.eu.org 1.4P NetBSD 1.4P (ROCHEBONNE) #0: Mon Nov 22 20:33:51 MET 1999 bouyer@rochebonne.antioche.eu.org:/usr/src/1.4M/src/sys/arch/i386/compile/ROCHEBONNE i386

rochebonne#/home/bouyer>mount
/dev/wd0a on / type ffs (NFS exported, local)
/dev/wd0e on /usr type ffs (NFS exported, local)
/dev/sd0g on /home type ffs (NFS exported, local)
/dev/wd0g on /usr/src type lfs (local)
mfs:161 on /tmp type mfs (asynchronous, local)
pid192@rochebonne:/auto on /auto type nfs
/dev/sd1d on /magn type ffs (asynchronous, NFS exported, local, read-only)

This system is configured with a large buffer cache:
total memory = 65148 KB
avail memory = 41364 KB
using 2822 buffers containing 20000 KB of memory
kernel compiled without SOFTDEP.

wd0 is a realy fast HD (> 22MB/s on the raw device), the CPU (Cyrix 133Mhz)
has difficulties keeping up.

>Description:

	After a frech newfs_lfs of /usr/src (without any special options)
	I was running 'zcat /magn/1.4P.tgz | tar xf -' in /usr/src (1.4P.tgz
	being the sources tarballs of today's -current). 'systat vm' shows
	that the CPU is busy (less than 1% idle), and spending more than
	50% in the kernel. wd0 gets about 2MB/s, disk is clearly not the
	bottleneck. 'top' show zcat, tar and lfs_cleanerd being the processes
	eating CPU (lfs_cleanerd for about 1%, but constantly running).

	after a while all activity stopped, any access to the filesystem
	hangs the process. I've been able to enter the debugger and get a
	core dump. Following are infos I've been able to get from it.

rochebonne#/var/crash>ps -M netbsd.3.core -axl
  UID   PID  PPID CPU PRI NI   VSZ  RSS WCHAN  STAT TT       TIME COMMAND
    0     0   173   0 -18  0     0    0 schedu DLs  ??    0:00.01 (swapper)
    0     1   173   0  10  0   236    0 wait   Is   ??    0:00.03 (init)
    0     2   173   0 -18  0     0    0 daemon DL   ??    0:00.01 (pagedaemon)
    0     3   173   0 -18  0     0    0 reaper DL   ??    0:00.03 (reaper)
    0     4   173   0  -5  0     0    0 lfs_di DL   ??    0:00.10 (ioflush)
    0   150   173   0   2  0    92    0 -      Rs   ??    0:00.10 (syslogd)
    0   154   173  25   2  0    60    0 select Is   ??    0:00.05 (portmap)
    0   161   173   0  10  0 20936    0 mfsidl Is   ??    0:00.03 (mount_mfs)
    0   165   173   0   2  0   116    0 select Is   ??    0:00.17 (mountd)
    0   167   173  18   2  0    40    0 netcon Is   ??    0:00.02 (nfsd)
    0   171   173  25  10  0    16    0 nfsidl IL   ??    0:00.00 (nfsiod)
    0   172   173  25  10  0    16    0 nfsidl IL   ??    0:00.00 (nfsiod)
    0   173   173  25  10  0    16    0 nfsidl IL   ??    0:00.00 (nfsiod)
    0   174   173  25  10  0    16    0 nfsidl IL   ??    0:00.00 (nfsiod)
    0   177   173  25   2  0    20    0 nfsd   IL   ??    0:00.01 (nfsd)
    0   178   173  25   2  0    20    0 nfsd   IL   ??    0:00.01 (nfsd)
    0   179   173  25   2  0    20    0 nfsd   IL   ??    0:00.01 (nfsd)
    0   181   173  25   2  0    20    0 nfsd   IL   ??    0:00.00 (nfsd)
    0   192   173   0   2  0   260    0 select Ss   ??    0:00.02 (amd)
    0   219   173   0 -14  0   220    0 ufs_ha Ds   ??    0:00.02 (cron)
    0   222   173  25   2  0   212    0 select Is   ??    0:00.01 (dhcpd)
    0   225   173  25   2  0    64    0 select Is   ??    0:00.01 (inetd)
    0   231   173  19   2  0   240    0 select Is   ??    0:02.89 (sshd1)
    0   257   173   0  10  0    44    0 wait   Is   ??    0:00.00 (lfs_cleanerd
    0   258   173   0  28  0   576    0 segmen I    ??    0:01.73 (lfs_cleanerd
 1210   233   173   0  18  0   524    0 pause  Is   E0    0:00.22 (tcsh)
 1210   266   173  20   2  0   500    0 netio  I+   E0    0:47.33 (zcat)
 1210   267   173  27  -5  0   588    0 lfs wr D+   E0    1:00.80 (tar)
    0   234   173   2  18  0   328    0 pause  Is   E1    0:00.12 (csh)
    0   245   173   0  18  0   548    0 pause  I    E1    0:00.16 (tcsh)
    0   265   173   0   3  0   292    0 ttyin  S+   E1    0:00.46 (systat)
    0   235   173   0  18  0   328    0 pause  Is   E2    0:00.12 (csh)
    0   270   173   0   2  0   204    0 select S+   E2    0:00.56 (top)
    0   236   173  17   3  0    48    0 ttyin  Is+  E3    0:00.03 (getty)
    0   237   173  17   3  0    48    0 ttyin  Is+  E4    0:00.03 (getty)
    0   238   173  17   3  0    48    0 ttyin  Is+  E5    0:00.03 (getty)
    0   239   173  17   3  0    48    0 ttyin  Is+  E6    0:00.03 (getty)

Note that tar, lfs_cleanerd and ioflush are all waiting in lfs code.
Other hung processes seems to wait on ufs_halock (verified in another case
I reproduced).

stack trace for lfs_cleanerd (pid 258):
(gdb) proc 0xfdf78c80
(gdb) where
#0  mi_switch () at ../../../../kern/kern_synch.c:748
#1  0xf0136495 in bpendtsleep () at ../../../../kern/kern_synch.c:415
#2  0xf01d9582 in sys_lfs_segwait (p=0xfdf78c80, v=0xfe062f88, 
    retval=0xfe062f80) at ../../../../ufs/lfs/lfs_syscalls.c:862
#3  0xf01fa86e in syscall (frame={tf_es = 134479903, tf_ds = -272695265, 
      tf_edi = -272638948, tf_esi = -272638956, tf_ebp = -272638936, 
      tf_ebx = 134549504, tf_edx = 1835, tf_ecx = 3, tf_eax = 187, 
      tf_trapno = 3, tf_err = 2, tf_eip = 1208583703, tf_cs = 23, 
      tf_eflags = 663, tf_esp = -272638996, tf_ss = 31, tf_vm86_es = 0, 
      tf_vm86_ds = 0, tf_vm86_fs = 0, tf_vm86_gs = 0})
    at ../../../../arch/i386/i386/trap.c:760
#4  0xf0100d7d in syscall1 ()
can not access 0xefbfdc28, invalid translation (invalid PDE)
can not access 0xefbfdc28, invalid translation (invalid PDE)
Cannot access memory at address 0xefbfdc28.

stack trace for tar (pid 267):
#0  mi_switch () at ../../../../kern/kern_synch.c:748
#1  0xf0136495 in bpendtsleep () at ../../../../kern/kern_synch.c:415
#2  0xf01d64ff in lfs_segwrite (mp=0xf04d4000, flags=5)
    at ../../../../ufs/lfs/lfs_segment.c:501
#3  0xf01d611c in lfs_vflush (vp=0xfe1dd700)
    at ../../../../ufs/lfs/lfs_segment.c:253
#4  0xf01d4ae3 in lfs_update (v=0xfe0beac4)
    at ../../../../ufs/lfs/lfs_inode.c:178
#5  0xf01db0d3 in lfs_fsync (v=0xfe0beb04) at ../../../../sys/vnode_if.h:1027
#6  0xf01542e7 in vinvalbuf (vp=0xfe1dd700, flags=1, cred=0xffffffff, 
    p=0xfe096c88, slpflag=0, slptimeo=0) at ../../../../sys/vnode_if.h:449
#7  0xf0154ea5 in vclean (vp=0xfe1dd700, flags=8, p=0xfe096c88)
    at ../../../../kern/vfs_subr.c:1349
#8  0xf0155083 in vgonel (vp=0xfe1dd700, p=0xfe096c88)
    at ../../../../kern/vfs_subr.c:1479
#9  0xf0154026 in getnewvnode (tag=VT_LFS, mp=0xf04d4000, vops=0xf0451c00, 
    vpp=0xfe0bec30) at ../../../../kern/vfs_subr.c:503
#10 0xf01d367e in lfs_vcreate (mp=0xf04d4000, ino=33107, vpp=0xfe0bec30)
    at ../../../../ufs/lfs/lfs_alloc.c:255
#11 0xf01d357b in lfs_valloc (v=0xfe0bec90)
    at ../../../../ufs/lfs/lfs_alloc.c:201
#12 0xf01e1792 in ufs_makeinode (mode=33188, dvp=0xfe1db09c, vpp=0xfe0beee8, 
    cnp=0xfe0beefc) at ../../../../sys/vnode_if.h:933
#13 0xf01deb7a in ufs_create (v=0xfe0bee08)
    at ../../../../ufs/ufs/ufs_vnops.c:118
#14 0xf01db462 in lfs_create (v=0xfe0bee08)
    at ../../../../ufs/lfs/lfs_vnops.c:449
#15 0xf015ae28 in vn_open (ndp=0xfe0beed8, fmode=1550, cmode=420)
    at ../../../../sys/vnode_if.h:96
#16 0xf01573a0 in sys_open (p=0xfe096c88, v=0xfe0bef88, retval=0xfe0bef80)
    at ../../../../kern/vfs_syscalls.c:999
#17 0xf01fa86e in syscall (frame={tf_es = 31, tf_ds = 31, tf_edi = 0, 
      tf_esi = 134813696, tf_ebp = -272639140, tf_ebx = 1549, tf_edx = -1, 
      tf_ecx = 2, tf_eax = 5, tf_trapno = 3, tf_err = 2, tf_eip = 134684535, 
      tf_cs = 23, tf_eflags = 514, tf_esp = -272639504, tf_ss = 31, 
      tf_vm86_es = 0, tf_vm86_ds = 0, tf_vm86_fs = 0, tf_vm86_gs = 0})
    at ../../../../arch/i386/i386/trap.c:760
#18 0xf0100d7d in syscall1 ()
can not access 0xefbfdb5c, invalid translation (invalid PDE)
can not access 0xefbfdb5c, invalid translation (invalid PDE)
Cannot access memory at address 0xefbfdb5c.

stack trace for ioflush (pid 4):
#0  mi_switch () at ../../../../kern/kern_synch.c:748
#1  0xf0136495 in bpendtsleep () at ../../../../kern/kern_synch.c:415
#2  0xf01da5ab in lfs_sync (mp=0xf04d4000, waitfor=3, cred=0xf044cf80, 
    p=0xfdf783c0) at ../../../../ufs/lfs/lfs_vfsops.c:616
#3  0xf015e957 in sync_fsync (v=0xfdf84f80)
    at ../../../../miscfs/syncfs/sync_vnops.c:136
#4  0xf015ec61 in sched_sync (v=0x0) at ../../../../sys/vnode_if.h:449
#5  0xf0100327 in proc_trampoline ()
#6  0xf02c6ed8 in ?? ()


>How-To-Repeat:

	On my system, untarring a large source tree to a LFS filesystem
	will reproduce it reliably, after variable time (but always before
	extract is done).

>Fix:
	Sounds like a deadlock condition between lfs_sync() and another part of
	the LFS code. Unfortunably I can track this down better for now
	(doing so would require for me to understand the LFS code first which
	is require more time than I can spend on it) but I can test
	any patch or provide any additionnal informations.
	I can easily reproduce the problem.
>Release-Note:
>Audit-Trail:

From: Konrad Schroder <perseant@hhhh.org>
To: Manuel Bouyer <bouyer@antioche.lip6.fr>
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: kern/8863: filesystem lookup under heavy LFS write
Date: Tue, 23 Nov 1999 13:08:26 -0800 (PST)

 I see the problem: lfs_create declares a dirop, then goes on to call
 getnewvnode; an lfs vnode has to be flushed, but oops, dirops are active,
 so we can't flush anything or we risk corrupting the filesystem.  The
 other processes are backed up waiting on this one.
 (And lfs_{mkdir,mknod,symlink} will also have the same problem.)

 It may be possible to flush non-dirop vnodes without corrupting anything,
 but I'll have to think more on that to be sure.  If not, this can still be
 fixed in a couple of ways: Rewrite ufs_makeinode and ufs_mkdir in two
 parts, so that the lfs function can declare dirops *after* getnewvnode is
 called; or simply inline it into the lfs functions.

 I'll try and get one of these three solutions worked into a patch this
 week.
 						Konrad Schroder
 						perseant@hhhh.org

 On Tue, 23 Nov 1999, Manuel Bouyer wrote:

 > stack trace for tar (pid 267):
 > #0  mi_switch () at ../../../../kern/kern_synch.c:748
 > #1  0xf0136495 in bpendtsleep () at ../../../../kern/kern_synch.c:415
 > #2  0xf01d64ff in lfs_segwrite (mp=0xf04d4000, flags=5)
 >     at ../../../../ufs/lfs/lfs_segment.c:501
 > #3  0xf01d611c in lfs_vflush (vp=0xfe1dd700)
 >     at ../../../../ufs/lfs/lfs_segment.c:253
 > #4  0xf01d4ae3 in lfs_update (v=0xfe0beac4)
 >     at ../../../../ufs/lfs/lfs_inode.c:178
 > #5  0xf01db0d3 in lfs_fsync (v=0xfe0beb04) at ../../../../sys/vnode_if.h:1027
 > #6  0xf01542e7 in vinvalbuf (vp=0xfe1dd700, flags=1, cred=0xffffffff, 
 >     p=0xfe096c88, slpflag=0, slptimeo=0) at ../../../../sys/vnode_if.h:449
 > #7  0xf0154ea5 in vclean (vp=0xfe1dd700, flags=8, p=0xfe096c88)
 >     at ../../../../kern/vfs_subr.c:1349
 > #8  0xf0155083 in vgonel (vp=0xfe1dd700, p=0xfe096c88)
 >     at ../../../../kern/vfs_subr.c:1479
 > #9  0xf0154026 in getnewvnode (tag=VT_LFS, mp=0xf04d4000, vops=0xf0451c00, 
 >     vpp=0xfe0bec30) at ../../../../kern/vfs_subr.c:503
 > #10 0xf01d367e in lfs_vcreate (mp=0xf04d4000, ino=33107, vpp=0xfe0bec30)
 >     at ../../../../ufs/lfs/lfs_alloc.c:255
 > #11 0xf01d357b in lfs_valloc (v=0xfe0bec90)
 >     at ../../../../ufs/lfs/lfs_alloc.c:201
 > #12 0xf01e1792 in ufs_makeinode (mode=33188, dvp=0xfe1db09c, vpp=0xfe0beee8, 
 >     cnp=0xfe0beefc) at ../../../../sys/vnode_if.h:933
 > #13 0xf01deb7a in ufs_create (v=0xfe0bee08)
 >     at ../../../../ufs/ufs/ufs_vnops.c:118
 > #14 0xf01db462 in lfs_create (v=0xfe0bee08)
 >     at ../../../../ufs/lfs/lfs_vnops.c:449
 > #15 0xf015ae28 in vn_open (ndp=0xfe0beed8, fmode=1550, cmode=420)
 >     at ../../../../sys/vnode_if.h:96
 > #16 0xf01573a0 in sys_open (p=0xfe096c88, v=0xfe0bef88, retval=0xfe0bef80)
 >     at ../../../../kern/vfs_syscalls.c:999
 > #17 0xf01fa86e in syscall (frame={tf_es = 31, tf_ds = 31, tf_edi = 0, 
 >       tf_esi = 134813696, tf_ebp = -272639140, tf_ebx = 1549, tf_edx = -1, 
 >       tf_ecx = 2, tf_eax = 5, tf_trapno = 3, tf_err = 2, tf_eip = 134684535, 
 >       tf_cs = 23, tf_eflags = 514, tf_esp = -272639504, tf_ss = 31, 
 >       tf_vm86_es = 0, tf_vm86_ds = 0, tf_vm86_fs = 0, tf_vm86_gs = 0})
 >     at ../../../../arch/i386/i386/trap.c:760
 > #18 0xf0100d7d in syscall1 ()


From: Konrad Schroder <perseant@hhhh.org>
To: Manuel Bouyer <bouyer@antioche.lip6.fr>
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: kern/8863: filesystem lookup under heavy LFS write
Date: Wed, 24 Nov 1999 14:43:46 -0800 (PST)

 Please try the enclosed patch, which I think should fix this problem.

 						Konrad Schroder
 						perseant@hhhh.org

 Index: lfs_segment.c
 ===================================================================
 RCS file: /cvsroot/syssrc/sys/ufs/lfs/lfs_segment.c,v
 retrieving revision 1.36
 diff -u -r1.36 lfs_segment.c
 --- lfs_segment.c	1999/11/17 20:08:30	1.36
 +++ lfs_segment.c	1999/11/24 22:37:42
 @@ -484,32 +484,30 @@
  	/*
  	 * If lfs_flushvp is non-NULL, we are called from lfs_vflush,
  	 * in which case we have to flush *all* buffers off of this vnode.
 +	 * We don't care about other nodes, but write any non-dirop nodes
 +	 * anyway in anticipation of another getnewvnode().
 +	 *
 +	 * If we're cleaning we only write cleaning and ifile blocks, and
 +	 * no dirops, since otherwise we'd risk corruption in a crash.
  	 */
 -	if((sp->seg_flags & SEGM_CLEAN) && !(fs->lfs_flushvp))
 +	if(fs->lfs_flushvp)
 +		lfs_writevnodes(fs, mp, sp, VN_REG);
 +	else if(sp->seg_flags & SEGM_CLEAN)
  		lfs_writevnodes(fs, mp, sp, VN_CLEAN);
  	else {
  		lfs_writevnodes(fs, mp, sp, VN_REG);
 -		/*
 -		 * XXX KS - If we're cleaning, we can't wait for dirops,
 -		 * because they might be waiting on us.  The downside of this
 -		 * is that, if we write anything besides cleaning blocks
 -		 * while cleaning, the checkpoint is not completely
 -		 * consistent.
 -		 */
 -		if(!(sp->seg_flags & SEGM_CLEAN)) {
 -			while(fs->lfs_dirops)
 -				if((error = tsleep(&fs->lfs_writer, PRIBIO + 1,
 -						"lfs writer", 0)))
 -				{
 -					free(sp->bpp, M_SEGMENT);
 -					free(sp, M_SEGMENT); 
 -					return (error);
 -				}
 -			fs->lfs_writer++;
 -			writer_set=1;
 -			lfs_writevnodes(fs, mp, sp, VN_DIROP);
 -			((SEGSUM *)(sp->segsum))->ss_flags &= ~(SS_CONT);
 -		}
 +		while(fs->lfs_dirops)
 +			if((error = tsleep(&fs->lfs_writer, PRIBIO + 1,
 +					"lfs writer", 0)))
 +			{
 +				free(sp->bpp, M_SEGMENT);
 +				free(sp, M_SEGMENT); 
 +				return (error);
 +			}
 +		fs->lfs_writer++;
 +		writer_set=1;
 +		lfs_writevnodes(fs, mp, sp, VN_DIROP);
 +		((SEGSUM *)(sp->segsum))->ss_flags &= ~(SS_CONT);
  	}	

  	/*


From: Manuel Bouyer <bouyer@antioche.lip6.fr>
To: Konrad Schroder <perseant@hhhh.org>
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: kern/8863: filesystem lookup under heavy LFS write
Date: Thu, 25 Nov 1999 21:41:04 +0100

 On Wed, Nov 24, 1999 at 02:43:46PM -0800, Konrad Schroder wrote:
 > Please try the enclosed patch, which I think should fix this problem.

 I seems to, but I have now another one. tar hang, as well as any access to
 the LFS but access to other filesystems still works (this is with fresh sources,
 updated from cvs.netbsd.org today, at about 17H GMT).
 I've got a core dump, here's the ps output:
 rochebonne#/var/crash>ps -axl -M netbsd.6.core 
   UID   PID  PPID CPU PRI NI   VSZ  RSS WCHAN  STAT TT       TIME COMMAND
     0     0   173   0 -18  0     0    0 schedu DLs  ??    0:00.01 (swapper)
     0     1   173   0  10  0   236    0 wait   Is   ??    0:00.04 (init)
     0     2   173   0 -18  0     0    0 daemon DL   ??    0:00.08 (pagedaemon)
     0     3   173   0 -18  0     0    0 reaper DL   ??    0:00.03 (reaper)
     0     4   173   0  18  0     0    0 syncer DL   ??    0:00.97 (ioflush)
     0   150   173   0   2  0    92    0 -      Rs   ??    0:00.10 (syslogd)
     0   154   173  24   2  0    60    0 select IWs  ??    0:00.00 (portmap)
     0   161   173   0  10  0 20936    0 mfsidl Is   ??    0:00.04 (mount_mfs)
     0   165   173   0   2  0   116    0 select Is   ??    0:00.18 (mountd)
     0   167   173  24   2  0    40    0 netcon IWs  ??    0:00.00 (nfsd)
     0   171   173  24  10  0    16    0 nfsidl IL   ??    0:00.00 (nfsiod)
     0   172   173  24  10  0    16    0 nfsidl IL   ??    0:00.00 (nfsiod)
     0   173   173  24  10  0    16    0 nfsidl IL   ??    0:00.00 (nfsiod)
     0   174   173  24  10  0    16    0 nfsidl IL   ??    0:00.00 (nfsiod)
     0   177   173  24   2  0    20    0 nfsd   IL   ??    0:00.01 (nfsd)
     0   178   173  24   2  0    20    0 nfsd   IL   ??    0:00.01 (nfsd)
     0   179   173  24   2  0    20    0 nfsd   IL   ??    0:00.00 (nfsd)
     0   180   173  24   2  0    20    0 nfsd   IL   ??    0:00.00 (nfsd)
     0   192   173   0   2  0   260    0 select Is   ??    0:00.03 (amd)
     0   219   173   0  10  0   220    0 nanosl Ss   ??    0:00.03 (cron)
     0   222   173  16   2  0   212    0 select IWs  ??    0:00.00 (dhcpd)
     0   225   173  24   2  0    64    0 select IWs  ??    0:00.00 (inetd)
     0   231   173  24   2  0   240    0 select IWs  ??    0:00.00 (sshd1)
     0   265   173   0   2  0  3988    0 select S    ??    0:14.99 (XF86_SVGA)
     0   365   173   2  10  0    44    0 wait   Is   ??    0:00.00 (lfs_cleanerd
     0   366   173   1  29  0   128    0 segmen I    ??    0:00.01 (lfs_cleanerd
  1210   287   173   2   3  0   532    0 ttyin  Is+  p0    0:00.13 (tcsh)
  1210   288   173   8   3  0   524    0 ttyin  IWs+ p1    0:00.00 (tcsh)
  1210   294   173  10   3  0   524    0 ttyin  IWs+ p2    0:00.00 (tcsh)
  1210   283   173   7   3  0   524    0 ttyin  IWs+ p3    0:00.00 (tcsh)
  1210   282   173   6   3  0   524    0 ttyin  IWs+ p4    0:00.00 (tcsh)
     0   290   173   0  18  0   512    0 pause  Is   p5    0:00.19 (tcsh)
     0   355   173   0   2  0   220    0 select S+   p5    0:00.68 (top)
  1210   296   173  13   3  0   524    0 ttyin  IWs+ p6    0:00.00 (tcsh)
     0   313   173   0  18  0   552    0 pause  Is   p7    0:00.29 (tcsh)
     0   354   173   0   3  0   296    0 ttyin  S+   p7    0:00.52 (systat)
     0   326   173   2  18  0   552    0 pause  Is   p8    0:00.24 (tcsh)
     0   367   173   6   2  0   496    0 netio  I+   p8    0:00.33 (zcat)
     0   368   173   6  34  0   340    0 lfs_ma I+   p8    0:00.30 (tar)
  1210   254   173   4  18  0   524    0 pause  IWs  E0    0:00.00 (tcsh)
  1210   259   173  18  10  0   364    0 wait   IW+  E0    0:00.00 (sh)
  1210   264   173   4  10  0    68    0 wait   IW+  E0    0:00.00 (xinit)
     0   268   173   0   2  0   524    0 -      R    E0    0:00.28 (xterm)
  1210   270   173   0   2  0   236    0 select I    E0    0:00.42 (fvwm)
     0   271   173   0   2  0   536    0 select I    E0    0:01.29 (xterm)
  1210   272   173   0   2  0   156    0 select S    E0    0:00.29 (xload)
  1210   273   173   0   2  0   124    0 select S    E0    0:00.16 (oclock)
     0   276   173   1   2  0   532    0 select IW   E0    0:00.00 (xterm)
     0   277   173   1   2  0   532    0 select IW   E0    0:00.00 (xterm)
     0   278   173   1   2  0   532    0 select IW   E0    0:00.00 (xterm)
     0   279   173   1   2  0   532    0 select IW   E0    0:00.00 (xterm)
     0   280   173   0   2  0   532    0 select S    E0    0:00.72 (xterm)
     0   312   173   0   2  0   532    0 select S    E0    0:00.56 (xterm)
     0   325   173   0   2  0   532    0 select S    E0    0:00.31 (xterm)
     0   234   173  16   3  0    48    0 ttyin  IWs+ E1    0:00.00 (getty)
     0   235   173  16   3  0    48    0 ttyin  IWs+ E2    0:00.00 (getty)
     0   236   173  16   3  0    48    0 ttyin  IWs+ E3    0:00.00 (getty)
     0   237   173  16   3  0    48    0 ttyin  IWs+ E4    0:00.00 (getty)
     0   238   173  16   3  0    48    0 ttyin  IWs+ E5    0:00.00 (getty)
     0   239   173  16   3  0    48    0 ttyin  IWs+ E6    0:00.00 (getty)

 Stack trace for 'tar':
 (gdb) where
 #0  mi_switch () at ../../../../kern/kern_synch.c:748
 #1  0xf0136495 in bpendtsleep () at ../../../../kern/kern_synch.c:415
 #2  0xf01db2b5 in lfs_set_dirop (fs=0xf04fdc00)
     at ../../../../ufs/lfs/lfs_vnops.c:311
 #3  0xf01db59d in lfs_create (v=0xfed79e08)
     at ../../../../ufs/lfs/lfs_vnops.c:446
 #4  0xf015aee8 in vn_open (ndp=0xfed79ed8, fmode=1550, cmode=420)
     at ../../../../sys/vnode_if.h:96
 #5  0xf0157460 in sys_open (p=0xfe0f8dd0, v=0xfed79f88, retval=0xfed79f80)
     at ../../../../kern/vfs_syscalls.c:999
 #6  0xf01fa9fa in syscall (frame={tf_es = 31, tf_ds = 31, tf_edi = 0, 
       tf_esi = 134811648, tf_ebp = -272640156, tf_ebx = 1549, tf_edx = -1, 
       tf_ecx = 2, tf_eax = 5, tf_trapno = 3, tf_err = 2, tf_eip = 134684535, 
       tf_cs = 23, tf_eflags = 514, tf_esp = -272640520, tf_ss = 31, 
       tf_vm86_es = 0, tf_vm86_ds = 0, tf_vm86_fs = 0, tf_vm86_gs = 0})
     at ../../../../arch/i386/i386/trap.c:760
 #7  0xf0100d7d in syscall1 ()
 can not access 0xefbfd764, invalid translation (invalid PDE)
 can not access 0xefbfd764, invalid translation (invalid PDE)
 Cannot access memory at address 0xefbfd764.

 stack trace for 'lfs_cleanerd' (pid 366):
 (gdb) where
 #0  mi_switch () at ../../../../kern/kern_synch.c:748
 #1  0xf0136495 in bpendtsleep () at ../../../../kern/kern_synch.c:415
 #2  0xf01d96f2 in sys_lfs_segwait (p=0xfdf78c80, v=0xfe0d4f88, 
     retval=0xfe0d4f80) at ../../../../ufs/lfs/lfs_syscalls.c:864
 #3  0xf01fa9fa in syscall (frame={tf_es = -272695265, tf_ds = 31, 
       tf_edi = -272640140, tf_esi = -272640148, tf_ebp = -272640128, 
       tf_ebx = 134549504, tf_edx = 1835, tf_ecx = 3, tf_eax = 187, 
       tf_trapno = 3, tf_err = 2, tf_eip = 1208583703, tf_cs = 23, 
       tf_eflags = 663, tf_esp = -272640188, tf_ss = 31, tf_vm86_es = 0, 
       tf_vm86_ds = 0, tf_vm86_fs = 0, tf_vm86_gs = 0})
     at ../../../../arch/i386/i386/trap.c:760
 #4  0xf0100d7d in syscall1 ()
 can not access 0xefbfd780, invalid translation (invalid PDE)
 can not access 0xefbfd780, invalid translation (invalid PDE)
 Cannot access memory at address 0xefbfd780.

 The other lfs_cleanerd proc is waiting in sys_wait4(), nothing interesting
 here :)
 When in this situation, an attempt to 'reboot' (I got the core dump from
 ddb with 'call cpu_reboot 0x104', i.e. no sync and no umount) causes a panic:
 "Panic: bawrite LFS buffer" when the kernel umounts the LFS.
 A userland umount will just return EBUSY.

 Also I still get a panic when running a 'rm -rf' on a LFS filesystem
 (even a fresh one, just populated with kernel sources):
 panic: pool_get(lfsinopl): free list modified: magic=0; page 0xfe0cc000; item addr 0xfe0cc850
 syncing disks... 5 3 done

 I have a crash dump, here's the output of 'ps -alx':
 rochebonne#/var/crash>ps -axl -M netbsd.5.core
   UID   PID  PPID CPU PRI NI   VSZ  RSS WCHAN  STAT TT       TIME COMMAND
     0     0   173   0 -18  0     0    0 schedu DLs  ??    0:00.00 (swapper)
     0     1   173   1  10  0   236    0 wait   Is   ??    0:00.03 (init)
     0     2   173   0 -18  0     0    0 daemon DL   ??    0:00.00 (pagedaemon)
     0     3   173   0 -18  0     0    0 reaper DL   ??    0:00.02 (reaper)
     0     4   173   0  18  0     0    0 syncer DL   ??    0:00.09 (ioflush)
     0   150   173   0   2  0    92    0 select Ss   ??    0:00.08 (syslogd)
     0   154   173  18   2  0    60    0 select Is   ??    0:00.05 (portmap)
     0   158   173  15  10  0    44    0 wait   Is   ??    0:00.00 (lfs_cleanerd
     0   160   173   0  28  0   320    0 segmen I    ??    0:00.02 (lfs_cleanerd
     0   161   173   0  10  0 20936    0 mfsidl Ss   ??    0:00.03 (mount_mfs)
     0   165   173  15   2  0   100    0 select Is   ??    0:00.01 (mountd)
     0   167   173  15   2  0    40    0 netcon Is   ??    0:00.02 (nfsd)
     0   169   173  15   2  0    20    0 nfsd   IL   ??    0:00.01 (nfsd)
     0   170   173  15   2  0    20    0 nfsd   IL   ??    0:00.01 (nfsd)
     0   171   173  15   2  0    20    0 nfsd   IL   ??    0:00.01 (nfsd)
     0   172   173  15   2  0    20    0 nfsd   IL   ??    0:00.01 (nfsd)
     0   177   173  15  10  0    16    0 nfsidl IL   ??    0:00.00 (nfsiod)
     0   178   173  15  10  0    16    0 nfsidl IL   ??    0:00.00 (nfsiod)
     0   179   173  15  10  0    16    0 nfsidl IL   ??    0:00.00 (nfsiod)
     0   180   173  15  10  0    16    0 nfsidl IL   ??    0:00.00 (nfsiod)
     0   192   173   0   2  0   260    0 select Is   ??    0:00.02 (amd)
     0   220   173   0  10  0   220    0 nanosl Ss   ??    0:00.02 (cron)
     0   223   173  18   2  0   212    0 select Is   ??    0:00.01 (dhcpd)
     0   226   173  18   2  0    64    0 select Is   ??    0:00.01 (inetd)
     0   232   173   9   2  0   240    0 select Is   ??    0:05.81 (sshd1)
  1210   234   173   0  18  0   524    0 pause  Ss   E0    0:00.22 (tcsh)
  1210   274   173   5  33  0   236    0 -      R+   E0    0:00.11 (rm)
     0   235   173   9   3  0    48    0 ttyin  Is+  E1    0:00.03 (getty)
     0   236   173   9   3  0    48    0 ttyin  Is+  E2    0:00.03 (getty)
     0   237   173   9   3  0    48    0 ttyin  Is+  E3    0:00.03 (getty)
     0   238   173   9   3  0    48    0 ttyin  Is+  E4    0:00.03 (getty)
     0   239   173   9   3  0    48    0 ttyin  Is+  E5    0:00.03 (getty)
     0   240   173   9   3  0    48    0 ttyin  Is+  E6    0:00.03 (getty)

 The stack trace look like:
 #0  cpu_reboot (howto=11, bootstr=0xfe0b2af8 " +\013þø©\020ðD+\013þ")
     at ../../../../arch/i386/i386/machdep.c:1192
 #1  0xf01f4647 in cpu_reboot (howto=256, bootstr=0x0)
     at ../../../../arch/i386/i386/machdep.c:1215
 #2  0xf010ac71 in db_fncall (addr=-32822460, have_addr=0, count=0, 
     modif=0xfe0b2b28 " ð&ð\001") at ../../../../ddb/db_command.c:561
 #3  0xf010a9f8 in db_command (last_cmdp=0xf025d878, cmd_table=0xf025d6b8)
     at ../../../../ddb/db_command.c:295
 #4  0xf010ab52 in db_command_loop () at ../../../../ddb/db_command.c:486
 #5  0xf010d212 in db_trap (type=1, code=0) at ../../../../ddb/db_trap.c:78
 #6  0xf01f2570 in kdb_trap (type=1, code=0, regs=0xfe0b2c54)
     at ../../../../arch/i386/i386/db_interface.c:120
 #7  0xf01fa338 in trap (frame={tf_es = 1667432464, tf_ds = -32833520, 
       tf_edi = -266090624, tf_esi = 256, tf_ebp = -32822132, 
       tf_ebx = -32822100, tf_edx = 981, tf_ecx = 768, tf_eax = 0, 
       tf_trapno = 1, tf_err = 0, tf_eip = -266394068, tf_cs = -266141688, 
       tf_eflags = 514, tf_esp = -32822112, tf_ss = -267139359, 
       tf_vm86_es = -32714792, tf_vm86_ds = -265718368, tf_vm86_fs = -32716720, 
       tf_vm86_gs = -32822044}) at ../../../../arch/i386/i386/trap.c:298
 #8  0xf0100cc5 in calltrap ()
 #9  0xf013c6e1 in panic (
     fmt=0xf023c780 "pool_get(%s): free list modified: magic=%x; page %p; item addr %p\n") at ../../../../kern/subr_prf.c:215
 #10 0xf013b708 in _pool_get (pp=0xf02975a0, flags=2, 
     file=0xf0243f20 "../../../../ufs/lfs/lfs_alloc.c", line=264)
     at ../../../../kern/subr_pool.c:745
 #11 0xf01d3761 in lfs_vcreate (mp=0xf04a9400, ino=282, vpp=0xfe0b2d34)
     at ../../../../ufs/lfs/lfs_alloc.c:264
 #12 0xf01da870 in lfs_vget (mp=0xf04a9400, ino=282, vpp=0xfe0b2dcc)
     at ../../../../ufs/lfs/lfs_vfsops.c:681
 #13 0xf01dda72 in ufs_lookup (v=0xfe0b2e3c)
     at ../../../../ufs/ufs/ufs_lookup.c:494
 #14 0xf0153623 in lookup (ndp=0xfe0b2ef0) at ../../../../sys/vnode_if.h:71
 #15 0xf0153317 in namei (ndp=0xfe0b2ef0) at ../../../../kern/vfs_lookup.c:153
 #16 0xf01588a1 in sys_unlink (p=0xfe0998c8, v=0xfe0b2f88, retval=0xfe0b2f80)
     at ../../../../kern/vfs_syscalls.c:1560
 #17 0xf01fa9fa in syscall (frame={tf_es = 31, tf_ds = 31, tf_edi = 0, 
       tf_esi = 134713344, tf_ebp = -272639128, tf_ebx = 134736512, 
       tf_edx = 134734208, tf_ecx = 0, tf_eax = 10, tf_trapno = 3, tf_err = 2, 
       tf_eip = 134606939, tf_cs = 23, tf_eflags = 582, tf_esp = -272639148, 
       tf_ss = 31, tf_vm86_es = 0, tf_vm86_ds = 0, tf_vm86_fs = 0, 
       tf_vm86_gs = 0}) at ../../../../arch/i386/i386/trap.c:760
 #18 0xf0100d7d in syscall1 ()
 can not access 0xefbfdb68, invalid translation (invalid PDE)
 can not access 0xefbfdb68, invalid translation (invalid PDE)
 Cannot access memory at address 0xefbfdb68.

 I'm not sure if this is related to the previous problem or not.
 Let me know if you need trace of other processes (but I'm not sure we'll
 have something interesting here), or other infos.

 --
 Manuel Bouyer <bouyer@antioche.eu.org>
 --

From: Konrad Schroder <perseant@hhhh.org>
To: Manuel Bouyer <bouyer@antioche.lip6.fr>
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: kern/8863: filesystem lookup under heavy LFS write
Date: Mon, 29 Nov 1999 11:17:12 -0800 (PST)

 On Thu, 25 Nov 1999, Manuel Bouyer wrote:

 > I seems to, but I have now another one. tar hang, as well as any access to
 > the LFS but access to other filesystems still works (this is with fresh sources,
 > updated from cvs.netbsd.org today, at about 17H GMT).

 >     0   368   173   6  34  0   340    0 lfs_ma I+   p8    0:00.30 (tar)

 Here's a new patch, which seems to fix the problem here.
 (OTOH I still have no idea on the pool panic problem.)

 						Konrad Schroder
 						perseant@hhhh.org

 Index: lfs_vnops.c
 ===================================================================
 RCS file: /cvsroot/syssrc/sys/ufs/lfs/lfs_vnops.c,v
 retrieving revision 1.32
 diff -u -r1.32 lfs_vnops.c
 --- lfs_vnops.c	1999/11/15 18:49:14	1.32
 +++ lfs_vnops.c	1999/11/29 19:14:44
 @@ -304,9 +304,16 @@
  	while (fs->lfs_writer || lfs_dirvcount>LFS_MAXDIROP) {
  		if(fs->lfs_writer)
  			tsleep(&fs->lfs_dirops, PRIBIO + 1, "lfs_dirop", 0);
 +		if(lfs_dirvcount > LFS_MAXDIROP) {
 +			lfs_flush(fs,0);
 +			/* We might be busying the fs, flush it explicitly */
 +			lfs_flush_fs(fs->lfs_ivnode->v_mount,0);
 +		}
 +
 +		/* If it's still too high, e.g. other dirops are active, sleep */
  		if(lfs_dirvcount > LFS_MAXDIROP) {		
  #ifdef DEBUG_LFS
 -			printf("(dirvcount=%d)\n",lfs_dirvcount); 
 +			printf("lfs_set_dirop: sleeping with dirops=%d, dirvcount=%d\n",fs->lfs_dirops,lfs_dirvcount); 
  #endif
  			if((error=tsleep(&lfs_dirvcount, PCATCH|PUSER, "lfs_maxdirop", 0))!=0)
  				return error;


From: Manuel Bouyer <bouyer@antioche.lip6.fr>
To: Konrad Schroder <perseant@hhhh.org>
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: kern/8863: filesystem lookup under heavy LFS write
Date: Wed, 1 Dec 1999 10:49:31 +0100

 On Mon, Nov 29, 1999 at 11:17:12AM -0800, Konrad Schroder wrote:
 > Here's a new patch, which seems to fix the problem here.

 Ok, with this one + the one you sent me previsoulsy I've been able to
 extract a whole src tree several times without problems.

 > (OTOH I still have no idea on the pool panic problem.)

 I looked a bit at this. The pool magic number has been set to 0, which
 in the inode structure correspond to the le_next member of i_hash.
 This one is set to 0 in ufs_ihashrem().
 I'll try to add more code to confirm this theory and detect the corruption
 earlier. Maybe this way I'll get a usefull stack trace.
State-Changed-From-To: open->feedback 
State-Changed-By: perseant 
State-Changed-When: Sat Jan 15 15:27:13 PST 2000 
State-Changed-Why:  
Problem solved, patch committed some time ago 


Responsible-Changed-From-To: kern-bug-people->perseant 
Responsible-Changed-By: perseant 
Responsible-Changed-When: Sat Jan 15 15:27:13 PST 2000 
Responsible-Changed-Why:  
I have responsibility for LFS-related problems. 
State-Changed-From-To: feedback->closed 
State-Changed-By: bouyer 
State-Changed-When: Thu Apr 6 23:21:11 PDT 2000 
State-Changed-Why:  
Solved, I haven't got this problem for months now. 
>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.