NetBSD Problem Report #33406

From simonb@thistledown.com.au  Mon May  1 14:18:12 2006
Return-Path: <simonb@thistledown.com.au>
Received: from thoreau.thistledown.com.au (thoreau.thistledown.com.au [203.217.30.154])
	by narn.netbsd.org (Postfix) with ESMTP id 9305063B882
	for <gnats-bugs@gnats.NetBSD.org>; Mon,  1 May 2006 14:18:11 +0000 (UTC)
Message-Id: <20060501141803.B89972356F@thoreau.thistledown.com.au>
Date: Tue,  2 May 2006 00:18:03 +1000 (EST)
From: Simon Burge <simonb@wasabisystems.com>
Reply-To: Simon Burge <simonb@wasabisystems.com>
To: gnats-bugs@netbsd.org
Subject: softdeps get stuck in endless loop
X-Send-Pr-Version: 3.95

>Number:         33406
>Category:       kern
>Synopsis:       softdeps get stuck in endless loop
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    ad
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon May 01 14:20:00 +0000 2006
>Closed-Date:    Tue Apr 29 18:21:32 +0000 2008
>Last-Modified:  Tue Apr 29 18:21:32 +0000 2008
>Originator:     Simon Burge <simonb@wasabisystems.com>
>Release:        NetBSD 3.99.18, sources from around Apr 30, 2006.
>Organization:
Wasabi Systems
>Environment:
System: NetBSD iq80321 3.99.18 NetBSD 3.99.18 (IQ80321) #6: Fri Apr 28 14:06:50 EST 2006 simonb@bigkev:/home/simonb/netbsd/src-virgin/sys/arch/evbarm/compile/IQ80321
Architecture: arm
Machine: evbarm
 and
System: NetBSD athlon 3.99.18 NetBSD 3.99.18 (GENERIC) #4: Mon May 1 12:54:17 EST 2006 simonb@bigkev:/home/simonb/netbsd/src-virgin/sys/arch/i386/compile/GENERIC i386
Architecture: i386
Machine: i386
>Description:

        Copying lots of files on to a filesystem with softdeps enabled
        on an ARM architecture with a modernish disk results in
        vflusbuf() getting stuck in a loop writing out the same two
        blocks over and over.  The lat_fs test from lmbench is also a
        reliable way to reproduce the problem.

        Adding a printf to bwrite() and softdep_disk_write_complete()
        and using a serial console also makes an i386 and a evbmips
        machine fail in the same way.  I'll send a patch to the PR
        separately.

        What I think appears to be happening is that vflushbuf() is
        calling bwrite() for a dirty buffer.  I think when vflushbuf()
        is does the splx(s) just before calling bwrite() for the next
        dirty buffer, the biodone() comes through for the IO that has
        just completed, and for some reason the softdep code decides
        that there is a rollback in the inode block, and calls bdirty()
        to reinsert the previous block back on to the dirty list.  The
        next buffer writen goes through the same process, except that
        the rollback is because the bitmap is incomplete, and we end up
        writing the same two blocks out one after another in an endless
        loop

        In twist, Steve Woodford reports that he can reproduce the
        problem on an IQ31244 with a fast modern disk, but that the
        problem goes away when he uses an ancient 400MB non-DMA disk.

	This also appears to be the cause of kern/32330.

>How-To-Repeat:

        The easiest way I've found to reproduce the problem is to use
        the following recipe.  Sometimes more than one invocation of
        lat_fs may be needed.

        newfs -I -i 524288 /dev/rwd0d
        mount -o softdep -t ffs /dev/wd0d /mnt
	cd /mnt
	.../lmbench/bin/.../lat_fs

>Fix:

	None given.  This seems to be timing related somehow though.

>Release-Note:

>Audit-Trail:
From: Simon Burge <simonb@wasabisystems.com>
To: gnats-bugs@netbsd.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org
Subject: Re: kern/33406: softdeps get stuck in endless loop 
Date: Tue, 02 May 2006 00:52:09 +1000

 With the following patches running on an i386 with a 115200 baud serial
 console, I can reproduce this problem using the lat_fs recipe in the PR.

 The Debugger call in vfs_subr.c drops you to ddb after a while.  You
 may need to "c"ontinue a couple of times before you see the problem.  I
 added the Debugger check since sending a BREAK to the serial console
 didn't drop in to ddb while the console was busying sending so much
 output.


 Index: kern/vfs_bio.c
 ===================================================================
 RCS file: /cvsroot/src/sys/kern/vfs_bio.c,v
 retrieving revision 1.159
 diff -d -p -u -r1.159 vfs_bio.c
 --- kern/vfs_bio.c	5 Apr 2006 00:52:16 -0000	1.159
 +++ kern/vfs_bio.c	1 May 2006 14:06:31 -0000
 @@ -735,7 +735,10 @@ bwrite(struct buf *bp)
  	 * vnode queue.
  	 */
  	if (wasdelayed)
 +{
 +printf("bwrite: reassignbuf\n");
  		reassignbuf(bp, bp->b_vp);
 +}
  	else
  		p->p_stats->p_ru.ru_oublock++;

 Index: kern/vfs_subr.c
 ===================================================================
 RCS file: /cvsroot/src/sys/kern/vfs_subr.c,v
 retrieving revision 1.265
 diff -d -p -u -r1.265 vfs_subr.c
 --- kern/vfs_subr.c	25 Feb 2006 07:11:31 -0000	1.265
 +++ kern/vfs_subr.c	1 May 2006 14:06:32 -0000
 @@ -862,6 +862,9 @@ vflushbuf(struct vnode *vp, int sync)
  loop:
  	s = splbio();
  	for (bp = LIST_FIRST(&vp->v_dirtyblkhd); bp; bp = nbp) {
 +static int loops = 0;
 +if (loops++ % 1000 == 0)
 + Debugger();
  		nbp = LIST_NEXT(bp, b_vnbufs);
  		simple_lock(&bp->b_interlock);
  		if ((bp->b_flags & B_BUSY)) {
 Index: ufs/ffs/ffs_softdep.c
 ===================================================================
 RCS file: /cvsroot/src/sys/ufs/ffs/ffs_softdep.c,v
 retrieving revision 1.73
 diff -d -p -u -r1.73 ffs_softdep.c
 --- ufs/ffs/ffs_softdep.c	24 Dec 2005 20:45:10 -0000	1.73
 +++ ufs/ffs/ffs_softdep.c	1 May 2006 14:06:33 -0000
 @@ -4032,6 +4032,7 @@ softdep_disk_write_complete(bp)
  	/*
  	 * Reattach any requests that must be redone.
  	 */
 +printf("reattach loop\n");
  	while ((wk = LIST_FIRST(&reattach)) != NULL) {
  		WORKLIST_REMOVE(wk);
  		WORKLIST_INSERT(&bp->b_dep, wk);

From: YAMAMOTO Takashi <yamt@mwd.biglobe.ne.jp>
To: gnats-bugs@netbsd.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org,
	netbsd-bugs@netbsd.org
Subject: Re: kern/33406: softdeps get stuck in endless loop
Date: Mon, 08 May 2006 11:02:22 +0900

 > >How-To-Repeat:
 > 
 >         The easiest way I've found to reproduce the problem is to use
 >         the following recipe.  Sometimes more than one invocation of
 >         lat_fs may be needed.
 > 
 >         newfs -I -i 524288 /dev/rwd0d
 >         mount -o softdep -t ffs /dev/wd0d /mnt
 > 	cd /mnt
 > 	.../lmbench/bin/.../lat_fs

 your /dev is not ffs?

 YAMAMOTO Takashi

From: Simon Burge <simonb@wasabisystems.com>
To: YAMAMOTO Takashi <yamt@mwd.biglobe.ne.jp>
Cc: gnats-bugs@netbsd.org, kern-bug-people@netbsd.org,
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/33406: softdeps get stuck in endless loop 
Date: Mon, 08 May 2006 12:11:39 +1000

 YAMAMOTO Takashi wrote:

 > > >How-To-Repeat:
 > > 
 > >         The easiest way I've found to reproduce the problem is to use
 > >         the following recipe.  Sometimes more than one invocation of
 > >         lat_fs may be needed.
 > > 
 > >         newfs -I -i 524288 /dev/rwd0d
 > >         mount -o softdep -t ffs /dev/wd0d /mnt
 > > 	cd /mnt
 > > 	.../lmbench/bin/.../lat_fs
 > 
 > your /dev is not ffs?

 Correct.  / and /home are nfs mounts, and /mnt is the only local
 (ffs) filesystem.

 Cheers,
 Simon.
 --
 Simon Burge                            <simonb@wasabisystems.com>
 NetBSD Support and Service:         http://www.wasabisystems.com/

From: YAMAMOTO Takashi <yamt@mwd.biglobe.ne.jp>
To: simonb@wasabisystems.com
Cc: gnats-bugs@netbsd.org, kern-bug-people@netbsd.org,
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/33406: softdeps get stuck in endless loop 
Date: Mon, 08 May 2006 11:25:12 +0900

 > Correct.  / and /home are nfs mounts, and /mnt is the only local
 > (ffs) filesystem.

 only ffs has special code for softdep, like
 softdep_fsync_mountdev and B_SCANNED.  see ffs_full_fsync.

 YAMAMOTO Takashi

From: Simon Burge <simonb@wasabisystems.com>
To: YAMAMOTO Takashi <yamt@mwd.biglobe.ne.jp>
Cc: gnats-bugs@netbsd.org, kern-bug-people@netbsd.org,
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/33406: softdeps get stuck in endless loop 
Date: Mon, 08 May 2006 12:33:13 +1000

 YAMAMOTO Takashi wrote:

 > > Correct.  / and /home are nfs mounts, and /mnt is the only local
 > > (ffs) filesystem.
 > 
 > only ffs has special code for softdep, like
 > softdep_fsync_mountdev and B_SCANNED.  see ffs_full_fsync.

 The lat_fs test is being run inside /mnt, and fails with /mnt is mounted
 with softdeps and works when mounted without softdeps.

 Simon.
 --
 Simon Burge                            <simonb@wasabisystems.com>
 NetBSD Support and Service:         http://www.wasabisystems.com/

From: YAMAMOTO Takashi <yamt@mwd.biglobe.ne.jp>
To: simonb@wasabisystems.com
Cc: gnats-bugs@netbsd.org, kern-bug-people@netbsd.org,
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/33406: softdeps get stuck in endless loop 
Date: Mon, 08 May 2006 11:44:00 +0900

 > > > Correct.  / and /home are nfs mounts, and /mnt is the only local
 > > > (ffs) filesystem.
 > > 
 > > only ffs has special code for softdep, like
 > > softdep_fsync_mountdev and B_SCANNED.  see ffs_full_fsync.
 > 
 > The lat_fs test is being run inside /mnt, and fails with /mnt is mounted
 > with softdeps and works when mounted without softdeps.

 i meant code necessary for softdep-mounted block device.

 YAMAMOTO Takashi

Responsible-Changed-From-To: kern-bug-people->ad
Responsible-Changed-By: ad@NetBSD.org
Responsible-Changed-When: Tue, 29 Apr 2008 17:51:29 +0000
Responsible-Changed-Why:
take


From: Andrew Doran <ad@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/33406 CVS commit: src/sys
Date: Tue, 29 Apr 2008 18:18:09 +0000 (UTC)

 Module Name:	src
 Committed By:	ad
 Date:		Tue Apr 29 18:18:09 UTC 2008

 Modified Files:
 	src/sys/coda: coda_vfsops.c
 	src/sys/fs/adosfs: advfsops.c
 	src/sys/fs/cd9660: cd9660_vfsops.c
 	src/sys/fs/filecorefs: filecore_vfsops.c
 	src/sys/fs/hfs: hfs_vfsops.c hfs_vnops.c
 	src/sys/fs/msdosfs: msdosfs_vfsops.c
 	src/sys/fs/ntfs: ntfs_vfsops.c
 	src/sys/fs/ptyfs: ptyfs_vfsops.c
 	src/sys/fs/puffs: puffs_vfsops.c
 	src/sys/fs/smbfs: smbfs_vfsops.c
 	src/sys/fs/sysvbfs: sysvbfs.c
 	src/sys/fs/tmpfs: tmpfs_vfsops.c
 	src/sys/fs/udf: udf_vfsops.c
 	src/sys/fs/union: union_vfsops.c
 	src/sys/fs/unionfs: unionfs_vfsops.c
 	src/sys/miscfs/fdesc: fdesc_vfsops.c
 	src/sys/miscfs/kernfs: kernfs_vfsops.c
 	src/sys/miscfs/nullfs: null_vfsops.c
 	src/sys/miscfs/overlay: overlay_vfsops.c
 	src/sys/miscfs/portal: portal_vfsops.c
 	src/sys/miscfs/procfs: procfs_vfsops.c
 	src/sys/miscfs/specfs: spec_vnops.c
 	src/sys/miscfs/umapfs: umap_vfsops.c
 	src/sys/nfs: nfs_vfsops.c
 	src/sys/sys: mount.h vnode.h
 	src/sys/ufs/ext2fs: ext2fs_vfsops.c ext2fs_vnops.c
 	src/sys/ufs/ffs: ffs_softdep.c ffs_vfsops.c ffs_vnops.c
 	src/sys/ufs/lfs: lfs_vfsops.c
 	src/sys/ufs/mfs: mfs_vfsops.c

 Log Message:
 PR kern/38057 ffs makes assuptions about devvp file system
 PR kern/33406 softdeps get stuck in endless loop

 Introduce VFS_FSYNC() and call it when syncing a block device, if it
 has a mounted file system.


 To generate a diff of this commit:
 cvs rdiff -r1.64 -r1.65 src/sys/coda/coda_vfsops.c
 cvs rdiff -r1.49 -r1.50 src/sys/fs/adosfs/advfsops.c
 cvs rdiff -r1.57 -r1.58 src/sys/fs/cd9660/cd9660_vfsops.c
 cvs rdiff -r1.49 -r1.50 src/sys/fs/filecorefs/filecore_vfsops.c
 cvs rdiff -r1.16 -r1.17 src/sys/fs/hfs/hfs_vfsops.c
 cvs rdiff -r1.9 -r1.10 src/sys/fs/hfs/hfs_vnops.c
 cvs rdiff -r1.61 -r1.62 src/sys/fs/msdosfs/msdosfs_vfsops.c
 cvs rdiff -r1.65 -r1.66 src/sys/fs/ntfs/ntfs_vfsops.c
 cvs rdiff -r1.31 -r1.32 src/sys/fs/ptyfs/ptyfs_vfsops.c
 cvs rdiff -r1.78 -r1.79 src/sys/fs/puffs/puffs_vfsops.c
 cvs rdiff -r1.78 -r1.79 src/sys/fs/smbfs/smbfs_vfsops.c
 cvs rdiff -r1.10 -r1.11 src/sys/fs/sysvbfs/sysvbfs.c
 cvs rdiff -r1.39 -r1.40 src/sys/fs/tmpfs/tmpfs_vfsops.c
 cvs rdiff -r1.35 -r1.36 src/sys/fs/udf/udf_vfsops.c
 cvs rdiff -r1.52 -r1.53 src/sys/fs/union/union_vfsops.c
 cvs rdiff -r1.1 -r1.2 src/sys/fs/unionfs/unionfs_vfsops.c
 cvs rdiff -r1.73 -r1.74 src/sys/miscfs/fdesc/fdesc_vfsops.c
 cvs rdiff -r1.83 -r1.84 src/sys/miscfs/kernfs/kernfs_vfsops.c
 cvs rdiff -r1.73 -r1.74 src/sys/miscfs/nullfs/null_vfsops.c
 cvs rdiff -r1.48 -r1.49 src/sys/miscfs/overlay/overlay_vfsops.c
 cvs rdiff -r1.71 -r1.72 src/sys/miscfs/portal/portal_vfsops.c
 cvs rdiff -r1.77 -r1.78 src/sys/miscfs/procfs/procfs_vfsops.c
 cvs rdiff -r1.117 -r1.118 src/sys/miscfs/specfs/spec_vnops.c
 cvs rdiff -r1.75 -r1.76 src/sys/miscfs/umapfs/umap_vfsops.c
 cvs rdiff -r1.196 -r1.197 src/sys/nfs/nfs_vfsops.c
 cvs rdiff -r1.173 -r1.174 src/sys/sys/mount.h
 cvs rdiff -r1.192 -r1.193 src/sys/sys/vnode.h
 cvs rdiff -r1.131 -r1.132 src/sys/ufs/ext2fs/ext2fs_vfsops.c
 cvs rdiff -r1.81 -r1.82 src/sys/ufs/ext2fs/ext2fs_vnops.c
 cvs rdiff -r1.109 -r1.110 src/sys/ufs/ffs/ffs_softdep.c
 cvs rdiff -r1.223 -r1.224 src/sys/ufs/ffs/ffs_vfsops.c
 cvs rdiff -r1.98 -r1.99 src/sys/ufs/ffs/ffs_vnops.c
 cvs rdiff -r1.256 -r1.257 src/sys/ufs/lfs/lfs_vfsops.c
 cvs rdiff -r1.92 -r1.93 src/sys/ufs/mfs/mfs_vfsops.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->closed
State-Changed-By: ad@NetBSD.org
State-Changed-When: Tue, 29 Apr 2008 18:21:32 +0000
State-Changed-Why:
I think there are two parts to this. One is fixed by the change that should
be noted in the PR within a few minutes. The other was widespread access to
vnode and buf structures that was not atomic WRT interrupts, which was fixed
with the merge of vmlocking.



>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.