NetBSD Problem Report #10448

Received: (qmail 3110 invoked from network); 26 Jun 2000 05:42:25 -0000
Message-Id: <y9jog4pc820.wl@vega.sat.t.u-tokyo.ac.jp>
Date: Mon, 26 Jun 2000 14:42:15 +0900
From: IWAMOTO Toshihiro <iwamoto@sat.t.u-tokyo.ac.jp>
Reply-To: IWAMOTO Toshihiro <iwamoto@sat.t.u-tokyo.ac.jp>
To: gnats-bugs@gnats.netbsd.org
Subject: softdep panics under heavy load at lockmgr
X-Send-Pr-Version: 3.95

>Number:         10448
>Category:       kern
>Synopsis:       softdep panics under heavy load at lockmgr
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    kern-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Jun 26 05:43:00 +0000 2000
>Closed-Date:    Mon Feb 25 00:56:50 +0000 2002
>Last-Modified:  Mon Feb 25 00:56:50 +0000 2002
>Originator:     IWAMOTO Toshihiro
>Release:        NetBSD-current of 2 weeks ago, also confirmed on recent 1.5_ALPHA
>Organization:
	Univ. of Tokyo
>Environment:

System: NetBSD yomogi.sat.t.u-tokyo.ac.jp 1.4ZA NetBSD 1.4ZA (YOMOGI) #18: Mon Jun 12 17:50:29 JST 2000 iwamoto@yomogi.sat.t.u-tokyo.ac.jp:/usr/src/syssrc/sys/arch/i386/compile/YOMOGI i386


>Description:
	With softdep enabled and heavy filesystem activity,
	ffs_vget() can call itself, resulting in the following panic.

	As shown below, the detailed situation of this panic is:

	1. ffs_vget() calls getnewvnode() with ufs_hashlock locked
	2. getnewvnode() calls vgonel() to recycle a vnode entry
	3. flush_pagedep_deps() needs to call ffs_vget() to add
	   directory entries "." and ".." before flushing the vnode
	4. ufs_hashlock is already locked

#10 0xc015ecc9 in panic (fmt=0xc0304380 "lockmgr: locking against myself")
    at ../../../../kern/subr_prf.c:219
#11 0xc0151296 in lockmgr (lkp=0xc03b67e0, flags=34, interlkp=0x0)
    at ../../../../kern/kern_lock.c:518
#12 0xc023c43f in ffs_vget (mp=0xc08c3600, ino=1075358, vpp=0xd88f3a64)
    at ../../../../ufs/ffs/ffs_vfsops.c:968
#13 0xc02ea224 in flush_pagedep_deps (pvp=0xd8765448, mp=0xc08c3600,
    diraddhdp=0xc0a675b0) at ../../../../../gnu/sys/ufs/ffs/ffs_softdep.c:4296
#14 0xc02e9cad in softdep_sync_metadata (v=0xd88f3b50)
    at ../../../../../gnu/sys/ufs/ffs/ffs_softdep.c:4026
#15 0xc023d45d in ffs_fsync (v=0xd88f3b50)
    at ../../../../ufs/ffs/ffs_vnops.c:312
#16 0xc0176f16 in vinvalbuf (vp=0xd8765448, flags=1, cred=0xffffffff,
    p=0xd88ec018, slpflag=0, slptimeo=0) at ../../../../sys/vnode_if.h:449
#17 0xc0177bfd in vclean (vp=0xd8765448, flags=8, p=0xd88ec018)
    at ../../../../kern/vfs_subr.c:1424
#18 0xc0177ddb in vgonel (vp=0xd8765448, p=0xd88ec018)
    at ../../../../kern/vfs_subr.c:1554
#19 0xc0176cbd in getnewvnode (tag=VT_UFS, mp=0xc08c3600, vops=0xc0874200,
    vpp=0xd88f3c4c) at ../../../../kern/vfs_subr.c:508
#20 0xc023c45e in ffs_vget (mp=0xc08c3600, ino=1394840, vpp=0xd88f3cb0)
    at ../../../../ufs/ffs/ffs_vfsops.c:971
#21 0xc0235dae in ffs_valloc (v=0xd88f3cb4)
    at ../../../../ufs/ffs/ffs_alloc.c:605
#22 0xc024c700 in ufs_makeinode (mode=33188, dvp=0xd8793014, vpp=0xd88f3ee8,
    cnp=0xd88f3efc) at ../../../../sys/vnode_if.h:933
#23 0xc0249a82 in ufs_create (v=0xd88f3e08)
---Type <return> to continue, or q <return> to quit---
    at ../../../../ufs/ufs/ufs_vnops.c:117
#24 0xc017df98 in vn_open (ndp=0xd88f3ed8, fmode=1550, cmode=420)
    at ../../../../sys/vnode_if.h:96
#25 0xc017a3ae in sys_open (p=0xd88ec018, v=0xd88f3f88, retval=0xd88f3f80)
    at ../../../../kern/vfs_syscalls.c:1004
#26 0xc02699e7 in syscall (frame={tf_es = 134873119, tf_ds = -1078001633,
      tf_edi = 0, tf_esi = 134882304, tf_ebp = -1077946548, tf_ebx = 1549,
      tf_edx = -1, tf_ecx = -35, tf_eax = 5, tf_trapno = 3, tf_err = 2,
      tf_eip = 134730351, tf_cs = 23, tf_eflags = 514, tf_esp = -1077946912,
      tf_ss = 31, tf_vm86_es = 0, tf_vm86_ds = 0, tf_vm86_fs = 0,
      tf_vm86_gs = 0}) at ../../../../arch/i386/i386/trap.c:766
#27 0xc0100dc1 in syscall1 ()
can not access 0xbfbfd74c, invalid translation (invalid PDE)
can not access 0xbfbfd74c, invalid translation (invalid PDE)

>How-To-Repeat:
	To reproduce the above panic, an incomplete directory vnode
	should be vgonel()'ed from getnewvnode().
	Such a situation can be easily reproducable.

	1. Boot with a large amount (~50MB) of buffers and the default
	   value of numvnodes (778).
	2. Generate a heavy disk activity by issueing the following commands
	   simultaneously.

	$ cd /usr/pkgsrc/www/mozilla; make    (tarball extraction)
	# cd /usr/src/syssrc; cvs update -Pd

>Fix:
	A workaround is to configure a large number of vnodes.

	I think a lock to ufs_hashlock is unnecessary when calling
	getnewvnode(), and the patch below seems to work.
	Am I missing the point?

--- /sys/ufs/ffs/ffs_vfsops.c	Sun Jun 25 16:57:51 2000
+++ ./ffs_vfsops.c	Mon Jun 26 13:47:58 2000
@@ -1014,15 +1014,16 @@
 	} while (lockmgr(&ufs_hashlock, LK_EXCLUSIVE|LK_SLEEPFAIL, 0));

 	/* Allocate a new vnode/inode. */
+	lockmgr(&ufs_hashlock, LK_RELEASE, 0);
 	if ((error = getnewvnode(VT_UFS, mp, ffs_vnodeop_p, &vp)) != 0) {
 		*vpp = NULL;
-		lockmgr(&ufs_hashlock, LK_RELEASE, 0);
 		return (error);
 	}
 	/*
 	 * XXX MFS ends up here, too, to allocate an inode.  Should we
 	 * XXX create another pool for MFS inodes?
 	 */
+	lockmgr(&ufs_hashlock, LK_EXCLUSIVE|LK_SLEEPFAIL, 0);
 	ip = pool_get(&ffs_inode_pool, PR_WAITOK);
 	memset((caddr_t)ip, 0, sizeof(struct inode));
 	vp->v_data = ip;
>Release-Note:
>Audit-Trail:

From: Frank van der Linden <frank@wins.uva.nl>
To: IWAMOTO Toshihiro <iwamoto@sat.t.u-tokyo.ac.jp>
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: kern/10448: softdep panics under heavy load at lockmgr
Date: Wed, 28 Jun 2000 01:29:43 +0200

 On Mon, Jun 26, 2000 at 02:42:15PM +0900, IWAMOTO Toshihiro wrote:
 > 	With softdep enabled and heavy filesystem activity,
 > 	ffs_vget() can call itself, resulting in the following panic.
 > 
 > 	As shown below, the detailed situation of this panic is:
 > 
 > 	1. ffs_vget() calls getnewvnode() with ufs_hashlock locked
 > 	2. getnewvnode() calls vgonel() to recycle a vnode entry
 > 	3. flush_pagedep_deps() needs to call ffs_vget() to add
 > 	   directory entries "." and ".." before flushing the vnode
 > 	4. ufs_hashlock is already locked

 [...]

 The diff you append in the PR isn't quite right; it may lead to
 ddoubly used inode numbers under some circumstances. Could you
 perhaps try the following (which I will also shortly check into
 -current)?

 Thanks,

 - Frank


 Index: ufs/ffs/ffs_vfsops.c
 ===================================================================
 RCS file: /cvsroot/syssrc/sys/ufs/ffs/ffs_vfsops.c,v
 retrieving revision 1.67
 diff -c -r1.67 ffs_vfsops.c
 *** ffs_vfsops.c	2000/06/16 05:45:14	1.67
 --- ffs_vfsops.c	2000/06/27 23:04:26
 ***************
 *** 1008,1024 ****

   	ump = VFSTOUFS(mp);
   	dev = ump->um_dev;
 - 	do {
 - 		if ((*vpp = ufs_ihashget(dev, ino, LK_EXCLUSIVE)) != NULL)
 - 			return (0);
 - 	} while (lockmgr(&ufs_hashlock, LK_EXCLUSIVE|LK_SLEEPFAIL, 0));

   	/* Allocate a new vnode/inode. */
   	if ((error = getnewvnode(VT_UFS, mp, ffs_vnodeop_p, &vp)) != 0) {
   		*vpp = NULL;
 - 		lockmgr(&ufs_hashlock, LK_RELEASE, 0);
   		return (error);
   	}
   	/*
   	 * XXX MFS ends up here, too, to allocate an inode.  Should we
   	 * XXX create another pool for MFS inodes?
 --- 1008,1034 ----

   	ump = VFSTOUFS(mp);
   	dev = ump->um_dev;

 + 	if ((*vpp = ufs_ihashget(dev, ino, LK_EXCLUSIVE)) != NULL)
 + 		return (0);
 + 
   	/* Allocate a new vnode/inode. */
   	if ((error = getnewvnode(VT_UFS, mp, ffs_vnodeop_p, &vp)) != 0) {
   		*vpp = NULL;
   		return (error);
   	}
 + 
 + 	/*
 + 	 * If someone beat us to it while sleeping in getnewvnode(),
 + 	 * push back the freshly allocated vnode we don't need, and return.
 + 	 */
 + 	do {
 + 		if ((*vpp = ufs_ihashget(dev, ino, LK_EXCLUSIVE)) != NULL) {
 + 			vinsheadfree(vp);
 + 			return (0);
 + 		}
 + 	} while (lockmgr(&ufs_hashlock, LK_EXCLUSIVE|LK_SLEEPFAIL, 0));
 + 
   	/*
   	 * XXX MFS ends up here, too, to allocate an inode.  Should we
   	 * XXX create another pool for MFS inodes?
 Index: sys/vnode.h
 ===================================================================
 RCS file: /cvsroot/syssrc/sys/sys/vnode.h,v
 retrieving revision 1.79
 diff -c -r1.79 vnode.h
 *** vnode.h	2000/06/26 15:04:52	1.79
 --- vnode.h	2000/06/27 23:05:21
 ***************
 *** 486,491 ****
 --- 486,492 ----
   int 	cdevvp __P((dev_t dev, struct vnode **vpp));
   int 	getnewvnode __P((enum vtagtype tag, struct mount *mp,
   			 int (**vops) __P((void *)), struct vnode **vpp));
 + void	vinsheadfree __P((struct vnode *));
   int	getvnode __P((struct filedesc *fdp, int fd, struct file **fpp));
   void	vfs_getnewfsid __P((struct mount *));
   int	speedup_syncer __P((void));
 Index: kern/vfs_subr.c
 ===================================================================
 RCS file: /cvsroot/syssrc/sys/kern/vfs_subr.c,v
 retrieving revision 1.128
 diff -c -r1.128 vfs_subr.c
 *** vfs_subr.c	2000/06/10 18:44:44	1.128
 --- vfs_subr.c	2000/06/27 23:06:05
 ***************
 *** 542,547 ****
 --- 542,577 ----
   }

   /*
 +  * This is really just the reverse of getnewvnode(). Needed for
 +  * VFS_VGET functions who may need to push back a vnode in case
 +  * of a locking race.
 +  */
 + void
 + vinsheadfree(vp)
 + 	struct vnode *vp;
 + {
 + #ifdef DIAGNOSTIC
 + 	if (vp->v_usecount != 1)
 + 		panic("vinsheadfree: busy vnode");
 + #endif
 + 	vp->v_usecount--;
 + 	insmntque(vp, NULL);
 + 	vp->v_type = VBAD;
 + 
 + 	simple_lock(&vp->v_interlock);
 + 	/* 
 + 	 * Insert at head of LRU list
 + 	 */
 + 	simple_lock(&vnode_free_list_slock);
 + 	if (vp->v_holdcnt > 0)
 + 		TAILQ_INSERT_HEAD(&vnode_hold_list, vp, v_freelist);
 + 	else
 + 		TAILQ_INSERT_HEAD(&vnode_free_list, vp, v_freelist);
 + 	simple_unlock(&vnode_free_list_slock); 
 + 	simple_unlock(&vp->v_interlock);
 + }
 + 
 + /*
    * Move a vnode from one mount queue to another.
    */
   void

From: IWAMOTO Toshihiro <iwamoto@sat.t.u-tokyo.ac.jp>
To: frank@wins.uva.nl
Cc: iwamoto@sat.t.u-tokyo.ac.jp, gnats-bugs@gnats.netbsd.org
Subject: Re: kern/10448: softdep panics under heavy load at lockmgr
Date: Thu, 29 Jun 2000 14:17:53 +0900

 At Wed, 28 Jun 2000 01:29:43 +0200,
 Frank van der Linden <frank@wins.uva.nl> wrote:
 >> 	1. ffs_vget() calls getnewvnode() with ufs_hashlock locked
 >> 	2. getnewvnode() calls vgonel() to recycle a vnode entry
 >> 	3. flush_pagedep_deps() needs to call ffs_vget() to add
 >> 	   directory entries "." and ".." before flushing the vnode
 >> 	4. ufs_hashlock is already locked

 > The diff you append in the PR isn't quite right; it may lead to
 > ddoubly used inode numbers under some circumstances. Could you
 > perhaps try the following (which I will also shortly check into
 > -current)?

 Hmm..., I see. We should make sure that there are no duplicates, at
 the point of locking ufs_ihashlock. I'm going to try your patch.
 BTW, as getnewvnode() calls are waste of CPU time in case of cache hits,
 shouldn't we try ufs_ihashget() once before calling getnewvnode()?

 At a first glance over other ufs filesystems, it seems like that
 ext2fs_vget and lfs_vget (lfs_vcreate calls getnewvnode) can also
 cause the same panic, as getnewvnode() called from these functions can
 pick a softdep vnode.

 --
 IWAMOTO Toshihiro
State-Changed-From-To: open->closed 
State-Changed-By: chs 
State-Changed-When: Sun Feb 24 16:56:03 PST 2002 
State-Changed-Why:  
fixed by frank's changes long ago. 
>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.