NetBSD Problem Report #10448
Received: (qmail 3110 invoked from network); 26 Jun 2000 05:42:25 -0000
Message-Id: <y9jog4pc820.wl@vega.sat.t.u-tokyo.ac.jp>
Date: Mon, 26 Jun 2000 14:42:15 +0900
From: IWAMOTO Toshihiro <iwamoto@sat.t.u-tokyo.ac.jp>
Reply-To: IWAMOTO Toshihiro <iwamoto@sat.t.u-tokyo.ac.jp>
To: gnats-bugs@gnats.netbsd.org
Subject: softdep panics under heavy load at lockmgr
X-Send-Pr-Version: 3.95
>Number: 10448
>Category: kern
>Synopsis: softdep panics under heavy load at lockmgr
>Confidential: no
>Severity: critical
>Priority: medium
>Responsible: kern-bug-people
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Mon Jun 26 05:43:00 +0000 2000
>Closed-Date: Mon Feb 25 00:56:50 +0000 2002
>Last-Modified: Mon Feb 25 00:56:50 +0000 2002
>Originator: IWAMOTO Toshihiro
>Release: NetBSD-current of 2 weeks ago, also confirmed on recent 1.5_ALPHA
>Organization:
Univ. of Tokyo
>Environment:
System: NetBSD yomogi.sat.t.u-tokyo.ac.jp 1.4ZA NetBSD 1.4ZA (YOMOGI) #18: Mon Jun 12 17:50:29 JST 2000 iwamoto@yomogi.sat.t.u-tokyo.ac.jp:/usr/src/syssrc/sys/arch/i386/compile/YOMOGI i386
>Description:
With softdep enabled and heavy filesystem activity,
ffs_vget() can call itself, resulting in the following panic.
As shown below, the detailed situation of this panic is:
1. ffs_vget() calls getnewvnode() with ufs_hashlock locked
2. getnewvnode() calls vgonel() to recycle a vnode entry
3. flush_pagedep_deps() needs to call ffs_vget() to add
directory entries "." and ".." before flushing the vnode
4. ufs_hashlock is already locked
#10 0xc015ecc9 in panic (fmt=0xc0304380 "lockmgr: locking against myself")
at ../../../../kern/subr_prf.c:219
#11 0xc0151296 in lockmgr (lkp=0xc03b67e0, flags=34, interlkp=0x0)
at ../../../../kern/kern_lock.c:518
#12 0xc023c43f in ffs_vget (mp=0xc08c3600, ino=1075358, vpp=0xd88f3a64)
at ../../../../ufs/ffs/ffs_vfsops.c:968
#13 0xc02ea224 in flush_pagedep_deps (pvp=0xd8765448, mp=0xc08c3600,
diraddhdp=0xc0a675b0) at ../../../../../gnu/sys/ufs/ffs/ffs_softdep.c:4296
#14 0xc02e9cad in softdep_sync_metadata (v=0xd88f3b50)
at ../../../../../gnu/sys/ufs/ffs/ffs_softdep.c:4026
#15 0xc023d45d in ffs_fsync (v=0xd88f3b50)
at ../../../../ufs/ffs/ffs_vnops.c:312
#16 0xc0176f16 in vinvalbuf (vp=0xd8765448, flags=1, cred=0xffffffff,
p=0xd88ec018, slpflag=0, slptimeo=0) at ../../../../sys/vnode_if.h:449
#17 0xc0177bfd in vclean (vp=0xd8765448, flags=8, p=0xd88ec018)
at ../../../../kern/vfs_subr.c:1424
#18 0xc0177ddb in vgonel (vp=0xd8765448, p=0xd88ec018)
at ../../../../kern/vfs_subr.c:1554
#19 0xc0176cbd in getnewvnode (tag=VT_UFS, mp=0xc08c3600, vops=0xc0874200,
vpp=0xd88f3c4c) at ../../../../kern/vfs_subr.c:508
#20 0xc023c45e in ffs_vget (mp=0xc08c3600, ino=1394840, vpp=0xd88f3cb0)
at ../../../../ufs/ffs/ffs_vfsops.c:971
#21 0xc0235dae in ffs_valloc (v=0xd88f3cb4)
at ../../../../ufs/ffs/ffs_alloc.c:605
#22 0xc024c700 in ufs_makeinode (mode=33188, dvp=0xd8793014, vpp=0xd88f3ee8,
cnp=0xd88f3efc) at ../../../../sys/vnode_if.h:933
#23 0xc0249a82 in ufs_create (v=0xd88f3e08)
---Type <return> to continue, or q <return> to quit---
at ../../../../ufs/ufs/ufs_vnops.c:117
#24 0xc017df98 in vn_open (ndp=0xd88f3ed8, fmode=1550, cmode=420)
at ../../../../sys/vnode_if.h:96
#25 0xc017a3ae in sys_open (p=0xd88ec018, v=0xd88f3f88, retval=0xd88f3f80)
at ../../../../kern/vfs_syscalls.c:1004
#26 0xc02699e7 in syscall (frame={tf_es = 134873119, tf_ds = -1078001633,
tf_edi = 0, tf_esi = 134882304, tf_ebp = -1077946548, tf_ebx = 1549,
tf_edx = -1, tf_ecx = -35, tf_eax = 5, tf_trapno = 3, tf_err = 2,
tf_eip = 134730351, tf_cs = 23, tf_eflags = 514, tf_esp = -1077946912,
tf_ss = 31, tf_vm86_es = 0, tf_vm86_ds = 0, tf_vm86_fs = 0,
tf_vm86_gs = 0}) at ../../../../arch/i386/i386/trap.c:766
#27 0xc0100dc1 in syscall1 ()
can not access 0xbfbfd74c, invalid translation (invalid PDE)
can not access 0xbfbfd74c, invalid translation (invalid PDE)
>How-To-Repeat:
To reproduce the above panic, an incomplete directory vnode
should be vgonel()'ed from getnewvnode().
Such a situation can be easily reproducable.
1. Boot with a large amount (~50MB) of buffers and the default
value of numvnodes (778).
2. Generate a heavy disk activity by issueing the following commands
simultaneously.
$ cd /usr/pkgsrc/www/mozilla; make (tarball extraction)
# cd /usr/src/syssrc; cvs update -Pd
>Fix:
A workaround is to configure a large number of vnodes.
I think a lock to ufs_hashlock is unnecessary when calling
getnewvnode(), and the patch below seems to work.
Am I missing the point?
--- /sys/ufs/ffs/ffs_vfsops.c Sun Jun 25 16:57:51 2000
+++ ./ffs_vfsops.c Mon Jun 26 13:47:58 2000
@@ -1014,15 +1014,16 @@
} while (lockmgr(&ufs_hashlock, LK_EXCLUSIVE|LK_SLEEPFAIL, 0));
/* Allocate a new vnode/inode. */
+ lockmgr(&ufs_hashlock, LK_RELEASE, 0);
if ((error = getnewvnode(VT_UFS, mp, ffs_vnodeop_p, &vp)) != 0) {
*vpp = NULL;
- lockmgr(&ufs_hashlock, LK_RELEASE, 0);
return (error);
}
/*
* XXX MFS ends up here, too, to allocate an inode. Should we
* XXX create another pool for MFS inodes?
*/
+ lockmgr(&ufs_hashlock, LK_EXCLUSIVE|LK_SLEEPFAIL, 0);
ip = pool_get(&ffs_inode_pool, PR_WAITOK);
memset((caddr_t)ip, 0, sizeof(struct inode));
vp->v_data = ip;
>Release-Note:
>Audit-Trail:
From: Frank van der Linden <frank@wins.uva.nl>
To: IWAMOTO Toshihiro <iwamoto@sat.t.u-tokyo.ac.jp>
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: kern/10448: softdep panics under heavy load at lockmgr
Date: Wed, 28 Jun 2000 01:29:43 +0200
On Mon, Jun 26, 2000 at 02:42:15PM +0900, IWAMOTO Toshihiro wrote:
> With softdep enabled and heavy filesystem activity,
> ffs_vget() can call itself, resulting in the following panic.
>
> As shown below, the detailed situation of this panic is:
>
> 1. ffs_vget() calls getnewvnode() with ufs_hashlock locked
> 2. getnewvnode() calls vgonel() to recycle a vnode entry
> 3. flush_pagedep_deps() needs to call ffs_vget() to add
> directory entries "." and ".." before flushing the vnode
> 4. ufs_hashlock is already locked
[...]
The diff you append in the PR isn't quite right; it may lead to
ddoubly used inode numbers under some circumstances. Could you
perhaps try the following (which I will also shortly check into
-current)?
Thanks,
- Frank
Index: ufs/ffs/ffs_vfsops.c
===================================================================
RCS file: /cvsroot/syssrc/sys/ufs/ffs/ffs_vfsops.c,v
retrieving revision 1.67
diff -c -r1.67 ffs_vfsops.c
*** ffs_vfsops.c 2000/06/16 05:45:14 1.67
--- ffs_vfsops.c 2000/06/27 23:04:26
***************
*** 1008,1024 ****
ump = VFSTOUFS(mp);
dev = ump->um_dev;
- do {
- if ((*vpp = ufs_ihashget(dev, ino, LK_EXCLUSIVE)) != NULL)
- return (0);
- } while (lockmgr(&ufs_hashlock, LK_EXCLUSIVE|LK_SLEEPFAIL, 0));
/* Allocate a new vnode/inode. */
if ((error = getnewvnode(VT_UFS, mp, ffs_vnodeop_p, &vp)) != 0) {
*vpp = NULL;
- lockmgr(&ufs_hashlock, LK_RELEASE, 0);
return (error);
}
/*
* XXX MFS ends up here, too, to allocate an inode. Should we
* XXX create another pool for MFS inodes?
--- 1008,1034 ----
ump = VFSTOUFS(mp);
dev = ump->um_dev;
+ if ((*vpp = ufs_ihashget(dev, ino, LK_EXCLUSIVE)) != NULL)
+ return (0);
+
/* Allocate a new vnode/inode. */
if ((error = getnewvnode(VT_UFS, mp, ffs_vnodeop_p, &vp)) != 0) {
*vpp = NULL;
return (error);
}
+
+ /*
+ * If someone beat us to it while sleeping in getnewvnode(),
+ * push back the freshly allocated vnode we don't need, and return.
+ */
+ do {
+ if ((*vpp = ufs_ihashget(dev, ino, LK_EXCLUSIVE)) != NULL) {
+ vinsheadfree(vp);
+ return (0);
+ }
+ } while (lockmgr(&ufs_hashlock, LK_EXCLUSIVE|LK_SLEEPFAIL, 0));
+
/*
* XXX MFS ends up here, too, to allocate an inode. Should we
* XXX create another pool for MFS inodes?
Index: sys/vnode.h
===================================================================
RCS file: /cvsroot/syssrc/sys/sys/vnode.h,v
retrieving revision 1.79
diff -c -r1.79 vnode.h
*** vnode.h 2000/06/26 15:04:52 1.79
--- vnode.h 2000/06/27 23:05:21
***************
*** 486,491 ****
--- 486,492 ----
int cdevvp __P((dev_t dev, struct vnode **vpp));
int getnewvnode __P((enum vtagtype tag, struct mount *mp,
int (**vops) __P((void *)), struct vnode **vpp));
+ void vinsheadfree __P((struct vnode *));
int getvnode __P((struct filedesc *fdp, int fd, struct file **fpp));
void vfs_getnewfsid __P((struct mount *));
int speedup_syncer __P((void));
Index: kern/vfs_subr.c
===================================================================
RCS file: /cvsroot/syssrc/sys/kern/vfs_subr.c,v
retrieving revision 1.128
diff -c -r1.128 vfs_subr.c
*** vfs_subr.c 2000/06/10 18:44:44 1.128
--- vfs_subr.c 2000/06/27 23:06:05
***************
*** 542,547 ****
--- 542,577 ----
}
/*
+ * This is really just the reverse of getnewvnode(). Needed for
+ * VFS_VGET functions who may need to push back a vnode in case
+ * of a locking race.
+ */
+ void
+ vinsheadfree(vp)
+ struct vnode *vp;
+ {
+ #ifdef DIAGNOSTIC
+ if (vp->v_usecount != 1)
+ panic("vinsheadfree: busy vnode");
+ #endif
+ vp->v_usecount--;
+ insmntque(vp, NULL);
+ vp->v_type = VBAD;
+
+ simple_lock(&vp->v_interlock);
+ /*
+ * Insert at head of LRU list
+ */
+ simple_lock(&vnode_free_list_slock);
+ if (vp->v_holdcnt > 0)
+ TAILQ_INSERT_HEAD(&vnode_hold_list, vp, v_freelist);
+ else
+ TAILQ_INSERT_HEAD(&vnode_free_list, vp, v_freelist);
+ simple_unlock(&vnode_free_list_slock);
+ simple_unlock(&vp->v_interlock);
+ }
+
+ /*
* Move a vnode from one mount queue to another.
*/
void
From: IWAMOTO Toshihiro <iwamoto@sat.t.u-tokyo.ac.jp>
To: frank@wins.uva.nl
Cc: iwamoto@sat.t.u-tokyo.ac.jp, gnats-bugs@gnats.netbsd.org
Subject: Re: kern/10448: softdep panics under heavy load at lockmgr
Date: Thu, 29 Jun 2000 14:17:53 +0900
At Wed, 28 Jun 2000 01:29:43 +0200,
Frank van der Linden <frank@wins.uva.nl> wrote:
>> 1. ffs_vget() calls getnewvnode() with ufs_hashlock locked
>> 2. getnewvnode() calls vgonel() to recycle a vnode entry
>> 3. flush_pagedep_deps() needs to call ffs_vget() to add
>> directory entries "." and ".." before flushing the vnode
>> 4. ufs_hashlock is already locked
> The diff you append in the PR isn't quite right; it may lead to
> ddoubly used inode numbers under some circumstances. Could you
> perhaps try the following (which I will also shortly check into
> -current)?
Hmm..., I see. We should make sure that there are no duplicates, at
the point of locking ufs_ihashlock. I'm going to try your patch.
BTW, as getnewvnode() calls are waste of CPU time in case of cache hits,
shouldn't we try ufs_ihashget() once before calling getnewvnode()?
At a first glance over other ufs filesystems, it seems like that
ext2fs_vget and lfs_vget (lfs_vcreate calls getnewvnode) can also
cause the same panic, as getnewvnode() called from these functions can
pick a softdep vnode.
--
IWAMOTO Toshihiro
State-Changed-From-To: open->closed
State-Changed-By: chs
State-Changed-When: Sun Feb 24 16:56:03 PST 2002
State-Changed-Why:
fixed by frank's changes long ago.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.