NetBSD Problem Report #37191

From martin@duskware.de  Wed Oct 24 08:05:19 2007
Return-Path: <martin@duskware.de>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by narn.NetBSD.org (Postfix) with ESMTP id 7882763B8FE
	for <gnats-bugs@gnats.netbsd.org>; Wed, 24 Oct 2007 08:05:19 +0000 (UTC)
Message-Id: <20071023232212.0A7BF63B8FE@narn.NetBSD.org>
Date: Tue, 23 Oct 2007 23:22:12 +0000 (UTC)
From: ad@netbsd.org
Reply-To: ad@netbsd.org
To: netbsd-bugs-owner@NetBSD.org
Subject: softdep: locking against myself
X-Send-Pr-Version: www-1.0

>Number:         37191
>Category:       kern
>Synopsis:       softdep: locking against myself
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Oct 24 08:10:00 +0000 2007
>Closed-Date:    Wed Apr 01 04:09:04 +0000 2009
>Last-Modified:  Wed Apr 01 04:09:04 +0000 2009
>Originator:     Andrew Doran
>Release:        4.99.34
>Organization:
The NetBSD Project
>Environment:
N/A
>Description:
Locking against myself panic from softdep. This is with a kernel
from the vmlocking branch, but it probably also applies to HEAD:

panic: lockmgr: locking against myself (type *0* flags 400, sharecount 0, exclusivecount 1, recurselevel 0, waitcount 0, wmesg vnlock, lock_addr 0xc020632c, unlock_addr 0xc0206248)

Stopped in pid 20214.1 (nbconfig) at    netbsd:breakpoint+0x1:  ret
db{0}> bt
breakpoint(c04412b5,cc09f298,cc09f28c,c029fad7,30303498) at netbsd:breakpoint+0x1
panic(c043d038,cc09f35a,c043d432,cc09f2c4,0) at netbsd:panic+0xc1
lockmgr(cf8b7320,10002,cf8b728c,0,0) at netbsd:lockmgr+0x4c1
ffs_lock(cc09f49c,0,0,d83c9c40,0) at netbsd:ffs_lock+0xdc   
VOP_LOCK(cf8b728c,10002,c04d2760,c04b478c,cf8b728c) at netbsd:VOP_LOCK+0x7b
vn_lock(cf8b728c,10002,0,488aa,488aa) at netbsd:vn_lock+0x9c
vget(cf8b728c,10002,80,0,0) at netbsd:vget+0x1d2
ufs_ihashget(c,488aa,0,2,0) at netbsd:ufs_ihashget+0x94
ffs_vget(c1b74000,488aa,0,cc09f65c,d83c9c40) at netbsd:ffs_vget+0x42
softdep_sync_metadata(cdbc7358,cdbc7358,0,c02775ad,c04b5528) at netbsd:softdep_sync_metadata+0x440
ffs_full_fsync(cdbc7358,5,1,c027aa0b,100001) at netbsd:ffs_full_fsync+0x21d
ffs_fsync(cc09f794,0,cc09f7bc,c02df612,cc09f79c) at netbsd:ffs_fsync+0x67  
VOP_FSYNC(cdbc7358,ffffffff,5,0,0) at netbsd:VOP_FSYNC+0x9f
vinvalbuf(cdbc7358,1,ffffffff,d83c9c40,0) at netbsd:vinvalbuf+0xb6
vclean(cdbc7358,8,0,c027a1b4,0) at netbsd:vclean+0x163
getcleanvnode(c04d3b2c,200000,0,c02789b9,a7c0) at netbsd:getcleanvnode+0x143
getnewvnode(1,c1b74000,c1a38700,cc09f934,1f) at netbsd:getnewvnode+0xe4
ffs_vget(c1b74000,49ac5,0,cc09fbb8,c00) at netbsd:ffs_vget+0x72
ffs_valloc(cf8b728c,81a4,cad640c0,cc09fbb8,c1c4d604) at netbsd:ffs_valloc+0x5d3
ufs_makeinode(81a4,cf8b728c,cc09fbb8,cc09fbcc,cf8b728c) at netbsd:ufs_makeinode+0x71
ufs_create(cc09fab0,c02673a0,0,0,0) at netbsd:ufs_create+0x55
VOP_CREATE(cf8b728c,cc09fbb8,cc09fbcc,cc09fb00,0) at netbsd:VOP_CREATE+0x88
vn_open(cc09fba4,602,1a4,3,9c3c) at netbsd:vn_open+0x115
sys_open(d83c9c40,cc09fc50,cc09fc48,cc09fc80,c03167af) at netbsd:sys_open+0xba
syscall_plain() at netbsd:syscall_plain+0x134
--- syscall (number 5) ---

This is with kern.maxvnodes set to 2000. The newly created file
needs a vnode, so we reclaim an inactive one from ffs. That means
flushing any data + softdep dependencies from the vnode. It looks
like softdep is trying to push out a modification to the parent
directory (?) of the vnode which we are flushing. The directory is
already locked because the flush is the result of creating a file
in it.

>How-To-Repeat:
Exercise a file system with softdep enabled. Set kern.maxvnodes to a
value low enough to not make the test I/O bound, but enough to push up
the rate at which vnodes are recycled.


>Fix:
Journaling :-)

>Release-Note:

>Audit-Trail:
From: Antti Kantee <pooka@cs.hut.fi>
To: gnats-bugs@NetBSD.org
Cc: ad@netbsd.org
Subject: Re: kern/37191: softdep: locking against myself
Date: Wed, 24 Oct 2007 12:36:43 +0300

 This is kind of the same problem that puffs has if the file server makes a
 system call which tries to clean a puffs vnode for the same file system.
 There I am just very careful not to do blocking operations if VXLOCK
 is set.  That would obviously not work here.

 Anyway, we were discussing with Bill (not groo) about moving the whole
 vnode garbage collection to a separate context to prevent weird corner
 case deadlocks.

 -- 
 Antti Kantee <pooka@iki.fi>                     Of course he runs NetBSD
 http://www.iki.fi/pooka/                          http://www.NetBSD.org/
     "la qualité la plus indispensable du cuisinier est l'exactitude"

From: Andrew Doran <ad@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/37191 CVS commit: src
Date: Sun, 22 Feb 2009 20:28:07 +0000 (UTC)

 Module Name:	src
 Committed By:	ad
 Date:		Sun Feb 22 20:28:07 UTC 2009

 Modified Files:
 	src/doc: CHANGES
 	src/lib/libp2k: p2k.c
 	src/sbin/fsck_lfs: lfs.c
 	src/sbin/mount: mount.8
 	src/sbin/newfs_lfs: make_lfs.c
 	src/sbin/tunefs: tunefs.8 tunefs.c
 	src/sys/arch/vax/conf: VAX780
 	src/sys/conf: files
 	src/sys/kern: sys_aio.c vfs_bio.c vfs_subr.c vfs_syscalls.c
 	src/sys/miscfs/specfs: spec_vnops.c
 	src/sys/miscfs/syncfs: sync_subr.c
 	src/sys/modules/ffs: Makefile
 	src/sys/rump/fs/lib/libffs: Makefile
 	src/sys/rump/include/rump: rump.h
 	src/sys/rump/librump/rumpvfs: rump_vfs.c vm_vfs.c
 	src/sys/sys: buf.h vnode.h
 	src/sys/ufs: files.ufs
 	src/sys/ufs/ffs: ffs_alloc.c ffs_balloc.c ffs_extern.h ffs_inode.c
 	    ffs_snapshot.c ffs_vfsops.c ffs_vnops.c ffs_wapbl.c
 	src/sys/ufs/lfs: lfs_rfw.c lfs_vfsops.c lfs_vnops.c
 	src/sys/ufs/ufs: inode.h ufs_dirhash.c ufs_extern.h ufs_inode.c
 	    ufs_lookup.c ufs_readwrite.c ufs_vnops.c ufs_wapbl.c
 	src/sys/uvm: uvm_pager.c
 Removed Files:
 	src/sys/rump/librump/rumpkern/opt: opt_softdep.h
 	src/sys/ufs/ffs: ffs_softdep.c ffs_softdep.stub.c softdep.h

 Log Message:
 PR kern/26878 FFSv2 + softdep = livelock (no free ram)
 PR kern/16942 panic with softdep and quotas
 PR kern/19565 panic: softdep_write_inodeblock: indirect pointer #1 mismatch
 PR kern/26274 softdep panic: allocdirect_merge: ...
 PR kern/26374 Long delay before non-root users can write to softdep partitions
 PR kern/28621 1.6.x "vp != NULL" panic in ffs_softdep.c:4653 while unmounting a softdep (+quota) filesystem
 PR kern/29513 FFS+Softdep panic with unfsck-able file-corruption
 PR kern/31544 The ffs softdep code appears to fail to write dirty bits to disk
 PR kern/31981 stopping scsi disk can cause panic (softdep)
 PR kern/32116 kernel panic in softdep (assertion failure)
 PR kern/32532 softdep_trackbufs deadlock
 PR kern/37191 softdep: locking against myself
 PR kern/40474 Kernel panic after remounting raid root with softdep

 Retire softdep, pass 2. As discussed and later formally announced on the
 mailing lists.


 To generate a diff of this commit:
 cvs rdiff -r1.1191 -r1.1192 src/doc/CHANGES
 cvs rdiff -r1.8 -r1.9 src/lib/libp2k/p2k.c
 cvs rdiff -r1.29 -r1.30 src/sbin/fsck_lfs/lfs.c
 cvs rdiff -r1.65 -r1.66 src/sbin/mount/mount.8
 cvs rdiff -r1.13 -r1.14 src/sbin/newfs_lfs/make_lfs.c
 cvs rdiff -r1.37 -r1.38 src/sbin/tunefs/tunefs.8 src/sbin/tunefs/tunefs.c
 cvs rdiff -r1.1 -r1.2 src/sys/arch/vax/conf/VAX780
 cvs rdiff -r1.942 -r1.943 src/sys/conf/files
 cvs rdiff -r1.22 -r1.23 src/sys/kern/sys_aio.c
 cvs rdiff -r1.215 -r1.216 src/sys/kern/vfs_bio.c
 cvs rdiff -r1.368 -r1.369 src/sys/kern/vfs_subr.c
 cvs rdiff -r1.388 -r1.389 src/sys/kern/vfs_syscalls.c
 cvs rdiff -r1.122 -r1.123 src/sys/miscfs/specfs/spec_vnops.c
 cvs rdiff -r1.36 -r1.37 src/sys/miscfs/syncfs/sync_subr.c
 cvs rdiff -r1.2 -r1.3 src/sys/modules/ffs/Makefile
 cvs rdiff -r1.6 -r1.7 src/sys/rump/fs/lib/libffs/Makefile
 cvs rdiff -r1.9 -r1.10 src/sys/rump/include/rump/rump.h
 cvs rdiff -r1.1 -r0 src/sys/rump/librump/rumpkern/opt/opt_softdep.h
 cvs rdiff -r1.12 -r1.13 src/sys/rump/librump/rumpvfs/rump_vfs.c
 cvs rdiff -r1.3 -r1.4 src/sys/rump/librump/rumpvfs/vm_vfs.c
 cvs rdiff -r1.110 -r1.111 src/sys/sys/buf.h
 cvs rdiff -r1.200 -r1.201 src/sys/sys/vnode.h
 cvs rdiff -r1.18 -r1.19 src/sys/ufs/files.ufs
 cvs rdiff -r1.121 -r1.122 src/sys/ufs/ffs/ffs_alloc.c
 cvs rdiff -r1.51 -r1.52 src/sys/ufs/ffs/ffs_balloc.c
 cvs rdiff -r1.74 -r1.75 src/sys/ufs/ffs/ffs_extern.h
 cvs rdiff -r1.102 -r1.103 src/sys/ufs/ffs/ffs_inode.c
 cvs rdiff -r1.91 -r1.92 src/sys/ufs/ffs/ffs_snapshot.c
 cvs rdiff -r1.116 -r0 src/sys/ufs/ffs/ffs_softdep.c
 cvs rdiff -r1.23 -r0 src/sys/ufs/ffs/ffs_softdep.stub.c
 cvs rdiff -r1.242 -r1.243 src/sys/ufs/ffs/ffs_vfsops.c
 cvs rdiff -r1.110 -r1.111 src/sys/ufs/ffs/ffs_vnops.c
 cvs rdiff -r1.11 -r1.12 src/sys/ufs/ffs/ffs_wapbl.c
 cvs rdiff -r1.11 -r0 src/sys/ufs/ffs/softdep.h
 cvs rdiff -r1.11 -r1.12 src/sys/ufs/lfs/lfs_rfw.c
 cvs rdiff -r1.269 -r1.270 src/sys/ufs/lfs/lfs_vfsops.c
 cvs rdiff -r1.219 -r1.220 src/sys/ufs/lfs/lfs_vnops.c
 cvs rdiff -r1.55 -r1.56 src/sys/ufs/ufs/inode.h
 cvs rdiff -r1.27 -r1.28 src/sys/ufs/ufs/ufs_dirhash.c
 cvs rdiff -r1.60 -r1.61 src/sys/ufs/ufs/ufs_extern.h
 cvs rdiff -r1.77 -r1.78 src/sys/ufs/ufs/ufs_inode.c
 cvs rdiff -r1.100 -r1.101 src/sys/ufs/ufs/ufs_lookup.c
 cvs rdiff -r1.93 -r1.94 src/sys/ufs/ufs/ufs_readwrite.c
 cvs rdiff -r1.172 -r1.173 src/sys/ufs/ufs/ufs_vnops.c
 cvs rdiff -r1.4 -r1.5 src/sys/ufs/ufs/ufs_wapbl.c
 cvs rdiff -r1.93 -r1.94 src/sys/uvm/uvm_pager.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->closed
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Wed, 01 Apr 2009 04:09:04 +0000
State-Changed-Why:
softdep (softupdates) has been removed.


>Unformatted:

Home	PR Database Search