NetBSD Problem Report #28621

From woods@building.weird.com  Sun Dec 12 03:32:55 2004
Return-Path: <woods@building.weird.com>
Received: from building.weird.com (building.weird.com [204.92.254.24])
	by narn.netbsd.org (Postfix) with ESMTP id F2E61251EB0
	for <gnats-bugs@gnats.netbsd.org>; Sun, 12 Dec 2004 03:32:54 +0000 (UTC)
Message-Id: <m1CdKTe-0024gMC@building.weird.com>
Date: Sat, 11 Dec 2004 22:32:54 -0500 (EST)
From: "Greg A. Woods" <woods@weird.com>
Reply-To: "Greg A. Woods" <woods@planix.com>
To: gnats-bugs@netbsd.org
Subject: 1.6.x "vp != NULL" panic in ffs_softdep.c:4653 while unmounting a softdep (+quota) filesystem
X-Send-Pr-Version: 3.95

>Number:         28621
>Category:       kern
>Synopsis:       1.6.x "vp != NULL" panic in ffs_softdep.c:4653 while unmounting a softdep (+quota) filesystem
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Dec 12 03:33:00 +0000 2004
>Closed-Date:    Wed Apr 01 04:08:33 +0000 2009
>Last-Modified:  Wed Apr 01 04:08:33 +0000 2009
>Originator:     Greg A. Woods
>Release:        NetBSD 1.6.2_STABLE
>Organization:
Planix, Inc.; Toronto, Ontario; Canada
>Environment:
System: NetBSD 1.6.2_STABLE
Architecture: i386
Machine: i386
>Description:

	I was unmounting a filesystem after playing with quotas.  It
	was mounted with softdeps, but I had just run "quotaoff -v -a",
	so quotas should have been disabled by then.

	there hadn't been any file I/O, other than by the kernel to the
	quota file, on that filesystem for quite some time....

>How-To-Repeat:

panic: kernel diagnostic assertion "vp != NULL" failed: file "/building/work/woods/m-NetBSD-1.6/sys/ufs/ffs/ffs_softdep.c", line 4653

Stopped in pid 27332 (umount) at        cpu_Debugger+0x4:       movl    %ebp,%esp
db> where
No such command  (someday I'm going to make that an alias! :-)
db> trace
cpu_Debugger(c04ca5ed,ffffffff,e412a1d4,c025789d,e4122b64) at cpu_Debugger+0x4
panic(c062b1e0,c04ca5ed,c04cbee7,c04cbe80,122d) at panic+0xb0
__main(c04ca5ed,c04cbe80,122d,c04cbee7,e554699c) at __main
flush_inodedep_deps(c1e14000,454401,e4122c20,c029584f,e412a1f8) at flush_inodedep_deps+0x3c
softdep_sync_metadata(e4122dac,0,e4122c90,c029c9f2) at softdep_sync_metadata+0x2fb
ffs_full_fsync(e4122dac,0,e4122d10,c0251fa9,e4122dac) at ffs_full_fsync+0x260
ffs_fsync(e4122dac,20002,0,10,0) at ffs_fsync+0x3f
ffs_flushfiles(c225f400,0,e4244cac,c02c2bb1,0) at ffs_flushfiles+0xfd
softdep_flushfiles(c225f400,0,e4244cac,c02c824d,0) at softdep_flushfiles+0x56
ffs_unmount(c225f400,0,e4244cac,e4244cac,0) at ffs_unmount+0x3d
dounmount(c225f400,0,e4244cac,0,e4122f80) at dounmount+0xea
sys_unmount(e4244cac,e4122f80,e4122f78,c033dde0) at sys_unmount+0xf5
syscall_plain(1f,1f,1f,1f,bfbfcfb4) at syscall_plain+0xa7
db> 

I (dyoung@netbsd.org) see this, too, on a Soekris board. Below is the Soekris
panic.  I ran umount(8) with the attached script.  This is a very -current
kernel, 2.99.11.  I see this often.  I can provide console access to a Soekris
board where it occurs.  It is really too bad if I cannot use softdep because
it speeds up the script by several minutes. -dcy

  Stopped in pid 26781.1 (umount) at      netbsd:cpu_Debugger+0x4:        popl    %
  ebp
  db> trace/u
  cpu_Debugger(0,c385e934,2,c3978d0c,c0271fb9) at netbsd:cpu_Debugger+0x4
  panic(c02c50a0,c029f1fc,c02a206d,c02b3340,1408) at netbsd:panic+0xa9
  __assert(c029f1fc,c02b3340,1408,c02a206d,0) at netbsd:__assert+0x19
  flush_inodedep_deps(c0546800,80,0,0,0) at netbsd:flush_inodedep_deps+0x3b
  softdep_sync_metadata(c3978e1c,0,0,4,c385e934) at netbsd:softdep_sync_metadata+0
  x6b
  ffs_full_fsync(c3978e1c,c3a5a2b8,c05f7000,c3a5c04c,0) at netbsd:ffs_full_fsync+0
  x20e
  ffs_fsync(c3978e1c,c02800a0,c385e934,c2e3f0fc,1) at netbsd:ffs_fsync+0x48
  VOP_FSYNC(c385e934,c2e3f0fc,1,0,0) at netbsd:VOP_FSYNC+0x4c
  softdep_flushworklist(c05f7000,c3978ea4,c3906008,0,0) at netbsd:softdep_flushwor
  klist+0x87
  ffs_sync(c05f7000,1,c2e3f0fc,c3906008,0) at netbsd:ffs_sync+0x10e
  dounmount(c05f7000,0,c3906008,c3906008,bfbfe930) at netbsd:dounmount+0xc3
  sys_unmount(c36efe70,c3978f70,c3978f68,c02c9ce8,0) at netbsd:sys_unmount+0xf2
  syscall_plain() at netbsd:syscall_plain+0xc2
  --- syscall (number 22) ---
  ?(4807053c,bfbfedbc,0,8068000,8065d68) at 0x4807b78b
  Bad user frame pointer: 0x48078200
  db>

>Fix:

	unknown

	In my application, a very unpleasant workaround is to disable
	softdep. -dcy

>Release-Note:

>Audit-Trail:

From: David Young <dyoung@pobox.com>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/28621: 1.6.x "vp != NULL" crash in ffs_sfotdep.c:4653 while unmounting a softdep (+quota) filesystem
Date: Fri, 7 Jan 2005 22:39:55 -0600

 I (dyoung@netbsd.org) see this, too, on a Soekris board. Below is the
 Soekris panic.  I ran umount(8) with the attached script.  This is a very
 -current kernel, 2.99.11.  I see this often.  I can provide console access
 to a Soekris board where it occurs.  It is really too bad if I cannot
 use softdep because it speeds up the script by several minutes. -dcy

   Stopped in pid 26781.1 (umount) at      netbsd:cpu_Debugger+0x4:        popl  
   %
   ebp
   db> trace/u
   cpu_Debugger(0,c385e934,2,c3978d0c,c0271fb9) at netbsd:cpu_Debugger+0x4
   panic(c02c50a0,c029f1fc,c02a206d,c02b3340,1408) at netbsd:panic+0xa9
   __assert(c029f1fc,c02b3340,1408,c02a206d,0) at netbsd:__assert+0x19
   flush_inodedep_deps(c0546800,80,0,0,0) at netbsd:flush_inodedep_deps+0x3b
   softdep_sync_metadata(c3978e1c,0,0,4,c385e934) at netbsd:softdep_sync_metadata
 +0
   x6b
   ffs_full_fsync(c3978e1c,c3a5a2b8,c05f7000,c3a5c04c,0) at netbsd:ffs_full_fsync
 +0
   x20e
   ffs_fsync(c3978e1c,c02800a0,c385e934,c2e3f0fc,1) at netbsd:ffs_fsync+0x48
   VOP_FSYNC(c385e934,c2e3f0fc,1,0,0) at netbsd:VOP_FSYNC+0x4c
   softdep_flushworklist(c05f7000,c3978ea4,c3906008,0,0) at netbsd:softdep_flushw
 or
   klist+0x87
   ffs_sync(c05f7000,1,c2e3f0fc,c3906008,0) at netbsd:ffs_sync+0x10e
   dounmount(c05f7000,0,c3906008,c3906008,bfbfe930) at netbsd:dounmount+0xc3
   sys_unmount(c36efe70,c3978f70,c3978f68,c02c9ce8,0) at netbsd:sys_unmount+0xf2
   syscall_plain() at netbsd:syscall_plain+0xc2
   --- syscall (number 22) ---
   ?(4807053c,bfbfedbc,0,8068000,8065d68) at 0x4807b78b
   Bad user frame pointer: 0x48078200
   db>

 *************
 *************

 This is a representative mount(8) output from one of the Soekris boxes:

 # mount 
 /dev/wd0a on / type ffs (read-only, local)
 mfs:10 on /dev type mfs (synchronous, local)
 /etc on /permanent/etc type null (local)
 mfs:1390 on /etc type mfs (synchronous, noatime, local)
 /home on /permanent/home type null (local)
 mfs:1398 on /home type mfs (synchronous, noatime, local)
 /tmp on /permanent/tmp type null (local)
 mfs:1420 on /tmp type mfs (synchronous, noexec, nosuid, nodev, noatime, local)
 /var on /permanent/var type null (local)
 mfs:1429 on /var type mfs (synchronous, noatime, local)

 *************
 *************

 Here is the script:

 #!/bin/sh
 # $Id: upgrade 2288 2004-12-23 07:16:30Z dyoung $

 [ "$(whoami)" = "root" ] || { 
 	echo "This script is intended to be run as root on a CUW node." 1>&2; 
 	exit 1; 
 }
 [ $1 ] || { echo "Usage: $0 user@host:/path/to/tar" 1>&2; exit 1; }

 gripe () {
 	echo "$*" 1>&2
 }

 bomb () {
 	gripe "$*"
 	cd
 	if mount | grep -q "on /mnt " ; then
 		umount /mnt
 	fi
 	exit 1
 }

 attempt () {
 	eval $* || bomb "Upgrade failed on $* [$?]"
 }

 set -u

 extract_scp_format="\([^@]*\)@\([^:]*\):\(.*\)"
 user=$(echo $1 | sed -n -e "s/$extract_scp_format/\1/p")
 host=$(echo $1 | sed -n -e "s/$extract_scp_format/\2/p")
 tar=$(echo $1 | sed -n -e "s/$extract_scp_format/\3/p")

 current=$(mount | sed -n -e "s/^[^ ]*\([ae]\) on \/ .*/\1/p")
 if [ $current = 'a' ] ; then
 	setactive=1
 	dev="/dev/wd0e"
 	rdev="/dev/rwd0e"
 elif [ $current = 'e' ] ; then
 	setactive=0
 	dev="/dev/wd0a"
 	rdev="/dev/rwd0a"
 fi
 ddev="/dev/rwd0d"

 echo "Preparing for upgrade on $dev."

 attempt newfs $rdev
 attempt mount -o noatime $dev /mnt
 attempt cd /mnt

 echo "Installing the upgrade."
 attempt "ssh $user@$host cat $tar | pax -pe -r -z"

 echo "Updating etc/fstab"
 fstab=$(mktemp /var/tmp/$(basename $0).fstab.XXXXXX)
 sed -e "s|/dev/wd0[ae] / \(.*\)|$dev / \1|" /mnt/etc/fstab > $fstab
 attempt install -o root -g wheel -m 0644 $fstab /mnt/etc/fstab
 rm $fstab

 echo "Updating bootstrap"
 attempt fdisk -f -a -$setactive $ddev
 attempt mbrlabel -frw $ddev
 attempt installboot -o console=com0kbd,speed=19200 $rdev /mnt/usr/mdec/bootxx_ffsv1

 if [ -f /usr/share/cuw_config.subr ] ; then
 	. /usr/share/cuw_config.subr
 	if [ -f $cuw_conf_file ] ; then
 		echo "Copying existing $cuw_conf_file."
 		cp $cuw_conf_file /mnt/$cuw_conf_file
 	else
 		echo "No existing configuration found."
 	fi
 fi

 cd -
 umount /mnt

 echo "Upgrade complete ($dev)."

 # $Id: upgrade 2288 2004-12-23 07:16:30Z dyoung $

 -- 
 David Young             OJC Technologies
 dyoung@ojctech.com      Urbana, IL * (217) 278-3933

From: Andrew Doran <ad@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/28621 CVS commit: src
Date: Sun, 22 Feb 2009 20:28:07 +0000 (UTC)

 Module Name:	src
 Committed By:	ad
 Date:		Sun Feb 22 20:28:07 UTC 2009

 Modified Files:
 	src/doc: CHANGES
 	src/lib/libp2k: p2k.c
 	src/sbin/fsck_lfs: lfs.c
 	src/sbin/mount: mount.8
 	src/sbin/newfs_lfs: make_lfs.c
 	src/sbin/tunefs: tunefs.8 tunefs.c
 	src/sys/arch/vax/conf: VAX780
 	src/sys/conf: files
 	src/sys/kern: sys_aio.c vfs_bio.c vfs_subr.c vfs_syscalls.c
 	src/sys/miscfs/specfs: spec_vnops.c
 	src/sys/miscfs/syncfs: sync_subr.c
 	src/sys/modules/ffs: Makefile
 	src/sys/rump/fs/lib/libffs: Makefile
 	src/sys/rump/include/rump: rump.h
 	src/sys/rump/librump/rumpvfs: rump_vfs.c vm_vfs.c
 	src/sys/sys: buf.h vnode.h
 	src/sys/ufs: files.ufs
 	src/sys/ufs/ffs: ffs_alloc.c ffs_balloc.c ffs_extern.h ffs_inode.c
 	    ffs_snapshot.c ffs_vfsops.c ffs_vnops.c ffs_wapbl.c
 	src/sys/ufs/lfs: lfs_rfw.c lfs_vfsops.c lfs_vnops.c
 	src/sys/ufs/ufs: inode.h ufs_dirhash.c ufs_extern.h ufs_inode.c
 	    ufs_lookup.c ufs_readwrite.c ufs_vnops.c ufs_wapbl.c
 	src/sys/uvm: uvm_pager.c
 Removed Files:
 	src/sys/rump/librump/rumpkern/opt: opt_softdep.h
 	src/sys/ufs/ffs: ffs_softdep.c ffs_softdep.stub.c softdep.h

 Log Message:
 PR kern/26878 FFSv2 + softdep = livelock (no free ram)
 PR kern/16942 panic with softdep and quotas
 PR kern/19565 panic: softdep_write_inodeblock: indirect pointer #1 mismatch
 PR kern/26274 softdep panic: allocdirect_merge: ...
 PR kern/26374 Long delay before non-root users can write to softdep partitions
 PR kern/28621 1.6.x "vp != NULL" panic in ffs_softdep.c:4653 while unmounting a softdep (+quota) filesystem
 PR kern/29513 FFS+Softdep panic with unfsck-able file-corruption
 PR kern/31544 The ffs softdep code appears to fail to write dirty bits to disk
 PR kern/31981 stopping scsi disk can cause panic (softdep)
 PR kern/32116 kernel panic in softdep (assertion failure)
 PR kern/32532 softdep_trackbufs deadlock
 PR kern/37191 softdep: locking against myself
 PR kern/40474 Kernel panic after remounting raid root with softdep

 Retire softdep, pass 2. As discussed and later formally announced on the
 mailing lists.


 To generate a diff of this commit:
 cvs rdiff -r1.1191 -r1.1192 src/doc/CHANGES
 cvs rdiff -r1.8 -r1.9 src/lib/libp2k/p2k.c
 cvs rdiff -r1.29 -r1.30 src/sbin/fsck_lfs/lfs.c
 cvs rdiff -r1.65 -r1.66 src/sbin/mount/mount.8
 cvs rdiff -r1.13 -r1.14 src/sbin/newfs_lfs/make_lfs.c
 cvs rdiff -r1.37 -r1.38 src/sbin/tunefs/tunefs.8 src/sbin/tunefs/tunefs.c
 cvs rdiff -r1.1 -r1.2 src/sys/arch/vax/conf/VAX780
 cvs rdiff -r1.942 -r1.943 src/sys/conf/files
 cvs rdiff -r1.22 -r1.23 src/sys/kern/sys_aio.c
 cvs rdiff -r1.215 -r1.216 src/sys/kern/vfs_bio.c
 cvs rdiff -r1.368 -r1.369 src/sys/kern/vfs_subr.c
 cvs rdiff -r1.388 -r1.389 src/sys/kern/vfs_syscalls.c
 cvs rdiff -r1.122 -r1.123 src/sys/miscfs/specfs/spec_vnops.c
 cvs rdiff -r1.36 -r1.37 src/sys/miscfs/syncfs/sync_subr.c
 cvs rdiff -r1.2 -r1.3 src/sys/modules/ffs/Makefile
 cvs rdiff -r1.6 -r1.7 src/sys/rump/fs/lib/libffs/Makefile
 cvs rdiff -r1.9 -r1.10 src/sys/rump/include/rump/rump.h
 cvs rdiff -r1.1 -r0 src/sys/rump/librump/rumpkern/opt/opt_softdep.h
 cvs rdiff -r1.12 -r1.13 src/sys/rump/librump/rumpvfs/rump_vfs.c
 cvs rdiff -r1.3 -r1.4 src/sys/rump/librump/rumpvfs/vm_vfs.c
 cvs rdiff -r1.110 -r1.111 src/sys/sys/buf.h
 cvs rdiff -r1.200 -r1.201 src/sys/sys/vnode.h
 cvs rdiff -r1.18 -r1.19 src/sys/ufs/files.ufs
 cvs rdiff -r1.121 -r1.122 src/sys/ufs/ffs/ffs_alloc.c
 cvs rdiff -r1.51 -r1.52 src/sys/ufs/ffs/ffs_balloc.c
 cvs rdiff -r1.74 -r1.75 src/sys/ufs/ffs/ffs_extern.h
 cvs rdiff -r1.102 -r1.103 src/sys/ufs/ffs/ffs_inode.c
 cvs rdiff -r1.91 -r1.92 src/sys/ufs/ffs/ffs_snapshot.c
 cvs rdiff -r1.116 -r0 src/sys/ufs/ffs/ffs_softdep.c
 cvs rdiff -r1.23 -r0 src/sys/ufs/ffs/ffs_softdep.stub.c
 cvs rdiff -r1.242 -r1.243 src/sys/ufs/ffs/ffs_vfsops.c
 cvs rdiff -r1.110 -r1.111 src/sys/ufs/ffs/ffs_vnops.c
 cvs rdiff -r1.11 -r1.12 src/sys/ufs/ffs/ffs_wapbl.c
 cvs rdiff -r1.11 -r0 src/sys/ufs/ffs/softdep.h
 cvs rdiff -r1.11 -r1.12 src/sys/ufs/lfs/lfs_rfw.c
 cvs rdiff -r1.269 -r1.270 src/sys/ufs/lfs/lfs_vfsops.c
 cvs rdiff -r1.219 -r1.220 src/sys/ufs/lfs/lfs_vnops.c
 cvs rdiff -r1.55 -r1.56 src/sys/ufs/ufs/inode.h
 cvs rdiff -r1.27 -r1.28 src/sys/ufs/ufs/ufs_dirhash.c
 cvs rdiff -r1.60 -r1.61 src/sys/ufs/ufs/ufs_extern.h
 cvs rdiff -r1.77 -r1.78 src/sys/ufs/ufs/ufs_inode.c
 cvs rdiff -r1.100 -r1.101 src/sys/ufs/ufs/ufs_lookup.c
 cvs rdiff -r1.93 -r1.94 src/sys/ufs/ufs/ufs_readwrite.c
 cvs rdiff -r1.172 -r1.173 src/sys/ufs/ufs/ufs_vnops.c
 cvs rdiff -r1.4 -r1.5 src/sys/ufs/ufs/ufs_wapbl.c
 cvs rdiff -r1.93 -r1.94 src/sys/uvm/uvm_pager.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->closed
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Wed, 01 Apr 2009 04:08:33 +0000
State-Changed-Why:
softdep (softupdates) has been removed.


>Unformatted:

Home	PR Database Search