NetBSD Problem Report #39875
From lists@nerdbynature.de Fri Nov 7 10:33:25 2008
Return-Path: <lists@nerdbynature.de>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by narn.NetBSD.org (Postfix) with ESMTP id 2F5AD63BADB
for <gnats-bugs@gnats.NetBSD.org>; Fri, 7 Nov 2008 10:33:25 +0000 (UTC)
Message-Id: <20081107101010.104EF6C@netbsd.housecafe.de>
Date: Fri, 7 Nov 2008 11:10:10 +0100 (CET)
From: lists@nerdbynature.de
Reply-To: lists@nerdbynature.de
To: gnats-bugs@gnats.NetBSD.org
Subject: panic when booting off ffs+log
X-Send-Pr-Version: 3.95
>Number: 39875
>Category: kern
>Synopsis: panic when booting off ffs+log
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: kern-bug-people
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Fri Nov 07 10:35:00 +0000 2008
>Closed-Date: Sun Dec 04 14:46:11 +0000 2016
>Last-Modified: Sun Dec 04 14:46:11 +0000 2016
>Originator: Christian Kujau
>Release: NetBSD 5.0_BETA
>Organization:
>Environment:
System: NetBSD netbsd.housecafe.de 5.0_BETA NetBSD 5.0_BETA (XEN3_DOMU) #0: Mon Nov 3 22:50:22 PST 2008 builds@wb28:/home/builds/ab/netbsd-5/amd64/200811030002Z-obj/home/builds/ab/netbsd-5/src/sys/arch/amd64/compile/XEN3_DOMU amd64
Architecture: x86_64
Machine: amd64
>Description:
After installing from netbsd-INSTALL_XEN3_DOMU.gz in an Dom0
(Ubuntu Linux, amd64, Intel based MacMini) rebooting from
netbsd-XEN3_DOMU.gz, the kernel panics:
# xm create -c netbsd.cfg
Using config file "/etc/xen/netbsd.cfg".
Started domain netbsd
Loaded initial symtab at 0xffffffff80658464, strtab at 0xffffffff806a4a70, # entries 12943
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
NetBSD 5.0_BETA (XEN3_DOMU) #0: Mon Nov 3 22:50:22 PST 2008
builds@wb28:/home/builds/ab/netbsd-5/amd64/200811030002Z-obj/home/builds/ab/netbsd-5/src/sys/arch/amd64/compile/XEN3_DOMU
total memory = 65536 KB
avail memory = 55692 KB
mainbus0 (root)
hypervisor0 at mainbus0: Xen version 3.2
vcpu0 at hypervisor0: Intel 686-class, 1999MHz, id 0x6f6
xenbus0 at hypervisor0: Xen Virtual Bus Interface
xencons0 at hypervisor0: Xen Virtual Console Driver
xencons0: using event channel 2
xbd0 at xenbus0 id 3: Xen Virtual Block Device Interface
xennet0 at xenbus0 id 0: Xen Virtual Network Interface
xennet0: MAC address 00:16:3e:38:d3:03
unknown type console at xenbus0 id 0 not configured
raidattach: Asked for 8 units
Kernelized RAIDframe activated
Searching for RAID components...
boot device: <unknown>
root device:
use one of: xbd0[a-p] xennet0 md0[a-p] ddb halt reboot
root device: xbd0a
dump device (default xbd0b):
file system (default generic): ffs
root on xbd0a dumps on xbd0b
unknown type console at xenbus0 id 0 not configured
unknown type console at xenbus0 id 0 not configured
/: replaying log to memory
init path (default /sbin/init):
init: copying out path `/sbin/init' 11
Fri Nov 7 10:18:52 CET 2008
swapctl: adding /dev/xbd0b as swap device at priority 0
Starting file system checks:
/dev/rxbd0a: file system is journaled; not checking
/dev/rxbd0e: 1 files, 1 used, 4032278 free (14 frags, 504033 blocks, 0.0% fragmentation)
/dev/rxbd0e: MARKING FILE SYSTEM CLEAN
/ fs is journalled, ignoring soft update mode
/: replaying log to disk
Setting tty flags.
Setting sysctl variables:
Starting network.
Hostname: netbsd.housecafe.de
IPv6 mode: host
Configuring network interfaces: xennet0.
Adding interface aliases:.
Starting dhclient.
Internet Systems Consortium DHCP Client V3.0.3
Copyright 2004-2005 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/products/DHCP
Listening on BPF/xennet0/00:16:3e:38:d3:03
Sending on BPF/xennet0/00:16:3e:38:d3:03
Sending on Socket/fallback
DHCPDISCOVER on xennet0 to 255.255.255.255 port 67 interval 7
DHCPOFFER from 192.168.10.50
DHCPREQUEST on xennet0 to 255.255.255.255 port 67
DHCPACK from 192.168.10.50
bound to 192.168.10.102 -- renewal in 33844 seconds.
Building databases: dev, utmp, utmpx, services done
wsconscfg: Cannot open `/dev/ttyEcfg': Device not configured
wsconscfg: Cannot open `/dev/ttyEcfg': Device not configured
wsconscfg: Cannot open `/dev/ttyEcfg': Device not configured
wsconscfg: Cannot open `/dev/ttyEcfg': Device not configured
Starting syslogd.
Checking for core dump...
Mounting all filesystems...
/ fs is journalled, ignoring soft update mode
Clearing temporary files.
Checking quotas: done.
Starting virecover.
Starting local daemons:.
Updating motd.
panic: kernel diagnostic assertion "!rw_write_held(&wl->wl_rwlock)" failed: file "/home/builds/ab/netbsd-5/src/sys/kern/vfs_wapbl.c", line 1570
fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip ffffffff80375c95 cs e030 rflags 246 cr2 7f7ffdba1800 cpl 0 rsp ffffa00006772830
Stopped in pid 233.1 (rm) at netbsd:breakpoint+0x5: leave
breakpoint() at netbsd:breakpoint+0x5
panic() at netbsd:panic+0x255
__kernassert() at netbsd:__kernassert+0x2d
vn_lock() at netbsd:vn_lock+0x6e
vget() at netbsd:vget+0x132
ufs_ihashget() at netbsd:ufs_ihashget+0x91
ffs_vget() at netbsd:ffs_vget+0xc1
handle_workitem_remove() at netbsd:handle_workitem_remove+0x36
softdep_setup_remove() at netbsd:softdep_setup_remove+0x103
ufs_dirremove() at netbsd:ufs_dirremove+0x9f
ufs_remove() at netbsd:ufs_remove+0x16c
VOP_REMOVE() at netbsd:VOP_REMOVE+0x2d
do_sys_unlink() at netbsd:do_sys_unlink+0xa7
syscall() at netbsd:syscall+0xb4
ds 0x2840
es 0x6ddc
fs 0x2840
gs 0x6eb7
rdi 0
rsi 0xd
rbp 0xffffa00006772830
rbx 0xffffa00006772840
rdx 0
rcx 0
rax 0x1
r8 0xffffffff80591640 cpu_info_primary
r9 0x1
r10 0xffffa00006772750
r11 0xffffffff8038da70 xenconscn_putc
r12 0x100
r13 0xffffffff804820f8 copyright+0x8e5b8
r14 0xe5e1
r15 0x2
rip 0xffffffff80375c95 breakpoint+0x5
cs 0xe030
rflags 0x246
rsp 0xffffa00006772830
ss 0xe02b
netbsd:breakpoint+0x5: leave
The underlying device is provided by the Xen-3.2 Dom0, which in turn built
this device from an LVM2 volume.
>How-To-Repeat:
This happens every time I try to boot netbsd-XEN3_DOMU.gz from
NetBSD-daily/netbsd-5/200811030002Z/amd64. More details from
ddb is available.
>Fix:
The root filesystem was mounted with the "log" option. Removing
this option fixes this issue and booting continues.
FWIW, this looked like PR 39295 to me, but the filesystem was not
corrupted afterwards and it panicked always in line 1570 vfs_wapbl.c
>Release-Note:
>Audit-Trail:
From: Greg Oster <oster@cs.usask.ca>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/39875: panic when booting off ffs+log
Date: Fri, 07 Nov 2008 11:11:31 -0600
lists@nerdbynature.de writes:
> >Number: 39875
> >Category: kern
> >Synopsis: panic when booting off ffs+log
> >Confidential: no
> >Severity: serious
> >Priority: medium
> >Responsible: kern-bug-people
> >State: open
> >Class: sw-bug
> >Submitter-Id: net
> >Arrival-Date: Fri Nov 07 10:35:00 +0000 2008
> >Originator: Christian Kujau
> >Release: NetBSD 5.0_BETA
> >Organization:
> >Environment:
>
>
> System: NetBSD netbsd.housecafe.de 5.0_BETA NetBSD 5.0_BETA (XEN3_DOMU) #0: M
> on Nov 3 22:50:22 PST 2008 builds@wb28:/home/builds/ab/netbsd-5/amd64/2008110
> 30002Z-obj/home/builds/ab/netbsd-5/src/sys/arch/amd64/compile/XEN3_DOMU amd64
> Architecture: x86_64
> Machine: amd64
> >Description:
> After installing from netbsd-INSTALL_XEN3_DOMU.gz in an Dom0
> (Ubuntu Linux, amd64, Intel based MacMini) rebooting from
> netbsd-XEN3_DOMU.gz, the kernel panics:
>
> # xm create -c netbsd.cfg
> Using config file "/etc/xen/netbsd.cfg".
> Started domain netbsd
> Loaded initial symtab at 0xffffffff80658464, strtab at 0xffffffff806a4a70, #
> entries 12943
> Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
> 2006, 2007, 2008
> The NetBSD Foundation, Inc. All rights reserved.
> Copyright (c) 1982, 1986, 1989, 1991, 1993
> The Regents of the University of California. All rights reserved.
>
> NetBSD 5.0_BETA (XEN3_DOMU) #0: Mon Nov 3 22:50:22 PST 2008
> builds@wb28:/home/builds/ab/netbsd-5/amd64/200811030002Z-obj/home/build
> s/ab/netbsd-5/src/sys/arch/amd64/compile/XEN3_DOMU
> total memory = 65536 KB
> avail memory = 55692 KB
> mainbus0 (root)
> hypervisor0 at mainbus0: Xen version 3.2
> vcpu0 at hypervisor0: Intel 686-class, 1999MHz, id 0x6f6
> xenbus0 at hypervisor0: Xen Virtual Bus Interface
> xencons0 at hypervisor0: Xen Virtual Console Driver
> xencons0: using event channel 2
> xbd0 at xenbus0 id 3: Xen Virtual Block Device Interface
> xennet0 at xenbus0 id 0: Xen Virtual Network Interface
> xennet0: MAC address 00:16:3e:38:d3:03
> unknown type console at xenbus0 id 0 not configured
> raidattach: Asked for 8 units
> Kernelized RAIDframe activated
> Searching for RAID components...
> boot device: <unknown>
> root device:
> use one of: xbd0[a-p] xennet0 md0[a-p] ddb halt reboot
> root device: xbd0a
> dump device (default xbd0b):
> file system (default generic): ffs
> root on xbd0a dumps on xbd0b
> unknown type console at xenbus0 id 0 not configured
> unknown type console at xenbus0 id 0 not configured
> /: replaying log to memory
> init path (default /sbin/init):
> init: copying out path `/sbin/init' 11
> Fri Nov 7 10:18:52 CET 2008
> swapctl: adding /dev/xbd0b as swap device at priority 0
> Starting file system checks:
> /dev/rxbd0a: file system is journaled; not checking
> /dev/rxbd0e: 1 files, 1 used, 4032278 free (14 frags, 504033 blocks, 0.0% fra
> gmentation)
> /dev/rxbd0e: MARKING FILE SYSTEM CLEAN
> / fs is journalled, ignoring soft update mode
> /: replaying log to disk
> Setting tty flags.
> Setting sysctl variables:
> Starting network.
> Hostname: netbsd.housecafe.de
> IPv6 mode: host
> Configuring network interfaces: xennet0.
> Adding interface aliases:.
> Starting dhclient.
> Internet Systems Consortium DHCP Client V3.0.3
> Copyright 2004-2005 Internet Systems Consortium.
> All rights reserved.
> For info, please visit http://www.isc.org/products/DHCP
>
> Listening on BPF/xennet0/00:16:3e:38:d3:03
> Sending on BPF/xennet0/00:16:3e:38:d3:03
> Sending on Socket/fallback
> DHCPDISCOVER on xennet0 to 255.255.255.255 port 67 interval 7
> DHCPOFFER from 192.168.10.50
> DHCPREQUEST on xennet0 to 255.255.255.255 port 67
> DHCPACK from 192.168.10.50
> bound to 192.168.10.102 -- renewal in 33844 seconds.
> Building databases: dev, utmp, utmpx, services done
> wsconscfg: Cannot open `/dev/ttyEcfg': Device not configured
> wsconscfg: Cannot open `/dev/ttyEcfg': Device not configured
> wsconscfg: Cannot open `/dev/ttyEcfg': Device not configured
> wsconscfg: Cannot open `/dev/ttyEcfg': Device not configured
> Starting syslogd.
> Checking for core dump...
> Mounting all filesystems...
> / fs is journalled, ignoring soft update mode
> Clearing temporary files.
> Checking quotas: done.
> Starting virecover.
> Starting local daemons:.
> Updating motd.
> panic: kernel diagnostic assertion "!rw_write_held(&wl->wl_rwlock)" failed: f
> ile "/home/builds/ab/netbsd-5/src/sys/kern/vfs_wapbl.c", line 1570
> fatal breakpoint trap in supervisor mode
> trap type 1 code 0 rip ffffffff80375c95 cs e030 rflags 246 cr2 7f7ffdba1800
> cpl 0 rsp ffffa00006772830
> Stopped in pid 233.1 (rm) at netbsd:breakpoint+0x5: leave
> breakpoint() at netbsd:breakpoint+0x5
> panic() at netbsd:panic+0x255
> __kernassert() at netbsd:__kernassert+0x2d
> vn_lock() at netbsd:vn_lock+0x6e
> vget() at netbsd:vget+0x132
> ufs_ihashget() at netbsd:ufs_ihashget+0x91
> ffs_vget() at netbsd:ffs_vget+0xc1
> handle_workitem_remove() at netbsd:handle_workitem_remove+0x36
> softdep_setup_remove() at netbsd:softdep_setup_remove+0x103
It appears, somehow, that softdeps have been turned on for this
filesystem!?!? If you turn off softdeps does the problem go away?
Later...
Greg Oster
From: lists@nerdbynature.de
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/39875: panic when booting off ffs+log
Date: Fri, 7 Nov 2008 12:48:22 -0800 (PST)
On Fri, 7 Nov 2008, Greg Oster wrote:
> It appears, somehow, that softdeps have been turned on for this
> filesystem!?!? If you turn off softdeps does the problem go away?
I still have softdeps enabled for all disk based filesystems. Also, I was
under the impression that vfs_wapbl.c was related to the logging feature,
not softdeps?
Anyway, the workaround, as I wrote in the report:
> Fix:
> The root filesystem was mounted with the "log" option. Removing
> this option fixes this issue and booting continues.
HTH,
Christian.
--
BOFH excuse #445:
Browser's cookie is corrupted -- someone's been nibbling on it.
From: lists@nerdbynature.de
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/39875: panic when booting off ffs+log
Date: Sat, 8 Nov 2008 03:24:50 -0800 (PST)
On Fri, 7 Nov 2008, Greg Oster wrote:
> It appears, somehow, that softdeps have been turned on for this
> filesystem!?!? If you turn off softdeps does the problem go away?
To answer your question more accuratly than my last post: yes, turning off
softdeps helps indeed. I've tested again, same system, but already
running this time. I could mount the same partition with rw, log,
noatime and read and write from/to it. But after mounting with
log+softdep (the combination is important, as both are fine when used
alone!), the system panicked again. The slightly different panic and more
from ddb(4) at: http://nerdbynature.de/bits/PR-39875/
Thanks,
Christian.
From: Greg Oster <oster@cs.usask.ca>
To: lists@nerdbynature.de
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/39875: panic when booting off ffs+log
Date: Sat, 08 Nov 2008 11:47:46 -0600
lists@nerdbynature.de writes:
> On Fri, 7 Nov 2008, Greg Oster wrote:
> > It appears, somehow, that softdeps have been turned on for this
> > filesystem!?!? If you turn off softdeps does the problem go away?
>
> To answer your question more accuratly than my last post: yes, turning off
> softdeps helps indeed. I've tested again, same system, but already
> running this time. I could mount the same partition with rw, log,
> noatime and read and write from/to it. But after mounting with
> log+softdep (the combination is important, as both are fine when used
> alone!), the system panicked again. The slightly different panic and more
> from ddb(4) at: http://nerdbynature.de/bits/PR-39875/
It's my understanding that it shouldn't let you mount with both
log+softdep, so that would be the real bug here...
Later...
Greg Oster
State-Changed-From-To: open->closed
State-Changed-By: maya@NetBSD.org
State-Changed-When: Sun, 04 Dec 2016 14:46:11 +0000
State-Changed-Why:
issue was softdep+wapbl, which is no longer possible.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home