NetBSD Problem Report #50159

From gson@gson.org  Fri Aug 21 07:46:19 2015
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 9C4A7A6531
	for <gnats-bugs@gnats.NetBSD.org>; Fri, 21 Aug 2015 07:46:19 +0000 (UTC)
Message-Id: <20150821074611.4F839745874@guava.gson.org>
Date: Fri, 21 Aug 2015 10:46:11 +0300 (EEST)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@gnats.NetBSD.org
Subject: "panic: ifree: freeing free inode" mounting root fs
X-Send-Pr-Version: 3.95

>Number:         50159
>Category:       kern
>Synopsis:       "panic: ifree: freeing free inode" mounting root fs
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Aug 21 07:50:00 +0000 2015
>Last-Modified:  Sat Sep 05 23:40:00 +0000 2015
>Originator:     Andreas Gustafsson
>Release:        NetBSD 6.1.4
>Organization:

>Environment:
System: NetBSD
Architecture: x86_64
Machine: amd64
>Description:

One of my NetBSD/amd64 machines is failing to reboot after a crash.
When remounting the root file system read/write after fsck, it panics
with "panic: ifree: freeing free inode", apparently while replaying
the wapbl log.

Backtrace:

  http://www.gson.org/netbsd/bugs/ifree/ifree2.jpg

If I power cycle the machine, I just end up in the same situation
again.

This looks somewhat similar to kern/49419, but there the panic
occurred during an unlink() syscall, not mount().

Any hints how to recover from this situation would be appreciated.
Since this machine hosts my automated NetBSD test runs and most of my
other NetBSD related work, my ability to contribute to the project is
somewhat limited until I can get it to boot.

>How-To-Repeat:

>Fix:

>Audit-Trail:
From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, kern-bug-people@netbsd.org, 
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Cc: 
Subject: Re: kern/50159: "panic: ifree: freeing free inode" mounting root fs
Date: Fri, 21 Aug 2015 03:54:45 -0400

 On Aug 21,  7:50am, gson@gson.org (Andreas Gustafsson) wrote:
 -- Subject: kern/50159: "panic: ifree: freeing free inode" mounting root fs

 | >Number:         50159
 | >Category:       kern
 | >Synopsis:       "panic: ifree: freeing free inode" mounting root fs
 | >Confidential:   no
 | >Severity:       critical
 | >Priority:       high
 | >Responsible:    kern-bug-people
 | >State:          open
 | >Class:          sw-bug
 | >Submitter-Id:   net
 | >Arrival-Date:   Fri Aug 21 07:50:00 +0000 2015
 | >Originator:     Andreas Gustafsson
 | >Release:        NetBSD 6.1.4
 | >Organization:
 | 
 | >Environment:
 | System: NetBSD
 | Architecture: x86_64
 | Machine: amd64
 | >Description:

 tunefs -l 0 the filesystem, run fsck and reboot?

 christos

From: mlelstv@serpens.de (Michael van Elst)
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/50159: "panic: ifree: freeing free inode" mounting root fs
Date: Fri, 21 Aug 2015 08:05:43 +0000 (UTC)

 gson@gson.org (Andreas Gustafsson) writes:

 >When remounting the root file system read/write after fsck, it panics
 >with "panic: ifree: freeing free inode", apparently while replaying
 >the wapbl log.

 The panic comes from wapbl trying to free partially allocated inodes.
 It only occurs when the filesystem is read-write, maybe that panic
 should also be suppressed while the journal is played back.

 This might work:

 Boot in single user
 fsck /
 tunefs -l 0 /
 reboot


 -- 
 -- 
                                 Michael van Elst
 Internet: mlelstv@serpens.de
                                 "A potential Snark may lurk in every tree."

From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@NetBSD.org
Cc: kern-bug-people@netbsd.org,
    gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org
Subject: Re: kern/50159: "panic: ifree: freeing free inode" mounting root fs
Date: Fri, 21 Aug 2015 11:34:14 +0300

 Christos Zoulas wrote:
 >  tunefs -l 0 the filesystem, run fsck and reboot?

 That worked - thank you!
 -- 
 Andreas Gustafsson, gson@gson.org

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/50159: "panic: ifree: freeing free inode" mounting root fs
Date: Sat, 5 Sep 2015 23:35:29 +0000

 On Fri, Aug 21, 2015 at 07:50:00AM +0000, Andreas Gustafsson wrote:
  > One of my NetBSD/amd64 machines is failing to reboot after a crash.
  > When remounting the root file system read/write after fsck, it panics
  > with "panic: ifree: freeing free inode", apparently while replaying
  > the wapbl log.
  > 
  > Backtrace:
  > 
  >   http://www.gson.org/netbsd/bugs/ifree/ifree2.jpg

 wd0a is the / on that machine? Because it's rather curious that it
 would go through fsck, remount read-write, and croak at that point.
 After all, fsck is supposed to leave volumes in a good state.

 I am guessing that fsck and wapbl are both completing the same
 partially-finished unlink, with the result that the second try
 panics.

 This obviously shouldn't be happening; however, it's not all that
 unlikely. (With physical block journaling, you can't record arbitrary
 loose information in the journal, so instead you generally build extra
 on-disk structures for things like keeping track of which files have
 been unlinked but not reclaimed. wapbl has such a scheme; however, I
 don't know how it works. Someone who does ought to look into this.)

  > This looks somewhat similar to kern/49419, but there the panic
  > occurred during an unlink() syscall, not mount().

 It's more likely than not the same underlying issue. That one didn't
 run fsck though, so perhaps my theory's not so great.

 Alternate theory that would explain both: the code that enters
 unlinked files into the structure of unlinked but not reclaimed files
 has a bug or race such that it (maybe sometimes) enters the wrong
 inode number.

 -- 
 David A. Holland
 dholland@netbsd.org

>Unformatted:

Home	PR Database Search