NetBSD Problem Report #49244
From wiz@yt.nih.at Mon Sep 29 21:05:27 2014
Return-Path: <wiz@yt.nih.at>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
by mollari.NetBSD.org (Postfix) with ESMTPS id C4174A6551
for <gnats-bugs@gnats.NetBSD.org>; Mon, 29 Sep 2014 21:05:27 +0000 (UTC)
Message-Id: <20140929204858.B05CC2AC11C@yt.nih.at>
Date: Mon, 29 Sep 2014 22:48:58 +0200 (CEST)
From: Thomas Klausner <wiz@NetBSD.org>
Reply-To: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Subject: raid(4): reconstruct-on-spare-in-progress not a separate state
X-Send-Pr-Version: 3.95
>Number: 49244
>Category: kern
>Synopsis: raid(4): reconstruct-on-spare-in-progress not a separate state
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: oster
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Mon Sep 29 21:10:00 +0000 2014
>Closed-Date: Wed Dec 03 14:20:58 +0000 2014
>Last-Modified: Wed Dec 03 14:20:58 +0000 2014
>Originator: Thomas Klausner
>Release: NetBSD 5.1.2
>Organization:
Curiosity is the very basis of education and if you tell me that
curiosity killed the cat, I say only that the cat died nobly.
- Arnold Edinborough
>Environment:
Architecture: x86_64
Machine: amd64
>Description:
After replacing a hard disk in a 3-disk RAID 5, I added the replacement disk
as spare and started reconstructing on it (in single-user mode).
After having started that, I rebooted, because I thought it should
work (even if it might start from 0% again) and I could use the
machine in multi-user mode with a degraded raid.
When I rebooted, I noticed that the raid thought itself to be perfect;
i.e. even though reconstruction had been at 1%, it now said that
everything was fine for all 3 disks.
I immediately failed the new disk and started reconstruction to it again.
The whole procedure cost me one inode on the RAID, which fsck could
fix (I was lucky).
Greg Oster has analyzed:
> Basically, when reconstruction starts to a spare disk, the disk is
> marked as rf_used_spare -- and that is what we key off of when
> updating the component labels on shutdown, etc. So what happens
> when you reboot in the middle of a reconstruction is that the disk
> being rebuilt *to* gets updated with a component label as though
> the reconstruction is finished!!! I think all that's needed is
> another intermediate state (say rf_ds_rebuild_in_progress) that
> basically marks the spare disk as being in used, but not done with
> reconstruction until rf_used_spare is hit....
>How-To-Repeat:
See above.
>Fix:
Greg has agreed to fix this. Thanks, Greg :)
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: kern-bug-people->oster
Responsible-Changed-By: wiz@NetBSD.org
Responsible-Changed-When: Mon, 29 Sep 2014 21:10:56 +0000
Responsible-Changed-Why:
Greg agreed to handle this one. Thanks!
From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/49244: raid(4): reconstruct-on-spare-in-progress not a
separate state
Date: Tue, 14 Oct 2014 05:54:11 +0000
On Mon, Sep 29, 2014 at 09:10:03PM +0000, Thomas Klausner wrote:
> >Severity: serious
> >Priority: high
> [...]
>
> The whole procedure cost me one inode on the RAID, which fsck could
> fix (I was lucky).
If you don't mind, I'm going to change that to "critical". :-|
--
David A. Holland
dholland@netbsd.org
From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/49244: raid(4): reconstruct-on-spare-in-progress not a
separate state
Date: Tue, 14 Oct 2014 08:39:45 +0200
On Tue, Oct 14, 2014 at 05:55:01AM +0000, David Holland wrote:
>On Mon, Sep 29, 2014 at 09:10:03PM +0000, Thomas Klausner wrote:
> > >Severity: serious
> > >Priority: high
> > [...]
> >
> > The whole procedure cost me one inode on the RAID, which fsck could
> > fix (I was lucky).
>
>If you don't mind, I'm going to change that to "critical". :-|
Fine with me.
Thomas
From: Greg Oster <oster@cs.usask.ca>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/49244: raid(4): reconstruct-on-spare-in-progress not a
separate state
Date: Tue, 11 Nov 2014 18:04:52 -0600
On Mon, 29 Sep 2014 21:10:03 +0000 (UTC)
Thomas Klausner <wiz@NetBSD.org> wrote:
> >Number: 49244
> >Category: kern
> >Synopsis: raid(4): reconstruct-on-spare-in-progress not a
> >separate state Confidential: no
> >Severity: serious
> >Priority: high
> >Responsible: kern-bug-people
> >State: open
> >Class: sw-bug
> >Submitter-Id: net
> >Arrival-Date: Mon Sep 29 21:10:00 +0000 2014
> >Originator: Thomas Klausner
> >Release: NetBSD 5.1.2
> >Organization:
> Curiosity is the very basis of education and if you tell me that
> curiosity killed the cat, I say only that the cat died nobly.
> - Arnold Edinborough
> >Environment:
>
>
> Architecture: x86_64
> Machine: amd64
> >Description:
> After replacing a hard disk in a 3-disk RAID 5, I added the
> replacement disk as spare and started reconstructing on it (in
> single-user mode).
>
> After having started that, I rebooted, because I thought it should
> work (even if it might start from 0% again) and I could use the
> machine in multi-user mode with a degraded raid.
>
> When I rebooted, I noticed that the raid thought itself to be perfect;
> i.e. even though reconstruction had been at 1%, it now said that
> everything was fine for all 3 disks.
>
> I immediately failed the new disk and started reconstruction to it
> again.
>
> The whole procedure cost me one inode on the RAID, which fsck could
> fix (I was lucky).
>
> Greg Oster has analyzed:
> > Basically, when reconstruction starts to a spare disk, the disk
> > is marked as rf_used_spare -- and that is what we key off of
> > when updating the component labels on shutdown, etc. So what
> > happens when you reboot in the middle of a reconstruction is that
> > the disk being rebuilt *to* gets updated with a component label as
> > though the reconstruction is finished!!! I think all that's needed
> > is another intermediate state (say rf_ds_rebuild_in_progress)
> > that basically marks the spare disk as being in used, but not done
> > with reconstruction until rf_used_spare is
> > hit....
>
>
> >How-To-Repeat:
> See above.
> >Fix:
> Greg has agreed to fix this. Thanks, Greg :)
>
> >Unformatted:
I have a fix for this that I hope to commit very soon, and request
pullups. I'm just working on double-checking the changes to make sure
I havn't introduced any unintended consequences.
Later...
Greg Oster
From: "Greg Oster" <oster@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/49244 CVS commit: src/sys/dev/raidframe
Date: Fri, 14 Nov 2014 14:29:16 +0000
Module Name: src
Committed By: oster
Date: Fri Nov 14 14:29:16 UTC 2014
Modified Files:
src/sys/dev/raidframe: raidframevar.h rf_netbsdkintf.c rf_reconstruct.c
Log Message:
Fix a long-standing bug related to rebooting while a
reconstruct-to-spare is underway but not yet complete.
The issue was that a component was being marked as a used_spare when
the rebuild started, not when the rebuild was actually finished.
Marking it as a used_spare meant that the component label on the spare
was being updated such that after a reboot the component would be
considered up-to-date, regardless of whether the rebuild actually
completed!
This fix includes:
1) Add an additional state "rf_ds_rebuilding_spare" which is used
to denote that a spare is currently being rebuilt from the live
components.
2) Update the comments on the disk states, which were out-of-sync
with reality.
3) When rebuilding to a spare component, that spare now enters the
state rf_ds_rebuilding_spare instead of the state rf_ds_used_spare.
4) When the rebuild is actually complete then the spare component
enters the rf_ds_used_spare state. rf_ds_used_spare is now used
exclusively for the case where the rebuilding to the spare has
completed successfully.
XXX: Someday we need to teach raidctl(8) about this new state, and
take out the backwards compatibility code in rf_netbsdkintf.c (see
RAIDFRAME_GET_INFO in raidioctl()). For today, this fix needs to be
generic enough that it can get backported without major grief.
XXX: Needs pullup to netbsd-5*, netbsd-6*, and netbsd-7
Fixes PR#49244.
To generate a diff of this commit:
cvs rdiff -u -r1.16 -r1.17 src/sys/dev/raidframe/raidframevar.h
cvs rdiff -u -r1.315 -r1.316 src/sys/dev/raidframe/rf_netbsdkintf.c
cvs rdiff -u -r1.120 -r1.121 src/sys/dev/raidframe/rf_reconstruct.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Soren Jacobsen" <snj@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/49244 CVS commit: [netbsd-7] src/sys/dev/raidframe
Date: Tue, 18 Nov 2014 18:03:10 +0000
Module Name: src
Committed By: snj
Date: Tue Nov 18 18:03:10 UTC 2014
Modified Files:
src/sys/dev/raidframe [netbsd-7]: raidframevar.h rf_netbsdkintf.c
rf_reconstruct.c
Log Message:
Pull up following revision(s) (requested by oster in ticket #243):
sys/dev/raidframe/raidframevar.h: revision 1.17
sys/dev/raidframe/rf_netbsdkintf.c: revision 1.316
sys/dev/raidframe/rf_reconstruct.c: revision 1.121
Fix a long-standing bug related to rebooting while a
reconstruct-to-spare is underway but not yet complete.
The issue was that a component was being marked as a used_spare when
the rebuild started, not when the rebuild was actually finished.
Marking it as a used_spare meant that the component label on the spare
was being updated such that after a reboot the component would be
considered up-to-date, regardless of whether the rebuild actually
completed!
This fix includes:
1) Add an additional state "rf_ds_rebuilding_spare" which is used
to denote that a spare is currently being rebuilt from the live
components.
2) Update the comments on the disk states, which were out-of-sync
with reality.
3) When rebuilding to a spare component, that spare now enters the
state rf_ds_rebuilding_spare instead of the state rf_ds_used_spare.
4) When the rebuild is actually complete then the spare component
enters the rf_ds_used_spare state. rf_ds_used_spare is now used
exclusively for the case where the rebuilding to the spare has
completed successfully.
XXX: Someday we need to teach raidctl(8) about this new state, and
take out the backwards compatibility code in rf_netbsdkintf.c (see
RAIDFRAME_GET_INFO in raidioctl()). For today, this fix needs to be
generic enough that it can get backported without major grief.
XXX: Needs pullup to netbsd-5*, netbsd-6*, and netbsd-7
Fixes PR#49244.
To generate a diff of this commit:
cvs rdiff -u -r1.16 -r1.16.4.1 src/sys/dev/raidframe/raidframevar.h
cvs rdiff -u -r1.312.2.2 -r1.312.2.3 src/sys/dev/raidframe/rf_netbsdkintf.c
cvs rdiff -u -r1.120 -r1.120.2.1 src/sys/dev/raidframe/rf_reconstruct.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Stephen Borrill" <sborrill@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/49244 CVS commit: [netbsd-5] src/sys/dev/raidframe
Date: Thu, 20 Nov 2014 09:38:56 +0000
Module Name: src
Committed By: sborrill
Date: Thu Nov 20 09:38:56 UTC 2014
Modified Files:
src/sys/dev/raidframe [netbsd-5]: raidframevar.h rf_netbsdkintf.c
rf_reconstruct.c
Log Message:
Pull up the following revisions(s) (requested by oster in ticket #1933):
sys/dev/raidframe/raidframevar.h: revision 1.17
sys/dev/raidframe/rf_netbsdkintf.c: revision 1.316
sys/dev/raidframe/rf_reconstruct.c: revision 1.121 via patch
Fix a long-standing bug related to rebooting while a reconstruct-to-spare
is underway but not yet complete. Fixes PR kern/49244.
To generate a diff of this commit:
cvs rdiff -u -r1.12.10.3 -r1.12.10.4 src/sys/dev/raidframe/raidframevar.h
cvs rdiff -u -r1.250.4.13 -r1.250.4.14 src/sys/dev/raidframe/rf_netbsdkintf.c
cvs rdiff -u -r1.105.4.6 -r1.105.4.7 src/sys/dev/raidframe/rf_reconstruct.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Stephen Borrill" <sborrill@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/49244 CVS commit: [netbsd-5-2] src/sys/dev/raidframe
Date: Thu, 20 Nov 2014 12:25:10 +0000
Module Name: src
Committed By: sborrill
Date: Thu Nov 20 12:25:10 UTC 2014
Modified Files:
src/sys/dev/raidframe [netbsd-5-2]: raidframevar.h rf_netbsdkintf.c
rf_reconstruct.c
Log Message:
Pull up the following revisions(s) (requested by oster in ticket #1933):
sys/dev/raidframe/raidframevar.h: revision 1.17
sys/dev/raidframe/rf_netbsdkintf.c: revision 1.316
sys/dev/raidframe/rf_reconstruct.c: revision 1.121 via patch
Fix a long-standing bug related to rebooting while a reconstruct-to-spare
is underway but not yet complete. Fixes PR kern/49244.
To generate a diff of this commit:
cvs rdiff -u -r1.12.10.3 -r1.12.10.3.2.1 src/sys/dev/raidframe/raidframevar.h
cvs rdiff -u -r1.250.4.13 -r1.250.4.13.2.1 \
src/sys/dev/raidframe/rf_netbsdkintf.c
cvs rdiff -u -r1.105.4.6 -r1.105.4.6.2.1 \
src/sys/dev/raidframe/rf_reconstruct.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Stephen Borrill" <sborrill@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/49244 CVS commit: [netbsd-5-1] src/sys/dev/raidframe
Date: Thu, 20 Nov 2014 15:52:10 +0000
Module Name: src
Committed By: sborrill
Date: Thu Nov 20 15:52:10 UTC 2014
Modified Files:
src/sys/dev/raidframe [netbsd-5-1]: raidframevar.h rf_netbsdkintf.c
rf_reconstruct.c
Log Message:
Pull up the following revisions(s) (requested by oster in ticket #1933):
sys/dev/raidframe/raidframevar.h: revision 1.17
sys/dev/raidframe/rf_netbsdkintf.c: revision 1.316
sys/dev/raidframe/rf_reconstruct.c: revision 1.121 via patch
Fix a long-standing bug related to rebooting while a reconstruct-to-spare
is underway but not yet complete. Fixes PR kern/49244.
To generate a diff of this commit:
cvs rdiff -u -r1.12.10.1 -r1.12.10.1.2.1 src/sys/dev/raidframe/raidframevar.h
cvs rdiff -u -r1.250.4.6 -r1.250.4.6.2.1 \
src/sys/dev/raidframe/rf_netbsdkintf.c
cvs rdiff -u -r1.105.4.3 -r1.105.4.3.2.1 \
src/sys/dev/raidframe/rf_reconstruct.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Soren Jacobsen" <snj@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/49244 CVS commit: [netbsd-6] src/sys/dev/raidframe
Date: Tue, 2 Dec 2014 22:05:14 +0000
Module Name: src
Committed By: snj
Date: Tue Dec 2 22:05:14 UTC 2014
Modified Files:
src/sys/dev/raidframe [netbsd-6]: raidframevar.h rf_netbsdkintf.c
rf_reconstruct.c
Log Message:
Pull up following revision(s) (requested by oster in ticket #1194):
sys/dev/raidframe/raidframevar.h: revision 1.17
sys/dev/raidframe/rf_netbsdkintf.c: revision 1.316
sys/dev/raidframe/rf_reconstruct.c: revision 1.121
Fix a long-standing bug related to rebooting while a
reconstruct-to-spare is underway but not yet complete.
The issue was that a component was being marked as a used_spare when
the rebuild started, not when the rebuild was actually finished.
Marking it as a used_spare meant that the component label on the spare
was being updated such that after a reboot the component would be
considered up-to-date, regardless of whether the rebuild actually
completed!
This fix includes:
1) Add an additional state "rf_ds_rebuilding_spare" which is used
to denote that a spare is currently being rebuilt from the live
components.
2) Update the comments on the disk states, which were out-of-sync
with reality.
3) When rebuilding to a spare component, that spare now enters the
state rf_ds_rebuilding_spare instead of the state rf_ds_used_spare.
4) When the rebuild is actually complete then the spare component
enters the rf_ds_used_spare state. rf_ds_used_spare is now used
exclusively for the case where the rebuilding to the spare has
completed successfully.
XXX: Someday we need to teach raidctl(8) about this new state, and
take out the backwards compatibility code in rf_netbsdkintf.c (see
RAIDFRAME_GET_INFO in raidioctl()). For today, this fix needs to be
generic enough that it can get backported without major grief.
XXX: Needs pullup to netbsd-5*, netbsd-6*, and netbsd-7
Fixes PR#49244.
To generate a diff of this commit:
cvs rdiff -u -r1.15 -r1.15.10.1 src/sys/dev/raidframe/raidframevar.h
cvs rdiff -u -r1.295.6.2 -r1.295.6.3 src/sys/dev/raidframe/rf_netbsdkintf.c
cvs rdiff -u -r1.117.8.1 -r1.117.8.2 src/sys/dev/raidframe/rf_reconstruct.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Soren Jacobsen" <snj@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/49244 CVS commit: [netbsd-6-0] src/sys/dev/raidframe
Date: Tue, 2 Dec 2014 22:06:58 +0000
Module Name: src
Committed By: snj
Date: Tue Dec 2 22:06:58 UTC 2014
Modified Files:
src/sys/dev/raidframe [netbsd-6-0]: raidframevar.h rf_netbsdkintf.c
rf_reconstruct.c
Log Message:
Pull up following revision(s) (requested by oster in ticket #1194):
sys/dev/raidframe/raidframevar.h: revision 1.17
sys/dev/raidframe/rf_netbsdkintf.c: revision 1.316
sys/dev/raidframe/rf_reconstruct.c: revision 1.121
Fix a long-standing bug related to rebooting while a
reconstruct-to-spare is underway but not yet complete.
The issue was that a component was being marked as a used_spare when
the rebuild started, not when the rebuild was actually finished.
Marking it as a used_spare meant that the component label on the spare
was being updated such that after a reboot the component would be
considered up-to-date, regardless of whether the rebuild actually
completed!
This fix includes:
1) Add an additional state "rf_ds_rebuilding_spare" which is used
to denote that a spare is currently being rebuilt from the live
components.
2) Update the comments on the disk states, which were out-of-sync
with reality.
3) When rebuilding to a spare component, that spare now enters the
state rf_ds_rebuilding_spare instead of the state rf_ds_used_spare.
4) When the rebuild is actually complete then the spare component
enters the rf_ds_used_spare state. rf_ds_used_spare is now used
exclusively for the case where the rebuilding to the spare has
completed successfully.
XXX: Someday we need to teach raidctl(8) about this new state, and
take out the backwards compatibility code in rf_netbsdkintf.c (see
RAIDFRAME_GET_INFO in raidioctl()). For today, this fix needs to be
generic enough that it can get backported without major grief.
XXX: Needs pullup to netbsd-5*, netbsd-6*, and netbsd-7
Fixes PR#49244.
To generate a diff of this commit:
cvs rdiff -u -r1.15 -r1.15.16.1 src/sys/dev/raidframe/raidframevar.h
cvs rdiff -u -r1.295.6.2 -r1.295.6.2.4.1 \
src/sys/dev/raidframe/rf_netbsdkintf.c
cvs rdiff -u -r1.117.8.1 -r1.117.8.1.4.1 \
src/sys/dev/raidframe/rf_reconstruct.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Soren Jacobsen" <snj@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/49244 CVS commit: [netbsd-6-1] src/sys/dev/raidframe
Date: Tue, 2 Dec 2014 22:08:01 +0000
Module Name: src
Committed By: snj
Date: Tue Dec 2 22:08:01 UTC 2014
Modified Files:
src/sys/dev/raidframe [netbsd-6-1]: raidframevar.h rf_netbsdkintf.c
rf_reconstruct.c
Log Message:
Pull up following revision(s) (requested by oster in ticket #1194):
sys/dev/raidframe/raidframevar.h: revision 1.17
sys/dev/raidframe/rf_netbsdkintf.c: revision 1.316
sys/dev/raidframe/rf_reconstruct.c: revision 1.121
Fix a long-standing bug related to rebooting while a
reconstruct-to-spare is underway but not yet complete.
The issue was that a component was being marked as a used_spare when
the rebuild started, not when the rebuild was actually finished.
Marking it as a used_spare meant that the component label on the spare
was being updated such that after a reboot the component would be
considered up-to-date, regardless of whether the rebuild actually
completed!
This fix includes:
1) Add an additional state "rf_ds_rebuilding_spare" which is used
to denote that a spare is currently being rebuilt from the live
components.
2) Update the comments on the disk states, which were out-of-sync
with reality.
3) When rebuilding to a spare component, that spare now enters the
state rf_ds_rebuilding_spare instead of the state rf_ds_used_spare.
4) When the rebuild is actually complete then the spare component
enters the rf_ds_used_spare state. rf_ds_used_spare is now used
exclusively for the case where the rebuilding to the spare has
completed successfully.
XXX: Someday we need to teach raidctl(8) about this new state, and
take out the backwards compatibility code in rf_netbsdkintf.c (see
RAIDFRAME_GET_INFO in raidioctl()). For today, this fix needs to be
generic enough that it can get backported without major grief.
XXX: Needs pullup to netbsd-5*, netbsd-6*, and netbsd-7
Fixes PR#49244.
To generate a diff of this commit:
cvs rdiff -u -r1.15 -r1.15.24.1 src/sys/dev/raidframe/raidframevar.h
cvs rdiff -u -r1.295.6.2 -r1.295.6.2.6.1 \
src/sys/dev/raidframe/rf_netbsdkintf.c
cvs rdiff -u -r1.117.8.1 -r1.117.8.1.6.1 \
src/sys/dev/raidframe/rf_reconstruct.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->closed
State-Changed-By: oster@NetBSD.org
State-Changed-When: Wed, 03 Dec 2014 14:20:58 +0000
State-Changed-Why:
Pullups have been done, so this ticket is done.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home