NetBSD Problem Report #38471
From martin@duskware.de Sun Apr 20 19:16:29 2008
Return-Path: <martin@duskware.de>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by narn.NetBSD.org (Postfix) with ESMTP id BD06663B293
for <gnats-bugs@gnats.netbsd.org>; Sun, 20 Apr 2008 19:16:28 +0000 (UTC)
Message-Id: <20080420191345.B85A563B293@narn.NetBSD.org>
Date: Sun, 20 Apr 2008 19:13:45 +0000 (UTC)
From: thomas@t-online.de
Reply-To: thomas@t-online.de
To: netbsd-bugs-owner@NetBSD.org
Subject: Raidframe crashes on reconstruction of RAID5 (5 disks @ 298GB)
X-Send-Pr-Version: www-1.0
>Number: 38471
>Category: bin
>Synopsis: Raidframe crashes on reconstruction of RAID5 (5 disks @ 298GB)
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: oster
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sun Apr 20 19:20:00 +0000 2008
>Closed-Date: Mon Sep 22 15:13:26 +0000 2008
>Last-Modified: Mon Sep 22 15:13:26 +0000 2008
>Originator: Thomas Feddersen
>Release: 4.0_BETA2
>Organization:
Dipl.-Ing Thomas Feddersen, Beratender Ingenieur
>Environment:
NetBSD bremen 4.0_BETA2 NetBSD 4.0_BETA2 (XEN3_DOM0) #0: Thu Mar 1 04:57:05 PST 2007 builds@wb33:/home/builds/ab/netbsd-4/i386/200703010002Z-obj/home/builds/ab/netbsd-4/src/sys/arch/i386/compile/XEN3_DOM0 i386
>Description:
One disk of the raid-set has sectors pending.
Apr 20 18:09:26 bremen smartd[201]: Device: /dev/wd5d, 1 Currently unreadable (pending) sectors
Apr 20 18:09:26 bremen smartd[201]: Device: /dev/wd5d, 1 Offline uncorrectable sectors
I attempted an in-place reconstruction of the drive in question, but reconstruction lets the system drop into debugger.
Second attempt by adding a spare component and reconstructing to the spare has the same effect.
The problem occurs with XEN3 (128MB total memory) or i386 (1GB total Memory) kernel likewise.
Need assistance to restore the redundancy of the raid set.
>How-To-Repeat:
set up a raid5 set with 5 drives of 298 GB and fail one drive:
bremen# raidctl -s raid0
Components:
/dev/wd1a: optimal
/dev/wd2a: optimal
/dev/wd3a: optimal
/dev/wd4a: optimal
/dev/wd5a: failed
No spares.
Component label for /dev/wd1a:
Row: 0, Column: 0, Num Rows: 1, Num Columns: 5
Version: 2, Serial Number: 2007020501, Mod Counter: 1543
Clean: No, Status: 0
sectPerSU: 16, SUsPerPU: 1, SUsPerRU: 1
Queue size: 100, blocksize: 512, numBlocks: 625142320
RAID Level: 5
Autoconfig: Yes
Root partition: No
Last configured as: raid0
Component label for /dev/wd2a:
Row: 0, Column: 1, Num Rows: 1, Num Columns: 5
Version: 2, Serial Number: 2007020501, Mod Counter: 1543
Clean: No, Status: 0
sectPerSU: 16, SUsPerPU: 1, SUsPerRU: 1
Queue size: 100, blocksize: 512, numBlocks: 625142320
RAID Level: 5
Autoconfig: Yes
Root partition: No
Last configured as: raid0
Component label for /dev/wd3a:
Row: 0, Column: 2, Num Rows: 1, Num Columns: 5
Version: 2, Serial Number: 2007020501, Mod Counter: 1543
Clean: No, Status: 0
sectPerSU: 16, SUsPerPU: 1, SUsPerRU: 1
Queue size: 100, blocksize: 512, numBlocks: 625142320
RAID Level: 5
Autoconfig: Yes
Root partition: No
Last configured as: raid0
Component label for /dev/wd4a:
Row: 0, Column: 3, Num Rows: 1, Num Columns: 5
Version: 2, Serial Number: 2007020501, Mod Counter: 1543
Clean: No, Status: 0
sectPerSU: 16, SUsPerPU: 1, SUsPerRU: 1
Queue size: 100, blocksize: 512, numBlocks: 625142320
RAID Level: 5
Autoconfig: Yes
Root partition: No
Last configured as: raid0
/dev/wd5a status is: failed. Skipping label.
Parity status: DIRTY
Reconstruction is 100% complete.
Parity Re-write is 100% complete.
Copyback is 100% complete.
Assign plenty of swapspace:
bremen# swapctl -lh
Device Size Used Avail Capacity Priority
/dev/wd0b 129M 0B 129M 0% 0
/home/fed/swapfile 4.9G 0B 4.9G 0% 0
Total 5.0G 0B 5.0G
Give the command for in-place reconstruction:
bremen# raidctl -R /dev/wd5a raid0
the system will drop into debugger (green letters on console):
raid0: initiating in-place reconstruction on column 4
panic: malloc: out of space in kmem_map
stopped in pid 885.1 (raid_reconip) at netbsd:cpu_Debugger+0x4: popl
%
ebp
db>
the only way out is to reboot the system and operate the raid set in degraded mode
>Fix:
not known.
according to thread "NetBSD-users: Problem with raidframe under NetBSD-3 and NetBSD-4" the system must be taken out of operation.
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: bin-bug-people->oster
Responsible-Changed-By: oster@NetBSD.org
Responsible-Changed-When: Sun, 20 Apr 2008 21:50:20 +0000
Responsible-Changed-Why:
RAIDframe lossage is mine.
From: Greg Oster <oster@cs.usask.ca>
To: thomas@t-online.de
Cc: gnats-bugs@NetBSD.org
Subject: Re: bin/38471: Raidframe crashes on reconstruction of RAID5 (5 disks @ 298GB)
Date: Sun, 20 Apr 2008 15:47:32 -0600
thomas@t-online.de writes:
> >Number: 38471
> >Category: bin
> >Synopsis: Raidframe crashes on reconstruction of RAID5 (5 disks @ 298G
> B)
> >Confidential: no
> >Severity: critical
> >Priority: high
> >Responsible: bin-bug-people
> >State: open
> >Class: sw-bug
> >Submitter-Id: net
> >Arrival-Date: Sun Apr 20 19:20:00 +0000 2008
> >Originator: Thomas Feddersen
> >Release: 4.0_BETA2
> >Organization:
> Dipl.-Ing Thomas Feddersen, Beratender Ingenieur
> >Environment:
> NetBSD bremen 4.0_BETA2 NetBSD 4.0_BETA2 (XEN3_DOM0) #0: Thu Mar 1 04:57:05
> PST 2007 builds@wb33:/home/builds/ab/netbsd-4/i386/200703010002Z-obj/home/bu
> ilds/ab/netbsd-4/src/sys/arch/i386/compile/XEN3_DOM0 i386
> >Description:
> One disk of the raid-set has sectors pending.
> Apr 20 18:09:26 bremen smartd[201]: Device: /dev/wd5d, 1 Currently unreadable
> (pending) sectors
> Apr 20 18:09:26 bremen smartd[201]: Device: /dev/wd5d, 1 Offline uncorrectabl
> e sectors
>
> I attempted an in-place reconstruction of the drive in question, but reconstr
> uction lets the system drop into debugger.
>
> Second attempt by adding a spare component and reconstructing to the spare ha
> s the same effect.
>
> The problem occurs with XEN3 (128MB total memory) or i386 (1GB total Memory)
> kernel likewise.
>
> Need assistance to restore the redundancy of the raid set.
You'll likely have to build a kernel with a bumped up NKMEMPAGES
value... e.g.:
options NKMEMPAGES=32768
You can see what your current values are with:
sysctl vm.nkmempages
and use a larger value.
> >How-To-Repeat:
[snip]
> Give the command for in-place reconstruction:
>
> bremen# raidctl -R /dev/wd5a raid0
>
> the system will drop into debugger (green letters on console):
>
> raid0: initiating in-place reconstruction on column 4
> panic: malloc: out of space in kmem_map
> stopped in pid 885.1 (raid_reconip) at netbsd:cpu_Debugger+0x4: popl
> %
> ebp
> db>
>
> the only way out is to reboot the system and operate the raid set in degraded
> mode
I have a patch nearly done testing that will probably fix this...
Later...
Greg Oster
From: "Thomas Feddersen" <thomas@feddersen-engineering.de>
To: "'Greg Oster'" <oster@cs.usask.ca>
Cc: <gnats-bugs@NetBSD.org>
Subject: Re: bin/38471: Raidframe crashes on reconstruction of RAID5 (5 disks @ 298GB)
Date: Mon, 21 Apr 2008 20:33:58 +0200
Dear Greg,
Sorry, I have submitted an invalid email address in the web-form, the
correct one is on this email.
-------------------------------------------------------------------------=
---
-----------------------
> You'll likely have to build a kernel with a bumped up NKMEMPAGES=20
> value... e.g.:
> =20
> options NKMEMPAGES=3D32768
I built a kernel with=20
options NKMEMPAGES=3D65535
And: it is able to reconstruct the failed RAID component in place.
Thanks for your fast reply, which solved my problem.
I'll donate 100=88 to NetBSD.
Off course I'll still be interested in your fix.
Kind Regards
Thomas
From: Greg Oster <oster@cs.usask.ca>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: bin/38471: Raidframe crashes on reconstruction of RAID5 (5 disks @ 298GB)
Date: Tue, 20 May 2008 16:58:46 -0600
Greg Oster writes:
>
> > >How-To-Repeat:
> [snip]
> > Give the command for in-place reconstruction:
> >
> > bremen# raidctl -R /dev/wd5a raid0
> >
> > the system will drop into debugger (green letters on console):
> >
> > raid0: initiating in-place reconstruction on column 4
> > panic: malloc: out of space in kmem_map
> > stopped in pid 885.1 (raid_reconip) at netbsd:cpu_Debugger+0x4: popl
> > %
> > ebp
> > db>
> >
> > the only way out is to reboot the system and operate the raid set in degra
> ded
> > mode
>
> I have a patch nearly done testing that will probably fix this...
A likely fix for this has been applied to current in the following changes:
cvs rdiff -r1.30 -r1.31 src/sys/dev/raidframe/rf_reconmap.c
cvs rdiff -r1.10 -r1.11 src/sys/dev/raidframe/rf_reconmap.h
cvs rdiff -r1.103 -r1.104 src/sys/dev/raidframe/rf_reconstruct.c
cvs rdiff -r1.23 -r1.24 src/sys/dev/raidframe/rf_reconstruct.h
cvs rdiff -r1.24 -r1.25 src/sys/dev/raidframe/rf_revent.c
Pullups for netbsd-3 and netbsd-4 have been requested. (I forgot
there was an open PR on this, and neglected to put the PR number
in the commit message..)
Later...
Greg Oster
State-Changed-From-To: open->closed
State-Changed-By: oster@NetBSD.org
State-Changed-When: Mon, 22 Sep 2008 15:13:26 +0000
State-Changed-Why:
I believe this was fixed a while ago now.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.