NetBSD Problem Report #38471

From martin@duskware.de  Sun Apr 20 19:16:29 2008
Return-Path: <martin@duskware.de>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by narn.NetBSD.org (Postfix) with ESMTP id BD06663B293
	for <gnats-bugs@gnats.netbsd.org>; Sun, 20 Apr 2008 19:16:28 +0000 (UTC)
Message-Id: <20080420191345.B85A563B293@narn.NetBSD.org>
Date: Sun, 20 Apr 2008 19:13:45 +0000 (UTC)
From: thomas@t-online.de
Reply-To: thomas@t-online.de
To: netbsd-bugs-owner@NetBSD.org
Subject: Raidframe crashes on reconstruction of RAID5 (5 disks @ 298GB)
X-Send-Pr-Version: www-1.0

>Number:         38471
>Category:       bin
>Synopsis:       Raidframe crashes on reconstruction of RAID5 (5 disks @ 298GB)
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    oster
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Apr 20 19:20:00 +0000 2008
>Closed-Date:    Mon Sep 22 15:13:26 +0000 2008
>Last-Modified:  Mon Sep 22 15:13:26 +0000 2008
>Originator:     Thomas Feddersen
>Release:        4.0_BETA2
>Organization:
Dipl.-Ing Thomas Feddersen, Beratender Ingenieur
>Environment:
NetBSD bremen 4.0_BETA2 NetBSD 4.0_BETA2 (XEN3_DOM0) #0: Thu Mar  1 04:57:05 PST 2007  builds@wb33:/home/builds/ab/netbsd-4/i386/200703010002Z-obj/home/builds/ab/netbsd-4/src/sys/arch/i386/compile/XEN3_DOM0 i386
>Description:
One disk of the raid-set has sectors pending. 
Apr 20 18:09:26 bremen smartd[201]: Device: /dev/wd5d, 1 Currently unreadable (pending) sectors
Apr 20 18:09:26 bremen smartd[201]: Device: /dev/wd5d, 1 Offline uncorrectable sectors

I attempted an in-place reconstruction of the drive in question, but reconstruction lets the system drop into debugger.

Second attempt by adding a spare component and reconstructing to the spare has the same effect.

The problem occurs with XEN3 (128MB total memory) or i386 (1GB total Memory) kernel likewise.

Need assistance to restore the redundancy of the raid set.
>How-To-Repeat:
set up a raid5 set with 5 drives of 298 GB and fail one drive:

bremen# raidctl -s raid0
Components:
           /dev/wd1a: optimal
           /dev/wd2a: optimal
           /dev/wd3a: optimal
           /dev/wd4a: optimal
           /dev/wd5a: failed
No spares.
Component label for /dev/wd1a:
   Row: 0, Column: 0, Num Rows: 1, Num Columns: 5
   Version: 2, Serial Number: 2007020501, Mod Counter: 1543
   Clean: No, Status: 0
   sectPerSU: 16, SUsPerPU: 1, SUsPerRU: 1
   Queue size: 100, blocksize: 512, numBlocks: 625142320
   RAID Level: 5
   Autoconfig: Yes
   Root partition: No
   Last configured as: raid0
Component label for /dev/wd2a:
   Row: 0, Column: 1, Num Rows: 1, Num Columns: 5
   Version: 2, Serial Number: 2007020501, Mod Counter: 1543
   Clean: No, Status: 0
   sectPerSU: 16, SUsPerPU: 1, SUsPerRU: 1
   Queue size: 100, blocksize: 512, numBlocks: 625142320
   RAID Level: 5
   Autoconfig: Yes
   Root partition: No
   Last configured as: raid0
Component label for /dev/wd3a:
   Row: 0, Column: 2, Num Rows: 1, Num Columns: 5
   Version: 2, Serial Number: 2007020501, Mod Counter: 1543
   Clean: No, Status: 0
   sectPerSU: 16, SUsPerPU: 1, SUsPerRU: 1
   Queue size: 100, blocksize: 512, numBlocks: 625142320
   RAID Level: 5
   Autoconfig: Yes
   Root partition: No
   Last configured as: raid0
Component label for /dev/wd4a:
   Row: 0, Column: 3, Num Rows: 1, Num Columns: 5
   Version: 2, Serial Number: 2007020501, Mod Counter: 1543
   Clean: No, Status: 0
   sectPerSU: 16, SUsPerPU: 1, SUsPerRU: 1
   Queue size: 100, blocksize: 512, numBlocks: 625142320
   RAID Level: 5
   Autoconfig: Yes
   Root partition: No
   Last configured as: raid0
/dev/wd5a status is: failed.  Skipping label.
Parity status: DIRTY
Reconstruction is 100% complete.
Parity Re-write is 100% complete.
Copyback is 100% complete.

Assign plenty of swapspace:

bremen# swapctl -lh
Device             Size     Used    Avail Capacity  Priority
/dev/wd0b          129M       0B     129M     0%    0
/home/fed/swapfile 4.9G       0B     4.9G     0%    0
Total              5.0G       0B     5.0G 

Give the command for in-place reconstruction:

bremen# raidctl -R /dev/wd5a raid0

the system will drop into debugger (green letters on console):

raid0: initiating in-place reconstruction on column 4
panic: malloc: out of space in kmem_map
stopped in pid 885.1 (raid_reconip) at  netbsd:cpu_Debugger+0x4:  popl
%
ebp
db>

the only way out is to reboot the system and operate the raid set in degraded mode
>Fix:
not known.

according to thread "NetBSD-users: Problem with raidframe under NetBSD-3 and NetBSD-4" the system must be taken out of operation.


>Release-Note:

>Audit-Trail:

Responsible-Changed-From-To: bin-bug-people->oster
Responsible-Changed-By: oster@NetBSD.org
Responsible-Changed-When: Sun, 20 Apr 2008 21:50:20 +0000
Responsible-Changed-Why:
RAIDframe lossage is mine.


From: Greg Oster <oster@cs.usask.ca>
To: thomas@t-online.de
Cc: gnats-bugs@NetBSD.org
Subject: Re: bin/38471: Raidframe crashes on reconstruction of RAID5 (5 disks @ 298GB) 
Date: Sun, 20 Apr 2008 15:47:32 -0600

 thomas@t-online.de writes:
 > >Number:         38471
 > >Category:       bin
 > >Synopsis:       Raidframe crashes on reconstruction of RAID5 (5 disks @ 298G
 > B)
 > >Confidential:   no
 > >Severity:       critical
 > >Priority:       high
 > >Responsible:    bin-bug-people
 > >State:          open
 > >Class:          sw-bug
 > >Submitter-Id:   net
 > >Arrival-Date:   Sun Apr 20 19:20:00 +0000 2008
 > >Originator:     Thomas Feddersen
 > >Release:        4.0_BETA2
 > >Organization:
 > Dipl.-Ing Thomas Feddersen, Beratender Ingenieur
 > >Environment:
 > NetBSD bremen 4.0_BETA2 NetBSD 4.0_BETA2 (XEN3_DOM0) #0: Thu Mar  1 04:57:05 
 > PST 2007  builds@wb33:/home/builds/ab/netbsd-4/i386/200703010002Z-obj/home/bu
 > ilds/ab/netbsd-4/src/sys/arch/i386/compile/XEN3_DOM0 i386
 > >Description:
 > One disk of the raid-set has sectors pending. 
 > Apr 20 18:09:26 bremen smartd[201]: Device: /dev/wd5d, 1 Currently unreadable
 >  (pending) sectors
 > Apr 20 18:09:26 bremen smartd[201]: Device: /dev/wd5d, 1 Offline uncorrectabl
 > e sectors
 > 
 > I attempted an in-place reconstruction of the drive in question, but reconstr
 > uction lets the system drop into debugger.
 > 
 > Second attempt by adding a spare component and reconstructing to the spare ha
 > s the same effect.
 > 
 > The problem occurs with XEN3 (128MB total memory) or i386 (1GB total Memory) 
 > kernel likewise.
 > 
 > Need assistance to restore the redundancy of the raid set.

 You'll likely have to build a kernel with a bumped up NKMEMPAGES 
 value... e.g.:

  options NKMEMPAGES=32768

 You can see what your current values are with:

  sysctl vm.nkmempages

 and use a larger value.

 > >How-To-Repeat:
 [snip]
 > Give the command for in-place reconstruction:
 > 
 > bremen# raidctl -R /dev/wd5a raid0
 > 
 > the system will drop into debugger (green letters on console):
 > 
 > raid0: initiating in-place reconstruction on column 4
 > panic: malloc: out of space in kmem_map
 > stopped in pid 885.1 (raid_reconip) at  netbsd:cpu_Debugger+0x4:  popl
 > %
 > ebp
 > db>
 > 
 > the only way out is to reboot the system and operate the raid set in degraded
 >  mode

 I have a patch nearly done testing that will probably fix this...  

 Later...

 Greg Oster


From: "Thomas Feddersen" <thomas@feddersen-engineering.de>
To: "'Greg Oster'" <oster@cs.usask.ca>
Cc: <gnats-bugs@NetBSD.org>
Subject: Re: bin/38471: Raidframe crashes on reconstruction of RAID5 (5 disks @ 298GB)
Date: Mon, 21 Apr 2008 20:33:58 +0200

 Dear Greg,

 Sorry, I have submitted an invalid email address in the web-form, the
 correct one is on this email.

 -------------------------------------------------------------------------=
 ---
 -----------------------
  > You'll likely have to build a kernel with a bumped up NKMEMPAGES=20
 > value... e.g.:
 > =20
 >  options NKMEMPAGES=3D32768

 I built a kernel with=20
 	options NKMEMPAGES=3D65535

 And: it is able to reconstruct the failed RAID component in place.

 Thanks for your fast reply, which solved my problem.
 I'll donate 100=88 to NetBSD.

 Off course I'll still be interested in your fix.

 Kind Regards
 Thomas


From: Greg Oster <oster@cs.usask.ca>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: bin/38471: Raidframe crashes on reconstruction of RAID5 (5 disks @ 298GB) 
Date: Tue, 20 May 2008 16:58:46 -0600

 Greg Oster writes:
 >
 >  > >How-To-Repeat:
 >  [snip]
 >  > Give the command for in-place reconstruction:
 >  > 
 >  > bremen# raidctl -R /dev/wd5a raid0
 >  > 
 >  > the system will drop into debugger (green letters on console):
 >  > 
 >  > raid0: initiating in-place reconstruction on column 4
 >  > panic: malloc: out of space in kmem_map
 >  > stopped in pid 885.1 (raid_reconip) at  netbsd:cpu_Debugger+0x4:  popl
 >  > %
 >  > ebp
 >  > db>
 >  > 
 >  > the only way out is to reboot the system and operate the raid set in degra
 > ded
 >  >  mode
 >  
 >  I have a patch nearly done testing that will probably fix this...  

 A likely fix for this has been applied to current in the following changes:

 cvs rdiff -r1.30 -r1.31 src/sys/dev/raidframe/rf_reconmap.c
 cvs rdiff -r1.10 -r1.11 src/sys/dev/raidframe/rf_reconmap.h
 cvs rdiff -r1.103 -r1.104 src/sys/dev/raidframe/rf_reconstruct.c
 cvs rdiff -r1.23 -r1.24 src/sys/dev/raidframe/rf_reconstruct.h
 cvs rdiff -r1.24 -r1.25 src/sys/dev/raidframe/rf_revent.c

 Pullups for netbsd-3 and netbsd-4 have been requested.  (I forgot 
 there was an open PR on this, and neglected to put the PR number
 in the commit message..)

 Later...

 Greg Oster


State-Changed-From-To: open->closed
State-Changed-By: oster@NetBSD.org
State-Changed-When: Mon, 22 Sep 2008 15:13:26 +0000
State-Changed-Why:
I believe this was fixed a while ago now.


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.