NetBSD Problem Report #56008

From www@netbsd.org  Tue Feb 23 02:50:25 2021
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 36A531A921F
	for <gnats-bugs@gnats.NetBSD.org>; Tue, 23 Feb 2021 02:50:25 +0000 (UTC)
Message-Id: <20210223025023.C67891A9245@mollari.NetBSD.org>
Date: Tue, 23 Feb 2021 02:50:23 +0000 (UTC)
From: greg-org-netbsd@akua.com
Reply-To: greg-org-netbsd@akua.com
To: gnats-bugs@NetBSD.org
Subject: Zpool lost when disk is moved to another bus
X-Send-Pr-Version: www-1.0

>Number:         56008
>Category:       kern
>Synopsis:       Zpool lost when disk is moved to another bus
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Feb 23 02:55:00 +0000 2021
>Last-Modified:  Tue Feb 23 09:40:00 +0000 2021
>Originator:     Greg Kerr
>Release:        9.1_STABLE
>Organization:
Akua
>Environment:
NetBSD lion.mlb.akua.com 9.1_STABLE NetBSD 9.1_STABLE (Akua Lion lion-90-a $Revision: 1.531.2.10 $) #12: Mon Jan 11 16:27:59 EST 2021  akua@lion.mlb.akua.com:/usr/src/OUT/netbsd-9/kernel/lion-90-a amd64

>Description:
Created two zpools

One M.2/SATA 1TB SSD
One SATA 2TB SSD

Wanted to replace them
1. Shutdown 
2. Moved to USB enclosures. USB-SATA, USB-M.2-SATA
3. Inserted new M.2 NVMe
4. Booted (from mSATA)
5. One Zpool was there, the other not.


 $ sudo zpool status
  pool: M2
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        M2          ONLINE       0     0     0
          dk9       ONLINE       0     0     0

errors: No known data errors

  pool: QVO
 state: UNAVAIL
status: One or more devices could not be opened.  There are insufficient
        replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-3C
  scan: none requested
config:

        NAME                   STATE     READ WRITE CKSUM
        QVO                    UNAVAIL      0     0     0
          7099944016363518513  UNAVAIL      0     0     0  was /dev/wd1

 $ sudo zpool online QVO 7099944016363518513 sd0
cannot open 'QVO': pool is unavailable

 $ sudo zpool online QVO 7099944016363518513 rsd0
cannot open 'QVO': pool is unavailable

Difference was, M2 is GPT partitioned (created wedges), QVO was whole device. hexdump -C /dev/sd0 shows it is attached and reading fine.

 $ dmesg | grep sd0
[     3.150596] sd0 at scsibus0 target 0 lun 0: <Seagate, BACKUP+, 0304> disk fixed
[     3.190596] sd0: 1863 GB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 3907029167 sectors

 $ dmesg | grep sd1
[     3.240596] sd1 at scsibus1 target 0 lun 0: <X80 XDIS, , 2210> disk fixed
[     3.280596] sd1: fabricating a geometry
[     3.310596] sd1: 931 GB, 953869 cyl, 64 head, 32 sec, 512 bytes/sect x 1953525168 sectors
[     3.350596] sd1: fabricating a geometry
[     3.380596] sd1: GPT GUID: 2da609c5-3876-4e21-98c6-caa3ee7cf210
[     3.390596] dk6 at sd1: "ESP_M2", 614361 blocks at 40, type: msdos
[     3.410596] dk7 at sd1: "NetBSD", 134217728 blocks at 614440, type: ffs
[     3.440596] dk8 at sd1: "swap", 67108864 blocks at 134832200, type: swap
[     3.470596] dk9 at sd1: "NetBSD ZFS", 1751584055 blocks at 201941080, type: <unknown>
[     7.350596] boot device: sd1

>How-To-Repeat:
1. Create whole device zpool
2. Power down
3. Move to another bus
4. Power up
5. zpool status

>Fix:
Move back to original location

>Audit-Trail:
From: mlelstv@serpens.de (Michael van Elst)
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/56008: Zpool lost when disk is moved to another bus
Date: Tue, 23 Feb 2021 06:45:52 -0000 (UTC)

 greg-org-netbsd@akua.com writes:

 >1. Shutdown 
 >2. Moved to USB enclosures. USB-SATA, USB-M.2-SATA
 >3. Inserted new M.2 NVMe
 >4. Booted (from mSATA)
 >5. One Zpool was there, the other not.

 The devices are cached by their path names (i.e. /dev/dk9), when
 the paths change the cache becomes invalid.

 Workaround is to export and re-import the pool. When there is no
 cache information left (after export), all disks are scanned for
 ZFS labels to find the pool disks.

 -- 
 -- 
                                 Michael van Elst
 Internet: mlelstv@serpens.de
                                 "A potential Snark may lurk in every tree."

From: Greg Kerr <Greg@Akua.com>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/56008: Zpool lost when disk is moved to another bus
Date: Tue, 23 Feb 2021 03:22:57 -0500

 Worked. Thank you. Should have occurred to me :(

  - Greg

 > On 23 Feb 2021, at 01:50, Michael van Elst <mlelstv@serpens.de> wrote:
 > 
 > The following reply was made to PR kern/56008; it has been noted by GNATS.
 > 
 > From: mlelstv@serpens.de (Michael van Elst)
 > To: gnats-bugs@netbsd.org
 > Cc: 
 > Subject: Re: kern/56008: Zpool lost when disk is moved to another bus
 > Date: Tue, 23 Feb 2021 06:45:52 -0000 (UTC)
 > 
 > greg-org-netbsd@akua.com writes:
 > 
 >> 1. Shutdown 
 >> 2. Moved to USB enclosures. USB-SATA, USB-M.2-SATA
 >> 3. Inserted new M.2 NVMe
 >> 4. Booted (from mSATA)
 >> 5. One Zpool was there, the other not.
 > 
 > The devices are cached by their path names (i.e. /dev/dk9), when
 > the paths change the cache becomes invalid.
 > 
 > Workaround is to export and re-import the pool. When there is no
 > cache information left (after export), all disks are scanned for
 > ZFS labels to find the pool disks.
 > 
 > -- 
 > -- 
 >                                 Michael van Elst
 > Internet: mlelstv@serpens.de
 >                                 "A potential Snark may lurk in every tree."
 > 

Home	PR Database Search