NetBSD Problem Report #51116
From brad@anduin.eldar.org Thu May 5 20:23:15 2016
Return-Path: <brad@anduin.eldar.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
by mollari.NetBSD.org (Postfix) with ESMTPS id 1F3AE7A490
for <gnats-bugs@gnats.NetBSD.org>; Thu, 5 May 2016 20:23:15 +0000 (UTC)
Message-Id: <201605052023.u45KNAnG015145@anduin.eldar.org>
Date: Thu, 5 May 2016 16:23:10 -0400 (EDT)
From: brad@anduin.eldar.org
Reply-To: brad@anduin.eldar.org
To: gnats-bugs@NetBSD.org
Subject: resize_ffs has problems with non-zero filled expansion of an ffsv2 filesystem
X-Send-Pr-Version: 3.95
>Number: 51116
>Category: bin
>Synopsis: resize_ffs has problems with non-zero filled expansion of an ffsv2 filesystem
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: bin-bug-people
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Thu May 05 20:25:00 +0000 2016
>Closed-Date: Sat Sep 10 18:36:32 +0000 2016
>Last-Modified: Sat Sep 10 18:36:32 +0000 2016
>Originator: Brad Spencer
>Release: NetBSD 7.0_STABLE
>Organization:
Home
>Environment:
System: NetBSD anduin.eldar.org 7.0_STABLE NetBSD 7.0_STABLE (ANDUIN) #0: Thu Feb 25 12:38:29 EST 2016 brad@gimli.nat.eldar.org:/usr/src/sys/arch/amd64/compile/ANDUIN amd64
Architecture: x86_64
Machine: amd64
>Description:
I had noticed something messy at times when I would expand the
underling lvm size of a filesystem for a guest Xen VM. After
lvextending the size, and then presenting it to the VM, and running
resize_ffs in the guest, the filesystem would have inconsistency,
sometimes quite severe. I initially attributed this to having wapbl
enabled for the filesystem before expanding it, but I have found a
test case that illustrates this without wapbl enabled.
I narrowed the issue down to ffsv2 filesystems that are expanded in a
non zero-filled manor.
>How-To-Repeat:
Sorry for the length of this, I wanted to provide a couple of test cases that illustrate the issue:
First create some images that will be used for the test. Two are
random blobs and a zero filled blob to show that it doesn't happen
there.
% dd if=/dev/urandom of=random.fs bs=1048576 count=10
10+0 records in
10+0 records out
10485760 bytes transferred in 0.300 secs (34952533 bytes/sec)
% dd if=/dev/urandom of=morerandom.fs bs=1048576 count=10
10+0 records in
10+0 records out
10485760 bytes transferred in 0.263 secs (39869809 bytes/sec)
% dd if=/dev/zero of=zero.fs bs=1048576 count=10
10+0 records in
10+0 records out
10485760 bytes transferred in 0.040 secs (262144000 bytes/sec)
Second.. make a ffsv2 filesystem of 10megs in size and resize it to 20megs:
# cp random.fs test_ffsv2.fs
# vnconfig vnd0 test_ffsv2.fs
# disklabel -e vnd0
# disklabel vnd0 | grep '^ a'
a: 20480 0 4.2BSD 4096 32768 16 # (Cyl. 0 - 9)
# newfs -O2 vnd0a
/dev/rvnd0a: 10.0MB (20480 sectors) block size 4096, fragment size 512
using 4 cylinder groups of 2.50MB, 640 blks, 1136 inodes.
super-block backups (for fsck_ffs -b #) at:
144, 5264, 10384, 15504,
# dumpfs -s vnd0a
file system: /dev/rvnd0a
format FFSv2
endian little-endian
location 65536 (-b 128)
magic 19540119 time Thu May 5 15:46:37 2016
superblock location 65536 id [ 572ba31d 72225649 ]
cylgrp dynamic inodes FFSv2 sblock FFSv2 fslevel 5
nbfree 2244 ndir 1 nifree 4541 nffree 14
.
.
.
# fsck -fy vnd0a
** /dev/rvnd0a
** File system is already clean
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1 files, 1 used, 17966 free (14 frags, 2244 blocks, 0.1% fragmentation)
Expand it by concating on another 10meg blob:
# vnconfig -u vnd0
# cat morerandom.fs >> test_ffsv2.fs
# vnconfig vnd0 test_ffsv2.fs
# fsck -fy vnd0a
** /dev/rvnd0a
** File system is already clean
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1 files, 1 used, 17966 free (14 frags, 2244 blocks, 0.1% fragmentation)
# disklabel -e vnd0
# disklabel vnd0 | grep '^ a'
a: 40960 0 4.2BSD 4096 32768 16 # (Cyl. 0 - 19)
# fsck -fy vnd0a
** /dev/rvnd0a
** File system is already clean
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1 files, 1 used, 17966 free (14 frags, 2244 blocks, 0.1% fragmentation)
# resize_ffs /dev/rvnd0a
It's required to manually run fsck on file system before you can resize it
Did you run fsck on your disk (Yes/No) ? Yes
The "damage" that fsck will report will vary depending on the random
bytes that are present in the new expanded space. Sometimes, files
will be put in lost+found that are hard to remove as they can have any
sorts of flags sets including schg. Further in one instance it
required another fsck after the files were unlinked to completely
clean the filesystem.
Note that fsck said that the file system was clean, when it obviously
wasn't.
# fsck -fy vnd0a
** /dev/rvnd0a
** File system is already clean
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
UNKNOWN FILE TYPE I=4544
CLEAR? yes
UNKNOWN FILE TYPE I=4545
CLEAR? yes
.
.
.
UNKNOWN FILE TYPE I=7981
CLEAR? yes
UNKNOWN FILE TYPE I=7982
CLEAR? yes
PARTIALLY ALLOCATED INODE I=7983
CLEAR? yes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1 files, 1 used, 36078 free (14 frags, 4508 blocks, 0.0% fragmentation)
***** FILE SYSTEM WAS MODIFIED *****
The file system is clean now, and probably ok. I do not have any
knowledge of losing anything when this happens.
# fsck -fy vnd0a
** /dev/rvnd0a
** File system is already clean
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1 files, 1 used, 36078 free (14 frags, 4508 blocks, 0.0% fragmentation)
# mount /dev/vnd0a /mnt
# ls -l /mnt
# df /mnt
Filesystem 1K-blocks Used Avail %Cap Mounted on
/dev/vnd0a 18039 0 17137 0% /mnt
Now, do the same thing with a zero filled blob:
# cp zero.fs test_ffsv2_zero.fs
# vnconfig vnd0 test_ffsv2_zero.fs
# disklabel -e vnd0
# newfs -O2 vnd0a
/dev/rvnd0a: 10.0MB (20480 sectors) block size 4096, fragment size 512
using 4 cylinder groups of 2.50MB, 640 blks, 1136 inodes.
super-block backups (for fsck_ffs -b #) at:
144, 5264, 10384, 15504,
# fsck -fy vnd0a
** /dev/rvnd0a
** File system is already clean
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1 files, 1 used, 17966 free (14 frags, 2244 blocks, 0.1% fragmentation)
# vnconfig -u vnd0
# cat zero.fs >> test_ffsv2_zero.fs
# vnconfig vnd0 test_ffsv2_zero.fs
# fsck -fy vnd0a
** /dev/rvnd0a
** File system is already clean
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1 files, 1 used, 17966 free (14 frags, 2244 blocks, 0.1% fragmentation)
# disklabel -e vnd0
# disklabel vnd0 | grep '^ a'
a: 40960 0 4.2BSD 4096 32768 16 # (Cyl. 0 - 19)
# fsck -fy vnd0a
** /dev/rvnd0a
** File system is already clean
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1 files, 1 used, 17966 free (14 frags, 2244 blocks, 0.1% fragmentation)
# resize_ffs /dev/rvnd0a
It's required to manually run fsck on file system before you can resize it
Did you run fsck on your disk (Yes/No) ? Yes
Note that the file system is clean after the resize, which suggests
that something is not getting zero'ed out with resize_ffs.
# fsck -fy /dev/rvnd0a
** /dev/rvnd0a
** File system is already clean
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1 files, 1 used, 36078 free (14 frags, 4508 blocks, 0.0% fragmentation)
Final test, use the same random blobs and try this with a ffsv1 filesystem:
# cp random.fs test_ffsv1.fs
# vnconfig vnd0 test_ffsv1.fs
# disklabel -e vnd0
# disklabel vnd0 | grep '^ a'
a: 20480 0 4.2BSD 4096 32768 16 # (Cyl. 0 - 9)
# newfs vnd0a
/dev/rvnd0a: 10.0MB (20480 sectors) block size 4096, fragment size 512
using 4 cylinder groups of 2.50MB, 640 blks, 1216 inodes.
super-block backups (for fsck_ffs -b #) at:
32, 5152, 10272, 15392,
# dumpfs -s /dev/rvnd0a
file system: /dev/rvnd0a
format FFSv1
endian little-endian
magic 11954 time Thu May 5 15:58:21 2016
superblock location 8192 id [ 572ba5dd 3831964d ]
cylgrp dynamic inodes 4.4BSD sblock FFSv2 fslevel 4
# fsck -fy vnd0a
** /dev/rvnd0a
** File system is already clean
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1 files, 1 used, 19134 free (14 frags, 2390 blocks, 0.1% fragmentation)
# cat morerandom.fs >> test_ffsv1.fs
# vnconfig vnd0 test_ffsv1.fs
# fsck -fy vnd0a
** /dev/rvnd0a
** File system is already clean
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1 files, 1 used, 19134 free (14 frags, 2390 blocks, 0.1% fragmentation)
# disklabel -e vnd0
# disklabel vnd0 | grep '^ a'
a: 40960 0 4.2BSD 4096 32768 16 # (Cyl. 0 - 19)
# fsck -fy vnd0a
** /dev/rvnd0a
** File system is already clean
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1 files, 1 used, 19134 free (14 frags, 2390 blocks, 0.1% fragmentation)
# resize_ffs /dev/rvnd0a
It's required to manually run fsck on file system before you can resize it
Did you run fsck on your disk (Yes/No) ? Yes
Note that this is the exact same random blobs used in the ffsv2 case,
except here the filesystem is clean.
# fsck -fy vnd0a
** /dev/rvnd0a
** File system is already clean
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1 files, 1 used, 38302 free (14 frags, 4786 blocks, 0.0% fragmentation)
>Fix:
As an unreasonable work around one could always zero fill the space
before adding it to the filesystem.
>Release-Note:
>Audit-Trail:
From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: bin/51116: resize_ffs has problems with non-zero filled
expansion of an ffsv2 filesystem
Date: Sun, 14 Aug 2016 00:31:10 +0000
On Thu, May 05, 2016 at 08:25:00PM +0000, brad@anduin.eldar.org wrote:
> I narrowed the issue down to ffsv2 filesystems that are expanded in a
> non zero-filled manor.
I think the problem is that resize_ffs is not handling ffsv2's
deferred inode initialization correctly.
It does
cg->cg_initediblk = newsb->fs_ipg < 2 * FFS_INOPB(newsb) ?
newsb->fs_ipg : 2 * FFS_INOPB(newsb);
(line 497 in initcg) which declares that two blocks' worth of inodes
have been written out zeroed, but then later (at line 619) it doesn't
write the inodes out at all if ffsv2.
If this is the problem the following 100% untested patch might improve
the situation:
Index: resize_ffs.c
===================================================================
RCS file: /cvsroot/src/sbin/resize_ffs/resize_ffs.c,v
retrieving revision 1.46
diff -u -p -r1.46 resize_ffs.c
--- resize_ffs.c 17 Mar 2016 01:41:54 -0000 1.46
+++ resize_ffs.c 14 Aug 2016 00:29:11 -0000
@@ -105,7 +105,8 @@ union dinode {
} while (0)
/* a cg's worth of brand new squeaky-clean inodes */
-static struct ufs1_dinode *zinodes;
+static struct ufs1_dinode *zinodes1;
+static struct ufs2_dinode *zinodes2;
/* pointers to the in-core cgs, read off disk and possibly modified */
static struct cg **cgs;
@@ -615,10 +616,15 @@ initcg(int cgn)
newsb->fs_cstotal.cs_nffree += cg->cg_cs.cs_nffree;
newsb->fs_cstotal.cs_nbfree += cg->cg_cs.cs_nbfree;
newsb->fs_cstotal.cs_nifree += cg->cg_cs.cs_nifree;
- if (is_ufs2 == 0)
+ if (is_ufs2) {
+ /* Write out the cleared inodes. */
+ writeat(FFS_FSBTODB(newsb, cgimin(newsb, cgn)), zinodes2,
+ cg->cg_initediblk * sizeof(*zinodes2));
+ } else {
/* Write out the cleared inodes. */
- writeat(FFS_FSBTODB(newsb, cgimin(newsb, cgn)), zinodes,
- newsb->fs_ipg * sizeof(*zinodes));
+ writeat(FFS_FSBTODB(newsb, cgimin(newsb, cgn)), zinodes1,
+ newsb->fs_ipg * sizeof(*zinodes1));
+ }
/* Dirty the cg. */
cgflags[cgn] |= CGF_DIRTY;
}
@@ -993,8 +999,15 @@ grow(void)
/* Update the timestamp. */
newsb->fs_time = timestamp();
/* Allocate and clear the new-inode area, in case we add any cgs. */
- zinodes = alloconce(newsb->fs_ipg * sizeof(*zinodes), "zeroed inodes");
- memset(zinodes, 0, newsb->fs_ipg * sizeof(*zinodes));
+ if (is_ufs2) {
+ zinodes2 = alloconce(newsb->fs_ipg * sizeof(*zinodes2),
+ "zeroed inodes");
+ memset(zinodes2, 0, newsb->fs_ipg * sizeof(*zinodes2));
+ } else {
+ zinodes1 = alloconce(newsb->fs_ipg * sizeof(*zinodes1),
+ "zeroed inodes");
+ memset(zinodes1, 0, newsb->fs_ipg * sizeof(*zinodes1));
+ }
/* Check that the new last sector (frag, actually) is writable. Since
* it's at least one frag larger than it used to be, we know we aren't
--
David A. Holland
dholland@netbsd.org
From: Brad Spencer <brad@anduin.eldar.org>
To: gnats-bugs@NetBSD.org
Cc: gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: bin/51116: resize_ffs has problems with non-zero filled
expansion of an ffsv2 filesystem
Date: Tue, 16 Aug 2016 11:07:34 -0400
David Holland <dholland-bugs@netbsd.org> writes:
[snip]
> On Thu, May 05, 2016 at 08:25:00PM +0000, brad@anduin.eldar.org wrote:
> > I narrowed the issue down to ffsv2 filesystems that are expanded in a
> > non zero-filled manor.
>
> I think the problem is that resize_ffs is not handling ffsv2's
> deferred inode initialization correctly.
>
> It does
>
> cg->cg_initediblk = newsb->fs_ipg < 2 * FFS_INOPB(newsb) ?
> newsb->fs_ipg : 2 * FFS_INOPB(newsb);
>
> (line 497 in initcg) which declares that two blocks' worth of inodes
> have been written out zeroed, but then later (at line 619) it doesn't
> write the inodes out at all if ffsv2.
>
> If this is the problem the following 100% untested patch might improve
> the situation:
>
[snip]
>
> --
> David A. Holland
> dholland@netbsd.org
>
I applied the 100% untested patch to a NetBSD 7.x system and tested it.
I was not able to reproduce the problem I saw with the unpatched
resize_ffs which suggests that the patch improves the situation. I
would suggest that this patch should be applied to the code and pulled
into 7.x if possible.
Thanks for looking into this.
--
Brad Spencer - brad@anduin.eldar.org - KC8VKS
http://anduin.eldar.org - & - http://anduin.ipv6.eldar.org [IPv6 only]
From: "David A. Holland" <dholland@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/51116 CVS commit: src/sbin/resize_ffs
Date: Wed, 24 Aug 2016 07:44:05 +0000
Module Name: src
Committed By: dholland
Date: Wed Aug 24 07:44:05 UTC 2016
Modified Files:
src/sbin/resize_ffs: resize_ffs.c
Log Message:
Fix handling of ffsv2 inode initialization. Retires PR 51116.
To generate a diff of this commit:
cvs rdiff -u -r1.46 -r1.47 src/sbin/resize_ffs/resize_ffs.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->pending-pullups
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Fri, 26 Aug 2016 23:56:57 +0000
State-Changed-Why:
pullup-7 #1242
From: "Soren Jacobsen" <snj@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/51116 CVS commit: [netbsd-7] src/sbin/resize_ffs
Date: Sat, 10 Sep 2016 06:36:37 +0000
Module Name: src
Committed By: snj
Date: Sat Sep 10 06:36:37 UTC 2016
Modified Files:
src/sbin/resize_ffs [netbsd-7]: resize_ffs.c
Log Message:
Pull up following revision(s) (requested by dholland in ticket #1242):
sbin/resize_ffs/resize_ffs.c: revision 1.47
Fix handling of ffsv2 inode initialization. Retires PR 51116.
To generate a diff of this commit:
cvs rdiff -u -r1.38.6.1 -r1.38.6.2 src/sbin/resize_ffs/resize_ffs.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: pending-pullups->closed
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Sat, 10 Sep 2016 18:36:32 +0000
State-Changed-Why:
fixed, thanks
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home