NetBSD Problem Report #55834
From www@netbsd.org Tue Dec 1 07:26:34 2020
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id AA56F1A9217
for <gnats-bugs@gnats.NetBSD.org>; Tue, 1 Dec 2020 07:26:34 +0000 (UTC)
Message-Id: <20201201072633.47A321A921F@mollari.NetBSD.org>
Date: Tue, 1 Dec 2020 07:26:33 +0000 (UTC)
From: bernd.sieker@posteo.net
Reply-To: bernd.sieker@posteo.net
To: gnats-bugs@NetBSD.org
Subject: dump(8) stops reporting progress after 2 TB of data
X-Send-Pr-Version: www-1.0
>Number: 55834
>Category: bin
>Synopsis: dump(8) stops reporting progress after 2 TB of data
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: kre
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Tue Dec 01 07:30:00 +0000 2020
>Closed-Date: Mon Dec 07 21:27:13 +0000 2020
>Last-Modified: Mon Dec 07 21:27:13 +0000 2020
>Originator: Bernd Sieker
>Release: NetBSD 9.1_STABLE
>Organization:
>Environment:
NetBSD niob.bersie.loc 9.1_STABLE NetBSD 9.1_STABLE (NIOB) #37: Mon Nov 23 12:14:35 CET 2020 bernd@niob.bersie.loc:/usr/source/netbsd-9/src/sys/arch/amd64/compile/NIOB amd64
>Description:
When dumping a large file system, the dump program stops reporting progress when more than 2 TB of data have been dumped. It continues the dump process, the resulting file has a size of just under 3 TB:
$ ls -lh
-rw-r--r-- 1 bernd users 2.9T Nov 28 23:48 niob-home-L0-2020-11-27.dmp
After finishing, dump also reports a credible amount of (10k) blocks in the dump. The last lines of the dump output were as follows:
DUMP: 67.60% done, finished in 11:42
DUMP: 67.86% done, finished in 11:36
DUMP: 68.13% done, finished in 11:30
DUMP: 68.39% done, finished in 11:24
DUMP: 68.65% done, finished in 11:18
DUMP: 68.92% done, finished in 11:12
DUMP: 69.18% done, finished in 11:05
DUMP: 3096903227 tape blocks
DUMP: Volume 1 completed at: Sat Nov 28 23:48:42 2020
DUMP: Volume 1 took 35:24:00
DUMP: Volume 1 transfer rate: 24300 KB/s
DUMP: Date of this level 0 dump: Fri Nov 27 12:22:31 2020
DUMP: Date this dump completed: Sat Nov 28 23:48:42 2020
DUMP: Average transfer rate: 24300 KB/s
DUMP: DUMP IS DONE
Note that progress reports were made regularly until 69.18%, which corresponds to 2 Terabytes of data having been dumped. Reports stop, but dumping continues to the end, and the final report seems accurate.
Spot tests and hash comparisons indicate that the dump itself is also correct and complete.
>How-To-Repeat:
Use dump(8) to dump a filesystem on which more than 2 TB of data would be backed up, observe the status reports on the controlling terminal. See that reporting stops after 2 TB of data have been dumped, although dumping continues.
>Fix:
Unknown.
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: bin-bug-people->kre
Responsible-Changed-By: kre@NetBSD.org
Responsible-Changed-When: Tue, 01 Dec 2020 09:27:05 +0000
Responsible-Changed-Why:
I am looking into this PR
From: Robert Elz <kre@munnari.OZ.AU>
To: bernd.sieker@posteo.net, gnats-bugs@netbsd.org
Cc:
Subject: Re: bin/55834: dump(8) stops reporting progress after 2 TB of data
Date: Tue, 01 Dec 2020 16:26:23 +0700
Can you try the following patch to src/sbin/dump ?
I think it is likely to apply fairly cleanly to any (moderately recent)
source tree - but the sole change is to alter the type of the
blockswritten variable from int to u_int64_t.
A version of NetBSD old enough might be missing the "extern" in dump.h,
and the variable declaration in main.c entirely, that's OK, just
change the data type from int to u_int64_t (it should be the same as the
"tapesize" variable).
If this works for you I'll commit the change to the tree, and request pullups.
kre
Index: dump.h
===================================================================
RCS file: /cvsroot/src/sbin/dump/dump.h,v
retrieving revision 1.58
diff -u -r1.58 dump.h
--- dump.h 5 Apr 2020 15:25:39 -0000 1.58
+++ dump.h 1 Dec 2020 09:21:18 -0000
@@ -134,7 +134,7 @@
extern int density; /* density in 0.1" units */
extern int notify; /* notify operator flag */
extern int timestamp; /* timestamp messages */
-extern int blockswritten; /* number of blocks written on current tape */
+extern u_int64_t blockswritten; /* blocks written on current tape */
extern int tapeno; /* current tape number */
extern int is_ufs2;
Index: main.c
===================================================================
RCS file: /cvsroot/src/sbin/dump/main.c,v
retrieving revision 1.77
diff -u -r1.77 main.c
--- main.c 5 Apr 2020 15:25:39 -0000 1.77
+++ main.c 1 Dec 2020 09:21:19 -0000
@@ -108,7 +108,7 @@
int timestamp; /* print message timestamps */
int notify; /* notify operator flag */
-int blockswritten; /* number of blocks written on current tape */
+u_int64_t blockswritten; /* number of blocks written on current tape */
int tapeno; /* current tape number */
int density; /* density in bytes/0.1" */
int ntrec = NTREC; /* # tape blocks in each tape record */
From: bernd.sieker@posteo.net
To: gnats-bugs@netbsd.org, kre@netbsd.org
Cc:
Subject: Re: bin/55834: dump(8) stops reporting progress after 2 TB of data
Date: Wed, 02 Dec 2020 17:17:29 +0100
The patch works fine:
DUMP: 93.92% done, finished in 0:19
DUMP: 95.62% done, finished in 0:13
DUMP: 97.15% done, finished in 0:09
DUMP: 98.76% done, finished in 0:03
DUMP: 3139324907 tape blocks on 1 volume
DUMP: Volume 1 completed at: Wed Dec 2 16:41:09 2020
DUMP: Volume 1 took 5:18:50
DUMP: Volume 1 transfer rate: 164104 KB/s
DUMP: Date of this level 0 dump: Wed Dec 2 11:20:11 2020
DUMP: Date this dump completed: Wed Dec 2 16:41:09 2020
DUMP: Average transfer rate: 164104 KB/s
DUMP: Closing /dev/null
DUMP: DUMP IS DONE
Reporting all the way to the end.
Thanks a lot. It went a lot faster than the original dump, since I only
dumped to /dev/null, which went at 160 MB/s instead of 35 MB/s over the
network.
Bernd
State-Changed-From-To: open->needs-pullups
State-Changed-By: kre@NetBSD.org
State-Changed-When: Thu, 03 Dec 2020 08:27:04 +0000
State-Changed-Why:
Needs pullups to -8 and -9
From: "Robert Elz" <kre@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/55834 CVS commit: src/sbin/dump
Date: Thu, 3 Dec 2020 08:25:57 +0000
Module Name: src
Committed By: kre
Date: Thu Dec 3 08:25:57 UTC 2020
Modified Files:
src/sbin/dump: dump.h main.c
Log Message:
PR bin/55834
count blocks written in unsigned 64 bit counter
rather than signed int which overflows after 2^31-1
blocks (2TiB) after which neither the 5 minute
status updates or SIGINFO (^T) reports are issued
until the negative numbers increase past 0 and
wildly inaccurate reports would be written.
To generate a diff of this commit:
cvs rdiff -u -r1.58 -r1.59 src/sbin/dump/dump.h
cvs rdiff -u -r1.77 -r1.78 src/sbin/dump/main.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: needs-pullups->pending-pullups
State-Changed-By: kre@NetBSD.org
State-Changed-When: Thu, 03 Dec 2020 19:46:59 +0000
State-Changed-Why:
[pullup-9 #1139] has been requested.
This PR should return to needs-pullups state when that is handled,
as a pullup for -8 is still required (local logistical issues mean
it will take me longer to generate that one).
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/55834 CVS commit: [netbsd-9] src/sbin/dump
Date: Sun, 6 Dec 2020 10:31:02 +0000
Module Name: src
Committed By: martin
Date: Sun Dec 6 10:31:02 UTC 2020
Modified Files:
src/sbin/dump [netbsd-9]: dump.h main.c
Log Message:
Pull up following revision(s) (requested by kre in ticket #1139):
sbin/dump/dump.h: revision 1.59
sbin/dump/main.c: revision 1.78
PR bin/55834
count blocks written in unsigned 64 bit counter
rather than signed int which overflows after 2^31-1
blocks (2TiB) after which neither the 5 minute
status updates or SIGINFO (^T) reports are issued
until the negative numbers increase past 0 and
wildly inaccurate reports would be written.
To generate a diff of this commit:
cvs rdiff -u -r1.57 -r1.57.2.1 src/sbin/dump/dump.h
cvs rdiff -u -r1.75 -r1.75.2.1 src/sbin/dump/main.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: pending-pullups->needs-pullups
State-Changed-By: kre@NetBSD.org
State-Changed-When: Sun, 06 Dec 2020 12:44:40 +0000
State-Changed-Why:
Pullup to -9 complete. Still needs pullup to -8.
State-Changed-From-To: needs-pullups->pending-pullups
State-Changed-By: kre@NetBSD.org
State-Changed-When: Mon, 07 Dec 2020 00:16:34 +0000
State-Changed-Why:
[pullup-8 #1630] requested
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/55834 CVS commit: [netbsd-8] src/sbin/dump
Date: Mon, 7 Dec 2020 19:35:50 +0000
Module Name: src
Committed By: martin
Date: Mon Dec 7 19:35:50 UTC 2020
Modified Files:
src/sbin/dump [netbsd-8]: dump.h main.c
Log Message:
Pull up following revision(s) (requested by kre in ticket #1630):
sbin/dump/dump.h: revision 1.59
sbin/dump/main.c: revision 1.78
PR bin/55834
count blocks written in unsigned 64 bit counter
rather than signed int which overflows after 2^31-1
blocks (2TiB) after which neither the 5 minute
status updates or SIGINFO (^T) reports are issued
until the negative numbers increase past 0 and
wildly inaccurate reports would be written.
To generate a diff of this commit:
cvs rdiff -u -r1.54.8.1 -r1.54.8.2 src/sbin/dump/dump.h
cvs rdiff -u -r1.73.8.1 -r1.73.8.2 src/sbin/dump/main.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: pending-pullups->closed
State-Changed-By: kre@NetBSD.org
State-Changed-When: Mon, 07 Dec 2020 21:27:13 +0000
State-Changed-Why:
Pullups completed
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.46 2020/01/03 16:35:01 leot Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2020
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home