NetBSD Problem Report #10826

Received: (qmail 8180 invoked from network); 14 Aug 2000 00:51:46 -0000
Message-Id: <20000814005142.D96DC1FE0A@crasse.smd.ebone.net>
Date: Mon, 14 Aug 2000 00:51:42 +0000 (GMT)
From: smd@ebone.net
Reply-To: smd@ab.use.net
To: gnats-bugs@gnats.netbsd.org
Subject: restore does not DTRT at end of volume on DVD-RAM
X-Send-Pr-Version: 3.95

>Number:         10826
>Category:       bin
>Synopsis:       restore does not DTRT at end of volume on DVD-RAM
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    bin-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Aug 14 00:52:00 +0000 2000
>Closed-Date:    
>Last-Modified:  Tue Feb 05 17:24:14 +0000 2002
>Originator:     Sean Doran
>Release:        CVS 2000 08 13
>Organization:

>Environment:

System: NetBSD crasse.smd.ebone.net 1.5E NetBSD 1.5E (SCREAM) #0: Thu Aug 10 04:03:43 CEST 2000 smd@crasse.smd.ebone.net:/usr/src/sys/arch/i386/compile/SCREAM i386

>Description:

	dump to two dvd ram sides
	then try to restore everything; at the end of the first volume,
	  dump chokes

>How-To-Repeat:

dump -b 2 -B 2430000 -f /dev/rcd0d MD5 *.gz
  DUMP: Dumping sub files/directories from /scratch2
  DUMP: Dumping file/directory MD5
  DUMP: Dumping file/directory crasse.root.0.gz
  DUMP: Dumping file/directory crasse.u.0.gz
  DUMP: Dumping file/directory crasse.usr-X11R6.0.gz
  DUMP: Dumping file/directory crasse.usr-local.0.gz
  DUMP: Dumping file/directory crasse.usr-pkg.0.gz
  DUMP: Dumping file/directory crasse.usr.0.gz
  DUMP: Dumping file/directory crasse.var.0.gz
  DUMP: Date of this level 0 dump: Mon Aug 14 00:12:28 2000
  DUMP: Date of last level 0 dump: the epoch
  DUMP: Dumping a subset of /dev/rsd3a (a subset of /scratch2) to /dev/
  DUMP: Label: none
  DUMP: mapping (Pass I) [regular files]
  DUMP: mapping (Pass II) [directories]
  DUMP: estimated 3685151 tape blocks on 1.52 tape(s).
  DUMP: Volume 1 started at: Mon Aug 14 00:12:28 2000
  DUMP: dumping (Pass III) [directories]
  DUMP: dumping (Pass IV) [regular files]
  DUMP: 5.05% done, finished in 1:34
  DUMP: 10.06% done, finished in 1:29
  DUMP: 15.03% done, finished in 1:24
  DUMP: 20.01% done, finished in 1:19
  DUMP: 25.04% done, finished in 1:14
  DUMP: 30.07% done, finished in 1:09
  DUMP: 35.10% done, finished in 1:04
  DUMP: 40.09% done, finished in 0:59
  DUMP: 45.12% done, finished in 0:54
  DUMP: 50.07% done, finished in 0:49
  DUMP: 55.11% done, finished in 0:44
  DUMP: 60.10% done, finished in 0:39
  DUMP: 65.02% done, finished in 0:34
  DUMP: Closing /dev/rcd0d
  DUMP: Volume 1 completed at: Mon Aug 14 01:18:25 2000
  DUMP: Volume 1 took 1:05:57
  DUMP: Volume 1 transfer rate: 614 KB/s
  DUMP: Change Volumes: Mount volume #2
  DUMP: Is the new volume mounted and ready to go?: ("yes" or "no") yes
  DUMP: Volume 2 started at: Mon Aug 14 01:21:03 2000
  DUMP: Volume 2 begins with blocks from inode 224643
  DUMP: 67.37% done, finished in 0:32
  DUMP: 72.40% done, finished in 0:27
  DUMP: 77.41% done, finished in 0:22
  DUMP: 82.38% done, finished in 0:17
  DUMP: 87.29% done, finished in 0:12
  DUMP: 92.19% done, finished in 0:07
  DUMP: 97.00% done, finished in 0:03
  DUMP: 3684787 tape blocks on 2 volumes
  DUMP: Volume 2 completed at: Mon Aug 14 01:55:33 2000
  DUMP: Volume 2 took 0:34:30
  DUMP: Volume 2 transfer rate: 606 KB/s
  DUMP: Date of this level 0 dump: Mon Aug 14 00:12:28 2000
  DUMP: Date this dump completed:  Mon Aug 14 01:55:33 2000
  DUMP: Average transfer rate: 610 KB/s
  DUMP: Closing /dev/rcd0d
  DUMP: DUMP IS DONE
crasse# restore -x -b 2 -v -f /dev/rcd0d
Verify tape and initialize maps
Dump   date: Mon Aug 14 00:12:28 2000
Dumped from: the epoch
Level 0 dump of a subset of /scratch2 on crasse.smd.ebone.net:/dev/sd3a
Label: none
Extract directories from tape
Initialize symbol table.
Make node ./BACKUPS
Make node ./BACKUPS/20000813
Extract requested files
You have not read any tapes yet.
Unless you know which volume your file(s) are on you should start
with the last volume and work towards the first.
(Use 1 for the first volume/tape, etc.)
Specify next volume #: 1
extract file ./BACKUPS/20000813/crasse.u.0.gz
extract file ./BACKUPS/20000813/crasse.root.0.gz
extract file ./BACKUPS/20000813/crasse.var.0.gz
Tape read error while restoring ./BACKUPS/20000813/crasse.var.0.gz
continue? [yn] y
Tape read error while restoring ./BACKUPS/20000813/crasse.var.0.gz
continue? [yn] 
...

>Fix:

>Release-Note:
>Audit-Trail:

From: woods@weird.com (Greg A. Woods)
To: gnats-bugs@gnats.netbsd.org (NetBSD GNATS submissions and followups),
	netbsd-bugs@NetBSD.ORG (NetBSD Bugs and PR posting List)
Cc:  
Subject: Re: bin/10826: restore does not DTRT at end of volume on DVD-RAM
Date: Sun, 13 Aug 2000 21:13:43 -0400 (EDT)

 [ On Monday, August 14, 2000 at 00:51:42 (+0000), smd@ebone.net wrote: ]
 > Subject: bin/10826: restore does not DTRT at end of volume on DVD-RAM
 >
 > 	Note: There was a bad value `bin' for the field `>Confidential:'.
 > 	It was set to the default value of `yes'.

 oops!  ;-)

 > 	dump to two dvd ram sides
 > 	then try to restore everything; at the end of the first volume,
 > 	  dump chokes

 I can't be sure when I tried this last with other media, but I'm fairly
 sure it's a more generic bug -- nothing to do with the type of media or
 the device.

 -- 
 							Greg A. Woods

 +1 416 218-0098      VE3TCP      <gwoods@acm.org>      <robohack!woods>
 Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>

From: Hubert Feyrer <feyrer@rfhs8012.fh-regensburg.de>
To: smd@ebone.net
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: bin/10826: restore does not DTRT at end of volume on DVD-RAM
Date: Tue, 15 Aug 2000 04:45:27 +0200 (MET DST)

 On Mon, 14 Aug 2000 smd@ebone.net wrote:
 > dump -b 2 -B 2430000 -f /dev/rcd0d MD5 *.gz

 Out of curiosity, which DVD-RAM is that, and do you happen to know since
 when there's code to write to DVD-RAMs?

 I happen to have a Panasonic 5.6G DVD-RAM at work that didn't work some
 while ago (1.4U?). 

 Regarding your PR, did you have better success with tar? 
 Does the size of the archive you write to the DVD make a difference?
 E.g. a tar-archive with one small file vs. a tar-archive with several/many
 files? 


  - Hubert

 -- 
 NetBSD - because Unix isn't just #include <linux.h>, i386, ILP32, ELF, ...!




From: smd@ebone.net (Sean Doran)
To: hubert.feyrer@informatik.fh-regensburg.de
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: bin/10826: restore does not DTRT at end of volume on DVD-RAM
Date: Tue, 15 Aug 2000 10:53:06 +0200 (CEST)

 | Out of curiosity, which DVD-RAM is that, and do you happen to know since
 | when there's code to write to DVD-RAMs?

 ahc0: target 1 synchronous at 10.0MHz, offset = 0xf
 cd0 at scsibus1 target 1 lun 0: <MATSHITA, PD-2 LF-D100, A113> SCSI2 5/cdrom rem
 ovable

 It's been working "forever", at least since early 1.4 days.

 On another box:

 ahc1: target 2 synchronous at 10.0MHz, offset = 0xf
 scsipi_inqmatch: 2/5/1 <, , >
 cd1 at scsibus1 target 2 lun 0: <TOSHIBA, SD-W1101 DVD-RAM, 1033> SCSI2 5/cdrom re
 movable

 This also works.  I've not had any kind of problem writing to /dev/rcdXd
 since the days I used to use CD-R.

 | Regarding your PR, did you have better success with tar? 

 Both tar and restore fail in exactly the same way: if I do multivolume,
 I lose.  If I keep stuff only within the 2.6GB/side, things Just Work,
 and check out with MD5.

 Worse, I cannot use pax-as-tar or gtar to write out a 2.4GB dump file.
 I will raise a separate PR for this RSN.

 | Does the size of the archive you write to the DVD make a difference?
 | E.g. a tar-archive with one small file vs. a tar-archive with several/many
 | files? 

 Good question.  I will check that also.

 	Sean.

From: Sean Doran <smd@ebone.net>
To: hubert.feyrer@informatik.fh-regensburg.de
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: bin/10826: restore does not DTRT at end of volume on DVD-RAM
Date: 16 Aug 2000 19:43:22 +0200

 Hubert Feyrer <feyrer@rfhs8012.fh-regensburg.de> writes:

 > Regarding your PR, did you have better success with tar? 
 > Does the size of the archive you write to the DVD make a difference?
 > E.g. a tar-archive with one small file vs. a tar-archive with several/many
 > files? 

 I examined pax rather than tar.

 What I've found is that "pax -r" loses at the end of the
 DVD-RAM medium, where the kernel reports this:

 cd0(ahc0:1:0):  Check Condition on CDB: 0x08 12 99 92 04 00
     SENSE KEY:  Illegal Request
    INFO FIELD:  1218960
  COMMAND INFO:  83886080 (0x5000000)
      ASC/ASCQ:  Logical Block Address Out of Range

 which I would have expected pax would consider in an
 EOM-like way.   It kind-of does, asking for a volume
 change, but when the volume change happens, pax doesn't
 resynchronize, and loses the spilled-over portion of the
 last file on the previous volume.

 Examination with dd seems to show that all the right bits
 are actually on the medium.

 Is this the same with any removeable (re)writeable medium?

         Sean.

 pax -r -v -f /dev/rcd0d './Various Artists - In Dust We Trust - Invisible Compilation 1997'
 ...
 ./Various Artists - In Dust We Trust - Invisible Compilation 1997/14 Lick - Hollowed.mp3
 ./Various Artists - In Dust We Trust - Invisible Compilation 1997/15 Evil Mothers - Orisha (The Stork Mix).mp3
 pax: Failed read on archive volume 1 <Invalid argument>
 pax: Attempting to recover from an archive read failure.
 pax: Failed read on archive volume 1 <Invalid argument>
 pax: Attempting to recover from an archive read failure.
 pax: Failed read on archive volume 1 <Invalid argument>
 pax: Attempting to recover from an archive read failure.
 pax: Failed read on archive volume 1 <Invalid argument>
 pax: Attempting to recover from an archive read failure.
 pax: Failed read on archive volume 1 <Invalid argument>
 pax: Attempting to recover from an archive read failure.
 pax: Failed read on archive volume 1 <Invalid argument>
 pax: Attempting to recover from an archive read failure.
 pax: Failed read on archive volume 1 <Invalid argument>
 pax: Attempting to recover from an archive read failure.
 pax: Failed read on archive volume 1 <Invalid argument>
 pax: Attempting to recover from an archive read failure.
 pax: Failed read on archive volume 1 <Invalid argument>
 pax: Attempting to recover from an archive read failure.
 pax: Failed read on archive volume 1 <Invalid argument>
 pax: Attempting to recover from an archive read failure.
 pax: Failed read on archive volume 1 <Invalid argument>
 pax: Attempting to recover from an archive read failure.
 pax: Failed read on archive volume 1 <Invalid argument>
 pax: Archive read error limit (10) reached
 pax: ustar vol 1, 805 files, 2496430080 bytes read, 0 bytes written in 102 secs (24474804 bytes/sec)

 ATTENTION! pax archive volume change required.
 /dev/rcd0d ready for archive volume: 2
 Load the NEXT STORAGE MEDIA (if required)
 Type "y" to continue, "." to quit pax, or "s" to switch to new device.
 If you cannot change storage media, type "s"
 Is the device ready and online? >  unknown command, try again
 Type "y" to continue, "." to quit pax, or "s" to switch to new device.
 If you cannot change storage media, type "s"
 Is the device ready and online? > y
 pax: Archive I/O error. Trying to recover.
 ./Various Artists - In Dust We Trust - Invisible Compilation 1997/01 Sheep on Drugs - X-Lover (Critter Mix).mp3
 ./Various Artists - In Dust We Trust - Invisible Compilation 1997/06 Evil Mothers - You Had Enough (ripped Up).mp3
 ...

 stored versus extracted:

 15c15
 < MD5 (15 Evil Mothers - Orisha (The Stork Mix).mp3) = 62a5c4131c89253f9b149ef12e98aeea
 ---
 > MD5 (15 Evil Mothers - Orisha (The Stork Mix).mp3) = f061466706348b4e2d89f6800d8a0a17

 -r--r--r--  1 smd  wheel  4559868 Oct 23  1999 15 Evil Mothers - Orisha (The Stork Mix).mp3
 -r--------  1 smd  wheel  1568768 Oct 23  1999 15 Evil Mothers - Orisha (The Stork Mix).mp3


From: Sean Doran <smd@ebone.net>
To: hubert.feyrer@informatik.fh-regensburg.de
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: bin/10826: restore does not DTRT at end of volume on DVD-RAM
Date: 16 Aug 2000 19:58:09 +0200

 Hubert Feyrer <feyrer@rfhs8012.fh-regensburg.de> writes:

 > Does the size of the archive you write to the DVD make a difference?
 > E.g. a tar-archive with one small file vs. a tar-archive with several/many
 > files? 

 pax / tar/ cpio / dump / pkgsrc/archivers/gcpio / pkgsrc/archivers/gtar
 files written to the DVD-RAM such that the entire archive fits on one 
 volume work perfectly, and pass the MD5 test.

 archive files from the above formats that are written to
 the DVD-RAM such that multiple volumes are needed, all
 have synchronization problems at the start of the 2nd
 volume, losing the file that spills over.

 I imagine this could be tested with a vnd drive, yes?
         1. create empty file
         2. vndconfig over it
         3. archive into it until the archiver says "switch volumes"
            and waits for user input
         4. vndunconfig
         5. create next empty file
         6. goto 2.
 following the complementary process when reading.

 (incidentally, tar and gtar would not even write a 2.35GByte
 file into a tar file, whether argument to "-f" was a
 device or an ordinary file or pipe).

 : crasse (es) ; ls -l
 total 2433184
 -rw-r--r--  1 root  wheel    16028682 Aug 13 20:24 crasse.root.0.gz
 -rw-r--r--  1 root  wheel  2353193522 Aug 13 21:20 crasse.u.0.gz
 -rw-r--r--  1 root  wheel   121929728 Aug 14 02:40 crasse.var.0.gz
 : crasse (es) ; /usr/bin/time -l tar -cf - . | tar -tvf -
 drwxr-xr-x root/wheel        0 Aug 14 02:38 2000 ./
 -rw-r--r-- root/wheel -1941773774 Aug 13 21:20 2000 crasse.u.0.gz
 -rw-r--r-- root/wheel    16028682 Aug 13 20:24 2000 crasse.root.0.gz
 -rw-r--r-- root/wheel   121929728 Aug 14 02:40 2000 crasse.var.0.gz
         9.70 real         0.00 user         0.91 sys
          0  maximum resident set size
          0  average shared memory size
          0  average unshared data size
          0  average unshared stack size
         84  page reclaims
          0  page faults
          0  swaps
       2640  block input operations
          3  block output operations
      13473  messages sent
          0  messages received
          0  signals received
      34489  voluntary context switches
         21  involuntary context switches

 : crasse (es) ; /usr/bin/time -l pax -w . | pax -v 
 drwxr-xr-x  2 root     wheel          0 Aug 14 02:38 .
 -rw-r--r--  1 root     wheel  2353193522 Aug 13 21:20 ./crasse.u.0.gz
 -rw-r--r--  1 root     wheel   16028682 Aug 13 20:24 ./crasse.root.0.gz
 -rw-r--r--  1 root     wheel  121929728 Aug 14 02:40 ./crasse.var.0.gz
       251.63 real         0.18 user        18.34 sys
          0  maximum resident set size
          0  average shared memory size
          0  average unshared data size
          0  average unshared stack size
         95  page reclaims
          0  page faults
          0  swaps
      75267  block input operations
          3  block output operations
     243277  messages sent
          0  messages received
          0  signals received
     633037  voluntary context switches
        311  involuntary context switches
 pax: ustar vol 1, 4 files, 2491156480 bytes read, 0 bytes written in 251 secs (9924926 bytes/sec)

 /usr/bin/tar loses its brains totally when trying to read
 from the pax process above, presumably because the USTAR
 header is different?

 I suppose this last bit could be a separate PR. -:)

         Sean.

From: Hubert Feyrer <feyrer@rfhs8012.fh-regensburg.de>
To: Sean Doran <smd@ebone.net>
Cc: gnats-bugs@gnats.netbsd.org
Subject: Re: bin/10826: restore does not DTRT at end of volume on DVD-RAM
Date: Thu, 17 Aug 2000 01:23:05 +0200 (MET DST)

 On 16 Aug 2000, Sean Doran wrote:
 > Is this the same with any removeable (re)writeable medium?

 Sorry, I cannot tell you (I mainly wondered about support for DVD-?RAM, I
 don't know about the internals). I think you should ask again on
 tech-kern, pointing at your PR.


  - Hubert

 -- 
 NetBSD - because Unix isn't just #include <linux.h>, i386, ILP32, ELF, ...!


From: smd@ebone.net (Sean Doran by way of Erik E. Fair)
To: gnats-bugs@netbsd.org
Cc:  
Subject: Re: bin/10826: restore does not DTRT at end of volume on DVD-RAM
Date: Thu, 7 Sep 2000 14:08:53 -0700

 Also please see bin/10848.

 	Sean.
>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2007 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.