NetBSD Problem Report #48787
From www@NetBSD.org Tue May 6 14:14:51 2014
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
(using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits))
(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
by mollari.NetBSD.org (Postfix) with ESMTPS id 42677A5813
for <gnats-bugs@gnats.NetBSD.org>; Tue, 6 May 2014 14:14:51 +0000 (UTC)
Message-Id: <20140506141449.CFF50A5840@mollari.NetBSD.org>
Date: Tue, 6 May 2014 14:14:49 +0000 (UTC)
From: scdbackup@gmx.net
Reply-To: scdbackup@gmx.net
To: gnats-bugs@NetBSD.org
Subject: 32 bit rollover makes ISO multi-session medium unusable via mount
X-Send-Pr-Version: www-1.0
>Number: 48787
>Category: kern
>Synopsis: 32 bit rollover makes ISO multi-session medium unusable via mount
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: kern-bug-people
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Tue May 06 14:15:00 +0000 2014
>Closed-Date: Sun Jul 05 16:31:36 +0000 2015
>Last-Modified: Sun Jul 05 19:30:00 +0000 2015
>Originator: Thomas Schmitt
>Release: 6.1.3
>Organization:
>Environment:
NetBSD netbsd 6.1.3 NetBSD 6.1.3 (GENERIC) i386
>Description:
This bug shows up when mounting a multi-session ISO 9660 medium
which has its directory tree above 4 GiB. (Not possible with CD,
but with DVD or BD.)
netbsd# mount -t cd9660 /dev/cd0a /mnt
netbsd# ls -l /mnt
-r-xr-xr-x 1 root wheel 0 Jan 1 1970 /mnt
netbsd# umount /mnt
umount: /mnt: Invalid argument
netbsd# umount /dev/cd0a
umount: /mnt: Invalid argument
netbsd# mount
/dev/wd0a on / type ffs (local)
kernfs on /kern type kernfs (local)
ptyfs on /dev/pts type ptyfs (local)
procfs on /proc type procfs (local)
/dev/cd0a on /mnt type cd9660 (read-only, local)
I can get rid of the mount point only by reboot.
It does not happen with
mount -t cd9660 -o norrip /dev/cd0a /mnt
To my assessment it is caused by a 32 bit rollover in
sys/fs/cd9660/cd9660_node.c:isodirino()
which leads to a wrong reverse computed block address of the
directory extent in sys/fs/cd9660/cd9660_vfsops.c line 815:
ip->iso_start = ino >> imp->im_bshift;
Although sizeof(ino_t) == 8, the computation in isodirino() rolls over
at least on NetBSD i386 because none of the operands of the computation
is 64 bit wide.
A test in userspace confirms:
sizeof(ino_t) = 8
( 2113952 + 0 ) << 11 = 34406400
( ((uint64_t) 2113952) + 0 ) << 11 = 4329373696
Fixing the rollover makes the ISO mountable and umount-able.
Note:
Inode numbers above (2 exp 32 - 1) are shown rolled-over by ls -i
34445556 -rw-r--r-- 1 thomas dbus 396 Aug 26 2011 id_rsa.pub
>How-To-Repeat:
Create an appendable ISO 9660 DVD or BD with more than 4 GiB of data.
Then add another ISO session, eject, load, and mount the medium.
Or create such a multi-session ISO 9660 image and submit it to
a qemu-based NetBSD as -cdrom. This will be handled much like a
DVD+RW or BD-RE medium.
Programs which create such ISO media or images are growisofs out
of package dvd+rw-tools, or xorriso out of package libisoburn,
which was added to pkgsrc a few days ago.
>Fix:
Inject a (uint64_t) cast into the computation of isodirino().
>Release-Note:
>Audit-Trail:
From: "Thomas Schmitt" <scdbackup@gmx.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/48787
Date: Tue, 06 May 2014 16:19:45 +0200
--- sys/fs/cd9660/cd9660_node.c.orig 2014-05-05 11:56:09.000000000 +0000
+++ sys/fs/cd9660/cd9660_node.c 2014-05-06 08:20:51.000000000 +0000
@@ -440,7 +440,12 @@ isodirino(struct iso_directory_record *i
{
ino_t ino;
- ino = (isonum_733(isodir->extent) + isonum_711(isodir->ext_attr_length))
- << imp->im_bshift;
+ /* This is not only a file serial number, but also a message to
+ * cd9660_vfsops.c:cd9660_vget_internal() which computes this
+ * number back from ino_t by:
+ * ip->iso_start = ino >> imp->im_bshift;
+ */
+ ino = (((uint64_t) isonum_733(isodir->extent)) +
+ isonum_711(isodir->ext_attr_length)) << imp->im_bshift;
return (ino);
}
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/48787
Date: Tue, 6 May 2014 16:40:26 +0200
Looks good to me. Is it possible to create a failing image via some
simple script and makefs -f cd9660 (or other base system tools)?
Can a disk image hold multiple sessions?
I'm looking for a way to create a simple rump based test case for this.
Martin
From: "Thomas Schmitt" <scdbackup@gmx.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/48787
Date: Tue, 06 May 2014 18:43:26 +0200
Martin Husemann wrote:
> Is it possible to create a failing image via some
> simple script and makefs -f cd9660 (or other base system tools)?
I don't think so.
makefs(8) would need an option to load the directory tree of
an existing image, to modify this tree model, and to write
the new tree plus the data blocks of newly added files to
the end of the exiting image. Finally it would have to overwrite
the "superblock" at offset 0 (PVD at block 16) in order to
point mount(8) resp. cd9660_vfsops.c:iso_mountfs() to the new
root directory.
I don't see such options or capabilities in makefs(8).
growisofs operates mkisofs with options -M -C for this purpose.
My own program xorriso operates via libisoburn and libisofs.
I uploaded heavily zero-containing
http://scdbackup.webframe.org/large.iso.bz2
Only 4470 bytes, MD5 7d78dc3efaec8ea3f1801335329f410d.
It inflates to 4,329,897,984 bytes.
Submitted to my virtual NetBSD via qemu -cdrom, it causes with
kernel 6.1.3
netbsd# mount -t cd9660 /dev/cd0a /mnt
netbsd# ls -l /mnt
-r-xr-xr-x 1 root wheel 0 Jan 1 1970 /mnt
Whereas with my patched 6.99.40
netbsd# mount -t cd9660 /dev/cd0a /mnt
netbsd# ls -l /mnt
total 4
drwxr-xr-x 1 thomas dbus 2048 May 6 15:30 my
-rw-r--r-- 1 thomas dbus 6 May 6 15:34 small_file
-------------------------------------------------------------
The image was created by
# First session with enough input to surpass 4 GiB
dd if=/dev/zero bs=2048 count=1 seek=2113952 of=large_file
xorriso -outdev large.iso -blank as_needed \
-map large_file /my/large_file
rm large_file
# Second session
echo "hello" >small_file
xorriso -dev large.iso \
-map small_file /small_file
rm small_file
# Inspecting situation
xorriso -indev large.iso -toc -find / -exec lsdl --
reports:
...
TOC layout : Idx , sbsector , Size , Volume Id
ISO session : 1 , 32 , 2113977s , ISOIMAGE
ISO session : 2 , 2114016 , 25s , ISOIMAGE
Media summary: 2 sessions, 2114002 data blocks, 4129m data, 5158m free
Media nwa : 2114048s
drwxr-xr-x 1 0 0 0 May 6 17:31 '/'
drwxr-xr-x 1 1000 1000 0 May 6 17:30 '/my'
-rw-r--r-- 1 1000 1000 4329375744 May 6 17:30 '/my/large_file'
-rw-r--r-- 1 1000 1000 6 May 6 17:34 '/small_file'
(After remedy, look into directory ./my to see a problem
with files larger than 4 GiB - 1. ISO 9660 level 3 allows
multiple extents. cd9660 neither rejects nor does it work
correctly:
netbsd# ls -l /mnt/my
total 16777208
-rw-r--r-- 1 thomas dbus 4294965248 May 6 15:30 large_file
-rw-r--r-- 1 thomas dbus 4294965248 May 6 15:30 large_file
NetBSD is not alone with these bugs. See:
http://lists.freebsd.org/pipermail/freebsd-hackers/2012-April/038552.html
http://lists.freebsd.org/pipermail/freebsd-hackers/2012-April/038570.html
I was quite clueless, then. This time i dug with more success.
)
From: "Thomas Schmitt" <scdbackup@gmx.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/48787
Date: Tue, 06 May 2014 19:29:36 +0200
In case you want to convert the unpacked large.iso into a
sparse file with holes:
The data file extents are located between 2048-blocks
56 and 2114008.
Report layout: xt , Startlba , Blocks , Filesize , ISO image path
File data lba: 0 , 56 , 2097151 , 4329375744 , '/my/large_file'
File data lba: 1 , 2097207 , 16802 , 4329375744 , '/my/large_file'
I did on my virtual NetBSD:
dd if=/dev/cd0a bs=2048 count=56 of=/dvdbuffer/large.iso
dd conv=notrunc if=/dev/cd0a bs=2048 skip=2114008 seek=2114008 \
of=/dvdbuffer/large.iso
du increased by 864 blocks, file size and MD5 match the
original inflated ISO on the hosting system.
That's a cheap test object for a 4 GiB problem, i'd say.
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/48787
Date: Fri, 9 May 2014 17:52:04 +0200
Using the image you supplied I created a test case and it fails as expected
without the patch, but with the patch applied I get a duplicate file
(in the case w/o norrip):
ls -lR shows:
total 2
drwxr-xr-x 1 dbus dbus 2048 May 6 15:30 my
-rw-r--r-- 1 dbus dbus 6 May 6 15:34 small_file
/tmp/atf-run.03700b/mnt/my:
total 8388604
-rw-r--r-- 1 dbus dbus 4294965248 May 6 15:30 large_file
-rw-r--r-- 1 dbus dbus 4294965248 May 6 15:30 large_file
The norrip case works as expected:
total 2
dr-xr-xr-x 1 root wheel 2048 May 6 15:30 my
-r-xr-xr-x 1 root wheel 6 May 6 15:34 small_file
/tmp/atf-run.03700b/mnt/my:
total 33604
-r-xr-xr-x 1 root wheel 34410496 May 6 15:30 large_file
This is using rump_cd9660 on the bunzip2'd immage and -current.
Will dig deeper next week unless you beat me to it ;-)
Martin
From: "Thomas Schmitt" <scdbackup@gmx.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/48787
Date: Fri, 09 May 2014 18:42:04 +0200
> -rw-r--r-- 1 dbus dbus 4294965248 May 6 15:30 large_file
> -rw-r--r-- 1 dbus dbus 4294965248 May 6 15:30 large_file
This is a different bug. ISO 9660 level 3 allows to store files
larger than 4 GiB - 1. This is done by several directory entries
with the same name marking a byte interval ("extent") of the
overall file.
There is no global indication in a filesystem for ISO 9660 level 3.
You recognize it by encountering multi-extent files.
The real situation as reported by
xorriso -find /my/large_file -exec report_lba
is that the first extent starts at block 56 and has 2097151 blocks.
The second extent starts at block 2097207 and has 16802 blocks.
The combined file size is 4329375744 bytes.
> The norrip case works as expected:
> -r-xr-xr-x 1 root wheel 34410496 May 6 15:30 large_file
It does not, i fear. It just shows a different skewed projection
of the large file: 4329375744 - 4294965248 = 34410496.
I.e. we see the second extent with the remaining bytes of the
file above byte address 4 GiB - 2048.
(You can combine one file from with RRIP and the file of -norrip
to get the original large file. Easier would be to let xorriso
extract it to hard disk.)
When i assessed the problem in FreeBSD 2 years ago, i noticed
a 1:1 relation between iso_node and extent. It would have to
become 1:n, or multi-extent files would have to be handled
as reason for a well contained error. (Not easy to imagine how
the latter shall work consistently.)
I plan to look for a solution.
But that change will be a fat patch, i guess, because it will
include an adjustment of the object relationship graph in cd9660.
Between this quite clear PR 48787 and the PR for large files
i plan to post a PR for scaling down cd9660 inode numbers by a
factor of 32. See ls -i and its 32-bit rolled over inode number
display with the test image.
But i first want to gain more experience about NetBSD cooperation
by this PR 48787. Be invited to teach me how to make it convenient
for those who decide about commits.
(Also i have found one spot in libisofs which needs closer
inspection in respect to 64 bit ino_t. Pot, kettle, black.)
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/48787
Date: Fri, 9 May 2014 19:23:01 +0200
Ah, ok - I only ever use UDF in this cases.
Anyway, the test clearly shows the hang before and not hanging after
the patch, so I'll just clean up the check a bit and commit it, together
with the patch.
(Might still take 'till monday)
Martin
From: "Thomas Schmitt" <scdbackup@gmx.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/48787
Date: Sat, 10 May 2014 07:38:03 +0200
I obviously misperceived the ino_t-to-address computation as
a hack. Now, in the light of kern/48797, i rather see it as
making use of a feature of NetBSD.
So the comment in this patch seems inappropriate now.
Such a statement would still be helpful for future readers of
the code. But the function isodirino() is not the right place
for it, because symbolic links make use of the feature, too.
State-Changed-From-To: open->closed
State-Changed-By: martin@NetBSD.org
State-Changed-When: Sat, 10 May 2014 14:12:38 +0000
State-Changed-Why:
Applied, thanks!
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/48787 CVS commit: src/sys/fs/cd9660
Date: Sat, 10 May 2014 14:11:58 +0000
Module Name: src
Committed By: martin
Date: Sat May 10 14:11:58 UTC 2014
Modified Files:
src/sys/fs/cd9660: cd9660_node.c
Log Message:
PR kern/48787: inode calculation from ISO9660 block offset might get
truncated to 32bit - force the whole expression to be evaluated as ino_t.
Patch from Thomas Schmitt, with minor modifications (and reworded comment).
To generate a diff of this commit:
cvs rdiff -u -r1.30 -r1.31 src/sys/fs/cd9660/cd9660_node.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/48787 CVS commit: src
Date: Sat, 10 May 2014 14:15:25 +0000
Module Name: src
Committed By: martin
Date: Sat May 10 14:15:25 UTC 2014
Modified Files:
src/distrib/sets/lists/tests: mi
src/etc/mtree: NetBSD.dist.tests
src/tests/fs: Makefile
Added Files:
src/tests/fs/cd9660: Makefile pr_48787.image.bz2.uue
t_high_ino_big_file.sh
Log Message:
Add a test case for PR kern/48787.
To generate a diff of this commit:
cvs rdiff -u -r1.567 -r1.568 src/distrib/sets/lists/tests/mi
cvs rdiff -u -r1.104 -r1.105 src/etc/mtree/NetBSD.dist.tests
cvs rdiff -u -r1.22 -r1.23 src/tests/fs/Makefile
cvs rdiff -u -r0 -r1.1 src/tests/fs/cd9660/Makefile \
src/tests/fs/cd9660/pr_48787.image.bz2.uue \
src/tests/fs/cd9660/t_high_ino_big_file.sh
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: closed->pending-pullups
State-Changed-By: martin@NetBSD.org
State-Changed-When: Sat, 10 May 2014 14:43:24 +0000
State-Changed-Why:
Waiting on [pullup-5 #1904] and [pullup-6 #1062]
From: "Manuel Bouyer" <bouyer@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/48787 CVS commit: [netbsd-6] src/sys/fs/cd9660
Date: Wed, 21 May 2014 21:06:29 +0000
Module Name: src
Committed By: bouyer
Date: Wed May 21 21:06:29 UTC 2014
Modified Files:
src/sys/fs/cd9660 [netbsd-6]: cd9660_node.c
Log Message:
Pull up following revision(s) (requested by martin in ticket #1062):
sys/fs/cd9660/cd9660_node.c: revision 1.31
PR kern/48787: inode calculation from ISO9660 block offset might get
truncated to 32bit - force the whole expression to be evaluated as ino_t.
Patch from Thomas Schmitt, with minor modifications (and reworded comment).
To generate a diff of this commit:
cvs rdiff -u -r1.29 -r1.29.8.1 src/sys/fs/cd9660/cd9660_node.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Manuel Bouyer" <bouyer@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/48787 CVS commit: [netbsd-6-0] src/sys/fs/cd9660
Date: Wed, 21 May 2014 21:06:36 +0000
Module Name: src
Committed By: bouyer
Date: Wed May 21 21:06:36 UTC 2014
Modified Files:
src/sys/fs/cd9660 [netbsd-6-0]: cd9660_node.c
Log Message:
Pull up following revision(s) (requested by martin in ticket #1062):
sys/fs/cd9660/cd9660_node.c: revision 1.31
PR kern/48787: inode calculation from ISO9660 block offset might get
truncated to 32bit - force the whole expression to be evaluated as ino_t.
Patch from Thomas Schmitt, with minor modifications (and reworded comment).
To generate a diff of this commit:
cvs rdiff -u -r1.29 -r1.29.14.1 src/sys/fs/cd9660/cd9660_node.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Manuel Bouyer" <bouyer@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/48787 CVS commit: [netbsd-6-1] src/sys/fs/cd9660
Date: Wed, 21 May 2014 21:06:40 +0000
Module Name: src
Committed By: bouyer
Date: Wed May 21 21:06:40 UTC 2014
Modified Files:
src/sys/fs/cd9660 [netbsd-6-1]: cd9660_node.c
Log Message:
Pull up following revision(s) (requested by martin in ticket #1062):
sys/fs/cd9660/cd9660_node.c: revision 1.31
PR kern/48787: inode calculation from ISO9660 block offset might get
truncated to 32bit - force the whole expression to be evaluated as ino_t.
Patch from Thomas Schmitt, with minor modifications (and reworded comment).
To generate a diff of this commit:
cvs rdiff -u -r1.29 -r1.29.22.1 src/sys/fs/cd9660/cd9660_node.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Manuel Bouyer" <bouyer@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/48787 CVS commit: [netbsd-5] src/sys/fs/cd9660
Date: Wed, 21 May 2014 21:49:41 +0000
Module Name: src
Committed By: bouyer
Date: Wed May 21 21:49:41 UTC 2014
Modified Files:
src/sys/fs/cd9660 [netbsd-5]: cd9660_node.c
Log Message:
Pull up following revision(s) (requested by martin in ticket #1904):
sys/fs/cd9660/cd9660_node.c: revision 1.31
PR kern/48787: inode calculation from ISO9660 block offset might get
truncated to 32bit - force the whole expression to be evaluated as ino_t.
Patch from Thomas Schmitt, with minor modifications (and reworded comment).
To generate a diff of this commit:
cvs rdiff -u -r1.24 -r1.24.10.1 src/sys/fs/cd9660/cd9660_node.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Manuel Bouyer" <bouyer@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/48787 CVS commit: [netbsd-5-2] src/sys/fs/cd9660
Date: Wed, 21 May 2014 21:49:44 +0000
Module Name: src
Committed By: bouyer
Date: Wed May 21 21:49:44 UTC 2014
Modified Files:
src/sys/fs/cd9660 [netbsd-5-2]: cd9660_node.c
Log Message:
Pull up following revision(s) (requested by martin in ticket #1904):
sys/fs/cd9660/cd9660_node.c: revision 1.31
PR kern/48787: inode calculation from ISO9660 block offset might get
truncated to 32bit - force the whole expression to be evaluated as ino_t.
Patch from Thomas Schmitt, with minor modifications (and reworded comment).
To generate a diff of this commit:
cvs rdiff -u -r1.24 -r1.24.28.1 src/sys/fs/cd9660/cd9660_node.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Manuel Bouyer" <bouyer@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/48787 CVS commit: [netbsd-5-1] src/sys/fs/cd9660
Date: Wed, 21 May 2014 21:49:49 +0000
Module Name: src
Committed By: bouyer
Date: Wed May 21 21:49:49 UTC 2014
Modified Files:
src/sys/fs/cd9660 [netbsd-5-1]: cd9660_node.c
Log Message:
Pull up following revision(s) (requested by martin in ticket #1904):
sys/fs/cd9660/cd9660_node.c: revision 1.31
PR kern/48787: inode calculation from ISO9660 block offset might get
truncated to 32bit - force the whole expression to be evaluated as ino_t.
Patch from Thomas Schmitt, with minor modifications (and reworded comment).
To generate a diff of this commit:
cvs rdiff -u -r1.24 -r1.24.24.1 src/sys/fs/cd9660/cd9660_node.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Wolfgang Solfrank <Wolfgang@Solfrank.net>
To: gnats-bugs@NetBSD.org, kern-bug-people@netbsd.org,
gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, scdbackup@gmx.net
Cc:
Subject: Re: kern/48787
Date: Thu, 22 May 2014 01:23:28 +0200
Hi,
> > -rw-r--r-- 1 dbus dbus 4294965248 May 6 15:30 large_file
> > -rw-r--r-- 1 dbus dbus 4294965248 May 6 15:30 large_file
>
> This is a different bug. ISO 9660 level 3 allows to store files
> larger than 4 GiB - 1. This is done by several directory entries
> with the same name marking a byte interval ("extent") of the
> overall file.
...
> When i assessed the problem in FreeBSD 2 years ago, i noticed
> a 1:1 relation between iso_node and extent. It would have to
> become 1:n, or multi-extent files would have to be handled
> as reason for a well contained error. (Not easy to imagine how
> the latter shall work consistently.)
>
> I plan to look for a solution.
> But that change will be a fat patch, i guess, because it will
> include an adjustment of the object relationship graph in cd9660.
Be my guest. However, this will be a larger project than just
updating code in cd9660. The problem is that an extent (as you
indicate above) ist a "byte interval ... of the overall file". What this
means is that you can have a file with a first extent of, say, 3 bytes
and a second one of 5 bytes and so on (which looks absurd when
you first look at it, but does make sense if you think about multi-
session cds where you e.g. extend some logfile in a later session.)
The buffer cache in its current implementation doesn't support
this.kind of fragmentation.
Ciao,
Wolfgang
--
Wolfgang@Solfrank.net Wolfgang Solfrank
From: "Thomas Schmitt" <scdbackup@gmx.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/48787
Date: Thu, 22 May 2014 10:57:36 +0200
> However, this will be a larger project than just
> updating code in cd9660.
I really try to keep it enclosed there.
> The problem is that an extent (as you
> indicate above) ist a "byte interval ... of the overall file". What this
> means is that you can have a file with a first extent of, say, 3 bytes
> and a second one of 5 bytes
I plan to throw a suitable error in this case.
The existing examples in the code use EINVAL, which i deem
somewhat misleading. Also there are som ESTALE which seem to
stem from copied code that originally served for NFS.
> The buffer cache in its current implementation doesn't support
> this.kind of fragmentation.
Yes. I could only spot a file-to-device address translation which
is block-wise (512 bytes per block).
cd9660_bmap.c : cd9660_bmap()
I did not look yet, whether my host operating system would tolerate
file sections which are not aligned to block size and in some file
play a role as non-last filesection.
(The relation between extent and file section is 1:1, between
file section and iso_node it is m:n.)
First i want to develop a solution and only then i want to know
where others were smarter than me.
Somewhat an obstacle are my pending PRs 48808 and 48815, which
are currently the base of my development.
(I will have to look for instructions for re-syncing my source
with CVS without negative side effects.)
The 64-bit ino_t of NetBSD is a heaven's gift.
cd9660_vget_internal() needs to know the number of file sections
while having only an ino_t as clue. And ISO 9660 implements the
list of directory records as a block array in which each block
holds an individual linked list. Fuzzy end, if you do not know
the array end, which is stored far away from the array.
My todo list has as solution:
+++ Encode in ino_t:
48 bits of byte address of first record (as is done already now)
13 bits of file section count - 1.
Enough for filling the whole fs by 1 GiB pieces.
Possibly cd9660_vget_internal() should curb to a
much lower number.
count - 1 will keep numbers unchanged for single-extent
files. Count 0 is really impossible, not only illegal.
3 bits are still unused.
My assessment-and-plan text has 900 lines meanwhile.
The emerging patch is at 786 lines.
It is still not recognizing multiple extents but on the other
hand it still works properly.
Next point on my todo list is to develop a script that creates
a challenging ISO image. Not only for multi-extent but also for
regression tests with all (Rock Ridge) file types.
Then i will dare to let cd9660 recognize multi-extent files
and hope that it passes the tests.
(It is not without risk. I had a bug which got the whole VM stuck
just because it read undefined memory and maybe used the bad
bit pattern as block address for reading.)
State-Changed-From-To: pending-pullups->closed
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Sun, 05 Jul 2015 16:31:36 +0000
State-Changed-Why:
These pullups were done last year.
From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: Wolfgang Solfrank <Wolfgang@Solfrank.net>, scdbackup@gmx.net
Subject: Re: kern/48787
Date: Sun, 5 Jul 2015 16:34:14 +0000
On Wed, May 21, 2014 at 11:25:01PM +0000, Wolfgang Solfrank wrote:
> Be my guest. However, this will be a larger project than just
> updating code in cd9660. The problem is that an extent (as you
> indicate above) ist a "byte interval ... of the overall file". What this
> means is that you can have a file with a first extent of, say, 3 bytes
> and a second one of 5 bytes and so on (which looks absurd when
> you first look at it, but does make sense if you think about multi-
> session cds where you e.g. extend some logfile in a later session.)
> The buffer cache in its current implementation doesn't support
> this.kind of fragmentation.
For the record, I don't think this should require any more than a
custom getpages method. Mind you, that's not trivial.
--
David A. Holland
dholland@netbsd.org
From: "Thomas Schmitt" <scdbackup@gmx.net>
To: gnats-bugs@NetBSD.org
Cc: dholland-bugs@netbsd.org,Wolfgang@Solfrank.net
Subject: Re: kern/48787
Date: Sun, 05 Jul 2015 21:28:22 +0200
Hi,
i wrote:
> > > > ISO 9660 level 3 allows to store files larger than 4 GiB - 1.
Wolfgang Solfrank wrote:
> > > you can have a file with a first extent of, say, 3 bytes
> > > and a second one of 5 bytes and so on
i wrote:
> > I plan to throw a suitable error in this case.
David Holland wrote:
> For the record, I don't think this should require any more than a
> custom getpages method. Mind you, that's not trivial.
In any case somebody would have to review my proposal / research
result in PR 48959
"Misrepresentation of files of 4 GiB or larger in cd9660".
It would probably have to be adapted to changes in the cd9660
code made since july last year.
A method for exotic extent sizes could be added then.
A test challenge is available in the test tarball for 48959
(exoten.iso file "07").
Have a nice day :)
Thomas
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home