NetBSD Problem Report #50608
From www@NetBSD.org Sat Jan 2 12:07:33 2016
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.NetBSD.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
by mollari.NetBSD.org (Postfix) with ESMTPS id 57E0B7ACA4
for <gnats-bugs@gnats.NetBSD.org>; Sat, 2 Jan 2016 12:07:33 +0000 (UTC)
Message-Id: <20160102120732.6DE5E7ACC8@mollari.NetBSD.org>
Date: Sat, 2 Jan 2016 12:07:32 +0000 (UTC)
From: pooka@iki.fi
Reply-To: pooka@iki.fi
To: gnats-bugs@NetBSD.org
Subject: udf fails the fs/vfs/t_vnops::create_many test
X-Send-Pr-Version: www-1.0
>Number: 50608
>Category: kern
>Synopsis: udf fails the fs/vfs/t_vnops::create_many test
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: reinoud
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sat Jan 02 12:10:00 +0000 2016
>Closed-Date: Thu Jan 28 10:16:28 +0000 2016
>Last-Modified: Thu Jan 28 10:20:00 +0000 2016
>Originator: Antti Kantee
>Release:
>Organization:
>Environment:
>Description:
Not clear what exactly is going on, but it seems like under some conditions udf unlink() returns ENOENT for a recently created file.
>How-To-Repeat:
run tests/fs/vfs/t_vnops::create_many for udf
https://releng.netbsd.org/b5reports/i386/build/2016.01.02.01.24.44/test.html#fs_vfs_t_vnops_udf_create_many
>Fix:
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: kern-bug-people->reinoud
Responsible-Changed-By: dholland@NetBSD.org
Responsible-Changed-When: Sun, 03 Jan 2016 01:55:31 +0000
Responsible-Changed-Why:
over to udf maintainer
(if you can't deal with it, set it back)
From: Reinoud Zandijk <reinoud@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: reinoud@NetBSD.org, kern-bug-people@netbsd.org, netbsd-bugs@netbsd.org,
gnats-admin@netbsd.org, dholland@NetBSD.org, pooka@iki.fi
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Mon, 4 Jan 2016 09:59:17 +0100
Hi,
I've tried to replicate it in a NetBSD-current kernel on a vnd disc with sets
of various sizes but I couldn't replicate it. Still investigating it, might be
some emulation choices like dircache sizes that is preventing it to cache a
2500+(?) entry dircache where in a normal kernel this is considered trivial in
size.
With regards,
Reinoud
From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Mon, 4 Jan 2016 09:37:17 +0000
On Mon, Jan 04, 2016 at 09:00:01AM +0000, Reinoud Zandijk wrote:
> I've tried to replicate it in a NetBSD-current kernel on a vnd
> disc with sets of various sizes but I couldn't replicate it. Still
> investigating it, might be some emulation choices like dircache
> sizes that is preventing it to cache a 2500+(?) entry dircache
> where in a normal kernel this is considered trivial in size.
In order for the namecache to produce a wrong negative result it is
theoretically necessary to prime it with wrong information; just being
full isn't enough.
I wonder though if this is the same problem people have been seeing
intermittently with tmpfs. If so, chasing after it in this form is
definitely worthwhile.
--
David A. Holland
dholland@netbsd.org
From: Andreas Gustafsson <gson@gson.org>
To: reinoud@NetBSD.org
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Sat, 9 Jan 2016 16:36:44 +0200
reinoud wrote:
> I've tried to replicate it in a NetBSD-current kernel on a vnd disc with sets
> of various sizes but I couldn't replicate it.
This consistently replicates it for me:
$ sudo pkg_add qemu
$ sudo pkg_add py-anita
$ anita interact http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/201601090500Z/i386/
login: root
# cd /usr/tests/fs/vfs
# ./t_vnops udf_create_many
It's also failing on a physical amd64 machine (no qemu involved):
http://www.gson.org/netbsd/bugs/build/amd64-baremetal/2016/2016.01.05.10.20.22/test.html#fs_vfs_t_vnops_udf_create_many
--
Andreas Gustafsson, gson@gson.org
From: Reinoud Zandijk <reinoud@NetBSD.org>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Wed, 13 Jan 2016 17:39:00 +0100
I've tried your sequence and indeed it fails on an physical amd64. What I
meant is that i tried out a program that does exactly the same as the test
only then *without* RUMP and that passes fine.
I'll try to dig into the issue, might be the generic dirhash code misbehaving
though i still think its a rump internal problem.
On my machine this was set to the default:
vfs.generic.dirhash.maxmem = 1048576
Reinoud
From: Andreas Gustafsson <gson@gson.org>
To: Reinoud Zandijk <reinoud@NetBSD.org>
Cc: pooka@NetBSD.org,
gnats-bugs@NetBSD.org
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Wed, 13 Jan 2016 20:34:49 +0200
Reinoud Zandijk wrote:
> I've tried your sequence and indeed it fails on an physical amd64. What I
> meant is that i tried out a program that does exactly the same as the test
> only then *without* RUMP and that passes fine.
>
> I'll try to dig into the issue, might be the generic dirhash code misbehaving
> though i still think its a rump internal problem.
In your test, does the filename contain an embedded newline character
like the ones in t_vnops do? For example, the first file in the test
is actually called "testfile0\n", not "testfile0". This looks like a
bug in the test (hi pooka!), but it's a useful bug if it is also
exposing a bug in udf. I'm considering removing the newlines from the
create_many test and instead adding a separate test of files with
non-alphanumeric characters in their names.
Running t_vnops in the debugger, I see that for the filename
"testfile0\n", udf_to_unix_name() is returning the string
"testfile0_#2A44". The file is not found because those two
strings are not the same.
--
Andreas Gustafsson, gson@gson.org
From: Reinoud Zandijk <reinoud@NetBSD.org>
To: Andreas Gustafsson <gson@gson.org>
Cc: pooka@NetBSD.org, gnats-bugs@NetBSD.org
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Wed, 13 Jan 2016 20:17:52 +0100
On Wed, Jan 13, 2016 at 08:34:49PM +0200, Andreas Gustafsson wrote:
> I'm considering removing the newlines from the create_many test and instead
> adding a separate test of files with non-alphanumeric characters in their
> names.
It had a newline in the name?? Since when is this allowed?
> Running t_vnops in the debugger, I see that for the filename "testfile0\n",
> udf_to_unix_name() is returning the string "testfile0_#2A44". The file is
> not found because those two strings are not the same.
Thats how it is supposed to work in UDF; names are transcribed to and from
compressed unicode-16 in a standard prescribed way. This '\n' is seen as an
illegal character and replaced by a _ but to avoid aliases its appended with
'#' and a kind of checksum/crc character.
Is the code also trying to delete the files with names appened with '\n'?
Either this or some rump code is removing the '\n' in the remove code. If it
would pass the filename exactly the same I see no reason it should not have
found it. Last time I tested the regression test on names, all passed. They
also explicitly test for mangled filenames and UTF support.
Reinoud
From: Andreas Gustafsson <gson@gson.org>
To: Reinoud Zandijk <reinoud@NetBSD.org>
Cc: pooka@NetBSD.org,
gnats-bugs@NetBSD.org
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Wed, 13 Jan 2016 21:30:22 +0200
Reinoud Zandijk wrote:
> On Wed, Jan 13, 2016 at 08:34:49PM +0200, Andreas Gustafsson wrote:
> > I'm considering removing the newlines from the create_many test and instead
> > adding a separate test of files with non-alphanumeric characters in their
> > names.
>
> It had a newline in the name??
Yes. Don't look at me, I didn't write the test...
> Since when is this allowed?
Not to defend the test, but intro(2) says:
File Name
Names consisting of up to 255 (MAXNAMELEN) characters may be used
to name an ordinary file, special file, or directory.
These characters may be selected from the set of all ASCII char-
acter excluding 0 (NUL) and the ASCII code for `/' (slash). (The
parity bit, bit 7, must be 0).
> Thats how it is supposed to work in UDF; names are transcribed to and from
> compressed unicode-16 in a standard prescribed way. This '\n' is seen as an
> illegal character and replaced by a _ but to avoid aliases its appended with
> '#' and a kind of checksum/crc character.
>
> Is the code also trying to delete the files with names appened with '\n'?
Yes.
> Either this or some rump code is removing the '\n' in the remove code.
No.
> If it would pass the filename exactly the same I see no reason it
> should not have found it.
When the test tries to delete "testfile0\n", UDF ends up looking up
"testfile0\n" in the dirhash, but the dirhash doesn't contain an entry
for "testfile0\n", only for "testfile0_#2A44".
--
Andreas Gustafsson, gson@gson.org
From: Reinoud Zandijk <reinoud@NetBSD.org>
To: Andreas Gustafsson <gson@gson.org>
Cc: pooka@NetBSD.org, gnats-bugs@NetBSD.org
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Wed, 13 Jan 2016 21:54:34 +0100
On Wed, Jan 13, 2016 at 09:30:22PM +0200, Andreas Gustafsson wrote:
> > If it would pass the filename exactly the same I see no reason it
> > should not have found it.
>
> When the test tries to delete "testfile0\n", UDF ends up looking up
> "testfile0\n" in the dirhash, but the dirhash doesn't contain an entry
> for "testfile0\n", only for "testfile0_#2A44".
Ah!!! thanks for spotting the bug, it is indeed a bug in my UDF
implementation; the dir detach and lookup functions seem to assume there are
no illegal characters in the string. I guess the best way would be to do the
unix->UDF->unix name dance to get a consistent result.
I'll fix it :)
Reinoud
From: matthew green <mrg@eterna.com.au>
To: gnats-bugs@NetBSD.org
Cc: reinoud@NetBSD.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
pooka@iki.fi
Subject: re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Thu, 14 Jan 2016 18:17:52 +1100
> > Since when is this allowed?
>
> Not to defend the test, but intro(2) says:
i would like to strongly defend the test. this *should* work.
.mrg.
From: Andreas Gustafsson <gson@gson.org>
To: mrg@eterna.com.au
Cc: gnats-bugs@NetBSD.org, reinoud@NetBSD.org, pooka@NetBSD.org
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Thu, 14 Jan 2016 10:56:11 +0200
mrg wrote:
> i would like to strongly defend the test. this *should* work.
Yes, of course. What I meant was that you should not use a filename
containing a newline in a test case whose purpose is to test something
*else*. This should be a separate test case, and I will commit such
a test case shortly.
--
Andreas Gustafsson, gson@gson.org
From: "Andreas Gustafsson" <gson@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/50608 CVS commit: src/tests/fs/vfs
Date: Thu, 14 Jan 2016 08:58:02 +0000
Module Name: src
Committed By: gson
Date: Thu Jan 14 08:58:02 UTC 2016
Modified Files:
src/tests/fs/vfs: t_vnops.c
Log Message:
Don't use a filename with an embedded newline in the create_many test
case. Instead, add a separate test case "create_nonalphanum" for
filenames containing non-alphanumeric characters. The bug of
PR kern/50608 now causes a failure in create_nonalphanum rather than
create_many.
To generate a diff of this commit:
cvs rdiff -u -r1.53 -r1.54 src/tests/fs/vfs/t_vnops.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Antti Kantee <pooka@NetBSD.org>
To: Andreas Gustafsson <gson@gson.org>, mrg@eterna.com.au
Cc: gnats-bugs@NetBSD.org, reinoud@NetBSD.org
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Thu, 14 Jan 2016 13:12:11 +0000
On 14/01/16 08:56, Andreas Gustafsson wrote:
> Yes, of course. What I meant was that you should not use a filename
> containing a newline in a test case whose purpose is to test something
> *else*. This should be a separate test case, and I will commit such
> a test case shortly.
Thank you for improving the granularity of the tests!
And also for pinpointing the bug!
From: "Reinoud Zandijk" <reinoud@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/50608 CVS commit: src/sys/fs/udf
Date: Wed, 27 Jan 2016 00:06:49 +0000
Module Name: src
Committed By: reinoud
Date: Wed Jan 27 00:06:49 UTC 2016
Modified Files:
src/sys/fs/udf: udf_subr.c
Log Message:
Fix filename creation and deleting with illegal file names. UDF has a
machanism for it but it allowed to create a file with a name it considered
illegal that then couldn't be deleted with the same name.
Fixes PR kern/50608. When confirmed, it can be closed.
To generate a diff of this commit:
cvs rdiff -u -r1.135 -r1.136 src/sys/fs/udf/udf_subr.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->closed
State-Changed-By: reinoud@NetBSD.org
State-Changed-When: Thu, 28 Jan 2016 10:16:28 +0000
State-Changed-Why:
Fixed by commit
From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/50608 CVS commit: src/tests/fs/vfs
Date: Thu, 28 Jan 2016 10:10:10 +0000
Module Name: src
Committed By: martin
Date: Thu Jan 28 10:10:09 UTC 2016
Modified Files:
src/tests/fs/vfs: t_vnops.c
Log Message:
The test for PR 50608 works now, remove the expected failure.
To generate a diff of this commit:
cvs rdiff -u -r1.54 -r1.55 src/tests/fs/vfs/t_vnops.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.