NetBSD Problem Report #50608

From www@NetBSD.org  Sat Jan  2 12:07:33 2016
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.NetBSD.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 57E0B7ACA4
	for <gnats-bugs@gnats.NetBSD.org>; Sat,  2 Jan 2016 12:07:33 +0000 (UTC)
Message-Id: <20160102120732.6DE5E7ACC8@mollari.NetBSD.org>
Date: Sat,  2 Jan 2016 12:07:32 +0000 (UTC)
From: pooka@iki.fi
Reply-To: pooka@iki.fi
To: gnats-bugs@NetBSD.org
Subject: udf fails the fs/vfs/t_vnops::create_many test
X-Send-Pr-Version: www-1.0

>Number:         50608
>Category:       kern
>Synopsis:       udf fails the fs/vfs/t_vnops::create_many test
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    reinoud
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Jan 02 12:10:00 +0000 2016
>Closed-Date:    Thu Jan 28 10:16:28 +0000 2016
>Last-Modified:  Thu Jan 28 10:20:00 +0000 2016
>Originator:     Antti Kantee
>Release:        
>Organization:
>Environment:
>Description:
Not clear what exactly is going on, but it seems like under some conditions udf unlink() returns ENOENT for a recently created file.
>How-To-Repeat:
run tests/fs/vfs/t_vnops::create_many for udf

https://releng.netbsd.org/b5reports/i386/build/2016.01.02.01.24.44/test.html#fs_vfs_t_vnops_udf_create_many
>Fix:

>Release-Note:

>Audit-Trail:

Responsible-Changed-From-To: kern-bug-people->reinoud
Responsible-Changed-By: dholland@NetBSD.org
Responsible-Changed-When: Sun, 03 Jan 2016 01:55:31 +0000
Responsible-Changed-Why:
over to udf maintainer
(if you can't deal with it, set it back)


From: Reinoud Zandijk <reinoud@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: reinoud@NetBSD.org, kern-bug-people@netbsd.org, netbsd-bugs@netbsd.org,
	gnats-admin@netbsd.org, dholland@NetBSD.org, pooka@iki.fi
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Mon, 4 Jan 2016 09:59:17 +0100

 Hi,

 I've tried to replicate it in a NetBSD-current kernel on a vnd disc with sets
 of various sizes but I couldn't replicate it. Still investigating it, might be
 some emulation choices like dircache sizes that is preventing it to cache a
 2500+(?) entry dircache where in a normal kernel this is considered trivial in
 size.

 With regards,
 Reinoud

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Mon, 4 Jan 2016 09:37:17 +0000

 On Mon, Jan 04, 2016 at 09:00:01AM +0000, Reinoud Zandijk wrote:
  > I've tried to replicate it in a NetBSD-current kernel on a vnd
  > disc with sets of various sizes but I couldn't replicate it. Still
  > investigating it, might be some emulation choices like dircache
  > sizes that is preventing it to cache a 2500+(?) entry dircache
  > where in a normal kernel this is considered trivial in size.

 In order for the namecache to produce a wrong negative result it is
 theoretically necessary to prime it with wrong information; just being
 full isn't enough.

 I wonder though if this is the same problem people have been seeing
 intermittently with tmpfs. If so, chasing after it in this form is
 definitely worthwhile.

 -- 
 David A. Holland
 dholland@netbsd.org

From: Andreas Gustafsson <gson@gson.org>
To: reinoud@NetBSD.org
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Sat, 9 Jan 2016 16:36:44 +0200

 reinoud wrote:
 > I've tried to replicate it in a NetBSD-current kernel on a vnd disc with sets
 > of various sizes but I couldn't replicate it.

 This consistently replicates it for me:

   $ sudo pkg_add qemu
   $ sudo pkg_add py-anita
   $ anita interact http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/201601090500Z/i386/
   login: root
   # cd /usr/tests/fs/vfs
   # ./t_vnops udf_create_many

 It's also failing on a physical amd64 machine (no qemu involved):

   http://www.gson.org/netbsd/bugs/build/amd64-baremetal/2016/2016.01.05.10.20.22/test.html#fs_vfs_t_vnops_udf_create_many

 -- 
 Andreas Gustafsson, gson@gson.org

From: Reinoud Zandijk <reinoud@NetBSD.org>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Wed, 13 Jan 2016 17:39:00 +0100

 I've tried your sequence and indeed it fails on an physical amd64. What I
 meant is that i tried out a program that does exactly the same as the test
 only then *without* RUMP and that passes fine.

 I'll try to dig into the issue, might be the generic dirhash code misbehaving
 though i still think its a rump internal problem.

 On my machine this was set to the default:
 vfs.generic.dirhash.maxmem = 1048576

 Reinoud

From: Andreas Gustafsson <gson@gson.org>
To: Reinoud Zandijk <reinoud@NetBSD.org>
Cc: pooka@NetBSD.org,
    gnats-bugs@NetBSD.org
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Wed, 13 Jan 2016 20:34:49 +0200

 Reinoud Zandijk wrote:
 > I've tried your sequence and indeed it fails on an physical amd64. What I
 > meant is that i tried out a program that does exactly the same as the test
 > only then *without* RUMP and that passes fine.
 > 
 > I'll try to dig into the issue, might be the generic dirhash code misbehaving
 > though i still think its a rump internal problem.

 In your test, does the filename contain an embedded newline character
 like the ones in t_vnops do?  For example, the first file in the test
 is actually called "testfile0\n", not "testfile0".  This looks like a
 bug in the test (hi pooka!), but it's a useful bug if it is also
 exposing a bug in udf.  I'm considering removing the newlines from the
 create_many test and instead adding a separate test of files with
 non-alphanumeric characters in their names.

 Running t_vnops in the debugger, I see that for the filename
 "testfile0\n", udf_to_unix_name() is returning the string
 "testfile0_#2A44".  The file is not found because those two
 strings are not the same.
 -- 
 Andreas Gustafsson, gson@gson.org

From: Reinoud Zandijk <reinoud@NetBSD.org>
To: Andreas Gustafsson <gson@gson.org>
Cc: pooka@NetBSD.org, gnats-bugs@NetBSD.org
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Wed, 13 Jan 2016 20:17:52 +0100

 On Wed, Jan 13, 2016 at 08:34:49PM +0200, Andreas Gustafsson wrote:
 > I'm considering removing the newlines from the create_many test and instead
 > adding a separate test of files with non-alphanumeric characters in their
 > names.

 It had a newline in the name?? Since when is this allowed?

 > Running t_vnops in the debugger, I see that for the filename "testfile0\n",
 > udf_to_unix_name() is returning the string "testfile0_#2A44".  The file is
 > not found because those two strings are not the same.

 Thats how it is supposed to work in UDF; names are transcribed to and from
 compressed unicode-16 in a standard prescribed way. This '\n' is seen as an
 illegal character and replaced by a _ but to avoid aliases its appended with
 '#' and a kind of checksum/crc character.

 Is the code also trying to delete the files with names appened with '\n'?
 Either this or some rump code is removing the '\n' in the remove code. If it
 would pass the filename exactly the same I see no reason it should not have
 found it. Last time I tested the regression test on names, all passed. They
 also explicitly test for mangled filenames and UTF support.

 Reinoud

From: Andreas Gustafsson <gson@gson.org>
To: Reinoud Zandijk <reinoud@NetBSD.org>
Cc: pooka@NetBSD.org,
    gnats-bugs@NetBSD.org
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Wed, 13 Jan 2016 21:30:22 +0200

 Reinoud Zandijk wrote:
 > On Wed, Jan 13, 2016 at 08:34:49PM +0200, Andreas Gustafsson wrote:
 > > I'm considering removing the newlines from the create_many test and instead
 > > adding a separate test of files with non-alphanumeric characters in their
 > > names.
 > 
 > It had a newline in the name??

 Yes.  Don't look at me, I didn't write the test...

 > Since when is this allowed?

 Not to defend the test, but intro(2) says:

      File Name
              Names consisting of up to 255 (MAXNAMELEN) characters may be used
              to name an ordinary file, special file, or directory.

              These characters may be selected from the set of all ASCII char-
              acter excluding 0 (NUL) and the ASCII code for `/' (slash).  (The
              parity bit, bit 7, must be 0).

 > Thats how it is supposed to work in UDF; names are transcribed to and from
 > compressed unicode-16 in a standard prescribed way. This '\n' is seen as an
 > illegal character and replaced by a _ but to avoid aliases its appended with
 > '#' and a kind of checksum/crc character.
 > 
 > Is the code also trying to delete the files with names appened with '\n'?

 Yes.  

 > Either this or some rump code is removing the '\n' in the remove code.

 No.

 > If it would pass the filename exactly the same I see no reason it
 > should not have found it.

 When the test tries to delete "testfile0\n", UDF ends up looking up
 "testfile0\n" in the dirhash, but the dirhash doesn't contain an entry
 for "testfile0\n", only for "testfile0_#2A44".
 -- 
 Andreas Gustafsson, gson@gson.org

From: Reinoud Zandijk <reinoud@NetBSD.org>
To: Andreas Gustafsson <gson@gson.org>
Cc: pooka@NetBSD.org, gnats-bugs@NetBSD.org
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Wed, 13 Jan 2016 21:54:34 +0100

 On Wed, Jan 13, 2016 at 09:30:22PM +0200, Andreas Gustafsson wrote:
 > > If it would pass the filename exactly the same I see no reason it
 > > should not have found it.
 > 
 > When the test tries to delete "testfile0\n", UDF ends up looking up
 > "testfile0\n" in the dirhash, but the dirhash doesn't contain an entry
 > for "testfile0\n", only for "testfile0_#2A44".

 Ah!!! thanks for spotting the bug, it is indeed a bug in my UDF
 implementation; the dir detach and lookup functions seem to assume there are
 no illegal characters in the string. I guess the best way would be to do the
 unix->UDF->unix name dance to get a consistent result.

 I'll fix it :)
 Reinoud

From: matthew green <mrg@eterna.com.au>
To: gnats-bugs@NetBSD.org
Cc: reinoud@NetBSD.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
    pooka@iki.fi
Subject: re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Thu, 14 Jan 2016 18:17:52 +1100

 >  > Since when is this allowed?
 >  
 >  Not to defend the test, but intro(2) says:

 i would like to strongly defend the test.  this *should* work.


 .mrg.

From: Andreas Gustafsson <gson@gson.org>
To: mrg@eterna.com.au
Cc: gnats-bugs@NetBSD.org, reinoud@NetBSD.org, pooka@NetBSD.org
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Thu, 14 Jan 2016 10:56:11 +0200

 mrg wrote:
 > i would like to strongly defend the test.  this *should* work.

 Yes, of course.  What I meant was that you should not use a filename
 containing a newline in a test case whose purpose is to test something
 *else*.  This should be a separate test case, and I will commit such
 a test case shortly.
 -- 
 Andreas Gustafsson, gson@gson.org

From: "Andreas Gustafsson" <gson@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/50608 CVS commit: src/tests/fs/vfs
Date: Thu, 14 Jan 2016 08:58:02 +0000

 Module Name:	src
 Committed By:	gson
 Date:		Thu Jan 14 08:58:02 UTC 2016

 Modified Files:
 	src/tests/fs/vfs: t_vnops.c

 Log Message:
 Don't use a filename with an embedded newline in the create_many test
 case.  Instead, add a separate test case "create_nonalphanum" for
 filenames containing non-alphanumeric characters.  The bug of
 PR kern/50608 now causes a failure in create_nonalphanum rather than
 create_many.


 To generate a diff of this commit:
 cvs rdiff -u -r1.53 -r1.54 src/tests/fs/vfs/t_vnops.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: Antti Kantee <pooka@NetBSD.org>
To: Andreas Gustafsson <gson@gson.org>, mrg@eterna.com.au
Cc: gnats-bugs@NetBSD.org, reinoud@NetBSD.org
Subject: Re: kern/50608 (udf fails the fs/vfs/t_vnops::create_many test)
Date: Thu, 14 Jan 2016 13:12:11 +0000

 On 14/01/16 08:56, Andreas Gustafsson wrote:
 > Yes, of course.  What I meant was that you should not use a filename
 > containing a newline in a test case whose purpose is to test something
 > *else*.  This should be a separate test case, and I will commit such
 > a test case shortly.

 Thank you for improving the granularity of the tests!

 And also for pinpointing the bug!

From: "Reinoud Zandijk" <reinoud@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/50608 CVS commit: src/sys/fs/udf
Date: Wed, 27 Jan 2016 00:06:49 +0000

 Module Name:	src
 Committed By:	reinoud
 Date:		Wed Jan 27 00:06:49 UTC 2016

 Modified Files:
 	src/sys/fs/udf: udf_subr.c

 Log Message:
 Fix filename creation and deleting with illegal file names. UDF has a
 machanism for it but it allowed to create a file with a name it considered
 illegal that then couldn't be deleted with the same name.

 Fixes PR kern/50608. When confirmed, it can be closed.


 To generate a diff of this commit:
 cvs rdiff -u -r1.135 -r1.136 src/sys/fs/udf/udf_subr.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->closed
State-Changed-By: reinoud@NetBSD.org
State-Changed-When: Thu, 28 Jan 2016 10:16:28 +0000
State-Changed-Why:
Fixed by commit


From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/50608 CVS commit: src/tests/fs/vfs
Date: Thu, 28 Jan 2016 10:10:10 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Thu Jan 28 10:10:09 UTC 2016

 Modified Files:
 	src/tests/fs/vfs: t_vnops.c

 Log Message:
 The test for PR 50608 works now, remove the expected failure.


 To generate a diff of this commit:
 cvs rdiff -u -r1.54 -r1.55 src/tests/fs/vfs/t_vnops.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

>Unformatted:

Home	PR Database Search