NetBSD Problem Report #52724

From www@NetBSD.org  Mon Nov 13 11:01:02 2017
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 7AFF37A1D0
	for <gnats-bugs@gnats.NetBSD.org>; Mon, 13 Nov 2017 11:01:02 +0000 (UTC)
Message-Id: <20171113110101.5046E7A263@mollari.NetBSD.org>
Date: Mon, 13 Nov 2017 11:01:01 +0000 (UTC)
From: trebol4444@gmx.com
Reply-To: trebol4444@gmx.com
To: gnats-bugs@NetBSD.org
Subject: ntfs driver doen't allow ':' in file names.
X-Send-Pr-Version: www-1.0

>Number:         52724
>Category:       kern
>Synopsis:       ntfs driver doen't allow ':' in file names.
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    kern-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Nov 13 11:05:00 +0000 2017
>Closed-Date:    Tue Jan 23 22:00:53 +0000 2024
>Last-Modified:  Tue Jan 23 22:00:53 +0000 2024
>Originator:     trebol
>Release:        current
>Organization:
>Environment:
NetBSD npi 8.99.6 NetBSD 8.99.6 (RPI2) #2: Mon Nov 13 00:06:36 WET 2017  trebol@npi:/data/netbsd/src/sys/arch/evbarm/compile/obj/RPI2 evbarm
>Description:
In a POSIX environment, all unicode characters except '/' and '\0' must be allowed in file names of a ntfs file system.

The ntfs driver of NetBSD uses ':' for representing ntfs attributes, making impossible to properly read any file with ':' in its name.
>How-To-Repeat:
Write a file with ':' in its name using a POSIX OS without this problem, and try to read it in NetBSD with the kernel's ntfs driver.
>Fix:
The best solution in my opinion is to make a system utility for deal with ntfs attributes.

An invisible character, like ^G can be used instead of ':' to avoid problems with POSIX name space.

--- sys/fs/ntfs/ntfs_subr.c.orig        2017-11-12 11:44:53.152382608 +0000
+++ sys/fs/ntfs/ntfs_subr.c     2017-11-13 00:05:07.333116651 +0000
@@ -696,7 +696,7 @@
 }

 /*
- * Lookup attribute name in format: [[:$ATTR_TYPE]:$ATTR_NAME],
+ * Lookup attribute name in format: [['\a'$ATTR_TYPE]:$ATTR_NAME],
  * $ATTR_TYPE is searched in attrdefs read from $AttrDefs.
  * If $ATTR_TYPE not specified, ATTR_A_DATA assumed.
  */
@@ -782,14 +782,14 @@
        }

        /*
-        * Divide file name into: foofilefoofilefoofile[:attrspec]
+        * Divide file name into: foofilefoofilefoofile['\a'attrspec]
         * Store like this:       fname:fnamelen       [aname:anamelen]
         */
        fname = cnp->cn_nameptr;
        aname = NULL;
        anamelen = 0;
        for (fnamelen = 0; fnamelen < cnp->cn_namelen; fnamelen++)
-               if (fname[fnamelen] == ':') {
+               if (fname[fnamelen] == '\a') {
                        aname = fname + fnamelen + 1;
                        anamelen = cnp->cn_namelen - fnamelen - 1;
                        dprintf(("%s: %s (%d), attr: %s (%d)\n", __func__,

>Release-Note:

>Audit-Trail:
From: "J. Hannken-Illjes" <hannken@eis.cs.tu-bs.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/52724: ntfs driver doen't allow ':' in file names.
Date: Mon, 13 Nov 2017 12:51:07 +0100

 The correct fix seems to parse for colons from the end of the name
 and stop after the second colon, so you could use

 "File:Name:ATTRTYPE:ATTRNAME"

 to access the file named "File:Name".

 --
 J. Hannken-Illjes - hannken@eis.cs.tu-bs.de - TU Braunschweig (Germany)

From: trebol <trebol4444@gmx.com>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/52724: ntfs driver doen't allow ':' in file names.
Date: Mon, 13 Nov 2017 12:31:46 +0000 (WET)

 > The correct fix seems to parse for colons from the end of the name
 > and stop after the second colon, so you could use
 >
 > "File:Name:ATTRTYPE:ATTRNAME"
 >
 > to access the file named "File:Name".

 So instead of type a file name transparently, you have to type

 "File:Name:\$ATTRTYPE:\$ATTRNAME"

 And every applications that deal with files have to be rewritten to detect 
 the file system and in case of been in ntfs use

 "File:Name:\$ATTRTYPE:\$ATTRNAME"

 And this seems to be the correct fix.

 Sorry, but I don't get it.

From: Dave Huang <khym@azeotrope.org>
To: gnats-bugs@NetBSD.org, kern-bug-people@netbsd.org,
 gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Cc: 
Subject: Re: kern/52724: ntfs driver doen't allow ':' in file names.
Date: Mon, 13 Nov 2017 16:59:57 -0600

 On 11/13/2017 5:05, trebol4444@gmx.com wrote:
 > In a POSIX environment, all unicode characters except '/' and '\0' must be allowed in file names of a ntfs file system.
 > 
 > The ntfs driver of NetBSD uses ':' for representing ntfs attributes, making impossible to properly read any file with ':' in its name.
 >> How-To-Repeat:
 > Write a file with ':' in its name using a POSIX OS without this problem, and try to read it in NetBSD with the kernel's ntfs driver.

 Which POSIX OS lets you do that? While Windows NT used to have a POSIX 
 subsystem (https://en.wikipedia.org/wiki/Microsoft_POSIX_subsystem), 
 that was discontinued long ago.

 While perhaps technically correct that NTFS filenames can contain ":", 
 in practice, the vast majority of files on an NTFS filesystem will have 
 been written by some version of Windows, and the Win32 subsystem does 
 not allow ":" in filenames. Even the replacement for the POSIX 
 subsystem, the Windows Subsystem for Linux, doesn't actually store ":" 
 on disk; if you create a file with a ":" in the name, the actual on-disk 
 name has the ":" escaped as "#003A". So I would say that if you actually 
 have an NTFS filesystem with a file that has ":" in the name, you'd have 
 a lot of trouble reading it on an actual Windows machine.

 In Windows, ":" in filenames are used for Alternate Data Streams 
 (https://blogs.technet.microsoft.com/askcore/2013/03/24/alternate-data-streams-in-ntfs/). 
 E.g., you can "echo text > somefile.txt:somestream" and "more < 
 somefile.txt:somestream". And alternate data streams are just another 
 attribute of a file in NTFS. I don't know whether NetBSD's ntfs 
 implementation allows accessing ADSes, but it sounds like it might if 
 it's treating ":"s as special. In which case, I don't think it should be 
 changed to some other character, since that would make it work 
 differently than an actual Windows machine.

From: trebol <trebol4444@gmx.com>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/52724: ntfs driver doen't allow ':' in file names.
Date: Tue, 14 Nov 2017 00:26:38 +0000 (WET)

 I've read all those links, and sadly for my health, a lot more before 
 reporting this.

 It doesn't matter what games does windows with file names to avoid 
 intern conflicts, or which format use its programs.

 utfs support posix file names, that is, unicode characters except '\' 
 and '\0' and case sensitive.

 You can have problems (I suppose, I don't use Windows) with case 
 sensitive file names. But as it is supported, the implementation must 
 allow it. You can advise in the manual page to avoid using it. The same 
 for ':', '<', '\', etc.

 Utfs is used in removable media to assure compatibility between mac, linux 
 and windows, because the file size limitation of fat32. I'd prefer that 
 people use udf, but that is not the case, and once in a while a have to 
 stick a utfs pen drive in one of my computers. And as the implementation 
 of ntfs in mac and linux allow ':' in file names, it is normal to find 
 files with colon in its names.

 Any way. I can't care less. I don't use Windows. It makes me sick having 
 to read stupid documentation talking about the occult magic of microsoft, 
 with all the images of what explorer and that crap shell does.

 I only have to change two lines of code.

 Do whatever you want.

 trebol.

From: Dave Huang <khym@azeotrope.org>
To: gnats-bugs@NetBSD.org, kern-bug-people@netbsd.org,
 gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, trebol4444@gmx.com
Cc: 
Subject: Re: kern/52724: ntfs driver doen't allow ':' in file names.
Date: Tue, 14 Nov 2017 03:19:48 -0600

 On 11/13/2017 18:30, trebol wrote:
 >   I only have to change two lines of code.
 >   
 >   Do whatever you want.

 Yes, you can do whatever you want to the code you run, but I don't think 
 NetBSD should change to support whatever weird filenames you happen to 
 have at the expense of breaking support for a commonly-used NTFS feature.

From: Martin Husemann <martin@duskware.de>
To: Dave Huang <khym@azeotrope.org>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/52724: ntfs driver doen't allow ':' in file names.
Date: Tue, 14 Nov 2017 10:21:58 +0100

 On Tue, Nov 14, 2017 at 03:19:48AM -0600, Dave Huang wrote:
 > On 11/13/2017 18:30, trebol wrote:
 > >   I only have to change two lines of code.
 > >   Do whatever you want.
 > 
 > Yes, you can do whatever you want to the code you run, but I don't think
 > NetBSD should change to support whatever weird filenames you happen to have
 > at the expense of breaking support for a commonly-used NTFS feature.

 I'm confused - where does that "breaking support for a commonly-used NTFS
 feature" happen?

 Martin

From: Dave Huang <khym@azeotrope.org>
To: Martin Husemann <martin@duskware.de>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/52724: ntfs driver doen't allow ':' in file names.
Date: Tue, 14 Nov 2017 03:33:33 -0600

 On 11/14/2017 3:21, Martin Husemann wrote:
 > I'm confused - where does that "breaking support for a commonly-used NTFS
 > feature" happen?

 Well using a non-printing character to access alternate data streams 
 certainly violates the principle of least astonishment, at least... 
 Windows uses ":", and NTFS is primarily a Windows filesystem. If NetBSD 
 wants to be different, I think there should be a good reason for it, and 
 personally, I don't consider accessing a file that would also be 
 inaccessible on an actual Windows system to be a good reason.

 I'm still curious as to which POSIX OS is creating files on NTFS with 
 colons in their names.

From: trebol <trebol4444@gmx.com>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/52724: ntfs driver doen't allow ':' in file names.
Date: Tue, 14 Nov 2017 12:28:42 +0000 (WET)

 On Tue, 14 Nov 2017, Dave Huang wrote:
 > Well using a non-printing character to access alternate data streams
 > certainly violates the principle of least astonishment, at least...

 As I've said in my problem report, the best solution for me is to write a 
 system utility to deal with ntfs attributes (and ads). You can use in 
 that utility the format that windows uses in its programs, with ':' as a 
 separator. You can use in the implementation of that program the idea of 
 J. Hannken to parse for colons from the end of the name, so you can keep 
 naming your files without restrictions.

 Using ^G is a dirty hack that allow me to read files without restrictions, 
 and the same time access the attributes without any problems.


 > Windows uses ":", and NTFS is primarily a Windows filesystem.

 Yes, and case insensitive file names. So NetBSD must use case insensitive 
 file names in ntfs? And use '\' for paths in ntfs because window uses 
 '\'?

 > If NetBSD wants to be different, I think there should be a good reason
 for it

 Because is UNIX, and the file system permits the use of unicode characters 
 except '\' and '\0' in posix sytems.

 If an OS has restrictions in its file names, is that OS who has to deal 
 with it. As long as the file system permits it (and doesn't matter who 
 develop the file system) other OSes don't have to be tied to others 
 limitations.

 There is other restrictions in windows names apart of the use of some 
 characters. A lot of them. Do you think is a good idea to put these 
 restrictions in the ntfs implementation?

 > personally, I don't consider accessing a file that would also be 
 > inaccessible on an actual Windows system to be a good reason.

 I hope you have tested that, because if not, you are just trolling 
 around.

 > I'm still curious as to which POSIX OS is creating files on NTFS with 
 > colons in their names.

 - Every OS that uses ntfs-3g, for example NetBSD?
 - Tuxera NTFS (a comercial version from the ntfs-3g developers)
 - Linux ntfs driver.
 - Someone with a Mac can test the internal (unstable) driver, I don't know.

 I said "Do whatever you want", but nevertheless, I'm now going around 
 things I already have said and looking for a stupid windows machine to 
 test file names with colons...

 trebol.

From: trebol <trebol4444@gmx.com>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/52724: ntfs driver doen't allow ':' in file names.
Date: Tue, 14 Nov 2017 16:13:14 +0000 (WET)

 > I'm now going around things I already have said and looking for a stupid 
 > windows machine to test file names with colons...

 Wow, I'm not doing that. My head is upside down. As I explained above, it 
 doesn't matter, that was precisely my point.

 I described my problem (in my opinion). I gave to you an idea of a good 
 solution (in my opinion) and an actual workaround.

 It's up to you, the developers.
 I'm out of this discussion.

 trebol.

State-Changed-From-To: open->closed
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Tue, 23 Jan 2024 22:00:53 +0000
State-Changed-Why:
We keep the current use of ':' as the attribute separator. The alternative
is available in the PR for whoever is interested. Thanks for report.


>Unformatted:

Home	PR Database Search