NetBSD Problem Report #39445

From www@NetBSD.org  Mon Sep  1 07:13:37 2008
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by narn.NetBSD.org (Postfix) with ESMTP id 0004763BC80
	for <gnats-bugs@gnats.netbsd.org>; Mon,  1 Sep 2008 07:13:36 +0000 (UTC)
Message-Id: <20080901071336.8C8C163BC7D@narn.NetBSD.org>
Date: Mon,  1 Sep 2008 07:13:36 +0000 (UTC)
From: naruse@airemix.jp
Reply-To: naruse@airemix.jp
To: gnats-bugs@NetBSD.org
Subject: iconv returns wrong character on CP932
X-Send-Pr-Version: www-1.0

>Number:         39445
>Category:       lib
>Synopsis:       iconv returns wrong character on CP932
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    lib-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Sep 01 07:15:01 +0000 2008
>Closed-Date:    Sun Jun 07 15:55:38 +0000 2009
>Last-Modified:  Sun Jun 07 15:55:38 +0000 2009
>Originator:     NARUSE, Yui
>Release:        NetBSD 4.99.72
>Organization:
>Environment:
>Description:
Citrus iconv supports Microsoft CP932 (a.k.a. Windows-31J).
This codeset includes some duplicate mappings in Unicode to CP932 conversion.

Microsoft shows priority for those characters, but current NetBSD's implementation doesn't follow this.
http://support.microsoft.com/default.aspx?scid=kb;en-us;Q170559

Specification says priority is JISX0208 > NEC > NEC_IBM > IBM,
but implementation is JISX0208 > NEC > IBM > NEC_IBM.
http://www2d.biglobe.ne.jp/~msyk/charcode/cp932/uni2sjis.html (in japanese)


>How-To-Repeat:
Microsoft shows:

  0xeeef   -> U+2170   -> 0xfa40   Small Roman Numeral One

NetBSD 4.99.72:

  % printf '\x21\x70'|iconv -f utf-16be -t cp932|hexdump -C
  00000000  ee ef  

>Fix:
This is because mapper.dir.CP.src specifies wrong priority.
Following is a patch.

Index: mapper.dir.CP.src
===================================================================
RCS file: /cvsroot/src/share/i18n/csmapper/CP/mapper.dir.CP.src,v
retrieving revision 1.9
diff -u -p -r1.9 mapper.dir.CP.src
--- mapper.dir.CP.src   3 Dec 2006 14:28:46 -0000       1.9
+++ mapper.dir.CP.src   1 Sep 2008 06:57:42 -0000
@@ -13,6 +13,6 @@ CP942/UCS             mapper_parallel CP932UDA/UCS,
 CP943/UCS              mapper_parallel JISX0208VDC:NEC/UCS,CP932VDC:NEC_IBM/UCS,CP932VDC:IBM/UCS,CP932UDA/UCS,JISX0208:1990/UCS
 UCS/CP50220            mapper_parallel UCS/JISX0208:MS,UCS/JISX0208VDC:NEC,UCS/CP932VDC:NEC_IBM,UCS/CP932UDA
 UCS/CP51932            mapper_parallel UCS/JISX0208:MS,UCS/JISX0208VDC:NEC,UCS/CP932VDC:NEC_IBM
-UCS/CP932KANJI         mapper_parallel UCS/JISX0208:MS,UCS/JISX0208VDC:NEC,UCS/CP932VDC:NEC_IBM,UCS/CP932VDC:IBM,UCS/CP932UDA
+UCS/CP932KANJI         mapper_parallel UCS/JISX0208:MS,UCS/JISX0208VDC:NEC,UCS/CP932VDC:IBM,UCS/CP932VDC:NEC_IBM,UCS/CP932UDA
 UCS/CP942              mapper_parallel UCS/CP942EXT,UCS/CP932UDA
 UCS/CP943              mapper_parallel UCS/JISX0208:1990,UCS/JISX0208VDC:NEC,UCS/CP932VDC:NEC_IBM,UCS/CP932VDC:IBM,UCS/CP932UDA

>Release-Note:

>Audit-Trail:
From: Takehiko NOZAKI <tnozaki@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/39445 CVS commit: src/share/i18n/csmapper/CP
Date: Wed,  3 Sep 2008 15:42:45 +0000 (UTC)

 Module Name:	src
 Committed By:	tnozaki
 Date:		Wed Sep  3 15:42:45 UTC 2008

 Modified Files:
 	src/share/i18n/csmapper/CP: mapper.dir.CP.src

 Log Message:
 PR/39445, fix wrong charsets priority for CP932.
 reported and tested by NARUSE, Yui-san. thanks!


 To generate a diff of this commit:
 cvs rdiff -r1.9 -r1.10 src/share/i18n/csmapper/CP/mapper.dir.CP.src

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: open->closed
State-Changed-By: tnozaki@NetBSD.org
State-Changed-When: Wed, 03 Sep 2008 15:47:01 +0000
State-Changed-Why:
fixed, thanks!

From: Takahiro Kambe <taca@back-street.net>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: lib/39445 (iconv returns wrong character on CP932)
Date: Thu, 04 Sep 2008 18:26:24 +0900 (JST)

 Is it worth pulling up to netbsd-4 branch?

 -- 
 Takahiro Kambe <taca@back-street.net>

From: "Takehiko NOZAKI" <takehiko.nozaki@gmail.com>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: lib/39445 (iconv returns wrong character on CP932)
Date: Fri, 5 Sep 2008 19:04:15 +0900

 hi,  kambe-san.

 >  Is it worth pulling up to netbsd-4 branch?

 i think yes, many  japanese NetBSD user want
 right ucs4 -> cp932 conversion table for samba, i guess.

 i've already sent pullup request.
 http://releng.netbsd.org/cgi-bin/req-4.cgi?show=1194


 very truly yours.
 -- 
 Takehiko NOZAKI<takehiko.nozaki@gmail.com>

From: Takahiro Kambe <taca@back-street.net>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: lib/39445 (iconv returns wrong character on CP932)
Date: Tue, 09 Sep 2008 14:45:17 +0900 (JST)

 In message <20080905100503.0CE7C63B11D@narn.NetBSD.org>
 	on Fri,  5 Sep 2008 10:05:03 +0000 (UTC),
 	"Takehiko NOZAKI" <takehiko.nozaki@gmail.com> wrote:
 >  hi,  kambe-san.
 Hi,

 >  >  Is it worth pulling up to netbsd-4 branch?
 >  
 >  i think yes, many  japanese NetBSD user want
 >  right ucs4 -> cp932 conversion table for samba, i guess.
 >  
 >  i've already sent pullup request.
 >  http://releng.netbsd.org/cgi-bin/req-4.cgi?show=1194
 Oh, thanks.

 I asked the question because the PR's state "closed" not "pending-pullups".

 Best regards.

 -- 
 Takahiro Kambe <taca@back-street.net>

State-Changed-From-To: closed->pending-pullups
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Thu, 11 Sep 2008 15:56:33 +0000
State-Changed-Why:
Pullups are in the queue. (Not everyone knows about pending-pullups yet, I
think, and even when they do it's easy to forget.)


From: Manuel Bouyer <bouyer@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/39445 CVS commit: [netbsd-4] src/share/i18n/csmapper/CP
Date: Sun, 7 Jun 2009 11:58:47 +0000

 Module Name:	src
 Committed By:	bouyer
 Date:		Sun Jun  7 11:58:46 UTC 2009

 Modified Files:
 	src/share/i18n/csmapper/CP [netbsd-4]: mapper.dir.CP.src

 Log Message:
 Pull up following revision(s) (requested by tnozaki in ticket #1194):
 	share/i18n/csmapper/CP/mapper.dir.CP.src: revision 1.10 via patch
 PR/39445, fix wrong charsets priority for CP932.
 reported and tested by NARUSE, Yui-san. thanks!


 To generate a diff of this commit:
 cvs rdiff -u -r1.7 -r1.7.2.1 src/share/i18n/csmapper/CP/mapper.dir.CP.src

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: pending-pullups->closed
State-Changed-By: snj@NetBSD.org
State-Changed-When: Sun, 07 Jun 2009 15:55:38 +0000
State-Changed-Why:
Pullup complete.


>Unformatted:

Home	PR Database Search