NetBSD Problem Report #39193

From martin@duskware.de  Tue Jul 22 23:04:33 2008
Return-Path: <martin@duskware.de>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by narn.NetBSD.org (Postfix) with ESMTP id 3F46F63B907
	for <gnats-bugs@gnats.NetBSD.org>; Tue, 22 Jul 2008 23:04:33 +0000 (UTC)
Message-Id: <20080722230431.1427133A82@mail.duskware.de>
Date: Wed, 23 Jul 2008 01:04:26 +0200 (CEST)
From: martin
Reply-To: martin
To: gnats-bugs@gnats.NetBSD.org
Subject: sysinst lost umlauts
X-Send-Pr-Version: 3.95

>Number:         39193
>Category:       install
>Synopsis:       sysinst lost umlauts
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    install-manager
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Jul 22 23:05:00 +0000 2008
>Closed-Date:    Fri Aug 08 20:49:31 +0000 2008
>Last-Modified:  Fri Aug 08 20:49:31 +0000 2008
>Originator:     Martin Husemann
>Release:        NetBSD 4.99.69
>Organization:
The NetBSD Foundation, Inc.
>Environment:
System: NetBSD sunny-weather.duskware.de 4.99.69 NetBSD 4.99.69 (SUNNY.MP) #79: Tue Jul 15 10:21:57 CEST 2008 martin@sunny-weather.duskware.de:/usr/src/sys/arch/sparc64/compile/SUNNY.MP sparc64
Architecture: sparc64
Machine: sparc64
>Description:

When running any sysinst based install and selecting a non-english translation,
the display is missing all non-ASCII chars. The problem is old and had been
"fixed" (for now) by using CPPFLAGS.runetable.c+= -I ${HACKSRC}/../../../lib/libc -DALL_80_TO_FF_SW1
in distrib/utils/libhack/Makefile.inc.

Somehow this does not work any more.
>How-To-Repeat:
boot any version of sysinst and select (for example) german messages.
(I checked i386 miniroot.kmod, sparc rootfs, sparc64 netbsd-INSTALL)

>Fix:
n/a

>Release-Note:

>Audit-Trail:
From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: install/39193: sysinst lost umlauts
Date: Fri, 25 Jul 2008 01:01:20 +0200

 I double checked: the libhack runetable is still compiled with 
 ALL_80_TO_FF_SW1, and libhack seems to be linked correctly when building
 sysinst - yet wcwidth(0xE4) returns 0.

 I don't see it - anyone?

 Martin

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: install/39193: sysinst lost umlauts
Date: Mon, 28 Jul 2008 01:55:53 +0200

 It is a sign extension problem: a-umlaut is 0xe4, calling wcwidth(0x000000e4)
 returns 1, but when choosing the german sysinst translaton it is called
 with 0xffffffe4 and returns 0.

 Martin

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: install/39193: sysinst lost umlauts
Date: Mon, 28 Jul 2008 10:44:26 +0200

 I verified that this is unrelated to libhack and the restricted locale
 support - it also happens in a debug build of sysinst with full dynamic
 libc and locale support.

 This makes it a lot easier to debug, fortunately.

 Martin

From: "Takehiko NOZAKI" <takehiko.nozaki@gmail.com>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: install/39193: sysinst lost umlauts
Date: Mon, 28 Jul 2008 18:56:20 +0900

 >   It is a sign extension problem: a-umlaut is 0xe4, calling wcwidth(0x000000e4)
 >    returns 1, but when choosing the german sysinst translaton it is called
 >    with 0xffffffe4 and returns 0.

 hmmm, libhack's version's mbrtowc/wcrtomb(src/distrib/utils/libhack/multibyte.c)
 seems  directlty cast char -> wchar_t, i think this is wrong...


 Index: multibyte.c
 ===================================================================
 RCS file: /cvsroot/src/distrib/utils/libhack/multibyte.c,v
 retrieving revision 1.3
 diff -u -r1.3 multibyte.c
 --- multibyte.c 12 Jul 2008 19:20:03 -0000      1.3
 +++ multibyte.c 28 Jul 2008 09:48:32 -0000
 @@ -11,13 +11,13 @@
  size_t
  mbrtowc(wchar_t *wc, const char *str, size_t max_sz, mbstate_t *ps)
  {
 -       return str == NULL || (*wc = *str) == 0 ? 0 : 1;
 +       return str == NULL || (*wc = (unsigned char)*str) == 0 ? 0 : 1;
  }

  size_t
  wcrtomb(char *str, wchar_t wc, mbstate_t *ps)
  {
 -    *str = wc;
 +    *str = wc & 0xFF;
      return 1;
  }


 >   I verified that this is unrelated to libhack and the restricted locale
 >   support - it also happens in a debug build of sysinst with full dynamic
 >   libc and locale support.

 sysinst(8) should have to call setlocale(3) in full dynmic libc and
 locale support, i think.

 Index: main.c
 ===================================================================
 RCS file: /cvsroot/src/distrib/utils/sysinst/main.c,v
 retrieving revision 1.52
 diff -u -r1.52 main.c
 --- main.c      23 Oct 2006 19:45:56 -0000      1.52
 +++ main.c      28 Jul 2008 09:54:11 -0000
 @@ -47,6 +47,7 @@
  #include <unistd.h>
  #include <fcntl.h>
  #include <dirent.h>
 +#include <locale.h>

  #include "defs.h"
  #include "md.h"
 @@ -136,6 +137,7 @@
  #endif
         scripting = 0;

 +       setlocale(LC_ALL, "");
         /* Check for TERM ... */
         if (!getenv("TERM")) {
                 (void)fprintf(stderr,

 previous version of src/libcurses/acs.c call setlocale(3) internally,
 but that is bug.
 so i removed rev1.16.
 http://cvsweb.netbsd.org/bsdweb.cgi/src/lib/libcurses/acs.c.diff?r1=1.15&r2=1.16

 very truly yours.
 -- 
 Takehiko NOZAKI<takehiko.nozaki@gmail.com>

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: install/39193: sysinst lost umlauts
Date: Mon, 28 Jul 2008 12:07:25 +0200

 On Mon, Jul 28, 2008 at 10:00:05AM +0000, Takehiko NOZAKI wrote:
 >  sysinst(8) should have to call setlocale(3) in full dynmic libc and
 >  locale support, i think.

 Ah, true - I always wanted to add a string with the proper locale name
 to the msg files but somehow did not come around to it (and it would be
 useless in the libhack'd version for now).

 I'll test your suggestion, thanks!

 Martin

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: install/39193: sysinst lost umlauts
Date: Mon, 28 Jul 2008 15:44:14 +0200

 On Mon, Jul 28, 2008 at 12:07:25PM +0200, Martin Husemann wrote:
 > Ah, true - I always wanted to add a string with the proper locale name
 > to the msg files but somehow did not come around to it (and it would be
 > useless in the libhack'd version for now).

 Here is the part setting the locale depending on selected language - sounds
 "more correct" to me, and is kind of a preparation for the future when we
 might to get full locael support in the ramdisg environment. This calls
 would just fail for now and do no harm.

 Martin

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: install/39193: sysinst lost umlauts
Date: Mon, 28 Jul 2008 15:44:37 +0200

 --5vNYLRcllDrimb99
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline

 Ooops, forgot the patch...

 --5vNYLRcllDrimb99
 Content-Type: text/plain; charset=iso-8859-1
 Content-Disposition: attachment; filename=patch
 Content-Transfer-Encoding: 8bit

 Index: main.c
 ===================================================================
 RCS file: /cvsroot/src/distrib/utils/sysinst/main.c,v
 retrieving revision 1.52
 diff -u -p -r1.52 main.c
 --- main.c	23 Oct 2006 19:45:56 -0000	1.52
 +++ main.c	28 Jul 2008 13:33:25 -0000
 @@ -47,6 +47,7 @@
  #include <unistd.h>
  #include <fcntl.h>
  #include <dirent.h>
 +#include <locale.h>

  #include "defs.h"
  #include "md.h"
 @@ -307,6 +308,11 @@ select_language(void)
  	}
  	free(lang_msg);
  	free(fnames);
 +
 +	/* set locale according to selected language */
 +	cp = msg_string(MSG_sysinst_message_locale);
 +	setlocale(LC_CTYPE, cp);
 +	setenv("LC_CTYPE", cp, 1);
  }

  /* toplevel menu handler ... */
 Index: msg.mi.de
 ===================================================================
 RCS file: /cvsroot/src/distrib/utils/sysinst/msg.mi.de,v
 retrieving revision 1.43
 diff -u -p -r1.43 msg.mi.de
 --- msg.mi.de	25 Mar 2008 12:54:32 -0000	1.43
 +++ msg.mi.de	28 Jul 2008 13:33:25 -0000
 @@ -45,6 +45,9 @@ message usage
  message sysinst_message_language
  {Installation auf Deutsch}

 +message sysinst_message_locale
 +{de_DE.ISO8859-15}
 +
  message Yes {Ja}
  message No {Nein}
  message All {Alle(s)}
 Index: msg.mi.en
 ===================================================================
 RCS file: /cvsroot/src/distrib/utils/sysinst/msg.mi.en,v
 retrieving revision 1.149
 diff -u -p -r1.149 msg.mi.en
 --- msg.mi.en	4 Feb 2008 01:54:56 -0000	1.149
 +++ msg.mi.en	28 Jul 2008 13:33:26 -0000
 @@ -45,6 +45,9 @@ message usage
  message sysinst_message_language
  {Installation messages in English}

 +message sysinst_message_locale
 +{en_US.ISO8859-1}
 +
  message Yes {Yes}
  message No {No}
  message All {All}
 Index: msg.mi.es
 ===================================================================
 RCS file: /cvsroot/src/distrib/utils/sysinst/msg.mi.es,v
 retrieving revision 1.21
 diff -u -p -r1.21 msg.mi.es
 --- msg.mi.es	4 Feb 2008 01:54:56 -0000	1.21
 +++ msg.mi.es	28 Jul 2008 13:33:26 -0000
 @@ -45,6 +45,9 @@ message usage
  message sysinst_message_language
  {Mensajes de instalación en castellano}

 +message sysinst_message_locale
 +{es_ES.ISO8859-15}
 +
  message Yes {Sí}
  message No {No}
  message All {Todo}
 Index: msg.mi.fr
 ===================================================================
 RCS file: /cvsroot/src/distrib/utils/sysinst/msg.mi.fr,v
 retrieving revision 1.101
 diff -u -p -r1.101 msg.mi.fr
 --- msg.mi.fr	4 Feb 2008 01:54:56 -0000	1.101
 +++ msg.mi.fr	28 Jul 2008 13:33:26 -0000
 @@ -45,6 +45,9 @@ message usage
  message sysinst_message_language
  {Messages d'installation en français}

 +message sysinst_message_locale
 +{fr_FR.ISO8859-15}
 +
  message Yes {Oui}
  message No {Non}
  message All {Tout}
 Index: msg.mi.pl
 ===================================================================
 RCS file: /cvsroot/src/distrib/utils/sysinst/msg.mi.pl,v
 retrieving revision 1.60
 diff -u -p -r1.60 msg.mi.pl
 --- msg.mi.pl	4 Feb 2008 01:54:56 -0000	1.60
 +++ msg.mi.pl	28 Jul 2008 13:33:26 -0000
 @@ -47,6 +47,9 @@ message usage
  message sysinst_message_language
  {Komunikaty instalacyjne w jezyku polskim}

 +message sysinst_message_locale
 +{pl_PL.ISO8859-2}
 +
  message Yes {Tak}
  message No {Nie}
  message All {Wszystkie}

 --5vNYLRcllDrimb99--

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: install/39193: sysinst lost umlauts
Date: Mon, 28 Jul 2008 20:15:00 +0200

 On Mon, Jul 28, 2008 at 12:07:25PM +0200, Martin Husemann wrote:
 > I'll test your suggestion, thanks!

 Indeed this fixes it. Please commit it!

 Thanks,

 Martin

State-Changed-From-To: open->closed
State-Changed-By: martin@NetBSD.org
State-Changed-When: Fri, 08 Aug 2008 20:49:31 +0000
State-Changed-Why:
fixed


>Unformatted:

Home	PR Database Search