NetBSD Problem Report #56926

From www@netbsd.org  Fri Jul 15 13:49:25 2022
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 151E81A923B
	for <gnats-bugs@gnats.NetBSD.org>; Fri, 15 Jul 2022 13:49:25 +0000 (UTC)
Message-Id: <20220715134922.54C1E1A923C@mollari.NetBSD.org>
Date: Fri, 15 Jul 2022 13:49:22 +0000 (UTC)
From: rokuyama.rk@gmail.com
Reply-To: rokuyama.rk@gmail.com
To: gnats-bugs@NetBSD.org
Subject: curses: vi screen corrupted with UTF-8 multicolumn characters
X-Send-Pr-Version: www-1.0

>Number:         56926
>Category:       lib
>Synopsis:       curses: vi screen corrupted with UTF-8 multicolumn characters
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    blymn
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Jul 15 13:50:00 +0000 2022
>Closed-Date:    Wed Oct 19 21:21:15 +0000 2022
>Last-Modified:  Fri Oct 21 06:00:02 +0000 2022
>Originator:     Rin Okuyama
>Release:        9.99.97 as well as 9.99.98
>Organization:
Department of Physics, Meiji University
>Environment:
NetBSD rp64 9.99.97 NetBSD 9.99.97 (GENERIC64EB) #0: Mon Jun  6 12:27:47 JST 2022  rin@latipes:/build/src/sys/arch/evbarm/compile/GENERIC64EB evbarm
>Description:
As a recent regression, vi(1) screen gets corrupted for files
including UTF-8 multicolumn characters.

For example, edit this file with LC_CTYPE=ja_JP.UTF-8:

http://www.netbsd.org/~rin/curses_20220715/sample.txt

----
% ftp http://www.netbsd.org/~rin/curses_20220715/sample.txt
...snip...
% export LC_CTYPE=ja_JP.UTF-8
% locale
LANG=""
LC_CTYPE="ja_JP.UTF-8"
LC_COLLATE="C"
LC_TIME="C"
LC_NUMERIC="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_ALL=""
% vi sample.txt
----

Then, screen gets corrupted when cursor moves on multicolumn characters,
as observed in this movie:

http://www.netbsd.org/~rin/curses_20220715/vi.mp4

This occurs for all terminal emulators as far as I can see:
- xterm
- pkgsrc/x11/mlterm
- iTerm2 (for macOS)

By bisectioning, it turned out the regression starts with this commit:

http://www.nerv.org/netbsd/?q=id:20220125T030506Z.bcfcf004c0eda8a39471f720b258282e7df9f026

I've found two typos in addbytes.c:

----
Index: addbytes.c
===================================================================
RCS file: /home/netbsd/src/lib/libcurses/addbytes.c,v
retrieving revision 1.64
diff -p -u -r1.64 addbytes.c
--- addbytes.c	19 May 2022 07:41:26 -0000	1.64
+++ addbytes.c	15 Jul 2022 13:39:10 -0000
@@ -301,12 +301,12 @@ _cursesi_addbyte(WINDOW *win, __LINE **l
 	    *(*lp)->lastchp - win->ch_off);
 	if (win->bch != ' ' && c == ' ') {
 		(*lp)->line[*x].ch = win->bch;
-#ifdef HAVE_CHAR
+#ifdef HAVE_WCHAR
 		(*lp)->line[*x].wcols = win->wcols;
 #endif
 	} else {
 		(*lp)->line[*x].ch = c;
-#ifdef HAVE_CHAR
+#ifdef HAVE_WCHAR
 		(*lp)->line[*x].wcols = wcols;
 #endif
 	}
----

But this patch does not improve the situation.
>How-To-Repeat:
Described above.
>Fix:
N/A

>Release-Note:

>Audit-Trail:


From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: lib/56926: curses: vi screen corrupted with UTF-8 multicolumn
 characters
Date: Thu, 6 Oct 2022 22:42:11 +0000

 (this message didn't get sent to gnats originally)

    ------

 From: Rin Okuyama <rokuyama.rk@gmail.com>
 To: lib-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
 Cc: Brett Lymn <blymn@NetBSD.org>
 Subject: Re: lib/56926: curses: vi screen corrupted with UTF-8 multicolumn
 	characters
 Date: Fri, 15 Jul 2022 23:38:33 +0900

 On 2022/07/15 22:50, rokuyama.rk@gmail.com wrote:
 > % ftp http://www.netbsd.org/~rin/curses_20220715/sample.txt
 > ...snip...
 > % export LC_CTYPE=ja_JP.UTF-8
 > % locale
 > LANG=""
 > LC_CTYPE="ja_JP.UTF-8"
 > LC_COLLATE="C"
 > LC_TIME="C"
 > LC_NUMERIC="C"
 > LC_MONETARY="C"
 > LC_MESSAGES="C"
 > LC_ALL=""
 > % vi sample.txt

 Oops, this example is misleading. This is on SSH session with
 unicode-aware terminal emulators.

 If you reproduce the problem on xterm, LC_CTYPE should be set
 *before* you start xterm.

 Thanks,
 rin


Responsible-Changed-From-To: lib-bug-people->blymn
Responsible-Changed-By: rin@NetBSD.org
Responsible-Changed-When: Fri, 15 Jul 2022 13:55:55 +0000
Responsible-Changed-Why:
Over to committer; can you please look into?


From: RVP <rvp@SDF.ORG>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: lib/56926: curses: vi screen corrupted with UTF-8 multicolumn
 characters
Date: Mon, 3 Oct 2022 21:47:41 +0000 (UTC)

 On Fri, 15 Jul 2022, rokuyama.rk@gmail.com wrote:

 > As a recent regression, vi(1) screen gets corrupted for files
 > including UTF-8 multicolumn characters.
 >
 > For example, edit this file with LC_CTYPE=ja_JP.UTF-8:
 >
 > http://www.netbsd.org/~rin/curses_20220715/sample.txt
 >

 Not just a messy screen, vi dumps core:

 1. Edit sample.txt
 2. :set leftright
 3. Keep inserting text in-between Kanji chars. 
 4. core-dump after the screen is shifted once or twice.

 -RVP

From: Rin Okuyama <rokuyama.rk@gmail.com>
To: gnats-bugs@netbsd.org, blymn@netbsd.org, gnats-admin@netbsd.org,
 netbsd-bugs@netbsd.org, RVP <rvp@SDF.ORG>
Cc: 
Subject: Re: lib/56926: curses: vi screen corrupted with UTF-8 multicolumn
 characters
Date: Tue, 4 Oct 2022 08:07:02 +0900

 On 2022/10/04 6:50, RVP wrote:
 >   Not just a messy screen, vi dumps core:
 >   
 >   1. Edit sample.txt
 >   2. :set leftright
 >   3. Keep inserting text in-between Kanji chars.
 >   4. core-dump after the screen is shifted once or twice.

 This is independent problem to this PR; not related to curses, but
 problem of vi itself.

 leftright has long been broken for multi-column chars. It would be
 nice if we can fix it, but codes are *REALLY* complicated.

 Thanks,
 rin

From: RVP <rvp@SDF.ORG>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: lib/56926: curses: vi screen corrupted with UTF-8 multicolumn
 characters
Date: Mon, 3 Oct 2022 23:46:10 +0000 (UTC)

 On Mon, 3 Oct 2022, Rin Okuyama wrote:

 > This is independent problem to this PR; not related to curses, but
 > problem of vi itself.
 >
 > leftright has long been broken for multi-column chars. It would be
 > nice if we can fix it, but codes are *REALLY* complicated.
 >

 You're right. I had tested the latest nvi2 source against ncurses and
 didn't see the problem. But, the nvi2 sources in -HEAD compiled with
 ncurses also dumps core. So it's an nvi2 issue, but, seems to be fixed
 (at least, the lr-scrolling issue is) in the latest sources on Github.

 The messed up screen (with BSD curses) remains, and also now a new
 coredump issue when the program exits.

 -RVP

From: Rin Okuyama <rokuyama.rk@gmail.com>
To: gnats-bugs@netbsd.org, blymn@netbsd.org, gnats-admin@netbsd.org,
 netbsd-bugs@netbsd.org, RVP <rvp@SDF.ORG>
Cc: 
Subject: Re: lib/56926: curses: vi screen corrupted with UTF-8 multicolumn
 characters
Date: Tue, 4 Oct 2022 09:50:04 +0900

 On 2022/10/04 8:50, RVP wrote:
 >   You're right. I had tested the latest nvi2 source against ncurses and
 >   didn't see the problem. But, the nvi2 sources in -HEAD compiled with
 >   ncurses also dumps core. So it's an nvi2 issue, but, seems to be fixed
 >   (at least, the lr-scrolling issue is) in the latest sources on Github.

 Also note that we do *not* use nvi2. We use our fork of nvi 1.81.6.
 nvi2 has been forked after our fork, mainly for FreeBSD.

 M17N support has been improved independently for nvi2 and our nvi
 (mostly done by tnozaki@ for ours).

 Thanks,
 rin

From: Brett Lymn <blymn@internode.on.net>
To: gnats-bugs@netbsd.org
Cc: blymn@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
        rokuyama.rk@gmail.com
Subject: Re: lib/56926: curses: vi screen corrupted with UTF-8 multicolumn
 characters
Date: Wed, 5 Oct 2022 07:50:33 +1030

 On Mon, Oct 03, 2022 at 11:50:02PM +0000, RVP wrote:
 >  
 >  The messed up screen (with BSD curses) remains, and also now a new
 >  coredump issue when the program exits.
 >  

 I am progressing with the fixes for this - there are/were multiple
 overlapping bugs that cause the problem, mostly in the refresh code
 which is a nasty ball of hair.  I think I have all the positioning
 errors sorted but there is still a clearing efficiency issue.

 For the coredump issue please raise another PR, it isn't good to
 conflate a couple of issues into a single PR.

 -- 
 Brett Lymn
 --
 Sent from my NetBSD device.

 "We are were wolves",
 "You mean werewolves?",
 "No we were wolves, now we are something else entirely",
 "Oh"

State-Changed-From-To: open->closed
State-Changed-By: blymn@NetBSD.org
State-Changed-When: Wed, 19 Oct 2022 21:21:15 +0000
State-Changed-Why:
Fix has been committed for this bug.


From: Rin Okuyama <rokuyama.rk@gmail.com>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@NetBSD.org>
Cc: 
Subject: Re: lib/56926 (curses: vi screen corrupted with UTF-8 multicolumn
 characters)
Date: Fri, 21 Oct 2022 14:54:24 +0900

 -------- Forwarded Message --------
 Subject: CVS commit: src/lib/libcurses
 Date: Wed, 19 Oct 2022 06:09:27 +0000
 From: Brett Lymn <blymn@netbsd.org>
 Reply-To: source-changes-d@NetBSD.org
 To: source-changes-full@NetBSD.org

 Module Name:	src
 Committed By:	blymn
 Date:		Wed Oct 19 06:09:27 UTC 2022

 Modified Files:
 	src/lib/libcurses: add_wchstr.c addbytes.c background.c chgat.c
 	    clrtobot.c color.c cr_put.c curses.c delch.c erase.c ins_wch.c
 	    ins_wstr.c insch.c insdelln.c insstr.c refresh.c resize.c

 Log Message:
 Fixes for lib/56926 amongst other things:
   - plod now correctly accounts for wide characters when plodding
   - use erase line when in color mode if the terminal has the capability
   - ensure that the CA_CONTINUATION flag is applied consistently to the
     subsequent characters in a wide character.
   - fix a bunch of refresh bugs that caused inconsistent placement of
     wide characters.


 To generate a diff of this commit:
 cvs rdiff -u -r1.13 -r1.14 src/lib/libcurses/add_wchstr.c
 cvs rdiff -u -r1.64 -r1.65 src/lib/libcurses/addbytes.c
 cvs rdiff -u -r1.32 -r1.33 src/lib/libcurses/background.c
 cvs rdiff -u -r1.8 -r1.9 src/lib/libcurses/chgat.c
 cvs rdiff -u -r1.30 -r1.31 src/lib/libcurses/clrtobot.c \
      src/lib/libcurses/curses.c
 cvs rdiff -u -r1.46 -r1.47 src/lib/libcurses/color.c
 cvs rdiff -u -r1.39 -r1.40 src/lib/libcurses/cr_put.c
 cvs rdiff -u -r1.29 -r1.30 src/lib/libcurses/delch.c
 cvs rdiff -u -r1.36 -r1.37 src/lib/libcurses/erase.c
 cvs rdiff -u -r1.19 -r1.20 src/lib/libcurses/ins_wch.c
 cvs rdiff -u -r1.23 -r1.24 src/lib/libcurses/ins_wstr.c
 cvs rdiff -u -r1.27 -r1.28 src/lib/libcurses/insch.c
 cvs rdiff -u -r1.22 -r1.23 src/lib/libcurses/insdelln.c
 cvs rdiff -u -r1.10 -r1.11 src/lib/libcurses/insstr.c
 cvs rdiff -u -r1.123 -r1.124 src/lib/libcurses/refresh.c
 cvs rdiff -u -r1.35 -r1.36 src/lib/libcurses/resize.c

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.


From: Rin Okuyama <rokuyama.rk@gmail.com>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@NetBSD.org>
Cc: 
Subject: Re: lib/56926 (curses: vi screen corrupted with UTF-8 multicolumn
 characters)
Date: Fri, 21 Oct 2022 14:54:57 +0900

 -------- Forwarded Message --------
 Subject: CVS commit: src/tests/lib/libcurses
 Date: Wed, 19 Oct 2022 06:10:44 +0000
 From: Brett Lymn <blymn@netbsd.org>
 Reply-To: source-changes-d@NetBSD.org
 To: source-changes-full@NetBSD.org

 Module Name:	src
 Committed By:	blymn
 Date:		Wed Oct 19 06:10:44 UTC 2022

 Modified Files:
 	src/tests/lib/libcurses/check_files: add_wch3.chk copywin10.chk
 	    delch2.chk delch3.chk delch6.chk delch7.chk get_wstr.chk
 	    getn_wstr.chk ins_wch2.chk ins_wch3.chk redrawwin2.chk
 	    touchline2.chk wcolor_set.chk wget_wstr.chk wgetn_wstr.chk
 	    wins_wch2.chk wins_wch3.chk wins_wstr4.chk wvline_set.chk
 	src/tests/lib/libcurses/tests: ins_wch

 Log Message:
 Updates to account for changes resulting from fixes to libcurses.


 To generate a diff of this commit:
 cvs rdiff -u -r1.2 -r1.3 src/tests/lib/libcurses/check_files/add_wch3.chk \
      src/tests/lib/libcurses/check_files/delch3.chk \
      src/tests/lib/libcurses/check_files/get_wstr.chk \
      src/tests/lib/libcurses/check_files/getn_wstr.chk \
      src/tests/lib/libcurses/check_files/ins_wch2.chk \
      src/tests/lib/libcurses/check_files/ins_wch3.chk \
      src/tests/lib/libcurses/check_files/wcolor_set.chk \
      src/tests/lib/libcurses/check_files/wget_wstr.chk \
      src/tests/lib/libcurses/check_files/wgetn_wstr.chk \
      src/tests/lib/libcurses/check_files/wins_wch2.chk \
      src/tests/lib/libcurses/check_files/wvline_set.chk
 cvs rdiff -u -r1.3 -r1.4 src/tests/lib/libcurses/check_files/copywin10.chk \
      src/tests/lib/libcurses/check_files/wins_wch3.chk
 cvs rdiff -u -r1.1 -r1.2 src/tests/lib/libcurses/check_files/delch2.chk \
      src/tests/lib/libcurses/check_files/delch6.chk \
      src/tests/lib/libcurses/check_files/delch7.chk \
      src/tests/lib/libcurses/check_files/redrawwin2.chk \
      src/tests/lib/libcurses/check_files/touchline2.chk \
      src/tests/lib/libcurses/check_files/wins_wstr4.chk
 cvs rdiff -u -r1.2 -r1.3 src/tests/lib/libcurses/tests/ins_wch

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.


>Unformatted:

Home	PR Database Search