NetBSD Problem Report #56926
From www@netbsd.org Fri Jul 15 13:49:25 2022
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 151E81A923B
for <gnats-bugs@gnats.NetBSD.org>; Fri, 15 Jul 2022 13:49:25 +0000 (UTC)
Message-Id: <20220715134922.54C1E1A923C@mollari.NetBSD.org>
Date: Fri, 15 Jul 2022 13:49:22 +0000 (UTC)
From: rokuyama.rk@gmail.com
Reply-To: rokuyama.rk@gmail.com
To: gnats-bugs@NetBSD.org
Subject: curses: vi screen corrupted with UTF-8 multicolumn characters
X-Send-Pr-Version: www-1.0
>Number: 56926
>Category: lib
>Synopsis: curses: vi screen corrupted with UTF-8 multicolumn characters
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: blymn
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Fri Jul 15 13:50:00 +0000 2022
>Closed-Date: Wed Oct 19 21:21:15 +0000 2022
>Last-Modified: Fri Oct 21 06:00:02 +0000 2022
>Originator: Rin Okuyama
>Release: 9.99.97 as well as 9.99.98
>Organization:
Department of Physics, Meiji University
>Environment:
NetBSD rp64 9.99.97 NetBSD 9.99.97 (GENERIC64EB) #0: Mon Jun 6 12:27:47 JST 2022 rin@latipes:/build/src/sys/arch/evbarm/compile/GENERIC64EB evbarm
>Description:
As a recent regression, vi(1) screen gets corrupted for files
including UTF-8 multicolumn characters.
For example, edit this file with LC_CTYPE=ja_JP.UTF-8:
http://www.netbsd.org/~rin/curses_20220715/sample.txt
----
% ftp http://www.netbsd.org/~rin/curses_20220715/sample.txt
...snip...
% export LC_CTYPE=ja_JP.UTF-8
% locale
LANG=""
LC_CTYPE="ja_JP.UTF-8"
LC_COLLATE="C"
LC_TIME="C"
LC_NUMERIC="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_ALL=""
% vi sample.txt
----
Then, screen gets corrupted when cursor moves on multicolumn characters,
as observed in this movie:
http://www.netbsd.org/~rin/curses_20220715/vi.mp4
This occurs for all terminal emulators as far as I can see:
- xterm
- pkgsrc/x11/mlterm
- iTerm2 (for macOS)
By bisectioning, it turned out the regression starts with this commit:
http://www.nerv.org/netbsd/?q=id:20220125T030506Z.bcfcf004c0eda8a39471f720b258282e7df9f026
I've found two typos in addbytes.c:
----
Index: addbytes.c
===================================================================
RCS file: /home/netbsd/src/lib/libcurses/addbytes.c,v
retrieving revision 1.64
diff -p -u -r1.64 addbytes.c
--- addbytes.c 19 May 2022 07:41:26 -0000 1.64
+++ addbytes.c 15 Jul 2022 13:39:10 -0000
@@ -301,12 +301,12 @@ _cursesi_addbyte(WINDOW *win, __LINE **l
*(*lp)->lastchp - win->ch_off);
if (win->bch != ' ' && c == ' ') {
(*lp)->line[*x].ch = win->bch;
-#ifdef HAVE_CHAR
+#ifdef HAVE_WCHAR
(*lp)->line[*x].wcols = win->wcols;
#endif
} else {
(*lp)->line[*x].ch = c;
-#ifdef HAVE_CHAR
+#ifdef HAVE_WCHAR
(*lp)->line[*x].wcols = wcols;
#endif
}
----
But this patch does not improve the situation.
>How-To-Repeat:
Described above.
>Fix:
N/A
>Release-Note:
>Audit-Trail:
From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: lib/56926: curses: vi screen corrupted with UTF-8 multicolumn
characters
Date: Thu, 6 Oct 2022 22:42:11 +0000
(this message didn't get sent to gnats originally)
------
From: Rin Okuyama <rokuyama.rk@gmail.com>
To: lib-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Cc: Brett Lymn <blymn@NetBSD.org>
Subject: Re: lib/56926: curses: vi screen corrupted with UTF-8 multicolumn
characters
Date: Fri, 15 Jul 2022 23:38:33 +0900
On 2022/07/15 22:50, rokuyama.rk@gmail.com wrote:
> % ftp http://www.netbsd.org/~rin/curses_20220715/sample.txt
> ...snip...
> % export LC_CTYPE=ja_JP.UTF-8
> % locale
> LANG=""
> LC_CTYPE="ja_JP.UTF-8"
> LC_COLLATE="C"
> LC_TIME="C"
> LC_NUMERIC="C"
> LC_MONETARY="C"
> LC_MESSAGES="C"
> LC_ALL=""
> % vi sample.txt
Oops, this example is misleading. This is on SSH session with
unicode-aware terminal emulators.
If you reproduce the problem on xterm, LC_CTYPE should be set
*before* you start xterm.
Thanks,
rin
Responsible-Changed-From-To: lib-bug-people->blymn
Responsible-Changed-By: rin@NetBSD.org
Responsible-Changed-When: Fri, 15 Jul 2022 13:55:55 +0000
Responsible-Changed-Why:
Over to committer; can you please look into?
From: RVP <rvp@SDF.ORG>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: lib/56926: curses: vi screen corrupted with UTF-8 multicolumn
characters
Date: Mon, 3 Oct 2022 21:47:41 +0000 (UTC)
On Fri, 15 Jul 2022, rokuyama.rk@gmail.com wrote:
> As a recent regression, vi(1) screen gets corrupted for files
> including UTF-8 multicolumn characters.
>
> For example, edit this file with LC_CTYPE=ja_JP.UTF-8:
>
> http://www.netbsd.org/~rin/curses_20220715/sample.txt
>
Not just a messy screen, vi dumps core:
1. Edit sample.txt
2. :set leftright
3. Keep inserting text in-between Kanji chars.
4. core-dump after the screen is shifted once or twice.
-RVP
From: Rin Okuyama <rokuyama.rk@gmail.com>
To: gnats-bugs@netbsd.org, blymn@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, RVP <rvp@SDF.ORG>
Cc:
Subject: Re: lib/56926: curses: vi screen corrupted with UTF-8 multicolumn
characters
Date: Tue, 4 Oct 2022 08:07:02 +0900
On 2022/10/04 6:50, RVP wrote:
> Not just a messy screen, vi dumps core:
>
> 1. Edit sample.txt
> 2. :set leftright
> 3. Keep inserting text in-between Kanji chars.
> 4. core-dump after the screen is shifted once or twice.
This is independent problem to this PR; not related to curses, but
problem of vi itself.
leftright has long been broken for multi-column chars. It would be
nice if we can fix it, but codes are *REALLY* complicated.
Thanks,
rin
From: RVP <rvp@SDF.ORG>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: lib/56926: curses: vi screen corrupted with UTF-8 multicolumn
characters
Date: Mon, 3 Oct 2022 23:46:10 +0000 (UTC)
On Mon, 3 Oct 2022, Rin Okuyama wrote:
> This is independent problem to this PR; not related to curses, but
> problem of vi itself.
>
> leftright has long been broken for multi-column chars. It would be
> nice if we can fix it, but codes are *REALLY* complicated.
>
You're right. I had tested the latest nvi2 source against ncurses and
didn't see the problem. But, the nvi2 sources in -HEAD compiled with
ncurses also dumps core. So it's an nvi2 issue, but, seems to be fixed
(at least, the lr-scrolling issue is) in the latest sources on Github.
The messed up screen (with BSD curses) remains, and also now a new
coredump issue when the program exits.
-RVP
From: Rin Okuyama <rokuyama.rk@gmail.com>
To: gnats-bugs@netbsd.org, blymn@netbsd.org, gnats-admin@netbsd.org,
netbsd-bugs@netbsd.org, RVP <rvp@SDF.ORG>
Cc:
Subject: Re: lib/56926: curses: vi screen corrupted with UTF-8 multicolumn
characters
Date: Tue, 4 Oct 2022 09:50:04 +0900
On 2022/10/04 8:50, RVP wrote:
> You're right. I had tested the latest nvi2 source against ncurses and
> didn't see the problem. But, the nvi2 sources in -HEAD compiled with
> ncurses also dumps core. So it's an nvi2 issue, but, seems to be fixed
> (at least, the lr-scrolling issue is) in the latest sources on Github.
Also note that we do *not* use nvi2. We use our fork of nvi 1.81.6.
nvi2 has been forked after our fork, mainly for FreeBSD.
M17N support has been improved independently for nvi2 and our nvi
(mostly done by tnozaki@ for ours).
Thanks,
rin
From: Brett Lymn <blymn@internode.on.net>
To: gnats-bugs@netbsd.org
Cc: blymn@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
rokuyama.rk@gmail.com
Subject: Re: lib/56926: curses: vi screen corrupted with UTF-8 multicolumn
characters
Date: Wed, 5 Oct 2022 07:50:33 +1030
On Mon, Oct 03, 2022 at 11:50:02PM +0000, RVP wrote:
>
> The messed up screen (with BSD curses) remains, and also now a new
> coredump issue when the program exits.
>
I am progressing with the fixes for this - there are/were multiple
overlapping bugs that cause the problem, mostly in the refresh code
which is a nasty ball of hair. I think I have all the positioning
errors sorted but there is still a clearing efficiency issue.
For the coredump issue please raise another PR, it isn't good to
conflate a couple of issues into a single PR.
--
Brett Lymn
--
Sent from my NetBSD device.
"We are were wolves",
"You mean werewolves?",
"No we were wolves, now we are something else entirely",
"Oh"
State-Changed-From-To: open->closed
State-Changed-By: blymn@NetBSD.org
State-Changed-When: Wed, 19 Oct 2022 21:21:15 +0000
State-Changed-Why:
Fix has been committed for this bug.
From: Rin Okuyama <rokuyama.rk@gmail.com>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@NetBSD.org>
Cc:
Subject: Re: lib/56926 (curses: vi screen corrupted with UTF-8 multicolumn
characters)
Date: Fri, 21 Oct 2022 14:54:24 +0900
-------- Forwarded Message --------
Subject: CVS commit: src/lib/libcurses
Date: Wed, 19 Oct 2022 06:09:27 +0000
From: Brett Lymn <blymn@netbsd.org>
Reply-To: source-changes-d@NetBSD.org
To: source-changes-full@NetBSD.org
Module Name: src
Committed By: blymn
Date: Wed Oct 19 06:09:27 UTC 2022
Modified Files:
src/lib/libcurses: add_wchstr.c addbytes.c background.c chgat.c
clrtobot.c color.c cr_put.c curses.c delch.c erase.c ins_wch.c
ins_wstr.c insch.c insdelln.c insstr.c refresh.c resize.c
Log Message:
Fixes for lib/56926 amongst other things:
- plod now correctly accounts for wide characters when plodding
- use erase line when in color mode if the terminal has the capability
- ensure that the CA_CONTINUATION flag is applied consistently to the
subsequent characters in a wide character.
- fix a bunch of refresh bugs that caused inconsistent placement of
wide characters.
To generate a diff of this commit:
cvs rdiff -u -r1.13 -r1.14 src/lib/libcurses/add_wchstr.c
cvs rdiff -u -r1.64 -r1.65 src/lib/libcurses/addbytes.c
cvs rdiff -u -r1.32 -r1.33 src/lib/libcurses/background.c
cvs rdiff -u -r1.8 -r1.9 src/lib/libcurses/chgat.c
cvs rdiff -u -r1.30 -r1.31 src/lib/libcurses/clrtobot.c \
src/lib/libcurses/curses.c
cvs rdiff -u -r1.46 -r1.47 src/lib/libcurses/color.c
cvs rdiff -u -r1.39 -r1.40 src/lib/libcurses/cr_put.c
cvs rdiff -u -r1.29 -r1.30 src/lib/libcurses/delch.c
cvs rdiff -u -r1.36 -r1.37 src/lib/libcurses/erase.c
cvs rdiff -u -r1.19 -r1.20 src/lib/libcurses/ins_wch.c
cvs rdiff -u -r1.23 -r1.24 src/lib/libcurses/ins_wstr.c
cvs rdiff -u -r1.27 -r1.28 src/lib/libcurses/insch.c
cvs rdiff -u -r1.22 -r1.23 src/lib/libcurses/insdelln.c
cvs rdiff -u -r1.10 -r1.11 src/lib/libcurses/insstr.c
cvs rdiff -u -r1.123 -r1.124 src/lib/libcurses/refresh.c
cvs rdiff -u -r1.35 -r1.36 src/lib/libcurses/resize.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: Rin Okuyama <rokuyama.rk@gmail.com>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@NetBSD.org>
Cc:
Subject: Re: lib/56926 (curses: vi screen corrupted with UTF-8 multicolumn
characters)
Date: Fri, 21 Oct 2022 14:54:57 +0900
-------- Forwarded Message --------
Subject: CVS commit: src/tests/lib/libcurses
Date: Wed, 19 Oct 2022 06:10:44 +0000
From: Brett Lymn <blymn@netbsd.org>
Reply-To: source-changes-d@NetBSD.org
To: source-changes-full@NetBSD.org
Module Name: src
Committed By: blymn
Date: Wed Oct 19 06:10:44 UTC 2022
Modified Files:
src/tests/lib/libcurses/check_files: add_wch3.chk copywin10.chk
delch2.chk delch3.chk delch6.chk delch7.chk get_wstr.chk
getn_wstr.chk ins_wch2.chk ins_wch3.chk redrawwin2.chk
touchline2.chk wcolor_set.chk wget_wstr.chk wgetn_wstr.chk
wins_wch2.chk wins_wch3.chk wins_wstr4.chk wvline_set.chk
src/tests/lib/libcurses/tests: ins_wch
Log Message:
Updates to account for changes resulting from fixes to libcurses.
To generate a diff of this commit:
cvs rdiff -u -r1.2 -r1.3 src/tests/lib/libcurses/check_files/add_wch3.chk \
src/tests/lib/libcurses/check_files/delch3.chk \
src/tests/lib/libcurses/check_files/get_wstr.chk \
src/tests/lib/libcurses/check_files/getn_wstr.chk \
src/tests/lib/libcurses/check_files/ins_wch2.chk \
src/tests/lib/libcurses/check_files/ins_wch3.chk \
src/tests/lib/libcurses/check_files/wcolor_set.chk \
src/tests/lib/libcurses/check_files/wget_wstr.chk \
src/tests/lib/libcurses/check_files/wgetn_wstr.chk \
src/tests/lib/libcurses/check_files/wins_wch2.chk \
src/tests/lib/libcurses/check_files/wvline_set.chk
cvs rdiff -u -r1.3 -r1.4 src/tests/lib/libcurses/check_files/copywin10.chk \
src/tests/lib/libcurses/check_files/wins_wch3.chk
cvs rdiff -u -r1.1 -r1.2 src/tests/lib/libcurses/check_files/delch2.chk \
src/tests/lib/libcurses/check_files/delch6.chk \
src/tests/lib/libcurses/check_files/delch7.chk \
src/tests/lib/libcurses/check_files/redrawwin2.chk \
src/tests/lib/libcurses/check_files/touchline2.chk \
src/tests/lib/libcurses/check_files/wins_wstr4.chk
cvs rdiff -u -r1.2 -r1.3 src/tests/lib/libcurses/tests/ins_wch
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.47 2022/09/11 19:34:41 kim Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2022
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.