NetBSD Problem Report #55220
From wiz@yt.nih.at Thu Apr 30 22:00:26 2020
Return-Path: <wiz@yt.nih.at>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id A582E1A9217
for <gnats-bugs@gnats.NetBSD.org>; Thu, 30 Apr 2020 22:00:26 +0000 (UTC)
Message-Id: <20200430203403.B50AF1CB6882@yt.nih.at>
Date: Thu, 30 Apr 2020 22:34:03 +0200 (CEST)
From: Thomas Klausner <wiz@NetBSD.org>
Reply-To: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: kusumi.tomohiro@gmail.com, Roy Marples <roy@marples.name>
Subject: Possible color problem in curses and/or py-curses
X-Send-Pr-Version: 3.95
>Number: 55220
>Category: lib
>Synopsis: Possible color problem in curses and/or py-curses
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: blymn
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Thu Apr 30 22:05:00 +0000 2020
>Closed-Date: Wed Apr 13 22:07:50 +0000 2022
>Last-Modified: Wed Apr 13 22:07:50 +0000 2022
>Originator: Thomas Klausner
>Release: NetBSD 9.99.59
>Organization:
Curiosity is the very basis of education and if you tell me that
curiosity killed the cat, I say only that the cat died nobly.
- Arnold Edinborough
>Environment:
Architecture: x86_64
Machine: amd64
>Description:
editors/fileobj is a vi for binaries.
When starting it up on a new file (e.g., "fileobj /bin/ls"), it
starts in file position zero and only that cell is colored (black
on green). When moving around, cells that are entered are coloured
blue, lightgrey, or green depending on their value.
I understand that the color should be there from the beginning. tkusumi,
please correct me if I'm wrong.
It also comes with a test mode:
fileobj --test_screen
This prints, among others:
│This should look reversed.
|This may or may not look reversed.
|This should be in black,green.
All of these lines are just white-on-black (i.e. reversed) when I start this on an xterm
or in tmux configured with screen-256color.
I'll attach a screenshot later.
So the "black,green" is just black and white.
>How-To-Repeat:
See above.
>Fix:
Yes, please.
>Release-Note:
>Audit-Trail:
From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Fri, 1 May 2020 08:50:19 +0200
Here's the screenshot of what I see on 9.99.59:
https://www.netbsd.org/~wiz/fileobj-0.7.97-test-screen-netbsd.png
Here's tkusumi's example, taken on NetBSD 9.0:
https://leaf.dragonflybsd.org/~tkusumi/diff/netbsd/fileobj-0.7.97-test-screen.png
Thomas
Responsible-Changed-From-To: lib-bug-people->blymn
Responsible-Changed-By: blymn@NetBSD.org
Responsible-Changed-When: Mon, 25 Oct 2021 20:57:43 +0000
Responsible-Changed-Why:
I will claim this.
From: Brett Lymn <blymn@internode.on.net>
To: gnats-bugs@netbsd.org
Cc: lib-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
Thomas Klausner <wiz@NetBSD.org>
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Tue, 26 Oct 2021 07:35:09 +1030
On Fri, May 01, 2020 at 06:55:02AM +0000, Thomas Klausner wrote:
>
> Here's the screenshot of what I see on 9.99.59:
>
> https://www.netbsd.org/~wiz/fileobj-0.7.97-test-screen-netbsd.png
>
There seems to be a permission error on that file, I cannot access it.
Regardless, I have checked this on -current and fileobj --test_screen
appears correct to me, the colours/highlights match the word
descriptions in all cases.
Can you check again on -current just in case this has been fixed in the
interim?
--
Brett Lymn
--
Sent from my NetBSD device.
"We are were wolves",
"You mean werewolves?",
"No we were wolves, now we are something else entirely",
"Oh"
From: Thomas Klausner <wiz@NetBSD.org>
To: Brett Lymn <blymn@internode.on.net>
Cc: NetBSD bugtracking <gnats-bugs@NetBSD.org>
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Tue, 26 Oct 2021 11:09:14 +0200
On Tue, Oct 26, 2021 at 07:35:09AM +1030, Brett Lymn wrote:
> On Fri, May 01, 2020 at 06:55:02AM +0000, Thomas Klausner wrote:
> >
> > Here's the screenshot of what I see on 9.99.59:
> >
> > https://www.netbsd.org/~wiz/fileobj-0.7.97-test-screen-netbsd.png
> >
>
> There seems to be a permission error on that file, I cannot access it.
Sorry, I recently cleaned up the directory and had forgotten that the
file was referenced in a PR.
> Regardless, I have checked this on -current and fileobj --test_screen
> appears correct to me, the colours/highlights match the word
> descriptions in all cases.
Yes, --test_screen works fine now.
When I start fileobj on /bin/ls, the output looks ok too. Until I move the cursor.
The position where I'm at is reverse (in my case, gray on black). When
I move the cursor, the new byte under the cursor position gets that
color scheme, but the old one keeps it (it stays gray on black). I
don't think this is intended behaviour.
Can you please check that?
Thank you,
Thomas
From: Brett Lymn <blymn@internode.on.net>
To: gnats-bugs@netbsd.org
Cc: blymn@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
Thomas Klausner <wiz@NetBSD.org>
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Fri, 3 Dec 2021 07:51:43 +1030
On Tue, Oct 26, 2021 at 09:10:01AM +0000, Thomas Klausner wrote:
>
> The position where I'm at is reverse (in my case, gray on black). When
> I move the cursor, the new byte under the cursor position gets that
> color scheme, but the old one keeps it (it stays gray on black). I
> don't think this is intended behaviour.
>
> Can you please check that?
>
I agree, I don't think this is intended but I don't think it is a curses
problem. If I run this test code:
#!/usr/pkg/bin/python3.8
#
import curses
stdscr = curses.initscr()
curses.start_color()
win = curses.newwin(20, 80, 0, 0)
win.addstr(0, 0, "Current testing", curses.A_UNDERLINE)
win.getch()
curses.init_pair(1, curses.COLOR_RED, curses.COLOR_WHITE)
curses.init_pair(2, curses.COLOR_GREEN, curses.COLOR_WHITE)
win.addstr(1, 0, "Color testing", curses.color_pair(1))
win.getch()
win.chgat(1, 3, 4, curses.color_pair(2))
win.getch()
curses.endwin()
Then the win.chgat call results in the 4 characters are set to a black
background with white writing (default colour pair) instead of green on
a white background. Ramping up the debug in curses I see:
1638479121.598015: mvwchgat: x: 3 y: 1 count: 4 attr: 0x0 color pair 0
Which is also what I see when running debug on fileobj too - the
attributes and colour are _always_ 0 which translates to attribute
A_NORMAL and white text on a black background. Having a look at the
py-curses library code in the chgat handling there is:
color = (short)((attr >> 8) & 0xff);
attr = attr - (color << 8);
This really doesn't map well to the NetBSD curses colour and attributes
so it isn't surprising that the NetBSD curses is always getting 0's.
So, it looks like the py-curses needs to be fixed.
--
Brett Lymn
--
Sent from my NetBSD device.
"We are were wolves",
"You mean werewolves?",
"No we were wolves, now we are something else entirely",
"Oh"
From: Brett Lymn <blymn@internode.on.net>
To: gnats-bugs@netbsd.org
Cc: blymn@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
Thomas Klausner <wiz@NetBSD.org>
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Fri, 3 Dec 2021 17:12:02 +1030
--7+wQ6I/tGyRaxYE+
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
On Thu, Dec 02, 2021 at 09:35:01PM +0000, Brett Lymn wrote:
>
> Which is also what I see when running debug on fileobj too - the
> attributes and colour are _always_ 0 which translates to attribute
> A_NORMAL and white text on a black background. Having a look at the
> py-curses library code in the chgat handling there is:
>
>
> color = (short)((attr >> 8) & 0xff);
> attr = attr - (color << 8);
>
>
> This really doesn't map well to the NetBSD curses colour and attributes
> so it isn't surprising that the NetBSD curses is always getting 0's.
>
The attached patch works with the test code in the previous message and
improves the output in fileobj. With the patch the colour highlighting
works but the address fields get set to colour pair 0. I it not clear to
me if the ncurses version of chgat ignores colour pair 0, if this is the
case then NetBSD curses will need to emulate this too.
--
Brett Lymn
--
Sent from my NetBSD device.
"We are were wolves",
"You mean werewolves?",
"No we were wolves, now we are something else entirely",
"Oh"
--7+wQ6I/tGyRaxYE+
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=patch
--- _cursesmodule.c.orig 2021-12-03 16:48:08.869573389 +1030
+++ _cursesmodule.c 2021-12-03 16:52:32.269195559 +1030
@@ -1077,8 +1077,8 @@
return NULL;
}
- color = (short)((attr >> 8) & 0xff);
- attr = attr - (color << 8);
+ color = (short) PAIR_NUMBER(attr);
+ attr = attr & __ATTRIBUTES;
if (use_xy) {
rtn = mvwchgat(self->win,y,x,num,attr,color,NULL);
--7+wQ6I/tGyRaxYE+--
From: Thomas Klausner <wiz@NetBSD.org>
To: Brett Lymn <blymn@internode.on.net>
Cc: gnats-bugs@netbsd.org
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Sat, 4 Dec 2021 21:24:42 +0100
Hi Brett!
Thanks for analysing this and for the patch.
I tried fileobj with this, and I still see some weirdness (but perhaps
that is what you meant with the colour pair 0 question?):
When running `fileobj /bin/ls`, the values have three colors: green,
blue, black. The green and blue ones work fine. When I move the curses
over one of the black one, it becomes black-on-green (fine) but when I
move the cursor off, it becomes grey-on-black instead of the
black-on-white it was before.
My terminal is black-on-white by default in case it matters.
As for the patch itself: The color part should be fine to upstream,
since ncurses chgat(3) also documents PAIR_NUMBER. However, I can't
find __ATTRIBUTES in the ncurses man pages nor headers. Is there a
different way to get that?
Cheers,
Thomas
From: Brett Lymn <blymn@internode.on.net>
To: Thomas Klausner <wiz@NetBSD.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Mon, 6 Dec 2021 07:40:58 +1030
On Sat, Dec 04, 2021 at 09:24:42PM +0100, Thomas Klausner wrote:
>
> I tried fileobj with this, and I still see some weirdness (but perhaps
> that is what you meant with the colour pair 0 question?):
>
Yes, that is what I was talking about. I am sure it works fine in ncurses I suspect because
the ncurses chgat() won't update the colour if the pair number is 0. I am sure our colour
handling is correct because the output matches screenshots out on the internet, chgat is
only called when the cursor moves. I will have a look at the chgat code now the attributes
are coming through.
> As for the patch itself: The color part should be fine to upstream,
> since ncurses chgat(3) also documents PAIR_NUMBER. However, I can't
> find __ATTRIBUTES in the ncurses man pages nor headers. Is there a
> different way to get that?
>
Try A_ATTRIBUTES instead, that is defined in both NetBSD curses and ncurses.
--
Brett Lymn
--
Sent from my NetBSD device.
"We are were wolves",
"You mean werewolves?",
"No we were wolves, now we are something else entirely",
"Oh"
From: Thomas Klausner <wiz@NetBSD.org>
To: Brett Lymn <blymn@internode.on.net>
Cc: gnats-bugs@netbsd.org
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Mon, 6 Dec 2021 17:38:52 +0100
On Mon, Dec 06, 2021 at 07:40:58AM +1030, Brett Lymn wrote:
> On Sat, Dec 04, 2021 at 09:24:42PM +0100, Thomas Klausner wrote:
> >
> > I tried fileobj with this, and I still see some weirdness (but perhaps
> > that is what you meant with the colour pair 0 question?):
> >
>
> Yes, that is what I was talking about. I am sure it works fine in ncurses I suspect because
> the ncurses chgat() won't update the colour if the pair number is 0. I am sure our colour
> handling is correct because the output matches screenshots out on the internet, chgat is
> only called when the cursor moves. I will have a look at the chgat code now the attributes
> are coming through.
Ok, thanks. Let me know if the python code needs more changes.
> > As for the patch itself: The color part should be fine to upstream,
> > since ncurses chgat(3) also documents PAIR_NUMBER. However, I can't
> > find __ATTRIBUTES in the ncurses man pages nor headers. Is there a
> > different way to get that?
> >
>
> Try A_ATTRIBUTES instead, that is defined in both NetBSD curses and ncurses.
Thanks! I've added that to python39 and sent it upstream.
Thomas
From: Brett Lymn <blymn@internode.on.net>
To: Thomas Klausner <wiz@NetBSD.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Tue, 14 Dec 2021 17:04:51 +1030
--VyBCHPmlEpLtvs2o
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
On Mon, Dec 06, 2021 at 05:38:52PM +0100, Thomas Klausner wrote:
> On Mon, Dec 06, 2021 at 07:40:58AM +1030, Brett Lymn wrote:
> > On Sat, Dec 04, 2021 at 09:24:42PM +0100, Thomas Klausner wrote:
> > >
> > > I tried fileobj with this, and I still see some weirdness (but perhaps
> > > that is what you meant with the colour pair 0 question?):
> > >
> >
> > Yes, that is what I was talking about. I am sure it works fine in ncurses I suspect because
> > the ncurses chgat() won't update the colour if the pair number is 0. I am sure our colour
> > handling is correct because the output matches screenshots out on the internet, chgat is
> > only called when the cursor moves. I will have a look at the chgat code now the attributes
> > are coming through.
>
> Ok, thanks. Let me know if the python code needs more changes.
>
The attached patch fixes the weirdness. I haven't committed the fix yet
because it seems a bit hacky to me, I am consulting with the other
developers that work on curses to see what they think. It looks like
ncurses treats colour pair 0 as "special" sometimes, NetBSD curses
doesn't do that - it will be awkward if we don't follow the ncurses
model though.
--
Brett Lymn
--
Sent from my NetBSD device.
"We are were wolves",
"You mean werewolves?",
"No we were wolves, now we are something else entirely",
"Oh"
--VyBCHPmlEpLtvs2o
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="chgat.diff"
Index: chgat.c
===================================================================
RCS file: /cvsroot/src/lib/libcurses/chgat.c,v
retrieving revision 1.8
diff -r1.8 chgat.c
63a64,66
> __CTRACE(__CTRACE_ATTR, "mvwchgat: x: %d y: %d count: %d attr: 0x%x "
> "color pair %d\n", x, y, count, (attr & ~__COLOR),
> color);
69,70d71
< attr = (attr & ~__COLOR) | COLOR_PAIR(color);
<
74,76d74
< __CTRACE(__CTRACE_ATTR, "mvwchgat: x: %d y: %d count: %d attr: 0x%x "
< "color pair %d\n", x, y, count, (attr & ~__COLOR),
< PAIR_NUMBER(color));
89c87,92
< lc->attr = (lc->attr & ~WA_ATTRIBUTES) | attr;
---
> if (color == 0)
> lc->attr = (lc->attr & ~__COLOR) | __default_color
> | (lc->attr & ~WA_ATTRIBUTES) | attr;
> else
> lc->attr = (lc->attr & ~__COLOR) | COLOR_PAIR(color)
> | (lc->attr & ~WA_ATTRIBUTES) | attr;
--VyBCHPmlEpLtvs2o--
From: Brett Lymn <blymn@internode.on.net>
To: Thomas Klausner <wiz@NetBSD.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Tue, 12 Apr 2022 16:59:50 +0930
On Mon, Dec 06, 2021 at 05:38:52PM +0100, Thomas Klausner wrote:
> On Mon, Dec 06, 2021 at 07:40:58AM +1030, Brett Lymn wrote:
> >
> > Yes, that is what I was talking about. I am sure it works fine in ncurses I suspect because
> > the ncurses chgat() won't update the colour if the pair number is 0. I am sure our colour
> > handling is correct because the output matches screenshots out on the internet, chgat is
> > only called when the cursor moves. I will have a look at the chgat code now the attributes
> > are coming through.
>
> Ok, thanks. Let me know if the python code needs more changes.
>
Right - that was more complicated than I imagined. I needed to totally
rework the colour code because I realised that we should be using colour
pair 0 for the default colour (this is what fileobj was expecting), our
default colour pair was set to the last possible pair in the set and
there was no real way to use it.
I have committed the fixes for the default colour and now fileobj
appears to behave correctly. Can you please update your libcurses and
check?
--
Brett Lymn
--
Sent from my NetBSD device.
"We are were wolves",
"You mean werewolves?",
"No we were wolves, now we are something else entirely",
"Oh"
From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Wed, 13 Apr 2022 23:38:13 +0200
I played around a bit with fileobj and it all looks fine now.
Thank you for fixing this!
Thomas
State-Changed-From-To: open->closed
State-Changed-By: blymn@NetBSD.org
State-Changed-When: Wed, 13 Apr 2022 22:07:50 +0000
State-Changed-Why:
Confirmed fixed.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.46 2020/01/03 16:35:01 leot Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2020
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.