NetBSD Problem Report #55220

From wiz@yt.nih.at  Thu Apr 30 22:00:26 2020
Return-Path: <wiz@yt.nih.at>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id A582E1A9217
	for <gnats-bugs@gnats.NetBSD.org>; Thu, 30 Apr 2020 22:00:26 +0000 (UTC)
Message-Id: <20200430203403.B50AF1CB6882@yt.nih.at>
Date: Thu, 30 Apr 2020 22:34:03 +0200 (CEST)
From: Thomas Klausner <wiz@NetBSD.org>
Reply-To: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@NetBSD.org
Cc: kusumi.tomohiro@gmail.com, Roy Marples <roy@marples.name>
Subject: Possible color problem in curses and/or py-curses
X-Send-Pr-Version: 3.95

>Number:         55220
>Category:       lib
>Synopsis:       Possible color problem in curses and/or py-curses
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    blymn
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Apr 30 22:05:00 +0000 2020
>Closed-Date:    Wed Apr 13 22:07:50 +0000 2022
>Last-Modified:  Wed Apr 13 22:07:50 +0000 2022
>Originator:     Thomas Klausner
>Release:        NetBSD 9.99.59
>Organization:
Curiosity is the very basis of education and if you tell me that
curiosity killed the cat, I say only that the cat died nobly.
- Arnold Edinborough
>Environment:


Architecture: x86_64
Machine: amd64
>Description:
editors/fileobj is a vi for binaries.

When starting it up on a new file (e.g., "fileobj /bin/ls"), it
starts in file position zero and only that cell is colored (black
on green). When moving around, cells that are entered are coloured
blue, lightgrey, or green depending on their value.

I understand that the color should be there from the beginning. tkusumi,
please correct me if I'm wrong.

It also comes with a test mode:
fileobj --test_screen
This prints, among others:

│This should look reversed.
|This may or may not look reversed.
|This should be in black,green.

All of these lines are just white-on-black (i.e. reversed) when I start this on an xterm
or in tmux configured with screen-256color.

I'll attach a screenshot later.
So the "black,green" is just black and white.
>How-To-Repeat:
See above.
>Fix:
Yes, please.

>Release-Note:

>Audit-Trail:
From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Fri, 1 May 2020 08:50:19 +0200

 Here's the screenshot of what I see on 9.99.59:

 https://www.netbsd.org/~wiz/fileobj-0.7.97-test-screen-netbsd.png

 Here's tkusumi's example, taken on NetBSD 9.0:

 https://leaf.dragonflybsd.org/~tkusumi/diff/netbsd/fileobj-0.7.97-test-screen.png

  Thomas

Responsible-Changed-From-To: lib-bug-people->blymn
Responsible-Changed-By: blymn@NetBSD.org
Responsible-Changed-When: Mon, 25 Oct 2021 20:57:43 +0000
Responsible-Changed-Why:
I will claim this.


From: Brett Lymn <blymn@internode.on.net>
To: gnats-bugs@netbsd.org
Cc: lib-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
        Thomas Klausner <wiz@NetBSD.org>
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Tue, 26 Oct 2021 07:35:09 +1030

 On Fri, May 01, 2020 at 06:55:02AM +0000, Thomas Klausner wrote:
 > 
 >  Here's the screenshot of what I see on 9.99.59:
 >  
 >  https://www.netbsd.org/~wiz/fileobj-0.7.97-test-screen-netbsd.png
 >  

 There seems to be a permission error on that file, I cannot access it.

 Regardless, I have checked this on -current and fileobj --test_screen
 appears correct to me, the colours/highlights match the word
 descriptions in all cases.

 Can you check again on -current just in case this has been fixed in the
 interim?

 -- 
 Brett Lymn
 --
 Sent from my NetBSD device.

 "We are were wolves",
 "You mean werewolves?",
 "No we were wolves, now we are something else entirely",
 "Oh"

From: Thomas Klausner <wiz@NetBSD.org>
To: Brett Lymn <blymn@internode.on.net>
Cc: NetBSD bugtracking <gnats-bugs@NetBSD.org>
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Tue, 26 Oct 2021 11:09:14 +0200

 On Tue, Oct 26, 2021 at 07:35:09AM +1030, Brett Lymn wrote:
 > On Fri, May 01, 2020 at 06:55:02AM +0000, Thomas Klausner wrote:
 > > 
 > >  Here's the screenshot of what I see on 9.99.59:
 > >  
 > >  https://www.netbsd.org/~wiz/fileobj-0.7.97-test-screen-netbsd.png
 > >  
 > 
 > There seems to be a permission error on that file, I cannot access it.

 Sorry, I recently cleaned up the directory and had forgotten that the
 file was referenced in a PR.

 > Regardless, I have checked this on -current and fileobj --test_screen
 > appears correct to me, the colours/highlights match the word
 > descriptions in all cases.

 Yes, --test_screen works fine now.

 When I start fileobj on /bin/ls, the output looks ok too. Until I move the cursor.

 The position where I'm at is reverse (in my case, gray on black). When
 I move the cursor, the new byte under the cursor position gets that
 color scheme, but the old one keeps it (it stays gray on black). I
 don't think this is intended behaviour.

 Can you please check that?

 Thank you,
  Thomas


From: Brett Lymn <blymn@internode.on.net>
To: gnats-bugs@netbsd.org
Cc: blymn@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
        Thomas Klausner <wiz@NetBSD.org>
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Fri, 3 Dec 2021 07:51:43 +1030

 On Tue, Oct 26, 2021 at 09:10:01AM +0000, Thomas Klausner wrote:
 >  
 >  The position where I'm at is reverse (in my case, gray on black). When
 >  I move the cursor, the new byte under the cursor position gets that
 >  color scheme, but the old one keeps it (it stays gray on black). I
 >  don't think this is intended behaviour.
 >  
 >  Can you please check that?
 >  

 I agree, I don't think this is intended but I don't think it is a curses
 problem.  If I run this test code:

 #!/usr/pkg/bin/python3.8
 #
 import curses

 stdscr = curses.initscr()
 curses.start_color()

 win = curses.newwin(20, 80, 0, 0)

 win.addstr(0, 0, "Current testing", curses.A_UNDERLINE)
 win.getch()

 curses.init_pair(1, curses.COLOR_RED, curses.COLOR_WHITE)
 curses.init_pair(2, curses.COLOR_GREEN, curses.COLOR_WHITE)
 win.addstr(1, 0, "Color testing", curses.color_pair(1))
 win.getch()

 win.chgat(1, 3, 4, curses.color_pair(2))
 win.getch()

 curses.endwin()

 Then the win.chgat call results in the 4 characters are set to a black
 background with white writing (default colour pair) instead of green on
 a white background.  Ramping up the debug in curses I see:

 1638479121.598015: mvwchgat: x: 3 y: 1 count: 4 attr: 0x0 color pair 0

 Which is also what I see when running debug on fileobj too - the
 attributes and colour are _always_ 0 which translates to attribute
 A_NORMAL and white text on a black background.  Having a look at the
 py-curses library code in the chgat handling there is:


     color = (short)((attr >> 8) & 0xff);
     attr = attr - (color << 8);


 This really doesn't map well to the NetBSD curses colour and attributes
 so it isn't surprising that the NetBSD curses is always getting 0's.

 So, it looks like the py-curses needs to be fixed.

 -- 
 Brett Lymn
 --
 Sent from my NetBSD device.

 "We are were wolves",
 "You mean werewolves?",
 "No we were wolves, now we are something else entirely",
 "Oh"

From: Brett Lymn <blymn@internode.on.net>
To: gnats-bugs@netbsd.org
Cc: blymn@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
        Thomas Klausner <wiz@NetBSD.org>
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Fri, 3 Dec 2021 17:12:02 +1030

 --7+wQ6I/tGyRaxYE+
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline

 On Thu, Dec 02, 2021 at 09:35:01PM +0000, Brett Lymn wrote:
 >  
 >  Which is also what I see when running debug on fileobj too - the
 >  attributes and colour are _always_ 0 which translates to attribute
 >  A_NORMAL and white text on a black background.  Having a look at the
 >  py-curses library code in the chgat handling there is:
 >  
 >  
 >      color = (short)((attr >> 8) & 0xff);
 >      attr = attr - (color << 8);
 >  
 >  
 >  This really doesn't map well to the NetBSD curses colour and attributes
 >  so it isn't surprising that the NetBSD curses is always getting 0's.
 >  

 The attached patch works with the test code in the previous message and
 improves the output in fileobj.  With the patch the colour highlighting
 works but the address fields get set to colour pair 0. I it not clear to
 me if the ncurses version of chgat ignores colour pair 0, if this is the
 case then NetBSD curses will need to emulate this too.

 -- 
 Brett Lymn
 --
 Sent from my NetBSD device.

 "We are were wolves",
 "You mean werewolves?",
 "No we were wolves, now we are something else entirely",
 "Oh"

 --7+wQ6I/tGyRaxYE+
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: attachment; filename=patch

 --- _cursesmodule.c.orig	2021-12-03 16:48:08.869573389 +1030
 +++ _cursesmodule.c	2021-12-03 16:52:32.269195559 +1030
 @@ -1077,8 +1077,8 @@
          return NULL;
      }

 -    color = (short)((attr >> 8) & 0xff);
 -    attr = attr - (color << 8);
 +    color = (short) PAIR_NUMBER(attr);
 +    attr = attr & __ATTRIBUTES;

      if (use_xy) {
          rtn = mvwchgat(self->win,y,x,num,attr,color,NULL);

 --7+wQ6I/tGyRaxYE+--

From: Thomas Klausner <wiz@NetBSD.org>
To: Brett Lymn <blymn@internode.on.net>
Cc: gnats-bugs@netbsd.org
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Sat, 4 Dec 2021 21:24:42 +0100

 Hi Brett!

 Thanks for analysing this and for the patch.

 I tried fileobj with this, and I still see some weirdness (but perhaps
 that is what you meant with the colour pair 0 question?):

 When running `fileobj /bin/ls`, the values have three colors: green,
 blue, black. The green and blue ones work fine. When I move the curses
 over one of the black one, it becomes black-on-green (fine) but when I
 move the cursor off, it becomes grey-on-black instead of the
 black-on-white it was before.

 My terminal is black-on-white by default in case it matters.

 As for the patch itself: The color part should be fine to upstream,
 since ncurses chgat(3) also documents PAIR_NUMBER. However, I can't
 find __ATTRIBUTES in the ncurses man pages nor headers. Is there a
 different way to get that?

 Cheers,
  Thomas

From: Brett Lymn <blymn@internode.on.net>
To: Thomas Klausner <wiz@NetBSD.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Mon, 6 Dec 2021 07:40:58 +1030

 On Sat, Dec 04, 2021 at 09:24:42PM +0100, Thomas Klausner wrote:
 > 
 > I tried fileobj with this, and I still see some weirdness (but perhaps
 > that is what you meant with the colour pair 0 question?):
 > 

 Yes, that is what I was talking about.  I am sure it works fine in ncurses I suspect because
 the ncurses chgat() won't update the colour if the pair number is 0.  I am sure our colour
 handling is correct because the output matches screenshots out on the internet, chgat is
 only called when the cursor moves.  I will have a look at the chgat code now the attributes
 are coming through.


 > As for the patch itself: The color part should be fine to upstream,
 > since ncurses chgat(3) also documents PAIR_NUMBER. However, I can't
 > find __ATTRIBUTES in the ncurses man pages nor headers. Is there a
 > different way to get that?
 > 

 Try A_ATTRIBUTES instead, that is defined in both NetBSD curses and ncurses.

 -- 
 Brett Lymn
 --
 Sent from my NetBSD device.

 "We are were wolves",
 "You mean werewolves?",
 "No we were wolves, now we are something else entirely",
 "Oh"

From: Thomas Klausner <wiz@NetBSD.org>
To: Brett Lymn <blymn@internode.on.net>
Cc: gnats-bugs@netbsd.org
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Mon, 6 Dec 2021 17:38:52 +0100

 On Mon, Dec 06, 2021 at 07:40:58AM +1030, Brett Lymn wrote:
 > On Sat, Dec 04, 2021 at 09:24:42PM +0100, Thomas Klausner wrote:
 > > 
 > > I tried fileobj with this, and I still see some weirdness (but perhaps
 > > that is what you meant with the colour pair 0 question?):
 > > 
 > 
 > Yes, that is what I was talking about.  I am sure it works fine in ncurses I suspect because
 > the ncurses chgat() won't update the colour if the pair number is 0.  I am sure our colour
 > handling is correct because the output matches screenshots out on the internet, chgat is
 > only called when the cursor moves.  I will have a look at the chgat code now the attributes
 > are coming through.

 Ok, thanks. Let me know if the python code needs more changes.

 > > As for the patch itself: The color part should be fine to upstream,
 > > since ncurses chgat(3) also documents PAIR_NUMBER. However, I can't
 > > find __ATTRIBUTES in the ncurses man pages nor headers. Is there a
 > > different way to get that?
 > > 
 > 
 > Try A_ATTRIBUTES instead, that is defined in both NetBSD curses and ncurses.

 Thanks! I've added that to python39 and sent it upstream.
  Thomas

From: Brett Lymn <blymn@internode.on.net>
To: Thomas Klausner <wiz@NetBSD.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Tue, 14 Dec 2021 17:04:51 +1030

 --VyBCHPmlEpLtvs2o
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline

 On Mon, Dec 06, 2021 at 05:38:52PM +0100, Thomas Klausner wrote:
 > On Mon, Dec 06, 2021 at 07:40:58AM +1030, Brett Lymn wrote:
 > > On Sat, Dec 04, 2021 at 09:24:42PM +0100, Thomas Klausner wrote:
 > > > 
 > > > I tried fileobj with this, and I still see some weirdness (but perhaps
 > > > that is what you meant with the colour pair 0 question?):
 > > > 
 > > 
 > > Yes, that is what I was talking about.  I am sure it works fine in ncurses I suspect because
 > > the ncurses chgat() won't update the colour if the pair number is 0.  I am sure our colour
 > > handling is correct because the output matches screenshots out on the internet, chgat is
 > > only called when the cursor moves.  I will have a look at the chgat code now the attributes
 > > are coming through.
 > 
 > Ok, thanks. Let me know if the python code needs more changes.
 > 

 The attached patch fixes the weirdness.  I haven't committed the fix yet
 because it seems a bit hacky to me, I am consulting with the other
 developers that work on curses to see what they think.  It looks like
 ncurses treats colour pair 0 as "special" sometimes, NetBSD curses
 doesn't do that - it will be awkward if we don't follow the ncurses
 model though.

 -- 
 Brett Lymn
 --
 Sent from my NetBSD device.

 "We are were wolves",
 "You mean werewolves?",
 "No we were wolves, now we are something else entirely",
 "Oh"

 --VyBCHPmlEpLtvs2o
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: attachment; filename="chgat.diff"

 Index: chgat.c
 ===================================================================
 RCS file: /cvsroot/src/lib/libcurses/chgat.c,v
 retrieving revision 1.8
 diff -r1.8 chgat.c
 63a64,66
 > 	__CTRACE(__CTRACE_ATTR, "mvwchgat: x: %d y: %d count: %d attr: 0x%x "
 > 	    "color pair %d\n", x, y, count, (attr & ~__COLOR),
 > 	    color);
 69,70d71
 < 	attr = (attr & ~__COLOR) | COLOR_PAIR(color);
 < 
 74,76d74
 < 	__CTRACE(__CTRACE_ATTR, "mvwchgat: x: %d y: %d count: %d attr: 0x%x "
 < 	    "color pair %d\n", x, y, count, (attr & ~__COLOR),
 < 	    PAIR_NUMBER(color));
 89c87,92
 < 		lc->attr = (lc->attr & ~WA_ATTRIBUTES) | attr;
 ---
 > 		if (color == 0)
 > 			lc->attr = (lc->attr & ~__COLOR) | __default_color
 > 				    | (lc->attr & ~WA_ATTRIBUTES) | attr;
 > 		else
 > 			lc->attr = (lc->attr & ~__COLOR) | COLOR_PAIR(color)
 > 				    | (lc->attr & ~WA_ATTRIBUTES) | attr;

 --VyBCHPmlEpLtvs2o--

From: Brett Lymn <blymn@internode.on.net>
To: Thomas Klausner <wiz@NetBSD.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Tue, 12 Apr 2022 16:59:50 +0930

 On Mon, Dec 06, 2021 at 05:38:52PM +0100, Thomas Klausner wrote:
 > On Mon, Dec 06, 2021 at 07:40:58AM +1030, Brett Lymn wrote:
 > > 
 > > Yes, that is what I was talking about.  I am sure it works fine in ncurses I suspect because
 > > the ncurses chgat() won't update the colour if the pair number is 0.  I am sure our colour
 > > handling is correct because the output matches screenshots out on the internet, chgat is
 > > only called when the cursor moves.  I will have a look at the chgat code now the attributes
 > > are coming through.
 > 
 > Ok, thanks. Let me know if the python code needs more changes.
 > 

 Right - that was more complicated than I imagined.  I needed to totally
 rework the colour code because I realised that we should be using colour
 pair 0 for the default colour (this is what fileobj was expecting), our
 default colour pair was set to the last possible pair in the set and
 there was no real way to use it.

 I have committed the fixes for the default colour and now fileobj
 appears to behave correctly.  Can you please update your libcurses and
 check?

 -- 
 Brett Lymn
 --
 Sent from my NetBSD device.

 "We are were wolves",
 "You mean werewolves?",
 "No we were wolves, now we are something else entirely",
 "Oh"

From: Thomas Klausner <wiz@NetBSD.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: lib/55220: Possible color problem in curses and/or py-curses
Date: Wed, 13 Apr 2022 23:38:13 +0200

 I played around a bit with fileobj and it all looks fine now.

 Thank you for fixing this!
  Thomas

State-Changed-From-To: open->closed
State-Changed-By: blymn@NetBSD.org
State-Changed-When: Wed, 13 Apr 2022 22:07:50 +0000
State-Changed-Why:
Confirmed fixed.


>Unformatted:

Home	PR Database Search