NetBSD Problem Report #58875
From www@netbsd.org Thu Dec 5 14:43:41 2024
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
client-signature RSA-PSS (2048 bits) client-digest SHA256)
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 8AC281A9238
for <gnats-bugs@gnats.NetBSD.org>; Thu, 5 Dec 2024 14:43:41 +0000 (UTC)
Message-Id: <20241205144340.35C711A923B@mollari.NetBSD.org>
Date: Thu, 5 Dec 2024 14:43:40 +0000 (UTC)
From: campbell+netbsd@mumble.net
Reply-To: campbell+netbsd@mumble.net
To: gnats-bugs@NetBSD.org
Subject: python: terminal colours broke in some applications
X-Send-Pr-Version: www-1.0
>Number: 58875
>Category: pkg
>Synopsis: python: terminal colours broke in some applications
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: pkg-manager
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Thu Dec 05 14:45:01 +0000 2024
>Last-Modified: Fri Dec 06 06:20:01 +0000 2024
>Originator: Taylor R Campbell
>Release: netbsd-9, pkgsrc-current in the last year
>Organization:
The [0;33mNetBSD[0m Foundation
>Environment:
>Description:
Some time between when lang/python311 was at 3.11.8 and now, at 3.11.10nb3, terminal escape sequences for colours have broken, so that, for example, hg produces the following output:
$ hg amend -i
diff --git a/tests/lib/libc/sys/t_select.c b/tests/lib/libc/sys/t_select.c
1 hunks, 1 lines changed
[0;33mexamine changes to 'tests/lib/libc/sys/t_select.c'?[0m[0;33m(enter ? for help) [Ynesfdaq?][0m
(I originally filed this upstream with Mercurial at https://bz.mercurial-scm.org/show_bug.cgi?id=6930 before investigating closer today and finding that upgrading Python made the difference.)
>How-To-Repeat:
1. install devel/mercurial with recent python311 or python312 packages
2. interactively commit a change (hg commit -i, hg amend -i)
>Fix:
Yes, please!
>Release-Note:
>Audit-Trail:
From: Taylor R Campbell <riastradh@NetBSD.org>
To: gnats-bugs@NetBSD.org, pkgsrc-bugs@NetBSD.org
Cc: Aleksey Cheusov <cheusov@NetBSD.org>,
Christos Zoulas <christos@NetBSD.org>
Subject: Re: pkg/58875: python: terminal colours broke in some applications
Date: Thu, 5 Dec 2024 14:57:55 +0000
[cc committer cheusov@ and libedit wizard christos@]
Building lang/python311 with READLINE_TYPE=3Dreadline, instead of the
default READLINE_TYPE=3Deditline, fixes it. So this is probably a
regression caused by:
https://mail-index.netbsd.org/pkgsrc-changes/2024/04/30/msg298643.html
(and/or possibly caused by a bug in libedit which is yet to be
diagnosed)
Module Name: pkgsrc
Committed By: cheusov
Date: Tue Apr 30 17:01:15 UTC 2024
Modified Files:
pkgsrc/lang/python311: Makefile options.mk
Log Message:
Add option "readline" enabled by default.
Either --with-readline=3Deditline or --with-readline are added
to CONFIGURE_ARGS depending on READLINE_TYPE.
To generate a diff of this commit:
cvs rdiff -u -r1.25 -r1.26 pkgsrc/lang/python311/Makefile
cvs rdiff -u -r1.1 -r1.2 pkgsrc/lang/python311/options.mk
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
Modified files:
Index: pkgsrc/lang/python311/Makefile
diff -u pkgsrc/lang/python311/Makefile:1.25 pkgsrc/lang/python311/Makefile:=
1.26
--- pkgsrc/lang/python311/Makefile:1.25 Fri Feb 2 15:50:55 2024
+++ pkgsrc/lang/python311/Makefile Tue Apr 30 17:01:15 2024
@@ -1,4 +1,4 @@
-# $NetBSD: Makefile,v 1.25 2024/02/02 15:50:55 tsutsui Exp $
+# $NetBSD: Makefile,v 1.26 2024/04/30 17:01:15 cheusov Exp $
=20
.include "dist.mk"
=20
@@ -38,7 +38,6 @@ USE_CC_FEATURES+=3D c11
GNU_CONFIGURE=3D yes
CONFIGURE_ARGS+=3D --enable-shared
CONFIGURE_ARGS+=3D --with-openssl=3D${BUILDLINK_PREFIX.openssl}
-CONFIGURE_ARGS+=3D --with-readline=3Dreadline # XXX editline?
CONFIGURE_ARGS+=3D --with-system-expat
CONFIGURE_ARGS+=3D --with-system-ffi
CONFIGURE_ARGS+=3D --without-ensurepip
@@ -246,7 +245,6 @@ CONFIGURE_ENV+=3D ac_cv_lib_intl_textdoma
.endif
.include "../../devel/libffi/buildlink3.mk"
.include "../../devel/libuuid/buildlink3.mk"
-.include "../../devel/readline/buildlink3.mk"
.include "../../devel/zlib/buildlink3.mk"
.include "../../security/openssl/buildlink3.mk"
.include "../../textproc/expat/buildlink3.mk"
Index: pkgsrc/lang/python311/options.mk
diff -u pkgsrc/lang/python311/options.mk:1.1 pkgsrc/lang/python311/options.=
mk:1.2
--- pkgsrc/lang/python311/options.mk:1.1 Mon Oct 31 09:50:40 2022
+++ pkgsrc/lang/python311/options.mk Tue Apr 30 17:01:15 2024
@@ -1,13 +1,13 @@
-# $NetBSD: options.mk,v 1.1 2022/10/31 09:50:40 adam Exp $
+# $NetBSD: options.mk,v 1.2 2024/04/30 17:01:15 cheusov Exp $
=20
PKG_OPTIONS_VAR=3D PKG_OPTIONS.python311
-PKG_SUPPORTED_OPTIONS=3D dtrace pymalloc x11
-PKG_SUGGESTED_OPTIONS=3D x11
+PKG_SUPPORTED_OPTIONS=3D dtrace pymalloc x11 readline
+PKG_SUGGESTED_OPTIONS=3D x11 readline
=20
.include "../../mk/bsd.prefs.mk"
.include "../../mk/bsd.options.mk"
=20
-PLIST_VARS+=3D dtrace
+PLIST_VARS+=3D dtrace readline
=20
.if !empty(PKG_OPTIONS:Mdtrace)
CONFIGURE_ARGS+=3D --with-dtrace
@@ -39,3 +39,16 @@ CONFIGURE_ARGS+=3D --with-pymalloc
.else
CONFIGURE_ARGS+=3D --without-pymalloc
.endif
+
+# readline/editline
+.if empty(PKG_OPTIONS:Mreadline)
+CONFIGURE_ARGS+=3D --without-readline
+.else
+.include "../../mk/readline.buildlink3.mk"
+.if ${READLINE_TYPE} =3D=3D "editline"
+CONFIGURE_ARGS+=3D --with-readline=3Deditline
+.else
+CONFIGURE_ARGS+=3D --with-readline
+.endif
+PLIST.readline=3D yes
+.endif
From: Taylor R Campbell <riastradh@NetBSD.org>
To: gnats-bugs@NetBSD.org, pkgsrc-bugs@NetBSD.org
Cc: Aleksey Cheusov <cheusov@NetBSD.org>,
Christos Zoulas <christos@NetBSD.org>,
Jonathan Perkin <jperkin@mnx.io>
Subject: Re: pkg/58875: python: terminal colours broke in some applications
Date: Thu, 5 Dec 2024 15:29:52 +0000
This is a multi-part message in MIME format.
--=_SrNSPSgB67zgKSM708lGSVWnkRl5Kpmj
[cc pkgsrc-branch steward du jour jperkin}
The attached patch resolves the problem by using libreadline
unconditionally. Maybe in the future if libedit is changed to support
colours (which christos tells me it does not right now) we can put the
option back, but until then I think we should have applications work
out of the box.
OK to commit?
--=_SrNSPSgB67zgKSM708lGSVWnkRl5Kpmj
Content-Type: text/plain; charset="ISO-8859-1"; name="pr58875-pythonreadline"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment; filename="pr58875-pythonreadline.patch"
>From 8b3440b785cf4683bd91bad11ce055ce49c75897 Mon Sep 17 00:00:00 2001
From: Taylor R Campbell <riastradh@NetBSD.org>
Date: Thu, 5 Dec 2024 15:25:21 +0000
Subject: [PATCH] lang/python311, lang/python312: Use libreadline
unconditionally.
libedit breaks some applications.
PR pkg/58875: python: terminal colours broke in some applications
---
lang/python311/options.mk | 8 ++------
lang/python312/options.mk | 2 +-
2 files changed, 3 insertions(+), 7 deletions(-)
diff --git a/lang/python311/options.mk b/lang/python311/options.mk
index c057b81f38a2..4e4782124e27 100644
--- a/lang/python311/options.mk
+++ b/lang/python311/options.mk
@@ -41,12 +41,8 @@ CONFIGURE_ARGS+=3D --without-pymalloc
.endif
=20
.if ${PKG_OPTIONS:Mreadline}
-. include "../../mk/readline.buildlink3.mk"
-. if ${READLINE_TYPE} =3D=3D "editline"
-CONFIGURE_ARGS+=3D --with-readline=3Deditline
-. else
-CONFIGURE_ARGS+=3D --with-readline
-. endif
+. include "../../devel/readline/buildlink3.mk"
+CONFIGURE_ARGS+=3D --with-readline=3Dreadline
PLIST.readline=3D yes
.else
CONFIGURE_ARGS+=3D --without-readline
diff --git a/lang/python312/options.mk b/lang/python312/options.mk
index 3bc304fd8626..f09274e40f91 100644
--- a/lang/python312/options.mk
+++ b/lang/python312/options.mk
@@ -43,7 +43,7 @@ CONFIGURE_ARGS+=3D --without-pymalloc
.endif
=20
.if ${PKG_OPTIONS:Mreadline}
-. include "../../mk/readline.buildlink3.mk"
+. include "../../devel/readline/buildlink3.mk"
CONFIGURE_ARGS+=3D --with-readline=3Dreadline
PLIST.readline=3D yes
.else
--=_SrNSPSgB67zgKSM708lGSVWnkRl5Kpmj--
From: Jonathan Perkin <jperkin@mnx.io>
To: gnats-bugs@netbsd.org
Cc: pkg-manager@netbsd.org, gnats-admin@netbsd.org, pkgsrc-bugs@netbsd.org,
campbell+netbsd@mumble.net
Subject: Re: pkg/58875: python: terminal colours broke in some applications
Date: Thu, 5 Dec 2024 16:21:44 +0000
I initially said no on IRC, but on review the scope is very limited, and
the use of readline is well tested on non-NetBSD, so I'm happy for this
to go in as long as you pay close attention to bulk builds over the next
few days to ensure there is no fallout.
--
Jonathan Perkin - mnx.io - pkgsrc.smartos.org
Open Source Complete Cloud www.tritondatacenter.com
From: Christos Zoulas <christos@zoulas.com>
To: Taylor R Campbell <riastradh@netbsd.org>
Cc: gnats-bugs@netbsd.org, pkgsrc-bugs@netbsd.org, Aleksey Cheusov
<cheusov@netbsd.org>, Christos Zoulas <christos@netbsd.org>, Jonathan Perkin
<jperkin@mnx.io>
Subject: Re: pkg/58875: python: terminal colours broke in some applications
Date: Thu, 05 Dec 2024 11:36:23 -0500
On 2024-12-05 10:29 am, Taylor R Campbell wrote:
> [cc pkgsrc-branch steward du jour jperkin}
>
> The attached patch resolves the problem by using libreadline
> unconditionally. Maybe in the future if libedit is changed to support
> colours (which christos tells me it does not right now) we can put the
> option back, but until then I think we should have applications work
> out of the box.
>
> OK to commit?
I think it does support colors in the same way readline does according
to this:
https://wiki.hackzine.org/development/misc/readline-color-prompt.html
--
christos
From: Taylor R Campbell <riastradh@NetBSD.org>
To: Christos Zoulas <christos@zoulas.com>
Cc: gnats-bugs@NetBSD.org, pkgsrc-bugs@NetBSD.org,
Aleksey Cheusov <cheusov@NetBSD.org>,
Christos Zoulas <christos@NetBSD.org>, Jonathan Perkin <jperkin@mnx.io>
Subject: Re: pkg/58875: python: terminal colours broke in some applications
Date: Thu, 5 Dec 2024 17:44:37 +0000
> Date: Thu, 05 Dec 2024 11:36:23 -0500
> From: Christos Zoulas <christos@zoulas.com>
>=20
> On 2024-12-05 10:29 am, Taylor R Campbell wrote:
> > The attached patch resolves the problem by using libreadline
> > unconditionally. Maybe in the future if libedit is changed to support
> > colours (which christos tells me it does not right now) we can put the
> > option back, but until then I think we should have applications work
> > out of the box.
>=20
> I think it does support colors in the same way readline does according=20
> to this:
> https://wiki.hackzine.org/development/misc/readline-color-prompt.html
Interesting. If I patch Mercurial to surround the prompt by \001 and
\002, the prompt comes out just fine with libedit and libreadline:
diff -r 47eacd67ce16 mercurial/ui.py
--- a/mercurial/ui.py Thu Dec 05 13:48:22 2024 +0000
+++ b/mercurial/ui.py Thu Dec 05 17:18:57 2024 +0000
@@ -1745,7 +1745,7 @@
self.flush()
prompt =3D b' '
else:
- prompt =3D self.label(prompt, b'ui.prompt') + b' '
+ prompt =3D b'\001' + self.label(prompt, b'ui.prompt') + b'\002=
' + b' '
=20
# prompt ' ' must exist; otherwise readline may delete entire line
# - http://bugs.python.org/issue12833
But these are not required by libreadline.
It looks like the difference is that libedit simply discards wide
characters with wcwidth <=3D 0, like ESC:
85 w =3D wcwidth(end[1]); /* column width of the visible char */
86 *wp =3D (int)w;
87=20
88 if (w <=3D 0) /* we require something to be printed */
89 return 0;
...
100 for (n =3D 0, i =3D 0; i < len; i++)
101 n +=3D ct_encode_char(b + n, (size_t)(w - n), buf[i]);
102 n +=3D ct_encode_char(b + n, (size_t)(w - n), end[1]);
103 b[n] =3D '\0';
https://nxr.netbsd.org/xref/src/lib/libedit/literal.c?r=3D1.5#85
In contrast, I think libreadline preserves them, and just calculates
the column widths differently. (The logic in
https://git.savannah.gnu.org/cgit/readline.git/tree/display.c?h=3Dreadline-=
8.2&id=3Df7a382fd09319b20ef4435b9b554183b605468c1
is a lot more complicated, though, and I haven't stepped through it
carefully.)
On the one hand, it might be appropriate for Mercurial to use \001 and
\002 around any terminal-specific escape sequences it uses that take
up zero width -- particularly since, without that, readline might
compute the column widths wrong for strings like "\033[0;33m", where
it will treat the "\033" part as zero-width but the "[0;33m" as
nonzero-width.
On the other hand, I don't think Python really exposes this reasonably
as part of the documented and reliable builtin input() or readline
semantics. (There are macros RL_PROMPT_START/END_IGNORE in readline.h
but Python doesn't expose them. It's hard to search for applications
that handle this because what am I going to search for, the text
`\001' or `\x01' or something?) And what libedit is doing here isn't
quite the same as libreadline, so there is a compatibility issue.
So my inclination is still to switch back to unconditional libreadline
until we've had more time to digest this -- after the branch.
From: "Taylor R Campbell" <riastradh@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/58875 CVS commit: pkgsrc/lang
Date: Fri, 6 Dec 2024 06:19:12 +0000
Module Name: pkgsrc
Committed By: riastradh
Date: Fri Dec 6 06:19:12 UTC 2024
Modified Files:
pkgsrc/lang/python311: Makefile options.mk
pkgsrc/lang/python312: Makefile options.mk
pkgsrc/lang/python313: Makefile options.mk
Log Message:
lang/python*: Use libreadline unconditionally.
libedit breaks some applications. Maybe we can fix it later but for
now let's just go back to using libreadline unconditionally like we
did before.
PKGREVISION++ because this materially affects the build product -- it
built before this change but didn't work right for some applications.
(No change to platforms without libedit -- which is most non-NetBSD
platforms, probably -- but we don't have per-platform PKGREVISION, so
gotta bump it for everyone.)
No change to 2.7, 3.9, or 3.10 because they already use libreadline
unconditionally; respecting READLINE_TYPE is a new change in >=3.11
since April which broke applications.
ok jperkin
PR pkg/58875: python: terminal colours broke in some applications
To generate a diff of this commit:
cvs rdiff -u -r1.35 -r1.36 pkgsrc/lang/python311/Makefile
cvs rdiff -u -r1.3 -r1.4 pkgsrc/lang/python311/options.mk
cvs rdiff -u -r1.22 -r1.23 pkgsrc/lang/python312/Makefile
cvs rdiff -u -r1.8 -r1.9 pkgsrc/lang/python312/options.mk
cvs rdiff -u -r1.6 -r1.7 pkgsrc/lang/python313/Makefile
cvs rdiff -u -r1.1 -r1.2 pkgsrc/lang/python313/options.mk
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.47 2022/09/11 19:34:41 kim Exp $
$NetBSD: gnats_config.sh,v 1.9 2014/08/02 14:16:04 spz Exp $
Copyright © 1994-2024
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home