NetBSD Problem Report #61017
From juan.mendezr@proton.me Mon Nov 16 05:11:32 2026
Return-Path: <juan.mendezr@proton.me>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
client-signature RSA-PSS (2048 bits) client-digest SHA256)
(Client CN "mail.netbsd.org", Issuer "YR1" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id D6DCE1A923B
for <gnats-bugs@gnats.NetBSD.org>; Mon, 16 Nov 2026 05:11:32 +0000 (UTC)
Message-Id: <17c043e9da61c82964730957d828ae68@proton.me>
Date: Mon, 16 Nov 2026 05:11:21 +0000
From: =?utf-8?Q?Juan_Manuel_M=C3=A9ndez_Rey?= <juan.mendezr@proton.me>
Reply-To:
To: gnats-bugs@NetBSD.org
Subject: regexec(3): \B after a multibyte character walks past the end of the string
X-From4GNATS: "=?utf-8?Q?Juan_Manuel_M=C3=A9ndez_Rey?= via gnats" <gnats-admin@NetBSD.org>
>Number: 61017
>Category: lib
>Synopsis: regexec(3): \B after a multibyte character walks past the =
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: lib-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: unknown
>Arrival-Date: Mon Nov 16 05:15:00 +0000 2026
>Last-Modified: Mon Nov 16 17:40:02 +0000 2026
>Originator: =?utf-8?Q?Juan_Manuel_M=C3=A9ndez_Rey?= <juan.mendezr@proton.me>
>Release: NetBSD 10.0; the same code is in -current
>Environment:
System: NetBSD 10.0 (GENERIC) amd64, the pygolo/netbsd10 Vagrant box.
/usr/lib/libc.so.12 has engine.c 1.29, regexec.c 1.26, regcomp.c 1.46.
-current was tested by building the trunk lib/libc/regex sources
(engine.c 1.30, regcomp.c 1.50) into the test program on the same 10.0
system.
>Description:
Under LC_CTYPE=3DC.UTF-8, a pattern compiled with REG_EXTENDED|REG_GNU
that has \B in front of an alternation, such as "\B(a|b)b", makes
regexec(REG_STARTEND) on the 5-byte string 20 c3 83 62 62 (" ", U+00C3,
"b", "b") die with SIGSEGV, or return 0 with a match far outside the
string. The correct answer is rm_so=3D3, rm_eo=3D5.
Measured on 10.0 with the program below, 20 runs of each command,
stock libc:
"\B(a|b)b" rm_so=3D1 case: 17 SIGSEGV, 3 matches outside the string
"\B(a|b)b" rm_so=3D1 icase: 20 matches outside the string
"\B(a|b)[ab]" rm_so=3D1 case: 20 SIGSEGV
"\B(a|b)[ab]" rm_so=3D1 icase: 20 matches outside the string
"\B(a|b)[ab]" rm_so=3D0 case: 18 SIGSEGV, 2 matches outside the string
"\B(a|b)[ab]" rm_so=3D2 case: 18 SIGSEGV, 2 matches outside the string
A match outside the string looks like this (icase run):
rc=3D0 rm_so=3D8244 rm_eo=3D8246 (string is 5 bytes)
The netbsd-10 and trunk sources built into the program give the same
kinds of results (SIGSEGV in 16 to 20 runs of 20 for "case", a match
outside the string in 20 of 20 for "icase"). The backtrace of the first
command with the stock libc:
#0 _citrus_UTF8_mbrtowc_priv.constprop.0 () from /usr/lib/i18n/libUTF8.so=
.5.0
#1 _citrus_UTF8_ctype_mbrtowc () from /usr/lib/i18n/libUTF8.so.5.0
#2 mbrtowc_l () from /usr/lib/libc.so.12
#3 mwalk () from /usr/lib/libc.so.12
#4 regexec () from /usr/lib/libc.so.12
#5 main (argc=3D5, argv=3D...) at repro.c:46
Built from the netbsd-10 sources, the same crash shows the cause:
#3 xmbrtowc (..., n=3D18446744073709355008, ...) at regexec.c:84
#4 mwalk (..., start=3D..., stop=3D..., fast=3Dfalse) at engine.c:924
#5 mmatcher (...) at engine.c:327
(gdb) frame 4
(gdb) p m->coldp - m->endp
$1 =3D 196607
Cause. walk() (engine.c, trunk line 914) takes the character before
the start of the walk as one byte, (uch)*(start - 1); the comment there
says this is wrong for a multibyte character. Here the byte before "b"
is 0x83, the second byte of U+00C3, so a walk that starts at "b" sees
a non-word character in front of it, while the fast walk in matcher(),
which decodes U+00C3 whole, sees a word character. The fast walk finds
a match that starts at offset 3; the slow walks that look for the start
(engine.c:322-331) do not find it, and that loop advances m->coldp
until a walk "matches". assert(m->coldp < m->endp) is compiled out, so
m->coldp goes past m->endp, (size_t)(m->endp - p) becomes huge, and
mbrtowc reads past the string. Whether that ends in SIGSEGV or in a
match from unrelated memory depends on what follows the string.
The same cause gives wrong answers without a crash, because only the
walks that start after a multibyte character are affected. On the same
string with rm_so=3D1, stock 10.0 libc:
"\Bbb" rc=3D1 (correct: rm_so=3D3 rm_eo=3D5)
"\B(b)b" rc=3D1 (correct: rm_so=3D3 rm_eo=3D5)
"\B[ab]" rm_so=3D4 rm_eo=3D5 (correct: rm_so=3D3 rm_eo=3D4)
"\B" rm_so=3D4 rm_eo=3D4 (correct: rm_so=3D3 rm_eo=3D3)
The runaway also reads the byte at rm_eo. regex(3) says that with
REG_STARTEND the string ends "before the byte located at
string + pmatch[0].rm_eo". With the string placed so that rm_eo is the
first byte of an unmapped page ("edge" below), "\B(a|b)b" on the string
above (case) and "\B(a|b)[ab]" on 09 20 c3 83 62 62 20 62 61 (icase)
fault in 20 runs out of 20. For the second one gdb shows the fault in
mwalk at engine.c:914, (uch)*(start - 1), with m->coldp - m->endp =3D=3D 1:
the runaway walk reads the byte at rm_eo. The patch below removes this
read too, so it does not need a report of its own.
Found by the xwpe editor's search tests.
>How-To-Repeat:
repro.c:
/*
* usage: repro pattern hexstring rm_so icase|case [edge]
* Compiles pattern with REG_EXTENDED|REG_GNU (and REG_ICASE for "icase")
* under LC_CTYPE=3DC.UTF-8 and runs regexec(REG_STARTEND) on the string
* from rm_so to its end. With "edge" the string ends at an unmapped page.
*/
#include <sys/mman.h>
#include <locale.h>
#include <regex.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
int
main(int argc, char **argv)
{
=09char tmp[256], *s;
=09size_t n, i, ps =3D (size_t)sysconf(_SC_PAGESIZE);
=09regex_t re;
=09regmatch_t m;
=09int rc, cfl =3D REG_EXTENDED | REG_GNU;
=09unsigned v;
=09if (argc < 5 || setlocale(LC_CTYPE, "C.UTF-8") =3D=3D NULL)
=09=09return 2;
=09if (strcmp(argv[4], "icase") =3D=3D 0)
=09=09cfl |=3D REG_ICASE;
=09for (n =3D 0, i =3D 0; argv[2][i] && argv[2][i + 1]; i +=3D 2, n++) {
=09=09sscanf(argv[2] + i, "%2x", &v);
=09=09tmp[n] =3D (char)v;
=09}
=09s =3D mmap(NULL, 2 * ps, PROT_READ | PROT_WRITE, MAP_ANON | MAP_PRIVATE,
=09 -1, 0);
=09if (argc > 5 && strcmp(argv[5], "edge") =3D=3D 0)
=09=09mprotect(s + ps, ps, PROT_NONE);=09/* s[n] is unmapped */
=09else
=09=09s[ps] =3D '\0';
=09s +=3D ps - n;
=09memcpy(s, tmp, n);
=09if (regcomp(&re, argv[1], cfl) !=3D 0)
=09=09return 3;
=09m.rm_so =3D atoi(argv[3]);
=09m.rm_eo =3D (regoff_t)n;
=09rc =3D regexec(&re, s, 1, &m, REG_STARTEND);
=09printf("rc=3D%d rm_so=3D%jd rm_eo=3D%jd (string is %zu bytes)\n", rc,
=09 (intmax_t)m.rm_so, (intmax_t)m.rm_eo, n);
=09return 0;
}
$ cc -o repro repro.c
$ ./repro '\B(a|b)[ab]' 20c3836262 1 case
(killed by SIGSEGV, core dumped)
$ ./repro '\B(a|b)[ab]' 20c3836262 1 icase
rc=3D0 rm_so=3D8244 rm_eo=3D8246 (string is 5 bytes)
$ ./repro '\B[ab]' 20c3836262 1 case
rc=3D0 rm_so=3D4 rm_eo=3D5 (string is 5 bytes)
$ ./repro '\B(a|b)b' 20c3836262 1 case edge
(killed by SIGSEGV, core dumped)
$ ./repro '(a|b)[ab]' 20c3836262 1 case
rc=3D0 rm_so=3D3 rm_eo=3D5 (string is 5 bytes)
$ ./repro '\B(a|b)b' 20626262 1 case
rc=3D0 rm_so=3D2 rm_eo=3D4 (string is 4 bytes)
The last two show that the pattern without \B, and the string without
the multibyte character, are fine. The offset in the icase line
changes from run to run.
>Fix:
Decode the previous character whole in walk(), the same way the fast
walk decodes it, and stop the start-finding loop at the end of the
string so a disagreement can never walk off the string again.
Tested on 10.0, with the trunk sources and the patch below built into
the program:
- each command above, 20 runs each: rm_so=3D3 rm_eo=3D5 every time (4..6 fo=
r
the 9-byte string 0920c3836262206261 with "edge"), no faults;
- \B[ab] gives 3..4 and \B gives 3..3;
- 20000 random patterns built from a, b, [ab], ., (a|b), (a|b)+, [^ab],
\w, \W, a*, (b|ab) and \b \B \< \>, on random ASCII strings with
random rm_so, rm_eo, REG_ICASE, REG_NOTBOL and REG_NOTEOL, give the
same answer with the patch as with the stock 10.0 libc;
- the same 20000 cases on strings with U+00C3, U+00E9, U+00A0, U+20AC,
U+4E61 and U+1F600 give the same answer as the same string with each
of those characters replaced by an ASCII character of the same word
class (x or -), with the offsets mapped back. The stock libc does
not get past the 17th of these strings.
The patch also applies to netbsd-10 unchanged.
Index: lib/libc/regex/engine.c
--- a/lib/libc/regex/engine.c
+++ b/lib/libc/regex/engine.c
@@ -54,6 +54,7 @@
#ifdef SNAMES
#define=09stepback sstepback
+#define=09prevchar sprevchar
#define=09matcher=09smatcher
#define=09walk=09swalk
#define=09dissect=09sdissect
@@ -65,6 +66,7 @@
#endif
#ifdef LNAMES
#define=09stepback lstepback
+#define=09prevchar lprevchar
#define=09matcher=09lmatcher
#define=09walk=09lwalk
#define=09dissect=09ldissect
@@ -76,6 +78,7 @@
#endif
#ifdef MNAMES
#define=09stepback mstepback
+#define=09prevchar mprevchar
#define=09matcher=09mmatcher
#define=09walk=09mwalk
#define=09dissect=09mdissect
@@ -192,6 +195,34 @@
}
/*
+ * Return the character that ends at cur, without looking before start.
+ * A multibyte character is decoded whole and an invalid byte gives
+ * BADCHAR, so that a walk which starts at cur sees the same previous
+ * character as a walk which reached cur from further left.
+ */
+static wint_t
+prevchar(const char *start, const char *cur)
+{
+#ifdef NLS
+=09mbstate_t mbs;
+=09wchar_t wc;
+=09size_t mbc, clen;
+
+=09if (MB_CUR_MAX > 1) {
+=09=09for (mbc =3D 1; mbc <=3D MB_CUR_MAX &&
+=09=09 mbc <=3D (size_t)(cur - start); mbc++) {
+=09=09=09memset(&mbs, 0, sizeof(mbs));
+=09=09=09clen =3D mbrtowc(&wc, cur - mbc, mbc, &mbs);
+=09=09=09if (clen =3D=3D mbc || (clen =3D=3D 0 && mbc =3D=3D 1))
+=09=09=09=09return (wc);
+=09=09}
+=09=09return (BADCHAR);
+=09}
+#endif
+=09return ((uch)*(cur - 1));
+}
+
+/*
- matcher - the actual matching engine
=3D=3D static int matcher(struct re_guts *g, const char *string, \
=3D=3D=09size_t nmatch, regmatch_t pmatch[], int eflags);
@@ -328,6 +359,11 @@
=09=09=09if (endp !=3D NULL)
=09=09=09=09break;
=09=09=09assert(m->coldp < m->endp);
+=09=09=09if (m->coldp >=3D m->endp) {
+=09=09=09=09/* never walk past the end of the string */
+=09=09=09=09error =3D REG_NOMATCH;
+=09=09=09=09goto done;
+=09=09=09}
=09=09=09m->coldp +=3D XMBRTOWC(NULL, m->coldp,
=09=09=09 (size_t)(m->endp - m->coldp), &m->mbs, 0);
=09=09}
@@ -905,14 +941,8 @@
=09matchp =3D NULL;
=09if (start =3D=3D m->offp || (start =3D=3D m->beginp && !(m->eflags®_=
NOTBOL)))
=09=09c =3D OUT;
-=09else {
-=09=09/*
-=09=09 * XXX Wrong if the previous character was multi-byte.
-=09=09 * Newline never is (in encodings supported by FreeBSD),
-=09=09 * so this only breaks the ISWORD tests below.
-=09=09 */
-=09=09c =3D (uch)*(start - 1);
-=09}
+=09else
+=09=09c =3D prevchar(m->offp, start);
=09for (;;) {
=09=09/* next character */
=09=09lastc =3D c;
@@ -1258,6 +1288,7 @@
#endif
#undef=09stepback
+#undef=09prevchar
#undef=09matcher
#undef=09walk
#undef=09dissect
FreeBSD's lib/libc/regex/engine.c on main still has the same one-byte
lookup, with the same comment.
>Audit-Trail:
From: "Christos Zoulas" <christos@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/61017 CVS commit: src/tests/lib/libc/regex
Date: Mon, 16 Nov 2026 12:33:45 -0500
Module Name: src
Committed By: christos
Date: Mon Nov 16 17:33:45 UTC 2026
Modified Files:
src/tests/lib/libc/regex: Makefile t_regex.sh
Added Files:
src/tests/lib/libc/regex: h_match.c
Log Message:
PR/61017: Juan Manuel Mendez Rey: regexec reads past the string after a
multibyte character
To generate a diff of this commit:
cvs rdiff -u -r1.13 -r1.14 src/tests/lib/libc/regex/Makefile
cvs rdiff -u -r0 -r1.1 src/tests/lib/libc/regex/h_match.c
cvs rdiff -u -r1.1 -r1.2 src/tests/lib/libc/regex/t_regex.sh
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Christos Zoulas" <christos@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/61017 CVS commit: src/distrib/sets/lists
Date: Mon, 16 Nov 2026 12:35:20 -0500
Module Name: src
Committed By: christos
Date: Mon Nov 16 17:35:20 UTC 2026
Modified Files:
src/distrib/sets/lists/debug: mi
src/distrib/sets/lists/tests: mi
Log Message:
New regex rest for PR/61017
To generate a diff of this commit:
cvs rdiff -u -r1.525 -r1.526 src/distrib/sets/lists/debug/mi
cvs rdiff -u -r1.1435 -r1.1436 src/distrib/sets/lists/tests/mi
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: "Christos Zoulas" <christos@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/61017 CVS commit: src/lib/libc/regex
Date: Mon, 16 Nov 2026 12:36:41 -0500
Module Name: src
Committed By: christos
Date: Mon Nov 16 17:36:40 UTC 2026
Modified Files:
src/lib/libc/regex: engine.c
Log Message:
PR/61017: Juan Manuel Mendez Rey: regexec reads past the string after a
multibyte character. Provide a prevchar() function that fetches the previous
multibyte character.
To generate a diff of this commit:
cvs rdiff -u -r1.30 -r1.31 src/lib/libc/regex/engine.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
>Unformatted:
end of the string (SIGSEGV)
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.