NetBSD Problem Report #59761
From www@netbsd.org Sun Nov 16 00:57:20 2025
Return-Path: <www@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
client-signature RSA-PSS (2048 bits) client-digest SHA256)
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 133EE1A9239
for <gnats-bugs@gnats.NetBSD.org>; Sun, 16 Nov 2025 00:57:20 +0000 (UTC)
Message-Id: <20251116005718.9DADD1A923A@mollari.NetBSD.org>
Date: Sun, 16 Nov 2025 00:57:18 +0000 (UTC)
From: medwards-netbsd@mirkwood.net
Reply-To: medwards-netbsd@mirkwood.net
To: gnats-bugs@NetBSD.org
Subject: dmsetup broken in 10.1 with default GENERIC kernel
X-Send-Pr-Version: www-1.0
>Number: 59761
>Category: kern
>Synopsis: dmsetup broken in 10.1 with default GENERIC kernel
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: kern-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sun Nov 16 01:00:00 +0000 2025
>Last-Modified: Mon Nov 17 21:25:00 +0000 2025
>Originator: Mike Edwards
>Release: 10.1
>Organization:
N/A
>Environment:
NetBSD netbsd02 10.1 NetBSD 10.1 (GENERIC) #0: Mon Dec 16 13:08:11 UTC 2024 mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/amd64/compile/GENERIC amd64
>Description:
dmsetup fails to work, causing lvm to similarly break while attempting to create an LV, though PV and VG creation appear to work.
This is easily seen with dmsetup --version or dmsetup ls:
[ksh] root@netbsd02# dmsetup --version
Library version: 1.02.40 (2009-11-24)
ioctl version call failed with errno 25
Command failed
[ksh] root@netbsd02# dmsetup ls
ioctl version call failed with errno 25
Command failed
ktruss reports the ioctl that fails is NETBSD_DM_IOCTL:
1650 1650 dmsetup ioctl(0x4, NETBSD_DM_IOCTL, 0x7f7fffd5e1b0) Err#25 ENOTTY
Module 'dm' is loaded:
[ksh] root@netbsd02# modstat dm
NAME CLASS SOURCE FLAG REFS SIZE REQUIRES
dm driver filesys - 0 18216 dk_subr
Without checking the source, I can see references to NETBSD_DM_IOCTL in the dm module, as expected:
[ksh] root@netbsd02# grep -r NETBSD_DM_IOCTL /stand/
Binary file /stand/amd64/10.1/modules/dm/dm.kmod matches
I can't find a reason this ioctl should fail, given that the dm module is indeed loaded, which tells me that this driver appears to have broken at some point.
>How-To-Repeat:
Run dmstat --version or dmstat ls, watch for ioctl call to fail.
>Fix:
Unknown.
>Audit-Trail:
From: mlelstv@serpens.de (Michael van Elst)
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/59761: dmsetup broken in 10.1 with default GENERIC kernel
Date: Sun, 16 Nov 2025 17:24:06 -0000 (UTC)
medwards-netbsd@mirkwood.net writes:
>This is easily seen with dmsetup --version or dmsetup ls:
>[ksh] root@netbsd02# dmsetup --version
>Library version: 1.02.40 (2009-11-24)
>ioctl version call failed with errno 25
>Command failed
>[ksh] root@netbsd02# dmsetup ls
>ioctl version call failed with errno 25
Can't reproduce that. Please check the output of
ls -ld /dev/mapper/control
it should be like this:
crw-rw---- 1 root operator 194, 0 Sep 7 2022 /dev/mapper/control
Please also retry the dmsetup commands with two -v flags like:
# dmsetup -vv --version
Library version: 1.02.40 (2009-11-24)
Ioctl type version --- flags 4
Ioctl type version --- flags 4
Driver version: 4.16.0
# dmsetup -vv ls
Ioctl type version --- flags 4
Ioctl type names --- flags 4
No devices found
to see if any errors are reported.
From: Mike Edwards <mike@mobile.mirkwood.net>
To: gnats-bugs@netbsd.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
medwards-netbsd@mirkwood.net
Subject: Re: kern/59761: dmsetup broken in 10.1 with default GENERIC kernel
Date: Sun, 16 Nov 2025 18:58:35 +0000
Well, this is bizarre:
[ksh] root@netbsd02# ls -ld /dev/mapper/control
crw-rw---- 1 root operator 0, 0 Nov 15 19:50 /dev/mapper/control
0,0? Oddly, dmsetup doesn't agree with ls here:
[ksh] root@netbsd02# dmsetup -vv ls
/dev/mapper/control: Wrong device number: (33, 176) instead of (194, 0)
Creating device /dev/mapper/control (194, 0)
Ioctl type version --- flags 4
ioctl version call failed with errno 25
Command failed
Why would ls report 0,0, and dmsetup 33,176?
One other piece of information that it occurs to me may be relevant:
This is a pvh vm running under Xen.
On Sun, Nov 16, 2025 at 5:32=E2=80=AFPM Michael van Elst via gnats
<gnats-admin@netbsd.org> wrote:
>
> The following reply was made to PR kern/59761; it has been noted by GNATS=
.
>
> From: mlelstv@serpens.de (Michael van Elst)
> To: gnats-bugs@netbsd.org
> Cc:
> Subject: Re: kern/59761: dmsetup broken in 10.1 with default GENERIC kern=
el
> Date: Sun, 16 Nov 2025 17:24:06 -0000 (UTC)
>
> medwards-netbsd@mirkwood.net writes:
>
> >This is easily seen with dmsetup --version or dmsetup ls:
> >[ksh] root@netbsd02# dmsetup --version
> >Library version: 1.02.40 (2009-11-24)
> >ioctl version call failed with errno 25
>
> >Command failed
> >[ksh] root@netbsd02# dmsetup ls
> >ioctl version call failed with errno 25
>
>
> Can't reproduce that. Please check the output of
>
> ls -ld /dev/mapper/control
>
> it should be like this:
>
> crw-rw---- 1 root operator 194, 0 Sep 7 2022 /dev/mapper/control
>
>
> Please also retry the dmsetup commands with two -v flags like:
>
> # dmsetup -vv --version
> Library version: 1.02.40 (2009-11-24)
> Ioctl type version --- flags 4
> Ioctl type version --- flags 4
> Driver version: 4.16.0
>
> # dmsetup -vv ls
> Ioctl type version --- flags 4
> Ioctl type names --- flags 4
> No devices found
>
> to see if any errors are reported.
>
From: mlelstv@serpens.de (Michael van Elst)
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/59761: dmsetup broken in 10.1 with default GENERIC kernel
Date: Sun, 16 Nov 2025 22:43:34 -0000 (UTC)
gnats-admin@NetBSD.org ("Mike Edwards via gnats") writes:
> [ksh] root@netbsd02# ls -ld /dev/mapper/control
> crw-rw---- 1 root operator 0, 0 Nov 15 19:50 /dev/mapper/control
>
> 0,0? Oddly, dmsetup doesn't agree with ls here:
0, 0 would be "console". Something is wrong in your /dev directory.
> Why would ls report 0,0, and dmsetup 33,176?
The source reveals utter stupidity:
if (major && buf.st_rdev != MKDEV(major, minor)) {
log_verbose("%s: Wrong device number: (%u, %u) instead of "
"(%u, %u)", control,
MAJOR(buf.st_mode), MINOR(buf.st_mode),
major, minor);
...
The check is correct, but then it prints bits from buf.st_mode instead
of buf.st_rdev.
From: Mike Edwards <mike@mobile.mirkwood.net>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/59761: dmsetup broken in 10.1 with default GENERIC kernel
Date: Sun, 16 Nov 2025 23:32:32 +0000
Ok, well, that's funny - there's a bug here, but it's not the one I
thought it was!
Then again, there's still something odd:
[ksh] root@netbsd02# ls -al
total 48
drwxr-xr-x 2 root wheel 4096 Nov 16 18:19 .
drwxr-xr-x 11 root wheel 45056 Nov 12 22:03 ..
crw-rw---- 1 root operator 0, 0 Nov 16 18:19 control
[ksh] root@netbsd02# rm control
[ksh] root@netbsd02# ls -al
total 48
drwxr-xr-x 2 root wheel 4096 Nov 16 18:21 .
drwxr-xr-x 11 root wheel 45056 Nov 12 22:03 ..
[ksh] root@netbsd02# mknod -m 660 -u root -g operator control c 194 0
[ksh] root@netbsd02# ls -al
total 48
drwxr-xr-x 2 root wheel 4096 Nov 16 18:21 .
drwxr-xr-x 11 root wheel 45056 Nov 12 22:03 ..
crw-rw---- 1 root operator 0, 0 Nov 16 18:21 control
[ksh] root@netbsd02#
Perhaps unusually, I'm using ext2fs here, as it makes it easier to
mount the vm on the host if needed (Xen dom0 is currently Linux).
Other nodes in the top-level /dev seem to have correct numbering,
though I'm finding if I try to create a new copy of one of those nodes
with mknod, it likewise creates a device with address 0, 0.
If I recall correctly, a script running on the Linux host created the
nodes in /dev initially, so that may explain why those nodes have
correct numbering, but the inability to create a node with anything
but 0, 0 on the running vm is a bit on the strange side.
On Sun, Nov 16, 2025 at 10:45=E2=80=AFPM Michael van Elst via gnats
<gnats-admin@netbsd.org> wrote:
>
> The following reply was made to PR kern/59761; it has been noted by GNATS=
.
>
> From: mlelstv@serpens.de (Michael van Elst)
> To: gnats-bugs@netbsd.org
> Cc:
> Subject: Re: kern/59761: dmsetup broken in 10.1 with default GENERIC kern=
el
> Date: Sun, 16 Nov 2025 22:43:34 -0000 (UTC)
>
> gnats-admin@NetBSD.org ("Mike Edwards via gnats") writes:
>
> > [ksh] root@netbsd02# ls -ld /dev/mapper/control
> > crw-rw---- 1 root operator 0, 0 Nov 15 19:50 /dev/mapper/control
> >
> > 0,0? Oddly, dmsetup doesn't agree with ls here:
>
> 0, 0 would be "console". Something is wrong in your /dev directory.
>
>
> > Why would ls report 0,0, and dmsetup 33,176?
>
> The source reveals utter stupidity:
>
> if (major && buf.st_rdev !=3D MKDEV(major, minor)) {
> log_verbose("%s: Wrong device number: (%u, %u) instead o=
f "
> "(%u, %u)", control,
> MAJOR(buf.st_mode), MINOR(buf.st_mode),
> major, minor);
> ...
>
> The check is correct, but then it prints bits from buf.st_mode instead
> of buf.st_rdev.
>
>
From: Mike Edwards <mike@mobile.mirkwood.net>
To: gnats-bugs@netbsd.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
medwards-netbsd@mirkwood.net
Subject: Re: kern/59761: dmsetup broken in 10.1 with default GENERIC kernel
Date: Mon, 17 Nov 2025 00:28:46 +0000
Confirmed - if I shut down the VM, mount it's fs on the Linux Xen
host, and mknod the control file from there, it's created correctly,
and now dmsetup -vv ls works:
[ksh] root@netbsd02# dmsetup -vv ls
Ioctl type version --- flags 4
Ioctl type names --- flags 4
No devices found
I suppose it'll do as a workaround right now, but the inability to
create the node from the running netbsd VM is a bit odd. Perhaps an
issue with using ext2 for /?
On Sun, Nov 16, 2025 at 11:35=E2=80=AFPM Mike Edwards via gnats
<gnats-admin@netbsd.org> wrote:
>
> The following reply was made to PR kern/59761; it has been noted by GNATS=
.
>
> From: Mike Edwards <mike@mobile.mirkwood.net>
> To: gnats-bugs@netbsd.org
> Cc:
> Subject: Re: kern/59761: dmsetup broken in 10.1 with default GENERIC kern=
el
> Date: Sun, 16 Nov 2025 23:32:32 +0000
>
> Ok, well, that's funny - there's a bug here, but it's not the one I
> thought it was!
>
> Then again, there's still something odd:
> [ksh] root@netbsd02# ls -al
> total 48
> drwxr-xr-x 2 root wheel 4096 Nov 16 18:19 .
> drwxr-xr-x 11 root wheel 45056 Nov 12 22:03 ..
> crw-rw---- 1 root operator 0, 0 Nov 16 18:19 control
> [ksh] root@netbsd02# rm control
> [ksh] root@netbsd02# ls -al
> total 48
> drwxr-xr-x 2 root wheel 4096 Nov 16 18:21 .
> drwxr-xr-x 11 root wheel 45056 Nov 12 22:03 ..
> [ksh] root@netbsd02# mknod -m 660 -u root -g operator control c 194 0
> [ksh] root@netbsd02# ls -al
> total 48
> drwxr-xr-x 2 root wheel 4096 Nov 16 18:21 .
> drwxr-xr-x 11 root wheel 45056 Nov 12 22:03 ..
> crw-rw---- 1 root operator 0, 0 Nov 16 18:21 control
> [ksh] root@netbsd02#
>
> Perhaps unusually, I'm using ext2fs here, as it makes it easier to
> mount the vm on the host if needed (Xen dom0 is currently Linux).
> Other nodes in the top-level /dev seem to have correct numbering,
> though I'm finding if I try to create a new copy of one of those nodes
> with mknod, it likewise creates a device with address 0, 0.
>
> If I recall correctly, a script running on the Linux host created the
> nodes in /dev initially, so that may explain why those nodes have
> correct numbering, but the inability to create a node with anything
> but 0, 0 on the running vm is a bit on the strange side.
>
> On Sun, Nov 16, 2025 at 10:45=3DE2=3D80=3DAFPM Michael van Elst via gnat=
s
> <gnats-admin@netbsd.org> wrote:
> >
> > The following reply was made to PR kern/59761; it has been noted by GN=
ATS=3D
> .
> >
> > From: mlelstv@serpens.de (Michael van Elst)
> > To: gnats-bugs@netbsd.org
> > Cc:
> > Subject: Re: kern/59761: dmsetup broken in 10.1 with default GENERIC k=
ern=3D
> el
> > Date: Sun, 16 Nov 2025 22:43:34 -0000 (UTC)
> >
> > gnats-admin@NetBSD.org ("Mike Edwards via gnats") writes:
> >
> > > [ksh] root@netbsd02# ls -ld /dev/mapper/control
> > > crw-rw---- 1 root operator 0, 0 Nov 15 19:50 /dev/mapper/control
> > >
> > > 0,0? Oddly, dmsetup doesn't agree with ls here:
> >
> > 0, 0 would be "console". Something is wrong in your /dev directory.
> >
> >
> > > Why would ls report 0,0, and dmsetup 33,176?
> >
> > The source reveals utter stupidity:
> >
> > if (major && buf.st_rdev !=3D3D MKDEV(major, minor)) {
> > log_verbose("%s: Wrong device number: (%u, %u) instea=
d o=3D
> f "
> > "(%u, %u)", control,
> > MAJOR(buf.st_mode), MINOR(buf.st_mode),
> > major, minor);
> > ...
> >
> > The check is correct, but then it prints bits from buf.st_mode instea=
d
> > of buf.st_rdev.
> >
> >
>
From: mlelstv@serpens.de (Michael van Elst)
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/59761: dmsetup broken in 10.1 with default GENERIC kernel
Date: Mon, 17 Nov 2025 01:11:12 -0000 (UTC)
gnats-admin@NetBSD.org ("Mike Edwards via gnats") writes:
> Perhaps unusually, I'm using ext2fs here, as it makes it easier to
> mount the vm on the host if needed (Xen dom0 is currently Linux).
Looks like ext2fs does not initialize the rdev field for anything.
-rw-r--r-- 1 root wheel 0 Nov 17 02:08 foo
crw-r--r-- 1 root wheel 0, 0 Nov 17 02:09 bar
% stat -s foo
st_dev=4864 st_ino=12 st_mode=0100644 st_nlink=1 st_uid=0 st_gid=0 st_rdev=0 st_size=0 st_atime=1763341722 st_mtime=1763341722 st_ctime=1763341722 st_birthtime=0 st_blksize=1024 st_blocks=0 st_flags=0
% stat -s bar
st_dev=4864 st_ino=13 st_mode=020644 st_nlink=1 st_uid=0 st_gid=0 st_rdev=0 st_size=0 st_atime=1763341746 st_mtime=1763341746 st_ctime=1763341746 st_birthtime=0 st_blksize=65536 st_blocks=0 st_flags=0
From: Mike Edwards <mike@mobile.mirkwood.net>
To: gnats-bugs@netbsd.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
medwards-netbsd@mirkwood.net
Subject: Re: kern/59761: dmsetup broken in 10.1 with default GENERIC kernel
Date: Mon, 17 Nov 2025 08:56:43 -0500
--0000000000003028f50643cab846
Content-Type: text/plain; charset="UTF-8"
So, we effectively have two bugs here: dmsetup prints garbage, and ext2fs
does nothing with rdev.
What's the next step - create two new bugs and close this one?
Thanks for your help in tracking these down!
On Sun, Nov 16, 2025, 20:15 Michael van Elst via gnats <
gnats-admin@netbsd.org> wrote:
> The following reply was made to PR kern/59761; it has been noted by GNATS.
>
> From: mlelstv@serpens.de (Michael van Elst)
> To: gnats-bugs@netbsd.org
> Cc:
> Subject: Re: kern/59761: dmsetup broken in 10.1 with default GENERIC kernel
> Date: Mon, 17 Nov 2025 01:11:12 -0000 (UTC)
>
> gnats-admin@NetBSD.org ("Mike Edwards via gnats") writes:
>
> > Perhaps unusually, I'm using ext2fs here, as it makes it easier to
> > mount the vm on the host if needed (Xen dom0 is currently Linux).
>
>
> Looks like ext2fs does not initialize the rdev field for anything.
>
> -rw-r--r-- 1 root wheel 0 Nov 17 02:08 foo
> crw-r--r-- 1 root wheel 0, 0 Nov 17 02:09 bar
>
> % stat -s foo
> st_dev=4864 st_ino=12 st_mode=0100644 st_nlink=1 st_uid=0 st_gid=0
> st_rdev=0 st_size=0 st_atime=1763341722 st_mtime=1763341722
> st_ctime=1763341722 st_birthtime=0 st_blksize=1024 st_blocks=0 st_flags=0
>
> % stat -s bar
> st_dev=4864 st_ino=13 st_mode=020644 st_nlink=1 st_uid=0 st_gid=0
> st_rdev=0 st_size=0 st_atime=1763341746 st_mtime=1763341746
> st_ctime=1763341746 st_birthtime=0 st_blksize=65536 st_blocks=0 st_flags=0
>
>
>
--0000000000003028f50643cab846
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"auto">So, we effectively have two bugs here: dmsetup prints gar=
bage, and ext2fs does nothing with rdev.<div dir=3D"auto"><br></div><div di=
r=3D"auto">What's the next step - create two new bugs and close this on=
e?=C2=A0</div><div dir=3D"auto"><br></div><div dir=3D"auto">Thanks for your=
help in tracking these down!</div></div><br><div class=3D"gmail_quote gmai=
l_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Sun, Nov 16, 20=
25, 20:15 Michael van Elst via gnats <<a href=3D"mailto:gnats-admin@netb=
sd.org">gnats-admin@netbsd.org</a>> wrote:<br></div><blockquote class=3D=
"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding=
-left:1ex">The following reply was made to PR kern/59761; it has been noted=
by GNATS.<br>
<br>
From: <a href=3D"mailto:mlelstv@serpens.de" target=3D"_blank" rel=3D"norefe=
rrer">mlelstv@serpens.de</a> (Michael van Elst)<br>
To: <a href=3D"mailto:gnats-bugs@netbsd.org" target=3D"_blank" rel=3D"noref=
errer">gnats-bugs@netbsd.org</a><br>
Cc: <br>
Subject: Re: kern/59761: dmsetup broken in 10.1 with default GENERIC kernel=
<br>
Date: Mon, 17 Nov 2025 01:11:12 -0000 (UTC)<br>
<br>
=C2=A0gnats-admin@NetBSD.org ("Mike Edwards via gnats") writes:<b=
r>
<br>
=C2=A0> Perhaps unusually, I'm using ext2fs here, as it makes it eas=
ier to<br>
=C2=A0> mount the vm on the host if needed (Xen dom0 is currently Linux)=
.<br>
<br>
<br>
=C2=A0Looks like ext2fs does not initialize the rdev field for anything.<br=
>
<br>
=C2=A0-rw-r--r--=C2=A0 1 root=C2=A0 wheel=C2=A0 =C2=A0 =C2=A0 0 Nov 17 02:0=
8 foo<br>
=C2=A0crw-r--r--=C2=A0 1 root=C2=A0 wheel=C2=A0 =C2=A00, 0 Nov 17 02:09 bar=
<br>
<br>
=C2=A0% stat -s foo<br>
=C2=A0st_dev=3D4864 st_ino=3D12 st_mode=3D0100644 st_nlink=3D1 st_uid=3D0 s=
t_gid=3D0 st_rdev=3D0 st_size=3D0 st_atime=3D1763341722 st_mtime=3D17633417=
22 st_ctime=3D1763341722 st_birthtime=3D0 st_blksize=3D1024 st_blocks=3D0 s=
t_flags=3D0<br>
<br>
=C2=A0% stat -s bar<br>
=C2=A0st_dev=3D4864 st_ino=3D13 st_mode=3D020644 st_nlink=3D1 st_uid=3D0 st=
_gid=3D0 st_rdev=3D0 st_size=3D0 st_atime=3D1763341746 st_mtime=3D176334174=
6 st_ctime=3D1763341746 st_birthtime=3D0 st_blksize=3D65536 st_blocks=3D0 s=
t_flags=3D0<br>
<br>
<br>
</blockquote></div>
--0000000000003028f50643cab846--
From: Mike Edwards <mike@mobile.mirkwood.net>
To: gnats-bugs@netbsd.org
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
medwards-netbsd@mirkwood.net
Subject: Re: kern/59761: dmsetup broken in 10.1 with default GENERIC kernel
Date: Mon, 17 Nov 2025 16:23:34 -0500
Sorry about that - the Android gmail client seems to have lost the
ability to reply in plain text at some point. Re-sending, in a
distinctly more readable format:
So, we effectively have two bugs here: dmsetup prints garbage, and
ext2fs does nothing with rdev.
What's the next step - create two new bugs and close this one?
Thanks for your help in tracking these down!
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home