NetBSD Problem Report #54166
From gson@gson.org Mon May 6 07:28:39 2019
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
by mollari.NetBSD.org (Postfix) with ESMTPS id 89C117A169
for <gnats-bugs@gnats.NetBSD.org>; Mon, 6 May 2019 07:28:39 +0000 (UTC)
Message-Id: <20190506072834.4A1BC989F84@guava.gson.org>
Date: Mon, 6 May 2019 10:28:34 +0300 (EEST)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@NetBSD.org
Subject: NetBSD tests hang after lost IDE interrupt under qemu
X-Send-Pr-Version: 3.95
>Number: 54166
>Category: kern
>Synopsis: NetBSD tests hang after lost IDE interrupt under qemu
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: mlelstv
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Mon May 06 07:30:00 +0000 2019
>Closed-Date: Sat Jun 08 06:31:23 +0000 2019
>Last-Modified: Sat Jun 08 06:31:23 +0000 2019
>Originator: Andreas Gustafsson
>Release: NetBSD-current, source date >= 2019.03.19.16.56.29
>Organization:
>Environment:
System: NetBSD
Architecture: i386
Machine: i386
>Description:
Running the ATF tests on recent versions of NetBSD/i386, /amd64, or
/sparc64 under recent versions of qemu frequently ends with the NetBSD
IDE disk driver reporting a lost interrupt, after which the tests
produce no further output. For example:
lib/librt/t_sem (418/799): 4 test cases
basic: [0.285119s] Passed.
child: piixide0:0:0: lost interrupt
[ 5969.7132325] type: ata tc_bcount: 2048 tc_skip: 0
piixide0:0:0: bus-master DMA error: missing interrupt, status=0x61
[ 5969.7953625] wd0a: DMA error writing fsbn 128 of 128-131 (wd0 bn 2176; cn 2 tn 2 sn 34), xfer f5c, retry 0
[ 5971.7754220] wd0: soft error (corrected) xfer f5c
This was extracted from:
http://www.gson.org/netbsd/bugs/build/i386/2019/2019.04.26.11.51.56/test.log
More reports of similar hangs:
http://www.gson.org/netbsd/bugs/build/i386/2019/2019.04.11.14.47.06/test.log
http://www.gson.org/netbsd/bugs/build/i386/2019/2019.04.12.09.29.26/test.log
http://www.gson.org/netbsd/bugs/build/sparc64/2019/2019.04.26.14.36.40/test.log
http://releng.netbsd.org/b5reports/sparc64/2019/2019.03.05.15.18.59/test.log
This has been tricky to track down because it only happens with the
combination of a recent qemu *and* a recent -current, and even then it
does not fail in every test run, only most of them.
Specifically, the lost interrupts appear to be related to the qemu
version, whereas the inability to recover from those lost interrupts
appears to be related to the NetBSD-current version. I believe I have
now identified the commit where the latter problem started, namely
2019.03.19.16.56.29 mlelstv src/sys/dev/ata/wd.c 1.446
Log output showing the tests running to completion before said commit
and hanging after it can be found at
http://www.gson.org/netbsd/bugs/build/i386/commits-2019.03.html#2019.03.19.16.56.29
This PR is intended to track the specific issue of the test runs
hanging. For the lost interrupts themselves, there is already
port-sparc64/54035.
>How-To-Repeat:
Run the ATF tests on NetBSD-current/i386 under qemu 3.1.0 or 4.0.0,
possibly multiple times.
>Fix:
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: kern-bug-people->mlelstv
Responsible-Changed-By: mlelstv@NetBSD.org
Responsible-Changed-When: Sat, 18 May 2019 08:16:25 +0000
Responsible-Changed-Why:
Mine
From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/54166: NetBSD tests hang after lost IDE interrupt under qemu
Date: Tue, 4 Jun 2019 19:34:46 +0300
One more data point: the TNF i386 testbed on babylon5.netbsd.org,
which runs under qemu 2.11.1, rarely gets the "bus-master DMA error:
missing interrupt" errors, but they do happen, and one just happened
while testing sources from yesterday:
http://releng.netbsd.org/b5reports/i386/2019/2019.06.03.09.58.31/test.log
And sure enough, it caused the test run to hang.
The previous one was in October, and it did not cause the test run to hang:
http://releng.netbsd.org/b5reports/i386/2018/2018.10.16.11.28.30/test.log
--
Andreas Gustafsson, gson@gson.org
From: Andreas Gustafsson <gson@gson.org>
To: Michael van Elst <mlelstv@serpens.de>, gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/54166: NetBSD tests hang after lost IDE interrupt under qemu
Date: Thu, 6 Jun 2019 18:51:58 +0300
Since babylon5 was upgraded from NetBSD 7 to 8.1 a few days ago, most
i386 and amd64 test runs have reported lost IDE interrupts and hung,
even though it is still running qemu version 2.11.1 (rebuilt for
NetBSD 8).
So, strange as it may seem, it now looks like the key to reliably
reproducing the lost interrupts that in turn reproduce the hang is not
the qemu version, but the NetBSD version that qemu runs on (and/or was
built for), specifically, that version being NetBSD 8.x. This also
explains why I haven't been able to reproduce the problem on a Linux
host.
--
Andreas Gustafsson, gson@gson.org
From: Martin Husemann <martin@homeworld.netbsd.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: kern/54166: NetBSD tests hang after lost IDE interrupt under qemu
Date: Thu, 6 Jun 2019 15:57:30 +0000
On Thu, Jun 06, 2019 at 03:55:01PM +0000, Andreas Gustafsson wrote:
> So, strange as it may seem, it now looks like the key to reliably
> reproducing the lost interrupts that in turn reproduce the hang is not
> the qemu version, but the NetBSD version that qemu runs on (and/or was
> built for), specifically, that version being NetBSD 8.x. This also
> explains why I haven't been able to reproduce the problem on a Linux
> host.
More likely is the different compiler used to build the version now.
Martin
From: Michael van Elst <mlelstv@serpens.de>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/54166: NetBSD tests hang after lost IDE interrupt under qemu
Date: Thu, 6 Jun 2019 21:54:56 +0200
On Thu, Jun 06, 2019 at 06:51:58PM +0300, Andreas Gustafsson wrote:
>
> So, strange as it may seem, it now looks like the key to reliably
> reproducing the lost interrupts that in turn reproduce the hang is not
> the qemu version, but the NetBSD version that qemu runs on (and/or was
> built for), specifically, that version being NetBSD 8.x.
macallan@ got the same issue on macppc hardware and just found the bug.
Greetings,
--
Michael van Elst
Internet: mlelstv@serpens.de
"A potential Snark may lurk in every tree."
State-Changed-From-To: open->closed
State-Changed-By: gson@NetBSD.org
State-Changed-When: Sat, 08 Jun 2019 06:31:23 +0000
State-Changed-Why:
Fixed in src/sys/dev/ata/wd.c 1.451, thanks!
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.