NetBSD Problem Report #54166

From gson@gson.org  Mon May  6 07:28:39 2019
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 89C117A169
	for <gnats-bugs@gnats.NetBSD.org>; Mon,  6 May 2019 07:28:39 +0000 (UTC)
Message-Id: <20190506072834.4A1BC989F84@guava.gson.org>
Date: Mon,  6 May 2019 10:28:34 +0300 (EEST)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@NetBSD.org
Subject: NetBSD tests hang after lost IDE interrupt under qemu
X-Send-Pr-Version: 3.95

>Number:         54166
>Category:       kern
>Synopsis:       NetBSD tests hang after lost IDE interrupt under qemu
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    mlelstv
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon May 06 07:30:00 +0000 2019
>Closed-Date:    Sat Jun 08 06:31:23 +0000 2019
>Last-Modified:  Sat Jun 08 06:31:23 +0000 2019
>Originator:     Andreas Gustafsson
>Release:        NetBSD-current, source date >= 2019.03.19.16.56.29
>Organization:

>Environment:
System: NetBSD
Architecture: i386
Machine: i386
>Description:

Running the ATF tests on recent versions of NetBSD/i386, /amd64, or
/sparc64 under recent versions of qemu frequently ends with the NetBSD
IDE disk driver reporting a lost interrupt, after which the tests
produce no further output.  For example:

  lib/librt/t_sem (418/799): 4 test cases
      basic: [0.285119s] Passed.
      child: piixide0:0:0: lost interrupt
  [ 5969.7132325]		type: ata tc_bcount: 2048 tc_skip: 0
  piixide0:0:0: bus-master DMA error: missing interrupt, status=0x61
  [ 5969.7953625] wd0a: DMA error writing fsbn 128 of 128-131 (wd0 bn 2176; cn 2 tn 2 sn 34), xfer f5c, retry 0
  [ 5971.7754220] wd0: soft error (corrected) xfer f5c

This was extracted from:

  http://www.gson.org/netbsd/bugs/build/i386/2019/2019.04.26.11.51.56/test.log

More reports of similar hangs:

  http://www.gson.org/netbsd/bugs/build/i386/2019/2019.04.11.14.47.06/test.log
  http://www.gson.org/netbsd/bugs/build/i386/2019/2019.04.12.09.29.26/test.log
  http://www.gson.org/netbsd/bugs/build/sparc64/2019/2019.04.26.14.36.40/test.log
  http://releng.netbsd.org/b5reports/sparc64/2019/2019.03.05.15.18.59/test.log

This has been tricky to track down because it only happens with the
combination of a recent qemu *and* a recent -current, and even then it
does not fail in every test run, only most of them.

Specifically, the lost interrupts appear to be related to the qemu
version, whereas the inability to recover from those lost interrupts
appears to be related to the NetBSD-current version.  I believe I have
now identified the commit where the latter problem started, namely

  2019.03.19.16.56.29 mlelstv src/sys/dev/ata/wd.c 1.446

Log output showing the tests running to completion before said commit
and hanging after it can be found at

  http://www.gson.org/netbsd/bugs/build/i386/commits-2019.03.html#2019.03.19.16.56.29

This PR is intended to track the specific issue of the test runs
hanging.  For the lost interrupts themselves, there is already
port-sparc64/54035.

>How-To-Repeat:

Run the ATF tests on NetBSD-current/i386 under qemu 3.1.0 or 4.0.0,
possibly multiple times.

>Fix:

>Release-Note:

>Audit-Trail:

Responsible-Changed-From-To: kern-bug-people->mlelstv
Responsible-Changed-By: mlelstv@NetBSD.org
Responsible-Changed-When: Sat, 18 May 2019 08:16:25 +0000
Responsible-Changed-Why:
Mine


From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/54166: NetBSD tests hang after lost IDE interrupt under qemu
Date: Tue, 4 Jun 2019 19:34:46 +0300

 One more data point: the TNF i386 testbed on babylon5.netbsd.org,
 which runs under qemu 2.11.1, rarely gets the "bus-master DMA error:
 missing interrupt" errors, but they do happen, and one just happened
 while testing sources from yesterday:

   http://releng.netbsd.org/b5reports/i386/2019/2019.06.03.09.58.31/test.log

 And sure enough, it caused the test run to hang.

 The previous one was in October, and it did not cause the test run to hang:

   http://releng.netbsd.org/b5reports/i386/2018/2018.10.16.11.28.30/test.log

 -- 
 Andreas Gustafsson, gson@gson.org

From: Andreas Gustafsson <gson@gson.org>
To: Michael van Elst <mlelstv@serpens.de>, gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/54166: NetBSD tests hang after lost IDE interrupt under qemu
Date: Thu, 6 Jun 2019 18:51:58 +0300

 Since babylon5 was upgraded from NetBSD 7 to 8.1 a few days ago, most
 i386 and amd64 test runs have reported lost IDE interrupts and hung,
 even though it is still running qemu version 2.11.1 (rebuilt for
 NetBSD 8).

 So, strange as it may seem, it now looks like the key to reliably
 reproducing the lost interrupts that in turn reproduce the hang is not
 the qemu version, but the NetBSD version that qemu runs on (and/or was
 built for), specifically, that version being NetBSD 8.x.  This also
 explains why I haven't been able to reproduce the problem on a Linux
 host.
 -- 
 Andreas Gustafsson, gson@gson.org

From: Martin Husemann <martin@homeworld.netbsd.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: kern/54166: NetBSD tests hang after lost IDE interrupt under qemu
Date: Thu, 6 Jun 2019 15:57:30 +0000

 On Thu, Jun 06, 2019 at 03:55:01PM +0000, Andreas Gustafsson wrote:
 >  So, strange as it may seem, it now looks like the key to reliably
 >  reproducing the lost interrupts that in turn reproduce the hang is not
 >  the qemu version, but the NetBSD version that qemu runs on (and/or was
 >  built for), specifically, that version being NetBSD 8.x.  This also
 >  explains why I haven't been able to reproduce the problem on a Linux
 >  host.

 More likely is the different compiler used to build the version now.

 Martin

From: Michael van Elst <mlelstv@serpens.de>
To: Andreas Gustafsson <gson@gson.org>
Cc: gnats-bugs@NetBSD.org
Subject: Re: kern/54166: NetBSD tests hang after lost IDE interrupt under qemu
Date: Thu, 6 Jun 2019 21:54:56 +0200

 On Thu, Jun 06, 2019 at 06:51:58PM +0300, Andreas Gustafsson wrote:
 > 
 > So, strange as it may seem, it now looks like the key to reliably
 > reproducing the lost interrupts that in turn reproduce the hang is not
 > the qemu version, but the NetBSD version that qemu runs on (and/or was
 > built for), specifically, that version being NetBSD 8.x.

 macallan@ got the same issue on macppc hardware and just found the bug.


 Greetings,
 -- 
                                 Michael van Elst
 Internet: mlelstv@serpens.de
                                 "A potential Snark may lurk in every tree."

State-Changed-From-To: open->closed
State-Changed-By: gson@NetBSD.org
State-Changed-When: Sat, 08 Jun 2019 06:31:23 +0000
State-Changed-Why:
Fixed in src/sys/dev/ata/wd.c 1.451, thanks!


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.51 2026/08/10 02:28:17 riastradh Exp $
$NetBSD: gnats_config.sh,v 1.10 2026/05/13 22:00:09 riastradh Exp $
Copyright © 1994-2026 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.