NetBSD Problem Report #52199
From www@NetBSD.org Thu Apr 27 19:07:01 2017
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
by mollari.NetBSD.org (Postfix) with ESMTPS id 7875C7A209
for <gnats-bugs@gnats.NetBSD.org>; Thu, 27 Apr 2017 19:07:01 +0000 (UTC)
Message-Id: <20170427190700.4C86D7A2B0@mollari.NetBSD.org>
Date: Thu, 27 Apr 2017 19:07:00 +0000 (UTC)
From: code@boerschig.net
Reply-To: code@boerschig.net
To: gnats-bugs@NetBSD.org
Subject: vioscsi kernel panic on a KVM hypervisor
X-Send-Pr-Version: www-1.0
>Number: 52199
>Category: kern
>Synopsis: vioscsi kernel panic on a KVM hypervisor
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: jdolecek
>State: closed
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Thu Apr 27 19:10:00 +0000 2017
>Closed-Date: Wed Jul 05 07:51:55 +0000 2017
>Last-Modified: Wed Jul 05 07:51:55 +0000 2017
>Originator: MB
>Release: 7.1
>Organization:
>Environment:
>Description:
I installed NetBSD on a "virtual server" on an ISP's Linux/KVM-based hypervisor.
The installation using a 'rescue environment' works fine.
However, he KVM in normal operation supports only virtio-scsi for disks.
After a couple of minutes the machine dies, I tried updating the kernel but that corrupted the kernel file and/or the filesystem.
Using a VNC-console I captured the following (transcript from screenshot):
[...lots of HBA errors ...]
sd0(vioscsi0:0:1:0): generic HBA error
sd0(vioscsi0:0:1:0): adapter resource shortage
sd0(vioscsi0:0:1:0): generic HBA error
sd0(vioscsi0:0:1:0): generic HBA error
fatal page fault in supervisor mode
trap type 6 code 0 rip ffffffff80974049 cs 8 rflags 10286 cr7 70 ilevel 6 rsp fffffe8002faedc8
curlwp 0xfffffe803fe12000 pid 0.4 lowest kstack 0xfffffe8002fac2c0
kernel: page fault trap, code=0
Stopped in pid 0.4 (system) at netbsd:vioscsi_vq_done+0x13c: mov 70(%r12),%eax
db{0}> bt
vioscsi_vq_done() at netbsd:vioscsi_vq_done+0x13c
virtio_vq_intr() at netbsd:virtio_vq_intr+0x70
virtio_intr() at netbsd:virtio_intr+0x38
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x19
Xintr_ioapic_level5() at netbsd:Xintr_ioapic_level5+0xf2
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffffe8002faeff0
Xsoftintr() at netbsd:Xsoftintr+0x4f
--- interrupt ---
0:
db{0}>
>How-To-Repeat:
Run a virtual machine with NetBSD 7.1 on a Linux/KVM server with virtio-scsi enabled.
>Fix:
>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->feedback
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Thu, 27 Apr 2017 19:24:04 +0000
State-Changed-Why:
Can you please try booting with kernel from -current sources? You can download
it from http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/, for example
http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/201704270800Z/amd64/binary/kernel/
From: =?UTF-8?B?SmFyb23DrXIgRG9sZcSNZWs=?= <jaromir.dolecek@gmail.com>
To: gnats-bugs@netbsd.org
Cc: code@boerschig.net
Subject: Re: kern/52199 (vioscsi kernel panic on a KVM hypervisor)
Date: Thu, 27 Apr 2017 21:25:57 +0200
Forgot to say - there were some improvements done to the driver on
HEAD which were not ported to 7.1 branch yet, so using -current kernel
might resolve your problem.
Jaromir
2017-04-27 21:24 GMT+02:00 <jdolecek@netbsd.org>:
> Synopsis: vioscsi kernel panic on a KVM hypervisor
>
> State-Changed-From-To: open->feedback
> State-Changed-By: jdolecek@NetBSD.org
> State-Changed-When: Thu, 27 Apr 2017 19:24:04 +0000
> State-Changed-Why:
> Can you please try booting with kernel from -current sources? You can download
> it from http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/, for example
> http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/201704270800Z/amd64/binary/kernel/
>
>
>
From: MB <code@boerschig.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/52199 (vioscsi kernel panic on a KVM hypervisor)
Date: Thu, 27 Apr 2017 23:32:57 +0200
Well, booting a 7.99.70 kernel fixes the immediate crash, however
copying 160mb of tarballs takes forever , and a lot of the
"sd0(vioscsi0:0:1:0) adapter resource shortage" messages appear.
Nothing I could run in production...
I asked the ISP for more information on the KVM version used.
Am 27.04.2017 um 21:30 schrieb Jaromír Doleček:
> The following reply was made to PR kern/52199; it has been noted by GNATS.
>
> From: =?UTF-8?B?SmFyb23DrXIgRG9sZcSNZWs=?= <jaromir.dolecek@gmail.com>
> To: gnats-bugs@netbsd.org
> Cc: code@boerschig.net
> Subject: Re: kern/52199 (vioscsi kernel panic on a KVM hypervisor)
> Date: Thu, 27 Apr 2017 21:25:57 +0200
>
> Forgot to say - there were some improvements done to the driver on
> HEAD which were not ported to 7.1 branch yet, so using -current kernel
> might resolve your problem.
>
> Jaromir
>
> 2017-04-27 21:24 GMT+02:00 <jdolecek@netbsd.org>:
> > Synopsis: vioscsi kernel panic on a KVM hypervisor
> >
> > State-Changed-From-To: open->feedback
> > State-Changed-By: jdolecek@NetBSD.org
> > State-Changed-When: Thu, 27 Apr 2017 19:24:04 +0000
> > State-Changed-Why:
> > Can you please try booting with kernel from -current sources? You can download
> > it from http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/, for example
> > http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/201704270800Z/amd64/binary/kernel/
> >
> >
> >
>
>
From: MB <code@boerschig.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/52199 (vioscsi kernel panic on a KVM hypervisor)
Date: Fri, 28 Apr 2017 19:47:27 +0200
FWIW, the problem can be reproduced independent of the Linux version: my provider uses Linux 4.4.X/Qemu 2.5, and I can reproduce it on a Ubuntu 17.04 (Linux 4.10.X/Qemu 2.8) machine as follows:
$ qemu-img create nbsd.img 10G
$ qemu-system-x86_64 -drive if=none,id=hd,file=nbsd.img,format=raw -device virtio-scsi-pci,id=scsi --enable-kvm -device scsi-hd,drive=hd -cdrom NetBSD-7.99.70-amd64.iso
# for scsi-layer tracing add -d trace:*scsi*
the sysinst installation stalls and the "adapter resource shortage" messages appear.
Am 28.04.2017 um 00:50 schrieb MB:
> The following reply was made to PR kern/52199; it has been noted by GNATS.
>
> From: MB <code@boerschig.net>
> To: gnats-bugs@NetBSD.org
> Cc:
> Subject: Re: kern/52199 (vioscsi kernel panic on a KVM hypervisor)
> Date: Thu, 27 Apr 2017 23:32:57 +0200
>
> Well, booting a 7.99.70 kernel fixes the immediate crash, however
> copying 160mb of tarballs takes forever , and a lot of the
> "sd0(vioscsi0:0:1:0) adapter resource shortage" messages appear.
> Nothing I could run in production...
> I asked the ISP for more information on the KVM version used.
>
> Am 27.04.2017 um 21:30 schrieb Jaromír Doleček:
> > The following reply was made to PR kern/52199; it has been noted by GNATS.
> >
> > From: =?UTF-8?B?SmFyb23DrXIgRG9sZcSNZWs=?= <jaromir.dolecek@gmail.com>
> > To: gnats-bugs@netbsd.org
> > Cc: code@boerschig.net
> > Subject: Re: kern/52199 (vioscsi kernel panic on a KVM hypervisor)
> > Date: Thu, 27 Apr 2017 21:25:57 +0200
> >
> > Forgot to say - there were some improvements done to the driver on
> > HEAD which were not ported to 7.1 branch yet, so using -current kernel
> > might resolve your problem.
> >
> > Jaromir
> >
> > 2017-04-27 21:24 GMT+02:00 <jdolecek@netbsd.org>:
> > > Synopsis: vioscsi kernel panic on a KVM hypervisor
> > >
> > > State-Changed-From-To: open->feedback
> > > State-Changed-By: jdolecek@NetBSD.org
> > > State-Changed-When: Thu, 27 Apr 2017 19:24:04 +0000
> > > State-Changed-Why:
> > > Can you please try booting with kernel from -current sources? You can download
> > > it from http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/, for example
> > > http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/201704270800Z/amd64/binary/kernel/
> > >
> > >
> > >
> >
> >
>
>
Responsible-Changed-From-To: kern-bug-people->jdolecek
Responsible-Changed-By: jdolecek@NetBSD.org
Responsible-Changed-When: Sat, 29 Apr 2017 18:52:40 +0000
Responsible-Changed-Why:
I'll look at this. Unfortunately very likely only after 2017-05-13, busy
until then.
State-Changed-From-To: feedback->open
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Sat, 29 Apr 2017 18:52:40 +0000
State-Changed-Why:
Feedback provided. -current driver doesn't panic, but it also doesn't work.
In my previous testing I've seen these resource shortage messages appear
several times and then stop, then things worked. Behaviour might be different
with bigger disk or when using the kvm. It needs to be tracked down.
From: "Jaromir Dolecek" <jdolecek@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/52199 CVS commit: src/sys/dev/pci
Date: Sat, 13 May 2017 20:17:43 +0000
Module Name: src
Committed By: jdolecek
Date: Sat May 13 20:17:43 UTC 2017
Modified Files:
src/sys/dev/pci: vioscsi.c
Log Message:
fix number of maximum segments in the queue allocation so that
MAXPHYS-sized tranfer fits inside the indirect descriptor; avoids the
!indirect path in virtio_enqueue_reserve() for these transfers,
which runs out of the vq_entries and triggers XS_RESOURCE_SHORTAGE for
bigger number of outstanding tranfers
fixes PR kern/52199 by MB
To generate a diff of this commit:
cvs rdiff -u -r1.16 -r1.17 src/sys/dev/pci/vioscsi.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
State-Changed-From-To: open->feedback
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Sat, 13 May 2017 20:38:08 +0000
State-Changed-Why:
A fix was committed in src/sys/dev/pci/vioscsi.c revision 1.17. Can you
please check with updated -current kernel?
From: MB <code@boerschig.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/52199 (vioscsi kernel panic on a KVM hypervisor)
Date: Sun, 14 May 2017 21:21:42 +0200
Am 13.05.2017 um 22:38 schrieb jdolecek@NetBSD.org:
> Synopsis: vioscsi kernel panic on a KVM hypervisor
>
> State-Changed-From-To: open->feedback
> State-Changed-By: jdolecek@NetBSD.org
> State-Changed-When: Sat, 13 May 2017 20:38:08 +0000
> State-Changed-Why:
> A fix was committed in src/sys/dev/pci/vioscsi.c revision 1.17. Can you
> please check with updated -current kernel?
>
>
>
Thanks for your work.
The installation works now, allthough the "resource shortage" messages appear.
The 'resource shortage' messages also seem to affect the write performance.
However, I see some strange behavior and I'm not sure if it is related to the vioscsi driver:
After installation, I put some load on the virtual disk, and the whole system seems to freeze.
Once it continued after a couple of minutes, currently it is wedged for 10min+..
FWIW, I could produce this on a 10gb test image, default installation, with the following, but only on 3 out of 8 attempts:
dd if=/dev/urandom of=/home/test1.bin bs=1m count=128
ls -lahR /
for i in $(seq 0 8);do dd if=/home/test1.bin of=/home/foo$i ;done
#when the system freezes, here are no "adapter resource shortage" messages, they only appear when the system works without freeze
ls -lahR /
for i in $(seq 0 8);do dd if=/home/test1.bin of=/home/foo$i ;done
#this freezes, no keyboard input, not even '(qemu) sendkey ctrl-alt-f2'
From: =?UTF-8?B?SmFyb23DrXIgRG9sZcSNZWs=?= <jaromir.dolecek@gmail.com>
To: gnats-bugs@netbsd.org
Cc: code@boerschig.net
Subject: Re: kern/52199 (vioscsi kernel panic on a KVM hypervisor)
Date: Sun, 14 May 2017 22:24:06 +0200
--001a113f23fc66c3a5054f81b806
Content-Type: text/plain; charset="UTF-8"
Could you please transcribe the kernel messages which appear just before
the resource shortages? It should something along '%s: error %d loading DMA
map' or 'error reserving %d (nsegs %d)'.
Jaromir
2017-05-14 21:25 GMT+02:00 MB <code@boerschig.net>:
> The following reply was made to PR kern/52199; it has been noted by GNATS.
>
> From: MB <code@boerschig.net>
> To: gnats-bugs@NetBSD.org
> Cc:
> Subject: Re: kern/52199 (vioscsi kernel panic on a KVM hypervisor)
> Date: Sun, 14 May 2017 21:21:42 +0200
>
> Am 13.05.2017 um 22:38 schrieb jdolecek@NetBSD.org:
> > Synopsis: vioscsi kernel panic on a KVM hypervisor
> >
> > State-Changed-From-To: open->feedback
> > State-Changed-By: jdolecek@NetBSD.org
> > State-Changed-When: Sat, 13 May 2017 20:38:08 +0000
> > State-Changed-Why:
> > A fix was committed in src/sys/dev/pci/vioscsi.c revision 1.17. Can you
> > please check with updated -current kernel?
> >
> >
> >
> Thanks for your work.
> The installation works now, allthough the "resource shortage" messages
> appear.
> The 'resource shortage' messages also seem to affect the write
> performance.
> However, I see some strange behavior and I'm not sure if it is related to
> the vioscsi driver:
>
> After installation, I put some load on the virtual disk, and the whole
> system seems to freeze.
> Once it continued after a couple of minutes, currently it is wedged for
> 10min+..
> FWIW, I could produce this on a 10gb test image, default installation,
> with the following, but only on 3 out of 8 attempts:
>
> dd if=/dev/urandom of=/home/test1.bin bs=1m count=128
> ls -lahR /
> for i in $(seq 0 8);do dd if=/home/test1.bin of=/home/foo$i ;done
> #when the system freezes, here are no "adapter resource shortage"
> messages, they only appear when the system works without freeze
> ls -lahR /
> for i in $(seq 0 8);do dd if=/home/test1.bin of=/home/foo$i ;done
> #this freezes, no keyboard input, not even '(qemu) sendkey ctrl-alt-f2'
>
>
--001a113f23fc66c3a5054f81b806
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Could you please transcribe the kernel messages which appe=
ar just before the resource shortages? It should something along '<span=
style=3D"font-family:monaco;font-size:11px;font-variant-ligatures:no-commo=
n-ligatures;color:rgb(213,59,211)">%s</span><span style=3D"color:rgb(195,55=
,32);font-family:monaco;font-size:11px;font-variant-ligatures:no-common-lig=
atures">: error </span><span style=3D"font-family:monaco;font-size:11px;fon=
t-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">%d</span><sp=
an style=3D"color:rgb(195,55,32);font-family:monaco;font-size:11px;font-var=
iant-ligatures:no-common-ligatures"> loading DMA map</span>' or '<s=
pan style=3D"color:rgb(195,55,32);font-family:monaco;font-size:11px;font-va=
riant-ligatures:no-common-ligatures">error reserving </span><span style=3D"=
font-family:monaco;font-size:11px;font-variant-ligatures:no-common-ligature=
s;color:rgb(213,59,211)">%d</span><span style=3D"color:rgb(195,55,32);font-=
family:monaco;font-size:11px;font-variant-ligatures:no-common-ligatures"> (=
nsegs </span><span style=3D"font-family:monaco;font-size:11px;font-variant-=
ligatures:no-common-ligatures;color:rgb(213,59,211)">%d</span><span style=
=3D"color:rgb(195,55,32);font-family:monaco;font-size:11px;font-variant-lig=
atures:no-common-ligatures">)</span>'.<br><div><br></div><div>Jaromir</=
div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">2017-05=
-14 21:25 GMT+02:00 MB <span dir=3D"ltr"><<a href=3D"mailto:code@boersch=
ig.net" target=3D"_blank">code@boerschig.net</a>></span>:<br><blockquote=
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc soli=
d;padding-left:1ex"><span class=3D"">The following reply was made to PR ker=
n/52199; it has been noted by GNATS.<br>
<br>
From: MB <<a href=3D"mailto:code@boerschig.net">code@boerschig.net</a>&g=
t;<br>
To: gnats-bugs@NetBSD.org<br>
Cc:<br>
</span>Subject: Re: kern/52199 (vioscsi kernel panic on a KVM hypervisor)<b=
r>
Date: Sun, 14 May 2017 21:21:42 +0200<br>
<div><div class=3D"h5"><br>
=C2=A0Am 13.05.2017 um 22:38 schrieb jdolecek@NetBSD.org:<br>
=C2=A0> Synopsis: vioscsi kernel panic on a KVM hypervisor<br>
=C2=A0><br>
=C2=A0> State-Changed-From-To: open->feedback<br>
=C2=A0> State-Changed-By: jdolecek@NetBSD.org<br>
=C2=A0> State-Changed-When: Sat, 13 May 2017 20:38:08 +0000<br>
=C2=A0> State-Changed-Why:<br>
=C2=A0> A fix was committed in src/sys/dev/pci/vioscsi.c revision 1.17. =
Can you<br>
=C2=A0> please check with updated -current kernel?<br>
=C2=A0><br>
=C2=A0><br>
=C2=A0><br>
</div></div>=C2=A0Thanks for your work.<br>
=C2=A0The installation works now, allthough the "resource shortage&quo=
t; messages appear.<br>
=C2=A0The 'resource shortage' messages also seem to affect the writ=
e performance.<br>
=C2=A0However, I see some strange behavior and I'm not sure if it is re=
lated to the vioscsi driver:<br>
<br>
=C2=A0After installation, I put some load on the virtual disk, and the whol=
e system seems to freeze.<br>
=C2=A0Once it continued after a couple of minutes, currently it is wedged f=
or 10min+..<br>
=C2=A0FWIW, I could produce this on a 10gb test image, default installation=
, with the following, but only on 3 out of 8 attempts:<br>
<br>
=C2=A0dd if=3D/dev/urandom of=3D/home/test1.bin bs=3D1m count=3D128<br>
=C2=A0ls -lahR /<br>
=C2=A0for i in $(seq 0 8);do dd if=3D/home/test1.bin of=3D/home/foo$i ;done=
<br>
=C2=A0#when the system freezes, here are no "adapter resource shortage=
" messages, they only appear when the system works without freeze<br>
=C2=A0ls -lahR /<br>
=C2=A0for i in $(seq 0 8);do dd if=3D/home/test1.bin of=3D/home/foo$i ;done=
<br>
=C2=A0#this freezes, no keyboard input, not even '(qemu) sendkey ctrl-a=
lt-f2'<br>
<br>
</blockquote></div><br></div>
--001a113f23fc66c3a5054f81b806--
From: "Jaromir Dolecek" <jdolecek@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc:
Subject: PR/52199 CVS commit: src/sys/dev/pci
Date: Mon, 15 May 2017 21:30:37 +0000
Module Name: src
Committed By: jdolecek
Date: Mon May 15 21:30:37 UTC 2017
Modified Files:
src/sys/dev/pci: vioscsi.c
Log Message:
free the virtio slot _before_ calling scsipi_done(), so that driver is able
to accept another request immediatelly and doesn't trigger spurious
XS_RESOURCE_SHORTAGE
part of fix for PR kern/52199
To generate a diff of this commit:
cvs rdiff -u -r1.18 -r1.19 src/sys/dev/pci/vioscsi.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
From: =?UTF-8?B?SmFyb23DrXIgRG9sZcSNZWs=?= <jaromir.dolecek@gmail.com>
To: gnats-bugs@netbsd.org
Cc: code@boerschig.net
Subject: Re: kern/52199 (vioscsi kernel panic on a KVM hypervisor)
Date: Tue, 16 May 2017 00:02:15 +0200
--001a114904e844ac46054f973543
Content-Type: text/plain; charset="UTF-8"
Hello,
committed another fix for the resource shortage, messages and related i/o
delays should be gone for good. You can grab just the new kernel from
NetBSD-daily once it gets built.
Thanks for the reproduction instructions. I did see the resource shortages
during sysinst extracting, but unfortunately I wasn't able to trigger the
lockup using the dd's so far. I'm using QEMU 2.9, and I haven't try with
KVM yet. I don't see anything related in release notes for 2.6-2.9, so
maybe my non-kvm QEMU is too slow to trigger the problem.
Can you let me know if the latest fix by chance helps with the lockup too?
And can you please try without --enable-kvm too?
Another thing to try would be to compile&run custom kernel with LOCKDEBUG
and DEBUG options enabled. If the freeze is due to locking issue, those
options help to find the problematic spot.
Jaromir
2017-05-14 21:25 GMT+02:00 MB <code@boerschig.net>:
> Thanks for your work.
> The installation works now, allthough the "resource shortage" messages
> appear.
> The 'resource shortage' messages also seem to affect the write
> performance.
> However, I see some strange behavior and I'm not sure if it is related to
> the vioscsi driver:
>
> After installation, I put some load on the virtual disk, and the whole
> system seems to freeze.
> Once it continued after a couple of minutes, currently it is wedged for
> 10min+..
> FWIW, I could produce this on a 10gb test image, default installation,
> with the following, but only on 3 out of 8 attempts:
>
> dd if=/dev/urandom of=/home/test1.bin bs=1m count=128
> ls -lahR /
> for i in $(seq 0 8);do dd if=/home/test1.bin of=/home/foo$i ;done
> #when the system freezes, here are no "adapter resource shortage"
> messages, they only appear when the system works without freeze
> ls -lahR /
> for i in $(seq 0 8);do dd if=/home/test1.bin of=/home/foo$i ;done
> #this freezes, no keyboard input, not even '(qemu) sendkey ctrl-alt-f2'
>
>
--001a114904e844ac46054f973543
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Hello,<div><br></div><div>committed another fix for the re=
source shortage, messages and related i/o delays should be gone for good. Y=
ou can grab just the new kernel from NetBSD-daily once it gets built.</div>=
<div><br></div><div>Thanks for the reproduction instructions. I did see the=
resource shortages during sysinst extracting, but unfortunately I wasn'=
;t able to trigger the lockup using the dd's so far. I'm using QEMU=
2.9, and I haven't try with KVM yet. I don't see anything related =
in release notes for 2.6-2.9, so maybe my non-kvm QEMU is too slow to trigg=
er the problem.</div><div><br></div><div>Can you let me know if the latest =
fix by chance helps with the lockup too?</div><div><br></div><div>And can y=
ou please try without --enable-kvm too?</div><div><br></div><div>Another th=
ing to try would be to compile&run custom kernel with LOCKDEBUG and DEB=
UG options enabled. If the freeze is due to locking issue, those options he=
lp to find the problematic spot.</div><div><br></div><div>Jaromir</div><div=
><br></div><div><div class=3D"gmail_extra"><div class=3D"gmail_quote">2017-=
05-14 21:25 GMT+02:00 MB <span dir=3D"ltr"><<a href=3D"mailto:code@boers=
chig.net" target=3D"_blank">code@boerschig.net</a>></span>:<br><blockquo=
te class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc so=
lid;padding-left:1ex">=C2=A0Thanks for your work.<br>
=C2=A0The installation works now, allthough the "resource shortage&quo=
t; messages appear.<br>
=C2=A0The 'resource shortage' messages also seem to affect the writ=
e performance.<br>
=C2=A0However, I see some strange behavior and I'm not sure if it is re=
lated to the vioscsi driver:<br>
<br>
=C2=A0After installation, I put some load on the virtual disk, and the whol=
e system seems to freeze.<br>
=C2=A0Once it continued after a couple of minutes, currently it is wedged f=
or 10min+..<br>
=C2=A0FWIW, I could produce this on a 10gb test image, default installation=
, with the following, but only on 3 out of 8 attempts:<br>
<br>
=C2=A0dd if=3D/dev/urandom of=3D/home/test1.bin bs=3D1m count=3D128<br>
=C2=A0ls -lahR /<br>
=C2=A0for i in $(seq 0 8);do dd if=3D/home/test1.bin of=3D/home/foo$i ;done=
<br>
=C2=A0#when the system freezes, here are no "adapter resource shortage=
" messages, they only appear when the system works without freeze<br>
=C2=A0ls -lahR /<br>
=C2=A0for i in $(seq 0 8);do dd if=3D/home/test1.bin of=3D/home/foo$i ;done=
<br>
=C2=A0#this freezes, no keyboard input, not even '(qemu) sendkey ctrl-a=
lt-f2'<br>
<br>
</blockquote></div><br></div></div></div>
--001a114904e844ac46054f973543--
From: MB <code@boerschig.net>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: kern/52199 (vioscsi kernel panic on a KVM hypervisor)
Date: Tue, 16 May 2017 23:00:11 +0200
On 16.05.2017 00:05, Jaromír Doleček wrote:
> The following reply was made to PR kern/52199; it has been noted by GNATS.
>
> From: =?UTF-8?B?SmFyb23DrXIgRG9sZcSNZWs=?= <jaromir.dolecek@gmail.com>
> To: gnats-bugs@netbsd.org
> Cc: code@boerschig.net
> Subject: Re: kern/52199 (vioscsi kernel panic on a KVM hypervisor)
> Date: Tue, 16 May 2017 00:02:15 +0200
>
> --001a114904e844ac46054f973543
> Content-Type: text/plain; charset="UTF-8"
>
> Hello,
>
> committed another fix for the resource shortage, messages and related i/o
> delays should be gone for good. You can grab just the new kernel from
> NetBSD-daily once it gets built.
>
> Thanks for the reproduction instructions. I did see the resource shortages
> during sysinst extracting, but unfortunately I wasn't able to trigger the
> lockup using the dd's so far. I'm using QEMU 2.9, and I haven't try with
> KVM yet. I don't see anything related in release notes for 2.6-2.9, so
> maybe my non-kvm QEMU is too slow to trigger the problem.
>
> Can you let me know if the latest fix by chance helps with the lockup too?
>
> And can you please try without --enable-kvm too?
>
> Another thing to try would be to compile&run custom kernel with LOCKDEBUG
> and DEBUG options enabled. If the freeze is due to locking issue, those
> options help to find the problematic spot.
>
> Jaromir
Hi,
looks like you fixed the messages and the lockups.
Just tried to reproduce the freeze with vioscsi.c version 1.19 but could
not reproduce it. FWIW, i tried only with kvm enabled, and performance
looks reasonable good now, too.
I tried dd in an infinite loop (2hours...) and also added extracting the
pkgsrc tarballs for good measure -- so far not a single resource
shortage or lockup.
Again, thanks a lot.
PS: any chance these fixes will get pulled up in a stable release?
From: =?UTF-8?B?SmFyb23DrXIgRG9sZcSNZWs=?= <jaromir.dolecek@gmail.com>
To: gnats-bugs@netbsd.org
Cc: Jaromir Dolecek <jdolecek@netbsd.org>, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org,
code@boerschig.net
Subject: Re: kern/52199 (vioscsi kernel panic on a KVM hypervisor)
Date: Wed, 17 May 2017 00:48:21 +0200
Thanks for confirmation.
There were some intermediate changes to virtio drivers, so pullups to
netbsd-7 would be unfortunately a bit difficult.
Jaromir
2017-05-16 23:05 GMT+02:00 MB <code@boerschig.net>:
> The following reply was made to PR kern/52199; it has been noted by GNATS=
.
>
> From: MB <code@boerschig.net>
> To: gnats-bugs@NetBSD.org
> Cc:
> Subject: Re: kern/52199 (vioscsi kernel panic on a KVM hypervisor)
> Date: Tue, 16 May 2017 23:00:11 +0200
>
> On 16.05.2017 00:05, Jarom=C3=ADr Dole=C4=8Dek wrote:
> > The following reply was made to PR kern/52199; it has been noted by GN=
ATS.
> >
> > From: =3D?UTF-8?B?SmFyb23DrXIgRG9sZcSNZWs=3D?=3D <jaromir.dolecek@gmai=
l.com>
> > To: gnats-bugs@netbsd.org
> > Cc: code@boerschig.net
> > Subject: Re: kern/52199 (vioscsi kernel panic on a KVM hypervisor)
> > Date: Tue, 16 May 2017 00:02:15 +0200
> >
> > --001a114904e844ac46054f973543
> > Content-Type: text/plain; charset=3D"UTF-8"
> >
> > Hello,
> >
> > committed another fix for the resource shortage, messages and related=
i/o
> > delays should be gone for good. You can grab just the new kernel from
> > NetBSD-daily once it gets built.
> >
> > Thanks for the reproduction instructions. I did see the resource shor=
tages
> > during sysinst extracting, but unfortunately I wasn't able to trigger=
the
> > lockup using the dd's so far. I'm using QEMU 2.9, and I haven't try w=
ith
> > KVM yet. I don't see anything related in release notes for 2.6-2.9, s=
o
> > maybe my non-kvm QEMU is too slow to trigger the problem.
> >
> > Can you let me know if the latest fix by chance helps with the lockup=
too?
> >
> > And can you please try without --enable-kvm too?
> >
> > Another thing to try would be to compile&run custom kernel with LOCKD=
EBUG
> > and DEBUG options enabled. If the freeze is due to locking issue, tho=
se
> > options help to find the problematic spot.
> >
> > Jaromir
>
> Hi,
>
> looks like you fixed the messages and the lockups.
> Just tried to reproduce the freeze with vioscsi.c version 1.19 but could
> not reproduce it. FWIW, i tried only with kvm enabled, and performance
> looks reasonable good now, too.
> I tried dd in an infinite loop (2hours...) and also added extracting the
> pkgsrc tarballs for good measure -- so far not a single resource
> shortage or lockup.
>
> Again, thanks a lot.
>
> PS: any chance these fixes will get pulled up in a stable release?
>
State-Changed-From-To: feedback->closed
State-Changed-By: jdolecek@NetBSD.org
State-Changed-When: Wed, 05 Jul 2017 07:51:55 +0000
State-Changed-Why:
Fix is present on netbsd-8 branch, so will be part of 8.0.
>Unformatted:
(Contact us)
$NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014
The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
Home