NetBSD Problem Report #49395

From kilbi@kilbi.de  Sat Nov 15 19:07:50 2014
Return-Path: <kilbi@kilbi.de>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 466B1A65F7
	for <gnats-bugs@gnats.NetBSD.org>; Sat, 15 Nov 2014 19:07:50 +0000 (UTC)
Message-Id: <20141115190744.1A8F2390FD@mail.kilbi.de>
Date: Sat, 15 Nov 2014 20:07:43 +0100 (MET)
From: mk@kilbi.de
Reply-To: mk@kilbi.de
To: gnats-bugs@NetBSD.org
Subject: cubietruck freezes under i/o load on netbsd-7 and HEAD
X-Send-Pr-Version: 3.95

>Number:         49395
>Category:       port-evbarm
>Synopsis:       cubietruck freezes under i/o load on netbsd-7 and HEAD
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    port-evbarm-maintainer
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Nov 15 19:10:01 +0000 2014
>Closed-Date:    Thu Apr 30 03:53:45 +0000 2015
>Last-Modified:  Thu Apr 30 03:53:45 +0000 2015
>Originator:     Markus Kilbinger
>Release:        NetBSD 7.0_BETA
>Organization:

>Environment:


System: NetBSD ct1 7.0_BETA NetBSD 7.0_BETA (CUBIETRUCK) #0: Sat Nov 15 19:21:55 MET 2014 root@q:/usr/u/NetBSD/netbsd-7/src/sys/arch/evbarm/compile/CUBIETRUCK evbarm
Architecture: earmv7hf
Machine: evbarm
>Description:
	My cubietruck freezes during e. g. 'cvs -z2 update -dPA' in
	the machine's pkgsrc-tree.

	This happens under netbsd-7 BETA's _and_ HEAD's kernel (normal
	CUBIETRUCK configs) with a netbsd-7 userland.

	The NetBSD system is running on a connected 1 TB sata disk,
	the kernel is loaded from sdcard or via dhcp/tftp.

	The sata disk is populated with several 100 GB of data.

	The boot process runs flawlessly starting several daemons
	(e. g. postfix, dovecot, spamassassin, amavisd, clamd, pgsql).
	After that I can use the idle system quite normal.

	When I start 'cvs -z2 update -dPA' in its pkgsrc-tree folder
	(on the sata disk) the machine freezes after some time
	(secondes to some minutes): The network connections (ssh)
	become quite suddenly stuck. On the serial console pressing
	(pure) CRs repeats the login prompt quite normally, trying to
	login yields the same stuck situation.

	The hangup was not self-limited (waited about an hour).

	At that point I still can successfully ping the system (via
	ipv4 and ipv6) and enter DDB:

	Stopped in pid 1184.1 (getty) at        netbsd:cpu_Debugger+0x4:        bx      r14
	db{0}> show uvm
	Current UVM status:
	  pagesize=8192 (0x2000), pagemask=0x1fff, pageshift=13
	, ncolors=1  258898 VM pages: 152349 active, 78033 inactive, 8 wired, 7921 free
	  pages  45608 anon, 180741 file, 4041 exec
	  freemin=256, free-target=341, wired-max=86299
	  cpu0:
	    faults=267065, traps=559393, intrs=75562009, ctxswitch=744307153
	    softint=201536, syscalls=1116121
	  cpu1:
	    faults=258317, traps=3009273, intrs=305778776, ctxswitch=615049912
	    softint=69794, syscalls=1121207
	  fault counts:
	    noram=0, noanon=0, pgwait=0, pgrele=0
	    ok relocks(total)=1403(1403), anget(retrys)=35147795(0), amapcopy=27230
	    neighbor anon/obj pg=61493/238264, gets(lock/unlock)=71536/1403
	    cases: anon=35105352, anoncow=42444, obj=60644, prcopy=10891, przero=102748
	  daemon and swap counts:
	    woke=307478681, revs=10, scans=60772, obscans=10006, anscans=0
	    busy=0, freed=10006, reactivate=795, deactivate=138886
	    pageouts=0, pending=0, nswget=0
	    nswapdev=1, swpgavail=262143
	    swpages=262143, swpginuse=0, swpgonly=0, paging=0
	db{0}> bt
	0xbfd6d61c: netbsd:comintr+0xc
	0xbfd6d64c: netbsd:pic_dispatch+0xe8
	0xbfd6d6a4: netbsd:armgic_irq_handler+0xd4
	0xbfd6d714: netbsd:irq_entry+0x60
	0xbfd6d73c: netbsd:_splraise+0x3c
	0xbfd6d79c: netbsd:mutex_enter+0x294
	0xbfd6d7bc: netbsd:wakeup+0x44
	0xbfd6d7dc: netbsd:uvm_wait+0x5c
	0xbfd6d82c: netbsd:uvm_km_kmem_alloc+0x270
	0xbfd6d844: netbsd:pool_page_alloc+0x3c
	0xbfd6d884: netbsd:pool_grow+0x38
	0xbfd6d8bc: netbsd:pool_get+0x78
	0xbfd6d904: netbsd:pool_cache_get_slow+0x224
	0xbfd6d93c: netbsd:pool_cache_get_paddr+0x210
	0xbfd6d9bc: netbsd:ffs_loadvnode+0x250
	0xbfd6da24: netbsd:vcache_get+0x30c
	0xbfd6dad4: netbsd:ufs_lookup+0x7a8
	0xbfd6db0c: netbsd:VOP_LOOKUP+0x48
	0xbfd6db5c: netbsd:lookup_once+0x19c
	0xbfd6dc2c: netbsd:namei_tryemulroot+0x520
	0xbfd6dc64: netbsd:namei+0x34
	0xbfd6dd04: netbsd:check_exec+0x40
	0xbfd6dd6c: netbsd:execve_loadvm+0x1b8
	0xbfd6def4: netbsd:execve1+0x28
	0xbfd6df0c: netbsd:sys_execve+0x24
	0xbfd6df7c: netbsd:syscall+0x88
	0xbfd6dfac: netbsd:swi_handler+0x98
	db{0}> mach cpu 1
	kdb_trap: switching to cpu1
	Stopped in pid 3332.1 (cron) at netbsd:nullop:  mov     r0, #0x00000000
	db{1}> bt
	0xbdb4bd14: netbsd:mutex_enter+0xc
	0xbdb4bd64: netbsd:uvm_pagealloc_strat+0xb8
	0xbdb4bd84: netbsd:arm_pmap_alloc_poolpage+0x5c
	0xbdb4bdb4: netbsd:pmap_create+0x9c
	0xbdb4bddc: netbsd:uvmspace_init+0x6c
	0xbdb4be04: netbsd:uvmspace_alloc+0x44
	0xbdb4be34: netbsd:uvmspace_fork+0x2c
	0xbdb4be4c: netbsd:uvm_proc_fork+0x24
	0xbdb4bee4: netbsd:fork1+0x32c
	0xbdb4bf0c: netbsd:sys_fork+0x38
	0xbdb4bf7c: netbsd:syscall+0x88
	0xbdb4bfac: netbsd:swi_handler+0x98
	db{1}> reboot
	syncing disks...

	The 'syncing disks...' does not succeed, so I have to reset
	the machine.
	But: I've got _no_ filesystem corruption (UFS2) so far (did
	several fsck after the freeze / reset).

	Disabling MULTIPROCESSOR did not prevent from freeze.

>How-To-Repeat:

	Install / run a netbsd-7 evbarm-earmv7hf system on/from (a
	populated) sata-harddisk on a cubietruck and try to 'cvs
	update / checkout' a pkgsrc-tree on its harddisk. After some
	time (seconds to minutes) the system freezes ...

	A current HEAD kernel shows the same symptoms.

>Fix:
	Unknown.

>Release-Note:

>Audit-Trail:
From: Markus Kilbinger <mk@kilbi.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-evbarm/49395: cubietruck freezes under i/o load on netbsd-7
 and HEAD
Date: Sun, 16 Nov 2014 16:04:34 +0100

 After further playing with kernel configs I've found that deactivating

   options BUFQ_PRIOCSCAN

 prevents from the system freezes (cause / trigger?).

 Just as a hint for developers (change bug category?).

 Markus.

From: Markus Kilbinger <mk@kilbi.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: port-evbarm/49395: cubietruck freezes under i/o load on netbsd-7
 and HEAD
Date: Tue, 18 Nov 2014 22:51:00 +0100

 Todays HEAD and -netbsd-7 changes seem to have fixed the problem for me.

 So, you can close this PR.

 Markus.

State-Changed-From-To: open->closed
State-Changed-By: msaitoh@NetBSD.org
State-Changed-When: Wed, 19 Nov 2014 03:05:36 +0000
State-Changed-Why:
Submitter reported it's ok to close.
Thanks.


From: Markus Kilbinger <mk@kilbi.de>
To: gnats-bugs@netbsd.org
Cc: port-evbarm-maintainer@netbsd.org, netbsd-bugs@netbsd.org, 
	gnats-admin@netbsd.org, msaitoh@netbsd.org, Markus Kilbinger <mk@kilbi.de>
Subject: Re: port-evbarm/49395 (cubietruck freezes under i/o load on netbsd-7
 and HEAD)
Date: Thu, 4 Dec 2014 19:32:45 +0100

 2014-11-19 4:05 GMT+01:00  <msaitoh@netbsd.org>:
 > Synopsis: cubietruck freezes under i/o load on netbsd-7 and HEAD
 >
 > State-Changed-From-To: open->closed
 > State-Changed-By: msaitoh@NetBSD.org
 > State-Changed-When: Wed, 19 Nov 2014 03:05:36 +0000
 > State-Changed-Why:
 > Submitter reported it's ok to close.
 > Thanks.

 The 'freeze' is back for several days now, in netbsd-7 more than in HEAD!
 -> Can you re-open the PR, please!

 Markus.

State-Changed-From-To: closed->open
State-Changed-By: martin@NetBSD.org
State-Changed-When: Sat, 06 Dec 2014 11:16:55 +0000
State-Changed-Why:
Reopened on submitters request


From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: port-evbarm/49395 (cubietruck freezes under i/o load on netbsd-7 and HEAD)
Date: Sat, 6 Dec 2014 12:21:07 +0100

 On Thu, Dec 04, 2014 at 06:35:01PM +0000, Markus Kilbinger wrote:
 >  The 'freeze' is back for several days now, in netbsd-7 more than in HEAD!
 >  -> Can you re-open the PR, please!

 I am not sure this is the exact same thing, but I can reproduce it in -current
 even simpler and reliably:

 After building sets on my build machine, I scp them over to /var/tmp on the
 cubietruck, then move etc.tgz and xet.tgz to /tmp (for later postinstall
 use), and then do (in C shell syntax):

   cd /
   foreach tar (var/tmp/*.tgz)
      echo $tar; tar xpzf $tar
   end

 This reliably kills my machine. The issue may be KVA vs. WAPBL, haven't tried
 yet to do it w/o log mount. There is a minor variation that reliably works
 for me:

   cd /
   foreach tar (var/tmp/*.tgz)
      echo $tar; tar xpzf $tar; rm $tar; sync
   end


 Martin

From: Markus Kilbinger <mk@kilbi.de>
To: gnats-bugs@netbsd.org
Cc: port-evbarm-maintainer@netbsd.org, gnats-admin@netbsd.org, 
	netbsd-bugs@netbsd.org, Markus Kilbinger <mk@kilbi.de>
Subject: Re: port-evbarm/49395 (cubietruck freezes under i/o load on netbsd-7
 and HEAD)
Date: Sun, 7 Dec 2014 11:55:04 +0100

 2014-12-06 12:25 GMT+01:00 Martin Husemann <martin@duskware.de>:
 > The following reply was made to PR port-evbarm/49395; it has been noted by GNATS.
 >  I am not sure this is the exact same thing, but I can reproduce it in -current
 >  even simpler and reliably:
 >
 >  After building sets on my build machine, I scp them over to /var/tmp on the
 >  cubietruck, then move etc.tgz and xet.tgz to /tmp (for later postinstall
 >  use), and then do (in C shell syntax):
 >
 >    cd /
 >    foreach tar (var/tmp/*.tgz)
 >       echo $tar; tar xpzf $tar
 >    end
 >
 >  This reliably kills my machine. The issue may be KVA vs. WAPBL, haven't tried.

 WAPBL disabled (w/o 'log') didn't help during my tests.

 But: With that (KVA) hint I played with CUBIETRUCK's config and
 modifying it like (== preserve original std.awin settings and limit
 mem to 1 gb):

   #makeoptions   KERNEL_BASE_PHYS="0x80000000"
   #makeoptions   KERNEL_BASE_VIRT="0x80000000"
   options       MEMSIZE=1024

 produced a working netbsd-7 cubietruck kernel for me (with the side
 effect that 'bootargs' is no longer found / supported).

 Does this help to localize the problem?

 Markus.

From: matthew green <mrg@eterna.com.au>
To: mk@kilbi.de
Cc: port-evbarm-maintainer@netbsd.org, gnats-admin@netbsd.org,
    netbsd-bugs@netbsd.org, gnats-bugs@netbsd.org, para@netbsd.org
Subject: re: port-evbarm/49395 (cubietruck freezes under i/o load on netbsd-7 and HEAD)
Date: Mon, 08 Dec 2014 08:32:06 +1100

 i wonder if this value is insufficent for 2GB ram:

 include/arm32/param.h:#define NKMEMPAGES_MIN_DEFAULT  ((8 * 1024 * 1024) >> PAGE_SHIFT)

 could you try increasing it?  


 .mrg.

From: Markus Kilbinger <mk@kilbi.de>
To: matthew green <mrg@eterna.com.au>
Cc: Markus Kilbinger <mk@kilbi.de>, port-evbarm-maintainer@netbsd.org, gnats-admin@netbsd.org, 
	netbsd-bugs@netbsd.org, gnats-bugs@netbsd.org, para@netbsd.org
Subject: Re: port-evbarm/49395 (cubietruck freezes under i/o load on netbsd-7
 and HEAD)
Date: Mon, 8 Dec 2014 23:19:48 +0100

 2014-12-07 22:32 GMT+01:00 matthew green <mrg@eterna.com.au>:
 >
 > i wonder if this value is insufficent for 2GB ram:
 >
 > include/arm32/param.h:#define NKMEMPAGES_MIN_DEFAULT  ((8 * 1024 * 1024) >> PAGE_SHIFT)
 >
 > could you try increasing it?

 I increased it up to

   #define NKMEMPAGES_MIN_DEFAULT  ((48 * 1024 * 1024) >> PAGE_SHIFT)

 with no real change (maybe the freeze is a bit delayed, but still happens).

 Anything else I can try?

 Markus.

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: port-evbarm/49395 (cubietruck freezes under i/o load on netbsd-7 and HEAD)
Date: Fri, 6 Feb 2015 13:44:25 +0100

 On Sat, Dec 06, 2014 at 12:21:07PM +0100, Martin Husemann wrote:
 > After building sets on my build machine, I scp them over to /var/tmp on the
 > cubietruck, then move etc.tgz and xet.tgz to /tmp (for later postinstall
 > use), and then do (in C shell syntax):
 > 
 >   cd /
 >   foreach tar (var/tmp/*.tgz)
 >      echo $tar; tar xpzf $tar
 >   end
 > 
 > This reliably kills my machine.

 And I can't reproduce this hang any more with -current.

 Martin

From: mk@kilbi.de
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: port-evbarm/49395 (cubietruck freezes under i/o load on netbsd-7 and HEAD)
Date: Fri, 24 Apr 2015 22:18:44 +0200

 Hi!

 I can no longer reproduce this problem under netbsd-7 for several
 weeks now, so you can close this PR.

 Markus.

State-Changed-From-To: open->closed
State-Changed-By: msaitoh@NetBSD.org
State-Changed-When: Thu, 30 Apr 2015 03:53:45 +0000
State-Changed-Why:
Fixed. Thanks.


>Unformatted:

NetBSD Home
NetBSD PR Database Search

(Contact us) $NetBSD: query-full-pr,v 1.39 2013/11/01 18:47:49 spz Exp $
$NetBSD: gnats_config.sh,v 1.8 2006/05/07 09:23:38 tsutsui Exp $
Copyright © 1994-2014 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.