NetBSD Problem Report #53161

From martin@duskware.de  Fri Apr  6 07:55:41 2018
Return-Path: <martin@duskware.de>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 852E77A1B3
	for <gnats-bugs@gnats.NetBSD.org>; Fri,  6 Apr 2018 07:55:41 +0000 (UTC)
From: martin@NetBSD.org
Reply-To: martin@NetBSD.org
To: gnats-bugs@NetBSD.org
Subject: ATF test runs leave a rump_server process around
X-Send-Pr-Version: 3.95

>Number:         53161
>Category:       kern
>Synopsis:       ATF test runs leave a rump_server process around
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    kern-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Apr 06 08:00:00 +0000 2018
>Closed-Date:    Wed Sep 04 09:08:32 +0000 2019
>Last-Modified:  Wed Sep 04 09:08:32 +0000 2019
>Originator:     Martin Husemann
>Release:        NetBSD 8.0_BETA
>Organization:
The NetBSD Foundation, Inc.
>Environment:
System: NetBSD unpluged.duskware.de 8.0_BETA NetBSD 8.0_BETA (UNPLUGED) #30: Thu Apr 5 17:47:24 CEST 2018 martin@seven-days-to-the-wolves.aprisoft.de:/work/src-8/sys/arch/evbarm/compile/UNPLUGED evbarm
Architecture: earm
Machine: evbarm
>Description:

Since one or two weeks, on evbarm, after a full ATF test run, a rump_server
process is left running, eating lots of CPU.

I have not identified the test triggering this yet. This does not happen in
-current.

[Switching to LWP 521]
0xbbce4e8c in ___lwp_park60 () from /usr/lib/libc.so.12
(gdb) info thread
  Id   Target Id         Frame 
  1    LWP 1             0xbbce46c8 in _ksem_wait () from /usr/lib/libc.so.12
  2    LWP 1             0xbbce46c8 in _ksem_wait () from /usr/lib/libc.so.12
  3    LWP 520           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  4    LWP 518           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  5    LWP 515           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  6    LWP 512           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  7    LWP 511           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  8    LWP 510           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  9    LWP 509           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  10   LWP 508           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  11   LWP 507           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  12   LWP 504           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  13   LWP 496           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  14   LWP 492           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  15   LWP 490           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  16   LWP 486           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  17   LWP 482           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  18   LWP 480           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  19   LWP 479           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  20   LWP 476           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  21   LWP 471           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  22   LWP 470           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  23   LWP 464           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  24   LWP 459           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  25   LWP 457           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  26   LWP 456           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  27   LWP 453           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  28   LWP 449           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  29   LWP 443           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  30   LWP 442           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  31   LWP 439           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  32   LWP 433           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  33   LWP 406           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  34   LWP 403           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  35   LWP 384           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  36   LWP 379           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  37   LWP 356           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  38   LWP 352           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  39   LWP 348           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  40   LWP 345           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  41   LWP 339           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  42   LWP 333           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  43   LWP 327           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  44   LWP 299           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  45   LWP 280           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  46   LWP 275           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  47   LWP 274           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  48   LWP 217           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  49   LWP 211           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  50   LWP 137           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  51   LWP 70            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  52   LWP 68            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  53   LWP 55            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  54   LWP 50            0xbbe4aa78 in rump_schedule_cpu_interlock (
    l=l@entry=0xa27ca380, interlock=interlock@entry=0x0)
    at /work/src-8/lib/librump/../../sys/rump/librump/rumpkern/scheduler.c:313
  55   LWP 35            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  56   LWP 34            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  57   LWP 33            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  58   LWP 30            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  59   LWP 29            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  60   LWP 28            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  61   LWP 27            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  62   LWP 26            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  63   LWP 25            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  64   LWP 24            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  65   LWP 23            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  66   LWP 22            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  67   LWP 21            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  68   LWP 20            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  69   LWP 19            0xbbce42d8 in _sys___nanosleep50 ()
   from /usr/lib/libc.so.12
  70   LWP 18            0xbbce42d8 in _sys___nanosleep50 ()
   from /usr/lib/libc.so.12
  71   LWP 17            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  72   LWP 16            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  73   LWP 15            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  74   LWP 14            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  75   LWP 13            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  76   LWP 12            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  77   LWP 11            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  78   LWP 10            0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  79   LWP 9             0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  80   LWP 8             0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  81   LWP 7             0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  82   LWP 6             0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  83   LWP 5             0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  84   LWP 4             0xbbce48d8 in clock_nanosleep ()
   from /usr/lib/libc.so.12
  85   LWP 3             0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
  86   LWP 2             0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12
* 87   LWP 521           0xbbce4e8c in ___lwp_park60 ()
   from /usr/lib/libc.so.12


>How-To-Repeat:
s/a

>Fix:
n/a

>Release-Note:

>Audit-Trail:
From: Ryota Ozaki <ozaki-r@netbsd.org>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/53161: ATF test runs leave a rump_server process around
Date: Fri, 6 Apr 2018 17:14:51 +0900

 On Fri, Apr 6, 2018 at 5:00 PM,  <martin@netbsd.org> wrote:
 >>Number:         53161
 >>Category:       kern
 >>Synopsis:       ATF test runs leave a rump_server process around
 >>Confidential:   no
 >>Severity:       critical
 >>Priority:       high
 >>Responsible:    kern-bug-people
 >>State:          open
 >>Class:          sw-bug
 >>Submitter-Id:   net
 >>Arrival-Date:   Fri Apr 06 08:00:00 +0000 2018
 >>Originator:     Martin Husemann
 >>Release:        NetBSD 8.0_BETA
 >>Organization:
 > The NetBSD Foundation, Inc.
 >>Environment:
 > System: NetBSD unpluged.duskware.de 8.0_BETA NetBSD 8.0_BETA (UNPLUGED) #30: Thu Apr 5 17:47:24 CEST 2018 martin@seven-days-to-the-wolves.aprisoft.de:/work/src-8/sys/arch/evbarm/compile/UNPLUGED evbarm
 > Architecture: earm
 > Machine: evbarm
 >>Description:
 >
 > Since one or two weeks, on evbarm, after a full ATF test run, a rump_server
 > process is left running, eating lots of CPU.
 >
 > I have not identified the test triggering this yet. This does not happen in
 > -current.

 The last argument of the rump_server process may tell us which test
 starts the rump_server. It's a unix domain socket for rump kernels
 and many tests embed its test name into the filename.

   ozaki-r

From: Ryota Ozaki <ozaki-r@netbsd.org>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc: kern-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: kern/53161: ATF test runs leave a rump_server process around
Date: Fri, 6 Apr 2018 17:21:33 +0900

 On Fri, Apr 6, 2018 at 5:14 PM, Ryota Ozaki <ozaki-r@netbsd.org> wrote:
 > On Fri, Apr 6, 2018 at 5:00 PM,  <martin@netbsd.org> wrote:
 >>>Number:         53161
 >>>Category:       kern
 >>>Synopsis:       ATF test runs leave a rump_server process around
 >>>Confidential:   no
 >>>Severity:       critical
 >>>Priority:       high
 >>>Responsible:    kern-bug-people
 >>>State:          open
 >>>Class:          sw-bug
 >>>Submitter-Id:   net
 >>>Arrival-Date:   Fri Apr 06 08:00:00 +0000 2018
 >>>Originator:     Martin Husemann
 >>>Release:        NetBSD 8.0_BETA
 >>>Organization:
 >> The NetBSD Foundation, Inc.
 >>>Environment:
 >> System: NetBSD unpluged.duskware.de 8.0_BETA NetBSD 8.0_BETA (UNPLUGED) #30: Thu Apr 5 17:47:24 CEST 2018 martin@seven-days-to-the-wolves.aprisoft.de:/work/src-8/sys/arch/evbarm/compile/UNPLUGED evbarm
 >> Architecture: earm
 >> Machine: evbarm
 >>>Description:
 >>
 >> Since one or two weeks, on evbarm, after a full ATF test run, a rump_server
 >> process is left running, eating lots of CPU.
 >>
 >> I have not identified the test triggering this yet. This does not happen in
 >> -current.
 >
 > The last argument of the rump_server process may tell us which test
 > starts the rump_server. It's a unix domain socket for rump kernels
 > and many tests embed its test name into the filename.

 Oops. This is true only for tests/net. If the rump_server is started from
 tests of other categories, I have no idea to identify.

   ozaki-r

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: kern/53161: ATF test runs leave a rump_server process around
Date: Thu, 10 May 2018 08:33:54 +0200

 It happened again with a latest netbsd-8 test run on evbarm.
 The rump_server process has this command line:

 rump_server -lrumpvfs -lrumpnet -lrumpnet_net -lrumpnet_netinet -lrumpdev unix://commsock

 and I see two left over atf temp directories, one with empty stdout/stderr,
 one with this in stderr:

 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on lo0, link-type NULL (BSD loopback), capture size 262144 bytes


 Martin

State-Changed-From-To: open->closed
State-Changed-By: martin@NetBSD.org
State-Changed-When: Wed, 04 Sep 2019 09:08:32 +0000
State-Changed-Why:
Haven't seen this on ARM machines in a long time


>Unformatted:

Home	PR Database Search