NetBSD Problem Report #52007

From gson@gson.org  Sun Feb 26 11:36:18 2017
Return-Path: <gson@gson.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 7AC077A111
	for <gnats-bugs@gnats.NetBSD.org>; Sun, 26 Feb 2017 11:36:18 +0000 (UTC)
Message-Id: <20170226113612.2BA4C744F03@guava.gson.org>
Date: Sun, 26 Feb 2017 13:36:12 +0200 (EET)
From: gson@gson.org (Andreas Gustafsson)
Reply-To: gson@gson.org (Andreas Gustafsson)
To: gnats-bugs@NetBSD.org
Subject: New failures in lib/libevent/t_event test
X-Send-Pr-Version: 3.95

>Number:         52007
>Category:       lib
>Synopsis:       New failures in lib/libevent/t_event test
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    christos
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Feb 26 11:40:00 +0000 2017
>Last-Modified:  Thu Mar 01 19:05:00 +0000 2018
>Originator:     Andreas Gustafsson
>Release:        NetBSD-current, source date >= 2017.02.01.01.23.17
>Organization:

>Environment:
System: NetBSD
Architecture: x86_64
Machine: amd64
>Description:

Some test cases of the lib/libevent/t_event test have recently started
failing, even when run on real, Internet-connected hardware.  For
example, this shows a new failure of the "poll" test case on amd64:

  http://www.gson.org/netbsd/bugs/build/amd64-baremetal/commits-2017.02.html#2017.02.01.01.23.17

And this shows a new failure of both the "poll" and the "kqueue" test
cases on i386:

  http://www.gson.org/netbsd/bugs/build/i386-baremetal/commits-2017.02.html#2017.02.01.01.23.17

This started around the time of Christos' libevent commits of Jan 31 -
Feb 2.

Note that the logs from the TNF testbed are not helpful when it comes
to this particular test, because it has always been failing there, due
to qemu timing bugs and/or a lack of Internet connectivity.  Also, the
TNF amd64 testbed is currently not generating any test reports at all
due to PR 51934.

>How-To-Repeat:

Run the ATF tests on a physical machine connected to the Internet.

>Fix:

>Release-Note:

>Audit-Trail:

Responsible-Changed-From-To: lib-bug-people->christos
Responsible-Changed-By: gson@NetBSD.org
Responsible-Changed-When: Sun, 26 Feb 2017 11:41:55 +0000
Responsible-Changed-Why:
Problem appeared around the time of christos' libevent commits


From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: lib/52007: New failures in lib/libevent/t_event test
Date: Mon, 27 Feb 2017 11:55:29 +0700

     Date:        Sun, 26 Feb 2017 11:40:00 +0000 (UTC)
     From:        gson@gson.org (Andreas Gustafsson)
     Message-ID:  <20170226114000.71DE57A1F9@mollari.NetBSD.org>

   | >How-To-Repeat:
   | Run the ATF tests on a physical machine connected to the Internet.

 On amd64 (on a system with ipv6 connectivity, but with much ipv4 firewalled)
 what I see, during the poll test (as you said, kevent & select both work) is:

 tc-so:bufferevent/bufferevent_pair_release_lock: [forking] [warn] Trying to disa
 ble lock functions after they have been set up will probaby not work.
 tc-so:[warn] Trying to disable lock functions after they have been set up will p
 robaby not work.
 tc-so:
 tc-so:  FAIL /release/testing/src/external/bsd/libevent/dist/test/regress_buffer
 event.c:270: lock: lock error[err] /release/testing/src/external/bsd/libevent/di
 st/event.c:2557: Assertion evthread_is_debug_lock_held_((base)->th_base_lock) fa
 iled in event_add_nolock_
 tc-so:[Lost connection!]
 tc-so:  [bufferevent_pair_release_lock FAILED]

 and then the summary ...

 tc-so:1/308 TESTS FAILED. (21 skipped)

 Is that what you see too?


 kre

From: Andreas Gustafsson <gson@gson.org>
To: gnats-bugs@NetBSD.org
Cc: christos@NetBSD.org
Subject: Re: lib/52007: New failures in lib/libevent/t_event test
Date: Mon, 27 Feb 2017 19:11:15 +0200

 Robert Elz wrote:
 >  On amd64 (on a system with ipv6 connectivity, but with much ipv4 firewalled)
 >  what I see, during the poll test (as you said, kevent & select both work) is:
 >  
 >  tc-so:bufferevent/bufferevent_pair_release_lock: [forking] [warn] Trying to disa
 >  ble lock functions after they have been set up will probaby not work.
 [....]
 >  tc-so:1/308 TESTS FAILED. (21 skipped)
 >  
 >  Is that what you see too?

 Yes.  The full ATF output can be found at

   http://www.gson.org/netbsd/bugs/build/amd64-baremetal/2017/2017.02.01.01.23.17/test.tps

 -- 
 Andreas Gustafsson, gson@gson.org

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: christos@NetBSD.org
Subject: Re: lib/52007: New failures in lib/libevent/t_event test
Date: Wed, 22 Nov 2017 11:48:34 +0100

 I'd argue that the test progam should be removed (from the ATF suite)
 completely.

 The tests require:

  - pretty fast hardware
  - a working network connection
  - for best results a DNS on 127.0.0.1 (hard coded into the tests).

 The tests alway fail on all slower machines (that is in the order of
 less than 1.5 GHz cpu speed).

 They should be run after an import once, natively on a properly setup
 machine, but there is no point having them fail always and everywhere.

 Martin

From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: christos@NetBSD.org
Subject: Re: lib/52007: New failures in lib/libevent/t_event test
Date: Wed, 22 Nov 2017 11:57:52 +0100

 On Wed, Nov 22, 2017 at 11:48:34AM +0100, Martin Husemann wrote:
 > The tests alway fail on all slower machines (that is in the order of
 > less than 1.5 GHz cpu speed).

 To be more precise: the same binaries on the same arichtectures fail when
 run on a slower cpu.

 My sparc64 test bed machine has 2x 1002 MHz CPUs and consistently fails
 them (all three consistently time out at 300 seconds), my desktop
 sparc64 has 2x 1600 MHz CPUs and gets the same results as reported
 here: 1/308 TESTS FAILED. (21 skipped)

 If we would be talking about 500 MHz or something, I'd add a tc_skip for
 the affected machines. But all my ARM socs still faill them all always,
 so the requirements are a bit harsh, IMHO.

 Martin

From: christos@zoulas.com (Christos Zoulas)
To: Martin Husemann <martin@duskware.de>, gnats-bugs@NetBSD.org
Cc: 
Subject: Re: lib/52007: New failures in lib/libevent/t_event test
Date: Wed, 22 Nov 2017 09:14:02 -0500

 On Nov 22, 11:48am, martin@duskware.de (Martin Husemann) wrote:
 -- Subject: Re: lib/52007: New failures in lib/libevent/t_event test

 | I'd argue that the test progam should be removed (from the ATF suite)
 | completely.
 | 
 | The tests require:
 | 
 |  - pretty fast hardware
 |  - a working network connection
 |  - for best results a DNS on 127.0.0.1 (hard coded into the tests).
 | 
 | The tests alway fail on all slower machines (that is in the order of
 | less than 1.5 GHz cpu speed).
 | 
 | They should be run after an import once, natively on a properly setup
 | machine, but there is no point having them fail always and everywhere.

 Let's create a pre-test that tests if there is networking and DNS?

 christos

From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: lib/52007: New failures in lib/libevent/t_event test
Date: Thu, 23 Nov 2017 01:23:30 +0700

     Date:        Wed, 22 Nov 2017 14:15:00 +0000 (UTC)
     From:        christos@zoulas.com (Christos Zoulas)
     Message-ID:  <20171122141500.F24B87A1EE@mollari.NetBSD.org>

 (quoting Martin Husemann)
   |  | I'd argue that the test progam should be removed (from the ATF suite)
   |  | completely.

   |  Let's create a pre-test that tests if there is networking and DNS?

 I'd suggest just removing libevent from /usr/tests/lib/Atffile (add
 a little mechanism to bsd.test.mk to allow tests to be built, but
 excluded.)    Actually, is ".WAIT" already that mechanism?

 That is, leave the test there, and runnable manually as an individual
 test, but don't include it in the tests that ATF runs by default.

 kre

From: christos@zoulas.com (Christos Zoulas)
To: gnats-bugs@NetBSD.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org, 
	gson@gson.org (Andreas Gustafsson)
Cc: 
Subject: Re: lib/52007: New failures in lib/libevent/t_event test
Date: Wed, 22 Nov 2017 13:49:34 -0500

 On Nov 22,  6:25pm, kre@munnari.OZ.AU (Robert Elz) wrote:
 -- Subject: Re: lib/52007: New failures in lib/libevent/t_event test

 | The following reply was made to PR lib/52007; it has been noted by GNATS.
 | 
 | From: Robert Elz <kre@munnari.OZ.AU>
 | To: gnats-bugs@NetBSD.org
 | Cc: 
 | Subject: Re: lib/52007: New failures in lib/libevent/t_event test
 | Date: Thu, 23 Nov 2017 01:23:30 +0700
 | 
 |      Date:        Wed, 22 Nov 2017 14:15:00 +0000 (UTC)
 |      From:        christos@zoulas.com (Christos Zoulas)
 |      Message-ID:  <20171122141500.F24B87A1EE@mollari.NetBSD.org>
 |  
 |  (quoting Martin Husemann)
 |    |  | I'd argue that the test progam should be removed (from the ATF suite)
 |    |  | completely.
 |  
 |    |  Let's create a pre-test that tests if there is networking and DNS?
 |  
 |  I'd suggest just removing libevent from /usr/tests/lib/Atffile (add
 |  a little mechanism to bsd.test.mk to allow tests to be built, but
 |  excluded.)    Actually, is ".WAIT" already that mechanism?
 |  
 |  That is, leave the test there, and runnable manually as an individual
 |  test, but don't include it in the tests that ATF runs by default.

 Sounds good to me.

 christos

From: Robert Elz <kre@munnari.OZ.AU>
To: christos@zoulas.com (Christos Zoulas)
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@netbsd.org,
        gson@gson.org (Andreas Gustafsson)
Subject: Re: lib/52007: New failures in lib/libevent/t_event test
Date: Thu, 23 Nov 2017 09:28:08 +0700

     Date:        Wed, 22 Nov 2017 13:49:34 -0500
     From:        christos@zoulas.com (Christos Zoulas)
     Message-ID:  <20171122184934.E7CC217FDAB@rebar.astron.com>

 [Me...]
   | |  That is, leave the test there, and runnable manually as an individual
   | |  test, but don't include it in the tests that ATF runs by default.

 [Christos...]
   | Sounds good to me.

 There having been no objections, consider it done (in just a few minutes
 from now).

 kre

From: "Robert Elz" <kre@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/52007 CVS commit: src/share/mk
Date: Thu, 23 Nov 2017 02:39:28 +0000

 Module Name:	src
 Committed By:	kre
 Date:		Thu Nov 23 02:39:28 UTC 2017

 Modified Files:
 	src/share/mk: bsd.test.mk

 Log Message:
 PR lib/52007

 Provide a mechanism whereby a test sub-directory can be installed,
 without the test being scheduled to run by default (ie: keeping
 it out of the Atffile, and Kyuafile if Kyua is enabled.).

 The mechanism is perhaps a bit kludgey - anyone with a better idea
 how to make it happen, feel free to improve this (the one user as
 of about the time of this commit is (or will be) src/tests/lib/Makefile)


 To generate a diff of this commit:
 cvs rdiff -u -r1.24 -r1.25 src/share/mk/bsd.test.mk

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: "Robert Elz" <kre@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/52007 CVS commit: src/tests/lib
Date: Thu, 23 Nov 2017 02:40:01 +0000

 Module Name:	src
 Committed By:	kre
 Date:		Thu Nov 23 02:40:01 UTC 2017

 Modified Files:
 	src/tests/lib: Makefile

 Log Message:
 PR lib/52007

 Move libevent from being a test playing sub-directory, to a groupy,
 just hanging around, hoping someone will notice it, and throw it
 a bone...  (mixed metaphors?)


 To generate a diff of this commit:
 cvs rdiff -u -r1.29 -r1.30 src/tests/lib/Makefile

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: Robert Elz <kre@munnari.OZ.AU>
To: christos@zoulas.com (Christos Zoulas)
Cc: gnats-bugs@NetBSD.org, netbsd-bugs@netbsd.org,
        gson@gson.org (Andreas Gustafsson)
Subject: Re: lib/52007: New failures in lib/libevent/t_event test
Date: Thu, 23 Nov 2017 09:49:23 +0700

 OK, done.

 This is insufficient to close this PR, we still need to find out why
 the test fails in those circumstances where it should succeed.

 And it wouldn't hurt to do as Christos suggested, and make the test
 just skip if the environment is not suitable (if we can work out how
 to determine that properly, and accurately.)   Even with that, this
 test is so slow that it probably shouldn't be run by default (perhaps
 someday ATF/Kyua could be made able to have test modes of operation,
 and this one would be included in the "run everything" mode, but not
 the "run everytime" mode...)

 kre

From: "Martin Husemann" <martin@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/52007 CVS commit: [netbsd-8] src
Date: Thu, 1 Mar 2018 19:04:57 +0000

 Module Name:	src
 Committed By:	martin
 Date:		Thu Mar  1 19:04:57 UTC 2018

 Modified Files:
 	src/share/mk [netbsd-8]: bsd.test.mk
 	src/tests/lib [netbsd-8]: Makefile

 Log Message:
 Pull up following revision(s) (requested by kre in ticket #598):
 	tests/lib/Makefile: revision 1.30
 	share/mk/bsd.test.mk: revision 1.25
 PR lib/52007
 Provide a mechanism whereby a test sub-directory can be installed,
 without the test being scheduled to run by default (ie: keeping
 it out of the Atffile, and Kyuafile if Kyua is enabled.).
 The mechanism is perhaps a bit kludgey - anyone with a better idea
 how to make it happen, feel free to improve this (the one user as
 of about the time of this commit is (or will be) src/tests/lib/Makefile)
 PR lib/52007
 Move libevent from being a test playing sub-directory, to a groupy,
 just hanging around, hoping someone will notice it, and throw it
 a bone...  (mixed metaphors?)


 To generate a diff of this commit:
 cvs rdiff -u -r1.24 -r1.24.22.1 src/share/mk/bsd.test.mk
 cvs rdiff -u -r1.29 -r1.29.2.1 src/tests/lib/Makefile

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

>Unformatted:

Home	PR Database Search