NetBSD Problem Report #48624

From martin@duskware.de  Thu Feb 27 10:18:33 2014
Return-Path: <martin@duskware.de>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 822D1A583C
	for <gnats-bugs@gnats.NetBSD.org>; Thu, 27 Feb 2014 10:18:33 +0000 (UTC)
From: martin@NetBSD.org
Reply-To: martin@NetBSD.org
To: gnats-bugs@NetBSD.org
Subject: atf integration seriously broken
X-Send-Pr-Version: 3.95

>Number:         48624
>Category:       bin
>Synopsis:       atf integration seriously broken
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    jmmv
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Feb 27 10:20:00 +0000 2014
>Closed-Date:    Mon Mar 03 20:05:16 +0000 2014
>Last-Modified:  Mon Mar 03 20:20:01 +0000 2014
>Originator:     Martin Husemann
>Release:        NetBSD 6.99.32
>Organization:
The NetBSD Foundation, Inc.
>Environment:
System: NetBSD thirdstage.duskware.de 6.99.32 NetBSD 6.99.32 (MODULAR) #124: Wed Feb 26 22:03:53 CET 2014 martin@thirdstage.duskware.de:/usr/src/sys/arch/sparc64/compile/MODULAR sparc64
Architecture: sparc64
Machine: sparc64
>Description:

Since the last ATF import, many internal ATF tests fail:


Failed (bogus) test programs:
    atf-sh/misc_helpers, tools/zero_tcs_helper, tools/misc_helpers

Test cases for known bugs:
    test-programs/c_helpers:expect_death_and_exit: Exit case
    test-programs/c_helpers:expect_death_and_signal: Signal case
    test-programs/c_helpers:expect_exit_any_and_exit: Call will exit
    test-programs/c_helpers:expect_exit_code_and_exit: Call will exit
    test-programs/c_helpers:expect_fail_and_fail_check: And fail again: 2 checks failed as expected; see output for more details
    test-programs/c_helpers:expect_fail_and_fail_requirement: Fail reason: The failure
    test-programs/c_helpers:expect_signal_any_and_signal: Call will signal
    test-programs/c_helpers:expect_signal_no_and_signal: Call will signal
    test-programs/c_helpers:expect_timeout_and_hang: Will overrun
    test-programs/cpp_helpers:expect_death_and_exit: Exit case
    test-programs/cpp_helpers:expect_death_and_signal: Signal case
    test-programs/cpp_helpers:expect_exit_any_and_exit: Call will exit
    test-programs/cpp_helpers:expect_exit_code_and_exit: Call will exit
    test-programs/cpp_helpers:expect_fail_and_fail_check: And fail again: 2 checks failed as expected; see output for more details
    test-programs/cpp_helpers:expect_fail_and_fail_requirement: Fail reason: The failure
    test-programs/cpp_helpers:expect_signal_any_and_signal: Call will signal
    test-programs/cpp_helpers:expect_signal_no_and_signal: Call will signal
    test-programs/cpp_helpers:expect_timeout_and_hang: Will overrun
    test-programs/sh_helpers:expect_death_and_exit: Exit case
    test-programs/sh_helpers:expect_death_and_signal: Signal case
    test-programs/sh_helpers:expect_exit_any_and_exit: Call will exit
    test-programs/sh_helpers:expect_exit_code_and_exit: Call will exit
    test-programs/sh_helpers:expect_fail_and_fail_requirement: Fail reason: The failure
    test-programs/sh_helpers:expect_signal_any_and_signal: Call will signal
    test-programs/sh_helpers:expect_signal_no_and_signal: Call will signal
    test-programs/sh_helpers:expect_timeout_and_hang: Will overrun
    tools/expect_helpers:death_and_exit: Exit case
    tools/expect_helpers:death_and_signal: Signal case
    tools/expect_helpers:exit_any_and_exit: Call will exit
    tools/expect_helpers:exit_code_and_exit: Call will exit
    tools/expect_helpers:fail_and_fail_check: And fail again: 2 checks failed as expected; see output for more details
    tools/expect_helpers:fail_and_fail_requirement: Fail reason: The failure
    tools/expect_helpers:signal_any_and_signal: Call will signal
    tools/expect_helpers:signal_no_and_signal: Call will signal
    tools/expect_helpers:timeout_and_hang: Will overrun

Failed test cases:
    atf-c/macros_test:detect_unused_tests, atf-c/pkg_config_test:version, atf-c++/macros_test:detect_unused_tests, atf-c++/pkg_config_test:version, test-programs/c_helpers:cleanup_fail, test-programs/c_helpers:cleanup_pass, test-programs/c_helpers:cleanup_sigterm, test-programs/c_helpers:cleanup_skip, test-programs/c_helpers:config_empty, test-programs/c_helpers:config_multi_value, test-programs/c_helpers:config_value, test-programs/c_helpers:expect_death_but_pass, test-programs/c_helpers:expect_exit_but_pass, test-programs/c_helpers:expect_fail_but_pass, test-programs/c_helpers:expect_pass_but_fail_check, test-programs/c_helpers:expect_pass_but_fail_requirement, test-programs/c_helpers:expect_signal_but_pass, test-programs/c_helpers:expect_timeout_but_pass, test-programs/c_helpers:result_fail, test-programs/c_helpers:result_newlines_fail, test-programs/c_helpers:srcdir_exists, test-programs/cpp_helpers:config_empty, test-programs/cpp_helpers:config_multi_value, test-programs
 /cpp_hel
 pers:config_value, test-programs/cpp_helpers:expect_death_but_pass, test-programs/cpp_helpers:expect_exit_but_pass, test-programs/cpp_helpers:expect_fail_but_pass, test-programs/cpp_helpers:expect_pass_but_fail_check, test-programs/cpp_helpers:expect_pass_but_fail_requirement, test-programs/cpp_helpers:expect_signal_but_pass, test-programs/cpp_helpers:expect_timeout_but_pass, test-programs/cpp_helpers:result_exception, test-programs/cpp_helpers:result_fail, test-programs/cpp_helpers:result_newlines_fail, test-programs/cpp_helpers:srcdir_exists, test-programs/sh_helpers:cleanup_fail, test-programs/sh_helpers:cleanup_pass, test-programs/sh_helpers:cleanup_sigterm, test-programs/sh_helpers:config_multi_value, test-programs/sh_helpers:config_value, test-programs/sh_helpers:expect_death_but_pass, test-programs/sh_helpers:expect_exit_but_pass, test-programs/sh_helpers:expect_fail_and_fail_check, test-programs/sh_helpers:expect_fail_but_pass, test-programs/sh_helpers:expect_pass_bu
 t_fail_c
 heck, test-programs/sh_helpers:expect_pass_but_fail_requirement, test-programs/sh_helpers:expect_signal_but_pass, test-programs/sh_helpers:expect_timeout_but_pass, test-programs/sh_helpers:result_fail, test-programs/sh_helpers:srcdir_exists, tools/expect_helpers:death_but_pass, tools/expect_helpers:exit_but_pass, tools/expect_helpers:fail_but_pass, tools/expect_helpers:pass_but_fail_check, tools/expect_helpers:pass_but_fail_requirement, tools/expect_helpers:signal_but_pass, tools/expect_helpers:timeout_but_pass, tools/fail_helper:main

Summary for 78 test programs:
    700 passed test cases.
    58 failed test cases.
    35 expected failed test cases.
    6 skipped test cases.

Looking at the first of those, some blatant errors show up:

# pkg-config --modversion atf-c
$(cat atf-version.txt)
# atf-version
Automated Testing Framework  (atf-)
Copyright (c) 2007 The NetBSD Foundation, Inc.


Many of the other failures are due to things like (in C):

	atf_tc_get_config_var(tc, "tmpfile")

failing (tmpfile is apparently not defined).


>How-To-Repeat:
cd /usr/tests/atf
atf-run | atf-report

>Fix:
n/a - maybe we should go back to the last working state for now?
Alternatively we can skip the atf internal tests alltogether.

>Release-Note:

>Audit-Trail:

Responsible-Changed-From-To: bin-bug-people->jmmv
Responsible-Changed-By: martin@NetBSD.org
Responsible-Changed-When: Thu, 27 Feb 2014 10:29:26 +0000
Responsible-Changed-Why:
over to maintainer


State-Changed-From-To: open->analyzed
State-Changed-By: jmmv@NetBSD.org
State-Changed-When: Sun, 02 Mar 2014 20:35:02 +0000
State-Changed-Why:
This is with an unclean object tree, right?

After the upgrade, I changed the test-programs Makefile to auto-generate
an Atffile, which resulted in various helper programs being registered as
tests.  This is incorrect.

I later undid this change to rely on a manually-crafted Atffile.
Unfortunately, I suppose your object tree still has an Atffile and this is
being preferred over the one in the source tree.

Cleaning your object tree should fix this.

However, we should also fix bsd.test.mk to prevent this altogether.
The solution I applied to FreeBSD was to name the auto-generated files as
Atffile.auto or similar and only perform the rename during install.  This
way, the two can never be confused.  How does that sound?


From: Martin Husemann <martin@duskware.de>
To: gnats-bugs@NetBSD.org
Cc: jmmv@NetBSD.org, netbsd-bugs@netbsd.org, gnats-admin@netbsd.org,
	martin@NetBSD.org
Subject: Re: bin/48624 (atf integration seriously broken)
Date: Sun, 2 Mar 2014 21:39:41 +0100

 On Sun, Mar 02, 2014 at 08:35:02PM +0000, jmmv@NetBSD.org wrote:
 > Cleaning your object tree should fix this.

 No, this was not from an update build. Give me an example of wrong/right
 files and I can check for more details.

 However, this would not explain broken .pc files and atf-version displaying
 nonsense, or would it?

 Martin

From: "Julio Merino" <jmmv@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/48624 CVS commit: src/external/bsd/atf
Date: Sun, 2 Mar 2014 22:50:14 +0000

 Module Name:	src
 Committed By:	jmmv
 Date:		Sun Mar  2 22:50:13 UTC 2014

 Modified Files:
 	src/external/bsd/atf/dist/tools: atf-version.cpp
 	src/external/bsd/atf/lib/libatf-c: Makefile
 	src/external/bsd/atf/lib/libatf-c++: Makefile
 	src/external/bsd/atf/usr.bin/atf-sh: Makefile
 	src/external/bsd/atf/usr.bin/atf-version: Makefile

 Log Message:
 Fix bundling of the atf version into pkgconfig files and atf-version.

 Sigh; one more attempt.  This time I'm sure I've verified that the
 .pc files contain the right number and that atf-version also outputs
 the right stuff...  Both with a clean and non-clean obj directory.

 Should fix part of the problems reported in PR bin/48624.


 To generate a diff of this commit:
 cvs rdiff -u -r1.5 -r1.6 src/external/bsd/atf/dist/tools/atf-version.cpp
 cvs rdiff -u -r1.20 -r1.21 src/external/bsd/atf/lib/libatf-c/Makefile
 cvs rdiff -u -r1.23 -r1.24 src/external/bsd/atf/lib/libatf-c++/Makefile
 cvs rdiff -u -r1.11 -r1.12 src/external/bsd/atf/usr.bin/atf-sh/Makefile
 cvs rdiff -u -r1.12 -r1.13 src/external/bsd/atf/usr.bin/atf-version/Makefile

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: Julio Merino <jmmv@netbsd.org>
To: Martin Husemann <martin@duskware.de>
Cc: gnats-bugs@netbsd.org, netbsd-bugs@netbsd.org, gnats-admin@netbsd.org, 
	martin@netbsd.org
Subject: Re: bin/48624 (atf integration seriously broken)
Date: Sun, 2 Mar 2014 17:51:12 -0500

 On Sun, Mar 2, 2014 at 3:39 PM, Martin Husemann <martin@duskware.de> wrote:
 > On Sun, Mar 02, 2014 at 08:35:02PM +0000, jmmv@NetBSD.org wrote:
 >> Cleaning your object tree should fix this.
 >
 > No, this was not from an update build. Give me an example of wrong/right
 > files and I can check for more details.

 Check the Atffile from external/bsd/atf/tests/atf/test-programs/
 (copied to /usr/tests/atf/test-programs).  It should NOT list any of
 the *helper* binaries.  There should be an Atffile in the src
 directory with the right contents and no Atffile in the obj dir.

 Now that I think about it, a "make clean" would not delete the stray
 Atffiles because they are not referenced by the Makefiles any
 longer...

 > However, this would not explain broken .pc files and atf-version displaying
 > nonsense, or would it?

 Oh god, I didn't notice those in your report.  Bleh; should be fixed now.

From: Martin Husemann <martin@duskware.de>
To: Julio Merino <jmmv@netbsd.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: bin/48624 (atf integration seriously broken)
Date: Mon, 3 Mar 2014 12:05:54 +0100

 On Sun, Mar 02, 2014 at 05:51:12PM -0500, Julio Merino wrote:
 > Check the Atffile from external/bsd/atf/tests/atf/test-programs/
 > (copied to /usr/tests/atf/test-programs).  It should NOT list any of
 > the *helper* binaries.

 That one is fine, but /usr/tests/atf/atf-sh/Atffile is not. Same for atf-c++
 and atf-c. Haven't checked further...

 > Oh god, I didn't notice those in your report.  Bleh; should be fixed now.

 Yeah, thanks, that works now.

 Martin

From: "Julio Merino" <jmmv@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/48624 CVS commit: src/external/bsd/atf/tests/atf/atf-sh
Date: Mon, 3 Mar 2014 19:56:33 +0000

 Module Name:	src
 Committed By:	jmmv
 Date:		Mon Mar  3 19:56:32 UTC 2014

 Modified Files:
 	src/external/bsd/atf/tests/atf/atf-sh: Makefile
 Added Files:
 	src/external/bsd/atf/tests/atf/atf-sh: Atffile Kyuafile

 Log Message:
 Avoid running a helper program for atf-sh as a test.

 Should fix failures observed in PR bin/48624.


 To generate a diff of this commit:
 cvs rdiff -u -r0 -r1.1 src/external/bsd/atf/tests/atf/atf-sh/Atffile \
     src/external/bsd/atf/tests/atf/atf-sh/Kyuafile
 cvs rdiff -u -r1.7 -r1.8 src/external/bsd/atf/tests/atf/atf-sh/Makefile

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

State-Changed-From-To: analyzed->closed
State-Changed-By: jmmv@NetBSD.org
State-Changed-When: Mon, 03 Mar 2014 20:05:16 +0000
State-Changed-Why:
Per martin's response, we can consider this fixed.  Thanks for reporting!


From: Martin Husemann <martin@duskware.de>
To: Julio Merino <jmmv@netbsd.org>
Cc: gnats-bugs@netbsd.org
Subject: Re: bin/48624 (atf integration seriously broken)
Date: Mon, 3 Mar 2014 21:02:42 +0100

 On Mon, Mar 03, 2014 at 03:00:04PM -0500, Julio Merino wrote:
 > Hum... two of the failures here (pkg_config_test) should have been
 > fixed yesterday with the fixes to the version handling stuff. The
 > other two failures (detect_unused_tests) are possibly a result of your
 > gcc 4.8 work, but I cannot tell without further details.

 Yes, two of them are gcc 4.8 bugs, and it seems like you just fixed the
 last fallout, as a run from earlier today already looked pretty good:

 http://www.netbsd.org/~martin/sparc64-atf/285_atf.html

 Martin

From: Julio Merino <jmmv@netbsd.org>
To: Martin Husemann <martin@duskware.de>
Cc: gnats-bugs@netbsd.org
Subject: Re: bin/48624 (atf integration seriously broken)
Date: Mon, 3 Mar 2014 15:00:04 -0500

 On Mon, Mar 3, 2014 at 6:05 AM, Martin Husemann <martin@duskware.de> wrote:
 > On Sun, Mar 02, 2014 at 05:51:12PM -0500, Julio Merino wrote:
 >> Check the Atffile from external/bsd/atf/tests/atf/test-programs/
 >> (copied to /usr/tests/atf/test-programs).  It should NOT list any of
 >> the *helper* binaries.
 >
 > That one is fine, but /usr/tests/atf/atf-sh/Atffile is not.

 Should be fixed.

 >Same for atf-c++
 > and atf-c. Haven't checked further...

 Hum... two of the failures here (pkg_config_test) should have been
 fixed yesterday with the fixes to the version handling stuff. The
 other two failures (detect_unused_tests) are possibly a result of your
 gcc 4.8 work, but I cannot tell without further details.

>Unformatted:

Home	PR Database Search