NetBSD Problem Report #51667

From www@NetBSD.org  Mon Nov 28 07:15:00 2016
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id 772CE7A2E0
	for <gnats-bugs@gnats.NetBSD.org>; Mon, 28 Nov 2016 07:15:00 +0000 (UTC)
Message-Id: <20161128071459.169F17A322@mollari.NetBSD.org>
Date: Mon, 28 Nov 2016 07:14:59 +0000 (UTC)
From: ozaki-r@netbsd.org
Reply-To: ozaki-r@netbsd.org
To: gnats-bugs@NetBSD.org
Subject: atf-run wrongly passes even if atf_check fails in some cases
X-Send-Pr-Version: www-1.0

>Number:         51667
>Category:       bin
>Synopsis:       atf-run wrongly passes even if atf_check fails in some cases
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    bin-bug-people
>State:          closed
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Nov 28 07:20:00 +0000 2016
>Closed-Date:    Tue Apr 10 08:47:18 +0000 2018
>Last-Modified:  Tue Apr 10 08:47:18 +0000 2018
>Originator:     Ryota Ozaki
>Release:        -current
>Organization:
>Environment:
NetBSD rangeley 7.99.42 NetBSD 7.99.42 (RANGELEY) #34: Tue Nov 22 17:25:06 JST 2016  ozaki-r@rangeley:(hidden) amd64
>Description:
atf-run wrongly passes even if an atf_check fails in a case like the below sample test.

Normally we don't put atf_check at the end of a pipe, but we may want to write
a script like this:
  cat somefile | while read a b c; do
    atf_check -s exit:0 ...
  done

It doesn't work as well.
>How-To-Repeat:
atf_test_case should_fail
should_fail_body()
{

        atf_expect_fail "It should fail"
        echo zzz | atf_check -s exit:0 ls /tmp/non_exisiting_file
}

atf_init_test_cases()
{
        atf_add_test_case should_fail
}

# The test fails with "should_fail: [0.035389s] Failed: Test case was expecting a failure but none were raised"
>Fix:
n/a

>Release-Note:

>Audit-Trail:
From: "Ryota Ozaki" <ozaki-r@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/51667 CVS commit: src/tests/net
Date: Mon, 28 Nov 2016 07:29:56 +0000

 Module Name:	src
 Committed By:	ozaki-r
 Date:		Mon Nov 28 07:29:56 UTC 2016

 Modified Files:
 	src/tests/net: net_common.sh

 Log Message:
 Use redirection instead of pipeline

 This is a workaround for PR bin/51667.


 To generate a diff of this commit:
 cvs rdiff -u -r1.8 -r1.9 src/tests/net/net_common.sh

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: bin/51667
Date: Sun, 08 Apr 2018 03:31:33 +0700

 This is almost certainly because atf_check is just a shell function that
 calls atf_fail (if the checks fail), and atf_fail is just another shell function
 which (more or less) just does exit 1.

 ie: if the test is to fail, the shell running it exits with a "failed" status.

 Unfortunately, that is the shell running atf_fail (or atf_check) - some
 child sub-shell exiting with a failure status will only cause the correct
 behaviour for the test if the actual shell running the test also exits.

 In a pipeline, with /bin/sh (as allowed by posix, as it was the way the
 original Bourne sh did things) all the processes are run in a sub-shell
 (this is the same reason that

 	some_command | while whatever
 		do
 			var=foo
 		done

 neve results in var in the parent shell being set - the assignment happens
 in the sub-shell.

 That is, the sub-shell exits 1, causing a 1 exit status from the pipeline
 to be 1, but beyond that, the function just keeps on running.

 The same effect would be observed if one wanted to try

 	(atf-check -s exit:0 false)

 false exits with status 1, the "exit:0" test fails, so the test containg this
 check should fail, but it will not, because only the sub-shell exited, not
 the one running the test.

 I don't think there is anything that can be done about this, aside from
 just documenting it perhaps.   The alternative would be a major
 re-write of ATF.

 I would suggest closing this PR as "sh*t happens, can't be avoided"

 kre

From: Ryota Ozaki <ozaki-r@netbsd.org>
To: "gnats-bugs@NetBSD.org" <gnats-bugs@netbsd.org>
Cc: gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: bin/51667
Date: Tue, 10 Apr 2018 08:43:05 +0000

 On Sun, Apr 8, 2018 at 5:35 AM Robert Elz <kre@munnari.oz.au> wrote:

 > The following reply was made to PR bin/51667; it has been noted by GNATS.

 > From: Robert Elz <kre@munnari.OZ.AU>
 > To: gnats-bugs@netbsd.org
 > Cc:
 > Subject: Re: bin/51667
 > Date: Sun, 08 Apr 2018 03:31:33 +0700

 >   This is almost certainly because atf_check is just a shell function that
 >   calls atf_fail (if the checks fail), and atf_fail is just another shell
 function
 >   which (more or less) just does exit 1.

 >   ie: if the test is to fail, the shell running it exits with a "failed"
 status.

 >   Unfortunately, that is the shell running atf_fail (or atf_check) - some
 >   child sub-shell exiting with a failure status will only cause the correct
 >   behaviour for the test if the actual shell running the test also exits.

 >   In a pipeline, with /bin/sh (as allowed by posix, as it was the way the
 >   original Bourne sh did things) all the processes are run in a sub-shell
 >   (this is the same reason that

 >          some_command | while whatever
 >                  do
 >                          var=foo
 >                  done

 >   neve results in var in the parent shell being set - the assignment
 happens
 >   in the sub-shell.

 >   That is, the sub-shell exits 1, causing a 1 exit status from the pipeline
 >   to be 1, but beyond that, the function just keeps on running.

 >   The same effect would be observed if one wanted to try

 >          (atf-check -s exit:0 false)

 >   false exits with status 1, the "exit:0" test fails, so the test containg
 this
 >   check should fail, but it will not, because only the sub-shell exited,
 not
 >   the one running the test.

 >   I don't think there is anything that can be done about this, aside from
 >   just documenting it perhaps.   The alternative would be a major
 >   re-write of ATF.

 >   I would suggest closing this PR as "sh*t happens, can't be avoided"

 Thank you for the explanation. I got it.

 I'm ok to close this PR with "won't fix" because there is a workaround.

    ozaki-r

State-Changed-From-To: open->closed
State-Changed-By: ozaki-r@NetBSD.org
State-Changed-When: Tue, 10 Apr 2018 08:47:18 +0000
State-Changed-Why:
won't fix


>Unformatted:

Home	PR Database Search