NetBSD Problem Report #55452

From jruohone@gmail.com  Fri Jul  3 15:32:37 2020
Return-Path: <jruohone@gmail.com>
Received: from mail.netbsd.org (mail.netbsd.org [199.233.217.200])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.NetBSD.org", Issuer "mail.NetBSD.org CA" (not verified))
	by mollari.NetBSD.org (Postfix) with ESMTPS id D8E7B1A9217
	for <gnats-bugs@gnats.NetBSD.org>; Fri,  3 Jul 2020 15:32:37 +0000 (UTC)
Message-Id: <20200703153223.DECF91AECBA@kafka.localdomain>
Date: Fri,  3 Jul 2020 18:32:23 +0300 (EEST)
From: jruohonen@iki.fi
Sender: j ruohonen <jruohone@gmail.com>
Reply-To: jruohonen@iki.fi
To: gnats-bugs@NetBSD.org
Subject: ATF should not call clean-up routines for skipped tests
X-Send-Pr-Version: 3.95

>Number:         55452
>Category:       lib
>Synopsis:       ATF should not call clean-up routines for skipped tests
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Jul 03 15:35:00 +0000 2020
>Last-Modified:  Fri Jul 03 15:55:01 +0000 2020
>Originator:     jruohonen@iki.fi
>Release:        NetBSD 9.99.68
>Organization:
>Environment:
System: NetBSD kafka 9.99.68 NetBSD 9.99.68 (CUSTOM) #0: Tue Jun 23 11:22:56 EEST 2020 jruoho@kafka:/tmp/obj/sys/arch/amd64/compile/CUSTOM amd64
Architecture: x86_64
Machine: amd64
>Description:
ATF calls the registered clean-up routines even when a test case has
explicitly been skipped in its body.  Although there may be some use cases
that warrant this behavior, practically all of the NetBSD test cases that
use atf_tc_skip() or atf_skip() do it as the first thing in the test cases'
bodies. By implication, and at least in some cases (e.g., a heavy clean-up
routine; see tests/sbin/ifconfig/t_random_garbage) the same
skip-declarations need to be duplicated to the clean-up routines.
>How-To-Repeat:
See, e.g., /tests/sbin/ifconfig/t_tap:
http://releng.netbsd.org/b5reports/i386/2020/2020.06.30.07.37.32/test.html#sbin_envstat_t_envstat_zerotemp
>Fix:
Do not call the clean-up routines in these special cases?

>Audit-Trail:
From: Paul Goyette <paul@whooppee.com>
To: gnats-bugs@netbsd.org
Cc: lib-bug-people@netbsd.org, gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: lib/55452: ATF should not call clean-up routines for skipped
 tests
Date: Fri, 3 Jul 2020 08:39:48 -0700 (PDT)

 >...  the same
 > skip-declarations need to be duplicated to the clean-up routines.

 You still need to clean-up anything that might have been done
 in the test-case's *_head() routine...


 +--------------------+--------------------------+-----------------------+
 | Paul Goyette       | PGP Key fingerprint:     | E-mail addresses:     |
 | (Retired)          | FA29 0E3B 35AF E8AE 6651 | paul@whooppee.com     |
 | Software Developer | 0786 F758 55DE 53BA 7731 | pgoyette@netbsd.org   |
 +--------------------+--------------------------+-----------------------+

From: Jukka Ruohonen <jruohonen@iki.fi>
To: Paul Goyette <paul@whooppee.com>
Cc: gnats-bugs@netbsd.org, lib-bug-people@netbsd.org,
	gnats-admin@netbsd.org, netbsd-bugs@netbsd.org
Subject: Re: lib/55452: ATF should not call clean-up routines for skipped
 tests
Date: Fri, 3 Jul 2020 18:53:31 +0300

 On Fri, Jul 03, 2020 at 08:39:48AM -0700, Paul Goyette wrote:
 > >...  the same
 > > skip-declarations need to be duplicated to the clean-up routines.
 > 
 > You still need to clean-up anything that might have been done
 > in the test-case's *_head() routine...

 True enough, and the noted sysctl/random_garbage is one such case. Though, 
 I wonder why the skip-declarations are not done in the head-routines? Well,
 the answer is simple at least for atf-sh-api(3):

 atf_test_case random_garbage cleanup
 random_garbage_head() {
         sysctl -a > $tmp
         atf_set "require.user" "root"
         atf_set "descr" "Test writing random garbage " \
                 "to sysctl nodes (PR kern/55451)"
 }

 So perhaps the correct approach would be to declare these in the
 init-routine; cf. the following or something alike:

 atf_init_test_cases() {
         atf_add_test_case random_garbage atf_skip "PR kern/55451"
 }

 But I guess it is easier to just duplicate the skip-declarations for the
 time being.

 - Jukka

 PS. It would be nice to also have a field in the API for PR-references.

Home	PR Database Search