NetBSD Problem Report #43394

From www@NetBSD.org  Mon May 31 13:36:27 2010
Return-Path: <www@NetBSD.org>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
	by www.NetBSD.org (Postfix) with ESMTP id E477F63B8E3
	for <gnats-bugs@gnats.NetBSD.org>; Mon, 31 May 2010 13:36:26 +0000 (UTC)
Message-Id: <20100531133626.A600963B8CF@www.NetBSD.org>
Date: Mon, 31 May 2010 13:36:26 +0000 (UTC)
From: pooka@iki.fi
Reply-To: pooka@iki.fi
To: gnats-bugs@NetBSD.org
Subject: adding regression tests is too complicated
X-Send-Pr-Version: www-1.0

>Number:         43394
>Category:       bin
>Synopsis:       adding regression tests is too complicated
>Confidential:   no
>Severity:       non-critical
>Priority:       high
>Responsible:    bin-bug-people
>State:          open
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Mon May 31 13:40:01 +0000 2010
>Last-Modified:  Tue Jul 13 21:15:02 +0000 2010
>Originator:     Antti Kantee
>Release:        
>Organization:
Test'ng
>Environment:
>Description:
Adding a new atf tests requires (off the top of my head):

1) copypasting the test skeleton
2) writing the actual test
3) creating/editing a Makefile
4) creating/editing an Atffile
5) adding the test to several places in set lists

Additionally, if the test creates a new subdirectory in the
test hierarchy:

6) editing another Makefile
7) editing another Atffile
8) editing the mtree base
9) adding more lines to the set lists

Tests should be easy and simple to add.  Almost 50% of the time
in adding a test is spent appeasing various gods of build
mechanics.
>How-To-Repeat:
Attempt to make regression testing easy for yourself and others.
>Fix:
Most likely the easiest solution is to provide a script which
takes care of all the mechanical tasks.  It can take care of
everything apart from step 2.

>Audit-Trail:
From: Julio Merino <jmmv@netbsd.org>
To: gnats-bugs@gnats.NetBSD.org
Cc: 
Subject: PR/43394 CVS commit: src/share/mk
Date: Tue, 13 Jul 2010 21:11:42 +0000

 Module Name:	src
 Committed By:	jmmv
 Date:		Tue Jul 13 21:11:41 UTC 2010

 Modified Files:
 	src/share/mk: bsd.test.mk

 Log Message:
 Implement Atffile automatic generation.

 This change adds the ATFFILE variable, which can take the following values:
 - yes: The source directory contains a static Atffile to be installed.
 - auto: The Atffile is automatically generated.
 - no: Do not install any Atffile (replaces the previous NOATFFILE variable).

 The ugly bit of this is that I have to add a TESTS_SUBDIRS variable that
 lists the subdirectories that contain test cases (so that they can be listed
 in the Atffiles, without including auxiliary directories without tests).
 This is a subset of SUBDIR.

 Addresses a minor part of PR bin/43394.


 To generate a diff of this commit:
 cvs rdiff -u -r1.10 -r1.11 src/share/mk/bsd.test.mk

 Please note that diffs are not public domain; they are subject to the
 copyright notices on the relevant files.

Home	PR Database Search