Resource page for "A Technique for Agile and Automatic Interaction Testing for Product Lines"

On this website you will find information on how to reproduce the results from the ICTSS 2012 paper titled "A Technique for Agile and Automatic Interaction Testing for Product Lines" written by Martin Fagereng Johansen, Øystein Haugen, Franck Fleurey, Erik Carlson, Jan Endresen, and Tormod Wien.

Applying the Technique for Testing of Eclipse-based Product Lines

1. Setting up the Directory Structure

Decide on a root directory in which all files related to the testing will be stored. In this directory, create these directories:

2. Acquiring the Required Tools

Download and extract the following free tools in the "tools" directory.

  1. 7-Zip Command Line Version (9.20)
  2. Eclipse Platform (3.7.0)
  3. UnxUtils (of 2007-03-01), Port of the most important GNU utilities to Windows
  4. An implementation of Algorithm 3 from the paper (The source code is also available for both Eclipse and CVL versions of Algorithm 3.)

3. Building the Package Repository

  1. Download the mirroring-script and place it in the scripts directory. Download and edit the environment set up script in which the path to the java.exe, cmd.exe and the root directory must be specified. Executing the mirroring script will construct mirrors for the Eclipse Indigo repositories in "%basedir%/packages/repos/". This will take a while, and might require rerunning the script several times to ensure that all packages were downloaded correctly. The size of these repositories during the experiments presented in the paper was 3.6 GiB.
  2. Place a copy of the package containing the Eclipse Platform in "packages".
  3. Download the Eclipse Automated Tests for Eclipse 3.7.0. Inside it there is a zip-file called "eclipse-testing/eclipse-junit-tests-I20110613-1736.zip". Extract it in the "packages" directory.

4. Setting up the Required Models

The technique requires three files to be built.

  1. The feature model, for example such as the one for the Eclipse IDEs.
  2. The mapping between features are code artifacts, as for example the one used in the experiment in the paper.
  3. The mapping between features and tests, as for example the one used in the experiment in the paper.

5. Executing the Technique Described in the Paper

The following command will perform the technique described in the paper fully automatically. When the feature model, the features themselves or the test suites are changed, there is no additional required manual work to redo the testing other than executing this command. basedir is the root directory of the testing system, t is the strenght of the combinatorial interaction testing and timeout is the time in seconds after which a test suite execution will be aborted.

java no.sintef.ict.splcatool.lasplt.LTSPLT basedir t timeout

Output such as the following will be produced on the command line: 

Converting Package Mapping File to CSV [done]
Converting Feature-Test Mapping File to CSV [done]
Loading Mapping Files [done]
Generating 2-wise Covering Array [done]
Loading Covering Array [done] Products: 14
Test Product 0:
	Installing base product: [done]
	Installing features:
		Installing feature EclipseIDE [nothing to install]
		Installing feature RCP_Platform [nothing to install]
	Installing tests:
		Installing tests for EclipseIDE
		Installing tests for RCP_Platform
			Installing test org.eclipse.test.feature.group [done]
			...
			Installing test org.eclipse.search.tests [done]
	Running tests:
		Running tests for EclipseIDE
		Running tests for RCP_Platform
			Running test org.eclipse.ant.tests.core.AutomatedSuite [done] 100% (85/85)
			Running test org.eclipse.compare.tests.AllTests [done] 89% (101/113)
			Running test org.eclipse.core.internal.expressions.tests.AllTests [done] 100% (108/108)
			...
			Running test org.eclipse.search.tests.AllSearchTests [already exists] 78% (29/37)
Test Product 1:
	Installing base product: [already exists]
	Installing features:
		Installing feature WindowBuilder [done]
		Installing feature SVN15 [done]
		Installing feature EclipseIDE [nothing to install]
		...
		Installing feature RCP_Platform [nothing to install]
		Installing feature Datatools [done]
		Installing feature SVN [nothing to install]
...
Test Product 13:
...

The result produced by the experiment reported in the paper are available.

6. Generate the web-view for the results

The command in the previous step will produce logs containing the contents of stdour and stderr during test execution called failure-<product nr>-<test class name>.log and test result details in files called product<product nr>-<test class name>-test-result.xml. The result produced by the experiment reported in the paper are available.

These results can be compiled into a report by executing the following command.

java no.sintef.ict.splcatool.lasplt.GenerateWebView basedir

This will produce a HTML-document, such as the one for the Eclipse IDE experiment in the paper.

Applying the Technique for Testing of CVL-based Product Lines

The set up required to run the CVL-based version of the technique is found in CVLLASPLTInput.java (Template found in CVL toolchain).

It also requires the following tools:

Run the testing technique fully automatically by calling: java -jar cvlsplt.jar. The CVL-tool chain is available. Contact us for the source code.

When run, it will produce output as follows: 

Extracting Feature Model from CVL Model: Done
Generating 2-wise Covering Array: Done
Injecting Covering Array into CVL model: Done
Executing CVL:
	Generated %basepth%/model/product0.uml
	Generated %basepth%/model/product1.uml
	Generated %basepth%/model/product2.uml
	...
	Generated %basepth%/model/product10.uml
	Generated %basepth%/model/product11.uml
%basepth%/model/product0.uml
	Moving to %basepth%/JFWS/Product0/product.uml
	Running JavaFrame Generator: Exit code: 0
	Building workspace: Exit code: 0
	Building workspace: Exit code: 0
	Run Tests:
		SafetyDrive.GeneralStartUp: [success]
		SafetyDrive.Level3StartUpTest: [success]
Done
%basepth%/model/product1.uml
...