On this website you will find information on how to reproduce the results from the ICTSS 2012 paper titled "A Technique for Agile and Automatic Interaction Testing for Product Lines" written by Martin Fagereng Johansen, Øystein Haugen, Franck Fleurey, Erik Carlson, Jan Endresen, and Tormod Wien.
Decide on a root directory in which all files related to the testing will be stored. In this directory, create these directories:
Download and extract the following free tools in the "tools" directory.
The technique requires three files to be built.
The following command will perform the technique described in the paper fully automatically. When the feature model, the features themselves or the test suites are changed, there is no additional required manual work to redo the testing other than executing this command. basedir is the root directory of the testing system, t is the strenght of the combinatorial interaction testing and timeout is the time in seconds after which a test suite execution will be aborted.
java no.sintef.ict.splcatool.lasplt.LTSPLT basedir t timeout
Output such as the following will be produced on the command line:
Converting Package Mapping File to CSV [done] Converting Feature-Test Mapping File to CSV [done] Loading Mapping Files [done] Generating 2-wise Covering Array [done] Loading Covering Array [done] Products: 14 Test Product 0: Installing base product: [done] Installing features: Installing feature EclipseIDE [nothing to install] Installing feature RCP_Platform [nothing to install] Installing tests: Installing tests for EclipseIDE Installing tests for RCP_Platform Installing test org.eclipse.test.feature.group [done] ... Installing test org.eclipse.search.tests [done] Running tests: Running tests for EclipseIDE Running tests for RCP_Platform Running test org.eclipse.ant.tests.core.AutomatedSuite [done] 100% (85/85) Running test org.eclipse.compare.tests.AllTests [done] 89% (101/113) Running test org.eclipse.core.internal.expressions.tests.AllTests [done] 100% (108/108) ... Running test org.eclipse.search.tests.AllSearchTests [already exists] 78% (29/37) Test Product 1: Installing base product: [already exists] Installing features: Installing feature WindowBuilder [done] Installing feature SVN15 [done] Installing feature EclipseIDE [nothing to install] ... Installing feature RCP_Platform [nothing to install] Installing feature Datatools [done] Installing feature SVN [nothing to install] ... Test Product 13: ...
The result produced by the experiment reported in the paper are available.
The command in the previous step will produce logs containing the contents of stdour and stderr during test execution called failure-<product nr>-<test class name>.log and test result details in files called product<product nr>-<test class name>-test-result.xml. The result produced by the experiment reported in the paper are available.
These results can be compiled into a report by executing the following command.java no.sintef.ict.splcatool.lasplt.GenerateWebView basedir
This will produce a HTML-document, such as the one for the Eclipse IDE experiment in the paper.
The set up required to run the CVL-based version of the technique is found in CVLLASPLTInput.java (Template found in CVL toolchain).
Run the testing technique fully automatically by calling: java -jar cvlsplt.jar. The CVL-tool chain is available. Contact us for the source code.
When run, it will produce output as follows:
Extracting Feature Model from CVL Model: Done Generating 2-wise Covering Array: Done Injecting Covering Array into CVL model: Done Executing CVL: Generated %basepth%/model/product0.uml Generated %basepth%/model/product1.uml Generated %basepth%/model/product2.uml ... Generated %basepth%/model/product10.uml Generated %basepth%/model/product11.uml %basepth%/model/product0.uml Moving to %basepth%/JFWS/Product0/product.uml Running JavaFrame Generator: Exit code: 0 Building workspace: Exit code: 0 Building workspace: Exit code: 0 Run Tests: SafetyDrive.GeneralStartUp: [success] SafetyDrive.Level3StartUpTest: [success] Done %basepth%/model/product1.uml ...