MuCPP Mutation Tool

MuCPP is a mutation testing tool for the C++ programming language. The tool includes class-based and traditional-based mutation operators which are applied using the Clang compiler. MuCPP is not subject to a framework testing so that you can use your own test suite.

Installation

MuCPP has been developed and tested using:
  • Ubuntu 14.04 / 64 bits.
  • LLVM/Clang compiler 3.6 for 64 bits.

Steps:

  1. Download LLVM/Clang libraries:
    sudo apt-get install clang-3.6 libclang-3.6-dev
    

    Since the name of the binary files installed will contain the version, it is recommended that you create a simbolic link to use the binaries without the version as follows:
    sudo ln -s /usr/bin/clang-3.6 /usr/bin/clang
    sudo ln -s /usr/bin/clang++-3.6 /usr/bin/clang++
    sudo ln -s /usr/bin/llvm-config-3.6 /usr/bin/llvm-config
    
  2. Install the version control system git:
    sudo apt-get install git
    

    Configure git setting your name and email:
    git config --global user.name "Your name" 
    git config --global user.email "Your email" 
    
  3. Download the file "mucpp.gz" (see section "Downloads" at the end of this page) and unzip the file:
    cd download_directory
    gzip -d mucpp.gz
    
  4. Check that the executable of MuCPP (mucpp) has execution permission:
    ls -l mucpp
    chmod +x mucpp
    
  5. Copy the executable of MuCPP to /usr/bin:
    sudo cp mucpp /usr/bin
    

NB: Please note that, if you do not follow these steps, you may find error messages when using the tool about missing headers files because Clang may not be able to locate the path to built in header files. In case you get this kind of errors, you should add the "Clang include directory" when using the tool through the -isystem option (see "Analyze" in Section "Requirements"). This directory usually is:

dirname $(which clang++)/../lib/clang/CLANG_VERSION/include    where CLANG_VERSION depends on the installed Clang version.

Please, go to the following page to obtain further information:
http://clang.llvm.org/docs/FAQ.html#i-get-errors-about-some-headers-being-missing-stddef-h-stdarg-h

After these steps, the example program “cars.tar.gz” can be downloaded and unzipped to check the correct installation of the mutation tool (see Section “Example program” to observe an example of the usage of the tool).

tar xzvf cars.tar.gz
cd cars

Once inside the unzipped directory, type:

mucpp count cars.cpp -- -std=c++11

The result should be:

Total mutants: x

where 'x' is the number of mutants that can be generated in the program, which depends on the list of mutation operators (see Section “Mutation operators”).

Along with MuCPP, a library for the execution of tests is provided. If you want to use it, you will need to download “tests_execution.tar.gz”, unzip it and add the files (“TestLib.a” and “test_functions.h”) to the directory of your program (see Section “Test suite execution with the provided library”). The example program “cars.tar.gz” uses this library to execute the tests.

Requirements

Previous to using MuCPP

To avoid unintentional changes, it is recommended to create a copy of the system under test (SUT) before testing it with MuCPP; files will be under the version control system git once the mutation tool is executed.

Before using MuCPP, you should check that:
  1. Every file pertaining to the SUT is placed in the root directory or in its subdirectories, deleting the files not needed as much as possible to create a clean copy of the program.
  2. The SUT is not under a version control system. Also check that there is no a “.gitignore” file, as it might impede the correct operation of MuCPP.
  3. The files to be mutated should have write permission.
  4. The root directory, from where MuCPP will be executed, should not contain:
    • A directory called “reports_analyze”.
    • A file called “tests_output.txt”, “times_output.txt” or “comparison_results.txt”.

Remove/rename these files in case they exist before using MuCPP as they will be created when running the tool.

In addition, MuCPP processes a file called “list_operators.txt” which is in the root directory of the SUT. The purpose of these files will be explained later on in this document.

Analyze

In case of using headers located in other directories (out of the root directory), those header files should be added as system headers and not as user headers. Thus, MuCPP will not consider these files for mutating them as they are not under the version control system git. To that end:
  • Use option –I for directories with user header files (they will be under git and can be mutated).
  • Use option –isystem for directories with system header files (they won't be under git and cannot be mutated).

MuCPP can be executed through two different methods:

  • Method 1: fixed compilation. Provide "--" at the end of the different commands (see Section "Options"). After "--", clang++ compiler options can be supplied to indicate how the files are compiled. Example: -I for a location of a header file (type clang++ --help to see the options).
  • Method 2: using a compilation database file created previously in the root directory (with the name “compile_commands.json”). In this way, indicating the command to compile the files is not needed.
    Visit http://clang.llvm.org/docs/JSONCompilationDatabase.html to know how to create the compilation database file.

For a correct execution of MuCPP, header files should not be included in the source code files using a path with "./" or "../", but using the same relative path to the root directory of the program any time the header file is included in another file. In case of using the method 2, every file analyzed in the same execution should have the same “directory”, which is the root directory of the SUT.

Testing

MuCPP is not subject to a testing framework. Nevertheless, to execute the test suite it is necessary that you have a “Makefile” which includes the following rules:

  • mucpp_clean: clean all files that should be regenerated to execute the test suite.
  • mucpp_compilation: compile the mutated program.
  • mucpp_test: compile the test suite, generating executable files.
  • mucpp_execute: execute the test suite.

The tester must ensure that a file "tests_output.tmp" with the results of the test suite execution is generated in the process. Each line of that file contains, in the order of the execution of the tests, the result of each test case:

  • “0” means that the program passes the test case.
  • “1” means that the program fails the test case.

Optionally, a file “times_output.tmp” can be also generated by the tester, where each line represents the time spent by each test case. The unit of measure depends on the mechanism for the measurement of the times used by the tester.

Options

Previous notes:

  • To obtain information about the available options, operator codes..., type:
    mucpp --help
    
  • To remove the mutants previously created in the SUT and start from a clean copy, use the option:
    mucpp clean
    
  • In the examples shown for each option, the existence of a compilation database is supposed. Otherwise, “--” should be included at the end of the commands (as explained in section “Requirements”).
  • Options count, analyze and applyall: if no source file is provided, the files in the root directory will be processed (in this case, place test source files in a test directory as a subdirectory within the root directory).
  • Only the files with extension “c”, “cc”, “cxx” or “cpp” are permitted. Header files will be mutated for being included in a source code file. If a same header file is present in more than one source file, the tool will avoid the creation of duplicated mutants. The source files provided are sorted through “std::sort” so that all the executions with the same files always report the same mutants.

Count

Calculation of the total of mutants in the files supplied.

mucpp count [file1.cpp file2.cpp...]

Analyze:

Report with the mutants for each operator.
mucpp analyze [file1.cpp file2.cpp ...]

The report is displayed on the screen, but it is also saved in the directory called “reports_analyze”. The generated reports are:
  • A report per source file analyzed, with the name “analyze_file.txt” (where 'file' is the name of the source code file without extension).
  • A global report which computes the mutants of all the source code files as a whole, with the name “analyze_mucpp_global.txt”.

The output is a list with the next information for every mutation operator:

“Operator locations attributes”

Where:
  • Operator → name of the mutation operator
  • Locations → locations in the code where is possible to introduce a mutation.
  • Attributes → different mutations to insert in each location. For instance, if “attributes” shows “2”, that means that two different mutants can be produced per mutation location.

There are mutation operators whose number of attributes is not fixed, but variable. In other words, the number of mutants to inject in a concrete location depends entirely on the location within the code. In this case, the attribute is marked as “1” and each produced mutant is added to the location counter.

Applyall

Generation of all the mutants in the code.

mucpp applyall [file1.cpp file2.cpp ...]

The mutants are created as branches in the version control system git. The branches contain an exact copy of the original program except from the file or files mutated.

Format for the name of mutants:

m + code of operator + _ + order of location + _ + attribute selected to mutate the location + _ + name of file;

Where:
  • Code → operator identifier (see the code of the operators with mucpp -- help)
  • Location → number of location in the code.
  • Attribute → selected variant to be inserted in a location.
  • Name of file → name of the source code file without extension.
Notes:
  • When applyall is supplied, the existing branches are deleted previously.
  • The mutants are generated as the locations to mutate are found in the traversal of the code

Apply

Generation of a single mutant identified by operator, location, attribute and file.

mucpp apply operator location attribute [file1.cpp file2.cpp ...]

Example: to create the mutant of the operator IOD corresponding to the second location and the first attribute:
mucpp apply IOD 2 1 file.cpp

Note: When apply is supplied, only the corresponding branch is deleted previously.

Run

Execution of the test suite in a program. "test_directory" specifies the path to execute the rules in the “Makefile” (see section “Requirements”). The original program is executed when any mutants are supplied. In case that some mutants are provided, the test suite will be executed against those mutants.

mucpp run test_directory [mutant1 mutant2 ...]

A file called “tests_output.txt” is generated, where each line is the result of a test case:
  • “0” for success
  • “1” for fail

If the mutant is invalid, only a value “2” will be shown.

If the times of the test suite execution have been measured, a file “times_output.txt” will be created, where each line represents the time taken by each test case.

Notes:
  • The resulting files (“tests_output.txt” and “times_output.txt”) will be created in the root directory and will be versioned under git. If a mutant was executed, the results will be in its corresponding branch.
  • Make sure that the original program passes all the test cases before executing the mutants (fix the program if at least a test case fails).

Compare

Comparison between the results of the test suite execution against the original program and the results for the mutants supplied. "test_directory" specifies the path to execute the rules in the “Makefile” (see section “Requirements”). If no mutants are supplied, all existing mutants will be executed.
mucpp compare test_directory [mutant1 mutant2 ...]

Example: to compare the first and second mutants generated by the operator with code “01” in a source code file named “file.cpp”:
mucpp compare test_directory m01_1_1_file m01_2_1_file

This command displays a line per mutant, where for each test case prints:
  • “0” for the same result
  • “1” for a different result

The value “2” will be shown for invalid mutants.

If the times of the test suite execution have been measured, they will be displayed next to the results (a “T” is used to separate results and times).

The results of the test suite execution for the selected mutants will be also collected in a file called “comparsion_results.txt” in the root directory when the execution ends.

Notes:
  • When a mutant fails a test case (value 1), if the remaining test cases are executed depend on the configuration set by the tester in the test suite (see Section " Test suite execution with the provided library).
  • While the file “tests_output.txt” and “times_output.txt” are versioned in the corresponding branches in the root directory, the file “comparison_results.txt” will not be versioned, so it will disappear when executing any other option in MuCPP.

Mutation Operators

List of mutation operators

This version of MuCPP includes the next mutation operators at the class-level:

  • Inheritance: IHD, IHI, ISD, ISI, IOD, IOP, IOR, IPC and IMR
  • Polymorphism and dynamic binding: PVI, PCD, PCI, PCC, PPD, PMD, and PNC
  • Method overloading: OMD, OMR, OAN and OAO
  • Member and object replacement: MCO and MCI
  • Exception handling: EHC and EHR
  • Miscellany: CTD, CTI, CID, CDC, CDD and CCA

Note: Further information about this set of mutation operators can be found in:

Likewise, the following traditional operators are also included: ARB, ARU, ARS, AIU, AIS, ADS, ROR, COR, COI, COD, LOR and ASR.

Select a subset of mutation operators

A subset of the mutation operators can be used following these steps:

  1. Copy the file “list_operators.txt” (see Section "Downloads") in the root directory of the SUT.
  2. Remove the operators which are not desired.
  3. Maintain one operator per line in this file for a correct execution.

Handling of mutants with Git

  • Obtain a clean copy of the program without being under git:
    mucpp clean 
    
  • List generated mutants:
    git branch
    
  • If any changes have been carried out in the current branch, clean it before accessing another branch:
    git reset
    git checkout .
    git clean -xdf
    
  • If you want to maintain a change of a particular file in the current branch:
    git add file
    git commit -m "Observation" 
    
  • Observe the status of the current branch:
    git status
    
  • Access a mutant:
    git checkout mutant
    
  • Return to the original program directory:
    git checkout master
    
  • Observe the change represented by a mutant:
    git log –p mutant
    
  • Delete a mutant:
    git branch –D mutant
    
  • Compare the mutants of an operator, for instance, those of the operator with code “02”.
    mucpp compare . $(git branch | grep -w "m02.*") 
    

Test suite execution with the provided library

First, go to section “Requirements” to prepare the SUT correctly for the execution of the test suite. If you prefer to use your own test suite library or other testing framework, you will need to create an additional script in order to report the results of the test suite execution to MuCPP (in order to create the file "tests_output.tmp" properly).

This library automatically generates the files ("tests_output.tmp" and times_output.tmp") that MuCPP needs. The file "tests_execution.tar.gz" includes:
  • The library “TestLib.a”.
  • The header file called “tests_functions.h”.

The library "TestLib.a" defines a class “TestFunctions”. An object of this class can be constructed passing three arguments:

  1. First argument: whether a time measurement of the test suite execution is undertaken (the results will be saved in “times_output.tmp”) or not. In this library, the time is measured in nanoseconds.
  2. Second argument: to set a timeout (in seconds) so that a test case ends after the specified time. If the timeout is correctly set, it will indicate that a test case is taking too much time to perform and, therefore, an anomalous behavior is taking place.
  3. Third argument: to indicate whether the test suite execution should continue or not once a test case fails.

Note: The mutants for which the command “mucpp_test” fails cannot be executed against the test cases. In case that the third argument is set to true, only a value 1 will be shown for those mutants.

The class “TestFunctions” contains the next four public methods:

  • test_case starts the execution of the test case received as argument.
  • has_failed indicates if the last test case executed has failed.
  • continue_execution returns the third argument provided when constructing the object “TestFunctions”, indicating if the tester wants the test suite execution to continue, even when the last test case has failed.
  • set_timeout sets a new timeout; it will be applied to the rest of test cases.
In addition, “tests_functions.h” contains the following function to use within the body of a test case:
  • check_test_case receives a value as argument. A value “0” means that the test case is not passed and its execution ends. Otherwise, the test case continues its execution.

Example program

The file "cars.tar.gz" contains:
  1. The files “cars.cpp” and “cars.h”, which constitutes the SUT, that is, these files will be mutated by MuCPP.
  2. A file called “tests.cpp” where the different test cases are implemented, using the methods in the library “TestLib.a” (the declaration of these methods can be found in “tests_functions.h”).
  3. A file “execution_tests.cpp” invoking each of the test cases defined in “tests.cpp”. The test cases will be executed one after the other using the method “test_case” and, at the end of the execution of a test case, the method “has_failed” is used to check if that test case has failed. If so, the method “continue_execution” is used to know whether the rest of the test cases should be executed or the execution should be stopped to continue with the next mutant.
  4. “Makefile” with the following rules:
    • mucpp_clean: clean previous files.
    • mucpp_compilation: create the file “cars.o”
    • mucpp_test: create the file “tests.o”, “execution_tests.o” and the executable “test”.
    • mucpp_execute: execute “test”, running the test cases defined in “tests.cpp” in the order specified in “execution_tests.cpp”.

First, we can analyze the source code regarding the operators that will create mutants in the source file:

> mucpp analyze cars.cpp -- -std=c++11

...
IHI 3 1
IOD 3 1
PVI 3 1
CDC 2 1
AIU 2 1
COI 2 1
...

Note: We inform Clang to consider the c++11 standard when analyzing the code.

Then, the mutants can be generated:

> mucpp applyall cars.cpp -- -std=c++11
The mutant 'm28_1_1_cars' has been created.
The mutant 'm10_1_1_cars' has been created.
The mutant 'm02_1_1_cars' has been created.
The mutant 'm05_1_1_cars' has been created.
The mutant 'm10_2_1_cars' has been created.
The mutant 'm10_3_1_cars' has been created.
The mutant 'm34_1_1_cars' has been created.
The mutant 'm39_1_1_cars' has been created.
The mutant 'm28_2_1_cars' has been created.
The mutant 'm02_2_1_cars' has been created.
The mutant 'm02_3_1_cars' has been created.
The mutant 'm05_2_1_cars' has been created.
The mutant 'm05_3_1_cars' has been created.
The mutant 'm39_2_1_cars' has been created.
The mutant 'm34_2_1_cars' has been created.

As can be seen, the mutants are not generated in order of operator, but as the mutations are found in the traversal of the code, as it was commented in the option "applyall" in Section "Options".

We could have placed the file "list_operators.txt" in the root directory with this content:

IOD
CDC

Please, note that if you include the file "list_operators.txt" at this point (once MuCPP was already executed in this directory), it is required that either you first execute "mucpp clean" before including the file in the directory or add this file (git add list_operators.txt) and commit the change (git commit -m "Including file list_operators.txt") after including the file.

Therefore, only the mutants generated by those two operators would have been generated:

> mucpp applyall cars.cpp -- -std=c++11
The mutant 'm28_1_1_cars' has been created.
The mutant 'm05_1_1_cars' has been created.
The mutant 'm28_2_1_cars' has been created.
The mutant 'm05_2_1_cars' has been created.
The mutant 'm05_3_1_cars' has been created.

We could observe the mutation injected into a mutant:

> git log -p m05_1_1_cars

-       string type(){ return "car"; }
+       /*IOD*/

The method "type" is replaced by a comment that reflects that the operator IOD has been applied.

The use of the class “TestFunctions” defined in “test_functions.h” for the test suite execution can be observed in “execution_tests.cpp”. When executing the option "run", a file “tests_output.txt” will be created with the result of the two test cases defined in “tests.cpp” (MuCPP will print ‘0’ as result because they are passed with success):

> mucpp run . --
A file 'tests_output.txt' has been created with the results of the test suite execution.
> cat tests_output.txt
0
0

If you want to measure the times, edit the file "execution_tests.cpp" and change the first argument to true in the initialization of the "TestFunctions" object:

TestFunctions tf(true, 8, false);

Note: Again, clean the directory before modifying the file or add and commit the change after modifying the file.

As the program is set to measure the times, a file “times_output.txt” will also be generated when executing again "mucpp run . --". Each row in this file represents the time spent in executing each of the test cases:

> cat times_output.txt
30255541
30288507

When carrying out a comparison with the option "compare", the first mutant and the third mutant of the operator “05 - IOD” will provide a different output for the first and second test case respectively (represented with the value ‘1’); then, we can say that those mutants have been killed. The mutants remaining alive will need new test cases to detect those mutations inserted in the program or will turn out to be equivalent.

> mucpp applyall cars.cpp -- -std=c++11
The mutant 'm28_1_1_cars' has been created.
The mutant 'm05_1_1_cars' has been created.
The mutant 'm28_2_1_cars' has been created.
The mutant 'm05_2_1_cars' has been created.
The mutant 'm05_3_1_cars' has been created.

> mucpp compare . --
m05_1_1_cars 1 T 150488652
m05_2_1_cars 0 0 T 30184288 30253605 
m05_3_1_cars 0 1 T 30197093 150531361 
m28_1_1_cars 0 0 T 30212265 30257976 
m28_2_1_cars 0 0 T 30283844 30282076 

Note: We need to regenerate the mutants ("applyall" option) so that the mutants also contain the file "execution_tests.cpp" updated and the times can be shown. The time of each test case appears after the “T” in each mutant in case you generated the mutants with the option to measure the times.

As it can be seen in the result for the mutant "m05_1_1_cars", the execution of this mutant stopped when a test case detected the mutation. In other words, the first test case killed the mutant and therefore the second test case was not executed. In case you want to continue the execution when a test case detects the mutation, change to true the third argument of the "TestFunctions" object too:

TestFunctions tf(true, 8, true);

After adding and commiting this change and executing the comparison again, we can regenerate this mutant in particular (with the option "apply") and execute the option "compare" for this mutant in order to observe that all the test cases are executed:

> mucpp apply IOD 1 1 cars.cpp -- -std=c++11
The mutant 'm05_1_1_cars' has been created.

> mucpp compare . m05_1_1_cars --
m05_1_1_cars 1 0 T 150488652 30158942 

Finally, recall that the second argument of the "TestFunctions" object represents the timeout. In this case, the timeout is 8 seconds.

If you want to forget about compilation details, you can add the following JSON file to the root directory with the name "compile_commands.json":

[
{
        "directory": "/home/your_directory/cars",
        "command": "clang++ --std=c++11 /home/your_directory/cars/cars.cpp",
        "file" : "/home/your_directory/cars/cars.cpp" 
}
]

Note: Please, change "your_directory" to set your own absolute path.

In this way, you can omit "-- -std=c++11" when executing MuCPP from now on. For instance, to generate all the mutants, you only have to type:

> mucpp applyall cars.cpp

Contact

Report errors or ask for further information to Pedro Delgado Pérez

Downloads

MuCPP has been legally registered in the "Registro Territorial de la Propiedad Intelectual de la Junta de Andalucía" with the code: CA-452-14

cars.tar.gz - Example program (7.54 KB) Pedro Delgado Pérez, 02/29/2016 01:11 PM

tests_execution.tar.gz - Library to execute a test suite and produce the output required by MuCPP (6.53 KB) Pedro Delgado Pérez, 02/29/2016 01:11 PM

list_operators.txt Magnifier - List of available mutation operators in MuCPP (168 Bytes) Pedro Delgado Pérez, 02/29/2016 01:11 PM

mucpp.gz - MuCPP binary (9.32 MB) Pedro Delgado Pérez, 06/10/2016 02:09 PM