Coverage-Based Quality Metric

The Coverage-based quality metric addresses a special case: when there is a lack of coverage information for some of the mutants, we say that these mutants are difficult to reach. As we do not have enough coverage information about these mutants, it is an open question whether these mutants are easy or difficult to kill. We seek to categorise these mutants and favour them with the new quality metric because they might provide with necessary test cases.

Definition of difficult to reach mutants

Let MCOV be a constant representing the minimum number of coverage data required so that the coverage is regarded as significant. 
A dead mutant m is difficult to reach when the sum of the test cases reaching m and the non-equivalent mutants reached by those 
test cases is less or equal than MCOV.

As it can be seen from this definition, we need to set the value for MCOV. To this end, we carried out a sensitivity analysis for our six C++ programs under study. We calculated the coverage-based quality metric of each class mutation operator applicable to each program using different thresholds for MCOV (from 2 to 6). The results for each case study are presented below:

We can observe that, when using the values 2 and 3, several of the operators change their coverage-based quality metric. However, when using the values 5 and 6:
  • There are scattered changes.
  • These changes are often meaningful with respect to previous values for MCOV.

In the light of these results, we conclude that the value MCOV = 4 is an appropriate threshold to avoid overfitting the results to our programs under study.