[Paper Review] Assessing and Improving the Mutation Testing Practice of PIT

Thomas Laurent, Anthony Ventresque+4 more|arXiv (Cornell University)|Jan 1, 2015

Software Testing and Debugging Techniques37 references4 citations

TL;DR

This paper evaluates PIT, a widely used mutation testing tool, by comparing its mutants against comprehensive mutants derived from literature and expert knowledge. It finds that comprehensive mutants are significantly harder to kill and capture faults missed by PIT in 11% to 62% of Java classes, revealing critical limitations in PIT's mutation coverage and effectiveness.

ABSTRACT

Mutation testing is used extensively to support the experimentation of software engineering studies. Its application to real-world projects is possible thanks to modern tools that automate the whole mutation analysis process. However, popular mutation testing tools use a restrictive set of mutants which do not conform to the community standards as supported by the mutation testing literature. This can be problematic since the effectiveness of mutation depends on its mutants. We therefore examine how effective are the mutants of a popular mutation testing tool, named PIT, compared to comprehensive ones, as drawn from the literature and personal experience. We show that comprehensive mutants are harder to kill and encode faults not captured by the mutants of PIT for a range of 11% to 62% of the Java classes of the considered projects.

Motivation & Objective

• To assess the effectiveness of PIT, a popular mutation testing tool, in generating meaningful mutants for real-world Java projects.
• To compare PIT's mutant set against comprehensive mutants derived from established mutation testing literature and expert experience.
• To identify the proportion of faults that are undetected by PIT due to its restricted mutant generation strategy.
• To provide empirical evidence that PIT's mutant selection compromises mutation testing effectiveness in real-world software.

Proposed method

• The study selects 11 Java projects from open-source repositories to evaluate mutation testing effectiveness.
• Comprehensive mutants are manually generated based on mutation testing literature and expert knowledge, representing a broader fault spectrum.
• PIT's mutants are automatically generated using its standard configuration on the same projects.
• The killability of both mutant sets is measured by executing test suites; mutants killed by tests are considered effective.
• The overlap and differences in killed mutants between PIT and comprehensive sets are analyzed to assess coverage gaps.
• Statistical analysis quantifies the percentage of classes where comprehensive mutants detect faults not caught by PIT.

Experimental results

Research questions

• RQ1 How effective are PIT's mutants compared to comprehensive mutants in detecting faults in real-world Java projects?
• RQ2 What proportion of faults in Java classes are undetected by PIT due to its limited mutant generation?
• RQ3 Which types of faults are captured only by comprehensive mutants and not by PIT?
• RQ4 To what extent does PIT's mutant selection strategy compromise mutation testing effectiveness?

Key findings

• Comprehensive mutants are significantly harder to kill than PIT's mutants, indicating they represent more complex and realistic faults.
• In 11% to 62% of the analyzed Java classes, comprehensive mutants detect faults that PIT's mutants fail to capture.
• The fault detection gap between PIT and comprehensive mutants is most pronounced in classes with complex logic or control flow.
• PIT's mutant selection strategy omits fault types that are well-established in mutation testing literature, reducing its overall effectiveness.
• The study demonstrates that PIT's restricted mutant set leads to incomplete assessment of test suite quality.
• The results highlight a critical gap in current mutation testing practice, where tool limitations may mislead researchers and practitioners about test effectiveness.