[Paper Review] Causal testing: understanding defects' root causes

Brittany Johnson, Yuriy Brun+1 more|arXiv (Cornell University)|Jun 27, 2020

Software Engineering Research74 references17 citations

TL;DR

Causal Testing is a novel root-cause analysis method that uses counterfactual causality to identify executions revealing the true cause of software defects. Evaluated on Defects4J, it enabled developers to identify root causes in 86% of cases—up from 80% with standard tools—demonstrating significant improvement in debugging efficiency.

ABSTRACT

Understanding the root cause of a defect is critical to isolating and repairing buggy behavior. We present Causal Testing, a new method of root-cause analysis that relies on the theory of counterfactual causality to identify a set of executions that likely hold key causal information necessary to understand and repair buggy behavior. Using the Defects4J benchmark, we find that Causal Testing could be applied to 71% of real-world defects, and for 77% of those, it can help developers identify the root cause of the defect. A controlled experiment with 37 developers shows that Causal Testing improves participants' ability to identify the cause of the defect from 80% of the time with standard testing tools to 86% of the time with Causal Testing. The participants report that Causal Testing provides useful information they cannot get using tools such as JUnit. Holmes, our prototype, open-source Eclipse plugin implementation of Causal Testing, is available at http://holmes.cs.umass.edu/.

Motivation & Objective

• To address the challenge of identifying root causes of software defects, which is critical for effective debugging.
• To reduce the time and effort developers spend diagnosing bugs by providing targeted causal execution traces.
• To develop a method that reveals causal insights not available through standard testing tools like JUnit.
• To evaluate the effectiveness of causal analysis in real-world software defects using empirical benchmarks.

Proposed method

• Applies counterfactual causality theory to analyze program executions and identify causal factors behind defects.
• Generates a set of executions that differ only in the presence or absence of the defect to isolate causal variables.
• Uses a prototype Eclipse plugin, Holmes, to integrate causal analysis into the development workflow.
• Analyzes executions to identify minimal sets of conditions that trigger or prevent the defect.
• Prioritizes executions that reveal the most relevant causal information for debugging.
• Employs a trace analysis technique to compare executions and pinpoint causal discrepancies.

Experimental results

Research questions

• RQ1 Can causal testing identify root causes of real-world software defects more effectively than standard testing tools?
• RQ2 How does causal testing compare to traditional debugging methods in terms of developer accuracy?
• RQ3 What kind of causal information does causal testing provide that standard tools like JUnit cannot?

Key findings

• Causal Testing could be applied to 71% of real-world defects in the Defects4J benchmark.
• For 77% of the defects to which it was applicable, Causal Testing helped developers identify the root cause.
• In a controlled experiment, developers using Causal Testing identified the root cause in 86% of cases, up from 80% with standard tools.
• Participants reported that Causal Testing provided insights not available through JUnit or similar tools.
• The Holmes prototype, an open-source Eclipse plugin, successfully integrated causal testing into a real development environment.
• The method demonstrated practical viability and measurable improvement in debugging efficiency.