[Paper Review] CADET: A Systematic Method For Debugging Misconfigurations using Counterfactual Reasoning.

Rahul Krishna, Shahriar Iqbal+3 more|arXiv (Cornell University)|Oct 12, 2020

Software System Performance and Reliability73 references4 citations

TL;DR

CADET is a causal debugging toolkit that uses counterfactual reasoning over a learned causal model to systematically identify, explain, and repair non-functional faults in highly-configurable systems. It achieves up to 17% higher accuracy, 28% higher performance gain, and 40× speedup over ML-based debugging methods, while outperforming multi-objective optimization by up to 9× in repair speed.

ABSTRACT

Modern computing platforms are highly-configurable with thousands of interacting configurations. However, configuring these systems is challenging. Erroneous configurations can cause unexpected non-functional faults. This paper proposes CADET (short for Causal Debugging Toolkit) that enables users to identify, explain, and fix the root cause of non-functional faults early and in a principled fashion. CADET builds a causal model by observing the performance of the system under different configurations. Then, it uses casual path extraction followed by counterfactual reasoning over the causal model to: (a) identify the root causes of non-functional faults, (b) estimate the effects of various configurable parameters on the performance objective(s), and (c) prescribe candidate repairs to the relevant configuration options to fix the non-functional fault. We evaluated CADET on 5 highly-configurable systems deployed on 3 NVIDIA Jetson systems-on-chip. We compare CADET with state-of-the-art configuration optimization and ML-based debugging approaches. The experimental results indicate that CADET can find effective repairs for faults in multiple non-functional properties with (at most) 17% more accuracy, 28% higher gain, and $40 imes$ speed-up than other ML-based performance debugging methods. Compared to multi-objective optimization approaches, CADET can find fixes (at most) $9 imes$ faster with comparable or better performance gain. Our case study of non-functional faults reported in NVIDIA's forum show that CADET can find $14%$ better repairs than the experts' advice in less than 30 minutes.

Motivation & Objective

• To address the challenge of debugging non-functional faults in highly-configurable systems with thousands of interdependent configuration parameters.
• To enable early, principled identification of root causes for performance-related faults using causal reasoning.
• To provide actionable, data-driven repair recommendations for configuration parameters that fix non-functional issues.
• To outperform existing ML-based and multi-objective optimization approaches in accuracy, speed, and effectiveness of fault repair.

Proposed method

• CADET constructs a causal model by observing system performance across diverse configurations.
• It performs causal path extraction to trace the influence of configuration parameters on performance objectives.
• Counterfactual reasoning is applied to the causal model to simulate 'what-if' scenarios and identify root causes of faults.
• The system estimates the impact of individual configuration parameters on performance and recommends optimal fixes.
• It leverages observed configuration-performance data to infer causal relationships without requiring prior domain knowledge.
• The framework integrates causal inference with configuration debugging to enable systematic, explainable fault resolution.

Experimental results

Research questions

• RQ1 Can CADET effectively identify the root causes of non-functional faults in highly-configurable systems using causal reasoning?
• RQ2 How does CADET's accuracy and efficiency in fault repair compare to state-of-the-art ML-based debugging methods?
• RQ3 To what extent can CADET outperform multi-objective optimization techniques in repair speed and performance gain?
• RQ4 Can CADET recommend fixes that match or exceed expert-level advice in real-world debugging scenarios?

Key findings

• CADET achieved up to 17% higher accuracy than ML-based debugging methods in identifying effective repairs for non-functional faults.
• It delivered 28% higher performance gain compared to other ML-based approaches in the same fault repair tasks.
• CADET provided a 40× speedup in repair identification compared to state-of-the-art ML-based debugging techniques.
• In comparison to multi-objective optimization, CADET found fixes up to 9× faster while maintaining or improving performance gain.
• In a real-world case study, CADET recommended repairs that were 14% better than expert advice from NVIDIA’s forum, all within 30 minutes.
• CADET successfully diagnosed and repaired non-functional faults across five highly-configurable systems deployed on NVIDIA Jetson SoCs.