[Paper Review] Debugging Crashes using Continuous Contrast Set Mining

Rebecca Qian, Yang Yu+4 more|arXiv (Cornell University)|Nov 12, 2019

Network Security and Intrusion Detection13 references4 citations

TL;DR

This paper proposes Continuous Contrast Set Mining (CCSM), a novel method for debugging software crashes by directly analyzing continuous and mixed-data features without discretization. It introduces weighted anomaly scores and uncertainty measures to improve interpretability and scalability, achieving a 40x speedup over discretization-based baselines in real-world crash triage at Facebook.

ABSTRACT

Facebook operates a family of services used by over two billion people daily on a huge variety of mobile devices. Many devices are configured to upload crash reports should the app crash for any reason. Engineers monitor and triage millions of crash reports logged each day to check for bugs, regressions, and any other quality problems. Debugging groups of crashes is a manually intensive process that requires deep domain expertise and close inspection of traces and code, often under time constraints. We use contrast set mining, a form of discriminative pattern mining, to learn what distinguishes one group of crashes from another. Prior works focus on discretization to apply contrast mining to continuous data. We propose the first direct application of contrast learning to continuous data, without the need for discretization. We also define a weighted anomaly score that unifies continuous and categorical contrast sets while mitigating bias, as well as uncertainty measures that communicate confidence to developers. We demonstrate the value of our novel statistical improvements by applying it on a challenging dataset from Facebook production logs, where we achieve 40x speedup over baseline approaches using discretization.

Motivation & Objective

• To address the challenge of manually intensive crash triage in large-scale production systems with millions of crash reports daily.
• To enable effective contrast set mining on mixed data types—especially continuous variables—without the need for discretization, which distorts distributions and reduces interpretability.
• To improve developer insight by unifying categorical and continuous contrast sets through a weighted anomaly score and incorporating uncertainty estimates.
• To scale contrast mining to high-dimensional datasets with thousands of features, as encountered in real-world mobile application crash logs.
• To demonstrate the method’s effectiveness on real iOS out-of-memory crash data where traditional logging is absent.

Proposed method

• Proposes a direct application of contrast set mining to continuous data, bypassing discretization by modeling feature distributions using parametric statistical tests (e.g., t-test, ANOVA) on continuous values.
• Introduces a weighted anomaly score that combines effect size (Cohen’s d) and statistical significance to prioritize meaningful contrasts while reducing bias from rare or noisy features.
• Employs uncertainty measures derived from confidence intervals to communicate developer confidence in detected contrast sets, improving trust and usability.
• Uses TF-IDF vectorization to encode user navigation event sequences into continuous embeddings, enabling tractable analysis of high-dimensional behavioral logs.
• Applies one-way ANOVA and two-sample t-tests to detect significant deviations in continuous feature distributions between crash groups and the general population.
• Implements a recursive partitioning strategy to explore feature subsets and identify high-impact contrast sets, with pruning heuristics based on minimum group size and effect size thresholds.

Experimental results

Research questions

• RQ1 Can contrast set mining be effectively applied to continuous features in crash data without discretization, preserving distributional fidelity and interpretability?
• RQ2 How can anomaly scores be unified across categorical and continuous features to support holistic crash analysis?
• RQ3 What impact does uncertainty quantification have on developer trust and decision-making in crash triage?
• RQ4 Can continuous contrast mining detect subtle, high-impact root causes—such as build-specific memory issues—that are missed by traditional methods?
• RQ5 How does the performance of continuous contrast mining compare to discretization-based approaches in real-world, high-dimensional crash datasets?

Key findings

• CCSM achieved a 40x speedup over baseline discretization-based contrast mining approaches when applied to Facebook’s production crash logs.
• The method successfully identified a critical x86 build issue with a 56% higher-than-expected crash rate, which was previously undetected due to low signal in small device cohorts.
• Developers reported that contrast sets highlighting poor network conditions were highly actionable, confirming that continuous feature analysis provides meaningful diagnostic insights.
• The weighted anomaly score effectively prioritized high-impact features, reducing noise from rare or low-evidence patterns while preserving sensitivity to subtle but critical deviations.
• Uncertainty estimates helped developers assess the reliability of findings, reducing false alarms and increasing confidence in triage decisions.
• The approach demonstrated strong scalability on datasets with thousands of features, maintaining performance and interpretability where discretization fails.