[Paper Review] Understanding Differences among Executions with Variational Traces

Jens Meinicke, Chu-Pan Wong+2 more|arXiv (Cornell University)|Jul 10, 2018

Security and Verification in Computing1 references4 citations

TL;DR

This paper introduces variational traces—a compact, dynamic representation of control-flow and data-flow differences across multiple program executions—to help developers understand interaction faults in configurable software. By leveraging variational execution to scale trace comparison across exponentially many input configurations, the authors demonstrate in a user study that developers using their Eclipse plugin Varviz debug faults over twice as fast as with standard debugging tools.

ABSTRACT

One of the main challenges of debugging is to understand why the program fails for certain inputs but succeeds for others. This becomes especially difficult if the fault is caused by an interaction of multiple inputs. To debug such interaction faults, it is necessary to understand the individual effect of the input, how these inputs interact and how these interactions cause the fault. The differences between two execution traces can explain why one input behaves differently than the other. We propose to compare execution traces of all input options to derive explanations of the behavior of all options and interactions among them. To make the relevant information stand out, we represent them as variational traces that concisely represents control-flow and data-flow differences among multiple concrete traces. While variational traces can be obtained from brute-force execution of all relevant inputs, we use variational execution to scale the generation of variational traces to the exponential space of possible inputs. We further provide an Eclipse plugin Varviz that enables users to use variational traces for debugging and navigation. In a user study, we show that users of variational traces are more than twice as fast to finish debugging tasks than users of the standard Eclipse debugger. We further show that variational traces can be scaled to programs with many options.

Motivation & Objective

• To address the challenge of debugging interaction faults in configurable software where multiple inputs jointly cause failures.
• To overcome the limitations of standard debuggers that analyze only one input at a time, making it hard to compare behaviors across configurations.
• To provide a scalable, dynamic method for comparing execution traces across many input combinations to explain why certain inputs trigger faults.
• To develop an interactive tool (Varviz) that enables developers to navigate and understand differences in program behavior across configurations.
• To evaluate both the effectiveness and scalability of variational traces in real-world debugging scenarios involving complex feature interactions.

Proposed method

• Use variational execution (faceted execution) to simultaneously execute all relevant input configurations in a shared computation, avoiding redundant execution and trace recording.
• Dynamically track and annotate statements and states with their corresponding input configurations to preserve provenance and enable on-the-fly identification of differences.
• Construct variational traces by merging aligned traces from all executions, sharing common control and data flow while annotating divergent paths with their respective input contexts.
• Apply filtering techniques to reduce trace size by suppressing irrelevant or redundant information, focusing only on differences that explain fault behavior.
• Implement an interactive Eclipse plugin, Varviz, to visualize variational traces and support navigation, filtering, and exploration of execution differences.
• Leverage trace alignment and provenance tracking to enable dynamic, on-demand analysis without pre-recording all traces, thus avoiding memory explosion.

Experimental results

Research questions

• RQ1 How can developers effectively compare the behavior of multiple program executions to understand interaction faults in configurable systems?
• RQ2 What techniques can scale the comparison of execution traces across exponentially many input configurations without incurring prohibitive memory or computation costs?
• RQ3 Can a dynamic, variational approach to trace analysis outperform traditional debuggers in terms of debugging speed and accuracy?
• RQ4 How effective are variational traces in explaining complex fault causes involving multiple interacting inputs?
• RQ5 To what extent can variational traces be scaled and compressed while preserving essential differences for debugging?

Key findings

• Users of the Varviz plugin were more than twice as fast at completing debugging tasks compared to users of the standard Eclipse debugger.
• The average time to complete a debugging task was reduced to less than half with variational traces, indicating a significant improvement in developer efficiency.
• Variational traces effectively captured and explained interaction faults involving multiple inputs, such as the example where both `flag` and `ipv4` parameters jointly caused a crash.
• The authors demonstrated that variational execution enables scalable trace generation across exponentially large input spaces without full trace recording.
• Filters applied to variational traces successfully reduced trace size, maintaining conciseness while preserving fault-relevant differences.
• Evaluation on six highly configurable systems showed that variational traces remain practical and effective even in complex, real-world scenarios.