[Paper Review] FReD: Automated Debugging via Binary Search through a Process Lifetime

Kapil Arya, Tyler Denniston+2 more|arXiv (Cornell University)|Dec 20, 2012

Parallel Computing and Optimization Techniques31 references4 citations

TL;DR

FReD introduces an automated binary search technique through a program's execution lifetime to locate the root cause of difficult, non-temporally local bugs in complex multithreaded applications. By leveraging checkpointing, deterministic replay via a custom DMTCP plugin, and GDB integration, FReD enables reverse expression watchpoints that efficiently narrow down the exact source code statement where a program invariant transitions from correct to violated, achieving debug times between 29 and 406 seconds on a 16-core system.

ABSTRACT

Reversible debuggers have been developed at least since 1970. Such a feature is useful when the cause of a bug is close in time to the bug manifestation. When the cause is far back in time, one resorts to setting appropriate breakpoints in the debugger and beginning a new debugging session. For these cases when the cause of a bug is far in time from its manifestation, bug diagnosis requires a series of debugging sessions with which to narrow down the cause of the bug. For such "difficult" bugs, this work presents an automated tool to search through the process lifetime and locate the cause. As an example, the bug could be related to a program invariant failing. A binary search through the process lifetime suffices, since the invariant expression is true at the beginning of the program execution, and false when the bug is encountered. An algorithm for such a binary search is presented within the FReD (Fast Reversible Debugger) software. It is based on the ability to checkpoint, restart and deterministically replay the multiple processes of a debugging session. It is based on GDB (a debugger), DMTCP (for checkpoint-restart), and a custom deterministic record-replay plugin for DMTCP. FReD supports complex, real-world multithreaded programs, such as MySQL and Firefox. Further, the binary search is robust. It operates on multi-threaded programs, and takes advantage of multi-core architectures during replay.

Motivation & Objective

• To address the challenge of diagnosing 'difficult' bugs that lack good temporal locality, where the cause is far from the manifestation.
• To automate the process of narrowing down the root cause of such bugs without requiring manual reverse stepping through millions of instructions.
• To support complex, real-world multithreaded programs like MySQL and Firefox in a scalable and efficient debugging environment.
• To enable high-level, expression-based debugging queries (e.g., 'when did this invariant become false?') using automated binary search.
• To integrate with existing tools (GDB, DMTCP) while extending them with deterministic record-replay and reverse execution capabilities.

Proposed method

• FReD uses a combination of DMTCP for checkpoint-restart and a custom deterministic record-replay plugin to log system calls, pthread calls, and glibc library calls for determinism.
• It leverages GDB as the front-end debugger and integrates with it to allow reverse execution and expression evaluation.
• The core technique is a binary search over the program’s execution timeline, guided by user-defined expressions that transition from true to false.
• At each step, the system checkpoints, replays forward to a midpoint, and checks whether the target expression is still true, narrowing the search space by half.
• The algorithm locates the precise source-level statement that causes the expression to become false, avoiding reliance on manual breakpoints or step-by-step reversal.
• It supports multi-core replay by exploiting parallelism during deterministic replay, significantly improving performance on modern many-core architectures.

Experimental results

Research questions

• RQ1 Can automated binary search through a process lifetime efficiently locate the root cause of non-temporally local bugs in complex multithreaded programs?
• RQ2 How can deterministic replay and checkpointing be effectively combined with symbolic debuggers to enable scalable reverse debugging?
• RQ3 To what extent can reverse expression watchpoints reduce debugging time compared to manual reverse stepping in real-world applications?
• RQ4 Can the approach scale to production-grade, highly concurrent systems like MySQL and Firefox while maintaining correctness and performance?
• RQ5 What is the performance overhead of maintaining deterministic replay and enabling binary search in a reversible debugging environment?

Key findings

• FReD successfully locates the root cause of bugs in complex, multithreaded programs such as MySQL and Firefox using automated binary search.
• The system achieves debug times ranging from 29 to 406 seconds on a 16-core machine, demonstrating practical performance for real-world debugging.
• The reverse expression watchpoint feature enables precise localization of the statement that causes a program invariant to fail, without requiring manual reverse stepping.
• The integration with unmodified GDB and DMTCP ensures compatibility and extensibility while maintaining robustness across diverse workloads.
• The use of multi-core architectures during replay enables fast deterministic replay, making the binary search scalable and efficient.
• The approach outperforms manual reverse debugging by eliminating the need for iterative reverse-next or reverse-step commands over large execution traces.