Search papers, labs, and topics across Lattice.
This paper introduces DDB, a source-level interactive debugger specifically designed for distributed applications, addressing the limitations of traditional debuggers that struggle with multi-process environments. By implementing a Distributed Backtrace (DBT) for unified call stacks, an intent-preserving control plane for breakpoint management, and a Pause-Erased Time (PET) mechanism to prevent timeout cascades, DDB allows developers to conduct live debugging without the typical pitfalls of distributed systems. Evaluations demonstrate that DDB not only achieves rapid backtrace latencies and minimal throughput overhead but also significantly improves fault localization success rates compared to existing tools.
DDB achieves a staggering 100% fault localization success rate, revolutionizing interactive debugging in distributed systems.
Interactive debugging is an effective tool for understanding program behavior at the source level, allowing developers to pause execution, navigate the call stack, and inspect runtime state. However, interactive debuggers are designed for single-process execution, and interactive debugging has been widely considered impractical for distributed systems. Call stacks stop at process boundaries, debugging state fails to survive infrastructure dynamics, and, most critically, debugger-induced execution pauses trigger catastrophic timeout cascades that destroy the intended debug flow. Consequently, developers are forced to abandon live hypothesis testing in favor of unwieldy and iterative log-and-redeploy cycles. We present DDB, a source-level interactive debugger that extends interactive debugging capabilities to distributed applications. We show that each of these challenges admits a targeted solution. To bridge disjoint processes, Distributed Backtrace (DBT) embeds compact causality metadata in every RPC and reconstructs a unified call stack across RPC boundaries. To manage the lifecycle of a distributed session, an intent-preserving control plane automatically coordinates and propagates breakpoints across dynamic process sets. To make pausing safe, Pause-Erased Time (PET) virtualizes each process's clock, decoupling logical time from physical pauses and preventing timeout cascades. DDB integrates with an RPC framework in 20-60 lines of code. Evaluated on gRPC, ServiceWeaver, Nu, and Quicksand across up to 122 processes, DDB achieves 30ms median cross-RPC backtrace latency, sub-5 ms time jump under repeated execution pauses, and adds 1-5% throughput overhead, comparable to attaching a single-process debugger. In a controlled user study, DDB achieves a 100% fault localization success rate (compared to 38.5% for baseline tools) with a median localization time of ~8 minutes.
