Search papers, labs, and topics across Lattice.
UWByteDanceKong Center for Construction RoboticsUC DavisUC Santa CruzUniversity of California Santa CruzJun 22, 2026arXiv:2606.23521This paper introduces Concordia, a runtime system designed for fault-tolerant inference in long-running LLM agents by utilizing a GPU-resident persistent kernel. By implementing checkpoint hooks at device synchronization points and enabling JIT compilation of specialized delta-checkpoint handlers, Concordia allows for efficient recovery of valuable state without the need for extensive application-specific logic. The key finding demonstrates that this approach significantly reduces recovery time and resource overhead compared to existing methods, which often require restarting the entire serving stack or rely on CPU involvement.
Concordia achieves fault tolerance for LLM inference by seamlessly integrating persistent kernel checkpointing, enabling rapid recovery without CPU bottlenecks.
Long-running LLM agents keep valuable state resident on GPUs: KV caches, request schedulers, communication state, and sometimes online adapters. Losing this state after a GPU or communicator failure can discard minutes to hours of work, yet existing recovery mechanisms either restart the whole serving stack or require application-specific checkpoint logic inside every attention and runtime component. This paper argues that fault tolerance for such workloads needs a GPU-resident execution context: checkpoint hooks must run at device synchronization points, observe binary kernels that frameworks and libraries actually execute, and recover without putting the host CPU on the critical path. We present Concordia, a runtime that uses a device-resident persistent kernel as the substrate for fault-tolerant LLM inference. Concordia interposes on GPU module loading and supports PTX- and SASS-level instrumentation, allowing checkpoint and pause hooks to be inserted below framework code and library boundaries. For each registered LLM state region, Concordia JIT-compiles a specialized delta-checkpoint handler -- for example, a KV-block scanner, adapter-page scanner, or recovery applier -- and hot-swaps it into the persistent kernel's operator table. The persistent kernel consumes a lock-free ring buffer of compute, checkpoint, append-log, and recovery tasks, so the same always-on executor triggers dirty-page detection, stages deltas, and appends committed records to a CPU-visible log in CXL memory or host DRAM.
