Search papers, labs, and topics across Lattice.
This paper introduces SCOUT, a runtime failure-localization framework designed to identify and diagnose issues in LLM pre-training by leveraging strict-majority consensus among equivalent replicas. By aligning replica progress and using a Consensus Collective Communication (C3) abstraction, SCOUT effectively isolates outliers and ranks with divergent signatures, enabling real-time monitoring even when training jobs hang. The framework's integration with popular deep learning libraries and its ability to certify checkpoint numerical integrity significantly enhance the reliability of LLM training processes.
SCOUT reveals that real-time consensus among model replicas can pinpoint failure origins in LLM training, addressing critical issues that traditional methods overlook.
In LLM pre-training, synchronization propagates rank-local stalls, slowdowns, and numerical errors into job-wide symptoms, obscuring their origin. Existing diagnosis often relies on in-process monitors that cannot report after the trainer blocks or terminates, or on post-mortem logs that preserve only synchronized symptoms; offline health tests lose the workload and operating conditions that triggered the failure. We present SCOUT, a unified runtime failure-localization framework built on one design principle: identify outliers through strict-majority consensus among equivalent replicas. SCOUT aligns replica progress, timing, and numerical evidence, then uses its Consensus Collective Communication (C3) abstraction to identify ranks whose compact signatures disagree with their peers. An out-of-band CPU observer remains responsive when training hangs, whereas in-situ replay exercises recurring stragglers and silent data corruption (SDC) beside the live job with its model state, kernels, allocations, communication path, and thermal and memory pressure present. Collective fingerprints expose rank-local protocol divergence. Clean replay coverage certifies checkpoint numerical integrity, preventing recovery from selecting state corrupted by SDC. SCOUT integrates with PyTorch, TorchTitan, Megatron-Core, and DeepSpeed without training-loop or framework-source modifications. SCOUT is open source at https://github.com/LMResiliency/lm-resiliency.
