Search papers, labs, and topics across Lattice.
ARGUS is a low-overhead, always-on tracing and performance diagnosis system designed for large-scale LLM training in 10,000+ GPU production clusters. It achieves fine-grained observability with less than 2% overhead by decomposing observations into CPU call stacks, framework semantics, and GPU kernel execution, while compressing raw kernel events by approximately 3,700x. Over six months of deployment, ARGUS has effectively supported continuous fail-slow detection and performance optimization, demonstrating its capability to isolate various performance anomalies in real-time.
ARGUS achieves unprecedented fine-grained observability in 10,000+ GPU clusters with less than 2% overhead, revolutionizing performance diagnosis at scale.
Large-scale LLM training requires always-on, fine-grained observability for effective performance diagnosis at scale. Coarse resource monitors alone cannot localize root causes, and fine-grained profilers incur prohibitive (5%-30%) overheads and massive trace volumes, making always-on deployment impractical in large production clusters. We propose ARGUS, a low-overhead, fine-grained, always-on tracing and real-time analysis system for training workloads in 10,000+ GPU-scale production clusters. ARGUS decomposes observation along the training call hierarchy into CPU call stacks, framework semantics, and GPU kernel execution, with always-on collection under a combined overhead of less than 2%. It builds a unified data pipeline and compresses raw kernel events by approximately 3,700x from 10 MB to 2.7 KB per rank per step. Its progressive diagnosis framework automatically isolates anomalous windows, straggler ranks, and degraded kernels through iteration-time, phase-level, and kernel-level analysis. Deployed for over six months on a 10,000+ GPU production cluster, ARGUS has supported continuous fail-slow detection and performance optimization. Our case studies further demonstrate its effectiveness across representative anomalies, including compute stragglers, link degradation, pipeline-bubble amplification, FlashAttention JIT stalls, and compute stragglers masked by communication symptoms.
