Search papers, labs, and topics across Lattice.
The paper introduces SENTINEL, a novel verification mechanism for pipeline parallel (PP) decentralized training that addresses security risks arising from untrusted, geographically distributed nodes. SENTINEL uses lightweight momentum-based monitoring with exponential moving averages (EMAs) to detect corrupted inter-stage communication, verifying sequential activation/gradient transmission between layers. Theoretical convergence guarantees are provided, and experiments demonstrate successful training of up to 4B-parameter LLMs across untrusted distributed environments with up to 176 workers, maintaining model convergence and performance.
Training LLMs across 176 untrusted workers is now possible without computation duplication, thanks to a new verification mechanism that detects corrupted inter-stage communication in pipeline parallelism.
Decentralized training introduces critical security risks when executed across untrusted, geographically distributed nodes. While existing Byzantine-tolerant literature addresses data parallel (DP) training through robust aggregation methods, pipeline parallelism (PP) presents fundamentally distinct challenges. In PP, model layers are distributed across workers where the activations and their gradients flow between stages rather than being aggregated, making traditional DP approaches inapplicable. We propose SENTINEL, a verification mechanism for PP training without computation duplication. SENTINEL employs lightweight momentum-based monitoring using exponential moving averages (EMAs) to detect corrupted inter-stage communication. Unlike existing Byzantine-tolerant approaches for DP that aggregate parameter gradients across replicas, our approach verifies sequential activation/gradient transmission between layers. We provide theoretical convergence guarantees for this new setting that recovers classical convergence rates when relaxed to standard training. Experiments demonstrate successful training of up to 4B-parameter LLMs across untrusted distributed environments with up to 176 workers while maintaining model convergence and performance.
