Search papers, labs, and topics across Lattice.
This paper introduces NEURON-Fabric, a profile-guided runtime system designed to optimize low-bit gradient communication in distributed neural network training. By integrating calibrated operating profiles and model-aware runtime bindings, NEURON-Fabric effectively manages when to utilize low-bit aggregation versus full-precision communication, thereby maintaining training accuracy while significantly reducing communication costs. The results demonstrate that this approach preserves model performance across various architectures, validating its effectiveness in diverse workloads, including vision and language models.
Static low-bit gradient communication can destabilize training, but NEURON-Fabric ensures accuracy is preserved while slashing communication costs.
Large-scale neural-network training repeatedly aggregates gradients across devices, making communication a central cost in distributed learning. Low-bit gradient aggregation can reduce this cost, but applying it as a static replacement for full-precision communication can destabilize training because safe precision depends on training phase, model structure, runtime bucketization, and the communication substrate. This paper presents NEURON-Fabric, a profile-guided runtime system for controlled low-bit gradient communication. NEURON-Fabric uses calibrated operating profiles, model-aware runtime bindings, online training-health monitoring, and reducer-capacity checks to decide when low-bit aggregation should be admitted, when execution should fall back to FP32, and which model regions are eligible for each route. The runtime preserves model semantics inside mixed DDP buckets and treats reducer admission as an architecture-runtime co-design problem rather than as a standalone compression operator. Across vision, Transformer, and autoregressive language-model workloads, NEURON-Fabric validates the path from calibration to distributed communication-hook execution. Static low-bit communication can collapse training accuracy, while profile-guided control preserves accuracy near full-precision references or calibrated targets and reduces modeled gradient-communication traffic in the evaluated settings. Transformer and billion-parameter language-model checks show that the same routing and fallback mechanisms execute across model families and multi-node deployments. Reducer-side replay and reducer-path measurements identify when compact sign-count aggregation is expected to reduce communication cost and when endpoint capacity should trigger fallback.
