Distributed Systems & Hardware - Weekly Roundup

Asynchronous LLM training can be made robust to stale rollouts by minimizing a novel group-mass capping bias bound, outperforming existing methods on reasoning tasks.

A single controller can now robustly learn stochastic transport across a range of initial distribution shifts, outperforming standard SB and noise augmentation on challenging tasks.

LLM training CI is now 77.5% faster and 63.9% more GPU-efficient by intelligently pruning and prioritizing tests based on runtime evidence.

Semantic communication via VLM orchestrators unlocks robust, long-horizon coordination for multi-robot systems, outperforming prior single-robot approaches on complex manipulation tasks.

Quasi-Newton methods now conquer deep learning's non-convexity and scale, outperforming Adam and others on challenging tasks like 136M-parameter physics-informed models.

Asynchronous LLM training can be made robust to stale rollouts by minimizing a novel group-mass capping bias bound, outperforming existing methods on reasoning tasks.

A single controller can now robustly learn stochastic transport across a range of initial distribution shifts, outperforming standard SB and noise augmentation on challenging tasks.

LLM training CI is now 77.5% faster and 63.9% more GPU-efficient by intelligently pruning and prioritizing tests based on runtime evidence.

Semantic communication via VLM orchestrators unlocks robust, long-horizon coordination for multi-robot systems, outperforming prior single-robot approaches on complex manipulation tasks.

Quasi-Newton methods now conquer deep learning's non-convexity and scale, outperforming Adam and others on challenging tasks like 136M-parameter physics-informed models.

An RL agent learns to optimize SpTRSV parallelism by intelligently rewriting sparse matrix dependency graphs, achieving substantial performance gains with minimal structural changes.

A single auxiliary variable per group of failures unlocks near-optimal $T^{-4/5}$ convergence for large-scale Cox regression, outperforming stochastic baselines.

Lossy compression's inherent distortion patterns, not just update geometry, can reliably detect and mitigate model poisoning in federated learning.

Decentralized power allocation in complex MANETs is solved with a learned GNN that matches centralized optimization performance using only local, noisy channel data.

Consumer GPUs can now achieve up to 1.66x faster MoE inference by bypassing NCCL's CPU bottlenecks with a novel DMA-driven communication design.

LLM inference auctions unlock economic efficiency and user utility by enabling dynamic bidding for compute priority, outperforming static pricing tiers.

Parallel search agents can now achieve higher accuracy than generative aggregators by leveraging TRACE to intelligently select the best trajectory from aggregated cross-rollout evidence.

Splitting models into fully decoupled, communication-free shards cuts zero-order gradient variance by roughly $1/N$, enabling backprop-free pretraining to beat standard monolithic baselines using significantly fewer perturbations.

Static parallelism layouts leave massive post-training efficiency on the table, but dynamically re-slicing distributed model states across GPUs cuts RL step latency by 28% with less than 0.1% transition overhead.

NVIDIA's hand-crafted GEMM heuristics leave massive performance on the table: grounding learned kernel selectors in static hardware-behavior estimates cuts selection regret by up to 64.2% without running an autotuning benchmark.

Non-CUDA hardware no longer requires costly post-training pipelines: conditioning memory rewards strictly on adopted code enables zero-shot LLM agents to synthesize NPU kernels that outperform vendor libraries by up to 44x.

Billion-parameter LLMs no longer require global backpropagation to pretrain competitively: sharing a single delayed readout across decoupled modules matches standard zero-shot accuracy while yielding up to 1.44x pipeline throughput.

Interleaving expensive Muon spectral steps with cheap Lion sign descent not only cuts distributed training wall-clock time by 33%, but actually beats full Muon and AdamW on final perplexity.

Analog LLM accelerators can claw back over 11 percentage points in zero-shot accuracy while calibrating 12x faster by analytically co-optimizing ADC output quantization alongside weight and activation clipping.

Minimizing energy consumption in LLM inference does not ensure ecological sustainability: spatial and temporal deployment choices cause stark divergence across carbon, water, and biodiversity impacts that can double worst-case environmental regret.

Dynamic MoE load balancing can run at every single microbatch with zero CPU synchronization by having GPUs deterministically solve bitwise-identical expert replication plans locally, boosting cluster throughput by up to 11.4%.

Blackwell GPUs' FP4 Tensor Cores unlock 2.81x output throughput for long-context LLMs by strategically using mixed precision and rank-one smoothing, bypassing attention bottlenecks.

HeteroFold unlocks $10.7\times$ faster cross-family KV cache transfer for LLMs, eliminating receiver prefill and matching text-based communication performance.