Distributed Systems & Hardware - Weekly Roundup

An RL agent learns to optimize SpTRSV parallelism by intelligently rewriting sparse matrix dependency graphs, achieving substantial performance gains with minimal structural changes.

A single auxiliary variable per group of failures unlocks near-optimal $T^{-4/5}$ convergence for large-scale Cox regression, outperforming stochastic baselines.

Lossy compression's inherent distortion patterns, not just update geometry, can reliably detect and mitigate model poisoning in federated learning.

Decentralized power allocation in complex MANETs is solved with a learned GNN that matches centralized optimization performance using only local, noisy channel data.

Consumer GPUs can now achieve up to 1.66x faster MoE inference by bypassing NCCL's CPU bottlenecks with a novel DMA-driven communication design.

An RL agent learns to optimize SpTRSV parallelism by intelligently rewriting sparse matrix dependency graphs, achieving substantial performance gains with minimal structural changes.

A single auxiliary variable per group of failures unlocks near-optimal $T^{-4/5}$ convergence for large-scale Cox regression, outperforming stochastic baselines.

Lossy compression's inherent distortion patterns, not just update geometry, can reliably detect and mitigate model poisoning in federated learning.

Decentralized power allocation in complex MANETs is solved with a learned GNN that matches centralized optimization performance using only local, noisy channel data.

Consumer GPUs can now achieve up to 1.66x faster MoE inference by bypassing NCCL's CPU bottlenecks with a novel DMA-driven communication design.

LLM inference auctions unlock economic efficiency and user utility by enabling dynamic bidding for compute priority, outperforming static pricing tiers.

Parallel search agents can now achieve higher accuracy than generative aggregators by leveraging TRACE to intelligently select the best trajectory from aggregated cross-rollout evidence.

Splitting models into fully decoupled, communication-free shards cuts zero-order gradient variance by roughly $1/N$, enabling backprop-free pretraining to beat standard monolithic baselines using significantly fewer perturbations.

Static parallelism layouts leave massive post-training efficiency on the table, but dynamically re-slicing distributed model states across GPUs cuts RL step latency by 28% with less than 0.1% transition overhead.

NVIDIA's hand-crafted GEMM heuristics leave massive performance on the table: grounding learned kernel selectors in static hardware-behavior estimates cuts selection regret by up to 64.2% without running an autotuning benchmark.

Non-CUDA hardware no longer requires costly post-training pipelines: conditioning memory rewards strictly on adopted code enables zero-shot LLM agents to synthesize NPU kernels that outperform vendor libraries by up to 44x.

Billion-parameter LLMs no longer require global backpropagation to pretrain competitively: sharing a single delayed readout across decoupled modules matches standard zero-shot accuracy while yielding up to 1.44x pipeline throughput.

Interleaving expensive Muon spectral steps with cheap Lion sign descent not only cuts distributed training wall-clock time by 33%, but actually beats full Muon and AdamW on final perplexity.

Analog LLM accelerators can claw back over 11 percentage points in zero-shot accuracy while calibrating 12x faster by analytically co-optimizing ADC output quantization alongside weight and activation clipping.

Minimizing energy consumption in LLM inference does not ensure ecological sustainability: spatial and temporal deployment choices cause stark divergence across carbon, water, and biodiversity impacts that can double worst-case environmental regret.

Dynamic MoE load balancing can run at every single microbatch with zero CPU synchronization by having GPUs deterministically solve bitwise-identical expert replication plans locally, boosting cluster throughput by up to 11.4%.

Blackwell GPUs' FP4 Tensor Cores unlock 2.81x output throughput for long-context LLMs by strategically using mixed precision and rank-one smoothing, bypassing attention bottlenecks.

HeteroFold unlocks $10.7\times$ faster cross-family KV cache transfer for LLMs, eliminating receiver prefill and matching text-based communication performance.

Latent communication can be compressed by 75% without accuracy loss by conditioning KV cache transmissions on receiver information needs.