Distributed Systems & Hardware - Weekly Roundup

Remote KV caches can cripple LLM prefill latency, but sharding the KV store across parallel ranks and enabling asynchronous loading slashes p99 time from 83s to under 1s.

Hyperscalers can achieve 1.5x training throughput gains on AMD GPUs by moving beyond direct CUDA porting to hardware-model co-design and agentic optimization workflows.

Deep learning compilers are silently introducing bugs through state retention, with 35 stateful issues found in just four frameworks.

Fine-grained data balancing in heterogeneous pipelines unlocks 2x training speedups for large models by overcoming memory bottlenecks and coarse workload assignments.

Non-monotonic and fine-granular quantized activations are now efficiently implementable on custom DNN accelerators, unlocking broader hardware optimization.

Remote KV caches can cripple LLM prefill latency, but sharding the KV store across parallel ranks and enabling asynchronous loading slashes p99 time from 83s to under 1s.

Hyperscalers can achieve 1.5x training throughput gains on AMD GPUs by moving beyond direct CUDA porting to hardware-model co-design and agentic optimization workflows.

Deep learning compilers are silently introducing bugs through state retention, with 35 stateful issues found in just four frameworks.

Fine-grained data balancing in heterogeneous pipelines unlocks 2x training speedups for large models by overcoming memory bottlenecks and coarse workload assignments.

Non-monotonic and fine-granular quantized activations are now efficiently implementable on custom DNN accelerators, unlocking broader hardware optimization.

A novel PI-DeepONet implementation slashes training time by up to $3.75\times$ and makes 3D physics-informed learning feasible, overcoming critical computational barriers.

Decentralized agent swarms can learn to self-orchestrate their collective behavior and individual capabilities simultaneously, outperforming fixed coordination strategies.

Clients can achieve superior worst-decile accuracy in federated learning by stopping local training based on an optimal stopping framework that balances loss reduction against drift, outperforming tuned baselines and existing adaptive methods.

A neural evaluator predicts Lean 4 proof acceptance with 87.8% accuracy and achieves 4.2x the throughput of a multi-worker Lean baseline, drastically accelerating proof discovery.

Injecting uncertainty samples into DNNs during federated training unlocks robust, near-centralized performance for joint communication and computation resource allocation in XR systems, outperforming state-of-the-art.

Train LLMs with up to $1.455\times$ less memory and enable $3.5\times$ longer context lengths by losslessly compressing tensors on-the-fly.

VLMs on edge devices are surprisingly energy-intensive, with text generation dominating image processing costs, and NPU offloading drastically outperforming CPU backends.

Federated learning's bias-variance trade-off is precisely quantified in high dimensions, revealing how GTVMin regularization optimally balances local adaptation and global collaboration.

MoE serving can now scale 4.5x-11.7x faster by reusing existing runtime state, sidestepping costly instance reconstruction.

Larger minibatch optimal transport couplings in flow matching, enabled by a novel entropic approximation, significantly boost training stability and generation performance.

A simple greedy approach offers a $\frac{1}{\sqrt{e}}$-approximation for optimal policies in heterogeneous asynchronous multi-player bandits, a problem previously intractable.

Refresh timing, not just frequency, dictates Kronecker-preconditioned LLM training performance, with optimal placement flipping based on learning rate.

Equivariant MLIPs just got a 250x speedup: LunaTP unlocks efficient tensor products by rethinking parallel execution.

Distributed attention can achieve near-zero communication overhead for mask localization and compute-communication overlap, unlocking massive throughput gains for long-context LLMs.

Quantization can slash LLM inference energy costs by up to 4x with minimal accuracy loss, revealing a sweet spot for peak efficiency with long inputs and short outputs.