Inference & Quantization - Weekly Roundup

Remote KV caches can cripple LLM prefill latency, but sharding the KV store across parallel ranks and enabling asynchronous loading slashes p99 time from 83s to under 1s.

Distilling explicit, reward-guided skills into VLMs without the student model ever seeing the skills themselves unlocks superior multimodal performance.

A strong LLM can review grounded drafts from a weak model, retaining most task performance while halving latency and memory traffic.

Spatial memory via frustum slab routing slashes video world model inference time by over 4x without sacrificing generation quality.

Gating policy distillation with directional agreement between early and late RL shifts unlocks more effective knowledge transfer, boosting LLM performance on reasoning tasks.

Remote KV caches can cripple LLM prefill latency, but sharding the KV store across parallel ranks and enabling asynchronous loading slashes p99 time from 83s to under 1s.

Distilling explicit, reward-guided skills into VLMs without the student model ever seeing the skills themselves unlocks superior multimodal performance.

A strong LLM can review grounded drafts from a weak model, retaining most task performance while halving latency and memory traffic.

Spatial memory via frustum slab routing slashes video world model inference time by over 4x without sacrificing generation quality.

Gating policy distillation with directional agreement between early and late RL shifts unlocks more effective knowledge transfer, boosting LLM performance on reasoning tasks.

Lossless compression of sparse ReLU activations slashes LLM activation VRAM by up to 31% with near-zero overhead.

Transformers trained with ABNAT learn to mimic clean attention maps, achieving up to 85.77% less attention deviation and 11.28% higher accuracy on noisy analog hardware.

Frozen LLMs reject sender KV caches even when geometrically similar, failing due to competitive attention, not fidelity.

Deep learning compilers are silently introducing bugs through state retention, with 35 stateful issues found in just four frameworks.

A simple, fixed-budget best-arm identification algorithm can be made nearly as adaptive as state-of-the-art methods by optimizing its phase schedule, achieving $11/12$ of the optimal normalized rate for Gaussian arms.

A two-bit code and shared reconstruction program slash LLM conditional memory storage by 75%, enabling 4x larger tables on GPU.

A 32.5x smaller grid foundation model achieves superior reactive power prediction and significant active power improvements over its larger teacher, enabling real-time edge inference.

Non-monotonic and fine-granular quantized activations are now efficiently implementable on custom DNN accelerators, unlocking broader hardware optimization.

Byte-level LLMs can achieve a 25% jump in math accuracy by patching where text is *new*, not just surprising.

Sharing LLM internal states is efficient, but only if you augment the KV cache and replace activations, not swap them.

Neural audio watermarking detectors can now reliably detect legacy content even after updates, while simultaneously rejecting unauthorized embedders, by explicitly training for angular separation in the embedding space.

Consolidating specialist LLMs into a single, more accurate generalist is possible with *zero* gradient updates, relying solely on forward passes and input statistics.

Learned token visibility control dramatically boosts LLM reasoning accuracy, achieving state-of-the-art performance on GSM8K and MATH-500 with a tiny, frozen inference-time policy.

Train LLMs with up to $1.455\times$ less memory and enable $3.5\times$ longer context lengths by losslessly compressing tensors on-the-fly.

Aggressive KV cache compression can be achieved with near-zero accuracy loss by treating the cache as a trainable neural network optimized at test-time.

A simple, fixed query rule improves FID by over 2 points on CIFAR-10 without retraining, revealing prediction horizons are state-sensitive inference controls.

A single, stage-specific objective per expert in a decoupled diffusion distillation framework enables state-of-the-art video generation with a 10,000x reduction in training cost.

Teachers can be coaxed into recovering from their own errors during distillation, unlocking substantial gains in student model performance.

Distilling LLM agents by supervising semantic actions, not token sequences, unlocks significant performance gains and bypasses tokenization constraints.

Most self-distillation methods for agentic RL fail to consistently outperform a strong baseline, and impressive final scores often hide unstable training dynamics and substantial computational overhead.

Continuous integration over a piecewise linear memory curve unlocks exact normalized attention with analytical gradients, outperforming softmax and focal attention in causal Transformers.

A single, elastic transformer model can now dynamically adapt its compute budget across a continuum of sparsity levels, achieving up to 1.80x higher throughput and 1.26x lower latency on A100 GPUs without retraining.

A novel PPO variant leverages complementary temporal dynamics of self-teacher signals and critic value predictions to boost reasoning accuracy by up to 2.59pp.

LLM reasoning can be boosted at inference time by separately sharpening reasoning paths and final answers, outperforming existing power-sampling methods across multiple benchmarks.

Quantizing MoE models with mixed precision is significantly improved by calibrating expert error sensitivity using an information-theoretic contrast target, outperforming simple ranking-based methods.

Speculative decoding drafters can achieve over 2x speedup on 256K context by treating distant tokens as shared far memory with sharpened attention.

A simple reconstruction pre-training stage unlocks state-of-the-art one-step generative modeling with 10x less compute.

A closed-form, covariance-aware orthogonal transform for LLM quantization achieves state-of-the-art performance with a third of the memory and time of learned rotation methods.

Explicitly training LLM agents on runtime semantics via Harness Policy Distillation unlocks a 7.5x increase in autonomous task success and a 2.5x boost in runtime compliance on TRAJECT-Bench.

Deep learning DoA estimators can drastically cut inference time, but SVD-based beamforming remains the primary latency bottleneck in real-time ISAC.

Robot policies can now adapt to unseen terrains and payloads in real-time, boosting performance by up to 80% without needing rewards or retraining.

VLMs on edge devices are surprisingly energy-intensive, with text generation dominating image processing costs, and NPU offloading drastically outperforming CPU backends.

Amortized batch black-box optimization via flow matching generates variable-size batches in constant time, outperforming traditional methods on expensive optimization tasks.

A novel two-stage distillation framework achieves superior generalist reward model performance by first cultivating domain specialists and then integrating them, even surpassing individual teacher capabilities.

Region-level learning with granular balls dramatically boosts TSVR's robustness against noise and outliers, outperforming point-wise methods on challenging real-world regression tasks.

Prioritizing semantic integrity in early generative stages unlocks superior reward-mode balance in flow matching models.

Real-time video object removal is now possible at 26 FPS, enabling interactive editing of arbitrarily long videos without latency or quality degradation.

MoE serving can now scale 4.5x-11.7x faster by reusing existing runtime state, sidestepping costly instance reconstruction.

A novel bandit algorithm finds the optimal self-speculative decoding configuration for VLMs in real-time, boosting inference speed by up to 11% without retraining.

A single latent cache can unify diverse visual representations and spatiotemporal queries for efficient, consistent dynamic world modeling across multiple video generation tasks.

A single 24GB GPU can now perform untiled 4K VSR at 2x DiT speed, thanks to hybrid attention that balances local and global interactions.

A small, training-free "brief thinker" can cut LLM reasoning tokens by 31% without sacrificing accuracy.

Image tokens can be compressed by 94% by prioritizing semantic detail over spatial uniformity, unlocking efficient vision-language models.

Unbalanced Wasserstein flow elegantly resolves finite-batch marginal fluctuations in generative models, unlocking significant unconditional generation gains on complex datasets.

Quantum state discrimination errors are reduced by 2x by encoding auxiliary qubit readout signals and dynamically calibrating neural decoders to the evolving circuit environment.

Pruning LLMs while preserving their influence functions is now over 28x faster and more effective, unlocking better data selection for downstream tasks.

Training-free conditional sampling for Schrödinger Bridges is now possible, achieving state-of-the-art results on image generation and inverse problems by twisting SB's inherent endpoint prediction capabilities.

Summary tokens are not just compressed context, but also dynamic KV cache indices, unlocking flexible memory compression and reduced compute in hybrid attention.

Self-distillation methods fail to generalize reasoning gains to unseen problems and their gradients offer weak policy improvement signals, challenging their efficacy beyond in-distribution imitation.

LLMs can achieve superior KV cache compression by intelligently allocating quantization budgets between layer-wise and token-wise strategies, outperforming uniform or single-granularity approaches.

Few-step diffusion models can achieve state-of-the-art 4-NFE generation with simple on-policy velocity regression and a novel residual proxy, matching multi-step teacher performance without complex auxiliary models.

Causal inference estimation rates are surprisingly sensitive to covariate distribution smoothness, with rough designs demonstrably slowing down pointwise CATE estimation even when treatment effects and nuisance functions are well-behaved.

A single protein tokenizer now captures sequence, structure, and function simultaneously, unlocking joint generation and zero-shot retrieval across modalities.

Global optimization unlocks LLM fine-tuning with drastically reduced compute and memory, matching or exceeding gradient-based gains.

Circulant permutations unlock provably lower variance and storage for MinHash beyond CMinHash, with a new analytical tool that accurately predicts Minner estimator performance.

Random features achieve minimax optimality in kernel ridge regression even with misspecified models and heavy-tailed noise, requiring only a modest sample size increase for robustness.

Unknown GP hyperparameters can drastically distort tail-risk estimates in CVaR optimization, but a new algorithm robustly navigates this uncertainty to achieve near-oracle performance.

A novel attention mechanism slashes KV cache by 4x and training FLOPs by 7.9% while *improving* LLM quality, breaking the long-context "impossible triangle."

Generative image and video compression achieves significant memory and compute reduction by replacing dense Gaussian codebooks with implicit randomized Hadamard dictionaries, enabling real-time streaming video compression on a single GPU.

Quantization can slash LLM inference energy costs by up to 4x with minimal accuracy loss, revealing a sweet spot for peak efficiency with long inputs and short outputs.