Inference & Quantization - Weekly Roundup

A novel Kronecker Delta Rule enables associative memories to revise targeted information with minimal interference, unlocking stable, long-context linear attention.

A simple two-layer transformer with $O(\log\log N)$-bit precision can perfectly copy arbitrary strings up to length $N$ and terminate correctly, a feat impossible for standard transformers with finite-precision softmax.

Lifelong agents forget up to 40% of context under resource constraints, but learning memory worth unlocks significant performance gains.

Decentralized models can learn to route queries to specialized neighbors at inference time, unlocking significant performance gains without a central orchestrator.

Neural posterior estimation's overconfidence problem is solved by splitting the optimization of mean and variance, yielding better out-of-sample performance.

A novel Kronecker Delta Rule enables associative memories to revise targeted information with minimal interference, unlocking stable, long-context linear attention.

A simple two-layer transformer with $O(\log\log N)$-bit precision can perfectly copy arbitrary strings up to length $N$ and terminate correctly, a feat impossible for standard transformers with finite-precision softmax.

Lifelong agents forget up to 40% of context under resource constraints, but learning memory worth unlocks significant performance gains.

Decentralized models can learn to route queries to specialized neighbors at inference time, unlocking significant performance gains without a central orchestrator.

Neural posterior estimation's overconfidence problem is solved by splitting the optimization of mean and variance, yielding better out-of-sample performance.

Polynomial gradient orthogonalization achieves 15-18% faster runtimes than state-of-the-art methods by leveraging optimized polynomial circuits.

Near-autoregressive quality is achievable with parallel decoding, unlocking significant inference speedups across model scales.

Model merging is revealed as an optimal, closed-form stage of distillation, enabling efficient foundation model consolidation.

Adapting MoE experts directly, not through bypass adapters, unlocks the best downstream performance with minimal general capability loss.

Sparse attention in video diffusion models can be corrected using cached dense refinements, recovering fidelity without altering the reuse schedule.

Self-distillation in CTR prediction simplifies to a logit-averaged loss plus a disagreement penalty, enabling sample-free training and boosting single-tower accuracy.

Iterative GNNs trained on solver trajectories outperform single-pass SOCP solvers and enable warm-starting classical optimizers.

Adversarial attacks create provable geometric inconsistencies across prime-quantized and CNN embedding spaces, enabling theoretically guaranteed detection.

A novel minimax game formulation for multi-teacher distillation dynamically allocates optimization budget to ensure equitable progress across domains, even those with slower learning rates.

Agentic AI inference can be accelerated by 2-6x with history-free speculative decoding that eliminates KV cache overhead and boosts acceptance rates by up to 70%.

Remote KV caches can cripple LLM prefill latency, but sharding the KV store across parallel ranks and enabling asynchronous loading slashes p99 time from 83s to under 1s.

A strong LLM can review grounded drafts from a weak model, retaining most task performance while halving latency and memory traffic.

Distilling explicit, reward-guided skills into VLMs without the student model ever seeing the skills themselves unlocks superior multimodal performance.

Spatial memory via frustum slab routing slashes video world model inference time by over 4x without sacrificing generation quality.

Gating policy distillation with directional agreement between early and late RL shifts unlocks more effective knowledge transfer, boosting LLM performance on reasoning tasks.

Lossless compression of sparse ReLU activations slashes LLM activation VRAM by up to 31% with near-zero overhead.

Transformers trained with ABNAT learn to mimic clean attention maps, achieving up to 85.77% less attention deviation and 11.28% higher accuracy on noisy analog hardware.

Frozen LLMs reject sender KV caches even when geometrically similar, failing due to competitive attention, not fidelity.

Deep learning compilers are silently introducing bugs through state retention, with 35 stateful issues found in just four frameworks.

A simple, fixed-budget best-arm identification algorithm can be made nearly as adaptive as state-of-the-art methods by optimizing its phase schedule, achieving $11/12$ of the optimal normalized rate for Gaussian arms.

A two-bit code and shared reconstruction program slash LLM conditional memory storage by 75%, enabling 4x larger tables on GPU.

A 32.5x smaller grid foundation model achieves superior reactive power prediction and significant active power improvements over its larger teacher, enabling real-time edge inference.

Non-monotonic and fine-granular quantized activations are now efficiently implementable on custom DNN accelerators, unlocking broader hardware optimization.

Byte-level LLMs can achieve a 25% jump in math accuracy by patching where text is *new*, not just surprising.

Sharing LLM internal states is efficient, but only if you augment the KV cache and replace activations, not swap them.

Neural audio watermarking detectors can now reliably detect legacy content even after updates, while simultaneously rejecting unauthorized embedders, by explicitly training for angular separation in the embedding space.

Consolidating specialist LLMs into a single, more accurate generalist is possible with *zero* gradient updates, relying solely on forward passes and input statistics.

Learned token visibility control dramatically boosts LLM reasoning accuracy, achieving state-of-the-art performance on GSM8K and MATH-500 with a tiny, frozen inference-time policy.

Train LLMs with up to $1.455\times$ less memory and enable $3.5\times$ longer context lengths by losslessly compressing tensors on-the-fly.

A frozen, 16MB vocabulary can match or beat learned embeddings in small LLMs, unlocking dramatic speedups and memory reductions for edge deployment.

Aggressive KV cache compression can be achieved with near-zero accuracy loss by treating the cache as a trainable neural network optimized at test-time.

A simple, fixed query rule improves FID by over 2 points on CIFAR-10 without retraining, revealing prediction horizons are state-sensitive inference controls.

A single, stage-specific objective per expert in a decoupled diffusion distillation framework enables state-of-the-art video generation with a 10,000x reduction in training cost.

Teachers can be coaxed into recovering from their own errors during distillation, unlocking substantial gains in student model performance.

Distilling LLM agents by supervising semantic actions, not token sequences, unlocks significant performance gains and bypasses tokenization constraints.

Most self-distillation methods for agentic RL fail to consistently outperform a strong baseline, and impressive final scores often hide unstable training dynamics and substantial computational overhead.

Continuous integration over a piecewise linear memory curve unlocks exact normalized attention with analytical gradients, outperforming softmax and focal attention in causal Transformers.

A single, elastic transformer model can now dynamically adapt its compute budget across a continuum of sparsity levels, achieving up to 1.80x higher throughput and 1.26x lower latency on A100 GPUs without retraining.

A novel PPO variant leverages complementary temporal dynamics of self-teacher signals and critic value predictions to boost reasoning accuracy by up to 2.59pp.

LLM reasoning can be boosted at inference time by separately sharpening reasoning paths and final answers, outperforming existing power-sampling methods across multiple benchmarks.

Quantizing MoE models with mixed precision is significantly improved by calibrating expert error sensitivity using an information-theoretic contrast target, outperforming simple ranking-based methods.

Speculative decoding drafters can achieve over 2x speedup on 256K context by treating distant tokens as shared far memory with sharpened attention.

A simple reconstruction pre-training stage unlocks state-of-the-art one-step generative modeling with 10x less compute.

A closed-form, covariance-aware orthogonal transform for LLM quantization achieves state-of-the-art performance with a third of the memory and time of learned rotation methods.

Explicitly training LLM agents on runtime semantics via Harness Policy Distillation unlocks a 7.5x increase in autonomous task success and a 2.5x boost in runtime compliance on TRAJECT-Bench.

Deep learning DoA estimators can drastically cut inference time, but SVD-based beamforming remains the primary latency bottleneck in real-time ISAC.

Robot policies can now adapt to unseen terrains and payloads in real-time, boosting performance by up to 80% without needing rewards or retraining.

VLMs on edge devices are surprisingly energy-intensive, with text generation dominating image processing costs, and NPU offloading drastically outperforming CPU backends.

Amortized batch black-box optimization via flow matching generates variable-size batches in constant time, outperforming traditional methods on expensive optimization tasks.

A novel two-stage distillation framework achieves superior generalist reward model performance by first cultivating domain specialists and then integrating them, even surpassing individual teacher capabilities.

Region-level learning with granular balls dramatically boosts TSVR's robustness against noise and outliers, outperforming point-wise methods on challenging real-world regression tasks.

Prioritizing semantic integrity in early generative stages unlocks superior reward-mode balance in flow matching models.

Real-time video object removal is now possible at 26 FPS, enabling interactive editing of arbitrarily long videos without latency or quality degradation.

Speculative drafting can now dynamically query and integrate historical context from recurrent states, unlocking $3.16\times$ speedups on LLM generation.

Interactive video generation can now achieve high quality with large context windows without sacrificing real-time responsiveness.

MoE serving can now scale 4.5x-11.7x faster by reusing existing runtime state, sidestepping costly instance reconstruction.

A novel bandit algorithm finds the optimal self-speculative decoding configuration for VLMs in real-time, boosting inference speed by up to 11% without retraining.

A single latent cache can unify diverse visual representations and spatiotemporal queries for efficient, consistent dynamic world modeling across multiple video generation tasks.

A single 24GB GPU can now perform untiled 4K VSR at 2x DiT speed, thanks to hybrid attention that balances local and global interactions.

A small, training-free "brief thinker" can cut LLM reasoning tokens by 31% without sacrificing accuracy.

Image tokens can be compressed by 94% by prioritizing semantic detail over spatial uniformity, unlocking efficient vision-language models.

Unbalanced Wasserstein flow elegantly resolves finite-batch marginal fluctuations in generative models, unlocking significant unconditional generation gains on complex datasets.

Quantum state discrimination errors are reduced by 2x by encoding auxiliary qubit readout signals and dynamically calibrating neural decoders to the evolving circuit environment.

Pruning LLMs while preserving their influence functions is now over 28x faster and more effective, unlocking better data selection for downstream tasks.

Training-free conditional sampling for Schrödinger Bridges is now possible, achieving state-of-the-art results on image generation and inverse problems by twisting SB's inherent endpoint prediction capabilities.

Summary tokens are not just compressed context, but also dynamic KV cache indices, unlocking flexible memory compression and reduced compute in hybrid attention.

Self-distillation methods fail to generalize reasoning gains to unseen problems and their gradients offer weak policy improvement signals, challenging their efficacy beyond in-distribution imitation.

LLMs can achieve superior KV cache compression by intelligently allocating quantization budgets between layer-wise and token-wise strategies, outperforming uniform or single-granularity approaches.

Causal inference estimation rates are surprisingly sensitive to covariate distribution smoothness, with rough designs demonstrably slowing down pointwise CATE estimation even when treatment effects and nuisance functions are well-behaved.

A single protein tokenizer now captures sequence, structure, and function simultaneously, unlocking joint generation and zero-shot retrieval across modalities.

Global optimization unlocks LLM fine-tuning with drastically reduced compute and memory, matching or exceeding gradient-based gains.

Circulant permutations unlock provably lower variance and storage for MinHash beyond CMinHash, with a new analytical tool that accurately predicts Minner estimator performance.

Random features achieve minimax optimality in kernel ridge regression even with misspecified models and heavy-tailed noise, requiring only a modest sample size increase for robustness.

Unknown GP hyperparameters can drastically distort tail-risk estimates in CVaR optimization, but a new algorithm robustly navigates this uncertainty to achieve near-oracle performance.

A novel attention mechanism slashes KV cache by 4x and training FLOPs by 7.9% while *improving* LLM quality, breaking the long-context "impossible triangle."

Generative image and video compression achieves significant memory and compute reduction by replacing dense Gaussian codebooks with implicit randomized Hadamard dictionaries, enabling real-time streaming video compression on a single GPU.

Quantization strategies can achieve up to 4x energy savings for LLM inference, with peak efficiency observed for long inputs and short outputs.

Few-step diffusion models can achieve state-of-the-art 4-NFE generation without auxiliary score models or adversarial training, matching or exceeding multi-step teachers.