Inference & Quantization - Weekly Roundup

MoE serving can now scale 4.5x-11.7x faster by reusing existing runtime state, sidestepping costly instance reconstruction.

A novel bandit algorithm finds the optimal self-speculative decoding configuration for VLMs in real-time, boosting inference speed by up to 11% without retraining.

A single latent cache can unify diverse visual representations and spatiotemporal queries for efficient, consistent dynamic world modeling across multiple video generation tasks.

A single 24GB GPU can now perform untiled 4K VSR at 2x DiT speed, thanks to hybrid attention that balances local and global interactions.

A small, training-free "brief thinker" can cut LLM reasoning tokens by 31% without sacrificing accuracy.

MoE serving can now scale 4.5x-11.7x faster by reusing existing runtime state, sidestepping costly instance reconstruction.

A novel bandit algorithm finds the optimal self-speculative decoding configuration for VLMs in real-time, boosting inference speed by up to 11% without retraining.

A single latent cache can unify diverse visual representations and spatiotemporal queries for efficient, consistent dynamic world modeling across multiple video generation tasks.

A single 24GB GPU can now perform untiled 4K VSR at 2x DiT speed, thanks to hybrid attention that balances local and global interactions.

A small, training-free "brief thinker" can cut LLM reasoning tokens by 31% without sacrificing accuracy.

Image tokens can be compressed by 94% by prioritizing semantic detail over spatial uniformity, unlocking efficient vision-language models.

Unbalanced Wasserstein flow elegantly resolves finite-batch marginal fluctuations in generative models, unlocking significant unconditional generation gains on complex datasets.

Quantum state discrimination errors are reduced by 2x by encoding auxiliary qubit readout signals and dynamically calibrating neural decoders to the evolving circuit environment.

Pruning LLMs while preserving their influence functions is now over 28x faster and more effective, unlocking better data selection for downstream tasks.

Training-free conditional sampling for Schrödinger Bridges is now possible, achieving state-of-the-art results on image generation and inverse problems by twisting SB's inherent endpoint prediction capabilities.

Summary tokens are not just compressed context, but also dynamic KV cache indices, unlocking flexible memory compression and reduced compute in hybrid attention.

Self-distillation methods fail to generalize reasoning gains to unseen problems and their gradients offer weak policy improvement signals, challenging their efficacy beyond in-distribution imitation.

LLMs can achieve superior KV cache compression by intelligently allocating quantization budgets between layer-wise and token-wise strategies, outperforming uniform or single-granularity approaches.

Causal inference estimation rates are surprisingly sensitive to covariate distribution smoothness, with rough designs demonstrably slowing down pointwise CATE estimation even when treatment effects and nuisance functions are well-behaved.

A single protein tokenizer now captures sequence, structure, and function simultaneously, unlocking joint generation and zero-shot retrieval across modalities.

Global optimization unlocks LLM fine-tuning with drastically reduced compute and memory, matching or exceeding gradient-based gains.

Circulant permutations unlock provably lower variance and storage for MinHash beyond CMinHash, with a new analytical tool that accurately predicts Minner estimator performance.

Random features achieve minimax optimality in kernel ridge regression even with misspecified models and heavy-tailed noise, requiring only a modest sample size increase for robustness.

Unknown GP hyperparameters can drastically distort tail-risk estimates in CVaR optimization, but a new algorithm robustly navigates this uncertainty to achieve near-oracle performance.

A novel attention mechanism slashes KV cache by 4x and training FLOPs by 7.9% while *improving* LLM quality, breaking the long-context "impossible triangle."

Generative image and video compression achieves significant memory and compute reduction by replacing dense Gaussian codebooks with implicit randomized Hadamard dictionaries, enabling real-time streaming video compression on a single GPU.

Quantization strategies can achieve up to 4x energy savings for LLM inference, with peak efficiency observed for long inputs and short outputs.

Few-step diffusion models can achieve state-of-the-art 4-NFE generation without auxiliary score models or adversarial training, matching or exceeding multi-step teachers.

LLM decoder states exhibit distinct informational and geometric compartments, enabling principled layer selection for speculative decoding that surpasses heuristic approaches.

Task-specific training, not just model scale, unlocks effective relative policy difference transfer in distillation, and the reference policy is key to isolating residual capabilities.

LLM distillation can achieve 18.55% higher math accuracy and 13.8 points better ALFWorld success by intelligently branching student trajectories at uncertain decision points, not just at turn starts.

Batching future MH proposals unlocks significant speedups for LLM power sampling without altering the core reasoning process.

Small models can now learn new skills without forgetting old ones, thanks to a novel distillation technique that leverages a larger model to guide the learning process.

Explicit 3D geometry, rendered as text, unlocks superior spatial reasoning in RGB-only VLMs by guiding trajectory correction during distillation.

Token-level LLM routing, previously hindered by system inefficiencies, now achieves up to 64x higher throughput with a novel request-centric, model-centric serving architecture.

LLM value caches are surprisingly redundant, allowing for significant memory compression through simple symmetry exploitation without hurting inference speed.

Cloud workload forecasting models can now achieve near-optimal accuracy with only 20% of the compute cost of naive online learning by adaptively updating only when drift is detected.

Spatio-temporal Bayesian optimization for point-process data is now scalable and efficient, achieving accurate peak discovery with minimal evaluations.

Random features and projections unlock efficient quadratic entropy estimation for complex distributions and distances, revealing nuanced interdisciplinary research landscapes.

End-to-end diffusion models can achieve state-of-the-art precipitation nowcasting with over 17x faster inference by forecasting directly in radar space.

Iterative refinement in data space, not flow space, unlocks fast, high-fidelity generative sampling without manual timestep tuning.

Distillation inference can reliably detect unauthorized LLM distillation by comparing suspect model outputs against teacher reasoning traces via shadow models.

A reverse Hebbian-inspired pruning method achieves 99% sparsity on ResNet-18 with only a 2.4% accuracy drop, dynamically allocating sparsity without layer-specific targets.

Pruning LLMs for decoding speedup is significantly improved by calibrating sparsity with actual generation activations, not just static text, and optimizing SpMV kernels.

Quantizing AdamW's second moment in preconditioner space, not state space, slashes validation loss gaps by up to 70% in 4-bit training.

Diffusion models bypass Langevin's condition-number bottleneck by leveraging time-dependent score trajectories, achieving sampling error independent of target covariance ill-conditioning.

Parameter-efficient adaptation methods can match full fine-tuning gains for tabular foundation models, while selective aggregation offers a clear path to improved predictions over ensemble averaging.

Self-testing a KV cache with targeted Q&A pairs, rather than rereading the whole context, dramatically boosts memory retention, especially under severe budget constraints.

Softmax attention selectively exploits latent structure in Gaussian mixtures, solving nonlinear tasks beyond linear attention's reach while matching linear performance on tractable problems.

Posterior sampling recovers signals from noisy one-bit compressed sensing with near-optimal sample complexity, even with learned priors that are only approximately correct.

Speculative decoding can achieve up to 6.6x speedup by parallelizing token verification and recovering intra-block dependencies with a lightweight contextual corrector.

Key contribution not extracted.

Key contribution not extracted.

Latent recurrent models now match non-recurrent baselines at one loop, unlocking compute-for-accuracy scaling without the prior "one-loop tax."

AI agents can now navigate the intricate trade-offs of post-route power optimization, learning to balance speculative power recovery with costly repair actions under strict timing and physical constraints.

Standard two-level softmax sampling is demonstrably biased; corrected methods achieve near-exact softmax approximations with negligible overhead.

An 8x token reduction and 11.1x latency improvement in video generation is achieved by training autoencoders that are generation-aware, not just reconstruction-aware.

LLMs can now independently update factual knowledge without retraining, using conditional memory as an editable interface.

Amortized agentic policy discovery enables personalized test-time scaling to satisfy complex, multidimensional user requirements, outperforming single-objective optimization on unseen profiles.

Ollivier-Ricci curvature reveals and rectifies spectral representation errors in GNN-to-MLP distillation, unlocking significant accuracy gains for graph-free student models.

Self-distillation can degrade student performance if the teacher exploits privileged information unavailable to the student, but a novel KL-regularized objective ensures teacher guidance aligns with student capabilities.

Hybrid LLMs exhibit a "Seesaw Effect" in context extension, with linear attention models thriving on continual pretraining and sliding-window models excelling at extrapolation, a distinction rooted in their positional inductive biases.

2-bit KV cache quantization for Looped Transformers achieves 80% memory savings and 4x throughput gains with near-BF16 accuracy by quantizing residuals relative to the final loop's KV states.

Distilling teacher-relative logit shifts, not just final policies, unlocks substantial gains in multi-teacher on-policy distillation by preventing inherited base biases from dominating learned preferences.

Fresh, shallow network activations can correct diffusion model feature forecasts, unlocking significant speedups and improved fidelity without any parameter updates.

Probabilistic forecasting models can now achieve 65% CRPS reduction with drastically fewer samples by directly learning the distribution shift between successive predictions.

Embedding compression can be dramatically improved by learning the optimal coding geometry from data, not just optimizing within a fixed one.

A single objective directly optimizes speculative decoding efficiency, eliminating auxiliary hyperparameters and achieving state-of-the-art performance across multiple tasks.

Continual learning is achieved without any parameter updates by performing in-context Bayesian inference over a learned latent concept space.

LLM capabilities diverge under latent sequence compression; language modeling, decodability, and reasoning do not share a single compression threshold.

Unbiased dataset pruning can be framed as a linear program, enabling efficient, label-free subset selection that outperforms geometric methods at small budgets.

Reweighted normalizing flows enable exponential convergence to arbitrary Boltzmann distributions via a novel WFR JKO scheme, bypassing common structural assumptions.

LLM-generated rules, not full trees, unlock efficient and interpretable few-shot tabular classification.

Smoothness in treatment effects unlocks near-optimal policy learning in high-dimensional, low-signal digital experimentation.

Long-horizon reasoning models can now achieve $O(1)$ memory and $O(N)$ time complexity while surpassing larger Transformer baselines on complex tasks.

Black-box MIPS oracles can approximate attention with $O(1/\varepsilon^2)$ keys by augmenting queries and keys, bypassing $\sqrt{n}$ and $\log n$ dependencies.

Generative MARL teachers fail by interpolating coordination modes; a novel optimal transport approach forces them to commit to distinct strategies, boosting student performance.

A simple, training-free calibration wrapper boosts tensor-network LLM compression by 2-6x, closing up to 60% of the perplexity gap without altering the underlying decomposition solver.

On-policy distillation excels at teaching LLMs to *use* their existing knowledge for complex reasoning, but doesn't actually teach them *new facts*.

Reward modeling inference costs plummet by up to 7x by performing judgments entirely in a compressed latent space, bypassing token-by-token generation.

Interactive video generation can be sped up by 1.4x by intelligently caching transformer residuals based on detected control transitions, without sacrificing quality.

Storing knowledge directly in LLM parameters offers a flexible, efficient alternative to in-context learning, bypassing context length limitations and repeated encoding costs.

Broader tokenizer alignment in on-policy distillation can actively harm performance by introducing noisy supervision signals.

Latency in voice assistants plummets by 20% by speculatively executing tool calls *during* speech recognition, not after.

Optimal $O(\sqrt{d})$ regret in online inverse linear optimization is now achievable in polynomial time, shattering previous computational barriers.

Spatiotemporal time series forecasting now boasts valid confidence intervals even with imputed labels and temporal dependencies, overcoming a critical limitation of existing PPI.

Speculative decoding's efficiency gains come at a steep security price, with jailbreak success rates soaring far faster than utility drops.

Transformer uncertainty is now reliably calibrated with linear-time attention.

Edge CNN adaptation can now achieve near-zero activation memory overhead by decoupling backward computations from full-width layer inputs.

Precise long-range evidence retrieval is unlocked by coupling block-sparse attention with learned prefix states, overcoming the fixed local window limitations of prior hybrid architectures.

Heterogeneous load forecasting scales efficiently by learning a small bank of low-rank adaptations for a shared probabilistic model, outperforming independent models and monolithic Transformers.

A novel debiased estimator for sparse SVMs reveals statistically significant features beyond standard variable selection, even in high-dimensional, correlated settings.

A learned graph structure and structural energy term significantly improve selective forecasting reliability by explicitly penalizing violations of inter-variable dependencies, outperforming whole-forecast scoring methods on key benchmarks.

Collaborative edge-cloud LLM inference can achieve 275% higher throughput by dynamically synchronizing heterogeneous requests to batch their common cloud-side suffix.

LLM pruning can be task-aware: DIPrune's dual importance scoring jointly optimizes static feature saliency and dynamic semantic evolution, outperforming prior methods on LLaVA and Qwen-VL.

Training-free LQEC recovers 84% of the FP16 accuracy gap at 2-bit quantization by decoupling calibration and compensation, outperforming baselines by over 20%.

Agent experience, even when uniformly unrewarded, can be distilled into predictive foresight to dramatically improve learning in sparse reward settings.

Removing a mean shift from data representations requires a signal-to-noise ratio that is $O(d)$ times higher than simply detecting it, a gap that standard linear methods cannot close.

FP4 RL for MoE LLMs can now match BF16 rollout quality with 5.4x speedup by directly aligning train-rollout quantization.

Train sparse GNNs with dynamic graph topologies across epochs without re-extracting structure, achieving state-of-the-art performance even at extreme edge budgets.

Accelerating VLA models without sacrificing task success is now certifiable, guaranteeing over $85\%$ of reference performance with $10\times$ speedups.

Data-free diffusion distillation now achieves FID 2.04 at 1-NFE by unifying generation and score estimation in a single student network.