Inference & Quantization - Weekly Roundup

LLM decoder states exhibit distinct informational and geometric compartments, enabling principled layer selection for speculative decoding that surpasses heuristic approaches.

Task-specific training, not just model scale, unlocks effective relative policy difference transfer in distillation, and the reference policy is key to isolating residual capabilities.

LLM distillation can achieve 18.55% higher math accuracy and 13.8 points better ALFWorld success by intelligently branching student trajectories at uncertain decision points, not just at turn starts.

Batching future MH proposals unlocks significant speedups for LLM power sampling without altering the core reasoning process.

Small models can now learn new skills without forgetting old ones, thanks to a novel distillation technique that leverages a larger model to guide the learning process.

LLM decoder states exhibit distinct informational and geometric compartments, enabling principled layer selection for speculative decoding that surpasses heuristic approaches.

Task-specific training, not just model scale, unlocks effective relative policy difference transfer in distillation, and the reference policy is key to isolating residual capabilities.

LLM distillation can achieve 18.55% higher math accuracy and 13.8 points better ALFWorld success by intelligently branching student trajectories at uncertain decision points, not just at turn starts.

Batching future MH proposals unlocks significant speedups for LLM power sampling without altering the core reasoning process.

Small models can now learn new skills without forgetting old ones, thanks to a novel distillation technique that leverages a larger model to guide the learning process.

Explicit 3D geometry, rendered as text, unlocks superior spatial reasoning in RGB-only VLMs by guiding trajectory correction during distillation.

Token-level LLM routing, previously hindered by system inefficiencies, now achieves up to 64x higher throughput with a novel request-centric, model-centric serving architecture.

LLM value caches are surprisingly redundant, allowing for significant memory compression through simple symmetry exploitation without hurting inference speed.

Cloud workload forecasting models can now achieve near-optimal accuracy with only 20% of the compute cost of naive online learning by adaptively updating only when drift is detected.

Spatio-temporal Bayesian optimization for point-process data is now scalable and efficient, achieving accurate peak discovery with minimal evaluations.

Random features and projections unlock efficient quadratic entropy estimation for complex distributions and distances, revealing nuanced interdisciplinary research landscapes.

End-to-end diffusion models can achieve state-of-the-art precipitation nowcasting with over 17x faster inference by forecasting directly in radar space.

Iterative refinement in data space, not flow space, unlocks fast, high-fidelity generative sampling without manual timestep tuning.

Distillation inference can reliably detect unauthorized LLM distillation by comparing suspect model outputs against teacher reasoning traces via shadow models.

A reverse Hebbian-inspired pruning method achieves 99% sparsity on ResNet-18 with only a 2.4% accuracy drop, dynamically allocating sparsity without layer-specific targets.

Pruning LLMs for decoding speedup is significantly improved by calibrating sparsity with actual generation activations, not just static text, and optimizing SpMV kernels.

Quantizing AdamW's second moment in preconditioner space, not state space, slashes validation loss gaps by up to 70% in 4-bit training.

Diffusion models bypass Langevin's condition-number bottleneck by leveraging time-dependent score trajectories, achieving sampling error independent of target covariance ill-conditioning.

Parameter-efficient adaptation methods can match full fine-tuning gains for tabular foundation models, while selective aggregation offers a clear path to improved predictions over ensemble averaging.

Self-testing a KV cache with targeted Q&A pairs, rather than rereading the whole context, dramatically boosts memory retention, especially under severe budget constraints.

Softmax attention selectively exploits latent structure in Gaussian mixtures, solving nonlinear tasks beyond linear attention's reach while matching linear performance on tractable problems.

Posterior sampling recovers signals from noisy one-bit compressed sensing with near-optimal sample complexity, even with learned priors that are only approximately correct.

Speculative decoding can achieve up to 6.6x speedup by parallelizing token verification and recovering intra-block dependencies with a lightweight contextual corrector.

Key contribution not extracted.

Key contribution not extracted.

Latent recurrent models now match non-recurrent baselines at one loop, unlocking compute-for-accuracy scaling without the prior "one-loop tax."

AI agents can now navigate the intricate trade-offs of post-route power optimization, learning to balance speculative power recovery with costly repair actions under strict timing and physical constraints.

Standard two-level softmax sampling is demonstrably biased; corrected methods achieve near-exact softmax approximations with negligible overhead.

An 8x token reduction and 11.1x latency improvement in video generation is achieved by training autoencoders that are generation-aware, not just reconstruction-aware.

LLMs can now independently update factual knowledge without retraining, using conditional memory as an editable interface.

Amortized agentic policy discovery enables personalized test-time scaling to satisfy complex, multidimensional user requirements, outperforming single-objective optimization on unseen profiles.

Ollivier-Ricci curvature reveals and rectifies spectral representation errors in GNN-to-MLP distillation, unlocking significant accuracy gains for graph-free student models.

Self-distillation can degrade student performance if the teacher exploits privileged information unavailable to the student, but a novel KL-regularized objective ensures teacher guidance aligns with student capabilities.

Hybrid LLMs exhibit a "Seesaw Effect" in context extension, with linear attention models thriving on continual pretraining and sliding-window models excelling at extrapolation, a distinction rooted in their positional inductive biases.

2-bit KV cache quantization for Looped Transformers achieves 80% memory savings and 4x throughput gains with near-BF16 accuracy by quantizing residuals relative to the final loop's KV states.

Distilling teacher-relative logit shifts, not just final policies, unlocks substantial gains in multi-teacher on-policy distillation by preventing inherited base biases from dominating learned preferences.

Fresh, shallow network activations can correct diffusion model feature forecasts, unlocking significant speedups and improved fidelity without any parameter updates.

Probabilistic forecasting models can now achieve 65% CRPS reduction with drastically fewer samples by directly learning the distribution shift between successive predictions.

Embedding compression can be dramatically improved by learning the optimal coding geometry from data, not just optimizing within a fixed one.

A single objective directly optimizes speculative decoding efficiency, eliminating auxiliary hyperparameters and achieving state-of-the-art performance across multiple tasks.

Continual learning is achieved without any parameter updates by performing in-context Bayesian inference over a learned latent concept space.

LLM capabilities diverge under latent sequence compression; language modeling, decodability, and reasoning do not share a single compression threshold.

Unbiased dataset pruning can be framed as a linear program, enabling efficient, label-free subset selection that outperforms geometric methods at small budgets.

Reweighted normalizing flows enable exponential convergence to arbitrary Boltzmann distributions via a novel WFR JKO scheme, bypassing common structural assumptions.

LLM-generated rules, not full trees, unlock efficient and interpretable few-shot tabular classification.

Smoothness in treatment effects unlocks near-optimal policy learning in high-dimensional, low-signal digital experimentation.

Long-horizon reasoning models can now achieve $O(1)$ memory and $O(N)$ time complexity while surpassing larger Transformer baselines on complex tasks.

Black-box MIPS oracles can approximate attention with $O(1/\varepsilon^2)$ keys by augmenting queries and keys, bypassing $\sqrt{n}$ and $\log n$ dependencies.

Generative MARL teachers fail by interpolating coordination modes; a novel optimal transport approach forces them to commit to distinct strategies, boosting student performance.

A simple, training-free calibration wrapper boosts tensor-network LLM compression by 2-6x, closing up to 60% of the perplexity gap without altering the underlying decomposition solver.

On-policy distillation excels at teaching LLMs to *use* their existing knowledge for complex reasoning, but doesn't actually teach them *new facts*.

Reward modeling inference costs plummet by up to 7x by performing judgments entirely in a compressed latent space, bypassing token-by-token generation.

Interactive video generation can be sped up by 1.4x by intelligently caching transformer residuals based on detected control transitions, without sacrificing quality.

Storing knowledge directly in LLM parameters offers a flexible, efficient alternative to in-context learning, bypassing context length limitations and repeated encoding costs.

Broader tokenizer alignment in on-policy distillation can actively harm performance by introducing noisy supervision signals.

Latency in voice assistants plummets by 20% by speculatively executing tool calls *during* speech recognition, not after.

Optimal $O(\sqrt{d})$ regret in online inverse linear optimization is now achievable in polynomial time, shattering previous computational barriers.

Spatiotemporal time series forecasting now boasts valid confidence intervals even with imputed labels and temporal dependencies, overcoming a critical limitation of existing PPI.

Speculative decoding's efficiency gains come at a steep security price, with jailbreak success rates soaring far faster than utility drops.

Transformer uncertainty is now reliably calibrated with linear-time attention.

Edge CNN adaptation can now achieve near-zero activation memory overhead by decoupling backward computations from full-width layer inputs.

Precise long-range evidence retrieval is unlocked by coupling block-sparse attention with learned prefix states, overcoming the fixed local window limitations of prior hybrid architectures.

Heterogeneous load forecasting scales efficiently by learning a small bank of low-rank adaptations for a shared probabilistic model, outperforming independent models and monolithic Transformers.

A novel debiased estimator for sparse SVMs reveals statistically significant features beyond standard variable selection, even in high-dimensional, correlated settings.

A learned graph structure and structural energy term significantly improve selective forecasting reliability by explicitly penalizing violations of inter-variable dependencies, outperforming whole-forecast scoring methods on key benchmarks.

Collaborative edge-cloud LLM inference can achieve 275% higher throughput by dynamically synchronizing heterogeneous requests to batch their common cloud-side suffix.

LLM pruning can be task-aware: DIPrune's dual importance scoring jointly optimizes static feature saliency and dynamic semantic evolution, outperforming prior methods on LLaVA and Qwen-VL.

Training-free LQEC recovers 84% of the FP16 accuracy gap at 2-bit quantization by decoupling calibration and compensation, outperforming baselines by over 20%.

Agent experience, even when uniformly unrewarded, can be distilled into predictive foresight to dramatically improve learning in sparse reward settings.

Removing a mean shift from data representations requires a signal-to-noise ratio that is $O(d)$ times higher than simply detecting it, a gap that standard linear methods cannot close.

FP4 RL for MoE LLMs can now match BF16 rollout quality with 5.4x speedup by directly aligning train-rollout quantization.

Train sparse GNNs with dynamic graph topologies across epochs without re-extracting structure, achieving state-of-the-art performance even at extreme edge budgets.

Accelerating VLA models without sacrificing task success is now certifiable, guaranteeing over $85\%$ of reference performance with $10\times$ speedups.

Predicting subskill transition timing unlocks reliable execution of 4x longer action chunks in VLA models, drastically cutting robotic idle time by 5x.

Geometry-guided retrieval unlocks persistent memory in linear-attention video world models, boosting long-horizon consistency by 0.74 dB PSNR and 28.5% rotation error reduction without retraining.

Achieve 25x speedup in World Action Models with <1% performance drop by enabling one-step action generation and asynchronous expert execution.

A novel initialization technique for diffusion samplers bypasses initial steps by directly sampling a softened clean-space posterior, achieving order-of-magnitude gains on rare-mode inverse problems.

A single-voice TTS model with 10x fewer parameters and 15x less compute than its teacher achieves near-identical audio quality, runnable on a single laptop.

Key contribution not extracted.

Private prediction can achieve logarithmic dependence on the time horizon, while private learning requires polynomial dependence, revealing a fundamental gap in their sample complexity for differentially private settings.

Language models can learn to directly sample high-probability answers, achieving significant reasoning gains without parameter updates or costly search.

A flexible hyperbolic optimization framework unlocks 46-74% loss reduction on hierarchical graphs by decoupling embedding and optimization curvatures.

Tabular ICL models can achieve near full-context performance with drastically reduced inference costs by aligning intermediate activations, even with minimal training data and no GPU.

LLM inference cost plummets as a novel output-aware KV cache retrieval method bypasses attention score approximation errors, achieving competitive performance with minimal overhead.

Freezing the input factor A in LoRA, especially on top singular directions, unlocks substantial performance gains over freezing the output factor B, even at rank 2.

Metaheuristic search for Fourier fine-tuning coordinates unlocks substantial parameter efficiency gains, outperforming LoRA and fixed sampling strategies.

Prompt injection defenses can degrade LLM agent utility by altering benign behavior; RAISED achieves robustness without this cost through self-distillation on generated scenarios.

Merging transformers sequentially along their execution path, rather than layer-by-layer, dramatically reduces error accumulation and unlocks principled multi-task capacity expansion.

Deterministic midpoint methods for Langevin sampling can achieve superior sample complexity in high dimensions compared to their randomized counterparts.

Individual token selection and metadata caching unlock sparse attention for diffusion transformers without sacrificing generation quality, achieving up to $2.32\times$ speedups.

Achieve 4-bit quantization for diffusion LLMs without calibration data, slashing memory by 2.6x and inference by 2.2x while preserving accuracy.

Folding multi-branch redundancy slashes speculative decoding costs in diffusion LLMs, unlocking up to 2x faster generation.