Inference & Quantization - Weekly Roundup

Standard two-level softmax sampling is demonstrably biased; corrected methods achieve near-exact softmax approximations with negligible overhead.

An 8x token reduction and 11.1x latency improvement in video generation is achieved by training autoencoders that are generation-aware, not just reconstruction-aware.

LLMs can now independently update factual knowledge without retraining, using conditional memory as an editable interface.

Amortized agentic policy discovery enables personalized test-time scaling to satisfy complex, multidimensional user requirements, outperforming single-objective optimization on unseen profiles.

Ollivier-Ricci curvature reveals and rectifies spectral representation errors in GNN-to-MLP distillation, unlocking significant accuracy gains for graph-free student models.

Standard two-level softmax sampling is demonstrably biased; corrected methods achieve near-exact softmax approximations with negligible overhead.

An 8x token reduction and 11.1x latency improvement in video generation is achieved by training autoencoders that are generation-aware, not just reconstruction-aware.

LLMs can now independently update factual knowledge without retraining, using conditional memory as an editable interface.

Amortized agentic policy discovery enables personalized test-time scaling to satisfy complex, multidimensional user requirements, outperforming single-objective optimization on unseen profiles.

Ollivier-Ricci curvature reveals and rectifies spectral representation errors in GNN-to-MLP distillation, unlocking significant accuracy gains for graph-free student models.

Self-distillation can degrade student performance if the teacher exploits privileged information unavailable to the student, but a novel KL-regularized objective ensures teacher guidance aligns with student capabilities.

Hybrid LLMs exhibit a "Seesaw Effect" in context extension, with linear attention models thriving on continual pretraining and sliding-window models excelling at extrapolation, a distinction rooted in their positional inductive biases.

2-bit KV cache quantization for Looped Transformers achieves 80% memory savings and 4x throughput gains with near-BF16 accuracy by quantizing residuals relative to the final loop's KV states.

Distilling teacher-relative logit shifts, not just final policies, unlocks substantial gains in multi-teacher on-policy distillation by preventing inherited base biases from dominating learned preferences.

Fresh, shallow network activations can correct diffusion model feature forecasts, unlocking significant speedups and improved fidelity without any parameter updates.

Probabilistic forecasting models can now achieve 65% CRPS reduction with drastically fewer samples by directly learning the distribution shift between successive predictions.

Embedding compression can be dramatically improved by learning the optimal coding geometry from data, not just optimizing within a fixed one.

A single objective directly optimizes speculative decoding efficiency, eliminating auxiliary hyperparameters and achieving state-of-the-art performance across multiple tasks.

Continual learning is achieved without any parameter updates by performing in-context Bayesian inference over a learned latent concept space.

LLM capabilities diverge under latent sequence compression; language modeling, decodability, and reasoning do not share a single compression threshold.

Unbiased dataset pruning can be framed as a linear program, enabling efficient, label-free subset selection that outperforms geometric methods at small budgets.

Reweighted normalizing flows enable exponential convergence to arbitrary Boltzmann distributions via a novel WFR JKO scheme, bypassing common structural assumptions.

LLM-generated rules, not full trees, unlock efficient and interpretable few-shot tabular classification.

Smoothness in treatment effects unlocks near-optimal policy learning in high-dimensional, low-signal digital experimentation.

Long-horizon reasoning models can now achieve $O(1)$ memory and $O(N)$ time complexity while surpassing larger Transformer baselines on complex tasks.

Black-box MIPS oracles can approximate attention with $O(1/\varepsilon^2)$ keys by augmenting queries and keys, bypassing $\sqrt{n}$ and $\log n$ dependencies.

Generative MARL teachers fail by interpolating coordination modes; a novel optimal transport approach forces them to commit to distinct strategies, boosting student performance.

A simple, training-free calibration wrapper boosts tensor-network LLM compression by 2-6x, closing up to 60% of the perplexity gap without altering the underlying decomposition solver.

On-policy distillation excels at teaching LLMs to *use* their existing knowledge for complex reasoning, but doesn't actually teach them *new facts*.

Reward modeling inference costs plummet by up to 7x by performing judgments entirely in a compressed latent space, bypassing token-by-token generation.

Interactive video generation can be sped up by 1.4x by intelligently caching transformer residuals based on detected control transitions, without sacrificing quality.

Storing knowledge directly in LLM parameters offers a flexible, efficient alternative to in-context learning, bypassing context length limitations and repeated encoding costs.

Broader tokenizer alignment in on-policy distillation can actively harm performance by introducing noisy supervision signals.

Latency in voice assistants plummets by 20% by speculatively executing tool calls *during* speech recognition, not after.

Optimal $O(\sqrt{d})$ regret in online inverse linear optimization is now achievable in polynomial time, shattering previous computational barriers.

Spatiotemporal time series forecasting now boasts valid confidence intervals even with imputed labels and temporal dependencies, overcoming a critical limitation of existing PPI.

Speculative decoding's efficiency gains come at a steep security price, with jailbreak success rates soaring far faster than utility drops.

Transformer uncertainty is now reliably calibrated with linear-time attention.

Edge CNN adaptation can now achieve near-zero activation memory overhead by decoupling backward computations from full-width layer inputs.

Precise long-range evidence retrieval is unlocked by coupling block-sparse attention with learned prefix states, overcoming the fixed local window limitations of prior hybrid architectures.

Heterogeneous load forecasting scales efficiently by learning a small bank of low-rank adaptations for a shared probabilistic model, outperforming independent models and monolithic Transformers.

A novel debiased estimator for sparse SVMs reveals statistically significant features beyond standard variable selection, even in high-dimensional, correlated settings.

A learned graph structure and structural energy term significantly improve selective forecasting reliability by explicitly penalizing violations of inter-variable dependencies, outperforming whole-forecast scoring methods on key benchmarks.

Collaborative edge-cloud LLM inference can achieve 275% higher throughput by dynamically synchronizing heterogeneous requests to batch their common cloud-side suffix.

LLM pruning can be task-aware: DIPrune's dual importance scoring jointly optimizes static feature saliency and dynamic semantic evolution, outperforming prior methods on LLaVA and Qwen-VL.

Training-free LQEC recovers 84% of the FP16 accuracy gap at 2-bit quantization by decoupling calibration and compensation, outperforming baselines by over 20%.

Agent experience, even when uniformly unrewarded, can be distilled into predictive foresight to dramatically improve learning in sparse reward settings.

Removing a mean shift from data representations requires a signal-to-noise ratio that is $O(d)$ times higher than simply detecting it, a gap that standard linear methods cannot close.

FP4 RL for MoE LLMs can now match BF16 rollout quality with 5.4x speedup by directly aligning train-rollout quantization.

Train sparse GNNs with dynamic graph topologies across epochs without re-extracting structure, achieving state-of-the-art performance even at extreme edge budgets.

Accelerating VLA models without sacrificing task success is now certifiable, guaranteeing over $85\%$ of reference performance with $10\times$ speedups.

Predicting subskill transition timing unlocks reliable execution of 4x longer action chunks in VLA models, drastically cutting robotic idle time by 5x.

Geometry-guided retrieval unlocks persistent memory in linear-attention video world models, boosting long-horizon consistency by 0.74 dB PSNR and 28.5% rotation error reduction without retraining.

Achieve 25x speedup in World Action Models with <1% performance drop by enabling one-step action generation and asynchronous expert execution.

A novel initialization technique for diffusion samplers bypasses initial steps by directly sampling a softened clean-space posterior, achieving order-of-magnitude gains on rare-mode inverse problems.

A single-voice TTS model with 10x fewer parameters and 15x less compute than its teacher achieves near-identical audio quality, runnable on a single laptop.

Key contribution not extracted.

Private prediction can achieve logarithmic dependence on the time horizon, while private learning requires polynomial dependence, revealing a fundamental gap in their sample complexity for differentially private settings.

Language models can learn to directly sample high-probability answers, achieving significant reasoning gains without parameter updates or costly search.

A flexible hyperbolic optimization framework unlocks 46-74% loss reduction on hierarchical graphs by decoupling embedding and optimization curvatures.

Tabular ICL models can achieve near full-context performance with drastically reduced inference costs by aligning intermediate activations, even with minimal training data and no GPU.

LLM inference cost plummets as a novel output-aware KV cache retrieval method bypasses attention score approximation errors, achieving competitive performance with minimal overhead.

Freezing the input factor A in LoRA, especially on top singular directions, unlocks substantial performance gains over freezing the output factor B, even at rank 2.

Metaheuristic search for Fourier fine-tuning coordinates unlocks substantial parameter efficiency gains, outperforming LoRA and fixed sampling strategies.

Prompt injection defenses can degrade LLM agent utility by altering benign behavior; RAISED achieves robustness without this cost through self-distillation on generated scenarios.

Merging transformers sequentially along their execution path, rather than layer-by-layer, dramatically reduces error accumulation and unlocks principled multi-task capacity expansion.

Deterministic midpoint methods for Langevin sampling can achieve superior sample complexity in high dimensions compared to their randomized counterparts.

Individual token selection and metadata caching unlock sparse attention for diffusion transformers without sacrificing generation quality, achieving up to $2.32\times$ speedups.

Achieve 4-bit quantization for diffusion LLMs without calibration data, slashing memory by 2.6x and inference by 2.2x while preserving accuracy.

Folding multi-branch redundancy slashes speculative decoding costs in diffusion LLMs, unlocking up to 2x faster generation.

A few critical denoising steps in diffusion language models dictate the entire output, and training only these "pivots" dramatically boosts performance.

Adapters forget as much as they learn: a simple post-training normalization unlocks significant gains in both specialization and retained capabilities for LoRA.

A single tokenizer can now slash token counts for non-English scripts by 20-40% without increasing English tokenization cost.

Imperfect diffusion models can now achieve exact target distribution sampling at inference time by correcting inherent path and endpoint errors.

Real-time 3D Gaussian avatar animation is now achievable at 60fps on mobile by distilling complex neural inference into a linear blendshape approximation, reducing CPU cost by 1000x.

Adversarial distillation achieves state-of-the-art few-step visual generation by learning log-density ratios directly via classification, eliminating the need for auxiliary score fitting.

Teacher averaging in MOPD, not just raw gradients, dictates student learning, with Adam's momentum and BF16 precision masking significant parameter shifts.

Non-convex sampling from matrix spaces is now possible with provable exponential convergence using momentum preconditioning that spectrally tames gradients.

A single controller can now robustly learn stochastic transport across a range of initial distribution shifts, outperforming standard SB and noise augmentation on challenging tasks.

LLMs can learn effective dynamic embeddings for massive combinatorial optimization problems with a shallow policy by selectively propagating memory across rollout steps, bypassing the need for expensive full recomputation or external labels.

Harnessing earlier recurrent states as implicit weak supervision, a training-free contrastive decoding method boosts Looped Transformer performance by up to 11.45% and enables 22.5%-48.2% FLOP reduction.

Sampling weight updates, not just tokens, unlocks significant test-time adaptation gains for LLMs.

Stochastic rounding excels in Transformer MLPs, while round-to-nearest is better at the output head, enabling a 28% perplexity reduction via mixed-precision emulation.

Fine-tuning 30B+ LLMs on a single 80GB GPU is now possible with TACO, slashing optimizer state memory by $174\times$ without sacrificing accuracy.

Synthetic scenes with automatically grounded object identities unlock substantial real-world perceptual gains in MLLMs via on-policy self-distillation.

Coding agents can achieve over 4x throughput gains with a novel retrieval-based speculative decoding framework that intelligently sources and formats reusable text and adapts draft lengths.

A 0.9B parameter omni-modal embedder achieves state-of-the-art text retrieval quality by distilling a frozen text backbone's embeddings into new modalities, bypassing the common modality-extension performance drop.

Forecast errors, not drift alarms, directly signal when to retrain time series models, achieving state-of-the-art performance with a plug-in module.

LLM inference can be transformed from stateless recomputation to stateful recall, slashing per-question compute by 90% and enabling perfect recall over massive corpora.

Moreau--Yosida unadjusted Langevin sampling achieves $\widetilde O(\varepsilon^{-1})$ iterations for $\varepsilon$-accurate convergence to target distributions, bypassing the need for third derivatives.

Arbitrary multilevel models can now be amortized with neural networks that precisely preserve their full conditional independence structure, outperforming samplers on large-scale problems.

Sparse POT problems can now be solved efficiently with structured regularizers, outperforming entropic methods on cost, sparsity, and speed.

Inductive Gromov-Wasserstein embeddings are now achievable, enabling out-of-sample generalization without recomputing costly optimal transport plans.

Most failed multi-turn agent trajectories contain a single, recoverable pivotal mistake that, if addressed, can restore task success, a phenomenon PivotOPD exploits for significant performance gains.

LLM memory evolution reveals a shift from isolated attention to coordinated, multidimensional memory routing across depth and substrate, challenging the focus on single operators.

Multimodal models can achieve significant image generation improvements by learning from their own critiques during test-time compute, effectively acting as their own recursive self-improvement mechanism.

A novel decoupled autoencoder design unlocks 32x spatial compression for diffusion models while simultaneously boosting reconstruction fidelity and accelerating training convergence.

Diffusion language models can now generate high-quality text with up to 49% fewer network evaluations by matching output distributions.

Lossy compression's inherent distortion patterns, not just update geometry, can reliably detect and mitigate model poisoning in federated learning.

Consumer GPUs can now achieve up to 1.66x faster MoE inference by bypassing NCCL's CPU bottlenecks with a novel DMA-driven communication design.

End-to-end learned subspace projections shatter prior compression limits, achieving 70% weight reduction in Llama-2-7B with near-dense performance and drastically shrinking KV cache memory.

Zero-shot voice conversion now achieves $9\times$ faster inference with improved perceptual quality by jointly optimizing flow and content encoding.

LLM inference auctions unlock economic efficiency and user utility by enabling dynamic bidding for compute priority, outperforming static pricing tiers.

Exact incremental BPE achieves $O(\log t)$ amortized append time by weighting centroid search intervals, a significant speedup over prior $O(\log^2 t)$ methods.

A theoretical framework and diagnostic tool reveal that RoPE's long-context failures stem from a quantifiable semantic-positional tradeoff, with reasoning tasks suffering semantic reversal and retrieval tasks positional insensitivity.