Inference & Quantization - Weekly Roundup

Interactive video generation can be sped up by 1.4x by intelligently caching transformer residuals based on detected control transitions, without sacrificing quality.

Storing knowledge directly in LLM parameters offers a flexible, efficient alternative to in-context learning, bypassing context length limitations and repeated encoding costs.

Broader tokenizer alignment in on-policy distillation can actively harm performance by introducing noisy supervision signals.

Latency in voice assistants plummets by 20% by speculatively executing tool calls *during* speech recognition, not after.

Optimal $O(\sqrt{d})$ regret in online inverse linear optimization is now achievable in polynomial time, shattering previous computational barriers.

Interactive video generation can be sped up by 1.4x by intelligently caching transformer residuals based on detected control transitions, without sacrificing quality.

Storing knowledge directly in LLM parameters offers a flexible, efficient alternative to in-context learning, bypassing context length limitations and repeated encoding costs.

Broader tokenizer alignment in on-policy distillation can actively harm performance by introducing noisy supervision signals.

Latency in voice assistants plummets by 20% by speculatively executing tool calls *during* speech recognition, not after.

Optimal $O(\sqrt{d})$ regret in online inverse linear optimization is now achievable in polynomial time, shattering previous computational barriers.

Spatiotemporal time series forecasting now boasts valid confidence intervals even with imputed labels and temporal dependencies, overcoming a critical limitation of existing PPI.

Speculative decoding's efficiency gains come at a steep security price, with jailbreak success rates soaring far faster than utility drops.

Transformer uncertainty is now reliably calibrated with linear-time attention.

Edge CNN adaptation can now achieve near-zero activation memory overhead by decoupling backward computations from full-width layer inputs.

Precise long-range evidence retrieval is unlocked by coupling block-sparse attention with learned prefix states, overcoming the fixed local window limitations of prior hybrid architectures.

Heterogeneous load forecasting scales efficiently by learning a small bank of low-rank adaptations for a shared probabilistic model, outperforming independent models and monolithic Transformers.

A novel debiased estimator for sparse SVMs reveals statistically significant features beyond standard variable selection, even in high-dimensional, correlated settings.

A learned graph structure and structural energy term significantly improve selective forecasting reliability by explicitly penalizing violations of inter-variable dependencies, outperforming whole-forecast scoring methods on key benchmarks.

Collaborative edge-cloud LLM inference can achieve 275% higher throughput by dynamically synchronizing heterogeneous requests to batch their common cloud-side suffix.

LLM pruning can be task-aware: DIPrune's dual importance scoring jointly optimizes static feature saliency and dynamic semantic evolution, outperforming prior methods on LLaVA and Qwen-VL.

Training-free LQEC recovers 84% of the FP16 accuracy gap at 2-bit quantization by decoupling calibration and compensation, outperforming baselines by over 20%.

Agent experience, even when uniformly unrewarded, can be distilled into predictive foresight to dramatically improve learning in sparse reward settings.

Removing a mean shift from data representations requires a signal-to-noise ratio that is $O(d)$ times higher than simply detecting it, a gap that standard linear methods cannot close.

FP4 RL for MoE LLMs can now match BF16 rollout quality with 5.4x speedup by directly aligning train-rollout quantization.

Train sparse GNNs with dynamic graph topologies across epochs without re-extracting structure, achieving state-of-the-art performance even at extreme edge budgets.

Accelerating VLA models without sacrificing task success is now certifiable, guaranteeing over $85\%$ of reference performance with $10\times$ speedups.

Data-free diffusion distillation now achieves FID 2.04 at 1-NFE by unifying generation and score estimation in a single student network.

Predicting subskill transition timing unlocks reliable execution of 4x longer action chunks in VLA models, drastically cutting robotic idle time by 5x.

Geometry-guided retrieval unlocks persistent memory in linear-attention video world models, boosting long-horizon consistency by 0.74 dB PSNR and 28.5% rotation error reduction without retraining.

Achieve 25x speedup in World Action Models with <1% performance drop by enabling one-step action generation and asynchronous expert execution.

A novel initialization technique for diffusion samplers bypasses initial steps by directly sampling a softened clean-space posterior, achieving order-of-magnitude gains on rare-mode inverse problems.

A single-voice TTS model with 10x fewer parameters and 15x less compute than its teacher achieves near-identical audio quality, runnable on a single laptop.

Key contribution not extracted.

Private prediction can achieve logarithmic dependence on the time horizon, while private learning requires polynomial dependence, revealing a fundamental gap in their sample complexity for differentially private settings.

Language models can learn to directly sample high-probability answers, achieving significant reasoning gains without parameter updates or costly search.

A flexible hyperbolic optimization framework unlocks 46-74% loss reduction on hierarchical graphs by decoupling embedding and optimization curvatures.

Tabular ICL models can achieve near full-context performance with drastically reduced inference costs by aligning intermediate activations, even with minimal training data and no GPU.

LLM inference cost plummets as a novel output-aware KV cache retrieval method bypasses attention score approximation errors, achieving competitive performance with minimal overhead.

Freezing the input factor A in LoRA, especially on top singular directions, unlocks substantial performance gains over freezing the output factor B, even at rank 2.

Metaheuristic search for Fourier fine-tuning coordinates unlocks substantial parameter efficiency gains, outperforming LoRA and fixed sampling strategies.

Prompt injection defenses can degrade LLM agent utility by altering benign behavior; RAISED achieves robustness without this cost through self-distillation on generated scenarios.

Merging transformers sequentially along their execution path, rather than layer-by-layer, dramatically reduces error accumulation and unlocks principled multi-task capacity expansion.

Deterministic midpoint methods for Langevin sampling can achieve superior sample complexity in high dimensions compared to their randomized counterparts.

Individual token selection and metadata caching unlock sparse attention for diffusion transformers without sacrificing generation quality, achieving up to $2.32\times$ speedups.

Achieve 4-bit quantization for diffusion LLMs without calibration data, slashing memory by 2.6x and inference by 2.2x while preserving accuracy.

Folding multi-branch redundancy slashes speculative decoding costs in diffusion LLMs, unlocking up to 2x faster generation.

A few critical denoising steps in diffusion language models dictate the entire output, and training only these "pivots" dramatically boosts performance.

Adapters forget as much as they learn: a simple post-training normalization unlocks significant gains in both specialization and retained capabilities for LoRA.

A single tokenizer can now slash token counts for non-English scripts by 20-40% without increasing English tokenization cost.

Imperfect diffusion models can now achieve exact target distribution sampling at inference time by correcting inherent path and endpoint errors.

Real-time 3D Gaussian avatar animation is now achievable at 60fps on mobile by distilling complex neural inference into a linear blendshape approximation, reducing CPU cost by 1000x.

Adversarial distillation achieves state-of-the-art few-step visual generation by learning log-density ratios directly via classification, eliminating the need for auxiliary score fitting.

Teacher averaging in MOPD, not just raw gradients, dictates student learning, with Adam's momentum and BF16 precision masking significant parameter shifts.

Non-convex sampling from matrix spaces is now possible with provable exponential convergence using momentum preconditioning that spectrally tames gradients.

A single controller can now robustly learn stochastic transport across a range of initial distribution shifts, outperforming standard SB and noise augmentation on challenging tasks.

LLMs can learn effective dynamic embeddings for massive combinatorial optimization problems with a shallow policy by selectively propagating memory across rollout steps, bypassing the need for expensive full recomputation or external labels.

Harnessing earlier recurrent states as implicit weak supervision, a training-free contrastive decoding method boosts Looped Transformer performance by up to 11.45% and enables 22.5%-48.2% FLOP reduction.

Sampling weight updates, not just tokens, unlocks significant test-time adaptation gains for LLMs.

Stochastic rounding excels in Transformer MLPs, while round-to-nearest is better at the output head, enabling a 28% perplexity reduction via mixed-precision emulation.

Fine-tuning 30B+ LLMs on a single 80GB GPU is now possible with TACO, slashing optimizer state memory by $174\times$ without sacrificing accuracy.

Synthetic scenes with automatically grounded object identities unlock substantial real-world perceptual gains in MLLMs via on-policy self-distillation.

Coding agents can achieve over 4x throughput gains with a novel retrieval-based speculative decoding framework that intelligently sources and formats reusable text and adapts draft lengths.

A 0.9B parameter omni-modal embedder achieves state-of-the-art text retrieval quality by distilling a frozen text backbone's embeddings into new modalities, bypassing the common modality-extension performance drop.

Forecast errors, not drift alarms, directly signal when to retrain time series models, achieving state-of-the-art performance with a plug-in module.

LLM inference can be transformed from stateless recomputation to stateful recall, slashing per-question compute by 90% and enabling perfect recall over massive corpora.

Moreau--Yosida unadjusted Langevin sampling achieves $\widetilde O(\varepsilon^{-1})$ iterations for $\varepsilon$-accurate convergence to target distributions, bypassing the need for third derivatives.

Arbitrary multilevel models can now be amortized with neural networks that precisely preserve their full conditional independence structure, outperforming samplers on large-scale problems.

Sparse POT problems can now be solved efficiently with structured regularizers, outperforming entropic methods on cost, sparsity, and speed.

Inductive Gromov-Wasserstein embeddings are now achievable, enabling out-of-sample generalization without recomputing costly optimal transport plans.

Most failed multi-turn agent trajectories contain a single, recoverable pivotal mistake that, if addressed, can restore task success, a phenomenon PivotOPD exploits for significant performance gains.

LLM memory evolution reveals a shift from isolated attention to coordinated, multidimensional memory routing across depth and substrate, challenging the focus on single operators.

Multimodal models can achieve significant image generation improvements by learning from their own critiques during test-time compute, effectively acting as their own recursive self-improvement mechanism.

A novel decoupled autoencoder design unlocks 32x spatial compression for diffusion models while simultaneously boosting reconstruction fidelity and accelerating training convergence.

Diffusion language models can now generate high-quality text with up to 49% fewer network evaluations by matching output distributions.

Lossy compression's inherent distortion patterns, not just update geometry, can reliably detect and mitigate model poisoning in federated learning.

Consumer GPUs can now achieve up to 1.66x faster MoE inference by bypassing NCCL's CPU bottlenecks with a novel DMA-driven communication design.

End-to-end learned subspace projections shatter prior compression limits, achieving 70% weight reduction in Llama-2-7B with near-dense performance and drastically shrinking KV cache memory.

Zero-shot voice conversion now achieves $9\times$ faster inference with improved perceptual quality by jointly optimizing flow and content encoding.

LLM inference auctions unlock economic efficiency and user utility by enabling dynamic bidding for compute priority, outperforming static pricing tiers.

Exact incremental BPE achieves $O(\log t)$ amortized append time by weighting centroid search intervals, a significant speedup over prior $O(\log^2 t)$ methods.

A theoretical framework and diagnostic tool reveal that RoPE's long-context failures stem from a quantifiable semantic-positional tradeoff, with reasoning tasks suffering semantic reversal and retrieval tasks positional insensitivity.

Parallel diffusion language model generation can match autoregressive diversity and efficiency by strategically falling back to sequential sampling only at high-entropy decision points.

Distilling expert LLM behavior is now possible without reference solutions by using a fixed expert's next-token predictions on the student's *prefixes* as privileged context.

Quantized reasoning models overthink more, negating efficiency gains, but a new method precisely targets and penalizes these "overthinking" tokens without retraining.

A single backward pass can fundamentally alter training outcomes based on how it reuses low-precision forward-pass tensors, defying standard accuracy checks.

Low-precision matrix multiplication, even when repaired for accuracy, can break LLM prefix invariance and alter downstream task performance, necessitating a new certification approach.

Smaller, cheaper models generate superior negative examples for preference distillation, significantly boosting student performance without requiring larger teacher models.

Exploiting local parameter perturbations of a teacher model unlocks complementary supervision signals, boosting LLM mathematical reasoning accuracy beyond standard on-policy self-distillation.

Two shared scalar signals are sufficient to dynamically govern complex spectral memory, outperforming prior adaptive recurrences and accelerating training.

LLM agents can now adapt their action spaces dynamically from natural language, bypassing the need for pre-defined finite fields and accelerating inference through compositional policies.

A single inference engine now unifies 15 sparse attention methods, unlocking over 10x throughput gains for long-context LLMs through intelligent KV cache management.

Students trained with OPMD learn to mimic teacher reasoning processes, not just outputs, leading to demonstrably improved reasoning capabilities and altered solution strategies.

High-resolution 4K video upscaling no longer requires iterative diffusion: a single denoising step can outperform multi-step refiners while slashing latency by nearly 9×.

Sub-quadratic linear attention no longer requires full-precision state caching: 8-bit recurrent states can match FP32 accuracy while delivering up to 3.70× kernel speedups without any retraining.

Pushing KV cache quantization down to 2 bits without sacrificing downstream accuracy is now viable by rotating key and value matrices according to their joint second-order interaction statistics.

Treating every teacher token equally during on-policy distillation leaves massive gains on the table—dynamically reweighting supervision allows a smaller student model to outperform its own larger teacher on complex reasoning tasks.

Linear attention's recurrent state can be compressed down to 4 bits while outperforming uniform INT8 once precision is dynamically allocated to long-lived memory tracks and output-sensitive key rows.

Predicting dynamic expert activations two layers in advance eliminates transfer stalls, enabling massive MoE models to run up to 11.7× faster on a single commodity GPU.

Instead of struggling with fragile context tuning, treating divergent time series foundation model forecasts as complementary views in an identifiable latent space yields provably robust ensembles at test time without fine-tuning.

Diffusion language models can slash generation latency by up to 79% simply by treating latent semantic anchors as an intermittent cache rather than recomputing full network states at every denoising step.

Chain-of-thought token bloat can be slashed by 27% while actually boosting accuracy simply by recognizing that reasoning capacity lives almost entirely within the null space of a base model's dominant singular vectors.

Context expansion inflates an LLM's active token budget not to retrieve new facts, but simply to counteract softmax mass dilution caused by distractor token competition.

Allocating more precision bits to keys than values consistently beats symmetric KV caching across perplexity, prediction agreement, and generalization bounds, breaking through standard weight-only compression theory.

Native 8-bit LLM training silently destabilizes at billion-parameter scale due to softmax gradient drift, but restoring the zero-row-sum invariant completely eliminates the FP8 attention gap without architecture modifications.

Decoupling student rollout generation from teacher supervision kernels enables stable on-policy distillation for few-step flow models, outperforming multi-reward GRPO while accelerating convergence.