Inference & Quantization - Weekly Roundup

Forecast errors, not drift alarms, directly signal when to retrain time series models, achieving state-of-the-art performance with a plug-in module.

LLM inference can be transformed from stateless recomputation to stateful recall, slashing per-question compute by 90% and enabling perfect recall over massive corpora.

Moreau--Yosida unadjusted Langevin sampling achieves $\widetilde O(\varepsilon^{-1})$ iterations for $\varepsilon$-accurate convergence to target distributions, bypassing the need for third derivatives.

Arbitrary multilevel models can now be amortized with neural networks that precisely preserve their full conditional independence structure, outperforming samplers on large-scale problems.

Sparse POT problems can now be solved efficiently with structured regularizers, outperforming entropic methods on cost, sparsity, and speed.

Forecast errors, not drift alarms, directly signal when to retrain time series models, achieving state-of-the-art performance with a plug-in module.

LLM inference can be transformed from stateless recomputation to stateful recall, slashing per-question compute by 90% and enabling perfect recall over massive corpora.

Moreau--Yosida unadjusted Langevin sampling achieves $\widetilde O(\varepsilon^{-1})$ iterations for $\varepsilon$-accurate convergence to target distributions, bypassing the need for third derivatives.

Arbitrary multilevel models can now be amortized with neural networks that precisely preserve their full conditional independence structure, outperforming samplers on large-scale problems.

Sparse POT problems can now be solved efficiently with structured regularizers, outperforming entropic methods on cost, sparsity, and speed.

Inductive Gromov-Wasserstein embeddings are now achievable, enabling out-of-sample generalization without recomputing costly optimal transport plans.

Most failed multi-turn agent trajectories contain a single, recoverable pivotal mistake that, if addressed, can restore task success, a phenomenon PivotOPD exploits for significant performance gains.

LLM memory evolution reveals a shift from isolated attention to coordinated, multidimensional memory routing across depth and substrate, challenging the focus on single operators.

Multimodal models can achieve significant image generation improvements by learning from their own critiques during test-time compute, effectively acting as their own recursive self-improvement mechanism.

A novel decoupled autoencoder design unlocks 32x spatial compression for diffusion models while simultaneously boosting reconstruction fidelity and accelerating training convergence.

Diffusion language models can now generate high-quality text with up to 49% fewer network evaluations by matching output distributions.

Lossy compression's inherent distortion patterns, not just update geometry, can reliably detect and mitigate model poisoning in federated learning.

Consumer GPUs can now achieve up to 1.66x faster MoE inference by bypassing NCCL's CPU bottlenecks with a novel DMA-driven communication design.

End-to-end learned subspace projections shatter prior compression limits, achieving 70% weight reduction in Llama-2-7B with near-dense performance and drastically shrinking KV cache memory.

Zero-shot voice conversion now achieves $9\times$ faster inference with improved perceptual quality by jointly optimizing flow and content encoding.

LLM inference auctions unlock economic efficiency and user utility by enabling dynamic bidding for compute priority, outperforming static pricing tiers.

Exact incremental BPE achieves $O(\log t)$ amortized append time by weighting centroid search intervals, a significant speedup over prior $O(\log^2 t)$ methods.

A theoretical framework and diagnostic tool reveal that RoPE's long-context failures stem from a quantifiable semantic-positional tradeoff, with reasoning tasks suffering semantic reversal and retrieval tasks positional insensitivity.

Parallel diffusion language model generation can match autoregressive diversity and efficiency by strategically falling back to sequential sampling only at high-entropy decision points.

Distilling expert LLM behavior is now possible without reference solutions by using a fixed expert's next-token predictions on the student's *prefixes* as privileged context.

Quantized reasoning models overthink more, negating efficiency gains, but a new method precisely targets and penalizes these "overthinking" tokens without retraining.

A single backward pass can fundamentally alter training outcomes based on how it reuses low-precision forward-pass tensors, defying standard accuracy checks.

Low-precision matrix multiplication, even when repaired for accuracy, can break LLM prefix invariance and alter downstream task performance, necessitating a new certification approach.

Smaller, cheaper models generate superior negative examples for preference distillation, significantly boosting student performance without requiring larger teacher models.

Exploiting local parameter perturbations of a teacher model unlocks complementary supervision signals, boosting LLM mathematical reasoning accuracy beyond standard on-policy self-distillation.

Two shared scalar signals are sufficient to dynamically govern complex spectral memory, outperforming prior adaptive recurrences and accelerating training.

LLM agents can now adapt their action spaces dynamically from natural language, bypassing the need for pre-defined finite fields and accelerating inference through compositional policies.

A single inference engine now unifies 15 sparse attention methods, unlocking over 10x throughput gains for long-context LLMs through intelligent KV cache management.

High-resolution 4K video upscaling no longer requires iterative diffusion: a single denoising step can outperform multi-step refiners while slashing latency by nearly 9×.

Sub-quadratic linear attention no longer requires full-precision state caching: 8-bit recurrent states can match FP32 accuracy while delivering up to 3.70× kernel speedups without any retraining.

Pushing KV cache quantization down to 2 bits without sacrificing downstream accuracy is now viable by rotating key and value matrices according to their joint second-order interaction statistics.

Treating every teacher token equally during on-policy distillation leaves massive gains on the table—dynamically reweighting supervision allows a smaller student model to outperform its own larger teacher on complex reasoning tasks.

Linear attention's recurrent state can be compressed down to 4 bits while outperforming uniform INT8 once precision is dynamically allocated to long-lived memory tracks and output-sensitive key rows.

Predicting dynamic expert activations two layers in advance eliminates transfer stalls, enabling massive MoE models to run up to 11.7× faster on a single commodity GPU.

Instead of struggling with fragile context tuning, treating divergent time series foundation model forecasts as complementary views in an identifiable latent space yields provably robust ensembles at test time without fine-tuning.

Diffusion language models can slash generation latency by up to 79% simply by treating latent semantic anchors as an intermittent cache rather than recomputing full network states at every denoising step.

Chain-of-thought token bloat can be slashed by 27% while actually boosting accuracy simply by recognizing that reasoning capacity lives almost entirely within the null space of a base model's dominant singular vectors.

Context expansion inflates an LLM's active token budget not to retrieve new facts, but simply to counteract softmax mass dilution caused by distractor token competition.

Allocating more precision bits to keys than values consistently beats symmetric KV caching across perplexity, prediction agreement, and generalization bounds, breaking through standard weight-only compression theory.

Native 8-bit LLM training silently destabilizes at billion-parameter scale due to softmax gradient drift, but restoring the zero-row-sum invariant completely eliminates the FP8 attention gap without architecture modifications.

Decoupling student rollout generation from teacher supervision kernels enables stable on-policy distillation for few-step flow models, outperforming multi-reward GRPO while accelerating convergence.

Masked generative models can break the linear evaluation barrier during parallel decoding without ever needing to fully reconstruct the underlying token dependency graph.

Decoupling candidate actions into reusable cacheable embeddings yields a 5x inference speedup, but collapses real-world rule adherence by more than 35 percentage points compared to joint cross-attention.

A novel inference-time sampling strategy unlocks frontier LLM reasoning capabilities in smaller models by resolving the exploration-exploitation dilemma.

Independent chunk scoring in autoregressive video generation unlocks sustained visual quality over extended horizons, overcoming the temporal consistency trade-offs of prior methods.

A simple triadic outer product expansion of linear attention's state dramatically boosts long-context modeling capacity without a proportional parameter increase.

Optimizing soft memory directly for user behavior, not text reconstruction, unlocks state-of-the-art long-context personalization.

INT4 KV caching for MLA models is now possible, achieving 3.59x compression at 128K context and enabling 5.168x higher throughput in cache-pressure serving workloads.

Latent communication between LLMs was largely an illusion of interface shortcutting until Draft-KV, which routes genuine drafting KV caches to let a frozen 0.5B model leap from 37% to 78% MMLU-Redux accuracy by tapping an 8B model's latent thoughts.

Treating on-policy distillation as value-based RL slashes rollout compute by 75% while actively preserving the generation diversity needed for high-k reasoning.

Distilling multi-billion-parameter multi-vector retrievers no longer requires caching terabytes of document tokens: entropic optimal transport allows a lightweight 149M text model to capture 95% of VLM teacher performance at 26x query speed using queries alone.

Multi-agent LoRA systems can slash time-to-first-token by 3.1× without sacrificing role specialization by decoupling shared base KV caches from precomputed low-rank adapter updates.

LLM agent token consumption swings by over an order of magnitude on identical tasks, but dynamically tracking context inflation across execution segments slashes budget waste by 21.3% without making a single extra LLM call.

Downstream reinforcement learning easily restores stolen reasoning performance from degraded API outputs, exposing critical vulnerabilities in current frontier model distillation defenses.

Decoupling structured semantics from frame-level actions and distilling autoregressive rollouts via shared, compressed memory tokens solves the long-standing trade-off between real-time responsiveness and multi-step drift in interactive world models.

NVIDIA's hand-crafted GEMM heuristics leave massive performance on the table: grounding learned kernel selectors in static hardware-behavior estimates cuts selection regret by up to 64.2% without running an autotuning benchmark.

Shrinking an LLM's residual stream based on activation variance alone discards the wrong directions—weighting pruned subspaces by downstream output sensitivity dramatically improves compressed model performance without sacrificing closed-form computational efficiency.

Sub-3-bit LLM inference finally sheds its dequantization bottleneck: decoding Leech lattices on-the-fly via a 16 KiB Golay trellis yields over 113 tokens per second while remaining within 2.5 MMLU points of 4-bit AWQ.

Aligning ternary weight quantization directly with SIMD ciphertext packing layouts eliminates up to 80% of expensive FHE multiplications, unlocking sub-1% accuracy loss speedups across both ViTs and BERT.

Chasing large token-level teacher-student gaps in agent distillation is largely misguided: local divergence fails to predict task completion, whereas unpenalized minor deviations often trigger catastrophic multi-turn failure.

Standard early-exit architectures collapse completely outside their pre-selected exits, but stochastic prefix supervision transforms a normal Transformer into a valid, deployable language model at literally every intermediate layer depth.

Analog LLM accelerators can claw back over 11 percentage points in zero-shot accuracy while calibrating 12x faster by analytically co-optimizing ADC output quantization alongside weight and activation clipping.

Constrained decoding no longer has to stop at syntax: embedding type and scope checks directly into Earley parser descent prunes dead-end code mid-stream to deliver up to a 15-point gain in program correctness.

Minimizing energy consumption in LLM inference does not ensure ecological sustainability: spatial and temporal deployment choices cause stark divergence across carbon, water, and biodiversity impacts that can double worst-case environmental regret.

Chaining low-rank KV corrections directly across sequential agent workflows eliminates redundant prefills, slashing peak GPU memory by 3.7x and cutting time-to-first-token in half without sacrificing model accuracy.

Context compaction in agentic RL can run up to 5x faster simply by streaming the KV cache instead of flushing it—accidentally turning standard LLMs into recurrent agents that preserve evicted context purely through RL.

Standard feedforward networks require just $d+1$ total hidden neurons to approximate $d$-dimensional continuous functions to arbitrary precision, settling the exact theoretical minimum while maintaining near-optimal bit complexity.

Reading barely 20% of the KV cache is enough to match up to 99% of dense attention accuracy at 32K context, yielding an immediate 1.69× speedup over FlashAttention without any retraining.

Distilling long documents into compressed KV caches silently cripples base model capabilities on unrelated prompts, but a lightweight query router or modest off-target data mixing completely eliminates this derailment.

Step-exact constrained decoding in masked diffusion models quietly tilts the global trajectory distribution away from valid path laws—a distortion that twisted Sequential Monte Carlo can completely eliminate without extra dynamic programming overhead.

Uniform token-level distillation often punishes valid student reasoning, but dynamically reallocating teacher supervision using verified outcome agreement unlocks immediate gains across math and code tasks without relying on hard trajectory filtering.

Distilling with verbal feedback usually transfers the teacher's arbitrary stylistic biases rather than its genuine corrections—unless you isolate that feedback as formal probabilistic evidence against next-token hypotheses.

Multi-teacher distillation fails not because of bad routing, but because high-variance instruction feedback quietly drowns out precision domains like mathematics during shared student updates.

Distilling causal LLMs into parallel diffusion models fundamentally handicaps students when the teacher ignores visible future tokens—aligning this context yields up to 4-point accuracy gains at 1.5× faster training.

Severe oversaturation and artifacts in few-step video distillation stem directly from compounding critic errors, which can be neutralized via a single-line geometric projection without discarding the true generative signal.

Diffusion teachers and online score networks are no longer bottlenecks for distilling autoregressive video models: directly minimizing sample MMD against reference videos improves VBench scores while unlocking 14B post-training on just eight H200 GPUs.

Pre-trained Softmax ViTs can directly bootstrap linear-attention models without expensive pre-training from scratch: simply copy the operator-agnostic MLP weights and distill the attention routing behavior.

Distillation's success hinges more on KL direction and learning rate than on-policy rollouts, challenging conventional wisdom.

Blackwell GPUs' FP4 Tensor Cores unlock 2.81x output throughput for long-context LLMs by strategically using mixed precision and rank-one smoothing, bypassing attention bottlenecks.

HeteroFold unlocks $10.7\times$ faster cross-family KV cache transfer for LLMs, eliminating receiver prefill and matching text-based communication performance.

Latent communication can be compressed by 75% without accuracy loss by conditioning KV cache transmissions on receiver information needs.