Inference & Quantization - Weekly Roundup

Students trained with OPMD learn to mimic teacher reasoning processes, not just outputs, leading to demonstrably improved reasoning capabilities and altered solution strategies.

High-resolution 4K video upscaling no longer requires iterative diffusion: a single denoising step can outperform multi-step refiners while slashing latency by nearly 9×.

Sub-quadratic linear attention no longer requires full-precision state caching: 8-bit recurrent states can match FP32 accuracy while delivering up to 3.70× kernel speedups without any retraining.

Pushing KV cache quantization down to 2 bits without sacrificing downstream accuracy is now viable by rotating key and value matrices according to their joint second-order interaction statistics.

Treating every teacher token equally during on-policy distillation leaves massive gains on the table—dynamically reweighting supervision allows a smaller student model to outperform its own larger teacher on complex reasoning tasks.

Students trained with OPMD learn to mimic teacher reasoning processes, not just outputs, leading to demonstrably improved reasoning capabilities and altered solution strategies.

High-resolution 4K video upscaling no longer requires iterative diffusion: a single denoising step can outperform multi-step refiners while slashing latency by nearly 9×.

Sub-quadratic linear attention no longer requires full-precision state caching: 8-bit recurrent states can match FP32 accuracy while delivering up to 3.70× kernel speedups without any retraining.

Pushing KV cache quantization down to 2 bits without sacrificing downstream accuracy is now viable by rotating key and value matrices according to their joint second-order interaction statistics.

Treating every teacher token equally during on-policy distillation leaves massive gains on the table—dynamically reweighting supervision allows a smaller student model to outperform its own larger teacher on complex reasoning tasks.

Linear attention's recurrent state can be compressed down to 4 bits while outperforming uniform INT8 once precision is dynamically allocated to long-lived memory tracks and output-sensitive key rows.

Predicting dynamic expert activations two layers in advance eliminates transfer stalls, enabling massive MoE models to run up to 11.7× faster on a single commodity GPU.

Instead of struggling with fragile context tuning, treating divergent time series foundation model forecasts as complementary views in an identifiable latent space yields provably robust ensembles at test time without fine-tuning.

Diffusion language models can slash generation latency by up to 79% simply by treating latent semantic anchors as an intermittent cache rather than recomputing full network states at every denoising step.

Chain-of-thought token bloat can be slashed by 27% while actually boosting accuracy simply by recognizing that reasoning capacity lives almost entirely within the null space of a base model's dominant singular vectors.

Context expansion inflates an LLM's active token budget not to retrieve new facts, but simply to counteract softmax mass dilution caused by distractor token competition.

Allocating more precision bits to keys than values consistently beats symmetric KV caching across perplexity, prediction agreement, and generalization bounds, breaking through standard weight-only compression theory.

Native 8-bit LLM training silently destabilizes at billion-parameter scale due to softmax gradient drift, but restoring the zero-row-sum invariant completely eliminates the FP8 attention gap without architecture modifications.

Decoupling student rollout generation from teacher supervision kernels enables stable on-policy distillation for few-step flow models, outperforming multi-reward GRPO while accelerating convergence.

Masked generative models can break the linear evaluation barrier during parallel decoding without ever needing to fully reconstruct the underlying token dependency graph.

Decoupling candidate actions into reusable cacheable embeddings yields a 5x inference speedup, but collapses real-world rule adherence by more than 35 percentage points compared to joint cross-attention.

A novel inference-time sampling strategy unlocks frontier LLM reasoning capabilities in smaller models by resolving the exploration-exploitation dilemma.

Independent chunk scoring in autoregressive video generation unlocks sustained visual quality over extended horizons, overcoming the temporal consistency trade-offs of prior methods.

A simple triadic outer product expansion of linear attention's state dramatically boosts long-context modeling capacity without a proportional parameter increase.

Optimizing soft memory directly for user behavior, not text reconstruction, unlocks state-of-the-art long-context personalization.

INT4 KV caching for MLA models is now possible, achieving 3.59x compression at 128K context and enabling 5.168x higher throughput in cache-pressure serving workloads.

Latent communication between LLMs was largely an illusion of interface shortcutting until Draft-KV, which routes genuine drafting KV caches to let a frozen 0.5B model leap from 37% to 78% MMLU-Redux accuracy by tapping an 8B model's latent thoughts.

Treating on-policy distillation as value-based RL slashes rollout compute by 75% while actively preserving the generation diversity needed for high-k reasoning.

Distilling multi-billion-parameter multi-vector retrievers no longer requires caching terabytes of document tokens: entropic optimal transport allows a lightweight 149M text model to capture 95% of VLM teacher performance at 26x query speed using queries alone.

Multi-agent LoRA systems can slash time-to-first-token by 3.1× without sacrificing role specialization by decoupling shared base KV caches from precomputed low-rank adapter updates.

LLM agent token consumption swings by over an order of magnitude on identical tasks, but dynamically tracking context inflation across execution segments slashes budget waste by 21.3% without making a single extra LLM call.

Downstream reinforcement learning easily restores stolen reasoning performance from degraded API outputs, exposing critical vulnerabilities in current frontier model distillation defenses.

Decoupling structured semantics from frame-level actions and distilling autoregressive rollouts via shared, compressed memory tokens solves the long-standing trade-off between real-time responsiveness and multi-step drift in interactive world models.

NVIDIA's hand-crafted GEMM heuristics leave massive performance on the table: grounding learned kernel selectors in static hardware-behavior estimates cuts selection regret by up to 64.2% without running an autotuning benchmark.

Shrinking an LLM's residual stream based on activation variance alone discards the wrong directions—weighting pruned subspaces by downstream output sensitivity dramatically improves compressed model performance without sacrificing closed-form computational efficiency.

Sub-3-bit LLM inference finally sheds its dequantization bottleneck: decoding Leech lattices on-the-fly via a 16 KiB Golay trellis yields over 113 tokens per second while remaining within 2.5 MMLU points of 4-bit AWQ.

Aligning ternary weight quantization directly with SIMD ciphertext packing layouts eliminates up to 80% of expensive FHE multiplications, unlocking sub-1% accuracy loss speedups across both ViTs and BERT.

Chasing large token-level teacher-student gaps in agent distillation is largely misguided: local divergence fails to predict task completion, whereas unpenalized minor deviations often trigger catastrophic multi-turn failure.

Standard early-exit architectures collapse completely outside their pre-selected exits, but stochastic prefix supervision transforms a normal Transformer into a valid, deployable language model at literally every intermediate layer depth.

Analog LLM accelerators can claw back over 11 percentage points in zero-shot accuracy while calibrating 12x faster by analytically co-optimizing ADC output quantization alongside weight and activation clipping.

Constrained decoding no longer has to stop at syntax: embedding type and scope checks directly into Earley parser descent prunes dead-end code mid-stream to deliver up to a 15-point gain in program correctness.

Minimizing energy consumption in LLM inference does not ensure ecological sustainability: spatial and temporal deployment choices cause stark divergence across carbon, water, and biodiversity impacts that can double worst-case environmental regret.

Chaining low-rank KV corrections directly across sequential agent workflows eliminates redundant prefills, slashing peak GPU memory by 3.7x and cutting time-to-first-token in half without sacrificing model accuracy.

Context compaction in agentic RL can run up to 5x faster simply by streaming the KV cache instead of flushing it—accidentally turning standard LLMs into recurrent agents that preserve evicted context purely through RL.

Standard feedforward networks require just $d+1$ total hidden neurons to approximate $d$-dimensional continuous functions to arbitrary precision, settling the exact theoretical minimum while maintaining near-optimal bit complexity.

Reading barely 20% of the KV cache is enough to match up to 99% of dense attention accuracy at 32K context, yielding an immediate 1.69× speedup over FlashAttention without any retraining.

Distilling long documents into compressed KV caches silently cripples base model capabilities on unrelated prompts, but a lightweight query router or modest off-target data mixing completely eliminates this derailment.

Step-exact constrained decoding in masked diffusion models quietly tilts the global trajectory distribution away from valid path laws—a distortion that twisted Sequential Monte Carlo can completely eliminate without extra dynamic programming overhead.

Uniform token-level distillation often punishes valid student reasoning, but dynamically reallocating teacher supervision using verified outcome agreement unlocks immediate gains across math and code tasks without relying on hard trajectory filtering.

Distilling with verbal feedback usually transfers the teacher's arbitrary stylistic biases rather than its genuine corrections—unless you isolate that feedback as formal probabilistic evidence against next-token hypotheses.

Multi-teacher distillation fails not because of bad routing, but because high-variance instruction feedback quietly drowns out precision domains like mathematics during shared student updates.

Distilling causal LLMs into parallel diffusion models fundamentally handicaps students when the teacher ignores visible future tokens—aligning this context yields up to 4-point accuracy gains at 1.5× faster training.

Severe oversaturation and artifacts in few-step video distillation stem directly from compounding critic errors, which can be neutralized via a single-line geometric projection without discarding the true generative signal.

Diffusion teachers and online score networks are no longer bottlenecks for distilling autoregressive video models: directly minimizing sample MMD against reference videos improves VBench scores while unlocking 14B post-training on just eight H200 GPUs.

Pre-trained Softmax ViTs can directly bootstrap linear-attention models without expensive pre-training from scratch: simply copy the operator-agnostic MLP weights and distill the attention routing behavior.

Distillation's success hinges more on KL direction and learning rate than on-policy rollouts, challenging conventional wisdom.

Blackwell GPUs' FP4 Tensor Cores unlock 2.81x output throughput for long-context LLMs by strategically using mixed precision and rank-one smoothing, bypassing attention bottlenecks.

HeteroFold unlocks $10.7\times$ faster cross-family KV cache transfer for LLMs, eliminating receiver prefill and matching text-based communication performance.

Latent communication can be compressed by 75% without accuracy loss by conditioning KV cache transmissions on receiver information needs.

Experiments on multiple LLMs across different bit budgets show that CASA achieves lower perplexity than the latest scalar-proxy baselines, especially at ultra-low bit-widths ($<3$ bits per weight), confirming the distortion bound as a practical indicator of scalar-proxy failure.

This paper proposes a novel lightweight plastic-memory framework specifically designed for few-shot incremental learning on graphs, featuring an innovative evolving micro-clustering structure that dynamically updates representations of class prototypes, sub-prototypes, and their interaction weights.

STD is proposed, a hierarchical token pruning framework that adapts token selection mechanisms to the functional role of each network stage, and introduces a Stability-Adaptive Trigger in deep layers to execute pruning only during semantically stable phases.

RealFit is proposed, a framework that combines Unidirectional Information Flow (UIF) with Decoupled Timestep Modulation (DTM) that isolates the garment condition from stochastic noise to preserve garment identity, while DTM optimizes the modulation scale to maintain a strong conditional signal.

Embedding optical scattering priors directly into a lightweight neural pipeline yields a 10.8 dB PSNR gain over deep learning baselines under joint noise-scattering degradation, all within a 200 ms INT8 edge footprint.