Inference & Quantization - Weekly Roundup

System-aware optimizations in KV cache design could dramatically reduce the memory footprint and costs of serving large language models.

Switching between autoregressive and diffusion modes allows Nemotron-Labs-Diffusion to achieve unprecedented throughput and efficiency in language modeling.

Unbounded interaction horizons and real-time responsiveness redefine the possibilities for immersive AI-driven environments.

DSpark shifts the performance landscape of LLM inference, achieving up to 85% faster generation speeds while maintaining high throughput.

Targeting Super Weights in LLMs can lead to performance collapse, challenging assumptions about parameter importance and trainability.

Targeting Super Weights in LLMs can lead to performance collapse, challenging assumptions about parameter importance and trainability.

Achieving structured pruning that rivals unstructured methods in accuracy while significantly accelerating inference speed could redefine efficiency benchmarks for large language models.

Relaxed speculative decoding can significantly boost sampling speed, but it comes with hidden costs in capability evaluation and model quality.

Achieving extreme low-bit compression without lookup tables could revolutionize how we deploy large language models in memory-limited settings.

Achieving 205× latency reduction in FPGA-based neural networks could redefine the benchmarks for ultra-fast inference in latency-critical applications.

Compressing prompts into a single activation vector can cut computational costs while maintaining nearly full accuracy in LLM responses.

Quantized CFG models can achieve high fidelity without sacrificing sample quality, thanks to a new method that prevents branch drift.

Layer patching can dramatically enhance model performance in size interpolation, revealing that simple strategies often outperform complex methods.

System-aware optimizations in KV cache design could dramatically reduce the memory footprint and costs of serving large language models.

Memory compaction in LLMs is fundamentally flawed, with critical information often discarded before it's needed, revealing a systemic inefficiency across all layers.

MoE architectures outperform dense models in quantization resilience, revealing that structure trumps size in on-device VLM performance.

Quantization can induce significant behavioral changes in LLMs that traditional metrics fail to detect, revealing an illusion of equivalence between quantized and base models.

Distilling a reasoning model into a compact on-device version recovers significant summary quality while drastically reducing processing time from 39 seconds to just 0.8 seconds per article.

Mispredictions in power savings can occur, but they never lead to incorrect outputs, making Stochastic Activity Prediction a game-changer for tensor accelerator efficiency.

Quantization errors in speech enhancement models can be largely mitigated by robust spatial filtering, enabling efficient deployment on low-resource devices.

ZipDepth achieves real-time monocular depth estimation on resource-constrained devices while rivaling the accuracy of much larger models.

Small language models can achieve near state-of-the-art Text-to-SQL performance with just a fraction of the computational resources required by large models.

MAESTRO prunes MoE models more effectively by leveraging the interdependencies of expert routing, achieving superior performance retention and consistency across tasks.

Achieving up to 6.6x faster inference in LLMs, DominoTree redefines the limits of speculative decoding with path-dependent token drafting.

Eight-bit quantization allows a $1.2$-billion-parameter model to run on an $8$\,GB Raspberry Pi without sacrificing audio quality or speed.

OPSD-V enhances video generation by leveraging real video data for on-policy self-distillation, leading to superior visual quality and motion dynamics.

Delta-style networks outperform traditional methods by leveraging key-dependent projections, revealing a new path to efficient transformer inference.

Archiving quantized KV states can shrink memory usage by up to 54x without sacrificing retrieval speed or accuracy.

Achieving 97.6% of full performance with just 5.6% of the visual tokens, AnchorPrune redefines efficiency in multimodal inference.

Tailoring sparsity to layer importance can slash perplexity by over 1.9 points, challenging the one-size-fits-all approach in transformer pruning.

DeLS-Spec achieves faster inference and longer acceptance lengths by decoupling long and short context predictions, all while slashing training costs.

Transforming gradients into a near-isotropic space can cut LLM pretraining time by 7.6% while enhancing downstream task performance.

Achieving efficient multiple double arithmetic on NVIDIA tensor cores could significantly boost performance in high-precision applications.

The cost gap between new AI entrants and established incumbents is set to widen, with incumbents enjoying a 3-4x advantage by 2029-30.

Unbounded interaction horizons and real-time responsiveness redefine the possibilities for immersive AI-driven environments.

MoWorld achieves real-time interactive performance on low-cost hardware, revolutionizing the deployment of World Models in practical applications.

EeveeDark achieves unprecedented low-light video enhancement by fusing RAW data with event streams, outperforming prior methods while slashing computational costs.

Switching between autoregressive and diffusion modes allows Nemotron-Labs-Diffusion to achieve unprecedented throughput and efficiency in language modeling.

Estimation, not grid search, is key to optimizing LLM serving—this new approach reveals hidden performance potential in resource management.

Compressing the KV cache of speech tokens can enhance decoding speed by over 1.49 times while improving performance on key benchmarks.

Achieving an 83% reduction in vector count without sacrificing retrieval accuracy could revolutionize the deployment of late interaction models in real-world applications.

Weaver achieves a 4.37-fold speedup in autoregressive decoding while restoring crucial token dependencies, revolutionizing speculative decoding efficiency.

GeoSD counters the problematic drift in self-distillation, boosting out-of-distribution reasoning accuracy while preserving in-distribution gains.

FourTune slashes memory overhead by 2.25x while matching the performance of full-precision fine-tuning in diffusion models.

Achieving an 8.3x reduction in memory usage without sacrificing retrieval quality could redefine efficiency standards for long-context language models.

FreqDepthKV achieves a remarkable 3.9x effective compression ratio while preserving high accuracy in long-context LLM tasks, revolutionizing cache management.

A novel teacher-aligned repair stage enables significant parameter reduction in diffusion models while preserving image quality, achieving an FID of 3.12 with just one network evaluation.

NPUs can waste up to 40% of energy due to suboptimal configurations, but a new profiling tool reveals how to cut this waste significantly.

CGVQ achieves a remarkable 20% reduction in bits per pixel while maintaining visual quality, revolutionizing Gaussian-based image compression.

CAP achieves up to 86% higher throughput in MoE models while preserving accuracy, revolutionizing how we optimize expert placement and pruning.

RL-driven adaptive batching can yield a staggering 348% throughput improvement in multi-GPU settings, far surpassing traditional heuristics.

Weak models can supercharge strong models through efficient policy shift transfers, achieving significant performance gains without the usual rollout costs.

Privileged self-distillation can paradoxically hinder thinking models, leading to a 17% drop in accuracy on long reasoning tasks due to its impact on learning dynamics.

Quantized neurons can significantly enhance expressive capabilities over classical neurons, opening new avenues for quantum machine learning.

SNLP slashes the number of symbolic bootstraps in encrypted Transformer inference by over 2.5 times while maintaining performance, revealing a new avenue for efficient FHE applications.

TL-ANDI transforms how Tabular Foundation Models handle transfer learning by optimizing context selection to prevent negative transfer.

Elastic membership in LLM inference can boost throughput by up to 75% compared to static core allocations, all while maintaining bit-exact output.

A novel LRF gateway cuts operational misroutes by over 90% in LLM scheduling, preventing GPU crashes and optimizing resource allocation.

DSpark shifts the performance landscape of LLM inference, achieving up to 85% faster generation speeds while maintaining high throughput.

Learning to prune key-value caches based on future token utility allows KVpop to achieve up to 88% compression while retaining nearly full performance.

Achieving 98.7% accuracy with a deep binarized neural network, this spike sorter operates with unprecedented efficiency, making it ideal for low-power neural interfaces.