Inference & Quantization - Weekly Roundup

DSpark shifts the performance landscape of LLM inference, achieving up to 85% faster generation speeds while maintaining high throughput.

Set diffusion achieves faster and more flexible decoding by allowing arbitrary token ordering, outperforming traditional diffusion models in key tasks.

NPUs can waste up to 40% of energy due to suboptimal configurations, but a new profiling tool reveals how to cut this waste significantly.

CGVQ achieves a remarkable 20% reduction in bits per pixel while maintaining visual quality, revolutionizing Gaussian-based image compression.

CAP achieves up to 86% higher throughput in MoE models while preserving accuracy, revolutionizing how we optimize expert placement and pruning.

NPUs can waste up to 40% of energy due to suboptimal configurations, but a new profiling tool reveals how to cut this waste significantly.

CGVQ achieves a remarkable 20% reduction in bits per pixel while maintaining visual quality, revolutionizing Gaussian-based image compression.

CAP achieves up to 86% higher throughput in MoE models while preserving accuracy, revolutionizing how we optimize expert placement and pruning.

RL-driven adaptive batching can yield a staggering 348% throughput improvement in multi-GPU settings, far surpassing traditional heuristics.

Weak models can supercharge strong models through efficient policy shift transfers, achieving significant performance gains without the usual rollout costs.

Privileged self-distillation can paradoxically hinder thinking models, leading to a 17% drop in accuracy on long reasoning tasks due to its impact on learning dynamics.

Quantized neurons can significantly enhance expressive capabilities over classical neurons, opening new avenues for quantum machine learning.

SNLP slashes the number of symbolic bootstraps in encrypted Transformer inference by over 2.5 times while maintaining performance, revealing a new avenue for efficient FHE applications.

TL-ANDI transforms how Tabular Foundation Models handle transfer learning by optimizing context selection to prevent negative transfer.

Elastic membership in LLM inference can boost throughput by up to 75% compared to static core allocations, all while maintaining bit-exact output.

A novel LRF gateway cuts operational misroutes by over 90% in LLM scheduling, preventing GPU crashes and optimizing resource allocation.

DSpark shifts the performance landscape of LLM inference, achieving up to 85% faster generation speeds while maintaining high throughput.

Learning to prune key-value caches based on future token utility allows KVpop to achieve up to 88% compression while retaining nearly full performance.

Achieving 98.7% accuracy with a deep binarized neural network, this spike sorter operates with unprecedented efficiency, making it ideal for low-power neural interfaces.

Simple Best-of-$N$ sampling can outperform complex guided search methods in text-to-image generation, revealing a critical flaw in how we evaluate efficiency in diffusion models.

HiFA4 recovers 37.5% of accuracy lost due to quantization in LLMs while slashing MMLU regressions by over half.

Reorganizing expert groups in MoE models can slash accuracy degradation by over 71% while boosting throughput by more than twofold.

Sangam slashes latency for diffusion language models by intelligently managing prefill and decode processes, revealing a new paradigm for efficient LLM serving.

MxGLUT achieves up to 2.16x latency speedup and reduces energy consumption significantly while maintaining competitive perplexity levels in LLM inference.

Physical isolation in chiplet design can boost LLM serving performance by nearly 50% while slashing latency by over 60%.

WattGPU slashes prediction errors by up to 4x for unseen LLM-GPU combinations, revolutionizing how we optimize energy consumption in AI inference.

Self-Gating Attention achieves linear complexity in time series forecasting while maintaining competitive accuracy, revolutionizing the efficiency of attention mechanisms in this domain.

EfficientNet-B0 outperforms newer models in efficiency, achieving competitive accuracy with 79% fewer parameters and 86% fewer GMACs.

Achieving faster inference without compromising accuracy, Lynx enables immediate decoding by prioritizing the most significant bits of the KV cache.

Selectively repeating only the most informative tokens can dramatically enhance reasoning in LLMs while slashing computational costs.

Entropy-Aware Dense Pruning not only filters out textual noise but also ensures a comprehensive visual representation, leading to superior performance in vision-language tasks.

Prefill-deflecting scheduling can cut Time-to-First-Token by up to 81%, revolutionizing disaggregated LLM serving efficiency.

Spec-AUF boosts the average emitted length of masked block drafters by 8.75%, demonstrating that targeted supervision can outperform traditional methods without complicating the inference process.

Distilling a 688M-parameter model into a lightweight version that retains 93% of its accuracy while reducing size by 7x could revolutionize 3D reconstruction in resource-constrained environments.

Set diffusion achieves faster and more flexible decoding by allowing arbitrary token ordering, outperforming traditional diffusion models in key tasks.

Token compression falters under severe constraints, while structural pruning offers a more stable solution for robust ViT segmentation.

Heterogeneous knowledge distillation can retain critical spatial information, leading to superior performance across diverse model architectures.

Achieving up to 82x reduction in communication volume without sacrificing spatial accuracy could revolutionize collaborative perception in autonomous vehicles.

Binarization methods that ignore weight significance can lead to substantial performance losses, but SAB-LVLM optimizes this process, achieving superior efficiency without sacrificing accuracy.

Achieving a 0.20% accuracy loss while eliminating BRAM usage could revolutionize the deployment of Vision Transformers on resource-constrained edge devices.

OmniPilot can predict serving costs with remarkable accuracy while dynamically adapting to hardware uncertainties, ensuring optimal resource utilization in GPU clusters.

InduceKV achieves superior performance in continual adaptation of multimodal LLMs while strictly adhering to a fixed memory budget, challenging the notion that larger memory footprints are necessary for effective learning.

HOLA achieves a remarkable 16.1% reduction in perplexity while retaining critical information that traditional linear attention models typically forget.

Achieving 100% task success in closed-loop execution, Embodied.cpp revolutionizes how embodied AI models are deployed across diverse hardware platforms.

Achieving 10x faster text-to-image diffusion without any training, MrFlow redefines the limits of inference acceleration.

OrbitQuant achieves state-of-the-art post-training quantization for diffusion transformers, enabling efficient image and video generation without the need for data-specific calibration.

GSRQ achieves a remarkable 22.20 percentage point accuracy gain on LongBench tasks, revolutionizing KV cache efficiency for LLMs at sub-1-bit quantization.

Calibration-data composition can dramatically enhance quantization performance, with 3.5-bit models outperforming traditional 4-bit baselines by over 20 points.

CAT enables Large Reasoning Models to intelligently balance efficiency and accuracy, compressing confident responses while thoroughly processing uncertain queries.

DiT-Pruning achieves unprecedented image quality retention in Diffusion Transformers, maintaining a CLIP score loss of only 0.001 at 50% sparsity.

Log$_\text{b}$Quant achieves superior quantization performance, enabling efficient deployment of language models on consumer hardware without sacrificing accuracy.

A novel framework achieves unprecedented dataset distillation speed and accuracy by directly minimizing information loss, setting a new benchmark in the field.

Memory overload in autoregressive video generation can be tackled by absorbing historical context into model weights, achieving up to 50% cache reduction with minimal quality loss.

Hypic slashes time-to-first-token by 2.45x and doubles throughput for hybrid-attention LLMs, all while preserving near-full accuracy.

Achieving up to 16x faster attention computation with only a 1.76% accuracy loss could revolutionize the deployment of long-context LLMs in real-time applications.

BaseRT achieves unprecedented inference throughput on Apple Silicon, outperforming existing runtimes and setting a new standard for on-device LLM performance.

Compact diffusion models can now leverage the power of high-capacity Teachers without architectural changes, achieving remarkable performance gains.

Optimal block size selection can lead to a 4.20x speedup in diffusion-based speculative decoding, revolutionizing inference efficiency.

FlexViT achieves up to 2.74x speedup for Vision Transformer inference on edge devices, revolutionizing the deployment of complex models in resource-constrained environments.

ALO estimators cut the runtime of conformal prediction while preserving accuracy, making uncertainty quantification feasible for larger datasets.

Adopting adversarial distillation can boost certified accuracy by over 5% while maintaining robust performance, reshaping the landscape of certified training.

RaBitQCache accelerates long-context LLM inference while cutting memory I/O by intelligently adapting token budgets based on attention sparsity.

Generative models can transform the landscape of object detection quantization, achieving state-of-the-art performance even in extreme low-bit scenarios.

Late visual-token updates can be safely ignored, leading to a 33.7% reduction in computational load without sacrificing performance.

Open-weight language models can outperform costly proprietary APIs, slashing expenses by 390x and latency by 3.8x in database integrations.

SeKV achieves a remarkable 5.9% performance boost in long-context LLM inference while slashing GPU memory usage by over half.

ERA's innovative approach to visual token pruning preserves attention integrity, enabling efficient MLLMs without sacrificing performance.

REDI achieves a remarkable 46.8% reduction in token sequence length while boosting accuracy, showcasing the power of combining class-specific corpus statistics with attention mechanisms.

Pruning VLA models can reduce parameters by up to 30% while retaining 90% of performance, challenging the notion of parameter redundancy in these complex systems.

Achieving up to 82% higher energy efficiency while maintaining accuracy, MINT redefines the boundaries of CNN inference on FPGA platforms.

Omni-Flow revolutionizes multimodal inference by integrating flexible orchestration, efficient data sharing, and KV cache reuse into a single cohesive framework.

LASER cuts average latency by up to 38% while only sacrificing 1% accuracy, revolutionizing how we deploy reasoning models on edge devices.

Transforming context ahead of time can slash time-to-first-token by nearly 12x, revolutionizing LLM agent efficiency.

Staged knowledge distillation allows quantum agents to learn complex visual tasks without the pitfalls of direct pixel-based training, achieving near-optimal performance with significantly smaller models.

RMMD not only accelerates model inference by 7.5x but also outperforms its teacher model on nearly all target weather variables, showcasing a breakthrough in distillation techniques.

PRR slashes decoding latency by up to 40% in long-context LLMs while maintaining accuracy, revolutionizing the efficiency of dynamic sparse attention.

Relaxed and tree-based acceptance criteria can dramatically increase the certified acceptance region in speculative decoding, challenging traditional views on distribution preservation.

Off-policy distillation fails in multi-task settings, but a two-phase approach combining it with on-policy refinement can achieve single-task expert performance across multiple tasks.

Balancing forward and reverse KL divergence in knowledge distillation can lead to significant improvements in text generation quality, outperforming traditional methods by up to 0.6 points.

Achieving a 46.2% reduction in computational cost for CNNs without retraining could revolutionize how we deploy deep learning on edge devices.

COSM achieves a remarkable 2.8x improvement in PIM throughput while keeping CPU performance degradation under 2.0%.

Only a subset of design interactions in heterogeneous LLM inference are binding constraints, revealing critical insights for optimizing deployment strategies.

Achieving a 20.58x reduction in search cost for optimal neural network configurations on FPGAs could revolutionize resource-efficient AI deployment at the edge.

DOPD reveals that intelligently routing supervision based on advantage gaps can significantly enhance capability transfer in distillation, outperforming conventional methods.

Transforming probabilistic programs into dynamic graphs can drastically cut down on computation time, enabling faster and more precise MCMC inference.

HMA-Serve achieves 3.2x higher goodput and 4.8x better cost-efficiency by leveraging memory-heterogeneous accelerators, challenging the status quo of single-vendor LLM serving.

Budget-adaptive routing can outperform strong models while cutting per-frame latency by nearly 30%—a game changer for edge-cloud inference efficiency.

CAEE slashes inference latency in MoE models by intelligently pruning low-value experts, achieving efficiency gains without sacrificing accuracy.

A compact 4B model using DuoMem achieves a staggering 77.9% task success rate, rivaling a 72B teacher model while being over 3x faster in execution.