Inference & Quantization - Weekly Roundup

Edge service orchestration can achieve up to 64.5% lower decision latency than LLMs by using Jev decision models, even at high loads where LLMs fail.

This work introduces physical rank consistency (PRC) to measure how well tokenization preserves local physical distance rankings after reconstruction, and presents ActionPiece, which preserves physical action relationships through joint supervision of representation learning and quantization.

AMPLE-Math, a reusable suite of 5,319 mathematical problems with six reasoning views that share the same answer, is constructed and AMPLE-Math, a reusable suite of 5,319 mathematical problems with six reasoning views that share the same answer, is compared, suggesting that OPSD can improve access to existing reasoning capabilities through parameters shared by direct-response and thinking-enabled inference.

This work instantiates VDN on MiniMax H3, which combines local Softmax attention with bidirectional linear memory for long-range video context and introduces Video Delta Attention (VDA), which updates memory once per frame by jointly incorporating its spatial tokens.

Edge service orchestration can achieve up to 64.5% lower decision latency than LLMs by using Jev decision models, even at high loads where LLMs fail.

AMPLE-Math, a reusable suite of 5,319 mathematical problems with six reasoning views that share the same answer, is constructed and AMPLE-Math, a reusable suite of 5,319 mathematical problems with six reasoning views that share the same answer, is compared, suggesting that OPSD can improve access to existing reasoning capabilities through parameters shared by direct-response and thinking-enabled inference.

This work instantiates VDN on MiniMax H3, which combines local Softmax attention with bidirectional linear memory for long-range video context and introduces Video Delta Attention (VDA), which updates memory once per frame by jointly incorporating its spatial tokens.

Using realistic communication models reversed simulated traffic speed trends and more than doubled predicted hard-braking events, so researchers and transportation agencies can maintain existing simulation pipelines while accurately accounting for dense-traffic packet loss and denial-of-service impacts.

It is shown that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step, and an additive score centering term is derived that stabilizes RL under TIM by canceling drift.

The results support the conclusion: foundation-model supervision can amplify a fixed manual annotation budget into a substantially larger training set and yield a compact, deployable edge model.

This work proposes QUALS, a large-scale time series corpus equilibrium framework that significantly enhances data efficiency, enabling existing models to achieve superior performance using only a small fraction of the original training data.

The Depth Control Protocol (DCP) is proposed, a diagnostic suite that disentangles three quantities of depth: truncating depth at inference time, plotting quality against retained depth fraction, and reading off the slope.

DART, a training-free method that combines low-rank coordinate transport with target-schedule response calibration with forward evaluations and no source training videos is proposed, and Adapter-level results reveal positive functional effects for some adapters and strong attenuation with reduced negative functional effects for others.

SwitchSD is introduced, an adaptive framework that treats copying as a latent control signal of the LLM that allows the system to dynamically switch between neural drafting and context-based copying, effectively turning copying from a noisy heuristic into a principled, model-aware decoding regime.

D-Quant is proposed, a flexible KV cache quantization framework that introduces a Drift mechanism to convert entropy-coded representations of each token into fixed-size bitstreams, enabling regular memory access and parallel dequantization within attention kernels.

Zarya is introduced, a family of hybrid language models that jointly optimizes an autoregressive (AR) objective and a masked-diffusion objective within a single architecture, demonstrating performance on standard benchmarks while offering a principled integration of autoregressive and diffusion paradigms.

GS-PI is presented, a novel optimization-decoupled framework that casts PBR material generation as a geometry-conditioned diffusion process on 3D point clouds and introduces a multi-scale cross-view conditioning mechanism that integrates three complementary components.

This work presents the first systematic characterization of attention sparsity in autoregressive image generation across diverse workloads and representative open-source models and reveals several distinguishing properties, including a pronounced prefill-decode asymmetry, strong attention concentration on prompt and local tokens, and a unique diagonal attention sparsity pattern arising from the spatial locality of visual tokens.

Recency Forcing is proposed, which applies a non-positive, timestep-dependent bias, termed Temporal Response Bias (TRB), on pre-softmax attention logits derived directly from $R, closing the train-inference gap without modifying context length or training objectives.

QCPruner fuses two cross-modal cues into utility and applies it to both visual targets and candidate representatives within visual-affinity-based coverage, which achieves the highest average relative performance among evaluated complete-system pruning methods at every reported token budget.

Vision-language-action models let robots follow language instructions, but their language backbones of several billion parameters are the main obstacle to running them on robot hardware, and removing 63% of it from OpenVLA-OFT drops LIBERO-Long success from 93.2% to 0.8%.

Evaluated end-to-end on multiple VLMs, the 4.5-bit MiX formulation exhibits equivalent or superior accuracy on multi-modal benchmarks compared to NVFP4, proving the inverted-scaling datapath is physically superior for efficient VLM deployment.

This work presents RebarSim, a visual sim-to-real system trained entirely in simulation, a privileged state-based teacher is trained with reinforcement learning over procedurally generated rebar geometries, then distilled into a multi-view student that maps raw RGB and proprioception directly to actions under extensive domain randomization.

MAGMA-GEN is an on-policy data-generation pipeline that converts ambiguous failed rollouts into validated recovery supervision and improves task success and recovery capabilities, against distillation and trajectory-repair baselines under evolving task constraints in both simulation and real-robot execution.

Teacher Prediction Refinement Distillation (TPRD), a plug-and-play module that refines teacher predictions before distillation by exploiting stage-wise prediction information, and Maximum Dark Knowledge Preservation (MDKP), which selectively refines target-class logits while retaining non-target relations.

These findings support long-context inference in which pretrained models guide their own access to the information they retain, by introducing On-Demand Attention (ODA), a local-first decoding method that uses a lightweight recall head to selectively invoke global attention as its predicted benefit changes during generation.

This work proposes a backbone-agnostic compaction pipeline for single-view feed-forward 3DGS that couples an importance-score-based pruning mechanism with a trainable, lightweight recurrent refinement module, which iteratively updates the surviving primitives to restore image quality.

CrossUAV is introduced, a UAV benchmark for joint object detection and instance segmentation that provides a unified evaluation platform for cross-granularity task collaboration and Cross-Granularity Socialized Collaboration is proposed, a progressive and adaptive framework that regulates when, where, and how tasks exchange information across network hierarchies.

This work shows that view-invariant control can be achieved without explicit test-time 3D sensing by encoding geometric knowledge into the visual representation during simulation, and presents AnyViewDex, an asymmetric training pipeline that combines multi-view contrastive alignment with privileged 3D geometric supervision.

An alignment-path distillation framework for streaming automatic speech recognition (ASR) with large language models (LLMs) that extracts monotonic alignment paths from a non-streaming teacher's soft text-audio attention and uses them to construct interleaved training sequences.

This work proposes RetireOPD (Self-Retiring On-Policy Distillation), which first optimizes a decoupled, skill-conditioned teacher with environment rewards and then trains a skill-free student jointly with RL and OPD.

This work introduces physical rank consistency (PRC) to measure how well tokenization preserves local physical distance rankings after reconstruction, and presents ActionPiece, which preserves physical action relationships through joint supervision of representation learning and quantization.

This work proposes Colla-Q, a bit-allocation framework to maintain balanced performance across experts through an activation-entropy-based bit-width allocation algorithm that encourages each expert to operate collaboratively in the quantized model, thereby improving the overall MoE performance and reducing the dependence on the calibration dataset.

This work proposes that tool calls report their progress explicitly while they run, and measures what that takes, and shows that at the points where a KV cache decision is made, the reported progress is between several times and an order of magnitude more accurate than the best published predictors, and it stays accurate when the environment changes.

Experimental results demonstrate the proposed $\mathcal{M}^2$Tok-based VLA achieves superior reconstruction fidelity but also significantly boosts the success rate of VLA models, and comprehensive ablation studies further confirm the effectiveness of the multi-head and multi-codebook mechanisms.

It is argued that local LLM systems need explicit guarantees for prompt lifetime, persistent storage, and tenant isolation, because the surrounding serving software handles prompt data before, during, and after inference.

The results support response geometry as a conditional principle for structured pruning, with its benefit determined jointly by the observed response and the architecture in which that response is realized.

WISE was evaluated through in-orbit execution aboard the ISS-mounted IMAGIN-e payload, indicating that WISE is a practical candidate for low-latency wildfire smoke monitoring from space under onboard resource constraints.

ForceDelta-VLA is presented, a correction-distillation framework that constructs an explicit force-correction target using paired predictions from a frozen teacher's force-conditioned and learned force-agnostic modes, and reduces mean peak contact force over successful trials by approximately 26% on both platforms.

An SSD-native local MoE inference system that addresses challenges with an SSD I/O pipeline optimized for expert delivery, a native three-tier storage hierarchy that delivers and retains experts dynamically, and balanced CPU--GPU hybrid execution.

HBF memory system HBFlex, a full-HBF memory system with coordinated optimizations for KV reads, writes, and reclamation, is presented, benefiting from higher HBF bandwidth and more efficient management of dynamic KV-cache reads, writes, and erases.

Edge0 is presented, a streaming MoE inference engine that closes the gap with a prerouter: a per-layer head predicts the next layer's routing one token ahead, and the prediction is consumed as the routing itself, so the staged expert set equals the routed set and nothing is dropped.

Two new tokenisation algorithms are introduced by introducing two new tokenisation algorithms that complete this 2x2 design space: BottomUpLL, a bottom-up likelihood-based tokeniser, and TopDownComp, a top-down compression-based tokeniser.

This work proposes Randomized Quantization for Text (QuanText), a training-free and large-language-model-agnostic data release mechanism that protects global secrets in textual datasets while preserving data utility.

This work proposes MoQSplat, which maps 3DGS content onto the Media over QUIC (MoQ) transport hierarchy and evaluates the core components on a prototype implementation, showing that opacity-based pruning outperforms scale-based pruning for progressive delivery.

This work proposes ASPIRE, a non-synchronized batched self-speculative decoding framework built on three components, which achieves speedup in decoding throughput over autoregressive baselines and improves average speedup by approximately $27\% over the strongest prior self-speculative baselines.

Continuous autoregressive models operate directly on continuous representations and Bottlenecked Representation Alignment and Modality--Time RoPE are introduced to strengthen musical semantic structure in Composer hidden states and temporal alignments across modalities.

A multi-teacher knowledge distillation framework to train a lightweight, streaming content encoder that generalizes across healthy (HE) and EL speech that reduces the EL word error rate to 21.2%, compared to 39.3% for the strongest zero-shot SSL baseline.

This paper compares the most widely used HCLs using the same fixed design, the OCP MXFP4 block dot product, a quantization primitive at the heart of edge Physical-AI inference, implemented as a single 12-stage, II=1 pipeline.

MeshKV, a KV cache fabric that moves blocks as packetized flows over a lightweight NoC, co-designs TaKV affine striping to spread homes and cut hotspot load and co-designs Mare multicast with verified duplicate suppression, and Pad, which converts bisection back-pressure into useful KV transfer.

Experiments on synthetic scenes with tetrahedral and 12-channel Benchmark arrays show that teacher guidance consistently improves downstream SELD performance and substantially reduces localization error, indicating that the distilled representation is optimized primarily for task-relevant spatial information rather than strict FOA reconstruction.

Latent Chip Adaptation from Probes (LCAP), a population-informed framework that decomposes hardware adaptation into a transferable population correction and probe-inferred latent personalization, is proposed.

The proposed LePoKet (Learnable Parameter Optimization for Knowledge Transfer), a structural transfer framework that embeds knowledge inheritance directly into the forward computation, demonstrates that learnable structural transfer generalizes across recognition and motion perception tasks and provides a promising approach for efficient robotic vision.

G GrowMTP serves existing RL training frameworks as a modular component, particularly offering a from-scratch acceleration path for models without pretrained draft heads, particularly offering a from-scratch acceleration path for models without pretrained draft heads.

JustFit, an MLX-based inference runtime that combines KVExec for compressed KV execution, PhaseSwap for component residency, and StateTrans for state-preserving serving transitions is presented, showing how compact state and lifetime-aware execution expand local serving capacity while supporting extended generated reasoning.

This work introduces ReDraft (Reference-Driven Revision and Fine-Tuning), which obtains both from the model's own failures: using an expert response only as a reference, it has the model revise its own incorrect rollout, keeps the revision only if a verifier accepts it, and fine-tunes on what survives.

A hypergraph relational distillation loss is introduced that imposes complementary intra- and cross-modal constraints, enabling the event encoder to inherit image-derived semantic organization while preserving local relational consistency and event-specific characteristics.

By jointly optimizing distillation and quantization within a unified teacher-anchored framework, this method ensures consistent training under quantization, suppressing hard negatives and enlarging decision margins, and designs a lightweight cross-attention adapter that enhances non-RGB representations through RGB-guided semantic transfer, narrowing the modality gap.

The Channel-wise Token-balanced Output-Aware Clipping (CTOAC) method, which learns per-input-channel clipping bounds by minimizing a token-balanced reconstruction loss on the corresponding linear outputs, is proposed.

A reward-compatible bounded mixing mechanism for $\gamma\mathrm{OPD}$ that balances verifiable outcome feedback with the discounted OPD advantage to move beyond purely teacher-dependent optimization is developed.

FairCompressAgent (FCA) is proposed, an agentic framework that integrates fairness-aware pruning, incremental quantization, and sparse low-rank factorization through a common operator interface and demonstrates how measured feedback and explicit constraints support the selection and interactive refinement of fairness-aware compression configurations.

LACE (Layer-Adaptive Codec Encoding), a dynamic frame rate codec that applies an independent compression step at each quantization layer, enabling layer-specific segmentation boundaries, is proposed.

Fathom is presented, a key scan in which each query decides how many bits of each key channel to read, and on real coding-agent sessions Fathom reaches the step agreement of the most accurate 136-bit scan at 92 bits.

VC-Attention is proposed, a training-free low-bit attention framework that addresses diffusion Transformers and quantization scale by pairing Value smoothing with a fused probability Cast, and improves fidelity over low-bit baselines.

The Mind2Dialogue framework proposes a psychology-guided simulator that preserves personal characteristics while updating mental states through interaction to generate coherent conversations, and trains models on the Oracle's well-informed responses to assist users without direct access to their mental states at deployment.

DR-SL (Dehydrate-Rehydrate with Self-Learning loop), which formalizes de-identification completeness as two measurable conditions: de-identification sufficiency under Pufferfish semantics, and task-information preservation via QA probes, is proposed.

A global sensitivity analysis of key LLM functions, which bounds the required scale of differentially private noise and derives an upper bound on floating-point error from masking and noise cancellation in the TEE as a function of the privacy parameter epsilon.

The results show that the practical losslessness of Orthrus depends on numerical precision and that exact trajectory equivalence should be evaluated separately from downstream task performance.

RAPID is proposed, a self-referenced latent maximization framework that removes external dependencies and encourages the same model update to induce consistently disruptive effects across training samples, thereby improving reachability and establishing a paradigm for real-time protection against unauthorized distillation in deployed T2I systems.

QPriv-VL is proposed, a question-guided, privacy-aware token-pruning framework for FL, SL, and USL that prunes visual tokens before transmission based on task utility and privacy sensitivity and evaluates it on GQA, OK-VQA, VQAv2, SLAKE, VQA-RAD, and PathVQA against four privacy attack families.

This work establishes a skill anti-distillation paradigm for protecting personal privacy and labor autonomy against unauthorized skill distillation, and introduces CounterPersona, which constructs targeted counter-persona evidence, packs compatible behavioral states into compact realization units, and strengthens them through rationale-guided consistency rewriting.

Results show that THEMIS makes otherwise implicit requirement-to-repair transitions inspectable, enabling systematic examination of how repair decisions persist, align, and evolve across stages.

These results demonstrate that traditional codec traces persist even after neural transcoding, supporting the feasibility and necessity of neural-codec-aware audio forensics and proposing a Transformer-based framework designed to leverage the hierarchical and temporal dependencies inherent in RVQ sequences.

This work proposes DIDO, which distills the converged dynamics of a multi-step video model into a single denoising step and achieves an average success rate of 99.0\% on LIBERO, 76.6\% on LIBERO-Plus, and 92.0\% on RoboTwin, while also demonstrating effective transfer to long-horizon and generalization tasks in real-world robotic manipulation.

This paper provides a formalization of what constitutes a sufficient scene graph for planning by modeling planning over scene graphs within an information-spaces framework through the definition of scene graph transition systems and relevant action semantics for navigation and manipulation.

The results show that efficient DeepSeek-V4-Flash inference on CDNA2 is limited not only by memory bandwidth, but also by FP4 execution-format mismatch, low-M utilization, and per-layer synchronization costs.

FINNAS is presented, a FINN-guided hardware-aware evolutionary neural architecture search framework that jointly searches quantised MLP depth, width, and global precision settings, and ranks candidates using proxy validation accuracy together with FINN-estimated LUT usage and latency under a fully parallel mapping.

This work predicts single-sequence model throughput from GGUF metadata using roofline-shaped predictors with quantization-specific scale factors fitted on reference models with fitted efficiencies not universal.

Joint-Output On-Policy Distillation (JO-OPD) is proposed, which distills the model's stronger S2T policy into joint generation using student-generated S2TS trajectories, which reduces OMG across two interleaved generation architectures.

A self-evolving memory paradigm for generative recommendation is proposed, aiming to enable effective evolution across heterogeneous behavioral patterns, and three key principles for effective self-evolving recommendation systems are identified, including isolated memorization, reinforced evolution, and scalable application.