Multimodal Models - Weekly Roundup

This survey systematizes the EML landscape by introducing the first structured, model-to-system taxonomy and establishes a structured framework for multimodal systems that are not only high-performing and generalizable but natively efficient and ready for ubiquitous deployment.

The DeepSeek-V4.1-Flash model, a multimodal Mixture-of-Experts model with 552B backbone parameters and support for contexts of up to one million tokens, is introduced, substantially improving cost efficiency for agentic workloads and pushing the limits of KV cache compression.

RoboDawn is investigated, a human-intuitive interface that exposes robotic control to an agentic VLM through a compact set of discrete translation, rotation, and gripper commands, and an in-context learning (ICL) scheme is introduced that ground the VLM in both interface usage and task-solving strategies.

RoboDawn is investigated, a human-intuitive interface that exposes robotic control to an agentic VLM through a compact set of discrete translation, rotation, and gripper commands, and an in-context learning (ICL) scheme is introduced that ground the VLM in both interface usage and task-solving strategies.

This work finds that the wording of the question affects VLMs in two opposite ways: verbose paraphrasing reduces drift variance by 70--81% on the 8B models and the practical recipe---pad the prompt---further yields measurable gains in accuracy, even under image corruption.

This paper introduces multi-horizon multimodal prediction in ABBR, an agile tactile World Action Model for contact-rich robot control, which provides supervision for visual latent at a larger temporal offset while predicting the tactile latent in the next frame to capture fine-grained contact dynamics.

This work demonstrates how rigorous analytical frameworks can address complex humanitarian operations challenges characterised by information asymmetry and limited ground-truth data.

Unified Latent Diffusion Reasoner (Unified Latent Diffusion Reasoner), a framework that brings teacher reasoning steps from different modalities into shared thought tokens, trained to preserve the information needed for later reasoning steps and the final answer or action.

ERCPMP-Gx is presented, an endoscopic, histopathological, and genomic dataset developed to support the application of artificial intelligence (AI) in the recognition, characterization, and classification of colorectal polyposis.

Inspired by recent findings that self-attention acts as a low-pass filter and that its low-frequency components are informative, AttnPrint for white-box provenance is developed, which achieves strong derivative-model detection performance while remaining robust to five downstream modification techniques.

DeliveryGym is introduced, a 3D environment for evaluating and training agents on continuous courier shifts that couples multimodal tool interaction with persistent world dynamics and computes trajectory rewards from simulator events, making the costs of an agent's decisions available for reinforcement learning (RL).

It is shown that the workspace token can be used as a drop-in replacement for observations during deployment, enabling policies to solve memory-intensive tasks without the need for VLM reasoning in-the-loop.

A Retrieval-Augmented Generation framework for scientific image quality assessment, designed to simultaneously address both the understanding track (SIQA-U) and the scoring track (SIQA-S) of the SIQA challenge, is proposed.

AgriGround is introduced, a large-scale pixel-grounded agricultural multimodal instruction-tuning dataset containing over 500K images and 11M instruction-following samples spanning plant disease analysis, crop and weed identification, insect pest recognition, and fine-grained botanical understanding.

The reformulate city-scale 3D grounding as structured constraint reasoning, where description semantics are organized into computable cross-modal constraints over open-vocabulary 3D entities, attributes, and spatial relations, and introduces CitySTAR-3D, an enhanced benchmark that improves semantic coverage, instance completeness, bounding-box fidelity, and spatial-relation complexity in city-scale 3D grounding.

This work presents the first systematic characterization of attention sparsity in autoregressive image generation across diverse workloads and representative open-source models and reveals several distinguishing properties, including a pronounced prefill-decode asymmetry, strong attention concentration on prompt and local tokens, and a unique diagonal attention sparsity pattern arising from the spatial locality of visual tokens.

Recency Forcing is proposed, which applies a non-positive, timestep-dependent bias, termed Temporal Response Bias (TRB), on pre-softmax attention logits derived directly from $R, closing the train-inference gap without modifying context length or training objectives.

DocAttriBench (DAB) is introduced, a large-scale benchmark for fine-grained, element-level source attribution in Document VQA, grounding answers to specific layout elements such as text blocks, tables, and images.

QCPruner fuses two cross-modal cues into utility and applies it to both visual targets and candidate representatives within visual-affinity-based coverage, which achieves the highest average relative performance among evaluated complete-system pruning methods at every reported token budget.

This work adapts PP-OCRv6, a recent compact text recognizer without strong language modeling, for historical line recognition and compares it with a conventional CRNN across generalized pretraining, domain-specific training, corpus-level fine-tuning, and manuscript-specific few-shot adaptation on multilingual Latin- and Arabic-script material.

Experimental results show that SOLAR consistently outperforms state-of-the-art vision-only, vision-language, and task-specific models across all tasks, demonstrating superior accuracy, robustness, and multimodal reasoning.

Evaluated end-to-end on multiple VLMs, the 4.5-bit MiX formulation exhibits equivalent or superior accuracy on multi-modal benchmarks compared to NVFP4, proving the inverted-scaling datapath is physically superior for efficient VLM deployment.

This work introduces a spatial-semantic uncertainty formulation that maintains separate beliefs over each component and integrates probabilistic VLM evidence into a global target-identity posterior, including probability mass for undiscovered targets.

BinoGen is presented, an automated framework for generating large-scale, embodiment-aware egocentric binocular visual experiences in indoor environments and incorporating BinoGen data consistently improves real-world visual perception, including depth estimation, object detection, and video object tracking.

The segmentation ceiling gives a concrete design criterion for when mask-guided EF estimation is worthwhile, plus a simple, uncertainty-aware recipe for EF regression, and a heteroscedastic beta-NLL formulation yields informative, well-calibrated per-prediction uncertainty.

Experimental results and analyses show that IMFD improves multi-face forgery detection by integrating face bounding boxes into the instruction, and consistently outperforms various state-of-the-art methods.

By centering smart glasses as wearable-intelligence platforms, this survey provides a unified framework for organizing technologies, applications, and open challenges in this emerging area.

Inspired by human cognition, CRCD reformulates the task as counterfactual latent change captioning to bypass affirmation bias, and contrasts a synthesized safe expectation with reality to identify semantic omissions.

GPUB is introduced, a large-scale benchmark comprising 3,000 livestream instances with quality-controlled multi-moment temporal annotations and a catalog of over 31K fashion products that supports three evaluation tasks: the main task Grounded Product Understanding (GPrU) requires jointly identifying the target product and localizing its supporting moments from a livestream video and a candidate product set.

This work introduces PosteriorBench, a benchmark for evaluating the distributional accuracy of generative inverse solvers, and reveals substantial distribution-matching gaps across current solvers, while showing that neural operators improve resolution robustness, and guidance weights and generation noise are key to posterior-variance calibration.

FedQoS is proposed, a novel asynchronous, event-triggered FL framework that decouples local computation from global communication via a two-phase gating mechanism that achieves competitive personalized accuracy with only marginal performance loss compared to FedAvg, while substantially reducing QoS violations.

The proposed E-AVI framework improves predictive performance while providing inspectable support for assessment, feedback, and interactive analysis, and these results demonstrate that E-AVI consistently outperforms fine-tuned multimodal baselines in rank correlation.

Adaptive Uncertainty-Routed Analysis is introduced, a multimodal email threat detection system that analyzes both the content of an email and its embedded URLs and quantifies prediction uncertainty from a URL classifier.

A decomposed confidence layer along three interpretable channels, including perception, layout, and validation is introduced, together with a final conformal risk control, which can be used for reliable STP of financial documents.

V2-STRep is presented, a zero-shot framework that converts generated video motion into reusable robot skills through VLM-grounded structured task representations, and demonstrates improved execution success over baselines, reliable cross-scene transfer of successfully acquired skills, and adaptation to changed deployment instructions.

SnapPhysics enables physically interactive MR experiences without manual parameter tuning by combining instance-level 3D reconstruction and spatial alignment with a physics-aware scene graph that encodes these relationships and per-object metric geometry as structured context for VLM-based property reasoning.

Group Relative Return Policy Optimization (GR2PO), a critic-free reinforcement learning framework for continuous robot control, is proposed and significantly outperforms critic-free baselines that use immediate rewards and performs competitively against state-of-the-art actor-critic methods.

The collected dataset represents the first multimodal dataset that provides continuous visual, physiological, and operator's bilateral motion tracking of temporal affective state shifts during bimanual teleoperation and outperforms zero-shot baselines (Qwen, MiniCPM-V in tracking temporal state dynamics).

This work presents a multimodal conversational-context framework for LLM-based ASR that integrates a scenario-controlled data pipeline, scalable multimodal context training, and systematic evaluation and introduces MM-ContextASR Bench, which evaluates contextual understanding and entity error correction across five scenarios.

The DeepSeek-V4.1-Flash model, a multimodal Mixture-of-Experts model with 552B backbone parameters and support for contexts of up to one million tokens, is introduced, substantially improving cost efficiency for agentic workloads and pushing the limits of KV cache compression.

This work introduces PanoCaps, a human-annotated benchmark constructed from panoptic segmentation datasets, and introduces PANORAMA, a VLM that conditions a pretrained segmenter on contextualized phrase representations to obtain candidate masks and learns to select those corresponding to each phrase.

An online multimodal assessment framework that integrates interaction wrench, planar tool-center-point (TCP) kinematics, and skin conductance level (SCL) into four interpretable workload-related factors that keeps the assessment process transparent through explicit workload-related factors and defined weighting rules, while outperforming the corresponding fixed-weight formulation.

The Decoupled Embodiment Model (DEM), which pairs a fine-tuned DINOv3 encoder and a frozen NeoBERT encoder with a MeanFlow head that generates each action chunk in a single forward pass, achieves observed success comparable to state-of-the-art VLM-backbone policies under the evaluation protocol.

This survey systematizes the EML landscape by introducing the first structured, model-to-system taxonomy and establishes a structured framework for multimodal systems that are not only high-performing and generalizable but natively efficient and ready for ubiquitous deployment.

MCLC-NET is proposed, a multimodal continual learning framework for leaf counting that learns tasks sequentially using a memory-based strategy with a memory buffer to retain important samples from previous tasks.

Experimental results demonstrate the proposed $\mathcal{M}^2$Tok-based VLA achieves superior reconstruction fidelity but also significantly boosts the success rate of VLA models, and comprehensive ablation studies further confirm the effectiveness of the multi-head and multi-codebook mechanisms.

GuardEn (Guarding by Safety Rule Entailment), an executable safeguard framework that decomposes safety policies into atomic propositions through Safety-Rule Compilation, modeling their composition as executable code, is proposed.

CapMap-MS-TTA is presented, a training-free submission built on Microsoft Florence-2-base, combining caption keyword mapping with multi-scale flip test-time augmentation, and ranks 3rd on the final MUMU leaderboard.

It is demonstrated that reCAPTCHA can be defeated by a non-technical adversary, using only natural-language instructions to a commodity AI assistant, which collapses the practical attacker skill floor to near zero and fundamentally changes the threat model for challenge-based CAPTCHAs.

This work introduces a comprehensive evaluation framework organized around four complementary dimensions of physical world reasoning, and constructs a diverse set of novel tasks that require models to integrate complementary information across modalities.

The unexpected effects of visual cues on LVLMs highlight the need to understand the presence of an image and its framing when evaluating the robustness of LVLMs.

This work proposes a latency-aware metric for multi-object tracking called streaming-HOTA (s-HOTA), and introduces an ultra-High-resolution Airborne Remote-sensing Dataset (HARD) annotated at three levels for object detection, multi-object tracking, and scene-level visual question answering.

RAFAIL is introduced, a framework for detecting execution failures during robotic manipulation that requires no failure data and achieves 73.4% balanced accuracy across three real-world robotic manipulation tasks, outperforming the strongest evaluated OOD- and uncertainty-based baselines.

UMI-Bridge, which uses UMI as an intermediate domain to align representations according to action equivalence rather than pixel similarity, supports action-anchored latent alignment for data-efficient robot learning and UMI-to-robot task transfer.

CUEES (Correlation-gUided Encoder Selection), a lightweight heuristic that estimates complementarity through task- and category-level Pearson correlations between encoders'performance profiles, scoring a candidate set from single-encoder evaluations alone--without fusion training during selection.

GPT-Policy is introduced, a general-agent framework for in-context robot learning that integrates a context compiler that preserves task-relevant visual transitions, a VLM that proposes robot-tool actions, and a constrained controller that verifies and executes each action and reports its outcome.

ReFigBench, a benchmark and evaluation framework built on 1,000 real overview figures retrieved from arXiv papers with full provenance, is introduced, exposing the tension between fidelity and editability as the central challenge for practical multimodal document agents.

Across four models, attention heads causally necessary for OCR are identified, and it is discovered that these are in fact general-purpose heads that output interpretable semantic features across all image tokens.

A causal-enhanced approach orchestrated end-to-end by an Agentic Controller with self-reflective closed-loop control with self-reflective closed-loop control is proposed, which includes two core modules to address the above two challenges.

Video semantic fidelity (SF), a metric that quantifies how well a compressed video preserves the semantic content of its source, is introduced and shows that SF correlates more strongly with human judgments than conventional video metrics.

A Gated Noise-filtered Sentiment-Relevance Interaction (GNSRI) framework that employs a gated noise-filtering module to suppress sentiment-irrelevant features and enhance aspect-aware sentiment cues, and a sentiment-relevance interaction module to capture consistent and conflicting cross-modal signals at micro and macro levels.

This article categorizes existing UAV inspection architectures, identifies their key system challenges and architectural requirements, and experimentally assesses the feasibility of semantic edge intelligence on NVIDIA Jetson UAV-class hardware using the COCO-Bridge dataset.

Efficient Unified Multimodal Understanding (EUMU), the winning solution for the MUMU Track of the 8th LSVOS Challenge, builds on a shared pretrained multimodal model, using its prompt-based capabilities for detection and captioning and training lightweight heads on shared visual features to predict quality, scene, and event tags.

AVNet is a multimodal audio-visual transformer that classifies emergency vehicles (ambulance, fire engine, police car) and road background using both audio and video, while gracefully handling the absence of either modality at inference time.

This work revisits joint multimodal representation learning and generation to produce linearly interpolatable embeddings that are directly consumable by generative decoders and ensures its representations function as both discriminative semantic descriptors and generative conditions.

OPD-Aha is introduced, which reconstructs the distillation target directly from this isolated visual preference rather than relying on the fragile teacher-student discrepancy, yielding broad and consistent improvements across diverse fine-grained perception and complex multimodal reasoning benchmarks.

EgoPathBench, a dataset and five-task benchmark for first-person waypoint decision-making, and Fine-tuning Qwen 3.5 4B on the released training split improves all four reported evaluations across three external spatial benchmarks, with gains of 1.4--9.6 points.

ASDchat, a multimodal large language model designed for evidence-based ASD screening, which takes video, audio, and dialogue as input, provides a feasible path for large-scale, evidence-based early ASD screening in clinical practice.

JustFit, an MLX-based inference runtime that combines KVExec for compressed KV execution, PhaseSwap for component residency, and StateTrans for state-preserving serving transitions is presented, showing how compact state and lifetime-aware execution expand local serving capacity while supporting extended generated reasoning.

The developed BrainVLM model, which integrates an uncertainty quantification strategy to indicate prediction reliability and a module for generating radiology reports to elucidate the clinical rationale, was trained on multi-modal data from 40,043 individuals.

VideoMM is introduced, which marks a paradigm shift from model-centric downsizing to adaptive perceptual granularity and accelerates inference by 2.73 times over current leading methods, establishing a highly scalable paradigm for long-video understanding.