Multimodal Models - Weekly Roundup

MultiHuSE is introduced, a multi-modal dataset comprising 2,407 high-definition videos of 50 demographically diverse actors performing 1,463 text samples across four psychological humour styles (affiliative, aggressive, self-enhancing, and self-deprecating), as well as neutral content that provides empirical support for psychological theories linking humour and emotion.

This work presents RelateAnything, a 53M-parameter model taking an image and regions from any source and returning scored relations over a predicate vocabulary supplied at inference as strings, a 53M-parameter model taking an image and regions from any source and returning scored relations over a predicate vocabulary supplied at inference as strings.

It is shown that all six estimators identify correct annotations as most informative, and are able to audit the quality of mixed text corpora, choosing the annotations that result in the best downstream forecasting results without the need for model training.

MaP-WAM is introduced, a Memory-as-Plans framework that decomposes memory-dependent world-action modeling into memory-grounded planning and plan-conditioned execution, and uses long-term multimodal episodic context as planning-time evidence rather than repeatedly conditioning the executor on the full history.

This work presents RelateAnything, a 53M-parameter model taking an image and regions from any source and returning scored relations over a predicate vocabulary supplied at inference as strings, a 53M-parameter model taking an image and regions from any source and returning scored relations over a predicate vocabulary supplied at inference as strings.

It is shown that all six estimators identify correct annotations as most informative, and are able to audit the quality of mixed text corpora, choosing the annotations that result in the best downstream forecasting results without the need for model training.

MaP-WAM is introduced, a Memory-as-Plans framework that decomposes memory-dependent world-action modeling into memory-grounded planning and plan-conditioned execution, and uses long-term multimodal episodic context as planning-time evidence rather than repeatedly conditioning the executor on the full history.

The Feature Recovery Module (FRM), a plug-and-play module that maps degraded encoder features to pristine-aligned representations while keeping the host frozen, is proposed, which improves scene-level detection, CLIP/SigLIP2 feature recovery, and all four object-level VLM tasks, with larger gains under more severe degradation.

Modern VLMs already encode much of what is needed for specialized domains; the challenge is learning how to ask, suggests modern VLMs already encode much of what is needed for specialized domains; the challenge is learning how to ask.

This work proposes MomentUm SpEctral Clipping (Musec), which replaces Muon's spectral flattening with spectral clipping: rather than setting all singular values of the momentum matrix to approximately one, Musec clips singular values that exceed a threshold while preserving the underlying spectral structure of the momentum.

Generative Late-Interaction Embeddings (GLIE): k<<N vectors per page learned from the normalized centroids to serve as both a lightweight index and a basis for regenerating the page's full embedding set, with the decoder as its main design surface.

Native unified modelling is position as a promising path towards systems that perceive, reason and create within a fully end-to-end framework through SenseNova-U1.5, an 8B-MoT native unified multimodal model that understands, reasons about, and generates visual content within an encoder-free and VAE-free architecture.

The Logit Refiner is introduced, a lightweight autoregressive module that restores intra-scale dependencies by sequentially sampling tokens conditioned on frozen backbone features and generalizes to text-to-image generation, confirming that the mean-field bottleneck persists across VAR variants and is effectively alleviated by the method.

Findings show that foundation models require CGM-specific adaptation for reliable forecasting and that dietary context provides clinically meaningful signals beyond CGM alone, especially during postprandial periods.

This work generates synthetic plankton imagery conditioned on taxonomy: a CLIP encoder is adapted on a large plankton corpus with a ranked contrastive objective extended to deep, ragged taxonomies, then frozen to condition a parameter-efficient diffusion transformer.

This study examines how synthetic and real historical training data should be combined for low-resource OCR, and finds that joint and sequential training yield broadly similar archival accuracy under the tested practical pipelines.

Equal parameter count evaluations show that modulating the primitive's wavefront is an effective and lightweight enhancement for hologram representations, and frequency domain analysis illustrates that CVQPG has successfully preserved the mid-to-high frequency band of natural images.

This paper proposes a meta-learning framework that leverages a comprehensive set of meta-features capturing dataset complexity to predict classifier performance without exhaustive training, and achieves an average ranking prediction accuracy exceeding 86%, demonstrating its effectiveness in guiding model selection.

SolCloudLLM aligns sky-image patches with time-series patches and fuses their corresponding representations through bidirectional multimodal fusion, yielding a unified representation that is subsequently mapped into the embedding space of an LLM.

Control ablations show that gains in cross-regime adaptive propagation are not explained by single-basis propagation, an additional same-family branch, or within-family adaptive order alone, supporting complementary two-basis propagation as an effective design principle for visual representation learning.

Evaluation on a representative multimodal benchmark demonstrates that EMMI can reduce the communication payload by 32x while maintaining comparable downstream accuracy, resulting in up to a 3.4x reduction in estimated end-to-end inference latency under bandwidth-constrained edge conditions.

HALDETECT, the authors' system for the English hallucination-detection track (Task 1b) of ImageEval 2026, in which a system must identify, from an image and three culturally plausible statements, the single visually grounded one, is presented.

RAMamba-Net is proposed, a reliability-aware Mamba-based multimodal fusion network for AAD that effectively exploits complementary EEG-EOG information, yielding accuracy gains over unimodal baselines, and is robust to signal perturbation and parameter variation.

This work proposes Vid-PRE (Video Prompt Reasoner and Enhancer), a model-agnostic prompt rewriter that offloads the cognitive burden of reasoning to a dedicated VLM, and introduces VWG-Bench, a comprehensive benchmark spanning 9 reasoning dimensions and 38 fine-grained tasks.

This work empirically diagnose the root cause across six per-layer methods and three speech-LLM architectures and proposes a two-pool allocation that normalises encoder and LLM parameters into independent pools, and shows that joint $\ell_2$ sensitivity and the original $(\varepsilon,\delta)$-DP guarantee are unchanged.

MultiHuSE is introduced, a multi-modal dataset comprising 2,407 high-definition videos of 50 demographically diverse actors performing 1,463 text samples across four psychological humour styles (affiliative, aggressive, self-enhancing, and self-deprecating), as well as neutral content that provides empirical support for psychological theories linking humour and emotion.

A unified evaluation framework testing gradient and loss-based balancing strategies under controlled settings, and a theoretical diagnosis explaining why these methods fail, as they conflate fitting speed with discriminative contribution are provided.

ReGround is introduced, a large-scale dataset for reviewer comment grounding that links 10,267 reviewer comments to 16,274 evidence in the original anonymous submission of 3,656 papers and exposes grounding reviewer comments as a difficult and practically important problem for scientific document understanding.

This work proposes Caption-once, Frames-onDemand (CFD), a budget-aware edge-cloud agentic framework that turns visual access into a first-class, query-conditioned cost, capping per-query frame consumption regardless of video length.

This paper proposes MOCO, a novel diffusion-based framework capable of processing multiple simultaneous inputs, including speech audio, text descriptions, and trajectory data, to generate coherent and lifelike motions without requiring aligned multimodal data.

HighPerViT is presented, a compact vision-only architecture that injects an explicit second-order statistical prior into a transformer-based recognition pipeline, and integrates this token with first-order spatial representations through Perceiver-style latent distillation, supporting explicit statistical tokenization as an effective and robust design principle for texture-centric visual recognition.

IMLE-VLA is introduced, which replaces the iterative action head with a single-step conditional generator trained via conditional Implicit Maximum Likelihood Estimation (cIMLE), and promotes multimodal action coverage, avoiding the mode collapse of naive regression heads while eliminating multi-step sampling entirely.

These results validate the deployed navigation and inspection closed loop, while HROS provides an extensible software foundation for memory-augmented, voice-aware, and continuously improvable embodied inspection agents.

By decoupling haptic control loops from network bottlenecks, CARLAverse enables scalable, cross-institutional HITL experiments without compromising physical immersion.

This report presents a multi-view and confusion-guided ensemble framework for the Synthetic Image Attribution Challenge of the DLMMDD Workshop at ICANN 2026, and introduces a dedicated binary expert classifier that is selectively activated under low-confidence conditions.

Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate.

The model incorporates three core components: spatiotemporal embedding module, spatiotemporal fusion module, and LLM backbone, which adopts a differentiated parameter adaptation strategy to balance training efficiency and traffic data adaptability.

This paper model MTO as a sequence of sequential transfer optimization problems, concentrating evaluations on a single target per iteration, and proposes a likelihood-informed task prioritization mechanism to maximize transfer utility by identifying the task most likely ready for knowledge integration.

A novel semi-tensor product for third-order tensors under the t-product framework induced by arbitrary invertible linear transforms is introduced, yielding an accelerated multi-term randomized semi-tensor product SVD algorithm that achieves a balance between reconstruction accuracy and computational efficiency.

It is tested whether models in this deployment-relevant 2--8B range carry genuine taxonomic knowledge, evaluating four such VLMs against the domain-specific specialist BioCLIP on a 96-species task, and comparing clean iNaturalist photographs against camera-trap imagery from 6 LILA.

Whether vision language models can decide when to answer immediately and, when more evidence is needed, which experiment to perform is studied to reveal limitations in physical reasoning that conventional answer accuracy can overlook.

CoMA-DiT is proposed, a bidirectional cross-modal Diffusion Transformer for latent augmentation that treats paired modalities as sources of mutual generative supervision rather than merely as inputs to be fused.

A unified, patch-based framework that processes raw multimodal imaging through training-free registration, automated localization, and mask-filtered patch extraction that culminates in a hierarchical strategy that aggregates patch-level insights into subject-level diagnostics.

2AM shows that task memory can remain Agent-side and shows that Action Model capability depends not only on what the policy has learned, but on how precisely the Agent can steer it, and isolates this question through a deliberately constrained design.

An initial assessment of the suitability of VLM-based methods for V&V of defence applications is provided and a VLM-based error slice detection (ESD) method that independently groups and labels systematic errors made by a classification model is proposed.

The Eloquence team's approach to Task 2 of the 2nd MLC-SLM challenge at Interspeech 2026, which involves multilingual Multiple-Choice Question Answering (MCQA) across 21 languages, is detailed, which involves multilingual Multiple-Choice Question Answering across 21 languages.

This study forms continuous recognition of the Group Emotion at a one-second resolution, introduces the Mixed state, which captures the emotional divergence among participants in the group, and proposes a multimodal temporal framework that integrates audio and video information using a sliding-window context.

A new completeness estimation approach is introduced that quantifies the degree of sentiment-relevant information preserved in incomplete data to guide the reconstruction of missing semantics and enables more accurate semantic reconstruction, leading to more precise sentiment prediction.

M2Heat is introduced, a physics-inspired framework that investigates multimodal fusion through the lens of heat conduction and indicates the potential of heat-conduction-guided neural operators for efficient and interpretable RS multimodal fusion.

EHR (Emotion Hallucination Rate), an evaluator that quantifies emotion hallucinations across six facets, and HMER (Hallucination-aware Memory-guided Emotion Reasoning), a training-free framework for emotion hallucination mitigation, which enables fine-grained mitigation across diverse hallucination facets.

Vidu S2, which comprises Vidu S2-Avatar, a real-time interactive digital-character model, and Vidu S2-Editing, a real-time video editing model, is presented and the feasibility of real-time spatial video generation for both Vidu S2-Avatar and Vidu S2-Editing is explored.

This survey covers inference-efficiency mechanisms for visual and audiovisual VideoLLMs that report concrete reductions in parameter count, FLOPs per input, latency, memory, or visual and audio token count, and organize methods by the pipeline stage at which they act.

Extreme label skew degrades federated multimodal performance nearly three times more than a sevenfold increase in participating clients, exposing data heterogeneity rather than network scale as the primary failure mode for decentralized clinical AI.

Autonomous vehicles can maintain millimeter-wave beam alignment even when critical sensors drop out by leveraging generative cross-modal reconstruction across camera, LiDAR, and radar feeds.

Standard open-vocabulary detectors fail on nearly 70% of ambiguous, task-driven queries, but layering structured semantic retrieval and dynamic LLM concept expansion over frozen detectors pushes grounding success to 85%.

Unstructured conversational AI fails complex creative handoffs—anchoring revision tasks to a structured intent-evidence-action record prevents criteria drift while keeping human aesthetic authority intact.

Fragmented satellite data no longer means blind spots: learning over incomplete multi-sensor constellations nearly triples usable methane plume events while slashing false positives by 8.19 points.

Benchmark-average rankings mask massive sample-level disagreement among vision pruning techniques: dynamically routing inputs across existing pruning methods yields a 26.9% relative accuracy jump over any single fixed strategy.

Instead of mimicking gold-standard examples, MLLMs reason significantly better when shown suboptimal outputs alongside explicit contrastive paths to correct them.

Industrial warehouse safety exposes a massive capability gap in frontier LMMs, where even top proprietary models consistently fail to jointly ground visual hazards and reason over operational risks.

Monolithic federated updates are a fundamental bottleneck for embodied AI: decoupling vision, language, and action streams cuts uplink payloads by 96% and beats standard FedAvg by 22 percentage points under real-world wireless interference.

Treating distillation targets as dynamic on-policy decisions rather than static teacher outputs prevents catastrophic error propagation when adapting compact vision-language models to out-of-distribution multimodal data.

Probing internal two-token representations within a compact 4B VLM rivals massive model evaluation, providing high-resolution token-level hallucination detection when paired with a 400B judge.

Instead of forcing entire videos into a single, lossy feed-forward context, training an agent to actively read and write temporal memory yields 78.8% accuracy on MINERVA with virtually zero performance drop on long-duration clips.

Even top-performing voice MLLMs break down when forced to talk science, routinely failing to verbalize symbolic notation, decode technical jargon, and adapt across progressive multi-turn interactions.

Leading audio LLMs suffer up to a 41-percentage-point performance drop when prompted in native Southeast Asian languages compared to English, exposing severe blind spots in acoustic temporal grounding and paralinguistics.

Directly conditioning video diffusion on audio wastes massive capacity on static background and identity pixels—routing control transitively through a causal motion latent distilled under a single frozen video teacher achieves real-time streaming at 15.4 FPS with zero fidelity loss.

Standard pixel augmentations frequently corrupt delicate vision-language alignment, but injecting diffusion-style isotropic noise directly into embedding spaces breaks through the longstanding performance ceiling of stacked CutMix, Mixup, and RandAug recipes.

Supervised few-shot adaptation no longer has to break conformal prediction guarantees: aligning 1D nonconformity score distributions restores distribution-free calibration on unlabeled query sets without forcing practitioners to freeze model weights.

Standard multimodal fusion treats modality weights uniformly across emotional states, but coupling facial geometry with class-adaptive gating and valence-arousal transition priors drives up to a 4.36 F1 gain on conversational emotion tracking.

Leading MLLMs still struggle to articulate and localize physical changes across revisited scenes, exposing a fundamental blind spot in multi-image spatial grounding that targeted synthetic data can overcome.

Diffusion-generated thermal faces can effectively break the multi-modal data bottleneck in biometrics, outperforming models trained on scarce real pairs without requiring costly image translation at inference time.

Low-quality modalities can quietly sabotage multi-modal networks because they receive minimal training gradients yet dominate test-time skip connections—a failure mode solved by explicitly filtering and rebalancing degraded feature streams.

2D foundation model priors can be transferred directly into coordinate-free 3D point representations via conditional diffusion, forming an emergent multimodal bottleneck that achieves non-trivial 3D semantic understanding without native 3D labels.

Tiny models can conquer severe clinical domain shifts: grounding 3D kinematic sequences in LLM-generated semantic descriptions and merging source-domain weights delivers top-tier Parkinsonian gait severity prediction with just 637K active parameters.

Standard video caption metrics penalize valid visual descriptions for mere lexical divergence, a failure mode solved by testing whether captions contain sufficient factual signal to answer downstream multiple-choice video comprehension questions.

Wearable-free video models can match the precision of physical IMU sensors during severe turning-induced self-occlusion by distilling kinematic topologies directly into visual latent spaces.

A training-free framework that combines a query-independent layout-derived interaction prior with instruction relevance and feature novelty to rank visual evidence according to both potential future utility and diversity, allowing retained visual evidence to shrink monotonically across budgets while remaining reusable throughout the trajectory.

Shrinking the geometric modality gap in VLMs can paradoxically destroy retrieval performance, exposing a critical disconnect between representation geometry and task alignment that unbalanced optimal transport tracks with near-perfect fidelity (0.973 vs. 0.392 Spearman).

Zero-shot composed video retrieval can jump by over 35% absolute R@1 without any parameter updates—provided you route frozen foundation models as a dynamic compute ladder rather than a uniform scoring function.

Dense pixel-level segmentation can be performed directly inside an autoregressive token space without external decoders by serializing binary change masks into grammar-constrained quadtrees.

Today's specialized video hallucination detectors peak at an abysmal 34.63% accuracy under adversarial conditions, revealing that the verifiers meant to guardrail multimodal foundation models are fundamentally untrustworthy.

A simple, training-free framework in which a reasoning-capable vision-language model iteratively searches and reasons over Wikipedia, gathering evidence dynamically, shows that reasoning and retrieval are complementary on rare entities.