Multimodal Models - Weekly Roundup

Clustering training data by text embeddings significantly outperforms audio embeddings in generating coherent music structures.

Achieving up to 79-fold latency reductions in task-oriented image transmission without sacrificing classification accuracy could revolutionize communication under limited spectrum conditions.

Achieving top rankings in a competitive setting, this work reveals how hierarchical soft-label learning can significantly enhance multimodal sexism detection in memes.

Automated view scheduling in SceneFrom3D transforms the landscape of outdoor 3D scene generation, enabling unprecedented control over object appearance and geometry.

Simple Best-of-$N$ sampling can outperform complex guided search methods in text-to-image generation, revealing a critical flaw in how we evaluate efficiency in diffusion models.

Achieving top rankings in a competitive setting, this work reveals how hierarchical soft-label learning can significantly enhance multimodal sexism detection in memes.

Automated view scheduling in SceneFrom3D transforms the landscape of outdoor 3D scene generation, enabling unprecedented control over object appearance and geometry.

Simple Best-of-$N$ sampling can outperform complex guided search methods in text-to-image generation, revealing a critical flaw in how we evaluate efficiency in diffusion models.

UI-MOPD achieves a remarkable balance between retaining existing capabilities and adapting to new platforms, with task success rates that challenge conventional approaches in GUI agent learning.

Higher resolution in real-time audio-visual interactions can be achieved without sacrificing latency, enabling clearer agent representation in conversations.

A mere three poisoned samples can render a robot completely non-functional, highlighting a severe vulnerability in open-source robotics.

DynaVieW reveals that a schema-guided approach can drastically improve the modeling of visual dynamics, leading to superior performance in narrative generation and simulation tasks.

Current avatar systems are more diverse than ever, yet foundational prior learning is often overlooked in discussions of photorealistic digital humans.

Adversarial purification can be dramatically improved by focusing on patch-level semantics, leading to state-of-the-art performance in defending against adversarial attacks.

Membership inference attacks can now be effectively executed across multiple generative modalities with a single, unified framework.

Energy consumption in video generation models can be accurately predicted without direct access to model details, revealing a surprising adherence to scaling laws.

Surgical affordance maps can now be predicted with unprecedented accuracy, paving the way for true robotic autonomy in complex surgical environments.

CRISP achieves superior long-horizon point cloud forecasting and versatile downstream task performance by leveraging a unique forecasting-based pretraining approach with camera-radar fusion.

LiDAR NeRFs can be dramatically improved by optimizing volume sampling density, leading to superior 3D mapping results.

XS-VLA outperforms larger models by leveraging spatial distillation and generative flow control, achieving remarkable efficiency in robotic manipulation.

By focusing on low/mid-frequency patterns, UniSkip-Mamba outperforms existing models in forgery detection, achieving a remarkable 63.4% AP@0.95 on LAV-DF.

Visualizing cue-to-layer interactions reveals that biophonetic features are crucial for distinguishing genuine speech from spoofed audio, providing insights into model interpretability.

Automatically constructed data can dramatically enhance the temporal localization abilities of audio models, overcoming the limitations of manual annotation.

ResearchStudio-Reel not only automates research dissemination but does so with unprecedented quality, outperforming both traditional methods and leading LLMs in aesthetic appeal and information accuracy.

Siamese student encoders not only regularize the JEPA objective but also significantly enhance representation learning efficiency and accuracy.

Achieving a 40x speedup in image-to-video generation on mobile devices without sacrificing visual quality could revolutionize cinematic content creation on the go.

Tactile feedback can elevate visual robot policies from failure to near-perfect success in contact-rich tasks in under 80 minutes.

Attention dynamics reveal that MLLMs can fall back on language priors when visual context is disrupted, highlighting the fragility of multimodal integration.

CGGS achieves unprecedented coherence and accuracy in text-driven 3D scene generation, setting a new standard for ego-centric visual content creation.

A lightweight Q-value model can boost a 9B VLA's performance beyond that of a 27B model while reducing inference latency by 27%.

A unified decision process for multi-modal reasoning reveals that joint optimization of text and image generation can dramatically enhance performance in complex reasoning tasks.

Executable vector graphics enable MLLMs to achieve human-like spatial reasoning through a structured visual workspace.

Flex-Forcing achieves superior video generation quality and stability by unifying autoregressive and bidirectional methods, all while speeding up inference.

Unlocking lossless parallel generation in dense video captioning could redefine efficiency standards in multimodal AI systems.

Bridging the gap between probabilistic reasoning and deterministic execution could redefine safety standards in robotic applications.

Users can now generate and control high-quality videos in real-time using just their voice, revolutionizing interactive content creation.

Reducing sampling steps from 50 to just 8 without sacrificing quality could revolutionize how we approach generative modeling.

Predicting taste from audio embeddings not only surpasses human consensus but also redefines the benchmarks for music retrieval systems.

LIME turns ordinary egocentric video into a powerful tool for robots to dynamically adjust their camera poses based on user intent, revolutionizing how we think about robotic perception.

A frozen policy can achieve up to 86% success in manipulation tasks through guided inference, underscoring the power of learned critics in real-time decision-making.

Existing methods struggle with artifacts and consistency, but NeoMap achieves superior novel view synthesis without any training.

Bridge-WA achieves superior task performance by predicting where and how the world will change, enabling robots to focus on relevant scene dynamics rather than irrelevant visual details.

Learned visual front-ends can outperform classical tracking methods in specific scenarios, reducing trajectory errors by up to 38%.

CoRe achieves preference-aligned reinforcement learning by seamlessly integrating human-like reward decomposition, outperforming traditional methods in both simulated and real-world environments.

VLA-Corrector allows VLA models to adaptively replan actions in real-time, drastically reducing compounding errors in dynamic environments.

Combining language supervision with future latent alignment in VLA models leads to unprecedented stability and transfer performance across diverse robotic tasks.

Arachne achieves up to 65% faster iteration times for Text-to-Video model training by optimizing the orchestration of computational units across diverse data.

Clustering training data by text embeddings significantly outperforms audio embeddings in generating coherent music structures.

Achieving an impressive hierarchical F1 score of 81.25% through innovative multi-branch modeling and KNN post-processing reveals new potential in audio classification tasks.

Achieving 80% throughput with Classifier-Free Guidance challenges the assumption that CFG drastically reduces efficiency in multimodal audio processing.

Interleaving speech and text during ASR training boosts entity recognition accuracy and narrows the gap between modalities, challenging traditional training paradigms.

Non-verbal vocalizations can be effectively modeled in ASR, improving recognition of rare events without sacrificing lexical accuracy.

NEUROSYMLAND achieves a remarkable 61 successful UAV landing assessments in challenging terrains, showcasing a leap in safety and interpretability over existing methods.

Distribution-wise rewards can drastically enhance image diversity and quality in generative models, reducing mode collapse and reward hacking issues.

Achieving 97.7% accuracy with a quantum fusion module that slashes parameter counts by 10x could redefine efficiency in federated learning for multi-agent systems.

Object-centric LeJEPA achieves superior performance on key vision tasks while requiring significantly less training data than traditional image-level methods.

Pixelation can degrade VLM performance by over 34%, exposing a tension between patient privacy and diagnostic reliability in medical imaging.

Correlation-structured supervision from video-derived ordinal signals can replace traditional reward mechanisms, achieving robust policy learning across diverse tasks.

SelectTSL achieves precise localization of user-specified target sounds in complex acoustic environments, outperforming traditional methods that lack selectivity.

Entropy-Aware Dense Pruning not only filters out textual noise but also ensures a comprehensive visual representation, leading to superior performance in vision-language tasks.

Attention mechanisms can drastically improve pose sensing accuracy in the face of challenging visual conditions like occlusion and weak textures.

CoFL-S achieves superior navigation performance by leveraging language-conditioned flow fields, outperforming traditional action representations in both simulation and real-world applications.

MolSight achieves unprecedented accuracy in molecular image reasoning by seamlessly integrating chemical topology with vision-language processing.

Hidden evidence-use forgetting can lead to accurate answers that lack grounding, but a new framework shows how to preserve the evidence path behind those answers.

Achieving up to 79-fold latency reductions in task-oriented image transmission without sacrificing classification accuracy could revolutionize communication under limited spectrum conditions.

PairCoder boosts artifact verifiability by up to 3.9 times compared to traditional single-pass inference, revealing the power of collaborative programming in AI-generated outputs.

Models that excel in multiple-choice tasks can falter dramatically in open-ended and error identification formats, revealing a critical gap in art-historical reasoning capabilities.

Instruction-following speech models can be trained effectively without the cumbersome instruction tuning process, challenging the status quo in SLM development.

Text communication among LLMs can lead to significant output homogenization, challenging assumptions about diversity in multi-agent interactions.

Existing image comparison methods miss the mark in UI testing, but a new benchmark reveals that advanced techniques can drastically cut down on irrelevant noise.

ISU-Test reveals that systematic scene generation can dramatically improve the detection of failures in vision-language models used for critical in-car safety applications.

PanoSeeker achieves unprecedented search efficiency and segmentation accuracy in dynamic 360° environments, redefining how agents can actively perceive and interact with their surroundings.

Achieving accurate infrared and polarimetric renderings from just RGB images could revolutionize how we approach multimodal scene reconstruction.

GLEN not only outperforms traditional video models but also enables structured, interpretable predictions of how scenes evolve with human activities.

State-of-the-art vision-language models fail to leverage visual context, leading to biased outputs, but a new training framework shows they can learn to infer concepts from image sets effectively.

Align4D transforms any input modality into high-fidelity 4D content, setting a new standard for multimodal generative models.

GeoMix slashes localization errors by up to 90% in descriptor-free visual matching, challenging the dominance of traditional descriptor-based approaches.

EAGLE-360 redefines visual search in panoramic environments, achieving an unprecedented 8-fold increase in accuracy by integrating global context with local exploration.

A hybrid data collection strategy that blends moving and static viewpoints significantly boosts VLA models' ability to generalize spatially, countering the pitfalls of shortcut learning.

Large-scale structured academic visual data can transform image generation from mere aesthetic appeal to verifiable knowledge-grounded creation.

X-Splat achieves anatomically accurate 3D dental reconstructions from a single 2D panoramic image, outperforming traditional methods that struggle with detail retention.

Early-stage signal enhancement can unlock powerful global modifications in image editing while keeping backgrounds intact.

Synthesized videos reveal systematic differences in how the brain's visual pathways respond to dynamic stimuli, surpassing traditional methods.

InvSplat achieves unprecedented multi-view consistency and material recovery by predicting a structured 3D Gaussian representation in a single forward pass.

Bridging the semantic gap in fashion detail generation, DetailAnywhere sets a new standard by significantly outperforming existing methods in producing high-quality garment close-ups.

FlowCIR slashes training resource requirements by 90% while boosting robustness against negation in zero-shot image retrieval tasks.

Achieving 77.64% accuracy in distinguishing between breast fibroadenoma and phyllodes tumors could revolutionize preoperative decision-making in clinical settings.

QLoRA and BitFit deliver substantial energy savings for fine-tuning vision models without sacrificing accuracy, challenging the notion that more resources always yield better performance.

Achieving state-of-the-art zero-shot object counting without any training, AdaCount redefines how we leverage similarity maps for enhanced object detection in complex scenes.

Existing Video REC models falter dramatically when faced with the complexities of long-form egocentric videos, revealing a critical gap in current methodologies.

Ordinary RGB cameras can now transform trading card game streams into immersive augmented reality experiences without costly hardware.

Achieving a 46% boost in predictive performance for visual scanpath modeling, DeepGaze3.5-VL transforms how we understand human attention dynamics.

VLMs can outperform traditional LPR systems, achieving superior accuracy and robustness in challenging environments without the need for extensive annotated datasets.

Unconstrained egocentric video generation now achieves unprecedented fidelity and control by disentangling hand and camera motion with a novel 3D-aware representation.

LiZAD slashes memory and latency requirements for zero-shot anomaly detection by over 60%, making real-time defect detection feasible on edge devices.

STAR3 redefines automated radiology report generation by seamlessly integrating anatomical, temporal, and clinical context, leading to more relevant and accurate report retrieval.

Heterogeneous knowledge distillation can retain critical spatial information, leading to superior performance across diverse model architectures.

A simple data-driven approach outperforms complex models in ultrasound understanding, revealing the power of scale and alignment.

Binarization methods that ignore weight significance can lead to substantial performance losses, but SAB-LVLM optimizes this process, achieving superior efficiency without sacrificing accuracy.

Achieving high-fidelity motion control in video diffusion transformers without any training or extensive prompt engineering could revolutionize how we generate dynamic video content.

Organizing views into diversity-aware chunks can drastically enhance the performance of geometry transformers while slashing memory costs and inference times.

MMBench-Live achieves a high answer correctness rate while updating benchmarks at a fraction of the cost and time, revolutionizing how we assess VLMs.

Bypassing lossy latent compression, PixGS generates high-quality 3D Gaussian Splats in a single stage, outperforming multi-stage pipelines in both quality and speed.

Early collision prediction in low-visibility conditions can be dramatically improved with the LYNRED-MDS, which captures the complexities of real-world driving scenarios.

DCGNet not only overcomes the limitations of traditional SOD methods in underwater settings but also sets a new benchmark in saliency detection performance.