Search papers, labs, and topics across Lattice.
This paper introduces ROSE, a benchmark designed to evaluate how effectively multimodal large language models (MLLMs) can translate visual information into context-specific actions. By maintaining a fixed visual scene while varying task constraints, the study reveals a significant performance drop of up to 44.5 percentage points in MLLMs when transitioning from counting tasks to action-oriented tasks, despite near-perfect human performance. The findings highlight a critical model-dependent bottleneck in the ability to leverage shared visual evidence for contextually appropriate actions, underscoring the complexity of perception-to-action translation in AI systems.
MLLMs struggle to convert visual evidence into context-specific actions, with performance plummeting by over 44% in certain scenarios.
Multimodal large language models (MLLMs) are increasingly expected to act on visual information, yet the same scene may require different actions under different task contexts. How reliably can a model turn the same visual evidence into the action required by the current context? To answer this question, we introduce \textsc{ROSE} (\textbf{R}eference-conditioned \textbf{O}ddity and \textbf{S}ymbolic \textbf{E}xecution), a controlled benchmark that holds the visual scene fixed while varying region constraints and required symbolic outputs. Through coupled counting and coordinate-action tasks, \textsc{ROSE} tests whether models can infer an implicit majority reference and act on the resulting fine-grained visual evidence under changing contexts. Across nine recent MLLMs, performance drops by as much as 44.5 percentage points from counting-oriented tasks to region-conditioned action, despite 98.8\% human performance. The gap persists on paired scenes and regions for which the same model returns the correct count, while global-click and matched local controls show that coordinate grounding explains only part of the loss, revealing a distinct, model-dependent bottleneck in turning shared visual evidence into context-specific actions.
