Search papers, labs, and topics across Lattice.
This paper introduces Motion-as-Prompt (MaP), a novel framework that enhances motion reasoning in multimodal large language models (MLLMs) by utilizing track-guided cross-frame visual prompting. By recovering dense point trajectories and selecting motion-informative frames, MaP enables MLLMs to observe critical transitions and interactions that are typically lost in sparse sampling methods. Experiments reveal that MaP significantly boosts motion-reasoning accuracy on benchmark datasets, achieving improvements of 4.2% and 8.9% for GPT-5.5 without compromising non-motion understanding.
Motion-as-Prompt reveals that enhancing motion reasoning in MLLMs can lead to substantial accuracy gains without altering model architecture or requiring retraining.
Motion-centric video reasoning is fundamental to interactive applications such as robotic manipulation and autonomous navigation. However, multimodal large language models (MLLMs) typically process videos through sparse uniform sampling to control visual-token and attention costs. This strategy may discard critical transitions between sampled frames, limiting reasoning about object movement, collisions, and causal interactions. To mitigate this issue, we propose Motion-as-Prompt (MaP), a track-guided cross-frame visual prompting framework. MaP recovers dense point trajectories, selects motion-informative frames, and marks the trajectories accumulated between consecutive sampled frames directly onto the visual inputs, making otherwise hidden displacement, direction changes, and interactions observable to frozen MLLMs. Experiments on CLEVRER and Something-Something-v2 show that MaP consistently improves average motion-reasoning accuracy, yielding gains of 4.2% and 8.9% for GPT-5.5, respectively. Notably, these improvements are obtained without degrading non-motion understanding, highlighting the robustness of MaP. These results demonstrate that MaP provides a simple and effective solution for enhancing motion-centric video reasoning without model training or architectural modification. Project page:https://github.com/SunVictor23/MaP.
