Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces TraceVision, a vision-language model that incorporates trajectory-aware spatial understanding by bidirectionally fusing visual features with trajectory information using a Trajectory-aware Visual Perception (TVP) module. This approach addresses the limitations of existing LVLMs that struggle with human-like visual attention and region-specific explanations. TraceVision achieves state-of-the-art performance on trajectory-guided captioning, text-guided trajectory prediction, and segmentation, demonstrating improved spatial interaction and interpretable visual understanding, validated on the newly constructed Reasoning-based Interactive Localized Narratives (RILN) dataset.
By mimicking human visual attention, TraceVision significantly boosts spatial reasoning in vision-language models, outperforming existing methods on trajectory-guided tasks.
Recent Large Vision-Language Models (LVLMs) demonstrate remarkable capabilities in image understanding and natural language generation. However, current approaches focus predominantly on global image understanding, struggling to simulate human visual attention trajectories and explain associations between descriptions and specific regions. We propose TraceVision, a unified vision-language model integrating trajectory-aware spatial understanding in an end-to-end framework. TraceVision employs a Trajectory-aware Visual Perception (TVP) module for bidirectional fusion of visual features and trajectory information. We design geometric simplification to extract semantic keypoints from raw trajectories and propose a three-stage training pipeline where trajectories guide description generation and region localization. We extend TraceVision to trajectory-guided segmentation and video scene understanding, enabling cross-frame tracking and temporal attention analysis. We construct the Reasoning-based Interactive Localized Narratives (RILN) dataset to enhance logical reasoning and interpretability. Extensive experiments on trajectory-guided captioning, text-guided trajectory prediction, understanding, and segmentation demonstrate that TraceVision achieves state-of-the-art performance, establishing a foundation for intuitive spatial interaction and interpretable visual understanding.