Search papers, labs, and topics across Lattice.
This paper addresses the limitations of Multimodal Large Language Models (MLLMs) in spatial reasoning within complex egocentric scenes by introducing Ego Scene Augmentation (ESA), which utilizes an Ego-element Graph to enhance spatial perception. By constructing this graph to integrate egocentric spatial features from visual foundational models, the authors significantly improve MLLM performance on the EgoTextVQA benchmark, achieving gains of 8.14% in indoor settings and 8.72% in outdoor settings. Notably, the ESA framework excels particularly in the shopping subset of the indoor setting, showcasing its practical applicability in real-world scenarios.
MLLMs can now achieve over 8% improvement in spatial reasoning for egocentric scenes by leveraging a novel Ego-element Graph for enhanced perception.
Egocentric Visual Question Answering (VQA) has attracted widespread attention as an important task for enabling Multimodal Large Language Models (MLLMs) to interact with the real world. However, existing MLLMs struggle to perform effective spatial reasoning in complex egocentric scenes due to their limited spatial perception capabilities. To this end, we introduce Ego Scene Augmentation (ESA), an egocentric spatial perception framework, which actively enhances the spatial perception capabilities from the egocentric perspective, powered by the proposed Ego-element Graph. Our core insight is leveraging the Ego-element Graph as an intermediary representation to augment the egocentric spatial perception of MLLMs via visual foundational models. Specifically, we 1) construct the Ego-element Graph, which encapsulates and integrates egocentric spatial features enabled by visual foundational models; 2) enhance the spatial perception capabilities of MLLMs via the Ego-element Graph for ego-perspective scenes. Our proposed ESA framework presents significant performance improvement on the EgoTextVQA benchmark. We achieve an 8.14% gain on the indoor setting and an 8.72% gain on the outdoor setting. Furthermore, our ESA shows the most impressive performance improvement in the shopping subset of the indoor setting. The project code is publicly available.
