Search papers, labs, and topics across Lattice.
This paper introduces VistaRef, a framework that enhances spatial orientation awareness in pointing-to-object detection by addressing the limitations of Transformer-based visual models in capturing micro-geometric relationships. The Local Hand Entity Modeling (LHEM) module incorporates hand-pose embeddings to improve the model's sensitivity to finger deviations, while the Geometric Ray Modeling (GRM) module transforms implicit orientation data into explicit spatial features. Experimental results show that VistaRef achieves a 14-point absolute gain in grounding accuracy, effectively bridging the spatial perception gap in complex scenarios.
Achieving a 14-point boost in grounding accuracy, VistaRef redefines how we approach spatial orientation in AR and human-robot interaction.
Grounding deictic gestures in natural images is fundamental to AR and human-robot collaboration, providing a basis for seamless spatial interaction. While Transformer-based visual models have achieved significant progress in general object detection, their global attention mechanisms often neglect micro-geometric relationships, degrading orientation accuracy. In pointing tasks, this deficiency manifests as an inability to accurately capture the pointing ray implied by finger poses, which results in pointing drift and localization ambiguity when dealing with distant or densely packed objects. To address this, we propose VistaRef, a framework designed to explicitly enhance spatial orientation awareness. First, we develop the Local Hand Entity Modeling (LHEM) module, which incorporates hand-pose embeddings to strengthen the model's capability to capture subtle finger deviations. Second, drawing inspiration from multi-view geometry, we construct the Geometric Ray Modeling (GRM) module to transform implicit orientation information into explicit spatial geometric features, guiding feature aggregation and deep fusion via attention mechanisms. Furthermore, we introduce a novel Orientation-Consistent Alignment Loss (OCAL) to synergistically supervise hand presence and pointing consistency, ensuring that all architectural improvements collectively serve the core objective of spatial localization. Experimental results demonstrate that VistaRef significantly outperforms the baseline, achieving a 14-point absolute gain in grounding accuracy. Qualitative analysis further confirms that VistaRef effectively models the geometric correlation from hand to target, bridging the spatial perception gap inherent in traditional Transformers for complex scenarios. Code: https://github.com/lingli1724/VistaRef.
Tsinghua AI