Search papers, labs, and topics across Lattice.
This paper introduces a Structured Visual Compositional Representation (SVCR) framework for weakly supervised referring expression comprehension (WREC), addressing the limitations of existing anchor-level visual representations that fail to capture relational interactions. By explicitly modeling both unary object embeddings and pairwise relational embeddings, the SVCR creates a structured visual representation space that aligns more effectively with the complexity of natural language. Extensive experiments on benchmark datasets demonstrate that this approach achieves state-of-the-art performance, highlighting the importance of structured visual representations in enhancing visual-textual matching under weak supervision.
Explicitly modeling relational interactions in visual representations boosts weakly supervised referring expression comprehension to new heights.
Referring expression comprehension (REC) aims to localize the object in an image described by natural language. In Weakly supervised REC (WREC), existing approaches primarily operate on anchor-level visual representations. Even when enriched with auxiliary cues, relational interactions remain implicitly encoded within individual anchor features. The resulting visual representation remains flat and unary-only, limiting its ability to align with the structured nature of language. In this work, we propose a Structured Visual Compositional Representation (SVCR) learning framework for WREC. Rather than implicitly encoding relations within unary anchors, the proposed SVCR explicitly models both unary object embeddings and pairwise relational embeddings, forming a structured visual representation space. We further introduce a compositional alignment mechanism that matches unary and pairwise visual representations with their corresponding textual embeddings in a unified manner, enabling compositional visual-textual matching under weak supervision. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg show that the proposed SVCR achieves state-of-the-art performance. These results demonstrate the effectiveness of explicit structured visual representations and visual-textual alignment for WREC.
