Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
LaMER addresses weakly supervised Referring Expression Comprehension (REC) by dynamically routing frozen expert visual encoders based on language conditioning, mitigating the semantic gap and representation heterogeneity. The framework employs a Cross-Modal Routing Fusion module and a Cell-Text Matching Head with negative anchor augmentation, optimized via a multi-objective loss. This approach achieves state-of-the-art performance on RefCOCO, RefCOCO+, and RefCOCOg datasets, significantly outperforming prior weakly supervised methods.
Dynamically routing multiple frozen visual experts based on language context unlocks new state-of-the-art in weakly supervised referring expression comprehension.
Referring Expression Comprehension (REC) localizes the image region described by a natural language expression, underpinning applications such as autonomous navigation, human-robot interaction, and medical image analysis. Under weak supervision, REC is challenged by the visual-linguistic semantic gap and the difficulty of inferring region-text correspondences from coarse image-level signals alone. Moreover, no single visual encoder captures all complementary properties---semantic alignment, fine-grained texture, and boundary awareness---while naively fusing multiple pretrained encoders yields heterogeneous, hard-to-reconcile representations. We propose LaMER, a weakly supervised REC framework that dynamically aggregates four frozen expert encoders through a language-conditioned Cross-Modal Routing Fusion module, adapting expert selection to each referring expression without instance-level bounding-box supervision. A Cell-Text Matching Head with Negative Anchor Augmentation, trained with a multi-objective loss combining contrastive, reconstruction, alignment, and load-balancing terms, further stabilizes optimization. LaMER achieves 77.67\%, 62.81\%, and 69.49\% Acc@IoU\(\geq 0.5\) on RefCOCO, RefCOCO+, and RefCOCOg, surpassing DViN and WeakMCN by up to 16.49 and 14.49 points, with ablations validating each component.