Search papers, labs, and topics across Lattice.
This paper introduces a multi-modal extension of YOLOv5 for UI control detection, incorporating GPT-generated textual descriptions of UI images via cross-attention. The model aligns visual features with semantic information from text embeddings, improving robustness and context-awareness. Experiments on a large UI dataset show that convolutional fusion of visual and textual modalities significantly outperforms the baseline, especially for semantically complex elements.
Injecting LLM-generated text descriptions into a vision model boosts UI element detection, particularly for ambiguous cases where visual cues alone fall short.
Detecting user interface (UI) controls from software screenshots is a critical task for automated testing, accessibility, and software analytics, yet it remains challenging due to visual ambiguities, design variability, and the lack of contextual cues in pixel-only approaches. In this paper, we introduce a novel multi-modal extension of YOLOv5 that integrates GPT-generated textual descriptions of UI images into the detection pipeline through cross-attention modules. By aligning visual features with semantic information derived from text embeddings, our model enables more robust and context-aware UI control detection. We evaluate the proposed framework on a large dataset of over 16,000 annotated UI screenshots spanning 23 control classes. Extensive experiments compare three fusion strategies, i.e. element-wise addition, weighted sum, and convolutional fusion, demonstrating consistent improvements over the baseline YOLOv5 model. Among these, convolutional fusion achieved the strongest performance, with significant gains in detecting semantically complex or visually ambiguous classes. These results establish that combining visual and textual modalities can substantially enhance UI element detection, particularly in edge cases where visual information alone is insufficient. Our findings open promising opportunities for more reliable and intelligent tools in software testing, accessibility support, and UI analytics, setting the stage for future research on efficient, robust, and generalizable multi-modal detection systems.
