Search papers, labs, and topics across Lattice.
This paper introduces FRAME, a multi-modal, neuro-symbolic embedding technique designed to improve screen retrieval and clustering in UI programming by addressing the abstraction gap between code and graphical representations. By constructing symbolic, graph-based representations of UI components and leveraging large vision-language models, FRAME effectively captures feature patterns across diverse screens, achieving significant performance improvements. The method outperforms existing baselines by up to 13% in mean reciprocal rank (MRR) for search tasks and 7.6 percentage points in clustering accuracy, highlighting its potential to enhance automated UI design and testing tools.
FRAME achieves up to 13% better search accuracy by bridging the gap between UI code and graphical representations through innovative neuro-symbolic embeddings.
User Interface (UI) programming is challenging due to the complex abstraction gap between code and graphical software representations. To bridge this gap, UI programming tools often rely on screen retrieval and clustering, which require accurate similarity measures based on overlapping features. However, computing feature-oriented similarity is difficult because screens with similar functionality often exhibit design variations. To address this, we propose FRAME (ReinForced UseR InterfAce Screen EMbedding with Graphical Structural ComprEhension), a multi-modal, neuro-symbolic embedding technique. FRAME constructs symbolic, graph-based representations of UI components to encode salient relationships and capture feature patterns across different screens. It leverages large vision-language models for visual and lexical encoding, alongside a novel UI-specific computational geometry algorithm that enables weighted embedding propagation. Across three benchmarks, FRAME outperforms strong baselines by up to 13% MRR in search and 7.6 percentage points in clustering accuracy. A comprehensive ablation study further confirms the benefit of each component, demonstrating FRAME's potential for enhancing automated UI design and testing tools.
