Search papers, labs, and topics across Lattice.
This paper introduces UI2App, a benchmark designed to evaluate the interaction inference capabilities of large language models (LLMs) in generating executable web applications from UI screenshots. By systematically assessing artifacts across dimensions such as executability and interaction inference, the study reveals a significant gap between visual fidelity and the ability to realize complex interactions, with the top-performing model scoring only 7.5 on the interaction inference score (IIS). The findings highlight that while LLMs excel in visual reconstruction, they struggle with accurately inferring application behavior, particularly for high-complexity interactions like cross-page state management.
Visual fidelity in web app generation doesn't guarantee functional interaction, as evidenced by a leading model scoring 7.5 on interaction inference while trailing others by over 5x.
Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex prompts that impose substantial demands on users and offer limited expressivity for page layout and cross-page visual coherence. Image-driven paradigms, which take UI screenshots as input, align more closely with real development workflows. However, current benchmarks focus primarily on visual fidelity and lack a systematic evaluation of the interaction capabilities in generated artifacts. To address this gap, we introduce UI2App, the first benchmark targeting interaction inference, the ability to recover application behavior from screenshots alone, without any textual or behavioral guidance. UI2App comprises 327 screenshots grouped into 45 state-coherent screenshot sets for runnable multi-route web applications. We design an end-to-end pipeline that evaluates each artifact along four dimensions: executability, navigation reachability, visual fidelity, and interaction inference. The interaction metric (IIS) assesses inferred interactions by functional correctness and state-management complexity, crediting any valid implementation rather than matching a single reference. Experiments on six frontier vision-language models reveal a marked capability mismatch between visual reconstruction and interaction realization: the visual-fidelity leader scores only 7.5 on IIS, ranking fourth and trailing the IIS leader by 5.2x. High-complexity interactions such as cross-page state remain a pervasive bottleneck, with half of the evaluated models scoring exactly zero on this dimension. Overall, the results indicate that inferring complete interaction behavior from static screenshots remains a key challenge for models.
