Search papers, labs, and topics across Lattice.
This paper introduces EvoGenUI-Bench, a benchmark designed to evaluate the performance of large language models (LLMs) as multi-turn generative UI assistants through 150 five-turn tasks across various scenarios. The evaluation reveals that even the best-performing model achieves only 74.9% success in turn-level tasks while struggling significantly with episode-level completion and cross-turn retention, particularly in tool-grounded scenarios. The findings highlight critical failures in information architecture and state management, suggesting that traditional evaluation metrics may overlook essential aspects of interface behavior in dynamic contexts.
LLMs falter in maintaining coherent multi-turn interactions, with only 37.3% successfully completing five-turn episodes, revealing a major gap in generative UI capabilities.
Large language models can generate interactive web interfaces, but reliable generative UI requires maintaining an executable artifact as user requests evolve. We introduce EvoGenUI-Bench, a benchmark for multi-turn interface maintenance comprising 150 five-turn tasks and 750 turns across three scenarios: information presentation, executable interaction, and tool-grounded external state. We execute generated artifacts in a browser and evaluate them using screenshots, source and DOM evidence, actor traces, and runtime logs. Beyond turn-level and episode-level success, we measure cross-turn retention with Adjacent Pass Retention. Across eight models, even the strongest achieves 74.9% Turn Pass while completing only 37.3% of five-turn episodes; APR further falls to 52.4% on tool-grounded tasks. Diagnostic analysis shows that presentation failures center on information architecture, interaction failures on derived-state propagation and affordance binding, and tool-grounded failures additionally involve external-state grounding and requirement decomposition. These results reframe generative UI evaluation from judging isolated outputs to testing whether interface behavior, derived state, external state, and assistant claims remain synchronized as the artifact evolves.
