Search papers, labs, and topics across Lattice.
This study introduces UXBench, a comprehensive multimodal benchmark featuring 2,000 visual question-answering (VQA) samples aimed at evaluating the reasoning capabilities of multimodal large language models (MLLMs) in user interface contexts. The evaluation reveals that current MLLMs exhibit significant limitations in UI-based reasoning, prompting the development of UI-UX, an enhanced MLLM that employs a novel reward routing mechanism and an asymmetric transition reward to improve performance. The results demonstrate that UI-UX achieves state-of-the-art accuracy on UXBench, outperforming existing models while maintaining low inference latency, highlighting the potential for improved user experience assessments in real-world applications.
Current multimodal LLMs struggle with UI-based reasoning, but the new UI-UX model achieves a remarkable 0.7963 accuracy on the UXBench benchmark, setting a new standard.
User experience (UX) centered on usability, perceived consistency, and functional clarity is fundamental to real-world user interfaces (UI). The application of multimodal large language models (MLLMs) in the field of user interfaces is evolving rapidly, such as visual element grounding, graphical user interface (GUI) agents, and design-to-code generation. However, research efforts on evaluating UX based on UI screenshots are still immature. To address this, we propose UXBench, a novel multimodal benchmark consisting of 2,000 VQA data samples designed to assess MLLMs'ability to perform UI-based reasoning. UXBench includes 8 tasks based on real-world UI screenshots that require fine-grained diagnosis of UX issues across layout relationships, visual hierarchy, and content consistency. Our extensive evaluation of mainstream MLLMs shows that they remain fundamentally limited in their capacity for UI-based reasoning. The results underscore the need for further advancements in this area. To bridge this gap, we propose UI-UX, an MLLM based on Qwen3-VL-4B-Thinking foundation model and enhanced via reinforcement learning with two key innovations: a reward routing mechanism that dynamically balances perceptual understanding and logical reasoning during inference, and an asymmetric transition reward that suppresses redundant or insufficient reasoning steps. Experiments demonstrate that UI-UX achieves state-of-the-art (SOTA) performance on UXBench, attaining an accuracy of 0.7963 -- surpassing Claude-4.5-Sonnet's 0.6550 -- while exhibiting strong generalization across diverse UI tasks and maintaining low inference latency.
