Search papers, labs, and topics across Lattice.
The paper introduces WebCompass, a multimodal benchmark for evaluating code language models (CLMs) in end-to-end web coding across generation, editing, and repair tasks using text, image, and video inputs. It features a novel Agent-as-a-Judge paradigm that autonomously executes generated websites in a browser and synthesizes test cases to evaluate interactive behaviors. Experiments with closed-source and open-source models reveal that closed-source models perform better overall, editing and repair tasks have distinct difficulty profiles, and aesthetics remains a bottleneck, especially for open-source models.
Evaluating web coding LLMs with real-world fidelity reveals that even state-of-the-art models still struggle with aesthetics and framework-specific nuances.
Large language models are rapidly evolving into interactive coding agents capable of end-to-end web coding, yet existing benchmarks evaluate only narrow slices of this capability, typically text-conditioned generation with static-correctness metrics, leaving visual fidelity, interaction quality, and codebase-level reasoning largely unmeasured. We introduce WebCompass, a multimodal benchmark that provides unified lifecycle evaluation of web engineering capability. Recognizing that real-world web coding is an iterative cycle of generation, editing, and repair, WebCompass spans three input modalities (text, image, video) and three task types (generation, editing, repair), yielding seven task categories that mirror professional workflows. Through a multi-stage, human-in-the-loop pipeline, we curate instances covering 15 generation domains, 16 editing operation types, and 11 repair defect types, each annotated at Easy/Medium/Hard levels. For evaluation, we adopt a checklist-guided LLM-as-a-Judge protocol for editing and repair, and propose a novel Agent-as-a-Judge paradigm for generation that autonomously executes generated websites in a real browser, explores interactive behaviors via the Model Context Protocol (MCP), and iteratively synthesizes targeted test cases, closely approximating human acceptance testing. We evaluate representative closed-source and open-source models and observe that: (1) closed-source models remain substantially stronger and more balanced; (2) editing and repair exhibit distinct difficulty profiles, with repair preserving interactivity better but remaining execution-challenging; (3) aesthetics is the most persistent bottleneck, especially for open-source models; and (4) framework choice materially affects outcomes, with Vue consistently challenging while React and Vanilla/HTML perform more strongly depending on task type.
