Search papers, labs, and topics across Lattice.
This study investigates cross-environment compatibility in webpages generated by Multimodal Large Language Models (MLLMs), addressing a critical gap in existing evaluations that focus solely on visual fidelity. By constructing the WebCompat dataset, which includes 2,032 annotated instances of webpages rendered across nine browser-device combinations, the authors reveal that a staggering 68% of these pages suffer from compatibility issues, primarily affecting layout integrity. The research culminates in the development of XCompat, an advanced compatibility issue detector that significantly outperforms existing tools, achieving an F1 score of 0.903, thus highlighting the urgent need for improved reliability in MLLM-generated web content.
A staggering 68% of MLLM-generated webpages fail to render correctly across different environments, raising serious concerns about their reliability in real-world applications.
Multimodal Large Language Models (MLLMs) have been increasingly adopted to automate webpage generation from visual designs (e.g., screenshots). However, existing evaluations are limited to visual fidelity assessment under a fixed browser-device configuration. Such a setting overlooks the cross-environment rendering compatibility for real-world deployments. To address this gap, we present the first systematic empirical study of cross-environment compatibility in AI-generated webpages. Specifically, we construct WebCompat, a dataset of 2,032 annotated instances, comprising webpages generated by 8 representative AI tools, each rendered across 9 browser-and-device combinations. We analyze the prevalence of compatibility issues, their user-perceptible symptoms, and underlying code-level root causes. Our findings reveal that 68% of generated webpages exhibit at least one compatibility issue, underscoring the pervasive reliability concerns surrounding MLLM-generated front-end artifacts. The most prevalent symptoms are failures that disrupt the entire page layout (88.3%): pages shrink directly to fit the target screen with too small fonts, or exhibit scale mismatches that produce cut-off content. Failures localized to individual elements, such as image distortion or missing components, are comparatively less common (13.4%). Furthermore, although most MLLMs incorporate responsive design patterns into the generation, they fail to properly implement these codes. Guided by the findings, we develop XCompat, a lightweight offline compatibility issue detector that combines visual screenshots and the structural DOM tree for analysis. It achieves an F1 score of 0.903 on the WebCompat-test, outperforming the existing compatibility checking tools and LLM baselines. All datasets and tools are released to support future research on rendering reliability in MLLM-based front-end code generation.
