Search papers, labs, and topics across Lattice.
This paper introduces VideoVIBE, a benchmark that utilizes video-grounded diagnostics to assess the quality of one-shot generated interactive web applications, addressing the shortcomings of existing evaluation methods that focus on isolated artifacts. By analyzing approximately 1.7K diagnostic Video QA instances derived from 6,338 verified failures, the authors categorize failures into semantic-logical, visual-motion, structural-temporal, and functional domains. The proposed V2Lens system enhances the evaluation process by refining initial diagnoses through targeted verification, achieving a notable performance improvement over existing models, with V2Lens scoring 71.72 compared to the best standalone model at 64.54.
Video-grounded diagnostics reveal a staggering array of failure types in one-shot web application generation, pushing the boundaries of evaluation beyond mere artifact scoring.
Natural-language-driven "vibe coding" enables the one-shot generation of visually rich and interactive web applications, yet reliable assessment of their quality has not kept pace. Existing evaluations often score isolated artifacts or final task outcomes, offering limited evidence about which failures occur and why. We introduce VideoVIBE, a video-grounded benchmark that transforms human-operated webpage recordings into fine-grained diagnostic tasks. It contains approximately 1.7K diagnostic Video QA instances derived from 6,338 verified failures across generated webpages, spanning semantic-logical, visual-motion, structural-temporal, and functional failures. Diagnoses are grounded primarily in recorded presentation and behavior, with webpage source code used as complementary context. We further propose V2Lens, a training-free, evidence-grounded multi-agent system that challenges and selectively refines initial video-based diagnoses through targeted visual and source-code verification. Across thirteen closed-source and open-weight Video MLLMs, Gemini-2.5-Flash is the strongest standalone model with a score of 64.54, while V2Lens reaches 71.72, an improvement of 7.18 points. Together, our results show that video-grounded evaluation can move beyond isolated artifacts and aggregate outcomes toward a behaviorally faithful and diagnostically informative account of generated application quality.
