Search papers, labs, and topics across Lattice.
Vibe Code Bench is introduced as a new benchmark for evaluating AI models on end-to-end web application development, encompassing 100 application specifications and 964 browser-based workflows. Evaluating 16 frontier models, the benchmark reveals that the best model achieves only 58.0% accuracy, highlighting the challenge of reliable end-to-end application development. The study also identifies self-testing as a strong performance predictor and demonstrates the significant impact of evaluator selection on outcomes through a human alignment study.
Building a complete web application from scratch remains a surprisingly hard task for even the best AI models, with top performance at only 58% accuracy on a new end-to-end benchmark.
Code generation has emerged as one of AI's highest-impact use cases, yet existing benchmarks measure isolated tasks rather than the complete "zero-to-one" process of building a working application from scratch. We introduce Vibe Code Bench, a benchmark of 100 web application specifications (50 public validation, 50 held-out test) with 964 browser-based workflows comprising 10,131 substeps, evaluated against deployed applications by an autonomous browser agent. Across 16 frontier models, the best achieves only 58.0% accuracy on the test split, revealing that reliable end-to-end application development remains a frontier challenge. We identify self-testing during generation as a strong performance predictor (Pearson r=0.72), and show through a completed human alignment study that evaluator selection materially affects outcomes (31.8-93.6% pairwise step-level agreement). Our contributions include (1) a novel benchmark dataset and browser-based evaluation pipeline for end-to-end web application development, (2) a comprehensive evaluation of 16 frontier models with cost, latency, and error analysis, and (3) an evaluator alignment protocol with both cross-model and human annotation results.
MIT CSAIL