Search papers, labs, and topics across Lattice.
MMBench-Live introduces a dynamic benchmarking framework for vision-language models (VLMs) that evolves continuously through a multi-agent-driven automated pipeline. This approach mitigates issues of temporal staleness and data contamination by employing a task-guided dataset construction method, which includes real-time data acquisition and verifiable QA generation. The benchmark has been instantiated with 5.9K new evaluation instances, demonstrating stable model rankings and reduced memorization signals, indicating a scalable solution for sustainable multimodal evaluation.
MMBench-Live achieves a high answer correctness rate while updating benchmarks at a fraction of the cost and time, revolutionizing how we assess VLMs.
Evaluation benchmarks are essential for assessing vision-language models (VLMs), but most multimodal benchmarks are static, making them vulnerable to temporal staleness, data contamination, and costly maintenance. We present MMBench-Live, a continuously evolving multimodal benchmark built by a multi-agent-driven automated pipeline. Our framework treats benchmark evolution as task-guided dataset construction, integrating structured benchmark specification, feedback-controlled real-time data acquisition, and verifiable QA generation with executable reasoning. To maintain cross-version comparability, we introduce a distribution-consistent update strategy that extracts task-related visual patterns from the original benchmark to guide data collection and filtering. Instantiated from MMBench, MMBench-Live contains 5.9K newly generated evaluation instances with a high answer correctness rate, while each update costs about USD 30 and takes 1-2 hours. Extensive evaluations show that MMBench-Live preserves stable model rankings, maintains semantic alignment with the original benchmark, and exhibits weaker contamination-related memorization signals, suggesting a practical and scalable paradigm for sustainable multimodal benchmark evolution. The project is available at https://github.com/PRIS-CV/MMBench-Live.
