Search papers, labs, and topics across Lattice.
This paper introduces DBLifeBench, a comprehensive benchmark for evaluating Large Language Models (LLMs) across all phases of the database lifecycle, including Design, Implementation, Operation, Debugging, and Maintenance. The authors highlight a significant limitation in existing benchmarks that focus primarily on Text-to-SQL tasks, which do not adequately assess the diverse capabilities necessary for effective database management. Their findings reveal that while general-purpose models perform consistently across phases, specialized Text-to-SQL models exhibit "catastrophic forgetting" in non-coding tasks, underscoring the need for a more holistic evaluation framework.
Specialized Text-to-SQL models forget critical database management skills, while general-purpose LLMs maintain balanced performance across the entire database lifecycle.
Large Language Models (LLMs) are transforming database interaction paradigms, evolving from simple query translators to autonomous database administrators (DBAs). However, current evaluation benchmarks remain disproportionately fixated on Text-to-SQL tasks, neglecting the holistic Database Lifecycle-from initial schema design to post-deployment maintenance. This narrow focus fails to capture the diverse capabilities required for real-world database management. To bridge this gap, we introduce DBLifeBench, the first benchmark to evaluate LLMs across five critical lifecycle phases: Design, Implementation, Operation, Debugging, and Maintenance. Furthermore, addressing the cognitive mismatch between ambiguous natural language and complex SQL logic, we propose Progressive-Text2SQL, a novel task utilizing structured reasoning graphs to mimic human iterative problem-solving. Our extensive evaluation reveals a critical insight: while general-purpose models demonstrate balanced performance, specialized Text-to-SQL models suffer from ``catastrophic forgetting'' in non-coding phases like design and maintenance. DBLifeBench serves as a foundational step toward evaluating and building true full-stack database intelligence.
