Search papers, labs, and topics across Lattice.
This paper introduces PLSQLBench, the first benchmark specifically designed to evaluate LLMs' ability to generate executable PL/SQL programs through execution-based correctness testing. The benchmark comprises 2,865 instances, including both single-turn and multi-turn tasks, and addresses the underexplored area of procedural database programming, revealing significant challenges in schema grounding, dialect fidelity, and procedural control flow. Experiments with eight LLMs show that while tool-augmented agents can enhance performance, substantial gaps in capability persist, underscoring the limitations of current evaluations in this domain.
LLMs struggle with procedural database programming, revealing critical gaps in their capabilities that traditional benchmarks overlook.
We present PLSQLBench, to our knowledge the first benchmark for evaluating whether LLMs can write executable PL/SQL programs, with correctness measured through execution-based tests. Existing LLM evaluations largely target general-purpose code generation or declarative text-to-SQL, leaving procedural database programming underexplored. PLSQLBench contains 2,865 instances: 2,594 single-turn tasks and 271 multi-turn conversations spanning 978 turns. The benchmark combines complex schema-grounded tasks over enterprise-style Spider 2 databases, simpler schema-grounded tasks derived from Spider, and MBPP-derived procedural problems, covering varying levels of database grounding and procedural complexity. Experiments with eight LLMs reveal recurring difficulties in schema grounding, PL/SQL dialect fidelity, procedural control flow, exception handling, and cross-turn consistency. Tool-augmented LLM agents improve performance on several schema-grounded evaluations, although substantial gaps remain. These results highlight procedural database programming capabilities not directly assessed by conventional code generation or text-to-SQL benchmarks. Our code is available at https://github.com/oracle-samples/plsqlbench.
