Search papers, labs, and topics across Lattice.
This paper introduces DataClawEval, the first benchmark specifically designed to evaluate the end-to-end capabilities of autonomous agents in real-world data engineering tasks, addressing a significant gap in existing evaluation frameworks that focus on simplified scenarios. The benchmark comprises 100 complex tasks derived from production-grade code and spans five execution engines, providing a rigorous testing ground for LLM-based agents. Evaluation of 16 leading agents reveals that the highest-performing model achieves only 74.9, highlighting the persistent challenges and domain specialization in autonomous data engineering.
Autonomous data engineering agents struggle to achieve proficiency, with the best model scoring just 74.9 on a benchmark designed for real-world scenarios.
Large language models (LLMs) and LLM-based agents are increasingly being deployed to automate complex workflows, promising to revolutionize data management and processing. However, existing benchmarks predominantly focus on simplified Text-to-SQL translation or data analysis, leaving the critical and complex domain of end-to-end data engineering largely unexplored. To bridge this gap, we introduce DataClawEval, the first comprehensive benchmark designed specifically to evaluate the end-to-end task completion capabilities of autonomous agents in real-world data engineering scenarios. Built upon production-grade code authored by professional enterprise data engineers, it comprises 100 rigorous, end-to-end tasks spanning five execution engines: PySpark, MySQL, HiveSQL, PrestoSQL/Trino, and FlinkSQL. Rather than non-deterministic LLM-as-a-judge scoring, each task is executed within a case-specific, isolated sandbox and graded by deterministic, rule-based scripts. Evaluating 16 frontier agents exposes critical limitations: The strongest model attains only 74.9 overall, and no single model dominates, as each excels on a different engine, revealing strict domain specialization rather than omnipotent proficiency. Thus, autonomous data engineering remains a formidable, unresolved challenge. We release our dataset, containerized environments, and deterministic evaluation scripts at https://github.com/Dicemy/DataClawEval/tree/master
