Search papers, labs, and topics across Lattice.
This paper introduces a benchmark of 40 end-to-end data science projects with associated rubrics to evaluate the ability of generative AI models to complete comprehensive data science workflows. The authors built an automated grading pipeline to systematically assess AI-generated projects, revealing significant performance variations across models, particularly in tasks requiring judgment. The study indicates that while AI can handle structured data science tasks effectively, human verification remains crucial for tasks demanding subjective assessment.
Generative AI can approximate entry-level data scientists on routine tasks, but don't trust it to make complex judgments without human oversight.
Data science is an integrated workflow of technical, analytical, communication, and ethical skills, but current AI benchmarks focus mostly on constituent parts. We test whether AI models can generate end-to-end data science projects. To do this we create a benchmark of 40 end-to-end data science projects with associated rubric evaluations. We use these to build an automated grading pipeline that systematically evaluates the data science projects produced by generative AI models. We find the extent to which generative AI models can complete end-to-end data science projects varies considerably by model. Most recent models did well on structured tasks, but there were considerable differences on tasks that needed judgment. These findings suggest that while AI models could approximate entry-level data scientists on routine tasks, they require verification.
