Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces GRACE-DS, a novel evaluation framework designed to assess LLM-powered AutoML agents in data science workflows. By simulating realistic stages of machine learning tasks and incorporating hidden validators, GRACE-DS evaluates not only predictive performance but also critical aspects like leakage avoidance and reward alignment. The study finds that a flexible iterative interaction approach significantly outperforms traditional methods, establishing GRACE-DS as a vital tool for pre-deployment evaluation of AutoML agents in organizational contexts.
Flexible iterative interactions in GRACE-DS boost LLM-powered AutoML agents' performance, revealing critical insights into their operational capabilities.
We introduce GRACE-DS, a Guarded Reward-guided Agent Correction Environment in Data Science for pre-deployment evaluation of LLM-powered AutoML agents. GRACE-DS is a set of evaluation metrics in an isolated environment that can be applied to tabular ML tasks specific to a particular organization. It exposes agents to realistic workflow stages, from planning and data inspection through feature engineering, model development, validation, and code repair to final submission, while hidden executable validators measure not only final predictive performance but also leakage avoidance, reproducibility, protocol validity, correction behavior, and reward alignment. The strongest structured regime, flexible iterative interaction (our approach), achieves higher end-to-end normalized hidden-test quality than single-shot generation, unstructured interaction, and restart-based baselines, while also improving protocol-valid completion. Validated across more than 7,000 episodes, these results establish GRACE-DS as a robust platform for assessing the capacity of LLM-based AutoML agents to execute machine learning workflows under production-like conditions and in accordance with organization-specific requirements.