Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of LLM-generated skills in enhancing the performance of AI data scientists across four key data science workflows: data preparation, extraction, statistical analysis, and reporting. By conducting a comprehensive ablation study involving 56 tasks and 7,560 runs, the authors find that neither the full generated skills nor any of their components provide a significant performance boost over simple task prompting. The results suggest that relying on LLM-generated skills as a default strategy may not yield the expected improvements in data science tasks, highlighting a potential limitation in their utility.
LLM-generated skills fail to outperform basic task prompts in data science workflows, challenging the assumption that automated skill generation enhances AI performance.
Product data scientists often ask LLM-based agents to help with recurring execution tasks such as cleaning data, writing SQL, choosing statistical tests, and formatting results. Reusable skill files are meant to avoid prompting from scratch by packaging guidance for a task family. Expert-written skills can encode high-quality guidance, but writing and maintaining them across many data-science task families creates a manual bottleneck. We ask whether LLM-generated skills offer a useful low-curation alternative: do they improve performance over the task prompt alone? We test this question across four lifecycle stages: data preparation, data extraction, statistical analysis, and reporting, using one generated skill per stage. We find no reliable improvement from full generated skills over No-Skill prompting. We then ask whether any part of the skill is useful by ablating different skill components. The main ablation covers 56 tasks, nine model configurations, and three providers, yielding 7,560 runs. Compared with prompting using the task alone, neither the full generated skill nor any ablated skill variant significantly improves performance; all p-values are at least 0.396, and the total spread across variants is only 1.2 pp. A supplemental token-matched control adds 1,512 runs and finds that Full skills perform similarly to task-irrelevant skill-formatted content. The results caution against using one LLM-generated skill per data-science workflow as a default single-shot prompting strategy.
