Search papers, labs, and topics across Lattice.
This study investigates operationalization failures in analytical workflows generated by large language models (LLMs) across various domains, identifying 153 recurring issues despite successful execution. The analysis categorizes these failures into five classes: comparative grounding, process reasoning, quantitative reasoning, role confusion, and policy grounding, highlighting a significant semantic gap between user intentions and the capabilities of current workflow-generation systems. The findings underscore the need for enhanced semantic representations in agentic data systems to effectively align analytical intents with executable operations.
Despite successful workflow execution, LLMs fail to operationalize analytical intents 153 times across diverse domains, revealing critical semantic gaps.
Large language models (LLMs) are increasingly used to generate queries, invoke tools, and construct analytical workflows. Although recent advances have substantially improved workflow generation and execution, the semantic information required to operationalize analytical concepts often lies beyond what is explicitly represented in database schemas and data values. We present a cross-domain formative study of operationalization failures in agent-generated analytical workflows. Across 236 analytical intents spanning finance, human resources, and public safety domains, we identify 153 recurring failures despite successful workflow generation and execution. Our analysis reveals five recurring classes of failures: comparative grounding, process reasoning, quantitative reasoning, role confusion, and policy grounding. These findings suggest a semantic gap between user-level analytical concepts and the information available to workflow-generation systems. More broadly, they raise questions about the admissibility of analytical operations and suggest that future agentic data systems may require richer semantic representations to bridge the gap between analytical intent and executable computation.
