Search papers, labs, and topics across Lattice.
This study introduces a systematic experimental design framework for evaluating the autonomous model discovery behavior of stochastic coding agents like Codex and Claude Code. By analyzing various controlled factors such as reasoning effort, task type, and optimization metrics, the authors quantify variability in performance outcomes including output quality and cost. Key findings reveal a nuanced relationship between reasoning effort and performance-cost utility, providing insights into optimizing agent behavior in practical applications.
The framework reveals that optimizing reasoning effort can significantly impact both the cost and complexity of model discovery, challenging conventional benchmarks.
Large language model coding agents increasingly perform open-ended data modeling and analysis. These agents are stochastic and adaptive, and therefore their autonomous model discovery behavior cannot be adequately characterized by a single benchmark run. In this work, we propose an experimental design and analysis framework for systematically evaluating this discovery process, quantifying its variability, and identifying important factors. The proposed framework treats these agents as stochastic model-discovery operators, which map task-specific discovery data and an optimization target to a fitted model. Specifically, we investigate two such operators, Codex and Claude Code, under controlled experimental factors including agent's reasoning effort, task, optimization metric, and composition of training data. For each agent-task-metric combination, regression models and inference are conducted for multiple responses such as output quality, dollar cost, wall-clock time, and process complexity. Furthermore, we develop a utility-aligned canonical decomposition to characterize the dominant direction of the reasoning-effort effect and to assess whether that direction aligns with a performance-cost utility direction. The proposed framework is demonstrated on a testbed of networked word-forming games with insightful findings on reasoning effort with respect to cost and process complexity.
