Search papers, labs, and topics across Lattice.
This study investigates the capabilities of a large language model acting as an autonomous researcher in a long-horizon neural architecture design task, where it autonomously proposes, implements, and evaluates experiments over time. The agent successfully enhances a Vision Transformer model from a weak baseline to a more efficient version, achieving notable performance on small benchmarks and a usable model on ImageNet-1K. Key findings reveal that productivity follows a distinct phase structure influenced by workflow design, with the agent's hypothesis generation and evaluation methods significantly impacting its research outcomes.
A language model can autonomously navigate complex neural architecture design, achieving significant performance improvements while revealing the critical role of workflow design in research productivity.
We study what happens when a single general-purpose large language model acts as the sole researcher on a long-horizon neural architecture design problem. The agent receives a scientific question, an initial hypothesis and motivation, a compute budget, and research affordances (source and experiment management, experiment tracking, literature access, and persistent memory), then autonomously proposes, implements, evaluates, and records experiments over an extended period. The study comprises three phases, separated by human-declared transitions, that progressively expand the agent's tool surface or problem scale. Across approximately 100 sequential experiments, the agent improves a non-standard Vision Transformer from a weak baseline to a stronger, efficient model on small benchmarks and a usable but sub-SOTA model on ImageNet-1K, while producing a dense behavioural trace. We report four findings.(i)Productivity exhibits a clear phase structure: rapid early gains, a multi-dozen-hypothesis saturation wall, and recovery, with recovery triggered by expanding the action surface rather than changing the underlying model.(ii)A single early hypothesis contributes more to accuracy gain, with later improvements long-tailed.(iii)The preference for greedy, incremental hypotheses is largely workflow-induced: a commit-or-discard evaluation rule is isomorphic to greedy hill-climbing; the remainder reflects risk aversion after bold failures and anchoring on familiar literature. (iv)The agent independently rediscovers established results and, in the unfamiliar regime of pure channel attention, overturns a standard design choice. We conclude that workflow design was at least as influential as agent capability in this study and propose diversified search, budgeted moonshot hypotheses, explicit forks, and regime-aware re-validation as testable directions for future autonomous research.
