Search papers, labs, and topics across Lattice.
This paper introduces an automated adversarial collaboration framework that uses LLMs, program synthesis, and information-theoretic experimental design to adjudicate between competing cognitive theories. The framework iteratively generates theory agents, synthesizes experiments, and selects the most informative experiment to distinguish between theories. In a simulation study across three categorization theories, the framework successfully recovered the ground-truth theory, demonstrating its potential for in-silico theory adjudication.
LLMs can now automatically design and execute experiments to resolve debates between cognitive science theories, even discovering the models and experiments themselves.
Cognitive science often evaluates theories through narrow paradigms and local model comparisons, limiting the integration of evidence across tasks and realizations. We introduce an automated adversarial collaboration framework for adjudicating among competing theories even when the candidate models and experiments must be discovered during the adjudication process. The system combines LLM-based theory agents, program synthesis, and information-theoretic experimental design in a closed loop. In a simulation study spanning three classic categorization theories, the framework recovered the ground-truth theory across noise settings with weaker reliability in the hardest settings. Together, the framework and findings provide a concrete proof of concept for closed-loop, in-silico theory adjudication in cognitive science.
Stanford HAI