Search papers, labs, and topics across Lattice.
This study investigates the performance of LLM-based coding agents, specifically Claude Code and Codex, in social science analysis, focusing on their methodological diversity and interpretive biases. By executing 20 independent analyses on immigration and social policy, the authors found that while Codex matches human methodological diversity, Claude Code significantly exceeds it, with both agents aligning closely with human consensus on effect estimates. Notably, the agents' methodological choices are influenced by prompts, revealing that while they can produce diverse methodologies, they remain susceptible to biases in interpretation rather than estimation.
AI coding agents can achieve or surpass human methodological diversity while exposing vulnerabilities in interpretive bias that human analysts do not share.
The deployment of LLM-based agents in scientific analysis raises opposing concerns: that agents may reduce methodological diversity, or that they may amplify the analytic flexibility through which researchers reach motivated conclusions. We argue these worries target two empirically separable layers: a design layer of methodological choices, and a verdict layer in which a decision rule maps estimates to a substantive claim. We test both by running 20 independent executions of Claude Code and Codex on a prominent immigration and social-policy against a many-analysts human baseline. At the design layer, Codex matches human methodological diversity and Claude Code produces nearly three times as many specifications; both agents' effect estimates remain broadly aligned with the human consensus, and no agent model exactly matches any human model. A prompt-induced anti-immigration researcher prior reorganizes each agent's methodological decisions but, unlike for biased human analysts in the same data, does not shift aggregate estimates or final verdicts; nor do agents reroute along the methodological axes humans use to bias their estimates. At the verdict layer, an explicit confirmatory prompt flips Claude Code's verdicts from 10% to 90% support while leaving its coefficient distribution essentially unchanged, operating through rule omission rather than rule softening. AI agents can rival or exceed human methodological diversity at the design layer while remaining vulnerable at the verdict layer. In our setting, the locus of AI bias is not estimation but interpretation.
