Search papers, labs, and topics across Lattice.
This study analyzes the behavior of 400 human reviewers over a seven-month period to assess how scrutiny of AI-generated code changes with experience. The findings reveal a significant increase in approval rates for AI-generated pull requests, rising from 30.1% to 36.8%, indicating a trend of habituation where reviewers become less critical over time despite increased review latency and reduced inline comments. This suggests that as the workload increases, reviewers may reflexively lower their scrutiny rather than calibrate their trust in the AI agents, raising concerns about the quality assurance of AI-generated code.
Reviewers approve AI-generated code more often while actually engaging less, revealing a troubling trend of habituation that could compromise code quality.
As AI coding agents (e.g., GitHub Copilot, Devin, OpenAI Codex, Cursor) submit pull requests to open-source repositories at scale, a key question arises: do human reviewers gradually lower their scrutiny for AI-generated code over time? We conduct a longitudinal within-reviewer analysis using the AIDev dataset, studying 400 repeat reviewers who collectively submitted 11,429 reviews over a seven-month observation period. Comparing each reviewer's early and late review episodes, we observe a population-level shift in approval rate from 30.1% to 36.8% (Wilcoxon signed-rank p < 10^{-6} on paired shifts). Pooled by within-reviewer experience decile, the cumulative gap reaches +14.5 pp from first to tenth decile. This shift is experience-driven (persists after controlling for calendar time), agent-specific (human PR approval rates decline over the same period), and not explained by PR difficulty (median PR size is flat). However, review latency increases rather than decreases (+3.5x), while inline comment volume decreases (-22%, p=0.0014), suggesting reviewers spend more time in queue but less time actively inspecting code. The combination of rising approval, declining comment effort, and increasing queue time is most consistent with reflexive habituation under growing workload rather than rational trust calibration alone.
CMU ML