Search papers, labs, and topics across Lattice.
To determine whether API benchmarks accurately reflect user-facing deployments, the authors audited ChatGPT, Claude, and Gemini across seven systems and nine benchmarks evaluating capabilities, social bias, and sycophancy. API evaluations systematically outperform web interfaces by an average of 3.4 percentage points in accuracy and 2.1 percentage points in test-retest consistency, with the API-to-interface drop on ChatGPT exceeding the generational gap between GPT 5.3 and GPT 5.4. Furthermore, varying API system prompts, sampling hyperparameters, and reasoning configurations shifts model behavior but fails to reliably eliminate this context-validity gap.
Switching an LLM from an API to a consumer chat interface can degrade performance more than downgrading an entire model generation鈥攁nd standard API hyperparameter controls cannot reliably bridge the gap.
Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy. Yet, a key assumption underlying benchmark scores is that the model performance measured through APIs faithfully reflects the behavior of deployed systems. We challenge this assumption by auditing ChatGPT, Claude, and Gemini across seven systems and nine benchmarks spanning general capability, social bias, and sycophancy. We find systematic API--interface differences in both accuracy and consistency. On average, API evaluations score 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test--retest agreement than corresponding interface evaluations. For ChatGPT, the performance difference between API and interface access exceeds the API-only difference between GPT 5.3 and GPT 5.4. Put differently, switching access surfaces can degrade performance as much as downgrading a full model generation. We further test whether exposed API controls can reproduce interface behavior by varying system prompts, sampling parameters, and reasoning settings. These controls shift behavior in some cases but do not reliably eliminate the gap. Our findings document a context-validity gap: measurements obtained through APIs do not necessarily generalize to corresponding deployed interfaces, complicating the use of API evaluations as proxies for deployed systems.
Stanford HAI