Search papers, labs, and topics across Lattice.
This paper introduces OptimismBench, a novel framework for detecting directional bias in large language models (LLMs) by comparing probability judgments of success and failure in inverted pairs. The study reveals that fourteen out of sixteen evaluated models exhibit an optimistic bias, with only Anthropic's frontier tier showing signs of pessimism, indicating a systematic tendency in LLMs to overestimate positive outcomes. The findings underscore the importance of recognizing and addressing these biases, as they can significantly influence downstream decision-making processes in various applications.
Fourteen out of sixteen large language models exhibit a systematic optimism bias in their probability judgments, raising concerns about their reliability as decision aids.
Large language models are increasingly used as decision aids whose probability judgments shape downstream choices. Whether those judgments carry a systematic directional tilt has been hard to detect: calibration metrics aggregate unsigned errors, and naturalistic uncertainty offers no ground-truth probability. When an LLM rates a startup's success at 70% but its failure at 15%, the missing 15 points expose a distortion no aggregate score flags. We introduce OptimismBench, which detects directional bias with inverted pairs: each scenario elicits both P(success) and P(failure), and asymmetry between the two framings yields a signed bias score without ground truth. Across 16 models from 8 providers, fourteen are optimistic; pessimism appears only in Anthropic's frontier tier. Eleven matched base-versus-chat pairs across four families show post-training sets the sign of the bias, with opposite shifts in different families. The pattern survives prompt, temperature, perspective, and self-debiasing ablations. A seventeen-model six-language comparison further shows model identity dominates language, with inter-model variance at 4.7x inter-language variance. We release 3,870 items across 10 languages for per-model directional-bias auditing. When alignment makes a model more helpful, it also tilts its probabilities; downstream pipelines inherit the tilt by default.
