Search papers, labs, and topics across Lattice.
This study investigates the reliability of large language models (LLMs) in making confident predictions under conditions of hidden information, using a chess variant where the location of a royal piece is concealed. The results reveal a significant miscalibration in the models' confidence, with high-confidence predictions being correct only 1.6% of the time, indicating a severe disconnect between stated confidence and actual correctness. Furthermore, the findings suggest that traditional evaluation metrics do not correlate with belief quality, highlighting the potential pitfalls of relying solely on outcome-based assessments in agentic systems.
High-confidence predictions from LLMs in hidden information scenarios are alarmingly inaccurate, with only 1 in 62 correct, challenging the assumption that confidence reflects correctness.
Agentic systems increasingly gate actions on a model's own stated confidence, which assumes confidence tracks correctness at the moment of acting. We test this in a hidden-information chess variant where royal status can be secretly, repeatedly relocated between pieces, and where an agent's stated probability distribution over the opponent's hidden royal piece -- elicited every turn, separately from the move it chooses -- is scored against ground truth recoverable after the game. Across two independent batches, captures made at high stated confidence ($\geq 0.5$) about the hidden piece's location were correct in 1 of 62 cases. The calibration deficit is concentrated almost entirely in these events: 99.3% of it in the original batch, 98.7% in the replication. The same pattern, in weaker form, orders consistently (point estimates only; most pairwise gaps are not statistically distinguishable at this sample size) across four further model configurations spanning a second provider -- reported as scope for the finding, not as evidence that capability predicts calibration: a same-model comparison at a fixed external leaderboard score shows a deliberation-budget change alone moves the metric by nearly as much as a large cross-model gap. In a separate seat, conventional evaluation axes -- legality, cost, latency, completion rate -- can dissociate entirely from belief quality, with the configuration winning on every conventional axis producing the worst belief quality tested. A model exhibiting this pattern can still win the game its belief was about, which is why outcome-only evaluation would not detect it.
