Search papers, labs, and topics across Lattice.
This paper applies Signal Detection Theory (SDT) to analyze LLM calibration, decomposing it into sensitivity (discrimination ability) and bias (confidence tendency). The study treats LLMs as signal detectors in a factual discrimination task, examining whether temperature acts as a criterion shift similar to payoff manipulations in human psychophysics. Results show that temperature simultaneously increases sensitivity and shifts criterion, and that LLMs exhibit unequal-variance evidence distributions, with instruct models showing more asymmetry than base models.
Temperature scaling in LLMs isn't just a confidence knob; it unexpectedly boosts factual discrimination ability while shifting the decision threshold.
Large language models (LLMs) are evaluated for calibration using metrics such as Expected Calibration Error that conflate two distinct components: the model's ability to discriminate correct from incorrect answers (sensitivity) and its tendency toward confident or cautious responding (bias). Signal Detection Theory (SDT) decomposes these components. While SDT-derived metrics such as AUROC are increasingly used, the full parametric framework - unequal-variance model fitting, criterion estimation, z-ROC analysis - has not been applied to LLMs as signal detectors. In this pre-registered study, we treat three LLMs as observers performing factual discrimination across 168,000 trials and test whether temperature functions as a criterion shift analogous to payoff manipulations in human psychophysics. Critically, this analogy may break down because temperature changes the generated answer itself, not only the confidence assigned to it. Our results confirm the breakdown with temperature simultaneously increasing sensitivity (AUC) and shifting criterion. All models exhibited unequal-variance evidence distributions (z-ROC slopes 0.52-0.84), with instruct models showing more extreme asymmetry (0.52-0.63) than the base model (0.77-0.87) or human recognition memory (~0.80). The SDT decomposition revealed that models occupying distinct positions in sensitivity-bias space could not be distinguished by calibration metrics alone, demonstrating that the full parametric framework provides diagnostic information unavailable from existing metrics.
