Search papers, labs, and topics across Lattice.
This paper provides a theoretical analysis of temperature scaling, a widely used technique for controlling uncertainty in probabilistic models, particularly in classifiers and large language models (LLMs). The authors demonstrate that increasing temperature increases uncertainty (specifically entropy) in classifiers. However, they challenge the common belief that increasing temperature necessarily increases diversity in LLMs. Finally, they characterize temperature scaling as an information projection onto models with a given entropy and show it is the only linear scaler that preserves hard predictions.
Temperature scaling, the go-to method for uncertainty control, is formally proven to increase classifier uncertainty, but its impact on LLM diversity is questioned, revealing nuances in its application.
Temperature scaling is a simple method that allows to control the uncertainty of probabilistic models. It is mostly used in two contexts: improving the calibration of classifiers and tuning the stochasticity of large language models (LLMs). In both cases, temperature scaling is the most popular method for the job. Despite its popularity, a rigorous theoretical analysis of the properties of temperature scaling has remained elusive. We investigate here some of these properties. For classification, we show that increasing the temperature increases the uncertainty in the model in a very general sense (and in particular increases its entropy). However, for LLMs, we challenge the common claim that increasing temperature increases diversity. Furthermore, we introduce two new characterisations of temperature scaling. The first one is geometric: the tempered model is shown to be the information projection of the original model onto the set of models with a given entropy. The second characterisation clarifies the role of temperature scaling as a submodel of more general linear scalers such as matrix scaling and Dirichlet calibration: we show that temperature scaling is the only linear scaler that does not change the hard predictions of the model.
