Search papers, labs, and topics across Lattice.
This paper establishes a theoretical foundation for interpreting conformal prediction set size as a measure of information gain by introducing a decision-theoretic generalization of entropy. The authors derive a family of generalized information measures and show that Shannon mutual information admits an exact integral representation in terms of these measures. They then prove that in classification, the reduction in conformal set size from additional information is bounded by calibration-dependent measures and obeys a data processing inequality, formally linking set-size reduction to information gain.
Conformal prediction set size reduction is formally justified as an information gain metric, bridging uncertainty quantification with classical information theory.
Conformal prediction is a popular tool for uncertainty quantification that outputs prediction sets with finite-sample coverage guarantees. While prediction set size is commonly used as a heuristic measure of uncertainty, the information-theoretic basis for this interpretation remains poorly understood. In this work, we provide such a foundation using a decision-theoretic generalization of entropy tailored to set-valued prediction. In particular, we introduce a family of generalized information measures based on the size and coverage of conformal prediction sets. Notably, Shannon mutual information admits an exact integral representation in terms of these measures. We then show that, in standard classification settings, the reduction in conformal set size from additional information (i) is sandwiched between calibration-dependent members of this family and (ii) obeys a data processing inequality, both up to finite-sample calibration and model error terms. Together, our results formally relate conformal prediction to classical information-theoretic quantities and justify using set-size reduction as an information gain metric. Empirically, we validate our theory across 11 classification settings and show that set-size reduction and Shannon mutual information can rank features differently in a greedy feature selection experiment.
