Search papers, labs, and topics across Lattice.
This paper addresses the limitations of standard marginal conformal prediction (CP) in high-stakes decision systems, particularly its inadequate coverage of rare minority classes under severe class imbalance. By benchmarking marginal CP against class-conditional (Mondrian) CP and cost-controlled abstention across 15 real-world datasets, the authors demonstrate that Mondrian CP significantly improves minority-class coverage by an average of 61.7 percentage points. Additionally, the integration of Mondrian CP with cost-controlled abstention effectively reduces expected decision costs, offering a practical framework for uncertainty quantification in critical applications like healthcare and fraud detection.
Mondrian CP can boost minority-class coverage by over 60% while cutting decision costs in high-stakes scenarios.
High-stakes decision systems in credit scoring, fraud detection, healthcare, and industrial safety require reliable uncertainty quantification under severe class imbalance and asymmetric error costs. Standard marginal conformal prediction (CP) provides valid overall coverage guarantees; however, we show that it severely under-covers rare, costly minority classes, with minority-class coverage dropping to as low as 0.5% on certain datasets. To characterize and address this limitation, we conduct a comprehensive benchmark comparing marginal CP, class-conditional (Mondrian) CP, and cost-controlled abstention mechanisms across 15 real-world imbalanced tabular datasets, 7 classification models, 3 probability calibration techniques, and 10 random seeds, resulting in 3,150 experimental runs. Our results show that Mondrian CP restores valid minority-class coverage, achieving an average minority-coverage improvement of 61.7 percentage points over marginal CP (p < 1e-80). Furthermore, combining Mondrian CP with cost-controlled abstention significantly reduces expected decision cost compared with standard decision boundaries, confidence-based rejectors, and risk-controlled rejectors under realistic human review budgets. We further quantify dataset-specific break-even thresholds at which deferring ambiguous instances to human experts becomes cost-effective. These findings provide practical guidance for deploying distribution-free, cost-aware uncertainty quantification in high-stakes decision support systems.
