Search papers, labs, and topics across Lattice.
This paper systematically evaluates reject inference methods in credit scoring, revealing that improvements in model accuracy can mask deteriorating rejection quality due to survival bias. The authors introduce a controlled exploration strategy that allows lenders to approve a small fraction of rejected applicants to better assess model performance. Their findings indicate that traditional evaluation metrics can be misleading, as accuracy and rejection quality often provide conflicting recommendations regarding the need for exploration.
Models may appear to improve while actually worsening their ability to screen out defaulters, revealing a critical flaw in standard evaluation practices.
Reject inference methods are widely used to mitigate survival bias in credit scoring, yet their effectiveness remains poorly understood. We systematically evaluate several such methods and uncover a structural failure mode: in a natural retraining cycle, models whose accuracy improves while recall collapses create an illusion of improvement that leads practitioners to believe the system is getting better when, in fact, its rejection quality -- the ability to correctly screen out defaulters -- is deteriorating. We then propose a controlled exploration strategy that breaks the feedback loop without statistical assumptions: the lender deliberately approves a fraction of rejected applicants and observes their true outcomes. We show that accuracy and rejection quality give opposite recommendations on whether to explore: accuracy favors no exploration, while rejection quality improves with it, confirming that standard evaluation metrics are misleading under selection bias. Even minimal exploration rates (2--5\%) prove sufficient in our experiments to diagnose the severity of the feedback loop at near-zero cost. Our findings are consistent across two machine learning methods and three real-world datasets, and suggest that standard evaluation protocols are inadequate for assessing models trained under survival bias.
