Search papers, labs, and topics across Lattice.
This study investigates the phenomenon of predictive multiplicity in recidivism risk assessment, revealing that multiple accurate models can yield varying predictions for the same individual, raising concerns about decision-making arbitrariness. By constructing a dataset from over 15 years of inmate release data, the authors develop interpretable models that enhance predictive performance and reduce error-rate disparities while ensuring rehabilitative progress is reflected in risk scores. The findings indicate that, contrary to theoretical expectations, similarly accurate models can show significant predictive agreement, and a straightforward policy of assigning the lowest risk score among models effectively mitigates predictive arbitrariness.
Predictive multiplicity in recidivism assessments reveals that multiple accurate models can lead to significantly different outcomes, challenging the reliability of decision-making in high-stakes contexts.
Prediction tasks over individual futures, which are inherently noisy, often admit multiple similarly accurate models. When these models produce different predictions for the same individual, they raise concerns of arbitrariness in decision-making. How severe can this arbitrariness be, in theory and in practice? How can it be resolved to support high-stakes risk assessment? We address these questions through a study of a machine learning-based decision support system for recidivism risk assessment that has been in use for over 15 years. By translating complex legal rules into an algorithm for labeling post release outcomes (recidivist or non-recidivist), we first construct a dataset of thousands of inmate releases. Using this dataset, we learn interpretable models that improve predictive performance, reduce error-rate disparities between groups, and ensure that rehabilitative progress lowers risk scores. Next, we study predictive multiplicity, by first deriving a tight lower bound on the expected predictive agreement of any finite set of models over a dataset, and then by evaluating the extent to which structural diversity (e.g., different model coefficients) within this set translates to predictive multiplicity (i.e., different predictions for the same individual). Our experiments indicate that the existence of many similarly accurate models with comparable error-rate disparities does not necessarily translate into severe predictive multiplicity. Empirically, similarly performant models can exhibit substantially higher predictive agreement than worst-case theoretical guarantees suggest. We find that a simple policy that assigns each inmate the lowest risk among these models is effective for addressing predictive arbitrariness.
