Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the problem of selecting an optimal tradeoff ranking score when evaluating binary classifiers, acknowledging that common metrics like accuracy, recall, and precision induce divergent rankings. The authors introduce a novel framework where standard ranking scores are viewed as special cases of a parameterized continuum, and define optimality via minimizing Fréchet variance with respect to the Kendall distance between rankings. They derive an exact analytical solution for this optimal tradeoff score under equal class priors, demonstrating that it is not generally achieved by simple averaging of parameter importance values.
A single analytical solution precisely balances any two binary classification ranking metrics, revealing that optimal tradeoffs are rarely simple averages of importance parameters.
The accuracy, true negative rate (negative class recall), true positive rate (positive class recall), negative predictive value (negative class precision), positive predictive value (positive class precision), class-specific intersections over unions, class-specific F-scores, Cohen's kappa, and balanced accuracy are scores that one intuitively wants to maximize when developing machine learning methods. However, they lead to different rankings of such methods. So, how can we establish a ranking that is an optimal tradeoff between any pair of these scores? It turns out that all these scores are special cases of ranking scores: a continuum of scores parameterized by the relative importance values assigned to the true negative, false positive, false negative, and true positive outcomes. Since all these scores are known to induce meaningful rankings, we seek the optimal tradeoffs among them. We consider the tradeoff between two scores to be optimal when it minimizes the Fr\'echet variance regarding the Kendall distance (also known as the bubble-sort distance) between rankings. For any given set of classifiers to rank, the optimal tradeoff lies at the center of the shortest (geodesic) path on the manifold of all rankings induced by the ranking scores. However, in the parameter space of the ranking scores, it is rarely achieved by averaging the importance values corresponding to the initial scores. In this paper, we present an exact analytical solution for determining a ranking score that is an optimal tradeoff for any given pair of initial scores, in the common case where the class priors are the same for all compared performances. We also experimentally investigate what happens when performance follows some distributions with fixed priors.