Search papers, labs, and topics across Lattice.
This paper introduces and analyzes the focal-entropy, a focal-loss analogue of cross-entropy, to provide an information-theoretic understanding of the focal-loss's behavior in class-imbalanced classification. The analysis establishes conditions for finiteness, convexity, and continuity of the focal-entropy, along with asymptotic characterizations and proof of the existence and uniqueness of its minimizer. The key finding is that focal-entropy amplifies mid-range probabilities, suppresses high-probability outcomes, and can induce an over-suppression regime for very small probabilities, especially under extreme class imbalance.
Focal loss doesn't just re-weight examples; it fundamentally reshapes the learned probability distribution, potentially distorting it far from the true data distribution, especially with severe class imbalance.
The focal-loss has become a widely used alternative to cross-entropy in class-imbalanced classification problems, particularly in computer vision. Despite its empirical success, a systematic information-theoretic study of the focal-loss remains incomplete. In this work, we adopt a distributional viewpoint and study the focal-entropy, a focal-loss analogue of the cross-entropy. Our analysis establishes conditions for finiteness, convexity, and continuity of the focal-entropy, and provides various asymptotic characterizations. We prove the existence and uniqueness of the focal-entropy minimizer, describe its structure, and show that it can depart significantly from the data distribution. In particular, we rigorously show that the focal-loss amplifies mid-range probabilities, suppresses high-probability outcomes, and, under extreme class imbalance, induces an over-suppression regime in which very small probabilities are further diminished. These results, which are also experimentally validated, offer a theoretical foundation for understanding the focal-loss and clarify the trade-offs that it introduces when applied to imbalanced learning tasks.
