Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the challenge of imbalanced crowdsourcing by introducing a generative aggregation model that integrates class-dependent annotator accuracy with item difficulty, specifically targeting minority-class detection. The authors highlight the limitations of existing models that either overlook item difficulty or fail to account for class-dependent errors, thus providing a more nuanced approach to label aggregation in real-world applications. Evaluated on 33 diverse datasets, the model demonstrates superior minority recall while maintaining competitive balanced accuracy, underscoring its effectiveness in scenarios where rare labels are critical.
Achieving the highest minority recall across 33 datasets, this model reshapes how we approach label aggregation in imbalanced crowdsourcing scenarios.
We study imbalanced crowdsourcing with a focus on class-dependent annotator accuracy, a setting that, to the best of our knowledge, remains relatively underexplored despite its importance in real-world inspection systems where the labels of greatest operational importance are also the rarest ones. In this setting, annotators may be reliable on both classes, unreliable on both classes, majority-class specialists, or minority-class specialists. Existing models only partially address this problem: they either capture class-dependent errors but ignore item difficulty, or they model item difficulty without capturing class-dependent errors. To fill this gap for imbalanced datasets in crowdsourcing, we introduce a generative aggregation model combining item difficulty with class-dependent annotator competence. The model allows both annotator abilities and item difficulties to vary across classes. We then revisit Condorcet's Jury Theorem in the class-imbalanced setting. We also show that majority voting asymptotically preserves the underlying class proportion. We evaluate our model on $33$ real-world crowdsourcing datasets, covering multiclass tasks such as images and text, as well as two large-scale regimes: large-scale annotation datasets, with many annotations per item, and large-scale item datasets, with a large number of annotated instances. Across these diverse settings, our model consistently achieves the highest minority recall while remaining competitive in balanced accuracy, making it particularly relevant when rare-label recovery is the primary objective.