Search papers, labs, and topics across Lattice.
This paper introduces Ensemble Diversity Optimization (EDO), a novel framework designed to address systematic annotator disagreement in subjective NLP tasks by optimizing ensemble weights, cardinality, and calibration through a differentiable objective. By employing Gumbel-Softmax relaxation and a signed diversity regularizer, EDO effectively manages the trade-off between utility and calibration, preventing ensemble collapse while enhancing probabilistic calibration. Experiments across four subjective text-classification benchmarks reveal that EDO significantly reduces cross-entropy and Brier scores compared to existing methods, while maintaining competitive F1 scores and better alignment with annotator distributions.
EDO achieves up to 78% reduction in cross-entropy, revolutionizing how we model human subjectivity in NLP tasks.
Subjective NLP tasks often exhibit systematic annotator disagreement, requiring models that represent uncertainty rather than collapse it. We introduce Ensemble Diversity Optimization (EDO), a prediction-space framework that jointly optimizes ensemble weights, effective cardinality, and calibration through a unified differentiable objective. EDO learns ensemble composition and size end-to-end via Gumbel-Softmax relaxation and incorporates a signed diversity regularizer, tuned on validation data, to steer optimization toward either preserving or suppressing disagreement. This regularization prevents ensemble collapse and enables controlled navigation of the utility-calibration trade-off. The framework integrates a soft F1 surrogate, class-weighted cross-entropy to address imbalance, and reliability-weighted diversity to regulate intra-ensemble variability. Experiments on four subjective text-classification benchmarks (ArMIS, ConvAbuse, HS-Brexit, MD-Agreement) show that EDO substantially improves probabilistic calibration, reducing cross-entropy (40-78% depending on baseline) and lowering Brier scores relative to Soft-CE, Soft-MD, Top-5 Voting, and WEL, while maintaining competitive F1 and better alignment with annotator distributions. These results demonstrate that jointly optimizing ensemble structure with a signed diversity regularizer provides an efficient, model-agnostic approach for modeling human subjectivity in supervised learning.
