Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces MADE, a continuously updated multi-label text classification benchmark derived from medical device adverse event reports, designed to address limitations of existing benchmarks like saturation and data contamination. They evaluate over 20 encoder- and decoder-only models, including fine-tuned and few-shot variants, assessing both predictive performance and uncertainty quantification (UQ) using entropy/consistency-based and self-verbalized methods. The results reveal trade-offs between model architecture, fine-tuning strategy, and UQ reliability, highlighting the surprisingly weak UQ of large reasoning models.
Large language models may ace rare medical event classification, but their uncertainty estimates are surprisingly unreliable, especially compared to smaller, fine-tuned decoders.
Machine learning in high-stakes domains such as healthcare requires not only strong predictive performance but also reliable uncertainty quantification (UQ) to support human oversight. Multi-label text classification (MLTC) is a central task in this domain, yet remains challenging due to label imbalances, dependencies, and combinatorial complexity. Existing MLTC benchmarks are increasingly saturated and may be affected by training data contamination, making it difficult to distinguish genuine reasoning capabilities from memorization. We introduce MADE, a living MLTC benchmark derived from {m}edical device {ad}verse {e}vent reports and continuously updated with newly published reports to prevent contamination. MADE features a long-tailed distribution of hierarchical labels and enables reproducible evaluation with strict temporal splits. We establish baselines across more than 20 encoder- and decoder-only models under fine-tuning and few-shot settings (instruction-tuned/reasoning variants, local/API-accessible). We systematically assess entropy-/consistency-based and self-verbalized UQ methods. Results show clear trade-offs: smaller discriminatively fine-tuned decoders achieve the strongest head-to-tail accuracy while maintaining competitive UQ; generative fine-tuning delivers the most reliable UQ; large reasoning models improve performance on rare labels yet exhibit surprisingly weak UQ; and self-verbalized confidence is not a reliable proxy for uncertainty. Our work is publicly available at https://hhi.fraunhofer.de/aml-demonstrator/made-benchmark.