Search papers, labs, and topics across Lattice.
This study addresses the challenge of reliable confidence estimation in black-box large language models (LLMs) for hierarchical taxonomic reasoning within biodiversity monitoring. By employing hierarchy-aware supervision to train lightweight supervised estimators, the authors demonstrate significant improvements in micro AUROC performance, achieving scores between 0.75 and 0.80 compared to a baseline of 0.57. The findings underscore the importance of incorporating hierarchical output structures in uncertainty estimation, particularly when a unified rejection threshold is employed.
Hierarchical supervision can boost uncertainty estimation in LLMs, enhancing decision-making in biodiversity monitoring by improving prediction accuracy significantly.
Large language models (LLMs) are increasingly used for scientific decision support, yet reliable confidence estimation remains difficult in black-box settings. We study uncertainty estimation for hierarchical taxonomic reasoning generated by a black-box LLM in a long-tailed biodiversity monitoring pipeline. Using proxy features extracted by an open-source tool LLM, we train lightweight supervised estimators with hierarchy-aware supervision to predict rank-wise correctness. Across three tool LLMs, the supervised estimators consistently outperform a token-likelihood baseline for micro discrimination and selective prediction under a single global rejection threshold, improving micro AUROC from 0.57 to 0.75--0.80. The best results are achieved by a rank-specific multi-head design (H3), suggesting that accounting for hierarchical output structure is important when a unified abstention rule is required. Our code is publicly available at https://github.com/uoguelph-mlrg/hierarchy-aware-llm-uq
