Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper critically examines the robustness of tail-shape estimation in large language model (LLM) evaluations, specifically focusing on the tail-index parameter derived from extreme value theory. By implementing a pre-registered protocol that assesses admissibility, goodness-of-fit, threshold stability, and effect size, the authors identify and reject three distinct false positives in toxicity evaluations that would have otherwise gone unnoticed. The findings reveal that tail-shape estimation is more fragile than previously thought, underscoring the need for rigorous evaluation protocols in LLM assessments.
Tail-shape estimation in LLM evaluations is more prone to false positives than the literature suggests, challenging the reliability of current methodologies.
Recent work motivates moving large language model (LLM) evaluation from mean-based to tail-aware metrics, including conditional value-at-risk and tail-index estimates of reward-model error. We ask whether the canonical extreme-value-theory tail-index parameter, which isolates how heavy a tail is from how large the tail mass is, adds discriminative information beyond the mean and a standard tail-magnitude statistic in LLM evaluation. We pre-register a protocol covering admissibility, goodness-of-fit, threshold-stability, and effect-size requirements for any positive tail-shape claim. The protocol is the contribution of this paper; the empirical study below is a demonstration of what its gates catch. Applied to a standard LLM toxicity-evaluation setup under two structurally different scorer families, the protocol catches three distinct modes of false positives that a naive analysis would have published, and rejects the headline tail-shape claim on both scorers. We conclude that tail-shape estimation in the LLM toxicity-evaluation setups we examined is more fragile than the recent literature suggests, and recommend the protocol as a starting point for tail-index claims in similar setups.