Search papers, labs, and topics across Lattice.
The paper introduces BEATS, a framework and benchmark for evaluating bias, ethics, fairness, and factuality in LLMs across 29 metrics spanning demographic, cognitive, and social biases, ethical reasoning, group fairness, and misinformation risk. The goal is to quantitatively assess how LLMs perpetuate societal prejudices and systemic inequities. Empirical results show that 37.65% of outputs from leading LLMs contain some form of bias, demonstrating a significant risk in using these models for critical decision-making.
Despite advances in LLMs, over a third of their outputs still exhibit biases, as revealed by the new BEATS benchmark.
In this research, we introduce BEATS, a novel framework for evaluating Bias, Ethics, Fairness, and Factuality in Large Language Models (LLMs). Building upon the BEATS framework, we present a bias benchmark for LLMs that measure performance across 29 distinct metrics. These metrics span a broad range of characteristics, including demographic, cognitive, and social biases, as well as measures of ethical reasoning, group fairness, and factuality related misinformation risk. These metrics enable a quantitative assessment of the extent to which LLM generated responses may perpetuate societal prejudices that reinforce or expand systemic inequities. To achieve a high score on this benchmark a LLM must show very equitable behavior in their responses, making it a rigorous standard for responsible AI evaluation. Empirical results based on data from our experiment show that, 37.65\% of outputs generated by industry leading models contained some form of bias, highlighting a substantial risk of using these models in critical decision making systems. BEATS framework and benchmark offer a scalable and statistically rigorous methodology to benchmark LLMs, diagnose factors driving biases, and develop mitigation strategies. With the BEATS framework, our goal is to help the development of more socially responsible and ethically aligned AI models.
