Search papers, labs, and topics across Lattice.
The paper introduces JUBAKU-v2, a novel Japanese benchmark dataset designed to evaluate social biases in LLM reasoning, specifically focusing on in-group/out-group attribution biases. Unlike existing benchmarks that rely on translated English data and only assess bias in the conclusion, JUBAKU-v2 leverages attribution theory from social psychology to evaluate bias within the reasoning process itself. Experiments demonstrate that JUBAKU-v2 is more sensitive in detecting performance differences across models compared to existing Japanese bias benchmarks.
LLMs can harbor hidden biases in their reasoning processes, even when reaching unbiased conclusions, and a new Japanese benchmark exposes these subtle cultural biases.
In enhancing the fairness of Large Language Models (LLMs), evaluating social biases rooted in the cultural contexts of specific linguistic regions is essential. However, most existing Japanese benchmarks heavily rely on translating English data, which does not necessarily provide an evaluation suitable for Japanese culture. Furthermore, they only evaluate bias in the conclusion, failing to capture biases lurking in the reasoning. In this study, based on attribution theory in social psychology, we constructed a new dataset, ``JUBAKU-v2,''which evaluates the bias in attributing behaviors to in-groups and out-groups within reasoning while fixing the conclusion. This dataset consists of 216 examples reflecting cultural biases specific to Japan. Experimental results verified that it can detect performance differences across models more sensitively than existing benchmarks.
