Search papers, labs, and topics across Lattice.
YOMI-Bench is introduced as a specialized benchmark for assessing the kanji reading and phonological understanding capabilities of large language models (LLMs) in Japanese. The benchmark addresses the unique challenges posed by the multiple readings of kanji characters, which significantly hinder LLM performance. Evaluation results reveal that both Japanese-specific and commercial LLMs struggle with kanji reading tasks, highlighting a critical gap in their linguistic capabilities.
Even Japanese-specific LLMs fail to grasp kanji readings, revealing a significant shortcoming in their linguistic understanding.
We propose YOMI-Bench, a benchmark for evaluating kanji reading and phonological understanding of large language models (LLMs) for Japanese. In Japanese, a single kanji character often has multiple possible readings, making it difficult to infer the correct reading from surface-level text alone. Due to these linguistic characteristics, it is empirically known that LLMs exhibit low performance in kanji reading for Japanese. The proposed YOMI-Bench consists of four tasks specifically designed to evaluate kanji reading performance in Japanese. In our evaluation using YOMI-Bench, we assessed one multilingual open LLM, four Japanese-specific open LLMs, and five commercial LLMs. As a result, we found that even Japanese-specific models show low performance, and that commercial models also perform poorly on generation tasks that require consideration of kanji readings.
