Search papers, labs, and topics across Lattice.
K-MetBench, a new benchmark based on Korean meteorology qualification exams, was created to evaluate expert reasoning, multimodality, and Korean-specific knowledge in LLMs. Evaluation of 55 models revealed a significant modality gap in chart interpretation and a reasoning gap where models hallucinate logic despite accurate predictions. Surprisingly, smaller Korean-specific models outperformed larger global models in local contexts, highlighting the importance of cultural awareness beyond parameter scaling.
Scaling up LLMs doesn't guarantee expertise: Korean-specific models beat larger global models on a new meteorology benchmark, exposing critical gaps in multimodal reasoning and cultural understanding.
The development of practical (multimodal) large language model assistants for Korean weather forecasters is hindered by the absence of a multidimensional, expert-level evaluation framework grounded in authoritative sources. To address this, we introduce K-MetBench, a diagnostic benchmark grounded in national qualification exams. It exposes critical gaps across four dimensions: expert visual reasoning of charts, logical validity via expert-verified rationales, Korean-specific geo-cultural comprehension, and fine-grained domain analysis. Our evaluation of 55 models reveals a profound modality gap in interpreting specialized diagrams and a reasoning gap where models hallucinate logic despite correct predictions. Crucially, Korean models outperform significantly larger global models in local contexts, demonstrating that parameter scaling alone cannot resolve cultural dependencies. K-MetBench serves as a roadmap for developing reliable, culturally aware expert AI agents. The dataset is available at https://huggingface.co/datasets/soyeonbot/K-MetBench .
