Search papers, labs, and topics across Lattice.
This paper introduces BasPhyCo, the first non-question-answering physical commonsense reasoning dataset for Basque, available in both standard and dialectal variants, to address the lack of evaluation in low-resource languages. The authors evaluated multiple multilingual LLMs and models pretrained on Italian and Basque across three levels of commonsense understanding: plausibility, consistency, and verifiability. The results show that LLMs struggle with physical commonsense reasoning in Basque, particularly with dialectal variants, especially in terms of verifiability.
LLMs struggle with physical commonsense reasoning in low-resource languages like Basque, especially when presented with dialectal variations, revealing a significant limitation in their real-world applicability.
Physical commonsense reasoning represents a fundamental capability of human intelligence, enabling individuals to understand their environment, predict future events, and navigate physical spaces. Recent years have witnessed growing interest in reasoning tasks within Natural Language Processing (NLP). However, no prior research has examined the performance of Large Language Models (LLMs) on non-question-answering (non-QA) physical commonsense reasoning tasks in low-resource languages such as Basque. Taking the Italian GITA as a starting point, this paper addresses this gap by presenting BasPhyCo, the first non-QA physical commonsense reasoning dataset for Basque, available in both standard and dialectal variants. We evaluate model performance across three hierarchical levels of commonsense understanding: (1) distinguishing between plausible and implausible narratives (accuracy), (2) identifying the conflicting element that renders a narrative implausible (consistency), and (3) determining the specific physical state that creates the implausibility (verifiability). These tasks were assessed using multiple multilingual LLMs as well as models pretrained specifically for Italian and Basque. Results indicate that, in terms of verifiability, LLMs exhibit limited physical commonsense capabilities in low-resource languages such as Basque, especially when processing dialectal variants.
