Search papers, labs, and topics across Lattice.
This paper introduces MalayPrag, a new benchmark to evaluate LLMs' understanding of discourse particles in colloquial Malay, along with a framework of five attributes for interpreting their pragmatic functions. Experiments with ten LLMs on three prediction tasks reveal significant deficiencies in connecting discourse particles with their intended pragmatic functions. Providing the five attributes as structured scaffolding substantially improves performance, suggesting a path toward enhanced pragmatic competence in LLMs.
LLMs struggle to grasp the nuances of discourse particles in colloquial Malay, but targeted scaffolding can significantly boost their pragmatic understanding.
Discourse particles, such as \textit{well} and \textit{kind of}, are crucial components that enable LLMs to ``speak'' more like humans. They are used to convey emotions, intentions, and interpersonal meanings. However, existing studies have not yet built a comprehensive understanding of LLMs' capabilities in handling discourse particles. Moreover, the limited number of studies focuses primarily on high-resource languages such as English, with little attention paid to Southeast Asian languages. In this paper, we (1) propose \textsc{MalayPrag}, a benchmark designed to systematically evaluate and analyze LLMs' capabilities in handling discourse particles in colloquial Malay; and (2) introduce five attributes that provide a linguistically grounded, unified framework for interpreting the pragmatic functions of discourse particles. Applying these two contributions, we prompt ten off-the-shelf LLMs to perform three prediction tasks. The experimental results reveal substantial challenges for current LLMs in accurately connecting discourse particles with their pragmatic functions in Malay. The provision of the five attributes designed in this study is found to significantly improve these connections, highlighting the need for structured scaffolding for models' pragmatic competence.
