Search papers, labs, and topics across Lattice.
The authors introduce ALPS, a new expert-curated diagnostic dataset for evaluating deep semantic and pragmatic reasoning in Arabic NLP models. ALPS comprises 531 native Arabic questions across 15 tasks and 47 subtasks, designed to probe morpho-syntactic dependencies and compositional semantics. Evaluation of 23 diverse models reveals that while some commercial models surpass average human performance, a significant gap remains, particularly for Arabic-native models and in tasks requiring understanding of morpho-syntactic dependencies.
Despite high fluency, current Arabic NLP models stumble on fundamental morpho-syntactic dependencies, especially in diacritics-reliant tasks, highlighting a critical gap in deep linguistic understanding.
While recent Arabic NLP benchmarks focus on scale, they often rely on synthetic or translated data which may benefit from deeper linguistic verification. We introduce ALPS (Arabic Linguistic & Pragmatic Suite), a native, expert-curated diagnostic challenge set probing Deep Semantics and Pragmatics, capabilities that complement specialized large-scale benchmarks. While broad-coverage benchmarks prioritize scale and multi-task coverage, ALPS targets the depth of linguistic understanding through 531 rigorously crafted questions across 15 tasks and 47 subtasks. We developed the dataset with deep expertise in Arabic linguistics, guaranteeing cultural authenticity and eliminating translation artifacts. Evaluating 23 diverse models (commercial, open-source, and Arabic-native) against a single-pass human performance (avg. 84.6% accuracy) and an expert-adjudicated oracle (99.2%), we reveal a critical dissociation: models achieve high fluency but fail on fundamental morpho-syntactic dependencies, with elevated error rates on morpho-syntactic dependencies (36.5% across diacritics-reliant tasks) compared to compositional semantics. While top commercial models (Gemini-3-flash at 94.2%) surpass the average single human, a substantial gap persists between commercial giants and Arabic-native models, with the best Arabic-specific model (Jais-2-70B at 83.6%) approaching but not matching human performance.
