Search papers, labs, and topics across Lattice.
The paper introduces SAHM, a new benchmark for Arabic financial NLP and Shari'ah-compliant reasoning, comprising 14,380 expert-verified instances across seven tasks. Evaluation of 19 LLMs reveals that Arabic fluency doesn't guarantee strong, evidence-grounded financial reasoning, especially in generation and causal reasoning tasks. The benchmark, evaluation framework, and an instruction-tuned model are released to facilitate future research.
Arabic LLMs can speak the language of finance, but they often fail to reason about it, especially when it comes to causality and generation.
English financial NLP has progressed rapidly through benchmarks for sentiment, document understanding, and financial question answering, while Arabic financial NLP remains comparatively under-explored despite strong practical demand for trustworthy finance and Islamic-finance assistants. We introduce SAHM, a document-grounded benchmark and instruction-tuning dataset for Arabic financial NLP and Shari'ah-compliant reasoning. SAHM contains 14,380 expert-verified instances spanning seven tasks: AAOIFI standards QA, fatwa-based QA/MCQ, accounting and business exams, financial sentiment analysis, extractive summarization, and event-cause reasoning, curated from authentic regulatory, juristic, and corporate sources. We evaluate 19 strong open and proprietary LLMs using task-specific metrics and rubric-based scoring for open-ended outputs, and find that Arabic fluency does not reliably translate to evidence-grounded financial reasoning: models are substantially stronger on recognition-style tasks than on generation and causal reasoning, with the largest gaps on event-cause reasoning. We release the benchmark, evaluation framework, and an instruction-tuned model to support future research on trustworthy Arabic financial NLP.
