Search papers, labs, and topics across Lattice.
The paper introduces BankMathBench, a new benchmark dataset designed to evaluate and improve the numerical reasoning capabilities of LLMs in realistic banking scenarios involving tasks like payout estimation and interest calculation. The dataset is structured into three difficulty levels (basic, intermediate, advanced) to assess single-product reasoning, multi-product comparison, and multi-condition scenarios, respectively. Fine-tuning open-source LLMs on BankMathBench with tool augmentation resulted in significant accuracy improvements (57.6%p to 75.1%p) across all difficulty levels, demonstrating the benchmark's efficacy.
LLMs struggle with basic banking calculations, like comparing loans or estimating payouts, until fine-tuned on a new benchmark designed for realistic financial scenarios.
Large language models (LLMs)-based chatbots are increasingly being adopted in the financial domain, particularly in digital banking, to handle customer inquiries about products such as deposits, savings, and loans. However, these models still exhibit low accuracy in core banking computations-including total payout estimation, comparison of products with varying interest rates, and interest calculation under early repayment conditions. Such tasks require multi-step numerical reasoning and contextual understanding of banking products, yet existing LLMs often make systematic errors-misinterpreting product types, applying conditions incorrectly, or failing basic calculations involving exponents and geometric progressions. However, such errors have rarely been captured by existing benchmarks. Mathematical datasets focus on fundamental math problems, whereas financial benchmarks primarily target financial documents, leaving everyday banking scenarios underexplored. To address this limitation, we propose BankMathBench, a domain-specific dataset that reflects realistic banking tasks. BankMathBench is organized in three levels of difficulty-basic, intermediate, and advanced-corresponding to single-product reasoning, multi-product comparison, and multi-condition scenarios, respectively. When trained on BankMathBench, open-source LLMs exhibited notable improvements in both formula generation and numerical reasoning accuracy, demonstrating the dataset's effectiveness in enhancing domain-specific reasoning. With tool-augmented fine-tuning, the models achieved average accuracy increases of 57.6%p (basic), 75.1%p (intermediate), and 62.9%p (advanced), representing significant gains over zero-shot baselines. These findings highlight BankMathBench as a reliable benchmark for evaluating and advancing LLMs' numerical reasoning in real-world banking scenarios.
