Search papers, labs, and topics across Lattice.
The authors introduce FinForge, a semi-synthetic pipeline that generates finance-specific evaluation benchmarks by combining expert-guided data curation with controlled LM-based synthesis using Gemini 2.5 Flash. They construct FinForge-5k, a benchmark of 5,000 human-validated question-answer pairs across 11 finance subdomains, derived from 100,000 documents. Evaluations on FinForge-5k show significant performance differences between state-of-the-art models, highlighting the benchmark's utility for assessing and improving financial reasoning capabilities.
Current LLMs still struggle with financial reasoning, but FinForge offers a new, scalable way to generate high-quality, finance-specific benchmarks to drive improvements.
Evaluating Language Models (LMs) in specialized, high-stakes domains such as finance remains a significant challenge due to the scarcity of open, high-quality, and domain-specific datasets. Existing general-purpose benchmarks provide broad coverage but lack the depth and domain fidelity needed to assess LMs'capabilities for real-world financial reasoning, which requires both conceptual understanding and quantitative rigor. To address this gap, we introduce FinForge, a scalable, semi-synthetic pipeline for constructing finance-specific evaluation benchmarks through a hybrid of expert-guided data curation and controlled LM-based synthesis. FinForge combines manual and programmatic corpus construction from authoritative financial sources with structured question generation and validation using Gemini 2.5 Flash. To demonstrate the pipeline's efficacy, we produce FinForge-5k, a snapshot benchmark comprising over 5,000 human-validated question-answer pairs across 11 finance subdomains, derived from a curated corpus of 100,000 verified documents totaling 143M tokens. Evaluation of state-of-the-art open-source and closed-source models on FinForge-5k reveals significant differences in financial reasoning, with leading models achieving accuracy levels near 80%. These findings underscore the framework's utility for diagnosing current model limitations and guiding future improvements in financial domain competence. All code and data are available at https://github.com/gtfintechlab/FinForge.
