Search papers, labs, and topics across Lattice.
The paper introduces V-FiLLM, a novel framework for generating financial reasoning benchmarks based on executable computation trees derived from real tables, ensuring that answers are correct by construction. This benchmark allows for scalable item generation without the need for manual annotation, while also exposing four axes of difficulty that can be independently controlled. Evaluation of open-source models reveals significant accuracy drops鈥攗p to 51%鈥攚ith increased reasoning depth and adversarial perturbations, underscoring the challenges in robust financial reasoning, although fine-tuning with LoRA on verified traces shows promise for improving performance.
Financial reasoning accuracy plummets by up to 51% with deeper computations, revealing critical gaps in LLM capabilities.
While existing benchmarks have made substantial progress in evaluating LLMs across STEM domains, financial reasoning over structured data remains comparatively less explored. We introduce V-FiLLM, a framework that generates financial reasoning benchmarks from executable computation trees grounded in real tables, yielding items whose answers are correct by construction. Trees are evaluated symbolically to obtain ground truth and rendered into natural-language questions, removing any model from the labeling loop, so items can be generated at arbitrary scale without annotation cost and without inheriting a generator's error rate. V-FiLLM exposes four independently controllable axes of difficulty including computation depth, expression breadth, financial concept complexity, and context size. By evaluating on open-source models, we find that accuracy falls up to 51% as reasoning depth increases, and up to 47% points under adversarial numerical perturbations, highlighting remaining challenges in robust financial reasoning over tables. We further show that lightweight LoRA fine-tuning on verified chain-of-thought traces improves accuracy from 81.1% to 85.6% on held-out problems and outperforms the base model by 5% points on FinQA (Chen et al., 2022a), s), suggesting that targeted, low-cost adaptation is a promising direction for compositional reasoning in financial QA.
ETH