Search papers, labs, and topics across Lattice.
The paper introduces a new benchmark and evaluation methodology, Time Series Augmented Generation (TSAG), to assess LLMs' reasoning capabilities in financial time-series analysis by delegating tasks to external tools. They tested state-of-the-art LLMs (GPT-4o, Llama 3, Qwen2) on 100 financial questions, evaluating tool selection accuracy, faithfulness, and hallucination. Results show that capable agents can achieve near-perfect tool-use accuracy with minimal hallucination, supporting the tool-augmented approach for financial reasoning.
LLMs can achieve near-perfect tool use accuracy and minimal hallucination when reasoning about financial time series, but only if they're allowed to delegate to external tools.
Evaluating the reasoning capabilities of Large Language Models (LLMs) for complex, quantitative financial tasks is a critical and unsolved challenge. Standard benchmarks often fail to isolate an agent's core ability to parse queries and orchestrate computations. To address this, we introduce a novel evaluation methodology and benchmark designed to rigorously measure an LLM agent's reasoning for financial time-series analysis. We apply this methodology in a large-scale empirical study using our framework, Time Series Augmented Generation (TSAG), where an LLM agent delegates quantitative tasks to verifiable, external tools. Our benchmark, consisting of 100 financial questions, is used to compare multiple SOTA agents (e.g., GPT-4o, Llama 3, Qwen2) on metrics assessing tool selection accuracy, faithfulness, and hallucination. The results demonstrate that capable agents can achieve near-perfect tool-use accuracy with minimal hallucination, validating the tool-augmented paradigm. Our primary contribution is this evaluation framework and the corresponding empirical insights into agent performance, which we release publicly to foster standardized research on reliable financial AI.
