Search papers, labs, and topics across Lattice.
The paper introduces KTD-Fin, a new benchmark for evaluating LLM trading agents that mitigates data leakage and provides attribution analysis. KTD-Fin uses data-side masking to anonymize identifiers and calendar information, preventing LLMs from relying on memorized market data. Evaluation of ten frontier LLM agents on the CSI300 using KTD-Fin reveals that their returns are primarily driven by market and style exposure, with little evidence of stock-selection alpha, highlighting the importance of controlled evaluation in financial LLM benchmarks.
LLM trading agents might seem profitable, but a new benchmark reveals their returns are mostly just riding market trends, not actual stock-picking skill.
Evaluating whether large language model (LLM) agents can profit in capital markets is increasingly framed as end-to-end trading: place an agent in a historical market, let it trade, and measure portfolio returns. This setup is vulnerable to two evaluation failures. First, long backtests often overlap with the knowledge cutoffs of frontier LLMs, allowing memorized tickers, dates, prices, and market narratives to substitute for investment reasoning. Second, raw returns are a noisy proxy for stock-selection ability, since positive performance may come from market beta, style exposure, or favorable regimes rather than genuine alpha. We introduce KTD-Fin (Knowing-To-Doing Financial Benchmark), an end-to-end stock-market trading benchmark that addresses both issues. KTD-Fin uses a data-side masking protocol to anonymize key identifiers and calendar information consistently across prompts and tools, separating historical market memory from investment decision-making. It also incorporates a Barra-style performance attribution framework that decomposes portfolio returns into market, style, and stock-selection alpha components. Across ten frontier LLM agents evaluated on the Chinese CSI300 over a 2024--2026 window, masking substantially changes agent rationales, pushing them towards anonymized factor-based reasoning. Attribution analysis further shows that LLM agents' cumulative returns under leakage-controlled evaluation are largely explained by passive market and style exposure, with limited evidence of persistent stock-selection alpha. These findings suggest that financial LLM benchmarks should evaluate not only whether an agent makes money, but also whether the source of returns reflects transferable investment skill. We release KTD-Fin as a reproducible template for leakage-controlled and attribution-aware evaluation of LLM trading agents.
Tsinghua AI