Search papers, labs, and topics across Lattice.
The paper introduces FIDES, a novel measurement protocol that evaluates the concordance between the natural-language rationale, executable implementation, and performance track record of trading strategies generated by LLMs. By applying this protocol to 40 strategies across various models, the authors reveal that concordance does not correlate with profitability, as only 2 strategies outperformed a buy-and-hold approach, and self-assessments of performance were significantly overestimated. Additionally, the study highlights the impact of model variability on strategy evaluation, suggesting that the fidelity of measurement is crucial for understanding LLM-generated trading strategies.
Only 2 out of 40 LLM-generated trading strategies actually beat a simple buy-and-hold approach, despite claims of superiority from the models themselves.
An LLM asked for a trading strategy returns three artifacts at once: a natural-language rationale, an executable implementation, and once run, a track record. Whether these are the same object is rarely checked. We present FIDES, a measurement protocol that treats them as three views to be reconciled rather than one deliverable to be graded. Through dual delivery, a single model call returns both a natural-language strategy with an explicit claimed edge and a self-contained strategy(df) function. FIDES executes the code in a sandbox against a lag-one out-of-sample backtest and scores three concordance gaps: say to do, do to real, and say to result. On 8 liquid US ETFs across four models plus a two-stage elicitation arm, 40 strategies, 2023 to 2024 out-of-sample, three findings stand out. First, concordance does not predict profit: only 2 of 40 strategies beat buy-and-hold, and a plain sma(50,200) rule outperforms every model's mean Sharpe. Second, self-assessment is badly calibrated: 32 of 40 strategies claim to beat buy-and-hold and exactly one does. Third, swapping the language-code judge for a second model flips say to do on more than half of items. Injecting Close.shift(-1) drops do to real by 0.33 on average, while our runtime future-information probe fired on neither clean nor injected code. We frame FIDES as a protocol for measurement fidelity, not a claim about market performance.
