Search papers, labs, and topics across Lattice.
This paper introduces InvestLogicBench, a comprehensive benchmark designed to evaluate the investment reasoning of large language models (LLMs) using real-world data from 151 investors, encompassing over 201,247 documented decisions. By employing a structured trace of investor profiles, market events, reasoning, decisions, and outcomes (P鈫扙鈫扲鈫扗鈫扥), the study reveals that while LLMs maintain logical plausibility, their grounding in actual events is significantly lacking, with scores ranging from 0.8 to 2.8 out of 5. These findings highlight the inadequacy of traditional evaluation methods that focus solely on outcomes, suggesting a need for more nuanced assessment frameworks in AI-driven financial decision-making.
LLMs may sound convincing, but their investment reasoning often lacks grounding in real-world events, revealing a critical gap in evaluation methods.
Investment competence is inherently personalized: the same market evidence can justify different actions for investors with different goals, horizons, portfolios, and risk boundaries. Yet financial LLMs are evaluated either by static question answering or by terminal profit and loss. The former omits agency; the latter cannot reveal whether a profitable action was grounded, profile-consistent, or merely lucky. We ask whether the community is using the wrong ruler for consequential agents. We introduce \textsc{InvestLogicBench}, a process-native benchmark containing 201,247 documented decisions from 151 real-world investors. Each episode instantiates a \textbf{P$\rightarrow$E$\rightarrow$R$\rightarrow$D$\rightarrow$O} trace: investor \textit{Profile}, observable market \textit{Events}, investment \textit{Reasoning}, executable \textit{Decision}, and delayed \textit{Outcome}. The release includes profile construction, point-in-time event binding, structured logic, horizons, outcomes, and post-mortems, and supports comprehension, profile-conditioned generation, and end-to-end replay. Across four leading LLMs, logical plausibility remains near 4/5 while event grounding is only 0.8--2.8/5; return and process quality also disagree. These results expose polished but weakly grounded reasoning that outcome-only evaluation hides. We further argue that P$\rightarrow$E$\rightarrow$R$\rightarrow$D$\rightarrow$O should be a data-system interface, requiring versioned profiles, temporal provenance, inspectable retrieval, decision ledgers, and replayable outcomes. Finance is our stress test for a broader class of personalized, consequential agents.
