Search papers, labs, and topics across Lattice.
This paper introduces ShiJianBench, an innovative offline framework designed to evaluate conversational investment advisors by analyzing matched investor trajectories influenced by advisor interactions under historical market conditions. By employing a multi-agent investor simulator that incorporates evolving state variables and long-term memory, the framework assesses advisor effectiveness not just on immediate responses but on their impact on long-term investor behavior. The findings highlight a group of leading LLM advisors that excel in delivering personalized content while achieving superior investor trajectory outcomes, emphasizing the need for trajectory-aware evaluation methods in this domain.
Leading LLM investment advisors can significantly enhance long-term investor outcomes, revealing a critical gap in traditional evaluation methods.
Conversational investment advisors influence not only what users know, but also how they make subsequent decisions as market conditions evolve. Existing evaluations primarily assess response quality or observed outcomes, leaving the long-horizon pathway from advisor language to investor behavior difficult to audit. We introduce ShiJianBench, an offline framework for evaluating conversational investment advisors through matched investor trajectories under fixed historical market feedback. At its core is a multi-agent investor simulator with explicit evolving state variables, motive-driven deliberation, long-term memory, and dialogue-grounded updates. The simulator is calibrated against aggregate behavioral patterns from 7,199 real users, and advisor policies are evaluated using separate investor-side, service-side, and content-side metrics under a hard compliance gate. Experiments on Chinese fund-market traces from 2021 to 2026 identify a stable leading group of LLM advisors that combines substantially stronger personalized content with competitive investor-side trajectory outcomes. These results reveal a systematic distinction between producing a high-quality response and delivering an effective long-horizon intervention, motivating trajectory-aware evaluation of conversational advisors.
