Search papers, labs, and topics across Lattice.
This study investigates how user context, including memory and role prompts, influences the interpretation of financial documents by large language models (LLMs) using a dataset of 3,575 SEC filings. The findings reveal that the majority of context-related biases stem from the models' interpretative frameworks rather than differences in evidence retrieval. Two mitigation strategies were tested, showing a reduction in biases but highlighting that complete elimination is not achievable and effectiveness varies by model.
User context can significantly skew financial analysis in LLMs, with interpretation biases overshadowing evidence selection.
Large Language Models (LLMs) increasingly use user context such as memory, profiles, and role prompts to personalize their responses. This personalization can affect evidence-based judgment: the same evidence may lead to different conclusions under different user contexts. Finance provides a high-stakes setting to study this problem because decisions often depend on interpreting long and complex documents. We test this using 3,575 SEC filings across twelve LLMs. We compare persona-conditioned retrieval, neutral retrieval, and memory-framed context to separate the effect of evidence selection from the effect of interpretation. We find that most user-context spillover comes from how models interpret the same evidence under different roles, rather than from retrieving different evidence. We then test two simple mitigation strategies: expressing the same investor mindset as a user profile instead of an assistant role, and separating evidence-based and personalized outputs. Both reduce spillover, but neither removes it completely, and their effectiveness varies substantially across models.
