Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the detection of "confidently wrong" answers—hallucinations—produced by large language models (LLMs) in financial question answering by analyzing the models' internal activations. By training linear probes on the residual stream and measuring behavioral confidence through resampled answers, the research reveals that 15-23% of the most confident answers are incorrect, significantly outperforming traditional baseline methods in identifying these errors. The findings indicate that leveraging internal states can enhance quality control in high-stakes financial applications, providing a cost-effective way to flag potentially erroneous outputs for human review.
Confidently wrong answers in financial LLMs can be detected with up to 77% accuracy using internal model states, revealing a critical flaw in current QA systems.
Large language models (LLMs) in financial applications fail most consequentially when they are confidently wrong. Hedged, uncertain answers invite scrutiny, whereas confident errors silently degrade downstream decisions without warning. We ask how reliably such confidently wrong answers, or confident hallucinations, can be detected from a model's internal activations, and whether those activations carry information beyond its observable outputs. We train linear probes on the residual stream and evaluate them on two established question-answering (QA) benchmarks built from real filings, FinQA and TAT-QA. Behavioral confidence is measured as the agreement among eight resampled answers to the same question, and probe effectiveness is compared against baselines, such as token log-probabilities and the model's own True/False self-assessment of its answer. Our findings show that among confident answers, those for which all eight resamples agree, 15-23% are wrong on FinQA. There the probes have a significant advantage over baseline methods in detecting hallucinations, holding 0.68-0.77 AUROC while the best baselines fall to 0.55-0.63, across Qwen3-8B, Llama-3.1-8B, and Gemma-2-9B. Our results suggest that probing can be a cost-effective triage mechanism for routing LLM answers to human review and quality control procedures in high-stakes financial applications.