Search papers, labs, and topics across Lattice.
This paper introduces the Query Misalignment Framework, which posits that LLM hallucinations leading to failed abstention can be viewed as the model answering the wrong question. Based on this, they develop Trace Inversion, a method that reconstructs the query the model *actually* answered from its reasoning trace. By comparing the original query with the reconstructed query, Trace Inversion effectively identifies and flags instances where the model is likely to answer incorrectly, significantly improving abstention performance across various LLMs and QA datasets.
LLMs often answer a *different* question than the one you asked, and a clever "trace inversion" technique can reveal this misalignment, leading to state-of-the-art abstention performance.
For Large Language Models (LLMs) to be reliably deployed, models must effectively know when not to answer: abstain. Reasoning models, in particular, have gained attention for impressive performance on complex tasks. However, reasoning models have been shown to have worse abstention abilities. Taking the vulnerabilities of reasoning models into account, we propose our Query Misalignment Framework. Hallucinations resulting in failed abstention can be reinterpreted as LLMs answering the wrong question (rather than answering a question incorrectly). Based on this framework, we develop a new class of state-of-the-art abstention methods called Trace Inversion. First, we generate the reasoning trace of a model. Based on only the trace, we then reconstruct the most likely query that the model responded to. Finally, we compare the initial query with the reconstructed query. Low similarity score between the initial query and reconstructed query suggests that the model likely answered the question incorrectly and is flagged to abstain. Extensive experiments demonstrate that Trace Inversion effectively boosts abstention performance in four frontier LLMs across nine abstention QA datasets, beating competitive baselines in 33 out of 36 settings.
