Search papers, labs, and topics across Lattice.
This paper investigates the generalization of latent signals for detecting unanswerable questions across different domains and dialogue turns. By developing a turn-labeled multi-turn benchmark and an evaluation harness, the researchers demonstrate that unanswerability probes transfer well between datasets with shared grounds, such as missing information in math and reading comprehension tasks. While single-turn probes struggle with dynamic answerability in dialogue, in-structure probes show promise, though a calibrated probe gate's performance on underspecified turns still lags behind human-level decision-making.
Latent signals for unanswerability generalize robustly across diverse question-answering datasets, but their utility in multi-turn dialogue hinges on capturing conversational dynamics rather than just static missing information.
Large language models routinely answer questions that cannot be answered from the information given, and in dialogue they answer before enough has been said. Unanswerability is linearly decodable from hidden states, but it is unclear which of its forms share a representation and whether the signal is useful in dialogue. We contribute a turn-labeled multi-turn benchmark (423 conversations, 1,661 labeled turn-states) and an evaluation harness with a simulated user who answers clarifying questions, and use them with six datasets and six open-weight LLMs to test how far probes for unanswerability carry. Probes transfer robustly between datasets that share a ground of unanswerability: missing information in math (AUROC 0.77-0.97) and in a passage (SQuAD 2.0<->MuSiQue, 0.77-0.90). Probes for epistemic"known-unknowns"transfer poorly to math, but this separation weakens under lexical controls and changes with layer and coordinate system, so it remains unresolved. Single-turn probes fail zero-shot to detect when a conversation becomes answerable; in-structure probes recover it, but no better than a bag-of-words classifier. A gate on the calibrated probe, with no model fine-tuning, fires on underspecified turns far more precisely than chance, and its end-task success comes within 0.08 of a gate given the true labels. Yet across four models it does not reliably beat vanilla generation or prompted consolidation. The remaining gap lies mostly in how models use a clarification, not in detection.
