Search papers, labs, and topics across Lattice.
This paper decomposes multi-turn question answering with preference elicitation into clarification policy learning and post-clarification answering. Using the PACIFIC benchmark, they show that while supervised fine-tuning effectively improves clarification policy, post-clarification answering accuracy remains significantly lower. This highlights that the primary bottleneck in multi-turn QA is effectively understanding and utilizing user responses to clarifying questions.
Even when large language models know *what* clarifying question to ask, they still struggle to understand the answer and provide a correct final response, revealing a critical gap in multi-turn QA.
Pluralistic alignment requires systems to adapt to diverse user values, communication styles, and contextual assumptions. We believe that a foundational prerequisite for such alignment enabling accurate preference elicitation from people when their intent is under-specified or ambiguous. We study the problem of preference elicitation in multi-turn question answering by decomposing the problem into two components: a \textbf{clarification policy}, which decides whether to ask a clarifying question or answer directly, and \textbf{post-clarification answering}, which produces the correct final answer once the missing information is provided. We show, using the PACIFIC benchmark, that supervised fine-tuning rapidly improves the clarification policy, however, final answer accuracy remains substantially lower even when the model takes the correct action. This gap indicates that understanding and correctly interpreting the user's response is the critical gap in multi-turn question-answering systems.
