Search papers, labs, and topics across Lattice.
This paper reviews the current state of multi-turn conversational AI, highlighting the challenges of maintaining context and coherence in sustained multimodal interactions. It identifies gaps in existing systems, particularly in areas such as persistent memory, cross-turn grounding, and cultural alignment, despite advancements in multimodal capabilities. The authors propose a comprehensive research agenda aimed at addressing these challenges to enhance the effectiveness of conversational agents across diverse contexts and user interactions.
Current multimodal conversational AI systems excel in perception and action but falter in maintaining coherent dialogue across turns.
Conversational AI is moving beyond isolated text prompts toward sustained, multimodal interaction. In real conversations, users clarify goals, revise requests, interrupt responses, switch topics, and introduce new evidence while expecting systems to preserve context across turns. This makes multi-turn dialogue a distinct challenge requiring systems to maintain and update memory, ground responses across modalities, tools, and external knowledge, and adapt across languages and cultures. This study reviews multi-turn conversational AI across text-only dialogue, AudioLLMs and speech-native systems, multimodal and omni-modal systems, and tool-augmented agents. We organize the literature around datasets and benchmarks, modeling paradigms, training strategies, evaluation setups, and cross-cutting challenges. Our analysis shows that support for multiple modalities has advanced faster than the ability to sustain coherent interaction across a session. Despite stronger capabilities to perceive, speak, and act across modalities, current systems still struggle with persistent memory, cross-turn grounding, full-duplex interaction, robust evaluation, and cultural alignment. We conclude with a research agenda for systems that can remember, revise, ground, speak, listen, act, and adapt across turns, modalities, and cultures. (https://github.com/faiza-sfa/multiturn-conversational-ai-survey)
