Search papers, labs, and topics across Lattice.
This paper introduces conversational infill, a novel approach that allows a small talker model to generate immediate, contextually grounded responses while seamlessly integrating knowledge from a slower external reasoner during inference. By curating a large synthetic dataset and testing across various small language models, the authors demonstrate that their method can achieve millisecond-level response times while closing the accuracy gap to top-performing models. User studies indicate that this approach not only maintains high responsiveness but also matches the performance of leading models in retrieval-heavy tasks, marking a significant advancement in the design of conversational voice agents.
Conversational infill enables voice agents to achieve millisecond response times while closing the accuracy gap to state-of-the-art reasoning models, redefining the latency-capability trade-off.
Voice agents face a fundamental tension: the reasoning, retrieval, and tool use that make foundation models capable are iterative and slow, while conversational interaction demands responses on a millisecond timescale. Smaller, real-time models meet the latency bar but cannot match foundation models on complex tasks, leaving current voice agents to trade away either responsiveness or capability. We introduce conversational infill, where a small talker model both immediately generates contextually grounded responses to hide the latency of an external reasoner model and fluently integrates streamed reasoner knowledge into its responses during inference. We curate a 290,571-example synthetic dataset spanning six domains and demonstrate that this task is learnable across seven widely used small language models ranging from 135M to 1.7B parameters. Our system implementation, ConvFill, sustains millisecond-level time-to-first-response while closing the accuracy gap to within 6.3% of the corresponding frontier reasoner performance. In a live user study (n=18) with talker deployments running on an Apple M2 SoC, participants rank ConvFill on par with frontier models overall, prefer it for retrieval-heavy tasks, and rate it significantly more responsive. These results show that conversational infill unlocks a new point on the latency-capability Pareto frontier, offering a practical path toward voice agents that are both responsive and highly capable. Code, models, and datasets are available at https://github.com/vysri/conversational-infill.
