Search papers, labs, and topics across Lattice.
This paper introduces speculative tool execution for on-device cascaded voice agents to mitigate tool latency by initiating external tool calls during ongoing ASR, rather than after LLM inference. A novel Predictor module anticipates tool requests from partial ASR hypotheses, speculatively executes them, and caches results, which are then injected into the LLM prompt for faster responses. Evaluated on an Android voice assistant, this method reduces median time-to-first-audio from 5.79s to 4.60s and decreases latency standard deviation, enhancing user experience through more predictable response times.
Latency in voice assistants plummets by 20% by speculatively executing tool calls *during* speech recognition, not after.
Tool-augmented speech assistants typically serialize automatic speech recognition, large language model inference, and external tool execution. As a result, tool latency is incurred only after the user has finished speaking and the LLM has identified the required tool calls. We present speculative tool execution for on-device cascaded voice agents, which predicts tool requests from partial ASR hypotheses and initiates tool execution while speech is still being received, thereby reducing end-to-end response latency. Our approach introduces a Predictor module that anticipates tool calls during speech recognition, executes them speculatively, and caches the results. The cached outputs are then injected into the LLM prompt, enabling faster responses. Additionally, to mitigate errors caused by user self-corrections during speech, we employ a rule-based validation mechanism that selectively injects only valid cached results. As a final safeguard, the LLM retains the ability to issue tool calls directly, ensuring that the latency of our framework is upper-bounded by the baseline serial execution pipeline in the worst case. We evaluate our method using live measurements from a fully implemented Android voice assistant. Our approach reduces the median time-to-first-audio from 5.79,s to 4.60,s and decreases the standard deviation from 3.49,s to 2.81,s, resulting in more predictable response latency.
