Search papers, labs, and topics across Lattice.
This paper introduces an innovative approach to developing app-specific voice assistants by leveraging GUI test code, which serves as executable specifications of application functionality. By utilizing large language models (LLMs) to reinterpret these tests, the authors automate the creation of voice interaction components that are directly grounded in existing application logic. Preliminary results from the AppVA prototype demonstrate that this method significantly reduces development costs and enhances alignment between voice assistants and application behavior across multiple open-source applications.
Transforming GUI test code into app-specific voice assistants could revolutionize how developers create voice interactions, slashing costs and improving functionality.
Voice assistants are widely deployed on mobile platforms, yet most are designed as system-level services that remain poorly aligned with application-specific behavior. As a result, enabling voice interaction at the app level requires developers to manually reimplement application logic, leading to high development and maintenance costs. We propose an LLM-driven approach to automating the development of app-specific voice assistants by repurposing GUI test code, which encodes behavior-preserving, executable specifications of application functionality. In this paper, we present a perspective in which large language models reinterpret GUI tests as bridges between application behavior and conversational interaction. By transforming test methods into app-specific VA artifacts, such as voice intents, capability descriptions, and executable action plans, our approach grounds voice assistants directly in existing application logic rather than external specifications. We illustrate this vision through AppVA, a research prototype on Android. Our preliminary results across five open-source applications suggest that GUI test code can be reused beyond testing, enabling the synthesis of app-specific voice assistants and highlighting a broader research direction at the intersection of software testing, interaction design, and LLM-enabled automation.
