Search papers, labs, and topics across Lattice.
This paper introduces Voltron, a framework that enables the elastic execution of large language model (LLM) inference across multiple edge devices, addressing the limitations of centralized processing such as latency and privacy risks. By leveraging the combined resources of several user-end devices, Voltron enhances the accuracy of LLMs beyond what is achievable on a single device. The evaluation shows that Voltron achieves up to 16.5% higher accuracy compared to existing single-device LLMs, effectively meeting user quality of service requirements in diverse edge environments.
Voltron boosts LLM accuracy by 16.5% by harnessing the power of multiple edge devices, transforming how we think about local AI execution.
Large language models (LLMs) are widely used in intelligent services due to their remarkable capability in generative tasks. Typically, LLM-based services process the inference requests of the users in a centralized data center. Unfortunately, such centralized execution has limitations for end-users, such as increased response latency with communication overhead and privacy leakage risk. To alleviate the aforementioned limitations, there have been increasing pushes to execute LLM inference locally on user-end devices. However, the limited resources of a single edge device impose restrictions on achievable accuracy of LLMs. To overcome the issue, we first propose to leverage multiple user-end devices available at the edge for LLM inference, enabling the execution of larger models. Specifically, we propose Voltron, a novel on-device LLM inference framework that elastically utilizes multiple user-end devices for LLM inference execution while adapting to diverse real-world edge environments. In our evaluation, Voltron achieves up to 16.5% higher accuracy than state-of-the-art LLMs that can be executed on a single edge device, satisfying user QoS requirements.
