Search papers, labs, and topics across Lattice.
This paper introduces a hardware-aware framework for deploying a LLaMA-based multilingual LLM on Samsung Galaxy S24/S25 devices, enabling multiple use cases via application-specific LoRAs loaded at runtime. They achieve dynamic task switching and multi-stream decoding for stylistic variations within a single forward pass, reducing latency. By incorporating Dynamic Self-Speculative Decoding (DS2D) and INT4 quantization, the system achieves 4-6x improvements in memory and latency while maintaining accuracy across 9 languages and 8 tasks.
Run a single LLaMA model on your phone that speaks 9 languages, handles 8 tasks, and generates text in multiple styles *simultaneously* with minimal latency.
Deploying large language models (LLMs) on smartphones poses significant engineering challenges due to stringent constraints on memory, latency, and runtime flexibility. In this work, we present a hardware-aware framework for efficient on-device inference of a LLaMA-based multilingual foundation model supporting multiple use cases on Samsung Galaxy S24 and S25 devices with SM8650 and SM8750 Qualcomm chipsets respectively. Our approach integrates application-specific LoRAs as runtime inputs to a single frozen inference graph, enabling dynamic task switching without recompilation or memory overhead. We further introduce a multi-stream decoding mechanism that concurrently generates stylistic variations - such as formal, polite, or jovial responses - within a single forward pass, reducing latency by up to 6x. To accelerate token generation, we apply Dynamic Self-Speculative Decoding (DS2D), a tree-based strategy that predicts future tokens without requiring a draft model, yielding up to 2.3x speedup in decode time. Combined with quantization to INT4 and architecture-level optimizations, our system achieves 4-6x overall improvements in memory and latency while maintaining accuracy across 9 languages and 8 tasks. These results demonstrate practical feasibility of deploying multi-use-case LLMs on edge devices, advancing the commercial viability of Generative AI in mobile platforms.
