Search papers, labs, and topics across Lattice.
This paper introduces micro language models ($\mu$LMs), ultra-compact models (8M-30M parameters), designed to generate the initial 4-8 words of a response on edge devices, masking cloud latency by enabling a collaborative generation framework with larger cloud models. The authors demonstrate that these $\mu$LMs can match the performance of 70M-256M parameter models in initiating responses. They also introduce error correction methods for seamless mid-sentence handoffs between the $\mu$LM and the cloud model.
Instant AI assistants are now feasible on smartwatches: 8M-parameter models can kickstart responses locally, hiding cloud latency with surprisingly high quality.
Edge devices such as smartwatches and smart glasses cannot continuously run even the smallest 100M-1B parameter language models due to power and compute constraints, yet cloud inference introduces multi-second latencies that break the illusion of a responsive assistant. We introduce micro language models ($\mu$LMs): ultra-compact models (8M-30M parameters) that instantly generate the first 4-8 words of a contextually grounded response on-device, while a cloud model completes it; thus, masking the cloud latency. We show that useful language generation survives at this extreme scale with our models matching several 70M-256M-class existing models. We design a collaborative generation framework that reframes the cloud model as a continuator rather than a respondent, achieving seamless mid-sentence handoffs and structured graceful recovery via three error correction methods when the local opener goes wrong. Empirical results show that $\mu$LMs can initiate responses that larger models complete seamlessly, demonstrating that orders-of-magnitude asymmetric collaboration is achievable and unlocking responsive AI for extremely resource-constrained devices. The model checkpoint and demo are available at https://github.com/Sensente/micro_language_model_swen_project.
