Search papers, labs, and topics across Lattice.
This study investigates the compression of a tokenizer used in on-device speech transcription for Apple devices through a novel approach called latent-space distillation. By training a student encoder to replicate the pre-quantizer latent representation from a teacher model, the researchers achieve a significant 2.8x reduction in parameter count while maintaining a relative word error rate (WER) within 1.9% of the original model. This method not only enhances efficiency but also outperforms an independently trained tokenizer of the same size by 3.9% in terms of transcription accuracy.
Achieving 2.8x compression in on-device speech tokenizers without sacrificing accuracy could revolutionize mobile AI applications.
System-wide Dictation on Apple devices runs entirely on-device, and the speech it transcribes reaches the foundation model through a tokenizer: an encoder that maps short windows of waveform onto the representation the language model reads. Because that model is sparsely activated under Instruction-Following Pruning, only a small subset of its experts occupies DRAM at any time, so the always-on tokenizer competes for the same memory, and its parameter count bears directly on power and latency. In this work we study how to compress such a tokenizer by distillation, taking as the supervision target neither the discrete token nor the output distribution but the pre-quantizer latent the model actually consumes - the last representation the two token interfaces share. We train only the student encoder to regress the teacher's per-frame latent under a squared-error objective, with a single affine layer absorbing the teacher-student width mismatch. Because the target precedes both the quantizer and the language-model bridge, one recipe covers both token interfaces we support, and applies both to a tokenizer pretrained alone and to one jointly trained with a language model. At 2.8x compression the distilled student stays within 1.9% relative WER of its teacher on five of six teacher-student pairs without any fine-tuning, and improves on an independently trained tokenizer of identical capacity by 3.9% relative.
