Search papers, labs, and topics across Lattice.
This paper introduces BiMTokenizer, a low-bitrate speech codec that leverages a bidirectional state-space model combined with Residual Spherical Leech Quantization (RSLQ) to effectively balance acoustic fidelity and semantic preservation. By revisiting the single-tower architecture, BiMTokenizer achieves superior acoustic reconstruction and the lowest word error rate (WER) among low-bitrate codecs, while utilizing less than half the parameters of dual-tower designs. The results demonstrate that it can maintain semantic integrity in speech understanding tasks, challenging the assumption that more complex architectures are necessary for high performance.
A single-tower speech codec can outperform complex dual-tower architectures, achieving low-bitrate efficiency without sacrificing semantic quality.
Speech codecs serve as bridges between continuous speech signals and large language models, yet face an inherent conflict between acoustic fidelity and semantic preservation. To mitigate this conflict, recent works increasingly adopt dual-tower architectures to decouple semantic and acoustic modeling with separate encoders. However, these dual-tower designs incur substantial architectural overhead. To avoid such complexity, we revisit the single-tower paradigm and propose BiMTokenizer, a low-bitrate speech codec (around 1.1 kbps) combining a bidirectional state-space backbone with Residual Spherical Leech Quantization (RSLQ). The bidirectional backbone strengthens temporal modeling, while RSLQ offers a fixed, well-separated lattice bottleneck for robust semantic and acoustic tokenization without learned-codebook collapse. Experiments show that BiMTokenizer achieves superior acoustic reconstruction and the lowest WER among low-bitrate codec baselines across both clean and noisy environments, while using less than half the parameters of recent dual-tower baselines. Furthermore, its robust semantic representations yield strong performance on downstream speech understanding tasks, confirming that a well-designed single-tower codec can preserve the semantic-acoustic balance at low bitrates. The code and model weights are available at https://github.com/ZhangXinWhut/BiMTokenizer.
