Search papers, labs, and topics across Lattice.
This paper introduces TiCodec, a neural speech codec that utilizes a Time-Invariant Representation Extraction (TIRE) module to separate time-varying speech content from time-invariant information, thereby enhancing low-latency speech processing. Through probing tasks, the authors reveal that intermediate encoder layers capture complementary information related to speakers and environments, while linguistic content is minimized. The proposed Dual-TIRE architecture leverages these insights, demonstrating improved speech reconstruction quality and speaker similarity in a streaming inference setting without significant performance degradation.
Factorized neural codec representations can achieve low-latency speech generation without sacrificing reconstruction quality.
Neural speech codecs are increasingly used as intermediate representations in codec-based speech generation systems. TiCodec introduces a factorized representation that separates time-varying speech content from time-invariant information through a Time-Invariant Representation Extraction (TIRE) module, potentially reducing the amount of information that must be modeled at the frame-level. In this work, we investigate the nature of the information captured by TIRE representations and their suitability for low-latency speech processing. Using a series of probing tasks, we analyze the influence of the encoder layer and show that intermediate layers capture complementary speaker- and environment-related information while containing little linguistic content. We further study several segment selection strategies for TIRE training and demonstrate that cross-file sampling improves the robustness of invariant representations. Based on these findings, we propose Dual-TIRE, a multi-level architecture that exploits the complementarity of different encoder layers and improves speech reconstruction quality and speaker similarity. Finally, we evaluate TiCodec in a streaming inference setting using successive 660ms processing blocks. Results show that streaming operation can be achieved without significant degradation in reconstruction performance, highlighting the potential of factorized neural codec representations for future low-latency speech generation systems.
