Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces PINT (Parallel INvariant Tokenization), a method that fine-tunes a self-supervised learning encoder to align tokens across parallel utterances, effectively isolating linguistic content from non-linguistic variations such as speaker identity and prosody. By leveraging multiple speakers and varying conditions, PINT achieves a significant reduction in conditional entropy, resulting in more consistent token sequences for identical words. The method demonstrates a 98.7% reduction in speaker probe accuracy and lower error rates compared to existing baselines, underscoring the importance of focusing on linguistic invariance for efficient speech tokenization.
Achieving a 98.7% reduction in speaker identification accuracy reveals how focusing on linguistic content can drastically enhance speech tokenization efficiency.
Discrete speech tokenizers aim to disentangle semantic from acoustic information, yet targets from self-supervised learning (SSL) models like HuBERT retain non-linguistic variation: speaker identity, prosody, and channel conditions leak into the tokens, inflating entropy. Our key insight is that when enough speakers utter the same words under varying conditions, linguistic content is the only shared factor. We propose PINT (Parallel INvariant Tokenization), which fine-tunes an SSL encoder with alignment losses across parallel utterances and augmentations to distill this shared residual. PINT collapses identical words onto consistent token sequences, drastically reducing conditional entropy. Unlike ASR text, PINT tokens preserve frame-level temporal grounding and serve as drop-in semantic targets for audio codecs. Experiments show a 98.7% relative reduction in speaker probe accuracy (93.1% to 1.2%), a 42% lower ABX error rate, and 27-30% lower LM perplexity versus baselines, confirming that the right invariance is key to efficient learning.