Search papers, labs, and topics across Lattice.
This paper introduces Soft Posterior Speaker Injection (SPSI), a novel approach for multi-talker automatic speech recognition (MT-ASR) that enhances performance by predicting frame-level speaker posteriors and integrating them into the Whisper model using feature-wise linear modulation and decoder prompts. The method significantly reduces the constrained permutation word error rate (cpWER) from 50.7% to 49.6% on a two-speaker overlap task, with notable improvements in high-overlap scenarios. Additionally, SPSI outperforms traditional speaker-auxiliary objectives and voice activity detection pipelines, showcasing its effectiveness in challenging overlapping speech conditions.
Reducing cpWER in multi-talker speech recognition by over 1% demonstrates that soft speaker posteriors can effectively enhance model performance without the pitfalls of hard segmentation.
Multi-talker automatic speech recognition (MT-ASR) remains challenging under overlapping speech. Hard diarization-based segmentation introduces irreversible errors, whereas serialized output training (SOT) avoids explicit segmentation but does not condition a pretrained encoder on speaker activity. We propose Soft Posterior Speaker Injection (SPSI): a lightweight head predicts frame-level speaker posteriors $\hat{\mathbf{P}}$ and injects them into Whisper through multi-layer feature-wise linear modulation (FiLM) and decoder speaker-memory prompts. On controlled two-speaker LibriSpeech overlap, SPSI reduces utterance-mean constrained permutation word error rate (cpWER) from 50.7\% (SOT) to 49.6\% (one-sided paired bootstrap $p{\approx}0.006$), with a larger reduction in the high-overlap bin (60.4\%$\to$58.8\%). Same-backbone speaker-auxiliary objectives and voice activity detection (VAD) pipelines do not outperform SOT; zero-shot (ZS) LibriCSS is comparable. Freeze-posterior adaptation with overlap-heavy (OV-heavy) continuation reduces held-out LibriCSS cpWER (sessions 8--9) to 32.4\% (versus 37.5\% for SOT). Ablations indicate complementary encoder FiLM and decoder prompts, and that the effective signal is a \emph{soft} simplex-valued speaker share.
