Search papers, labs, and topics across Lattice.
This paper introduces diarization-conditioned spoken language models (SLMs) to enhance the performance of SLMs in multi-talker audio environments by conditioning the acoustic encoder on diarization masks while keeping the decoder frozen. The proposed model, Dixtral, incorporates a Diarization Conditioned Whisper (DiCoW) encoder and demonstrates significant improvements in speaker-attributed transcription accuracy across multiple datasets, outperforming existing models such as Gemini 3.0 Flash and Voxtral Mini Transcribe V2. Notably, Dixtral achieves competitive performance in long-form multi-speaker question-answering tasks, matching and even surpassing leading models in zero-shot and fine-tuned scenarios.
Dixtral achieves up to 29% absolute improvement in speaker-attributed transcription accuracy by leveraging diarization masks without risking catastrophic forgetting.
We propose diarization-conditioned spoken language models (SLMs), a strategy for extending SLMs to far-field multi-talker audio. Rather than adapting the decoder via Serialized Output Training, which risks catastrophic forgetting, we condition the acoustic encoder on diarization masks to extract target-speaker representations, keeping the decoder frozen. We instantiate this as Dixtral, integrating a Diarization Conditioned Whisper (DiCoW) encoder into the Voxtral SLM. On AMI, NOTSOFAR-1, LibriSpeechMix, and Mixer6, Dixtral outperforms Gemini 3.0 Flash, VibeVoice, and Voxtral Mini Transcribe V2 on speaker-attributed transcription by 29.0%, 19.8%, and 16.0% absolute cpWER respectively. On a novel long-form multi-speaker QA benchmark, zero-shot Dixtral matches Gemini on far-field content understanding, and when fine-tuned surpasses both Gemini and Voxtral operating on close-talk across all tasks.
CMU ML