Search papers, labs, and topics across Lattice.
This paper investigates speaker privacy leakage in end-to-end full-duplex speech dialogue models (SALM-Duplex and Moshi) using the VoicePrivacy 2024 protocol. It finds significant speaker identity leakage in the hidden states of these models across all transformer layers and turns. The authors then propose and evaluate two streaming anonymization methods, Anon-W2W (waveform-level) and Anon-W2F (feature-domain), demonstrating that Anon-W2F effectively reduces speaker identification while Anon-W2W maintains semantic content with low latency.
Turns out your always-on speech dialogue model is leaking speaker identity like a sieve, but a simple feature-domain anonymization technique can boost privacy by 3.5x with minimal impact on performance.
End-to-end full-duplex speech models feed user audio through an always-on LLM backbone, yet the speaker privacy implications of their hidden representations remain unexamined. Following the VoicePrivacy 2024 protocol with a lazy-informed attacker, we show that the hidden states of SALM-Duplex and Moshi leak substantial speaker identity across all transformer layers. Layer-wise and turn-wise analyses reveal that leakage persists across all layers, with SALM-Duplex showing stronger leakage in early layers while Moshi leaks uniformly, and that Linkability rises sharply within the first few turns. We propose two streaming anonymization setups using Stream-Voice-Anon: a waveform-level front-end (Anon-W2W) and a feature-domain replacement (Anon-W2F). Anon-W2F raises EER by over 3.5x relative to the discrete encoder baseline (11.2% to 41.0%), approaching the 50% random-chance ceiling, while Anon-W2W retains 78-93% of baseline sBERT across setups with sub-second response latency (FRL under 0.8 s).
