Search papers, labs, and topics across Lattice.
This study tackles the challenge of poor generalization in deepfake voice detection by integrating Audio Large Language Models (ALLMs) with a novel cross-modal prompting strategy that incorporates acoustic features as structured text tokens. The researchers found that using a frozen LLM as a baseline outperformed fine-tuning due to reduced risk of out-of-domain overfitting, leading to significant improvements in detection accuracy. Their approach achieved a 16.2% absolute improvement in Macro-F1 scores on out-of-domain benchmarks while maintaining competitive performance on in-domain tasks.
Bridging the gap between audio embeddings and LLMs, this method achieves a remarkable 16.2% boost in deepfake voice detection accuracy on unseen domains.
Deepfake voice detection suffers from poor generalization across unseen domains. While Audio Large Language Models (ALLMs) show promise, the modality gap between continuous audio embeddings which capture the subtle acoustic details necessary for deepfake detection and the semantic space of LLMs remains a critical, underexplored bottleneck. We address this by benchmarking diverse audio encoders integrated with Qwen LLMs (0.5B to 7B parameters). First, we demonstrate that fine-tuning the LLM alone risks out-of-domain overfitting, making a frozen LLM a stronger, resource-efficient baseline. Second, to explicitly bridge the modality gap, we introduce a cross-modal prompting strategy that injects linguistic-knowledge-driven acoustic features (via openSMILE) as structured text tokens. This explicit textual grounding not only enhances the frozen baseline but also makes LLM fine-tuning more effective. Ultimately, our approach demonstrates state-of-the-art resilience on the out-of-domain ITW and MLAAD benchmarks, yielding over \textbf{16.2\%} absolute improvement in Macro-F1 over existing ALLM baselines while maintaining competitive in-domain performance. All models reported in this work are \href{https://huggingface.co/01Yassine/AudioLLM-Deepfake-Detection}{publicly available}.
