Search papers, labs, and topics across Lattice.
This paper introduces a method to enhance text-to-audio-video (T2AV) generation models by integrating voice cloning capabilities through the addition of a single zero-initialized linear layer. By fine-tuning the modified model on a short training schedule and conditioning on reference audio, the authors achieve superior speaker-encoder cosine similarity (SECS) compared to five strong voice-cloning baselines across a diverse benchmark of 674 speaker-text pairs. Notably, the architecture allows for significant speed improvements, enabling audio evaluation independently of the video path, achieving a ~30x acceleration without sacrificing performance.
A single linear layer transforms T2AV models into effective voice-cloning systems, achieving record-breaking speaker similarity while slashing inference time by ~30x.
Text-to-audio-video (T2AV) generation models produce a video and its soundtrack from a textual description, but offer no control over whose voice speaks in the output. We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-initialized linear layer on top of its audio backbone, fine-tuning for a comparatively short training schedule, and conditioning on a short reference recording at inference time. The reference is injected through two complementary signals: its diffusion latents are prepended to the audio stream, and a global speaker embedding modulates token of the target audio. On a benchmark of 674 speaker-text pairs spanning 30 speakers we compare against five strong voice-cloning text-to-speech baselines: our enhanced 5B model attains the highest speaker-encoder cosine similarity (SECS) across three independent verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer), statistically significantly outperforming every baseline. A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a ~30x speed-up over the full audio-video diffusion loop while preserving the voice-cloning behaviour.
