Search papers, labs, and topics across Lattice.
The paper introduces Doppelganger, a benchmark designed to evaluate the ability of audio-conditioned generators to match synthetic sound effects with their corresponding real recordings. The study reveals that while off-the-shelf audio encoders struggle to effectively cross the synthetic-real boundary, training embeddings on pairs of real and synthetic clips significantly enhances matching performance, achieving an 80% accuracy on unseen sound events. Notably, the learned matching is specific to the generator used, indicating that different generators produce distinct audio characteristics that can be reliably detected.
Synthetic sound effects can fool listeners into mistaking them for real recordings nearly 29% of the time, but a generator-specific detector can perfectly distinguish the two.
Audio-conditioned generators now produce synthetic sound effects from real recordings, so the real and synthetic versions of an event increasingly coexist in sound libraries and in the corpora used to train audio models -- yet no benchmark measures whether a representation can match a synthetic clip to the specific real recording it was generated from. I introduce Doppelganger, a benchmark for matching sound effects across the synthetic-real boundary, pairing 10,420 real clips across 34 everyday sound events each with an audio-conditioned synthetic twin, alongside a controlled 7-class corpus. Off-the-shelf audio encoders do not cross the boundary cleanly. Making the embedding ignore the boundary the standard way -- training it on sound-event labels -- works on familiar sounds but backfires on new ones, dropping below the untrained encoder. Training on the pairs instead -- a clip and its own synthetic twin -- generalizes. On sound events held out of training, it recovers the exact real source about 80% of the time (up from 61% untrained; chance 0.03%), whereas no objective meaningfully improves category-level recognition on those unseen events. The learned matching is specific to one generator -- it survives changes to that generator's settings but not a switch to a different generator, and collapses for the text-only generators tested. A human annotation baseline (49 listeners) lands well above chance but below the models on the same trials. Synthetic twins fool people into calling them real about 29% of the time, yet a generator-specific detector separates these audio-conditioned twins from real recordings perfectly.
ETH