Search papers, labs, and topics across Lattice.
This paper introduces Anysynth, an embedding-free neural synthesizer that utilizes in-context flow matching to achieve zero-shot instrument cloning by directly conditioning a Diffusion Transformer on uncompressed reference audio and target MIDI. By allowing self-attention to dynamically retrieve acoustic details during generation, Anysynth significantly improves audio quality, timbre similarity, and melody adherence compared to existing embedding-based methods. The model's prompt-length scaling feature demonstrates that longer reference prompts enhance timbre fidelity, a capability not present in traditional systems, while the Asymmetric Hierarchical CFG optimizes controllability by decoupling MIDI and reference-timbre guidance.
Anysynth achieves superior zero-shot instrument cloning by dynamically retrieving acoustic details, outperforming traditional embedding-based methods in timbre fidelity and audio quality.
Zero-shot instrument cloning aims to render an arbitrary [Target MIDI] sequence with the acoustic identity of an unseen instrument given only a short [Reference Audio, Reference MIDI] pair. Existing methods rely on pre-trained embeddings (e.g., CLAP) that compress the reference audio into a fixed-length vector, discarding fine-grained acoustic cues essential for faithful timbre reconstruction. We present Anysynth, an embedding-free neural synthesizer based on in-context flow matching. By conditioning a Diffusion Transformer (DiT) directly on the uncompressed reference audio and target MIDI, our model allows self-attention to dynamically retrieve acoustic details at generation time. Experiments show that \tool outperforms embedding-based and auto-regressive baselines in audio quality, timbre similarity, and melody adherence. Notably, the model exhibits prompt-length scaling: longer reference prompts yield steadily better timbre fidelity, a property absent in embedding-based systems. To optimize controllability, we further propose Asymmetric Hierarchical CFG, which structurally decouples MIDI and reference-timbre guidance based on their natural semantic-acoustic dependency. This asymmetric formulation avoids gradient conflicts and improves both note accuracy and timbre fidelity, pushing the boundary of expressive, zero-shot instrument cloning. Demo audios are available at https://anysynth-demo.github.io/
