Search papers, labs, and topics across Lattice.
RAG-Audio addresses the challenge of prior domination in brain-to-audio reconstruction by employing a novel approach that combines fMRI decoding with retrieval of real-audio exemplars to initialize the sampling trajectory of a frozen generator. This method significantly enhances stimulus identification accuracy on the Brain2Music dataset, achieving a 10-way identification rate of 0.40–0.43, compared to 0.14–0.18 for direct generation. Additionally, it reduces the Fréchet Audio Distance from 13.49 to 1.25, indicating a substantial improvement in audio fidelity while maintaining generative capabilities.
Retrieval-guided initialization in RAG-Audio nearly matches the performance of direct retrieval in brain-to-audio tasks, revolutionizing how we reconstruct audio from neural signals.
Brain-to-audio reconstruction is limited by \emph{prior domination}: when a pretrained generator is conditioned on a weak neural signal, it produces realistic but stimulus-inaccurate audio. We introduce RAG-Audio, which decodes fMRI into a semantic audio embedding, retrieves a matching real-audio exemplar, and initializes the frozen generator's sampling trajectory from that exemplar while retaining the decoded embedding as conditioning. On Brain2Music, RAG-Audio improves 10-way stimulus identification from $0.14$--$0.18$ for direct generation, near the $0.10$ chance level, to $0.40$--$0.43$, comparable to retrieval. It also reduces Fr\'echet Audio Distance by roughly an order of magnitude, from $13.49$ to $1.25$ for AudioLDM. RAG-Audio approaches nearest-neighbor retrieval in identification while remaining generative; its higher FAD is expected because retrieval directly replays real audio. An autoregressive negative control, which lacks an initializable latent trajectory, shows no comparable gain, attributing the improvement to trajectory initialization. These results suggest that retrieval-guided initialization can mitigate prior domination in brain-to-audio generation.
