[Paper Review] Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from fMRI

Che Liu, Changde Du+2 more|arXiv (Cornell University)|May 29, 2024

Hearing Loss and Rehabilitation4 citations

TL;DR

This paper proposes a coarse-to-fine audio reconstruction framework from fMRI that mirrors the human auditory system's hierarchical processing: first decoding fMRI into low-dimensional CLAP semantic space, then refining it into AudioMAE latent space for fine-grained acoustic features, which condition a Latent Diffusion Model to generate high-quality audio. The method achieves state-of-the-art performance across three datasets (Brain2Sound, Brain2Music, Brain2Speech) in FD, FAD, and KL metrics, with semantic prompting further enhancing reconstruction quality when semantic decoding is suboptimal.

ABSTRACT

Drawing inspiration from the hierarchical processing of the human auditory system, which transforms sound from low-level acoustic features to high-level semantic understanding, we introduce a novel coarse-to-fine audio reconstruction method. Leveraging non-invasive functional Magnetic Resonance Imaging (fMRI) data, our approach mimics the inverse pathway of auditory processing. Initially, we utilize CLAP to decode fMRI data coarsely into a low-dimensional semantic space, followed by a fine-grained decoding into the high-dimensional AudioMAE latent space guided by semantic features. These fine-grained neural features serve as conditions for audio reconstruction through a Latent Diffusion Model (LDM). Validation on three public fMRI datasets-Brain2Sound, Brain2Music, and Brain2Speech-underscores the superiority of our coarse-to-fine decoding method over stand-alone fine-grained approaches, showcasing state-of-the-art performance in metrics like FD, FAD, and KL. Moreover, by employing semantic prompts during decoding, we enhance the quality of reconstructed audio when semantic features are suboptimal. The demonstrated versatility of our model across diverse stimuli highlights its potential as a universal brain-to-audio framework. This research contributes to the comprehension of the human auditory system, pushing boundaries in neural decoding and audio reconstruction methodologies.

Motivation & Objective

• To address the challenge of reconstructing high-fidelity, semantically accurate audio from non-invasive fMRI by mimicking the brain's hierarchical auditory processing pathway.
• To improve brain-to-audio reconstruction by decoupling semantic and acoustic decoding, avoiding direct high-dimensional fMRI-to-audio mapping.
• To enhance reconstruction quality when semantic features are weak or noisy by incorporating conditional prompts during decoding.
• To establish a universal brain-to-audio framework applicable across diverse audio modalities (speech, music, environmental sounds).

Proposed method

• The method begins by decoding fMRI data into a low-dimensional CLAP embedding space to extract coarse-grained semantic features.
• These semantic features guide a subsequent decoding of fMRI into the high-dimensional AudioMAE latent space, capturing fine-grained acoustic representations.
• The decoded AudioMAE features are used as conditional inputs to a Latent Diffusion Model (LDM) for mel-spectrogram generation.
• The generated mel-spectrograms are then converted into waveform using a vocoder to produce the final audio output.
• Conditional reconstruction is enabled via text or audio prompts, with hyperparameter $P_{gt}$ controlling the influence of ground-truth semantic features during training.
• The framework is validated on three public fMRI datasets: Brain2Sound, Brain2Music, and Brain2Speech, using metrics including FD, FAD, KL, and PCC.

Experimental results

Research questions

• RQ1 Can a coarse-to-fine decoding pipeline improve audio reconstruction quality from fMRI compared to direct fine-grained decoding?
• RQ2 How does leveraging semantic features from CLAP improve the fidelity and semantic accuracy of reconstructed audio?
• RQ3 To what extent can semantic prompts enhance reconstruction when the decoded semantic features are suboptimal?
• RQ4 Does the proposed method generalize across diverse audio types, including speech, music, and environmental sounds?

Key findings

• The coarse-to-fine method achieved state-of-the-art performance on all three datasets, with FD scores of 6.102 (Brain2Music), 11.636 (Brain2Speech), and 6.587 (Brain2Sound), and FAD scores of 1.504, 4.866, and 1.341, respectively.
• The method outperformed standalone fine-grained decoding in both perceptual and semantic quality, as evidenced by lower KL and KL-S values and higher PCC scores.
• Conditional reconstruction with $P_{gt} = 0.25$ yielded the highest PCC (0.454 on Brain2Music and 0.393 on Brain2Speech), indicating optimal balance between guidance and generalization.
• In the Brain2Speech dataset, where semantic decoding was weaker, audio prompts improved PCC from 0.379 to 0.315 and reduced FAD from 4.866 to 4.148, demonstrating the effectiveness of prompting in low-signal regimes.
• Text prompts in the Brain2Music dataset degraded performance due to loss of individuality among clips of the same genre, indicating that prompt design must align with semantic diversity.
• The integration of semantic prompts significantly enhanced reconstruction quality when semantic features were suboptimal, validating the method’s robustness and adaptability.