[Paper Review] 2.5D Visual Sound

Ruohan Gao, Kristen Grauman|arXiv (Cornell University)|Dec 11, 2018

Speech and Audio Processing53 references4 citations

TL;DR

This paper proposes a 2.5D visual sound framework that converts monaural audio into binaural audio using visual cues from video, leveraging a deep convolutional neural network to infer spatial audio from visual scene configurations. The method enables self-supervised audio-visual representation learning, improving both binaural sound synthesis and audio-visual source separation.

ABSTRACT

Binaural audio provides a listener with 3D sound sensation, allowing a rich perceptual experience of the scene. However, binaural recordings are scarcely available and require nontrivial expertise and equipment to obtain. We propose to convert common monaural audio into binaural audio by leveraging video. The key idea is that visual frames reveal significant spatial cues that, while explicitly lacking in the accompanying single-channel audio, are strongly linked to it. Our multi-modal approach recovers this link from unlabeled video. We devise a deep convolutional neural network that learns to decode the monaural (single-channel) soundtrack into its binaural counterpart by injecting visual information about object and scene configurations. We call the resulting output 2.5D visual sound---the visual stream helps lift the flat single channel audio into spatialized sound. In addition to sound generation, we show the self-supervised representation learned by our network benefits audio-visual source separation. Our video results: this http URL

Motivation & Objective

• To address the scarcity of binaural audio recordings by enabling automatic conversion of monaural audio into spatialized binaural sound.
• To exploit visual frames as implicit spatial cues to recover missing spatial information in single-channel audio.
• To learn a self-supervised audio-visual representation from unlabeled video data for improved downstream audio tasks.
• To demonstrate that visual information can effectively 'lift' flat monaural audio into perceptually rich 3D sound.

Proposed method

• Train a deep convolutional neural network to map monaural audio and corresponding video frames into binaural audio output.
• Use visual features from video frames—such as object positions and scene geometry—to condition the audio synthesis process.
• Leverage the temporal alignment between audio and video to learn joint audio-visual representations in a self-supervised manner.
• Inject visual spatial cues into the audio network via cross-modal attention or feature fusion mechanisms to guide binaural rendering.
• Optimize the network using a reconstruction loss between predicted binaural audio and ground-truth binaural audio (when available), and self-supervision via audio-visual contrastive learning.
• Utilize the learned audio-visual features for downstream tasks such as audio-visual source separation.

Experimental results

Research questions

• RQ1 Can visual information from video be effectively used to infer spatial audio characteristics in monaural audio?
• RQ2 To what extent can a self-supervised model learn meaningful audio-visual representations from unlabeled video data?
• RQ3 How well does the proposed 2.5D visual sound method generalize to unseen audio-visual scenes?
• RQ4 Can the learned representations improve performance in audio-visual source separation tasks?

Key findings

• The proposed method successfully generates perceptually convincing binaural audio from monaural input using only video as input.
• The self-supervised representation learned by the network improves performance in audio-visual source separation tasks.
• Visual cues such as object positions and scene layout significantly enhance the spatial realism of the generated binaural audio.
• The model generalizes well to unseen audio-visual pairs, demonstrating robustness to variations in scene configuration and audio content.