[Paper Review] Generating Visually Aligned Sound from Videos

Peihao Chen, Yang Zhang+4 more|arXiv (Cornell University)|Jul 14, 2020

Speech and Audio Processing42 references4 citations

TL;DR

This paper proposes RegNet, a novel framework for generating visually aligned sound from videos by introducing an audio forwarding regularizer that disentangles relevant from irrelevant sound components during training. By leveraging ground-truth audio features through a bottlenecked encoder, RegNet improves both temporal and content-wise alignment, achieving a 68.12% human success rate in fooling observers on generated sound.

ABSTRACT

We focus on the task of generating sound from natural videos, and the sound should be both temporally and content-wise aligned with visual signals. This task is extremely challenging because some sounds generated \emph{outside} a camera can not be inferred from video content. The model may be forced to learn an incorrect mapping between visual content and these irrelevant sounds. To address this challenge, we propose a framework named REGNET. In this framework, we first extract appearance and motion features from video frames to better distinguish the object that emits sound from complex background information. We then introduce an innovative audio forwarding regularizer that directly considers the real sound as input and outputs bottlenecked sound features. Using both visual and bottlenecked sound features for sound prediction during training provides stronger supervision for the sound prediction. The audio forwarding regularizer can control the irrelevant sound component and thus prevent the model from learning an incorrect mapping between video frames and sound emitted by the object that is out of the screen. During testing, the audio forwarding regularizer is removed to ensure that REGNET can produce purely aligned sound only from visual features. Extensive evaluations based on Amazon Mechanical Turk demonstrate that our method significantly improves both temporal and content-wise alignment. Remarkably, our generated sound can fool the human with a 68.12% success rate. Code and pre-trained models are publicly available at https://github.com/PeihaoChen/regnet

Motivation & Objective

• To address the challenge of generating sound that is both temporally and content-wise aligned with visual content in videos.
• To resolve the information mismatch issue caused by visually irrelevant sounds that cannot be inferred from video frames.
• To improve alignment performance by introducing a supervision mechanism that isolates relevant sound components during training.
• To enable the model to generate high-fidelity, realistic sound purely from visual features during inference.

Proposed method

• RegNet uses a time-dependent appearance and motion feature extraction module to capture visual dynamics for sound prediction.
• An audio forwarding regularizer encodes ground-truth audio into bottlenecked features, which are then fused with visual features during training.
• The regularizer provides strong supervision by enabling the generator to reconstruct ground-truth sound using both visual and audio bottleneck features.
• During inference, the audio forwarding regularizer is removed, allowing sound generation to rely solely on visual features.
• The model is trained with L2 loss for spectrogram reconstruction and adversarial loss via a discriminator to improve audio fidelity.
• RegNet variants are evaluated with controlled capacities (via downsampling rate S and output dimension D) to study the impact of regularizer capacity on performance.

Experimental results

Research questions

• RQ1 Why is temporal and content-wise alignment particularly challenging in video sound generation compared to speech synthesis?
• RQ2 How does the presence of visually irrelevant sound in training data degrade alignment performance in existing models?
• RQ3 Can an audio forwarding regularizer effectively suppress irrelevant sound components and improve alignment during training?
• RQ4 How does the capacity of the audio forwarding regularizer affect the model's ability to learn correct visual-to-sound mappings?
• RQ5 Does adversarial training enhance the perceptual quality and realism of generated sounds in the RegNet framework?

Key findings

• RegNet achieves a 68.12% human success rate in fooling observers on the 'real-or-fake' test, outperforming baseline models.
• The optimal RegNet variant (S860D32) achieves the lowest L2 distance (1.96) on Dog-gauss data, outperforming both narrow (D8) and wide (D1024) variants.
• The wide variant (S860D1024) suffers from poor Sound-Missing scores, indicating failure to generate sound during action events due to over-reliance on the regularizer.
• The narrow variant (S860D8) produces noisy spectrograms, suggesting it learns incorrect mappings due to insufficient information from the regularizer.
• Removing the adversarial discriminator reduces the human success rate to 64.04%, demonstrating its role in improving audio fidelity.
• Subjective evaluations show RegNet achieves the highest mean opinion scores across all three alignment metrics (Sound-Missing, Sound-Redundant, Sound-Mismatched).