[Paper Review] Soundify: Matching Sound Effects to Video

David Chuan-En Lin, Anastasis Germanidis+3 more|arXiv (Cornell University)|Dec 17, 2021

Music and Audio Processing35 references4 citations

TL;DR

Soundify is a novel system that leverages CLIP's activation maps to automatically detect, synchronize, and spatialize high-quality sound effects from professional libraries to video footage, significantly reducing workload and task completion time while improving usability in expert evaluations. It enables zero-shot sound matching without training audio generation models, outperforming manual editing in both preference and efficiency.

ABSTRACT

In the art of video editing, sound helps add character to an object and immerse the viewer within a space. Through formative interviews with professional editors (N=10), we found that the task of adding sounds to video can be challenging. This paper presents Soundify, a system that assists editors in matching sounds to video. Given a video, Soundify identifies matching sounds, synchronizes the sounds to the video, and dynamically adjusts panning and volume to create spatial audio. In a human evaluation study (N=889), we show that Soundify is capable of matching sounds to video out-of-the-box for a diverse range of audio categories. In a within-subjects expert study (N=12), we demonstrate the usefulness of Soundify in helping video editors match sounds to video with lighter workload, reduced task completion time, and improved usability.

Motivation & Objective

• To address the time-consuming and challenging task of manually matching sound effects to video in professional editing workflows.
• To develop a system that leverages existing studio-quality sound libraries instead of training audio generation models from scratch.
• To enable zero-shot sound matching by repurposing CLIP, an image-text embedding model, for audio-visual correspondence detection.
• To dynamically synchronize and spatialize sound effects (panning/volume) based on video content using activation map analysis.
• To evaluate the system's effectiveness and usability through large-scale human evaluation and expert studies.

Proposed method

• Repurpose CLIP, originally trained for image-text classification, as a zero-shot sound detector by analyzing its internal activation maps for audio-visual correspondence.
• Use the spatial coordinates of CLIP's activation maps to determine optimal panning (left-right positioning) of sound effects in the stereo field.
• Use the area of activation maps to estimate the relative volume of each sound effect, ensuring dynamic loudness based on visual salience.
• Retrieve matching sound clips from a curated library of professional-grade sound effects using label matching between video content and sound clip metadata.
• Automatically synchronize sound clips to video by aligning their temporal onset with the detected visual event in the video frame.
• Support stacking of multiple sound layers (e.g., foreground effects and ambient sounds) to create complex, immersive audio landscapes.

Experimental results

Research questions

• RQ1 Can a zero-shot approach using CLIP's activation maps effectively detect relevant sound effects in video without fine-tuning or audio generation training?
• RQ2 How well does Soundify perform in automatically matching diverse sound effects to complex video content compared to manual baseline methods?
• RQ3 To what extent does Soundify reduce workload, task completion time, and improve usability in professional video editing workflows?
• RQ4 How do professional editors perceive the accuracy, intuitiveness, and practicality of Soundify in real-world editing scenarios?

Key findings

• In a large-scale human evaluation with 889 raters, Soundify’s results were significantly preferred over the baseline in matching sounds to video across diverse audio categories.
• In a within-subjects expert study with 12 professional editors, Soundify reduced task completion time by a statistically significant margin compared to manual editing in Adobe Premiere Pro.
• Participants reported significantly lower workload when using Soundify (p=0.003), indicating a measurable reduction in cognitive and physical effort.
• Usability was significantly higher with Soundify (mean=6.03 vs. 4.70 on a 7-point scale, p=0.003), with users finding it easy to learn and use without instructions.
• Editors appreciated features such as automatic scene splitting, per-scene audio export, and dynamic panning, which eliminated repetitive keyframing in traditional DAWs.
• The system was effective in identifying relevant sound effects without keyword searches, with users highlighting the elimination of 'treasure hunting' for sounds as a major benefit.