[Paper Review] Multiple Sound Sources Localization from Coarse to Fine

Rui Qian, Di Hu+4 more|arXiv (Cornell University)|Jul 13, 2020

Speech and Audio Processing30 references8 citations

TL;DR

This paper proposes a two-stage audiovisual learning framework for multiple sound source localization in unconstrained videos, disentangling audio and visual representations via multi-task learning and Class Activation Mapping (CAM), then aligning them in a coarse-to-fine manner. It achieves state-of-the-art performance on public datasets and enables effective sound separation using localized visual cues.

ABSTRACT

How to visually localize multiple sound sources in unconstrained videos is a formidable problem, especially when lack of the pairwise sound-object annotations. To solve this problem, we develop a two-stage audiovisual learning framework that disentangles audio and visual representations of different categories from complex scenes, then performs cross-modal feature alignment in a coarse-to-fine manner. Our model achieves state-of-the-art results on public dataset of localization, as well as considerable performance on multi-source sound localization in complex scenes. We then employ the localization results for sound separation and obtain comparable performance to existing methods. These outcomes demonstrate our model's ability in effectively aligning sounds with specific visual sources. Code is available at https://github.com/shvdiwnkozbw/Multi-Source-Sound-Localization

Motivation & Objective

• To address the challenge of localizing multiple sound sources in complex, unconstrained videos without pairwise sound-object annotations.
• To disentangle audio and visual representations of different categories from cluttered scenes for improved alignment.
• To establish accurate audiovisual correspondence in multi-source scenarios where video-level supervision is too coarse.
• To develop a novel evaluation pipeline for multi-source sound localization that measures class-specific localization performance.
• To leverage localization results for effective sound source separation using visual guidance.

Proposed method

• A two-stage framework is proposed: first, a multi-task learning setup jointly performs audiovisual classification and correspondence learning.
• Class Activation Mapping (CAM) is applied to extract category-specific visual features, enabling disentanglement of sound-source objects from complex scenes.
• Coarse-to-fine audiovisual alignment is performed: category-level correspondence is refined into video- and category-level associations.
• Visual representations from localized objects are used as guidance for sound source separation in downstream tasks.
• The method uses weighted pooling to disentangle audio representations from mixed spectrograms, improving source-specific feature learning.
• The framework is trained end-to-end with self-supervision from temporal synchronization and category-level labels.

Experimental results

Research questions

• RQ1 Can audiovisual representations be effectively disentangled from complex scenes without one-to-one annotations?
• RQ2 How can coarse video-level supervision be leveraged to achieve fine-grained sound-source localization in multi-source environments?
• RQ3 Can a coarse-to-fine alignment strategy improve audiovisual correspondence accuracy in unconstrained videos?
• RQ4 To what extent can localization results enhance sound source separation performance?
• RQ5 How does the proposed evaluation metric reflect performance in multi-source localization compared to existing baselines?

Key findings

• The proposed method achieves state-of-the-art performance on public sound localization datasets, especially in multi-source scenarios.
• On the AudioSet level-2 benchmark, the model precisely localizes sound sources, e.g., correctly identifying the flute while avoiding mislocalization to the harp, unlike AVC and multi-task baselines.
• When used as visual guidance, the model's localization results achieve comparable sound separation performance to existing methods, with SDR of 6.53 and SIR of 12.15 on solo videos and SDR of 6.57 and SIR of 11.90 on multi-source (solo+duet) training.
• The visual representation-based guidance outperforms both Grad-CAM and audio representation-based guidance, which suffer from low resolution and limited disentanglement.
• The model achieves strong results with fewer audiovisual pairs than competing methods and does not require additional detectors for sound source extraction.
• The proposed evaluation pipeline enables fair and detailed assessment of class-specific localization performance in complex scenes.