[Paper Review] Unsupervised Sound Localization via Iterative Contrastive Learning

Yan-Bo Lin, Hung-Yu Tseng+3 more|arXiv (Cornell University)|Apr 1, 2021

Speech and Audio Processing46 references4 citations

TL;DR

This paper proposes an unsupervised sound localization framework using iterative contrastive learning that leverages pseudo-labels from predicted sounding/non-sounding regions across training epochs and inter-frame audio relationships to improve localization without annotations. The method achieves state-of-the-art performance on Flickr-Sound and MUSIC-Synthetic datasets using only self-supervised signals.

ABSTRACT

Sound localization aims to find the source of the audio signal in the visual scene. However, it is labor-intensive to annotate the correlations between the signals sampled from the audio and visual modalities, thus making it difficult to supervise the learning of a machine for this task. In this work, we propose an iterative contrastive learning framework that requires no data annotations. At each iteration, the proposed method takes the 1) localization results in images predicted in the previous iteration, and 2) semantic relationships inferred from the audio signals as the pseudo-labels. We then use the pseudo-labels to learn the correlation between the visual and audio signals sampled from the same video (intra-frame sampling) as well as the association between those extracted across videos (inter-frame relation). Our iterative strategy gradually encourages the localization of the sounding objects and reduces the correlation between the non-sounding regions and the reference audio. Quantitative and qualitative experimental results demonstrate that the proposed framework performs favorably against existing unsupervised and weakly-supervised methods on the sound localization task.

Motivation & Objective

• Address the challenge of limited annotated data for sound localization, where manual pairing of audio and visual signals is labor-intensive.
• Overcome limitations of existing weakly-supervised and self-supervised methods that rely on pre-defined clusters, single-source assumptions, or event-level labels.
• Develop a fully unsupervised framework that learns audio-visual correspondence without any ground-truth bounding boxes or event annotations.
• Improve localization accuracy by jointly modeling intra-frame (within-video) and inter-frame (cross-video) relationships between audio and visual signals.

Proposed method

• Use localization results from the previous training epoch as pseudo-labels for sounding and non-sounding regions to guide contrastive learning in the current epoch.
• Apply intra-frame sampling by treating predicted sounding regions as positive pairs and non-sounding regions as negative pairs within the same video frame.
• Introduce inter-frame relation by comparing audio embeddings across different videos; semantically similar audio clips are treated as positive pairs.
• Leverage a contrastive loss function that encourages alignment between audio and visual features within the same video and across videos based on audio similarity.
• Iteratively refine the model by updating pseudo-labels and representation learning across epochs, gradually focusing attention on true sounding objects.
• Use a backbone network (e.g., MSSL-based) with feature normalization and thresholding to generate reliable pseudo-labels from class activation maps.

Experimental results

Research questions

• RQ1 Can unsupervised sound localization be effectively achieved without any human-annotated data or prior assumptions about the number of sound sources?
• RQ2 How does iterative refinement using pseudo-labels from previous epochs improve localization accuracy compared to single-stage contrastive learning?
• RQ3 To what extent can inter-frame audio relationships—based on semantic similarity—enhance cross-video alignment in audio-visual representation learning?
• RQ4 Does jointly modeling both sounding and non-sounding regions improve the model's ability to suppress false positives and background noise?

Key findings

• The proposed iterative contrastive learning framework achieves state-of-the-art performance on the Flickr-Sound and MUSIC-Synthetic datasets without any annotations.
• The ablation study confirms that each component—iterative training, use of pseudo-non-sounding regions, and inter-frame relation—significantly improves localization performance.
• Localization results progressively focus on true sounding regions across training epochs, as visualized in Figure 6, demonstrating the effectiveness of iterative refinement.
• Audio-based retrieval results (Figure 7) validate that semantically similar audio clips across videos are correctly associated with visually similar scenes, confirming the utility of inter-frame audio relationships.
• The method outperforms existing weakly-supervised and unsupervised baselines in terms of consensus IoU and AUC metrics on benchmark datasets.
• Replacing the MSSL method’s second stage with the proposed unsupervised training leads to significant performance gains, even without event-level labels.