[Paper Review] Positive Sample Propagation along the Audio-Visual Event Line

Jinxing Zhou, Liang Zheng+3 more|arXiv (Cornell University)|Apr 1, 2021

Music and Audio Processing31 references6 citations

TL;DR

This paper proposes a Positive Sample Propagation (PSP) module that improves audio-visual event (AVE) localization by identifying and aggregating only high-similarity audio-visual segment pairs, filtering out irrelevant or weakly correlated pairs. By constructing an all-pair similarity map and applying a threshold-based selection, PSP enhances feature discriminability, achieving state-of-the-art accuracy of 77.8% under fully supervised and 73.5% under weakly supervised settings on the AVE dataset.

ABSTRACT

Visual and audio signals often coexist in natural environments, forming audio-visual events (AVEs). Given a video, we aim to localize video segments containing an AVE and identify its category. In order to learn discriminative features for a classifier, it is pivotal to identify the helpful (or positive) audio-visual segment pairs while filtering out the irrelevant ones, regardless whether they are synchronized or not. To this end, we propose a new positive sample propagation (PSP) module to discover and exploit the closely related audio-visual pairs by evaluating the relationship within every possible pair. It can be done by constructing an all-pair similarity map between each audio and visual segment, and only aggregating the features from the pairs with high similarity scores. To encourage the network to extract high correlated features for positive samples, a new audio-visual pair similarity loss is proposed. We also propose a new weighting branch to better exploit the temporal correlations in weakly supervised setting. We perform extensive experiments on the public AVE dataset and achieve new state-of-the-art accuracy in both fully and weakly supervised settings, thus verifying the effectiveness of our method.

Motivation & Objective

• Address the challenge of noisy and irrelevant audio-visual segment pairs during feature fusion in weakly supervised AVE localization.
• Improve feature discriminability by focusing only on high-correlation audio-visual pairs, even when temporally unsynchronized.
• Develop a method that generalizes across both fully and weakly supervised settings without requiring strong supervision.
• Reduce interference from negative or weak connections that degrade model performance during audio-visual fusion.
• Enhance temporal modeling by introducing a weighting branch in the weakly supervised setting to emphasize relevant segments.

Proposed method

• Construct an all-pair similarity map between every audio and visual segment to evaluate their cross-modal correlation.
• Apply a threshold-based filtering mechanism to retain only high-similarity audio-visual pairs, discarding low-scoring connections.
• Aggregate features from selected positive pairs in an online fashion to enrich discriminative representations.
• Introduce an audio-visual pair similarity loss in the fully supervised setting to encourage the network to learn highly correlated features for positive pairs.
• Design a weighting branch in the weakly supervised setting to assign temporal importance scores to segment features based on relevance.
• Use a Bi-LSTM followed by the PSP module to model long-range temporal dependencies and refine cross-modal representations.

Experimental results

Research questions

• RQ1 Can filtering out low-similarity audio-visual pairs improve the discriminative power of learned features in AVE localization?
• RQ2 How does positive sample propagation affect model performance when audio and visual segments are temporally unsynchronized?
• RQ3 Can a similarity-based selection mechanism outperform standard feature fusion that includes all pairs, even weakly correlated ones?
• RQ4 Does introducing a dedicated audio-visual pair similarity loss improve feature alignment in the fully supervised setting?
• RQ5 Can a learnable temporal weighting mechanism enhance performance in the weakly supervised setting by emphasizing relevant segments?

Key findings

• The proposed PSP method achieves 77.8% accuracy on the AVE dataset under fully supervised training, surpassing the previous state-of-the-art by 9.2 percentage points.
• In the weakly supervised setting, the method reaches 73.5% accuracy, outperforming the prior state-of-the-art by 6.8 percentage points.
• Visualizations show that PSP selectively retains only a small fraction of high-similarity audio-visual connections, which are strongly aligned with the target event.
• TSNE visualizations confirm that features become significantly better clustered after the PSP module, indicating stronger inter-class separability and intra-class compactness.
• Compared to AVEL, the proposed method attends to broader and more relevant visual regions, such as both the frying pot and chicken, even when audio and visual signals are not synchronized.
• The method outperforms AGSCA, a recent SOTA model, despite using a slightly less accurate backbone, demonstrating the effectiveness of the PSP and system design.