[Paper Review] Multi-granularity Correspondence Learning from Long-term Noisy Videos

Yijie Lin, Jie Zhang+4 more|arXiv (Cornell University)|Jan 30, 2024

Speech and Audio Processing4 citations

TL;DR

This paper proposes Norton, a noise-robust optimal transport framework that unifies multi-granularity correspondence learning for long-term video understanding, addressing both coarse-grained (clip-caption) and fine-grained (frame-word) misalignments through temporal optimal transport, soft-maximum alignment, and faulty negative rectification. Norton achieves state-of-the-art performance on video retrieval, videoQA, and action segmentation benchmarks by effectively modeling long-range temporal dependencies with minimal computational overhead.

ABSTRACT

Existing video-language studies mainly focus on learning short video clips, leaving long-term temporal dependencies rarely explored due to over-high computational cost of modeling long videos. To address this issue, one feasible solution is learning the correspondence between video clips and captions, which however inevitably encounters the multi-granularity noisy correspondence (MNC) problem. To be specific, MNC refers to the clip-caption misalignment (coarse-grained) and frame-word misalignment (fine-grained), hindering temporal learning and video understanding. In this paper, we propose NOise Robust Temporal Optimal traNsport (Norton) that addresses MNC in a unified optimal transport (OT) framework. In brief, Norton employs video-paragraph and clip-caption contrastive losses to capture long-term dependencies based on OT. To address coarse-grained misalignment in video-paragraph contrast, Norton filters out the irrelevant clips and captions through an alignable prompt bucket and realigns asynchronous clip-caption pairs based on transport distance. To address the fine-grained misalignment, Norton incorporates a soft-maximum operator to identify crucial words and key frames. Additionally, Norton exploits the potential faulty negative samples in clip-caption contrast by rectifying the alignment target with OT assignment to ensure precise temporal modeling. Extensive experiments on video retrieval, videoQA, and action segmentation verify the effectiveness of our method. Code is available at https://lin-yijie.github.io/projects/Norton.

Motivation & Objective

• To address the challenge of multi-granularity noisy correspondence (MNC) in long-term video understanding, where clip-caption and frame-word pairs are frequently misaligned.
• To reduce computational cost in learning long-range temporal dependencies by avoiding direct modeling of entire long videos.
• To improve video representation learning by jointly addressing coarse-grained (asynchronous and irrelevant misalignment) and fine-grained (partial correlation) noise in video-text alignment.
• To enhance temporal modeling accuracy by exploiting potentially faulty negative samples in clip-caption contrastive learning through optimal transport-based reassignment.
• To unify coarse- and fine-grained alignment into a single optimal transport framework for scalable and robust video-language pre-training.

Proposed method

• Norton employs a video-paragraph contrastive loss based on optimal transport (OT) to measure sequence-level distances between video clips and natural language paragraphs, enabling robust long-term temporal alignment.
• It introduces an alignable prompt bucket (APB) to filter out irrelevant clips and captions and re-align asynchronous pairs using transport distance, reducing coarse-grained misalignment.
• A token-wise soft-maximum operator is applied to identify crucial words and key frames within clip-caption pairs, improving fine-grained similarity measurement through log-sum-exp approximation.
• The method uses optimal transport to assign alignment targets between clips and captions, rectifying faulty negative samples and ensuring precise temporal modeling in clip-caption contrastive learning.
• The framework integrates both video-paragraph and clip-caption contrastive losses within a unified OT-based optimization, enabling end-to-end learning of multi-granularity correspondence.
• The model leverages a visual encoder with a one-layer classification head for action segmentation, and applies standard evaluation protocols for video retrieval and videoQA.

Experimental results

Research questions

• RQ1 How can optimal transport be leveraged to jointly address coarse-grained and fine-grained misalignments in long-form video-language learning?
• RQ2 To what extent can an alignable prompt bucket improve the robustness of video-paragraph retrieval under noisy clip-caption correspondence?
• RQ3 Can a soft-maximum operator improve fine-grained frame-word alignment by focusing on salient visual and textual elements?
• RQ4 How does exploiting faulty negative samples through optimal transport assignment enhance clip-caption contrastive learning in long videos?
• RQ5 What is the impact of different hyperparameter settings (e.g., α in log-sum-exp, p in APB) on the overall performance of multi-granularity correspondence learning?

Key findings

• Norton achieves state-of-the-art performance on video retrieval, with 76.1% R@1 and 95.0% R@5 on video-paragraph retrieval, outperforming prior methods like TempCLR and VideoCLIP.
• In action segmentation on YouCookII, Norton achieves 75.2% frame-wise accuracy, surpassing the best baseline (VideoCLIP) by 1.5 percentage points.
• Ablation studies show that incorporating the faulty negative exploitation strategy improves R@1 by 1.4% over the baseline, demonstrating its effectiveness in noisy settings.
• Using the soft-maximum operator with α = 1 yields the best performance, with a 0.3% improvement in R@1 over the mean-average baseline, validating its role in fine-grained alignment.
• The alignable prompt bucket with p = 30% (bottom 30% similarity threshold) yields the best performance, improving R@1 by 0.9% over the baseline in video-paragraph retrieval with backgrounds.
• The full Norton model (J) achieves 24.2% R@1 and 51.9% R@5 on clip-caption retrieval, showing consistent gains across all evaluation metrics.