[Paper Review] Labelling unlabelled videos from scratch with multi-modal self-supervision

Yuki M. Asano, Mandela Patrick+2 more|arXiv (Cornell University)|Jun 24, 2020

Human Pose and Action Recognition82 references71 citations

TL;DR

The paper introduces SeLaVi, a multi-modal self-supervised clustering method that jointly learns representations and pseudo-labels for unlabeled videos by leveraging audio-visual correspondence, achieving state-of-the-art unsupervised labeling on several video datasets.

ABSTRACT

A large part of the current success of deep learning lies in the effectiveness of data -- more precisely: labelled data. Yet, labelling a dataset with human annotation continues to carry high costs, especially for videos. While in the image domain, recent methods have allowed to generate meaningful (pseudo-) labels for unlabelled datasets without supervision, this development is missing for the video domain where learning feature representations is the current focus. In this work, we a) show that unsupervised labelling of a video dataset does not come for free from strong feature encoders and b) propose a novel clustering method that allows pseudo-labelling of a video dataset without any human annotations, by leveraging the natural correspondence between the audio and visual modalities. An extensive analysis shows that the resulting clusters have high semantic overlap to ground truth human labels. We further introduce the first benchmarking results on unsupervised labelling of common video datasets Kinetics, Kinetics-Sound, VGG-Sound and AVE.

Motivation & Objective

• Motivate reducing labeling costs for video data by enabling unsupervised labeling.
• Develop a clustering framework that learns from multi-modal (audio-visual) video data without human annotations.
• Ensure clusters reflect semantic content while handling imbalanced (Zipf-like) class distributions.
• Enable modality-robust clustering by treating audio and visual streams as augmentations and aligning them.
• Provide strong benchmarks on standard video datasets to establish unsupervised labeling performance.

Proposed method

• Formulate clustering as an optimal transport problem to prevent degenerate solutions (SeLa groundwork).
• Relax uniform cluster priors to accommodate skewed real-world distributions and allow arbitrary priors via Sinkhorn optimization.
• Introduce multi-modal single labeling by treating modalities as augmentations and learning modality-agnostic clusters.
• Synchronize modality-specific encoders at initialization to align outputs across modalities.
• Learn multiple decorrelated clustering heads to explore diverse, orthogonal labelings in parallel.
• Train with a pair of encoders (audio and visual) producing shared cluster assignments and apply modality splicing augmentations.

Experimental results

Research questions

• RQ1 Can multi-modal self-supervised clustering produce semantically meaningful video labels without human annotations?
• RQ2 How does incorporating audio-visual correspondence and modality alignment affect clustering quality compared to single-modality or post-hoc labeling?
• RQ3 Do multiple decorrelated clustering heads improve coverage of the valid labeling space for videos?
• RQ4 How robust is the learned clustering to degraded modalities (e.g., compressed visuals)?

Key findings

• SeLaVi achieves state-of-the-art clustering metrics on VGG-Sound, AVE, and Kinetics-Sound with substantially higher NMI, ARI, and accuracy than baselines.
• Using both audio and visual modalities yields higher clustering quality than either modality alone, with strong gains when modalities are aligned.
• Decorrelated clustering heads and modality alignment significantly improve clustering performance over single-head or simple concatenation baselines.
• The method can perfectly group 32% of VGG-Sound and 55% of AVE videos without labeled data, and achieves 57.9% accuracy on AVE.
• Unsupervised labels learned by SeLaVi support improved downstream representation learning, including improved video action retrieval performance.