[Paper Review] Evolving Losses for Unlabeled Video Representation Learning

AJ Piergiovanni, Anelia Angelova+1 more|arXiv (Cornell University)|Jun 7, 2019

Human Pose and Action Recognition9 references4 citations

TL;DR

This paper proposes an evolutionary algorithm to automatically discover optimal loss weighting for multi-modal, multi-task self-supervised video representation learning, distilling audio, optical flow, and temporal features into a single RGB-based network. The method outperforms ImageNet pretraining and supervised Kinetics fine-tuning, achieving 66.2% accuracy on HMDB51 and 92.4% on UCF101 using only unlabeled video data.

ABSTRACT

We present a new method to learn video representations from unlabeled data. Given large-scale unlabeled video data, the objective is to benefit from such data by learning a generic and transferable representation space that can be directly used for a new task such as zero/few-shot learning. We formulate our unsupervised representation learning as a multi-modal, multi-task learning problem, where the representations are also shared across different modalities via distillation. Further, we also introduce the concept of finding a better loss function to train such multi-task multi-modal representation space using an evolutionary algorithm; our method automatically searches over different combinations of loss functions capturing multiple (self-supervised) tasks and modalities. Our formulation allows for the distillation of audio, optical flow and temporal information into a single, RGB-based convolutional neural network. We also compare the effects of using additional unlabeled video data and evaluate our representation learning on standard public video datasets.

Motivation & Objective

• To improve unsupervised video representation learning by automatically discovering optimal loss function weights instead of manual tuning.
• To enable multi-modal, multi-task learning across RGB, audio, optical flow, and grayscale features using distillation to transfer knowledge into a single RGB network.
• To evaluate the impact of large-scale unlabeled video data on downstream recognition performance without any supervised pretraining.
• To demonstrate that evolving the loss function leads to better generalization than random or hand-crafted loss combinations.
• To analyze which tasks and modalities are most beneficial for action recognition through evolved loss weights.

Proposed method

• Formulates unsupervised video representation learning as a multi-modal, multi-task learning problem with shared representations across modalities via distillation.
• Uses a weighted sum of task-specific losses $\mathcal{L}_{m,t}$ and distillation losses $\mathcal{L}_d$ to form the total loss: $\mathcal{L} = \sum_{m}\sum_{t}\lambda_{m,t}\mathcal{L}_{m,t} + \sum_{d}\lambda_d\mathcal{L}_d$, where $\lambda_{m,t}, \lambda_d \in [0,1]$.
• Employs an evolutionary algorithm to search over the space of loss weights, maintaining a population of weight configurations and selecting the fittest based on clustering performance on a subset of HMDB.
• Applies distillation losses as $L_2$ differences between activations of the main RGB network and those from other modalities (e.g., audio, flow) at corresponding layers.
• Initializes the evolutionary population with random weights in $[0,1]$, and iteratively improves by mutating one weight per individual and selecting the best-performing configuration.
• Evaluates the final model on standard video benchmarks (HMDB51, UCF101) after fine-tuning with limited labeled data.

Experimental results

Research questions

• RQ1 Can an evolutionary algorithm effectively discover optimal loss weighting for multi-task, multi-modal self-supervised video representation learning?
• RQ2 How does the performance of the evolved loss compare to hand-crafted or random loss functions in terms of downstream action recognition accuracy?
• RQ3 Which self-supervised tasks and modalities (e.g., audio, flow, RGB) contribute most significantly to learning effective video representations?
• RQ4 To what extent can unlabeled video data alone, without any supervised pretraining, achieve performance comparable to ImageNet or Kinetics pretraining?
• RQ5 How do the timing and layer locations of distillation losses affect representation quality, and what patterns emerge in the evolved loss structure?

Key findings

• The evolved loss function achieves 66.2% accuracy on HMDB51 and 92.4% on UCF101 after fine-tuning with only 10% of Kinetics labeled data, outperforming both ImageNet pretraining and supervised Kinetics baselines.
• Using only 25k labeled samples (10% of Kinetics), the model reaches 58.1% accuracy, only 11% below the full supervised baseline, demonstrating strong data efficiency.
• The evolved loss assigns high weights to tasks like audio-video alignment, future frame prediction, and RGB-to-flow reconstruction, while assigning low weights to frame shuffle and colorization tasks.
• Distillation is applied earlier for grayscale features (more similar to RGB) and later for audio and flow features, suggesting modality-specific optimal fusion timing.
• The evolutionary process identifies that cross-modality reconstruction and alignment tasks are more beneficial than frame-order prediction or colorization for action recognition.
• Increasing the amount of unlabeled data consistently improves performance, with better gains when training time is fixed and more data is used, indicating scalability.