[Paper Review] Beyond Transfer Learning: Co-finetuning for Action Localisation

Anurag Arnab, Xuehan Xiong+7 more|arXiv (Cornell University)|Jul 8, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes co-finetuning, a training strategy that simultaneously fine-tunes a single model on multiple upstream video classification datasets and a downstream action localisation task (AVA/AVA-Kinetics), improving performance—especially on rare classes—by leveraging cross-dataset feature transfer and regularization. The method achieves state-of-the-art results, with 36.1 mAP on AVA and 36.2 mAP on AVA-Kinetics, outperforming complex architectures using traditional transfer learning.

ABSTRACT

Transfer learning is the predominant paradigm for training deep networks on small target datasets. Models are typically pretrained on large ``upstream'' datasets for classification, as such labels are easy to collect, and then finetuned on ``downstream'' tasks such as action localisation, which are smaller due to their finer-grained annotations. In this paper, we question this approach, and propose co-finetuning -- simultaneously training a single model on multiple ``upstream'' and ``downstream'' tasks. We demonstrate that co-finetuning outperforms traditional transfer learning when using the same total amount of data, and also show how we can easily extend our approach to multiple ``upstream'' datasets to further improve performance. In particular, co-finetuning significantly improves the performance on rare classes in our downstream task, as it has a regularising effect, and enables the network to learn feature representations that transfer between different datasets. Finally, we observe how co-finetuning with public, video classification datasets, we are able to achieve state-of-the-art results for spatio-temporal action localisation on the challenging AVA and AVA-Kinetics datasets, outperforming recent works which develop intricate models.

Motivation & Objective

• To address the limitations of traditional transfer learning in action localisation, where models are pre-trained on one upstream dataset and fine-tuned on a downstream task.
• To improve performance on long-tailed action localisation datasets like AVA and AVA-Kinetics, especially for rare classes that are prone to overfitting.
• To explore whether joint training on multiple upstream classification datasets can regularize the model and improve feature transfer to the downstream detection task.
• To demonstrate that a simple model with a modified training strategy can outperform complex architectures that rely on external memories or graph-based modeling.
• To evaluate the effectiveness of different upstream datasets (e.g., Kinetics, Moments in Time, SSv2, Instagram) in co-finetuning for action localisation.

Proposed method

• Co-finetuning trains a single model simultaneously on multiple upstream video classification datasets (e.g., Kinetics, Moments in Time, Something-Something V2) and a downstream action localisation dataset (AVA or AVA-Kinetics).
• The training objective combines classification losses from upstream datasets with detection loss from the downstream dataset, using a shared backbone and task-specific heads.
• The approach leverages the larger scale and diversity of multiple upstream datasets to regularize the model, especially benefiting rare classes in the downstream dataset.
• The method uses standard detection architectures (e.g., ViViT) without complex components like external memory or graph neural networks, focusing solely on the training strategy.
• Pretraining with large-scale datasets like WTS (60M videos) is used to further boost performance, with co-finetuning applied from the start.
• Inference is performed on a single view, and spatial resolution is increased (e.g., 512px) to improve accuracy, consistent with image detection practices.

Experimental results

Research questions

• RQ1 Can joint training on multiple upstream video classification datasets improve performance on downstream action localisation tasks compared to traditional transfer learning?
• RQ2 Does co-finetuning provide a regularizing effect that particularly benefits rare classes in long-tailed datasets like AVA?
• RQ3 How does the choice of upstream dataset (e.g., Kinetics vs. Moments in Time vs. SSv2) affect downstream performance in co-finetuning?
• RQ4 Can co-finetuning with public, weakly-labeled video datasets achieve state-of-the-art results without complex model architectures?
• RQ5 To what extent can feature representations learned from classification tasks transfer to the more complex detection task in action localisation?

Key findings

• Co-finetuning with Kinetics, Moments in Time, and Something-Something V2 improved mAP on AVA to 36.1, achieving state-of-the-art performance.
• The method achieved 36.2 mAP on AVA-Kinetics, representing the best-known result on this benchmark.
• Significant improvements were observed on rare classes (Tail and Mid), with co-finetuning acting as a regularizer to reduce overfitting.
• Using WTS pretraining (60M videos) in co-finetuning further improved performance, increasing mAP by 0.9% compared to baseline.
• Moments in Time was found to be more effective than Kinetics as an upstream dataset for action localisation, despite Kinetics being the standard choice.
• The model simultaneously performs video classification and action localisation, achieving competitive accuracy on upstream classification benchmarks.