[Paper Review] Temporal 3D ConvNets: New Architecture and Transfer Learning for Video Classification

Ali Diba, Mohsen Fayyaz+5 more|arXiv (Cornell University)|Nov 22, 2017

Human Pose and Action Recognition30 references187 citations

TL;DR

The paper introduces Temporal 3D ConvNets (T3D) with a Temporal Transition Layer (TTL) to capture multi-scale temporal dynamics, extends DenseNet to DenseNet3D, and proposes a 2D-to-3D supervision transfer to enable stable weight initialization and better performance with limited data. It achieves state-of-the-art results on HMDB51 and UCF101 and competitive results on Kinetics.

ABSTRACT

The work in this paper is driven by the question how to exploit the temporal cues available in videos for their accurate classification, and for human action recognition in particular? Thus far, the vision community has focused on spatio-temporal approaches with fixed temporal convolution kernel depths. We introduce a new temporal layer that models variable temporal convolution kernel depths. We embed this new temporal layer in our proposed 3D CNN. We extend the DenseNet architecture - which normally is 2D - with 3D filters and pooling kernels. We name our proposed video convolutional network `Temporal 3D ConvNet'~(T3D) and its new temporal layer `Temporal Transition Layer'~(TTL). Our experiments show that T3D outperforms the current state-of-the-art methods on the HMDB51, UCF101 and Kinetics datasets. The other issue in training 3D ConvNets is about training them from scratch with a huge labeled dataset to get a reasonable performance. So the knowledge learned in 2D ConvNets is completely ignored. Another contribution in this work is a simple and effective technique to transfer knowledge from a pre-trained 2D CNN to a randomly initialized 3D CNN for a stable weight initialization. This allows us to significantly reduce the number of training samples for 3D CNNs. Thus, by finetuning this network, we beat the performance of generic and recent methods in 3D CNNs, which were trained on large video datasets, e.g. Sports-1M, and finetuned on the target datasets, e.g. HMDB51/UCF101. The T3D codes will be released

Motivation & Objective

• Motivate exploiting temporal cues in video for improved action recognition.
• Develop an architecture that models variable temporal depths within 3D CNNs.
• Extend DenseNet to 3D with a novel TTL to capture short, mid, and long-range dynamics.
• Introduce a cross-architecture transfer learning method from pre-trained 2D CNNs to randomly initialized 3D CNNs to ease training.
• Evaluate on HMDB51, UCF101, and Kinetics to demonstrate performance and transferability.

Proposed method

• Introduce Temporal Transition Layer (TTL) that concatenates features from multiple temporal depths within a 3D convolutional framework.
• Extend DenseNet to DenseNet3D by using 3D filters and pooling kernels across densely connected blocks.
• Incorporate TTL into DenseNet3D to form Temporal 3D ConvNets (T3D) for learning short, mid, and long-range temporal dynamics.
• Propose supervision transfer from a pre-trained 2D CNN (ImageNet) to a randomly initialized 3D CNN by aligning image-video frame/clip pairs via an image-video correspondence task.
• Train T3D from scratch on Kinetics and fine-tune on target datasets (UCF101, HMDB51); compare against other 3D CNNs using RGB inputs only.
• Demonstrate that a 2D-to-3D transfer strategy provides stable weight initialization and improves data-efficient learning on small datasets.

Experimental results

Research questions

• RQ1 Can a 3D CNN capture long-range temporal information without fixed kernel depths?
• RQ2 Does a temporal transition layer with variable depth kernels improve action recognition over fixed-depth 3D convolutions?
• RQ3 Can knowledge learned by 2D CNNs be transferred to 3D CNNs to reduce the need for large labeled video datasets?
• RQ4 How does T3D perform relative to state-of-the-art 3D ConvNets on HMDB51, UCF101, and Kinetics?
• RQ5 What input configurations (frame rate, resolution) best support 3D video architectures?

Key findings

• T3D with TTL outperforms state-of-the-art 3D ConvNets on HMDB51 and UCF101 and is competitive on Kinetics.
• A 2D pre-trained CNN can act as a teacher to provide stable initialization for a randomly initialized 3D CNN, enabling effective transfer learning without large video datasets.
• T3D with TTL yields higher accuracy than DenseNet3D and other 3D architectures when trained from scratch on UCF101.
• Frame resolution and sampling rate significantly affect performance; 224x224 frames and a stride of 2 provide better results than smaller frames or larger strides.
• Transfer learning (2D→3D) improves performance on UCF101 and HMDB51, matching or exceeding models trained on large video datasets and finetuned on targets.