[Paper Review] Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition

Kensho Hara, Hirokatsu Kataoka+1 more|arXiv (Cornell University)|Aug 25, 2017

Human Pose and Action Recognition17 references79 citations

TL;DR

The paper extends ResNet to 3D convolutions to learn spatio-temporal features for action recognition and demonstrates strong performance on Kinetics, comparing with C3D and RGB-I3D baselines using large-scale pretraining.

ABSTRACT

Convolutional neural networks with spatio-temporal 3D kernels (3D CNNs) have an ability to directly extract spatio-temporal features from videos for action recognition. Although the 3D kernels tend to overfit because of a large number of their parameters, the 3D CNNs are greatly improved by using recent huge video databases. However, the architecture of 3D CNNs is relatively shallow against to the success of very deep neural networks in 2D-based CNNs, such as residual networks (ResNets). In this paper, we propose a 3D CNNs based on ResNets toward a better action representation. We describe the training procedure of our 3D ResNets in details. We experimentally evaluate the 3D ResNets on the ActivityNet and Kinetics datasets. The 3D ResNets trained on the Kinetics did not suffer from overfitting despite the large number of parameters of the model, and achieved better performance than relatively shallow networks, such as C3D. Our code and pretrained models (e.g. Kinetics and ActivityNet) are publicly available at https://github.com/kenshohara/3D-ResNets.

Motivation & Objective

• Motivate improving action recognition by applying residual networks to 3D CNNs to better model spatio-temporal features.
• Propose 3D ResNet architectures with 3D kernels and identity shortcuts to enable training of deep models.
• Evaluate 3D ResNets on large-scale video datasets (Kinetics) to assess overfitting and performance.
• Provide pretrained 3D ResNet models and training details for reproducibility.

Proposed method

• Extend ResNet architecture to 3D convolutions and pooling with 3x3x3 kernels.
• Use 16-frame RGB clips as input with standard ResNet-style residual blocks and identity shortcuts.
• Train with SGD on Kinetics using data augmentation including random temporal sampling, multi-scale cropping, and horizontal flipping.
• Evaluate using a sliding-window approach over 16-frame clips and average class probabilities across clips.
• Compare against C3D (with and without batch normalization) and RGB-I3D baselines to assess depth and pretraining effects.

Experimental results

Research questions

• RQ1 Does applying ResNet-style residual blocks to 3D CNNs improve action recognition performance on large-scale datasets?
• RQ2 How do deeper 3D ResNets compare to C3D and RGB-I3D baselines when trained from scratch on Kinetics?
• RQ3 What is the impact of pretraining and dataset size on overfitting for 3D CNNs in action recognition?
• RQ4 What are the practical training considerations (batch size, GPU usage) needed to train deep 3D ResNets effectively?

Key findings

• 3D ResNet-34 achieves higher accuracy than C3D baselines on Kinetics, indicating depth with 3D kernels benefits action recognition.
• 3D ResNets trained from scratch on Kinetics do not overfit and perform competitively with or surpass some RGB-I3D configurations, given adequate resources.
• Pretraining on Sports-1M helps shallower 3D CNNs (C3D) but deeper 3D ResNets still benefit from large-scale data without requiring Sports-1M pretraining.
• RGB-I3D without ImageNet pretraining achieves top performance among reported baselines, underscoring the value of deep 3D architectures with sufficient data.
• The authors provide publicly accessible code and pretrained 3D-ResNets models for reproducibility.