[Paper Review] Learning Spatiotemporal Features with 3D Convolutional Networks

Du Tran, Lubomir Bourdev+3 more|arXiv (Cornell University)|Dec 2, 2014

Human Pose and Action RecognitionComputer Science52 references155 citations

TL;DR

This paper proposes C3D, a 3D convolutional neural network that learns spatiotemporal features directly from video clips using small $3\times3\times3$ kernels. Trained on large-scale video datasets, C3D achieves state-of-the-art performance on multiple benchmarks with a simple linear classifier, offering compact, efficient, and generic video representations suitable for diverse video analysis tasks.

ABSTRACT

We propose a simple, yet effective approach for spatiotemporal feature learning using deep 3-dimensional convolutional networks (3D ConvNets) trained on a large scale supervised video dataset. Our findings are three-fold: 1) 3D ConvNets are more suitable for spatiotemporal feature learning compared to 2D ConvNets; 2) A homogeneous architecture with small 3x3x3 convolution kernels in all layers is among the best performing architectures for 3D ConvNets; and 3) Our learned features, namely C3D (Convolutional 3D), with a simple linear classifier outperform state-of-the-art methods on 4 different benchmarks and are comparable with current best methods on the other 2 benchmarks. In addition, the features are compact: achieving 52.8% accuracy on UCF101 dataset with only 10 dimensions and also very efficient to compute due to the fast inference of ConvNets. Finally, they are conceptually very simple and easy to train and use.

Motivation & Objective

• To develop a generic, compact, and efficient video descriptor that generalizes across diverse video analysis tasks.
• To investigate whether 3D ConvNets can outperform 2D ConvNets in modeling both spatial and temporal features simultaneously.
• To identify optimal architectural hyperparameters—particularly kernel size and network depth—for 3D ConvNets in video action recognition.
• To evaluate the effectiveness of C3D features with a simple linear classifier across multiple benchmarks without task-specific fine-tuning.

Proposed method

• Train a 3D ConvNet with $3\times3\times3$ convolutional kernels in all layers on a large-scale supervised video dataset.
• Use spatial and temporal convolutions to jointly learn appearance and motion patterns from video clips.
• Apply average pooling and a final fully connected layer to produce fixed-length video embeddings.
• Employ deconvolutional networks to visualize and interpret learned feature maps at different network depths.
• Train models with varying input resolutions ($64\times64$, $128\times128$, $256\times256$) to evaluate trade-offs in accuracy, parameters, and training time.
• Use a linear classifier on the final pooled features to evaluate performance on downstream video recognition tasks.

Experimental results

Research questions

• RQ1 Can 3D ConvNets learn more effective spatiotemporal features than 2D ConvNets for video action recognition?
• RQ2 What is the optimal kernel size and network depth for 3D ConvNets in learning video features?
• RQ3 How does input resolution affect the performance, parameter count, and training efficiency of 3D ConvNets?
• RQ4 To what extent can a simple linear classifier achieve strong performance when combined with C3D features?
• RQ5 What types of motion and appearance patterns do the learned filters in C3D detect at different network layers?

Key findings

• 3D ConvNets significantly outperform 2D ConvNets in modeling spatiotemporal features, as shown by a 3.1% accuracy gain on UCF101 when using 3D ConvNets with $128\times128$ input resolution.
• A homogeneous architecture with $3\times3\times3$ kernels in all layers achieves the best performance, outperforming larger or mixed-size kernels.
• C3D achieves 85.2% accuracy on UCF101 with only 10-dimensional features, demonstrating high compactness and efficiency.
• On the UCF100 benchmark, C3D achieves 90.4% accuracy when using all available features (e.g., optical flow), outperforming previous state-of-the-art methods.
• Deconvolution visualizations show that early layers detect low-level motion patterns (e.g., moving edges, blobs), while deeper layers detect complex actions (e.g., biking, circular motion, facial gestures).
• C3D features are more selective than optical flow, focusing only on salient motion patterns rather than all moving pixels, as confirmed by comparison in Figure 15.