[Paper Review] ConvNet Architecture Search for Spatiotemporal Feature Learning

Du Tran, Jamie Ray+3 more|arXiv (Cornell University)|Aug 16, 2017

Multimodal Machine Learning ApplicationsComputer Science1 references349 citations

TL;DR

The paper conducts an empirical ConvNet architecture search for video spatiotemporal features, yielding a deep 3D Residual ConvNet (Res3D) that outperforms C3D across multiple benchmarks while being faster and more compact.

ABSTRACT

Learning image representations with ConvNets by pre-training on ImageNet has proven useful across many visual understanding tasks including object detection, semantic segmentation, and image captioning. Although any image representation can be applied to video frames, a dedicated spatiotemporal representation is still vital in order to incorporate motion patterns that cannot be captured by appearance based models alone. This paper presents an empirical ConvNet architecture search for spatiotemporal feature learning, culminating in a deep 3-dimensional (3D) Residual ConvNet. Our proposed architecture outperforms C3D by a good margin on Sports-1M, UCF101, HMDB51, THUMOS14, and ASLAN while being 2 times faster at inference time, 2 times smaller in model size, and having a more compact representation.

Motivation & Objective

• Investigate how architectural choices affect spatiotemporal feature learning for video classification.
• Develop a deep 3D residual architecture tailored for video data.
• Constrain model capacity to attribute improvements to architecture rather than size.
• Demonstrate transferability of learned representations to diverse video tasks.

Proposed method

• Perform a controlled architecture search on UCF101 within a ResNet-inspired space.
• Propose the Res3D architecture (3D-ResNet-18) and train on Sports-1M for large-scale spatiotemporal learning.
• Systematically evaluate input sampling rates, input resolutions, and convolution types (3D vs 2D/2.5D) under fixed parameter budgets.
• Compare against C3D and 2D baselines to assess gains in accuracy, speed, and model size.

Experimental results

Research questions

• RQ1 What architectural changes most improve spatiotemporal feature learning for video classification?
• RQ2 Is a 3D convolutional approach superior to 2D or mixed convolutions for video representations?
• RQ3 How do sampling rate, input resolution, and network depth affect performance under capacity constraints?
• RQ4 Can the learned Res3D representation generalize to large-scale video benchmarks and different tasks?

Key findings

• Res3D outperforms C3D across multiple benchmarks (Sports-1M, UCF101, HMDB51, THUMOS14, ASLAN).
• On Sports-1M, Res3D achieves state-of-the-art results among non-long-term modeling methods and is 2x faster with 2x smaller model size than C3D.
• Res3D provides stronger RGB-only performance than C3D on UCF101 and HMDB51 (e.g., 3.5% and 3.3% relative gains in Table 9).
• 3D convolutions across all layers yield better performance than mixed 3D-2D or 2.5D variants, with Res3D's 3D design offering clearer advantages.
• A network depth around 18-26 layers offers a favorable accuracy-speed-memory trade-off for video classification.