[Paper Review] Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks

Zhaofan Qiu, Ting Yao+1 more|arXiv (Cornell University)|Nov 28, 2017

Human Pose and Action Recognition34 references251 citations

TL;DR

The paper introduces Pseudo-3D (P3D) blocks that simulate 3D convolutions using 2D spatial filters plus 1D temporal filters within a Residual Network, creating P3D ResNet variants that improve video representation over traditional 2D and 3D CNNs.

ABSTRACT

Convolutional Neural Networks (CNN) have been regarded as a powerful class of models for image recognition problems. Nevertheless, it is not trivial when utilizing a CNN for learning spatio-temporal video representation. A few studies have shown that performing 3D convolutions is a rewarding approach to capture both spatial and temporal dimensions in videos. However, the development of a very deep 3D CNN from scratch results in expensive computational cost and memory demand. A valid question is why not recycle off-the-shelf 2D networks for a 3D CNN. In this paper, we devise multiple variants of bottleneck building blocks in a residual learning framework by simulating $3 imes3 imes3$ convolutions with $1 imes3 imes3$ convolutional filters on spatial domain (equivalent to 2D CNN) plus $3 imes1 imes1$ convolutions to construct temporal connections on adjacent feature maps in time. Furthermore, we propose a new architecture, named Pseudo-3D Residual Net (P3D ResNet), that exploits all the variants of blocks but composes each in different placement of ResNet, following the philosophy that enhancing structural diversity with going deep could improve the power of neural networks. Our P3D ResNet achieves clear improvements on Sports-1M video classification dataset against 3D CNN and frame-based 2D CNN by 5.3% and 1.8%, respectively. We further examine the generalization performance of video representation produced by our pre-trained P3D ResNet on five different benchmarks and three different tasks, demonstrating superior performances over several state-of-the-art techniques.

Motivation & Objective

• Motivate efficient learning of spatio-temporal video representations without full 3D CNNs.
• Develop bottleneck blocks that simulate 3x3x3 convolutions with 1x3x3 spatial and 3x1x1 temporal filters.
• Explore different block designs (P3D-A/B/C) and mix them within a ResNet to improve performance.
• Demonstrate that P3D ResNet outperforms 3D CNNs and frame-based CNNs on multiple video datasets.
• Show that pre-training 2D spatial filters on images and learning 1D temporal filters on video data yields strong generalization.

Proposed method

• Define 3D convolutions and decouple into 2D spatial (1x3x3) and 1D temporal (3x1x1) components.
• Propose three P3D block designs (A, B, C) with different direct/indirect connections between S and T paths.
• Adopt a bottleneck scheme with 1x1 reductions/restorations around the spatial/temporal filters.
• Create a P3D ResNet by replacing ResNet blocks with P3D blocks and mix A/B/C blocks for structural diversity.
• Pre-train on Sports-1M (large-scale video) and evaluate as a generic video representation extractor across tasks.
• Compare against ResNet-50, C3D, and other baselines on UCF101, ActivityNet, ASLAN, YUPENN, and Dynamic Scene.

Experimental results

Research questions

• RQ1 Can pseudo-3D blocks effectively substitute full 3D convolutions to capture spatio-temporal information in videos?
• RQ2 Do different P3D block designs (A, B, C) offer complementary benefits, and does mixing them improve performance?
• RQ3 Is a P3D ResNet pre-trained on image data (for spatial) plus video data (for temporal) more effective than pure 3D CNNs or frame-based methods?
• RQ4 How does P3D ResNet perform as a general video representation across diverse datasets and tasks?

Key findings

• P3D variants outperform ResNet-50 and are competitive with or superior to C3D while adding modest model size and maintaining efficient runtime.
• Mixing P3D-A, P3D-B, and P3D-C (complete P3D ResNet) provides additional accuracy gains over any single variant, indicating value of architectural diversity.
• On Sports-1M, P3D ResNet achieves higher video-level accuracy (47.9% clip hit@1; 66.4% video hit@1; 87.4% video hit@5) compared with several baselines.
• On UCF101, P3D ResNet with only frame inputs reaches 88.6% top-1 accuracy, outperforming ResNet-152 and C3D; with IDT fusion it reaches 93.7%.
• On ActivityNet, P3D ResNet achieves Top-1 75.12%, Top-3 87.71%, MAP 78.86%—outperforming several baselines including IDT, C3D, and ResNet-152 baselines.
• Visualizations show P3D ResNet captures both spatial patterns and temporal motion, and t-SNE indicates semantically clearer clustering for P3D ResNet representations.