[論文レビュー] Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks

Zhaofan Qiu, Ting Yao+1 more|arXiv (Cornell University)|Nov 28, 2017

Human Pose and Action Recognition参考文献 34被引用数 251

ひとこと要約

本論文は Pseudo-3D (P3D) ブロックを導入し、Residual Network 内で 2D 空間フィルタと 1D 時間フィルタを組み合わせて 3D畳み込みを模倣し、従来の 2D および 3D CNN よりも動画表現を改善する P3D ResNet 変種を作成する。

ABSTRACT

Convolutional Neural Networks (CNN) have been regarded as a powerful class of models for image recognition problems. Nevertheless, it is not trivial when utilizing a CNN for learning spatio-temporal video representation. A few studies have shown that performing 3D convolutions is a rewarding approach to capture both spatial and temporal dimensions in videos. However, the development of a very deep 3D CNN from scratch results in expensive computational cost and memory demand. A valid question is why not recycle off-the-shelf 2D networks for a 3D CNN. In this paper, we devise multiple variants of bottleneck building blocks in a residual learning framework by simulating $3 imes3 imes3$ convolutions with $1 imes3 imes3$ convolutional filters on spatial domain (equivalent to 2D CNN) plus $3 imes1 imes1$ convolutions to construct temporal connections on adjacent feature maps in time. Furthermore, we propose a new architecture, named Pseudo-3D Residual Net (P3D ResNet), that exploits all the variants of blocks but composes each in different placement of ResNet, following the philosophy that enhancing structural diversity with going deep could improve the power of neural networks. Our P3D ResNet achieves clear improvements on Sports-1M video classification dataset against 3D CNN and frame-based 2D CNN by 5.3% and 1.8%, respectively. We further examine the generalization performance of video representation produced by our pre-trained P3D ResNet on five different benchmarks and three different tasks, demonstrating superior performances over several state-of-the-art techniques.

研究の動機と目的

• Motivate efficient learning of spatio-temporal video representations without full 3D CNNs.
• Develop bottleneck blocks that simulate 3x3x3 convolutions with 1x3x3 spatial and 3x1x1 temporal filters.
• Explore different block designs (P3D-A/B/C) and mix them within a ResNet to improve performance.
• Demonstrate that P3D ResNet outperforms 3D CNNs and frame-based CNNs on multiple video datasets.
• Show that pre-training 2D spatial filters on images and learning 1D temporal filters on video data yields strong generalization.

提案手法

• Define 3D convolutions and decouple into 2D spatial (1x3x3) and 1D temporal (3x1x1) components.
• Propose three P3D block designs (A, B, C) with different direct/indirect connections between S and T paths.
• Adopt a bottleneck scheme with 1x1 reductions/restorations around the spatial/temporal filters.
• Create a P3D ResNet by replacing ResNet blocks with P3D blocks and mix A/B/C blocks for structural diversity.
• Pre-train on Sports-1M (large-scale video) and evaluate as a generic video representation extractor across tasks.
• Compare against ResNet-50, C3D, and other baselines on UCF101, ActivityNet, ASLAN, YUPENN, and Dynamic Scene.]
• research_questions ["Can pseudo-3D blocks effectively substitute full 3D convolutions to capture spatio-temporal information in videos?","Do different P3D block designs (A, B, C) offer complementary benefits, and does mixing them improve performance?","Is a P3D ResNet pre-trained on image data (for spatial) plus video data (for temporal) more effective than pure 3D CNNs or frame-based methods?","How does P3D ResNet perform as a general video representation across diverse datasets and tasks?"]
• key_findings:[

実験結果

リサーチクエスチョン

• RQ1 Can pseudo-3D blocks effectively substitute full 3D convolutions to capture spatio-temporal information in videos?
• RQ2 Do different P3D block designs (A, B, C) offer complementary benefits, and does mixing them improve performance?
• RQ3 Is a P3D ResNet pre-trained on image data (for spatial) plus video data (for temporal) more effective than pure 3D CNNs or frame-based methods?
• RQ4 How does P3D ResNet perform as a general video representation across diverse datasets and tasks?

主な発見

Model size	Speed	Accuracy
ResNet-50	92MB	15.0 frame/s	80.8%
P3D-A ResNet	98MB	9.0 clip/s	83.7%
P3D-B ResNet	98MB	8.8 clip/s	82.8%
P3D-C ResNet	98MB	8.6 clip/s	83.0%
P3D ResNet	98MB	8.8 clip/s	84.2%

• P3D variants outperform ResNet-50 and are competitive with or superior to C3D while adding modest model size and maintaining efficient runtime.
• Mixing P3D-A, P3D-B, and P3D-C (complete P3D ResNet) provides additional accuracy gains over any single variant, indicating value of architectural diversity.
• On Sports-1M, P3D ResNet achieves higher video-level accuracy (47.9% clip hit@1; 66.4% video hit@1; 87.4% video hit@5) compared with several baselines.
• On UCF101, P3D ResNet with only frame inputs reaches 88.6% top-1 accuracy, outperforming ResNet-152 and C3D; with IDT fusion it reaches 93.7%.
• On ActivityNet, P3D ResNet achieves Top-1 75.12%, Top-3 87.71%, MAP 78.86%—outperforming several baselines including IDT, C3D, and ResNet-152 baselines.
• Visualizations show P3D ResNet captures both spatial patterns and temporal motion, and t-SNE indicates semantically clearer clustering for P3D ResNet representations.