[Paper Review] A Closer Look at Spatiotemporal Convolutions for Action Recognition

Du Tran, Heng Wang+4 more|arXiv (Cornell University)|Nov 30, 2017

Human Pose and Action Recognition26 references215 citations

TL;DR

The paper empirically analyzes various spatiotemporal convolutions for action recognition and introduces the R(2+1)D block, showing it achieves state-of-the-art results on Sports-1M, Kinetics, UCF101, and HMDB51. It demonstrates that factorizing 3D convolutions into separate spatial and temporal components improves accuracy and optimization, with mixed and (2+1)D variants offering trade-offs.

ABSTRACT

In this paper we discuss several forms of spatiotemporal convolutions for video analysis and study their effects on action recognition. Our motivation stems from the observation that 2D CNNs applied to individual frames of the video have remained solid performers in action recognition. In this work we empirically demonstrate the accuracy advantages of 3D CNNs over 2D CNNs within the framework of residual learning. Furthermore, we show that factorizing the 3D convolutional filters into separate spatial and temporal components yields significantly advantages in accuracy. Our empirical study leads to the design of a new spatiotemporal convolutional block "R(2+1)D" which gives rise to CNNs that achieve results comparable or superior to the state-of-the-art on Sports-1M, Kinetics, UCF101 and HMDB51.

Motivation & Objective

• Evaluate the impact of different spatiotemporal convolutions (2D, 3D, mixed, and (2+1)D) on action recognition performance.
• Assess optimization and accuracy benefits of factorizing 3D convolutions into separate spatial and temporal steps.
• Introduce and validate the R(2+1)D block within ResNet architectures across large-scale datasets.
• Compare against state-of-the-art methods on Sports-1M, Kinetics, UCF101, and HMDB51.
• Provide insights on clip length, training strategies, and video-level prediction in practice.

Proposed method

• Systematically evaluate multiple convolution variants: R2D (2D on clip), f-R2D (2D on frames), R3D (3D), MCx/rMCx (mixed 3D-2D), and R(2+1)D ((2+1)D) within ResNets.
• Propose (2+1)D blocks that replace each 3D filter Ni×t×d×d with a 2D spatial filter Ni×1×d×d followed by a 1D temporal filter Mi×t×1×1, with parameter-matching to 3D in channels.
• Analyze optimization and nonlinear capacity via training vs. testing error comparisons, demonstrating easier optimization and larger nonlinear capacity for (2+1)D vs. full 3D.
• Evaluate on large-scale benchmarks (Sports-1M, Kinetics) and transfer to UCF101/HMDB51 with clip-level and video-level metrics.
• Pretrain/fine-tune strategies and clip-length experiments to study video-level accuracy vs. clip-level accuracy.

Experimental results

Research questions

• RQ1 Does temporal modeling in convolutional networks improve action recognition over frame-wise or 2D-only models?
• RQ2 Can factorizing 3D convolutions into separate spatial and temporal components improve accuracy and optimization?
• RQ3 How do mixed and (2+1)D architectures compare to full 3D CNNs on large-scale action recognition datasets?
• RQ4 What is the impact of clip length and number of clips on video-level prediction performance?

Key findings

Net	# params	Clip@1 (8 frames)	Video@1 (8 frames)	Clip@1 (16 frames)	Video@1 (16 frames)
R2D	11.4M	46.7	59.5	47.0	58.9
f-R2D	11.4M	48.1	59.4	50.3	60.5
R3D	33.4M	49.4	61.8	52.5	64.2
MC2	11.4M	50.2	62.5	53.1	64.2
MC3	11.7M	50.7	62.9	53.7	64.7
MC4	12.7M	50.5	62.5	53.7	65.1
MC5	16.9M	50.3	62.5	53.7	65.1
rMC2	33.3M	49.8	62.1	53.1	64.9
rMC3	33.0M	49.8	62.3	53.2	65.0
rMC4	32.0M	49.9	62.3	53.4	65.1
rMC5	27.9M	49.4	61.2	52.1	63.1
R(2+1)D	33.3M	52.8	64.8	56.8	68.0

• R(2+1)D consistently achieves the best accuracy among tested variants on Kinetics with 8- and 16-frame inputs (Clip@1: 52.8–56.8; Video@1: 64.8–68.0).
• (2+1)D factorization yields higher accuracy and easier optimization than full 3D convolutions, especially as network depth increases.
• On Sports-1M, RGB 32-frame R(2+1)D achieves 57.0% clip@1 and 73.0% video@1, surpassing C3D and P3D baselines; video-level accuracy reaches 73.3% (best reported).
• R(2+1)D outperforms I3D and other baselines on Kinetics when trained from scratch on RGB, and pretraining on Sports-1M provides transfer advantages.
• Longer input clips improve clip-level accuracy but video-level gains saturate, with best video performance obtained by averaging predictions from multiple clips.
• R(2+1)D demonstrates favorable training vs. testing loss dynamics compared to R3D, particularly at deeper networks.