[Paper Review] A Comprehensive Study of Deep Video Action Recognition

Yi Zhu, Xinyu Li+8 more|arXiv (Cornell University)|Dec 11, 2020

Human Pose and Action Recognition274 references115 citations

TL;DR

This paper surveys over 200 deep learning methods for video action recognition, discusses datasets and challenges, benchmarks popular models, and releases code for reproducibility.

ABSTRACT

Video action recognition is one of the representative tasks for video understanding. Over the last decade, we have witnessed great advancements in video action recognition thanks to the emergence of deep learning. But we also encountered new challenges, including modeling long-range temporal information in videos, high computation costs, and incomparable results due to datasets and evaluation protocol variances. In this paper, we provide a comprehensive survey of over 200 existing papers on deep learning for video action recognition. We first introduce the 17 video action recognition datasets that influenced the design of models. Then we present video action recognition models in chronological order: starting with early attempts at adapting deep learning, then to the two-stream networks, followed by the adoption of 3D convolutional kernels, and finally to the recent compute-efficient models. In addition, we benchmark popular methods on several representative datasets and release code for reproducibility. In the end, we discuss open problems and shed light on opportunities for video action recognition to facilitate new research ideas.

Motivation & Objective

• Survey the landscape of deep learning for video action recognition across 200+ papers.
• Catalog datasets and their influence on model design and evaluation.
• Analyze model evolution from two-stream networks to 3D CNNs and compute-efficient architectures.
• Benchmark representative methods on standard datasets to compare accuracy and efficiency.
• Provide open problems and opportunities to guide future research and development.

Proposed method

• Chronological review of core developments in video action recognition (hand-crafted features to CNNs, two-stream networks, 3D CNNs, and compute-efficient models).
• Systematic discussion of datasets and challenges that shape model design and evaluation.
• Empirical benchmarking of popular methods on standard benchmarks to assess accuracy and efficiency.
• Release of model implementations in PyTorch and MXNet to ensure reproducibility.
• Analysis of open problems and opportunities for future research in video action recognition.

Experimental results

Research questions

• RQ1 What datasets and evaluation protocols have most influenced the design of video action recognition models?
• RQ2 How have model architectures evolved to address temporal modeling and computational efficiency in video action recognition?
• RQ3 What are the trade-offs between two-stream and 3D CNN approaches, and how do compute-efficient methods compare?
• RQ4 What open problems and opportunities remain to advance video action recognition?
• RQ5 How do multi-stream and multi-modality approaches (pose, objects, audio) contribute to recognition performance?

Key findings

• Over 200 papers and 17 influential datasets shaped the field and evaluation practices.
• Two-stream networks established the importance of combining appearance and motion information via RGB frames and optical flow.
• 3D CNNs (e.g., I3D) significantly advanced performance by directly modeling spatio-temporal features, especially after pretraining on large datasets like Kinetics400.
• Segment-based and compute-efficient models (e.g., TSN, TSM, X3D) enabled long-range temporal modeling and deployment on larger datasets.
• Results on core benchmarks show substantial gains when moving from shallow to deeper architectures and from 2D to 3D representations, with I3D achieving high performance on UCF101 and HMDB51 after pretraining.
• The authors release code to promote reproducibility and provide a model zoo for researchers.