[Paper Review] Video Action Recognition Via Neural Architecture Searching

Wei Peng, Xiaopeng Hong+1 more|arXiv (Cornell University)|Jul 10, 2019

Human Pose and Action Recognition22 references4 citations

TL;DR

This paper proposes the first neural architecture search (NAS) framework for video action recognition, using a differentiable, continuous search space modeled as a directed acyclic graph with pseudo-3D operators to efficiently explore spatio-temporal architectures. The method achieves 58.6% accuracy on UCF101 with only 0.67M parameters—surpassing larger manually designed models like 3D-ConvNet (79M params) by 7% in accuracy while using ~1% of the parameters.

ABSTRACT

Deep neural networks have achieved great success for video analysis and understanding. However, designing a high-performance neural architecture requires substantial efforts and expertise. In this paper, we make the first attempt to let algorithm automatically design neural networks for video action recognition tasks. Specifically, a spatio-temporal network is developed in a differentiable space modeled by a directed acyclic graph, thus a gradient-based strategy can be performed to search an optimal architecture. Nonetheless, it is computationally expensive, since the computational burden to evaluate each architecture candidate is still heavy. To alleviate this issue, we, for the video input, introduce a temporal segment approach to reduce the computational cost without losing global video information. For the architecture, we explore in an efficient search space by introducing pseudo 3D operators. Experiments show that, our architecture outperforms popular neural architectures, under the training from scratch protocol, on the challenging UCF101 dataset, surprisingly, with only around one percentage of parameters of its manual-design counterparts.

Motivation & Objective

• To automate the design of high-performing spatio-temporal neural architectures for video action recognition, reducing reliance on expert engineering.
• To address the high computational cost of NAS in video tasks by introducing a temporal segment strategy and efficient pseudo-3D operators.
• To enable end-to-end, differentiable architecture search in a continuous space, allowing gradient-based optimization.
• To achieve state-of-the-art performance under a training-from-scratch protocol with significantly smaller model size.

Proposed method

• Model the search space as a directed acyclic graph (DAG) where each edge is associated with a set of candidate operators, and operator selection is differentiable via learnable architecture parameters α.
• Use a proxyless search strategy: train a shallow network with shared weights to evaluate architecture performance, avoiding costly retraining.
• Introduce a temporal segment network (TSN)-like input processing scheme with Ns=4 segments and Nr=2 random crops per segment to reduce computation while preserving global video context.
• Employ pseudo-3D operators (e.g., depthwise separable convolutions, dilated convolutions) in a search space that decouples spatial and temporal operations for efficiency.
• Apply a softmax over architecture parameters α to select the top-2 connections per node and the most important operator per connection, forming a discrete, final architecture.
• Train the final network from scratch using the searched architecture with depth=6, input resolution 112×112, and a cosine decay learning rate schedule over 600 epochs.

Experimental results

Research questions

• RQ1 Can neural architecture search be effectively applied to video action recognition to automate the design of spatio-temporal architectures?
• RQ2 How can the computational cost of NAS for video models be reduced without sacrificing performance?
• RQ3 Can a differentiable, continuous search space with pseudo-3D operators yield better architectures than manual design with fewer parameters?
• RQ4 Does the proposed method outperform manually designed models when trained from scratch on standard benchmarks?

Key findings

• The proposed NAS method achieves 58.6% top-1 accuracy on UCF101 split 1, outperforming 3D-ConvNet (51.6%) by 7 percentage points despite having only 0.67M parameters.
• The searched architecture is approximately 1% the size of 3D-ConvNet (79M parameters), demonstrating a dramatic reduction in model size with significant accuracy gain.
• The method achieves state-of-the-art performance under a training-from-scratch protocol, eliminating the need for ImageNet pre-training.
• The search process completes in 25 hours on a single V100 GPU, making it feasible for practical deployment.
• The final architecture favors operators with larger感受 field, such as dilated and depthwise separable convolutions, indicating effective inductive bias learning.
• The performance gain is consistent across multiple evaluation metrics, with the searched model significantly outperforming 3D-ResNet18, 3D-ResNet101, and STC-ResNet variants in both accuracy and parameter efficiency.