[Paper Review] Spatiotemporal Residual Networks for Video Action Recognition

Christoph Feichtenhofer, Axel Pinz+1 more|arXiv (Cornell University)|Nov 7, 2016

Human Pose and Action RecognitionComputer Science494 citations

TL;DR

Introduces spatiotemporal ResNets that fuse a two-stream architecture with residual connections and temporal convolutions, achieving state-of-the-art action recognition on UCF101 and HMDB51.

ABSTRACT

Two-stream Convolutional Networks (ConvNets) have shown strong performance for human action recognition in videos. Recently, Residual Networks (ResNets) have arisen as a new technique to train extremely deep architectures. In this paper, we introduce spatiotemporal ResNets as a combination of these two approaches. Our novel architecture generalizes ResNets for the spatiotemporal domain by introducing residual connections in two ways. First, we inject residual connections between the appearance and motion pathways of a two-stream architecture to allow spatiotemporal interaction between the two streams. Second, we transform pretrained image ConvNets into spatiotemporal networks by equipping these with learnable convolutional filters that are initialized as temporal residual connections and operate on adjacent feature maps in time. This approach slowly increases the spatiotemporal receptive field as the depth of the model increases and naturally integrates image ConvNet design principles. The whole model is trained end-to-end to allow hierarchical learning of complex spatiotemporal features. We evaluate our novel spatiotemporal ResNet using two widely used action recognition benchmarks where it exceeds the previous state-of-the-art.

Motivation & Objective

• Extend ResNets to the spatiotemporal domain for video action recognition.
• Integrate appearance (RGB) and motion (optical flow) streams with residual connections across streams.
• Transform pretrained image ConvNets into spatiotemporal networks via temporal residual initializations.
• Enable end-to-end training to learn hierarchical spatiotemporal features.
• Demonstrate state-of-the-art performance on standard action recognition benchmarks.

Proposed method

• Adopt a two-stream ResNet-50 architecture for appearance and motion streams pre-trained on ImageNet.
• Introduce residual connections between the streams to enable spatiotemporal interaction (motion residuals).
• Transform spatial 1x1 dimensionality-mapping filters into temporal filters initialized as temporal residual connections (equations 5 and related).
• Stack temporal convolutions to enlarge the spatiotemporal receptive field while preserving image-based design principles.
• Train in three phases: separate stream pretraining, joint ST-ResNet training with cross-stream residuals, and ST-ResNet* with temporal max-pooling.
• Use fully convolutional inference with 25-frame chunks and temporal max-pooling for longer temporal extent.

Experimental results

Research questions

• RQ1 Can residual connections across appearance and motion streams improve spatiotemporal feature learning for video action recognition?
• RQ2 Does extending ResNets with temporal convolutions and pretrained image-net-based initialization improve performance on standard benchmarks?
• RQ3 What is the impact of end-to-end training and temporal max-pooling on recognition accuracy?
• RQ4 How do temporal strides and receptive fields affect action recognition in longer video sequences?

Key findings

• ST-ResNet significantly improves over the two-stream baseline with cross-stream residuals and temporal convolutions.
• ST-ResNet* and ST-ResNet* with temporal max-pooling achieve higher accuracy than ST-ResNet on both benchmarks.
• On UCF101 and HMDB51, ST-ResNet* yields state-of-the-art results compared to prior ConvNet approaches.
• Incorporating IDT features with ST-ResNet* further boosts HMDB51 performance (notable gains).
• End-to-end training of a spatiotemporal network leveraging pretrained image nets demonstrates strong generalization and performance gains.