[Paper Review] Low-Latency Human Action Recognition with Weighted Multi-Region Convolutional Neural Network

Yunfeng Wang, Wengang Zhou+3 more|arXiv (Cornell University)|May 8, 2018

Human Pose and Action Recognition19 references3 citations

TL;DR

This paper proposes a low-latency human action recognition system, WMR ConvNet, which uses a 2D ConvNet with weighted multi-region feature extraction from both foreground (primary) and background (secondary) regions to capture spatio-temporal context without frame accumulation. It achieves state-of-the-art accuracy among low-latency methods on UCF101 and HMDB51, outperforming even 3D ConvNet-based C3D despite not requiring video clip aggregation or LSTMs.

ABSTRACT

Spatio-temporal contexts are crucial in understanding human actions in videos. Recent state-of-the-art Convolutional Neural Network (ConvNet) based action recognition systems frequently involve 3D spatio-temporal ConvNet filters, chunking videos into fixed length clips and Long Short Term Memory (LSTM) networks. Such architectures are designed to take advantage of both short term and long term temporal contexts, but also requires the accumulation of a predefined number of video frames (e.g., to construct video clips for 3D ConvNet filters, to generate enough inputs for LSTMs). For applications that require low-latency online predictions of fast-changing action scenes, a new action recognition system is proposed in this paper. Termed "Weighted Multi-Region Convolutional Neural Network" (WMR ConvNet), the proposed system is LSTM-free, and is based on 2D ConvNet that does not require the accumulation of video frames for 3D ConvNet filtering. Unlike early 2D ConvNets that are based purely on RGB frames and optical flow frames, the WMR ConvNet is designed to simultaneously capture multiple spatial and short term temporal cues (e.g., human poses, occurrences of objects in the background) with both the primary region (foreground) and secondary regions (mostly background). On both the UCF101 and HMDB51 datasets, the proposed WMR ConvNet achieves the state-of-the-art performance among competing low-latency algorithms. Furthermore, WMR ConvNet even outperforms the 3D ConvNet based C3D algorithm that requires video frame accumulation. In an ablation study with the optical flow ConvNet stream removed, the ablated WMR ConvNet nevertheless outperforms competing algorithms.

Motivation & Objective

• To address the high latency in existing action recognition systems that require video frame accumulation for 3D ConvNet or LSTM processing.
• To improve low-latency online action recognition by leveraging spatial and temporal context from both foreground and background regions in video frames.
• To develop a lightweight, LSTM-free architecture that maintains high accuracy while enabling real-time inference.
• To demonstrate that multi-region spatial sampling with weighted fusion can outperform standard two-stream or single-region approaches in low-latency settings.

Proposed method

• Uses a fine-tuned Faster R-CNN to detect and localize the primary region (foreground) in each video frame.
• Applies Selective Search to extract multiple secondary regions (background) with varying IoU values relative to the primary region.
• Applies ROI-pooling to extract features from both primary and secondary regions using 2D ConvNets on RGB and optical flow inputs.
• Fuses features from the RGB and optical flow streams via a weighted sum operation to combine spatial and short-term temporal cues.
• Processes frames sequentially in the spatial stream and in 10-frame windows in the temporal stream, enabling a maximum processing delay of ~0.3 seconds at 30 fps.
• Employs a late fusion strategy using scores from the softmax layer, with weighted averaging to combine predictions from multiple regions.

Experimental results

Research questions

• RQ1 Can a 2D ConvNet-based system achieve high action recognition accuracy without requiring video frame accumulation or 3D spatio-temporal convolutions?
• RQ2 Does incorporating multiple spatial regions—especially background regions—improve recognition performance in low-latency settings?
• RQ3 How effective is the weighted fusion of features from primary and secondary regions compared to alternative fusion methods like SVM or concatenation?
• RQ4 Can a system without optical flow or LSTM components still outperform state-of-the-art models that rely on these components?

Key findings

• The WMR ConvNet achieves 85.7% accuracy on UCF101 and 66.7% on HMDB51, outperforming all competing low-latency 2D ConvNet methods and matching or exceeding iDT and C3D on UCF101.
• Even without the optical flow stream, the ablated WMR ConvNet achieves 78.8% accuracy on UCF101, surpassing dynamic image networks and other RGB-only methods.
• The weighted sum fusion of score-level features outperforms both SVM-based fusion and feature-level concatenation, with 75.9% accuracy on UCF101 split-1.
• Using bounding boxes from RGB frames as primary regions yields better results than selecting regions based on optical flow magnitude, indicating RGB cues are more informative.
• The method achieves a maximum processing delay of only 0.3 seconds, significantly lower than clip-based 3D ConvNet or LSTM systems that may incur delays of several to tens of seconds.