[Paper Review] Interaction-aware Spatio-temporal Pyramid Attention Networks for Action Classification

Yang Du, Chunfeng Yuan+4 more|arXiv (Cornell University)|Aug 3, 2018

Human Pose and Action Recognition53 references43 citations

TL;DR

The paper proposes an interaction-aware PCA-inspired self-attention with a multi-scale spatial pyramid, extended to spatio-temporal attention for video action classification, achieving state-of-the-art results on popular datasets.

ABSTRACT

Local features at neighboring spatial positions in feature maps have high correlation since their receptive fields are often overlapped. Self-attention usually uses the weighted sum (or other functions) with internal elements of each local feature to obtain its weight score, which ignores interactions among local features. To address this, we propose an effective interaction-aware self-attention model inspired by PCA to learn attention maps. Furthermore, since different layers in a deep network capture feature maps of different scales, we use these feature maps to construct a spatial pyramid and then utilize multi-scale information to obtain more accurate attention scores, which are used to weight the local features in all spatial positions of feature maps to calculate attention maps. Moreover, our spatial pyramid attention is unrestricted to the number of its input feature maps so it is easily extended to a spatio-temporal version. Finally, our model is embedded in general CNNs to form end-to-end attention networks for action classification. Experimental results show that our method achieves the state-of-the-art results on the UCF101, HMDB51 and untrimmed Charades.

Motivation & Objective

• Motivate improving action recognition by capturing interactions among neighboring local features in CNN feature maps.

Proposed method

• Introduce an interaction-aware self-attention mechanism inspired by PCA to leverage feature interactions.
• Construct a spatial feature pyramid from multiple CNN layers to obtain multi-scale attention maps.
• Perform a channel-wise self-attention across pyramid features and use a fusion function to compute attention maps.
• Extend the spatial pyramid attention to a spatio-temporal version to handle multiple video frames.
• Incorporate interactive and attention losses to regularize the attention and enable end-to-end training within CNNs.

Experimental results

Research questions

• RQ1 How can feature interactions among local CNN features be incorporated into self-attention for action recognition?
• RQ2 Does a spatial pyramid over multi-scale feature maps yield more accurate attention scores for video frames?
• RQ3 Can the attention mechanism be extended to a spatio-temporal form that works with sequences of frames of arbitrary length?
• RQ4 Do the proposed losses and multi-scale fusion improve action classification across different backbone networks?

Key findings

• The approach achieves state-of-the-art results on UCF101, HMDB51, and untrimmed Charades datasets.
• A 3-scale spatial pyramid provides notable performance gains over single-scale attention.
• Element-wise multiplication as the fusion function yields strong performance among candidates.
• The method demonstrates generality across CNN backbones (VGGNet-16, BN-Inception, Inception-ResNet-V2).
• Temporal extension allows aggregation over K frames with improvements when more frames are used for testing.