[Paper Review] Attentional Pooling for Action Recognition

Rohit Girdhar, Deva Ramanan|arXiv (Cornell University)|Nov 4, 2017

Human Pose and Action RecognitionComputer Science51 references208 citations

TL;DR

Introduces an attentional pooling module that learns image-specific attention maps to weight spatial features, improving action recognition across images and videos with minimal extra cost. It achieves state-of-the-art or competitive results on MPII, HMDB51, and HICO and analyzes the link between attention and second-order pooling.

ABSTRACT

We introduce a simple yet surprisingly powerful model to incorporate attention in action recognition and human object interaction tasks. Our proposed attention module can be trained with or without extra supervision, and gives a sizable boost in accuracy while keeping the network size and computational cost nearly the same. It leads to significant improvements over state of the art base architecture on three standard action recognition benchmarks across still images and videos, and establishes new state of the art on MPII dataset with 12.5% relative improvement. We also perform an extensive analysis of our attention module both empirically and analytically. In terms of the latter, we introduce a novel derivation of bottom-up and top-down attention as low-rank approximations of bilinear pooling methods (typically used for fine-grained classification). From this perspective, our attention formulation suggests a novel characterization of action recognition as a fine-grained recognition problem.

Motivation & Objective

• Motivate an attention mechanism that can be trained with or without extra supervision for action recognition.
• Incorporate attention into standard CNNs with negligible computational overhead.
• Show that attentional pooling aligns with a low-rank, bilinear pooling interpretation and improves benchmarks.
• Analyze how pose supervision affects attention, especially in video data with non-iconic frames.

Proposed method

• Derives attentional pooling as a rank-1 approximation of second-order pooling, enabling a bottom-up saliency map Xb and a top-down, class-specific map Xa.
• Implements the attention module as a replacement for standard pooling in CNNs, producing an attention-weighted feature via h = Xb and score = a^T(X^T h).
• Allows class-specific (top-down) attention via weights Wk or a_k, while sharing a class-agnostic bottom-up component b for saliency.
• Extends the model to pose-regularized attention by adding a pose-prediction head that guides attention without requiring pose at test time.
• Explores architectural choices, including base networks (ResNet-101 vs BN-Inception) and input resolutions, to maximize attention gains.

Experimental results

Research questions

• RQ1 Can an attention mechanism learned with minimal supervision improve action recognition on images and videos?
• RQ2 How does bottom-up saliency interact with top-down, class-specific attention in action recognition?
• RQ3 What is the impact of pose-guided regularization on attention and action recognition performance, especially for non-iconic video frames?
• RQ4 How does the attentional pooling relate to second-order/bilinear pooling, and can rank-1 approximations capture most benefits?
• RQ5 Does attention yield consistent gains across different base architectures and datasets (MPII, HICO, HMDB51)?

Key findings

• The attentional pooling module yields significant accuracy gains with negligible increase in computation across baselines.
• On MPII, the pose-regularized variant achieves the best performance, with notable gains over baselines and previous methods.
• On HICO, the method outperforms full-image baselines and most bounding-box-based methods, demonstrating strong HOI classification.
• On HMDB51, the pose-regularized attention significantly improves RGB single-frame performance, achieving state-of-the-art among RGB-only methods.
• Rank-1 (low-rank) attention via Xa and Xb is equivalent to a bilinear/pairwise attention mechanism, linking attention with second-order pooling concepts.
• The approach performs best with ResNet-101 and larger spatial resolution; Inception-based backbones show smaller gains due to receptive field characteristics.