[Paper Review] Pose-conditioned Spatio-Temporal Attention for Human Action Recognition

Fabien Baradel, Christian Wolf+1 more|arXiv (Cornell University)|Mar 29, 2017

Human Pose and Action Recognition51 references73 citations

TL;DR

The paper combines a pose-based CNN and a pose-conditioned spatio-temporal attention mechanism over RGB video, achieving state-of-the-art results on NTU-RGB+D and SBU Kinect Interaction, with knowledge transfer to smaller datasets.

ABSTRACT

We address human action recognition from multi-modal video data involving articulated pose and RGB frames and propose a two-stream approach. The pose stream is processed with a convolutional model taking as input a 3D tensor holding data from a sub-sequence. A specific joint ordering, which respects the topology of the human body, ensures that different convolutional layers correspond to meaningful levels of abstraction. The raw RGB stream is handled by a spatio-temporal soft-attention mechanism conditioned on features from the pose network. An LSTM network receives input from a set of image locations at each instant. A trainable glimpse sensor extracts features on a set of predefined locations specified by the pose stream, namely the 4 hands of the two people involved in the activity. Appearance features give important cues on hand motion and on objects held in each hand. We show that it is of high interest to shift the attention to different hands at different time steps depending on the activity itself. Finally a temporal attention mechanism learns how to fuse LSTM features over time. We evaluate the method on 3 datasets. State-of-the-art results are achieved on the largest dataset for human activity recognition, namely NTU-RGB+D, as well as on the SBU Kinect Interaction dataset. Performance close to state-of-the-art is achieved on the smaller MSR Daily Activity 3D dataset.

Motivation & Objective

• Motivate accurate action recognition using multimodal data: articulated pose and RGB frames.
• Develop a pose-aware two-stream architecture that leverages pose for guiding RGB attention.
• Encode pose information into a convolutional representation over time.
• Enable adaptive temporal pooling to fuse temporal information effectively.
• Demonstrate transfer of learned representations from a large dataset to smaller benchmarks.

Proposed method

• Encode pose data as 3D tensors organized along a topologically informed joint ordering for CNN processing.
• Use a pose-only CNN to extract hierarchical pose features from pose sub-sequences.
• Implement a spatial attention mechanism on RGB frames conditioned on pose features, using a trainable glimpse sensor focusing on four hands.
• Process glimpse outputs with an LSTM; integrate an attention-based fusion across hands at each time step.
• Apply a temporal attention mechanism to adaptively pool LSTM features over time.
• Fuse pose and RGB streams at the logit level and train end-to-end (with certain components frozen during RGB stream training).

Experimental results

Research questions

• RQ1 Can pose-driven features improve action recognition when combined with RGB video via attention mechanisms?
• RQ2 Does conditioning RGB spatial attention on pose features improve the model’s focus on informative regions (e.g., hands and manipulated objects)?
• RQ3 Can temporal attention effectively fuse features across time for improved accuracy?
• RQ4 Is knowledge transfer from a large dataset (NTU) beneficial for smaller datasets (MSR Daily Activity 3D, SBU Kinect Interaction)?
• RQ5 What is the impact of joint ordering and pose representation on recognition performance?

Key findings

• Achieved state-of-the-art results on NTU RGB+D with both pose-only and pose+RGB models.
• Achieved state-of-the-art results on the SBU Kinect Interaction dataset with the full model.
• Performance on MSR Daily Activity 3D is competitive, illustrating challenges with very small datasets.
• Joint ordering with a topological, neighborhood-preserving sequence yields >1 percentage point improvement on NTU over random ordering.
• The pose-conditioned spatial attention significantly improves RGB-only performance, with larger gains in RGB-only settings (~1–12 points) compared to multi-modal setups.
• Knowledge transfer from NTU to MSR and SBU enables meaningful improvements on smaller datasets.