[Paper Review] An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data

Sijie Song, Cuiling Lan+3 more|arXiv (Cornell University)|Nov 18, 2016

Human Pose and Action RecognitionComputer Science31 references481 citations

TL;DR

The paper presents an end-to-end LSTM-based architecture with spatial joint-attention and temporal frame-attention for skeleton-based action recognition, trained with a regularized loss and a joint training strategy, achieving state-of-the-art results on SBU and NTU datasets.

ABSTRACT

Human action recognition is an important task in computer vision. Extracting discriminative spatial and temporal features to model the spatial and temporal evolutions of different actions plays a key role in accomplishing this task. In this work, we propose an end-to-end spatial and temporal attention model for human action recognition from skeleton data. We build our model on top of the Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM), which learns to selectively focus on discriminative joints of skeleton within each frame of the inputs and pays different levels of attention to the outputs of different frames. Furthermore, to ensure effective training of the network, we propose a regularized cross-entropy loss to drive the model learning process and develop a joint training strategy accordingly. Experimental results demonstrate the effectiveness of the proposed model,both on the small human action recognition data set of SBU and the currently largest NTU dataset.

Motivation & Objective

• Motivate robust action recognition from skeleton data by modeling both spatial joint relevance and temporal frame importance.
• Develop an end-to-end architecture that learns to attend to discriminative joints within frames and to important frames over time.
• Introduce regularized loss terms and a joint training strategy to stabilize learning of the coupled attention-enabled network.
• Demonstrate effectiveness on public skeleton datasets, including SBU Kinect Interaction and NTU RGB+D.

Proposed method

• Propose an LSTM-based network with a spatial attention module that computes joint-selection gates to weigh joints within each frame.
• Implement a temporal attention module that assigns frame-selection gates to weigh frame contributions to final sequence classification.
• Formulate a regularized cross-entropy loss incorporating spatial and temporal attention regularizers and a weight sparsity term.
• Adopt a two-stage joint training procedure that pre-trains spatial/temporal attention components before full network fine-tuning.
• Use three LSTM layers for the main network and one LSTM layer for each attention subnetwork with 100 units per layer.
• Evaluate on SBU Kinect Interaction and NTU RGB+D datasets under CS and CV settings.

Experimental results

Research questions

• RQ1 Can end-to-end spatio-temporal attention improve skeleton-based action recognition compared to non-attention baselines?
• RQ2 Do spatial joint-attention and temporal frame-attention provide complementary improvements when used together?
• RQ3 How do regularization terms and the proposed joint training strategy affect learning stability and performance?
• RQ4 How does the proposed STA-LSTM compare with prior state-of-the-art methods on SBU and NTU datasets?

Key findings

• Spatial attention and temporal attention separately improve accuracy over a baseline LSTM, by up to about 5.1% and 6.4% on SBU/NTU respectively.
• Combining both spatial and temporal attention (STA-LSTM) yields the best results across datasets.
• Regularization terms improve performance for spatial and temporal attention modules, and the joint training strategy enhances convergence.
• STA-LSTM achieves substantial accuracy gains over prior methods on NTU (CS and CV) and competitive results on SBU.