[Paper Review] An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition

Chenyang Si, Wentao Chen+3 more|arXiv (Cornell University)|Feb 25, 2019

Human Pose and Action RecognitionComputer Science41 references97 citations

TL;DR

This paper introduces AGC-LSTM, an attention-enhanced graph convolutional LSTM network for skeleton-based action recognition that captures spatial-temporal features and their co-occurrence, using a temporal hierarchical architecture to boost temporal receptive fields and reduce computation, achieving state-of-the-art results on NTU RGB+D and Northwestern-UCLA datasets.

ABSTRACT

Skeleton-based action recognition is an important task that requires the adequate understanding of movement characteristics of a human action from the given skeleton sequence. Recent studies have shown that exploring spatial and temporal features of the skeleton sequence is vital for this task. Nevertheless, how to effectively extract discriminative spatial and temporal features is still a challenging problem. In this paper, we propose a novel Attention Enhanced Graph Convolutional LSTM Network (AGC-LSTM) for human action recognition from skeleton data. The proposed AGC-LSTM can not only capture discriminative features in spatial configuration and temporal dynamics but also explore the co-occurrence relationship between spatial and temporal domains. We also present a temporal hierarchical architecture to increases temporal receptive fields of the top AGC-LSTM layer, which boosts the ability to learn the high-level semantic representation and significantly reduces the computation cost. Furthermore, to select discriminative spatial information, the attention mechanism is employed to enhance information of key joints in each AGC-LSTM layer. Experimental results on two datasets are provided: NTU RGB+D dataset and Northwestern-UCLA dataset. The comparison results demonstrate the effectiveness of our approach and show that our approach outperforms the state-of-the-art methods on both datasets.

Motivation & Objective

• Motivate robust skeleton-based action recognition that leverages spatial configuration and temporal dynamics.
• Propose a unified model that captures co-occurrence between spatial and temporal domains.
• Incorporate an attention mechanism to emphasize discriminative joints across time steps.
• Introduce a temporal hierarchical architecture to enlarge temporal receptive fields and reduce computation.
• Demonstrate state-of-the-art performance on standard benchmarks (NTU RGB+D and Northwestern-UCLA).

Proposed method

• Map 3D joint coordinates to spatial features using a linear layer per joint.
• Compute augmented features by concatenating joint position features with frame-difference features and normalize scale via a shared LSTM.
• Model sequences with three stacked AGC-LSTM layers using graph convolutions to capture spatial-temporal patterns.
• Apply an attention network to emphasize key joints at each time step and blend attended and non-attended features.
• Introduce temporal average pooling to create a temporal hierarchy, increasing receptive fields and reducing computation.
• Fuse global (all joints) and local (attended joints) features from the last AGC-LSTM layer for classification.

Experimental results

Research questions

• RQ1 How to effectively extract discriminative spatial and temporal features from skeleton sequences using graph-based methods?
• RQ2 Can an attention mechanism on joints improve the discrimination of action-relevant spatial configurations?
• RQ3 Does a temporal hierarchical architecture improve high-level spatiotemporal representations while reducing computation?
• RQ4 How does joint-level and part-level modeling (and their combination) compare on skeleton-based action recognition?

Key findings

• AGC-LSTM with attention achieves state-of-the-art accuracy on NTU RGB+D (Joint/Part/Joint&Part: 93.5/93.8/95.0 CV, 87.5/87.5/89.2 CS) and Northwestern-UCLA (Joint/Part/Joint&Part: 93.3/?/? in reported table).
• Joint-level and part-level variants both reach top performance, with joint&part fusion yielding the best results on NTU RGB+D.
• In ablation analyses, replacing LSTM with GC-LSTM and adding temporal hierarchy improve accuracy substantially (e.g., GC-LSTM+TH vs GC-LSTM, and AGC-LSTM vs GC-LSTM).
• Attention embeddings progressively emphasize key joints (e.g., elbow, wrist, hand) across layers, shown via attention visualizations.
• Temporal hierarchical architecture increases temporal receptive fields and reduces computation without sacrificing accuracy.
• Hybrid joint&part modeling provides further performance gains over single-branch variants.