[Paper Review] Non-Local Graph Convolutional Networks for Skeleton-Based Action Recognition

Lei Shi, Yifan Zhang+2 more|arXiv (Cornell University)|May 20, 2018

Human Pose and Action Recognition17 references22 citations

TL;DR

This paper proposes a two-stream adaptive graph convolutional network (2s-AGCN) for skeleton-based action recognition that learns graph topology end-to-end via backpropagation, enabling data-driven adaptation to diverse samples. By jointly modeling first- and second-order skeleton information (joint positions and bone vectors), the method achieves state-of-the-art performance on NTU-RGBD and Kinetics-Skeleton datasets.

ABSTRACT

In skeleton-based action recognition, graph convolutional networks (GCNs), which model the human body skeletons as spatiotemporal graphs, have achieved remarkable performance. However, in existing GCN-based methods, the topology of the graph is set manually, and it is fixed over all layers and input samples. This may not be optimal for the hierarchical GCN and diverse samples in action recognition tasks. In addition, the second-order information (the lengths and directions of bones) of the skeleton data, which is naturally more informative and discriminative for action recognition, is rarely investigated in existing methods. In this work, we propose a novel two-stream adaptive graph convolutional network (2s-AGCN) for skeleton-based action recognition. The topology of the graph in our model can be either uniformly or individually learned by the BP algorithm in an end-to-end manner. This data-driven method increases the flexibility of the model for graph construction and brings more generality to adapt to various data samples. Moreover, a two-stream framework is proposed to model both the first-order and the second-order information simultaneously, which shows notable improvement for the recognition accuracy. Extensive experiments on the two large-scale datasets, NTU-RGBD and Kinetics-Skeleton, demonstrate that the performance of our model exceeds the state-of-the-art with a significant margin.

Motivation & Objective

• To address the limitation of fixed, manually designed graph topologies in existing GCNs for skeleton-based action recognition.
• To improve model generalization by enabling end-to-end learning of graph structures per sample or uniformly across layers.
• To exploit second-order skeleton information (bone lengths and directions) which is underutilized in prior methods.
• To develop a unified framework that simultaneously models both first-order (joint positions) and second-order (bone features) spatial relationships.

Proposed method

• Proposes a two-stream architecture where one stream processes joint positions (first-order features) and the other processes bone vectors (second-order features).
• Introduces learnable adjacency matrices that are optimized via backpropagation, allowing the graph topology to adapt to input samples and network depth.
• Employs adaptive graph convolution layers that dynamically adjust the graph structure during training without requiring manual design.
• Uses a residual learning framework with identity shortcuts to stabilize training of deep GCN architectures.
• Applies spatiotemporal graph convolutions that preserve temporal coherence across frames while modeling spatial relationships.
• Combines features from both streams via element-wise concatenation before final classification.

Experimental results

Research questions

• RQ1 Can end-to-end learned graph topologies improve performance in skeleton-based action recognition compared to fixed, handcrafted graphs?
• RQ2 How does modeling both first-order and second-order skeleton features affect recognition accuracy?
• RQ3 Does the proposed adaptive graph learning mechanism generalize across diverse action samples and complex human poses?
• RQ4 To what extent does the two-stream design enhance discriminative representation learning in GCNs?

Key findings

• The proposed 2s-AGCN achieves state-of-the-art accuracy on the NTU-RGBD dataset, outperforming prior methods with a significant margin.
• On the Kinetics-Skeleton dataset, the model achieves new state-of-the-art performance, demonstrating strong generalization to large-scale, diverse action categories.
• Ablation studies confirm that both the adaptive graph learning and the two-stream design contribute significantly to performance gains.
• The inclusion of second-order information (bone vectors) leads to a noticeable improvement in recognition accuracy over models using only joint positions.
• End-to-end learned graph topologies consistently improve performance across all evaluation protocols on both datasets.
• The model shows robustness to variations in pose and action complexity due to the flexible, data-driven graph construction.