[Paper Review] Two-person Graph Convolutional Network for Skeleton-based Human Interaction Recognition

Zhengcen Li, Yueran Li+3 more|arXiv (Cornell University)|Aug 12, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes a Two-person Graph Convolutional Network (2P-GCN) that models human interactions by unifying two individuals' skeletons into a single, joint graph structure, capturing both intra-body and inter-body relationships. By introducing novel edge labeling strategies and a two-person graph convolution block, 2P-GCN achieves state-of-the-art performance on four benchmarks, outperforming prior methods by up to 3.08% accuracy on NTU-120-Interaction while using only 1.47M parameters and 3.74G FLOPs.

ABSTRACT

Graph convolutional networks (GCNs) have been the predominant methods in skeleton-based human action recognition, including human-human interaction recognition. However, when dealing with interaction sequences, current GCN-based methods simply split the two-person skeleton into two discrete graphs and perform graph convolution separately as done for single-person action classification. Such operations ignore rich interactive information and hinder effective spatial inter-body relationship modeling. To overcome the above shortcoming, we introduce a novel unified two-person graph to represent inter-body and intra-body correlations between joints. Experiments show accuracy improvements in recognizing both interactions and individual actions when utilizing the proposed two-person graph topology. In addition, We design several graph labeling strategies to supervise the model to learn discriminant spatial-temporal interactive features. Finally, we propose a two-person graph convolutional network (2P-GCN). Our model achieves state-of-the-art results on four benchmarks of three interaction datasets: SBU, interaction subsets of NTU-RGB+D and NTU-RGB+D 120.

Motivation & Objective

• To address the limitation of existing GCN-based methods that treat two-person interactions as isolated single-person graphs, thereby losing critical interactive information.
• To design a unified two-person graph topology that simultaneously encodes intra-body joint connections and inter-body relationships between two individuals.
• To develop graph labeling strategies that guide the model to learn discriminative spatial-temporal interactive features.
• To propose a two-person graph convolution block (2P-GCB) that enables effective feature extraction within a single graph convolution operation.
• To achieve state-of-the-art performance on multiple benchmark datasets for skeleton-based human interaction recognition.

Proposed method

• The paper introduces a two-person graph where each frame's interaction is represented as a single graph with 2N joints (N per person), enabling joint modeling of intra- and inter-body connections.
• A relational adjacency matrix is constructed to encode both physical skeleton connections (intra-body) and inter-personal connections (inter-body), with edge weights reflecting correlation strength.
• Graph edge labeling strategies are designed to supervise the model in learning meaningful spatial-temporal interactive features, enhancing representation discrimination.
• A two-person graph convolution block (2P-GCB) is developed to perform graph convolutions on the unified two-person graph, preserving interaction-aware features across layers.
• The 2P-GCN model is evaluated on multiple GCN architectures, demonstrating consistent accuracy gains across different backbones.
• The method is scalable and generalizable, with potential extension to multi-person and human-object interaction scenarios.

Experimental results

Research questions

• RQ1 Can a unified two-person graph representation improve the modeling of inter-body and intra-body relationships in skeleton-based human interaction recognition?
• RQ2 How do graph labeling strategies affect the model's ability to learn discriminative spatial-temporal features for interaction recognition?
• RQ3 Does replacing isolated single-person graphs with a joint two-person graph lead to consistent performance gains across different GCN architectures?
• RQ4 To what extent does the proposed 2P-GCN outperform state-of-the-art methods on standard interaction recognition benchmarks?
• RQ5 Can the two-person graph representation also improve performance on single-person action recognition tasks?

Key findings

• 2P-GCN achieves state-of-the-art accuracy on four benchmarks, including 97.05% on X-Sub and 98.80% on X-View of NTU-Interaction, outperforming 2S-AIGCN by 1.71% and 0.80% respectively.
• On NTU120-Interaction, 2P-GCN achieves 93.47% and 93.73% accuracy on X-Sub120 and X-Set120, surpassing 2S-AIGCN by 2.76% and 3.08%.
• The model achieves 98.90% accuracy on the SBU dataset, outperforming the previous SOTA method (AIGCN) by 0.20% and DR-GCN by 0.14%.
• Despite modeling richer interaction features, 2P-GCN maintains low complexity with only 1.47M parameters and 3.74G FLOPs, significantly lower than many SOTA models.
• The two-person graph representation improves performance not only on interaction tasks but also on single-person action recognition, demonstrating broader utility.
• The ablation study confirms that the proposed two-person graph topology consistently improves accuracy across all tested GCN models, validating its effectiveness and generalization capability.