[Paper Review] Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition

Pengfei Zhang, Cuiling Lan+4 more|arXiv (Cornell University)|Apr 2, 2019

Human Pose and Action RecognitionComputer Science61 references42 citations

TL;DR

SGN introduces joint-type and frame-index semantics into a hierarchical model with joint-level GCNs and frame-level CNNs, achieving state-of-the-art accuracy on NTU60/NTU120/SYSU with significantly fewer parameters.

ABSTRACT

Skeleton-based human action recognition has attracted great interest thanks to the easy accessibility of the human skeleton data. Recently, there is a trend of using very deep feedforward neural networks to model the 3D coordinates of joints without considering the computational efficiency. In this paper, we propose a simple yet effective semantics-guided neural network (SGN) for skeleton-based action recognition. We explicitly introduce the high level semantics of joints (joint type and frame index) into the network to enhance the feature representation capability. In addition, we exploit the relationship of joints hierarchically through two modules, i.e., a joint-level module for modeling the correlations of joints in the same frame and a framelevel module for modeling the dependencies of frames by taking the joints in the same frame as a whole. A strong baseline is proposed to facilitate the study of this field. With an order of magnitude smaller model size than most previous works, SGN achieves the state-of-the-art performance on the NTU60, NTU120, and SYSU datasets. The source code is available at https://github.com/microsoft/SGN.

Motivation & Objective

• Motivate skeleton-based action recognition using easily accessible 3D joints.
• Explicitly incorporate high-level joint semantics (joint type and frame index) to enhance feature representations.
• Model joint-level correlations with GCNs and frame-level dependencies with CNNs in a hierarchical framework.
• Provide a lightweight strong baseline and demonstrate state-of-the-art performance with fewer parameters.

Proposed method

• Embed joint position and velocity into a unified dynamics representation, then fuse with joint semantics.
• Use a joint-level GCN with a content-adaptive graph learned from joint dynamics and joint-type semantics to model intra-frame relations.
• Construct a frame-level module that incorporates frame-index semantics and performs spatial pooling over joints, followed by temporal CNNs for inter-frame dynamics.
• Employ frame index and joint-type embeddings to enrich node and frame representations.
• Develop a lightweight strong baseline without semantics to benchmark performance, including data augmentation and pooling strategies.
• Compare SGN against state-of-the-art on NTU60, NTU120, and SYSU datasets, reporting parameter efficiency.

Experimental results

Research questions

• RQ1 Can explicit modeling of joint-type semantics improve graph construction and message passing in GCNs for skeleton data?
• RQ2 Does incorporating frame-index semantics at the frame level enhance temporal modeling and action classification accuracy?
• RQ3 Is a hierarchical joint-level and frame-level architecture more effective than non-hierarchical or global approaches for skeleton action recognition?
• RQ4 How does SGN perform with a lightweight baseline compared to heavier state-of-the-art models?

Key findings

• SGN achieves state-of-the-art results on NTU60 CS (89.0%) and CV (94.5%) settings.
• SGN outperforms its non-semantic baseline by 2.1% (CS) and 1.7% (CV).
• Frame-index semantics improve performance, especially when temporal convolution is limited, and provide additive gains when used with temporal kernels.
• Hierarchical modeling of intra-frame joint correlations (joint-level) and inter-frame frame correlations (frame-level) yields better accuracy than non-hierarchical or global graph approaches.
• SGN with semantics uses an order of magnitude fewer parameters than many prior methods while achieving competitive or superior accuracy.
• A strong lightweight baseline without semantics benefits significantly from data augmentation and max-pooling over joints, highlighting efficiency gains.