[Paper Review] Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition

Yuxin Chen, Ziqi Zhang+4 more|arXiv (Cornell University)|Jul 26, 2021

Human Pose and Action Recognition33 references49 citations

TL;DR

CTR-GC dynamically learns channel-specific topologies by refining a shared topology, enabling more flexible feature aggregation in skeleton-based action recognition; when combined into CTR-GCN, it achieves state-of-the-art results on NTU RGB+D, NTU RGB+D 120, and NW-UCLA.

ABSTRACT

Graph convolutional networks (GCNs) have been widely used and achieved remarkable results in skeleton-based action recognition. In GCNs, graph topology dominates feature aggregation and therefore is the key to extracting representative features. In this work, we propose a novel Channel-wise Topology Refinement Graph Convolution (CTR-GC) to dynamically learn different topologies and effectively aggregate joint features in different channels for skeleton-based action recognition. The proposed CTR-GC models channel-wise topologies through learning a shared topology as a generic prior for all channels and refining it with channel-specific correlations for each channel. Our refinement method introduces few extra parameters and significantly reduces the difficulty of modeling channel-wise topologies. Furthermore, via reformulating graph convolutions into a unified form, we find that CTR-GC relaxes strict constraints of graph convolutions, leading to stronger representation capability. Combining CTR-GC with temporal modeling modules, we develop a powerful graph convolutional network named CTR-GCN which notably outperforms state-of-the-art methods on the NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets.

Motivation & Objective

• Motivate improved topology modeling in skeleton-based action recognition by accounting for channel-wise variations in motion features.
• Propose CTR-GC to refine a shared topology with channel-specific correlations for each channel.
• Demonstrate that CTR-GC can be integrated into a CTR-GCN backbone for superior performance.

Proposed method

• Model channel-wise topology as refining a shared adjacency A with channel-specific correlations Q to produce R.
• Use correlation modeling functions M1 or M2 to compute channel-wise relationships from reduced representations of joint features.
• Refine A into R via R = A + alpha * Q and perform channel-wise aggregation on per-channel graphs to produce Z.
• Unify four graph convolution styles into a common formulation to argue CTR-GC relaxes certain constraints and enhances representational power.
• Construct CTR-GCN with CTR-GC blocks plus a temporal modeling module and residual connections for skeleton sequence action recognition.

Figure 1: Channel-wise topology refinement. Lines of different colors correspond to topologies in different channels and the thickness of lines indicates the correlation strength between joints.

Experimental results

Research questions

• RQ1 Can channel-wise refinements of a shared topology improve the expressiveness of graph convolutions for skeleton-based action recognition?
• RQ2 Does incorporating channel-specific correlations (Q) alongside a shared topology (A) yield measurable gains over static/sharable topologies?
• RQ3 How does CTR-GCN perform compared with state-of-the-art methods on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets?

Key findings

• CTR-GC significantly outperforms other graph convolutions with comparable parameters and computation cost.
• CTR-GCN exceeds state-of-the-art methods on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets.
• Ablations show that removing channel-specific correlations (Q) or the shared topology refinement (A) reduces performance, highlighting the importance of channel-wise topology refinement.
• Various configurations of the correlation modeling functions (M1, M2) and reduction rate r consistently outperform the baseline, demonstrating robustness and effectiveness of CTR-GC.

Figure 2: Framework of the proposed channel-wise topology refinement graph convolution. The channel-wise topology modeling refines the trainable shared topology with inferred channel-specific correlations. The feature transformation aims at transforming input features into high-level representations